mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-24 23:10:41 +02:00
Chris Lu
aeb5db1b13
* tests: pin weed mini admin ports and disable unused aux services Every weed mini that does not configure them still binds the fixed default ports for the admin server (12535 + derived 22535 gRPC), its legacy compat listener (23646 + 33646), WebDAV (7333), Lance (9101) and the Iceberg REST catalog (8181). Sequential or concurrent minis on one host then collide on them: the port check warns and shifts, and the admin's late bind can land on a stolen ephemeral port, adding noise and flakiness to test logs (e.g. run 40017181958, job 120659267479). Add testutil.MiniAdminArgs to pin the admin server to an allocated mini-style port (HTTP + port+10000 gRPC, both below the Linux ephemeral port floor since admin binds late) and disable the legacy compat listener. Every test that spawns weed mini now also disables the aux services it does not use: WebDAV, Lance and the Iceberg REST endpoint get port=0 unless the test actually exercises them, in which case their ports stay pinned as before. * tests: take mini admin ports from the below-ephemeral range * tests: reuse MiniAdminArgs for the remote-s3 fetch mini * tests: fully disable webdav and co-allocate mini admin ports - -webdav.port=0 only makes the service bind a random port; add -webdav=false so minis that do not test WebDAV skip it entirely. - Spark, RisingWave and s3 Spark launchers allocated the admin port in a second MustFreeMiniPort call, which could pick a port already handed out by the first batch. Fold 'Admin' into the same MustFreeMiniPorts call. - test/fuse_integration is a separate Go module and cannot import testutil; allocate the admin ports with its local freePort helper instead. * test/fuse: derive the mini admin gRPC port from its HTTP port Two independent freePort calls could return the same port, making admin.port equal admin.port.grpc (or collide with the filer port). Allocate once and derive the gRPC port with the +10000 convention. * test/fuse: pin the mini admin port in the fuse_test framework copy The fuse_test package framework spawns weed mini too; without a pinned admin port it still binds the fixed defaults 12535/23646 and collides with other minis. * tests: drop the redundant -webdav.port=0 flag -webdav=false alone disables the service; the port flag is dead config.
Dremio Iceberg Catalog Integration Test
This directory contains a Dremio integration smoke test for SeaweedFS's Iceberg REST Catalog implementation.
What It Tests
TestDremioIcebergCatalog verifies the Dremio path end to end:
- Starts a local SeaweedFS mini cluster with S3 Tables and Iceberg REST enabled.
- Creates a SeaweedFS table bucket.
- Creates an Iceberg namespace and empty table through the SeaweedFS REST catalog OAuth flow.
- Starts
dremio/dremio-oss:25.2.0. - Bootstraps a Dremio admin user and logs in.
- Creates a Dremio
RESTCATALOGsource that points at the SeaweedFS catalog. - Submits Dremio SQL through
/api/v3/sql, polls the job API, and reads job results. - Runs subtests against the SeaweedFS-backed Iceberg table:
BasicSelect: Dremio is alive and answering SQL.CountEmptyTable: catalog-to-table resolution and a scan of an empty table.ColumnProjection:SELECT id, labelsucceeds and the response schema reports both columns. Failure here means Dremio could not parse the schema returned by the SeaweedFS catalog.InformationSchemaColumns: the table's columns are exposed through Dremio's metadata layer with the expected ordinal order.InformationSchemaTables: the table is registered in Dremio'sINFORMATION_SCHEMA.MultiLevelNamespace: a 2-level Iceberg namespace (created via the REST API) is exposed by Dremio as nested folders, and a table inside it is queryable with dot-separated identifiers.ReadWrittenDataCountandReadWrittenDataValues: a separate table is populated with three rows by a PyIceberg writer container (Dockerfile.writer+append_rows.py) before Dremio bootstraps; Dremio reads the data back and the values are verified. This exercises the actual data path, not just metadata.
The PyIceberg writer image is built on demand via Docker layer caching. The first build pulls python:3.11-slim and pip-installs PyIceberg + PyArrow (~1-2 min in CI); subsequent invocations are cheap.
Running Locally
Build or install weed, then run:
cd test/s3tables/catalog_dremio
go test -v -timeout 20m .
The test requires Docker. The GitHub Actions job runs on ubuntu-22.04 and executes the test for pull requests.
Configuration
The test uses these fixed credentials for the local SeaweedFS IAM config:
- S3 access key:
AKIAIOSFODNN7EXAMPLE - S3 secret key:
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY - Region:
us-west-2 - Warehouse bucket:
iceberg-tables
The Dremio source is configured via POST /api/v3/catalog; it is not configured in dremio.conf.
The Dremio container starts with the plugins.restcatalog.enabled support key enabled, which is required for the Iceberg REST Catalog source in Dremio OSS 25.2.
Troubleshooting
- Ensure Docker is running:
docker version - Ensure
weedis built or available onPATH - Check host-gateway routing if Dremio cannot reach SeaweedFS:
docker run --add-host host.docker.internal:host-gateway --rm alpine getent hosts host.docker.internal - Check Dremio logs from the failed test output; the harness prints the Dremio container tail on Dremio startup, source setup, or job failures.