Chris Lu
aeb5db1b13
* tests: pin weed mini admin ports and disable unused aux services Every weed mini that does not configure them still binds the fixed default ports for the admin server (12535 + derived 22535 gRPC), its legacy compat listener (23646 + 33646), WebDAV (7333), Lance (9101) and the Iceberg REST catalog (8181). Sequential or concurrent minis on one host then collide on them: the port check warns and shifts, and the admin's late bind can land on a stolen ephemeral port, adding noise and flakiness to test logs (e.g. run 40017181958, job 120659267479). Add testutil.MiniAdminArgs to pin the admin server to an allocated mini-style port (HTTP + port+10000 gRPC, both below the Linux ephemeral port floor since admin binds late) and disable the legacy compat listener. Every test that spawns weed mini now also disables the aux services it does not use: WebDAV, Lance and the Iceberg REST endpoint get port=0 unless the test actually exercises them, in which case their ports stay pinned as before. * tests: take mini admin ports from the below-ephemeral range * tests: reuse MiniAdminArgs for the remote-s3 fetch mini * tests: fully disable webdav and co-allocate mini admin ports - -webdav.port=0 only makes the service bind a random port; add -webdav=false so minis that do not test WebDAV skip it entirely. - Spark, RisingWave and s3 Spark launchers allocated the admin port in a second MustFreeMiniPort call, which could pick a port already handed out by the first batch. Fold 'Admin' into the same MustFreeMiniPorts call. - test/fuse_integration is a separate Go module and cannot import testutil; allocate the admin ports with its local freePort helper instead. * test/fuse: derive the mini admin gRPC port from its HTTP port Two independent freePort calls could return the same port, making admin.port equal admin.port.grpc (or collide with the filer port). Allocate once and derive the gRPC port with the +10000 convention. * test/fuse: pin the mini admin port in the fuse_test framework copy The fuse_test package framework spawns weed mini too; without a pinned admin port it still binds the fixed defaults 12535/23646 and collides with other minis. * tests: drop the redundant -webdav.port=0 flag -webdav=false alone disables the service; the port flag is dead config.
DuckDB Lance Integration Test
Reads SeaweedFS Lance tables from DuckDB's lance core extension, the
counterpart of the DuckDB Iceberg tests in ../catalog/.
What this one proves that the others do not
The LanceDB and Spark suites go through the catalog. DuckDB does not: it reaches the data over S3 with no namespace involved. That exercises the other half of the design — a table bucket's layout is a valid Lance dataset directory, so a table stays readable when the catalog is not in the path.
What it does
TestDuckDBLance:
- Starts a
weed minicluster with S3 and the Lance Namespace enabled. - Creates a table bucket declared
LANCE, then declares a table through the namespace and writes a dataset into it with pylance. - Runs
duckdb_lance_ops.sqlinduckdb/duckdb:latest.
| Step | What it proves |
|---|---|
__lance_scan(s3://…) |
DuckDB reads a table this catalog created |
DESCRIBE |
the schema survived, vector column included |
WHERE id < 5 |
the filter path |
lance_vector_search |
vector search over data behind SeaweedFS |
a .lance path |
the replacement scan works on a suffixed path |
| a suffix-less path | and does not see one without the suffix |
The .lance suffix
DuckDB's replacement scan — SELECT * FROM 's3://…' — recognises a Lance dataset
by a .lance path suffix. Tables created through this catalog deliberately have
none: the catalog entry is the dataset directory, table names may not contain a
dot, and a suffix would leak into ARNs and policies.
So from DuckDB, a table in a SeaweedFS table bucket is read with
__lance_scan('s3://bucket/namespace/table') rather than the bare FROM 's3://…'
form. The test asserts both halves, so if the extension ever recognises a
suffix-less directory the test fails and tells us to update the documentation.
Credentials
CREATE SECRET seaweedfs (
TYPE lance,
ACCESS_KEY_ID '…', SECRET_ACCESS_KEY '…',
REGION 'us-east-1',
ENDPOINT 'http://seaweed:8333',
ALLOW_HTTP true,
VIRTUAL_HOSTED_STYLE_REQUEST false
);
Those are object_store's key names, the same as everywhere else Lance touches storage.
Running it
cd test/s3tables/catalog_duckdb_lance
(cd ../../../weed && go build .) # the harness runs this binary
go test -run TestDuckDBLance -v -timeout 30m .
Skipped without Docker, in -short mode, and if the DuckDB image cannot load the
extension.