Chris Lu
aeb5db1b13
* tests: pin weed mini admin ports and disable unused aux services Every weed mini that does not configure them still binds the fixed default ports for the admin server (12535 + derived 22535 gRPC), its legacy compat listener (23646 + 33646), WebDAV (7333), Lance (9101) and the Iceberg REST catalog (8181). Sequential or concurrent minis on one host then collide on them: the port check warns and shifts, and the admin's late bind can land on a stolen ephemeral port, adding noise and flakiness to test logs (e.g. run 40017181958, job 120659267479). Add testutil.MiniAdminArgs to pin the admin server to an allocated mini-style port (HTTP + port+10000 gRPC, both below the Linux ephemeral port floor since admin binds late) and disable the legacy compat listener. Every test that spawns weed mini now also disables the aux services it does not use: WebDAV, Lance and the Iceberg REST endpoint get port=0 unless the test actually exercises them, in which case their ports stay pinned as before. * tests: take mini admin ports from the below-ephemeral range * tests: reuse MiniAdminArgs for the remote-s3 fetch mini * tests: fully disable webdav and co-allocate mini admin ports - -webdav.port=0 only makes the service bind a random port; add -webdav=false so minis that do not test WebDAV skip it entirely. - Spark, RisingWave and s3 Spark launchers allocated the admin port in a second MustFreeMiniPort call, which could pick a port already handed out by the first batch. Fold 'Admin' into the same MustFreeMiniPorts call. - test/fuse_integration is a separate Go module and cannot import testutil; allocate the admin ports with its local freePort helper instead. * test/fuse: derive the mini admin gRPC port from its HTTP port Two independent freePort calls could return the same port, making admin.port equal admin.port.grpc (or collide with the filer port). Allocate once and derive the gRPC port with the +10000 convention. * test/fuse: pin the mini admin port in the fuse_test framework copy The fuse_test package framework spawns weed mini too; without a pinned admin port it still binds the fixed defaults 12535/23646 and collides with other minis. * tests: drop the redundant -webdav.port=0 flag -webdav=false alone disables the service; the port flag is dead config.
LanceDB Integration Test
Drives the SeaweedFS Lance Namespace with LanceDB, the way
catalog_spark, catalog_trino and catalog_clickhouse drive the Iceberg REST
catalog with their engines.
Why a real client
Every serious bug in this catalog so far looked correct to a request written by hand: a deregister that deleted the dataset, an S3 door that refused every Lance file, a namespace that listed tables it would then deny. A hand-built HTTP test checks the shape of a response. A client checks whether the response is usable — that the location it hands back, the storage options beside it and the layout rules on the S3 door all line up at once.
LanceDB connects with connect_namespace("rest", ...), which speaks the routes
this catalog implements, so what runs here is the protocol rather than our own
idea of it.
What it does
TestLanceDBNamespace:
- Starts a
weed minicluster with S3 and the Lance Namespace enabled. - Creates a table bucket declared
LANCE, so the catalog refuses tables of any other format in it. - Builds
Dockerfile.client(LanceDB, pylance, lance-namespace) and runslancedb_ops.pyagainst the namespace.
Inside the container:
| Step | What it proves |
|---|---|
| seed a table | the namespace's location and credentials are enough to write |
table_names |
the catalog is browsable through LanceDB |
open_table |
LanceDB resolves a table through the catalog and reads it |
| schema check | the vector column survived the round trip |
create_index |
an IVF_PQ index builds, and its files land through the S3 door's layout rules |
search(...) |
approximate search over that index, on data behind SeaweedFS |
where("id < 5") |
so does the scan path, not only the index |
create_table |
LanceDB declares through the namespace and writes the data itself |
create_table with pushdown |
either the client falls back and the table is complete, or the catalog refuses with the spec's Unsupported and leaves nothing behind |
direct lance.dataset(uri) |
the catalog stays optional; the dataset opens without it |
The seeding is pylance rather than LanceDB, because the namespace records where a table lives and does not carry its data. That split is the design, not a limitation of the test.
Running it
cd test/s3tables/catalog_lancedb
(cd ../../../weed && go build .) # the harness runs this binary
go test -run TestLanceDBNamespace -v -timeout 30m .
Skipped without Docker, and in -short mode. The first run builds the client
image, which takes a few minutes; later runs reuse it.