Chris Lu
aeb5db1b13
* tests: pin weed mini admin ports and disable unused aux services Every weed mini that does not configure them still binds the fixed default ports for the admin server (12535 + derived 22535 gRPC), its legacy compat listener (23646 + 33646), WebDAV (7333), Lance (9101) and the Iceberg REST catalog (8181). Sequential or concurrent minis on one host then collide on them: the port check warns and shifts, and the admin's late bind can land on a stolen ephemeral port, adding noise and flakiness to test logs (e.g. run 40017181958, job 120659267479). Add testutil.MiniAdminArgs to pin the admin server to an allocated mini-style port (HTTP + port+10000 gRPC, both below the Linux ephemeral port floor since admin binds late) and disable the legacy compat listener. Every test that spawns weed mini now also disables the aux services it does not use: WebDAV, Lance and the Iceberg REST endpoint get port=0 unless the test actually exercises them, in which case their ports stay pinned as before. * tests: take mini admin ports from the below-ephemeral range * tests: reuse MiniAdminArgs for the remote-s3 fetch mini * tests: fully disable webdav and co-allocate mini admin ports - -webdav.port=0 only makes the service bind a random port; add -webdav=false so minis that do not test WebDAV skip it entirely. - Spark, RisingWave and s3 Spark launchers allocated the admin port in a second MustFreeMiniPort call, which could pick a port already handed out by the first batch. Fold 'Admin' into the same MustFreeMiniPorts call. - test/fuse_integration is a separate Go module and cannot import testutil; allocate the admin ports with its local freePort helper instead. * test/fuse: derive the mini admin gRPC port from its HTTP port Two independent freePort calls could return the same port, making admin.port equal admin.port.grpc (or collide with the filer port). Allocate once and derive the gRPC port with the +10000 convention. * test/fuse: pin the mini admin port in the fuse_test framework copy The fuse_test package framework spawns weed mini too; without a pinned admin port it still binds the fixed defaults 12535/23646 and collides with other minis. * tests: drop the redundant -webdav.port=0 flag -webdav=false alone disables the service; the port flag is dead config.
Spark Lance Integration Test
Drives the SeaweedFS Lance Namespace with Spark, through the Lance Spark
connector's DSV2 catalog. The Lance counterpart of catalog_spark, which does
the same for the Iceberg REST catalog.
Why Spark
Spark is the engine most likely to be pointed at a lakehouse, and it reaches the
catalog over the same Lance Namespace routes as every other client. Between this
and catalog_lancedb, the catalog is exercised by the two clients people
actually use, rather than only by the protocol's reference implementation.
What it does
TestSparkLanceNamespace:
- Starts a
weed minicluster with S3 and the Lance Namespace enabled. - Creates a table bucket declared
LANCE. - Runs
spark_lance_ops.pyinside the stockapache/spark:3.5.1image, with the connector pulled from Maven at submit time.
Inside Spark:
| Step | What it proves |
|---|---|
CREATE NAMESPACE / SHOW NAMESPACES |
the bucket is the first level of the path, and the catalog is writable |
CREATE TABLE ... USING lance |
the connector declares through the namespace and writes the data itself |
INSERT / SELECT count(*) |
a write lands and reads back through the catalog |
| schema check | the vector column survived the round trip |
WHERE id >= 2 |
the filter path, not only a full scan |
a second INSERT |
the commit a store that cannot order commits fails on |
Configuration
The connector's catalog properties. The host names below are placeholders — the
test passes dynamically allocated host.docker.internal ports, and a real
deployment uses whatever address the gateway answers on:
spark.sql.catalog.lance org.lance.spark.LanceNamespaceSparkCatalog
spark.sql.catalog.lance.impl rest
spark.sql.catalog.lance.uri http://seaweed:9101
spark.sql.catalog.lance.storage.aws_endpoint http://seaweed:8333
spark.sql.catalog.lance.storage.allow_http true
spark.sql.catalog.lance.storage.aws_access_key_id …
spark.sql.catalog.lance.storage.aws_secret_access_key …
Anything under storage. is handed to lance as object_store options, so those
are object_store's key names rather than Spark's s3a ones. Credentials belong
there because a gateway without STS vends none — see the note in
../catalog_lancedb/README.md.
Running it
cd test/s3tables/catalog_spark_lance
(cd ../../../weed && go build .) # the harness runs this binary
go test -run TestSparkLanceNamespace -v -timeout 40m .
Skipped without Docker, and in -short mode. The first run downloads the Spark
image and the connector bundle, which is a few hundred megabytes; later runs
reuse both.