Chris Lu
49a7d4ff6a
Three from review. The client's dependencies were unpinned, so an unrelated upstream release could change what an old commit reproduces. Pinned to the versions this suite was verified against; the client is as much the thing under test as the server. The search was called ANN and was not: without an index LanceDB scans. The test now builds an IVF_PQ index over 1024 rows first, which is worth more than the wording fix - an index writes into a directory of the table that the S3 door has to admit, and that guard has refused a Lance directory before. It builds, covers all 1024 rows, and searches. The assertion moved with it. Demanding the exact nearest neighbour was right for a brute-force scan and wrong for a quantized index, which answered 0 as readily as 1; both are correct, so the check is now the neighbourhood. And the pushdown check accepted any failure. It now requires the refusal to be the catalog's Unsupported and requires that nothing was left behind, or, when the client falls back, that the table is complete. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
LanceDB Integration Test
Drives the SeaweedFS Lance Namespace with LanceDB, the way
catalog_spark, catalog_trino and catalog_clickhouse drive the Iceberg REST
catalog with their engines.
Why a real client
Every serious bug in this catalog so far looked correct to a request written by hand: a deregister that deleted the dataset, an S3 door that refused every Lance file, a namespace that listed tables it would then deny. A hand-built HTTP test checks the shape of a response. A client checks whether the response is usable — that the location it hands back, the storage options beside it and the layout rules on the S3 door all line up at once.
LanceDB connects with connect_namespace("rest", ...), which speaks the routes
this catalog implements, so what runs here is the protocol rather than our own
idea of it.
What it does
TestLanceDBNamespace:
- Starts a
weed minicluster with S3 and the Lance Namespace enabled. - Creates a table bucket declared
LANCE, so the catalog refuses tables of any other format in it. - Builds
Dockerfile.client(LanceDB, pylance, lance-namespace) and runslancedb_ops.pyagainst the namespace.
Inside the container:
| Step | What it proves |
|---|---|
| seed a table | the namespace's location and credentials are enough to write |
table_names |
the catalog is browsable through LanceDB |
open_table |
LanceDB resolves a table through the catalog and reads it |
| schema check | the vector column survived the round trip |
create_index |
an IVF_PQ index builds, and its files land through the S3 door's layout rules |
search(...) |
approximate search over that index, on data behind SeaweedFS |
where("id < 5") |
so does the scan path, not only the index |
create_table |
LanceDB declares through the namespace and writes the data itself |
create_table with pushdown |
either the client falls back and the table is complete, or the catalog refuses with the spec's Unsupported and leaves nothing behind |
direct lance.dataset(uri) |
the catalog stays optional; the dataset opens without it |
The seeding is pylance rather than LanceDB, because the namespace records where a table lives and does not carry its data. That split is the design, not a limitation of the test.
Running it
cd test/s3tables/catalog_lancedb
(cd ../../../weed && go build .) # the harness runs this binary
go test -run TestLanceDBNamespace -v -timeout 30m .
Skipped without Docker, and in -short mode. The first run builds the client
image, which takes a few minutes; later runs reuse it.