Chris Lu
5e7ab43ddd
* test: read Lance tables from DuckDB
The LanceDB and Spark suites go through the catalog. DuckDB does not: its
lance extension reaches the data over S3 with no namespace involved, which
exercises the other half of the design - a table bucket's layout is a
valid Lance dataset directory, so a table stays readable when the catalog
is not in the path.
scan_rows=128
scan_columns=id,title,vector
filtered_rows=5
nearest=1,0,2
It also pins the one place the layout costs us. DuckDB's replacement scan
recognises a dataset by a .lance path suffix, and tables created through
this catalog deliberately have none: the catalog entry is the dataset
directory, a table name may not contain a dot, and a suffix would leak
into ARNs and policies. So __lance_scan is the way in, and the bare
SELECT ... FROM 's3://...' form does not see these tables.
The test asserts both halves - a suffixed path is read, a suffix-less one
is not - so if the extension ever recognises a bare directory, it fails
and says to update the documentation rather than leaving it wrong.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: require the catalog error from the suffix-less read
Any failure satisfied the old check - a missing extension, bad credentials,
an unreachable endpoint - so the assertion could pass without the
replacement scan ever classifying the path.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame DuckDB for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: order the aggregates the assertions read
string_agg over an unordered relation may return the names, and the vector
search's ids, in any order, so the expectations could fail on a run where
nothing changed.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the DuckDB Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
DuckDB Lance Integration Test
Reads SeaweedFS Lance tables from DuckDB's lance core extension, the
counterpart of the DuckDB Iceberg tests in ../catalog/.
What this one proves that the others do not
The LanceDB and Spark suites go through the catalog. DuckDB does not: it reaches the data over S3 with no namespace involved. That exercises the other half of the design — a table bucket's layout is a valid Lance dataset directory, so a table stays readable when the catalog is not in the path.
What it does
TestDuckDBLance:
- Starts a
weed minicluster with S3 and the Lance Namespace enabled. - Creates a table bucket declared
LANCE, then declares a table through the namespace and writes a dataset into it with pylance. - Runs
duckdb_lance_ops.sqlinduckdb/duckdb:latest.
| Step | What it proves |
|---|---|
__lance_scan(s3://…) |
DuckDB reads a table this catalog created |
DESCRIBE |
the schema survived, vector column included |
WHERE id < 5 |
the filter path |
lance_vector_search |
vector search over data behind SeaweedFS |
a .lance path |
the replacement scan works on a suffixed path |
| a suffix-less path | and does not see one without the suffix |
The .lance suffix
DuckDB's replacement scan — SELECT * FROM 's3://…' — recognises a Lance dataset
by a .lance path suffix. Tables created through this catalog deliberately have
none: the catalog entry is the dataset directory, table names may not contain a
dot, and a suffix would leak into ARNs and policies.
So from DuckDB, a table in a SeaweedFS table bucket is read with
__lance_scan('s3://bucket/namespace/table') rather than the bare FROM 's3://…'
form. The test asserts both halves, so if the extension ever recognises a
suffix-less directory the test fails and tells us to update the documentation.
Credentials
CREATE SECRET seaweedfs (
TYPE lance,
ACCESS_KEY_ID '…', SECRET_ACCESS_KEY '…',
REGION 'us-east-1',
ENDPOINT 'http://seaweed:8333',
ALLOW_HTTP true,
VIRTUAL_HOSTED_STYLE_REQUEST false
);
Those are object_store's key names, the same as everywhere else Lance touches storage.
Running it
cd test/s3tables/catalog_duckdb_lance
(cd ../../../weed && go build .) # the harness runs this binary
go test -run TestDuckDBLance -v -timeout 30m .
Skipped without Docker, in -short mode, and if the DuckDB image cannot load the
extension.