Chris LuandDevin
d15396fc4d
* s3tables: scope the resolved anonymous identity instead of defaulting to allow Catalog authorization fell back to the open default whenever the request resolved to the configured anonymous identity, ignoring its grant. The fallback now applies only when no principal was resolved (zero-config), so anonymous requests are checked against the identity's configured actions like any other principal. The S3 Tables API and the Iceberg and Lance catalogs share this handler, so all three are covered. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: authenticate the catalog clients in the s3tables suites Anonymous calls into the Lance and Iceberg catalogs are now scoped by the anonymous identity's grant, so the suites that used to rely on the anonymous pass authenticate like their real deployments do: x-api-key for the Lance probes and REST drivers, client-credentials OAuth for pyiceberg. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * worker(lance): authenticate namespace calls with the storage credential Anonymous catalog calls are now scoped by the anonymous grant, so a worker that holds an S3 credential sends it as x-api-key on the namespace requests instead of relying on the anonymous pass. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: authenticate the pushdown catalog connection in catalog_lancedb Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: follow catalog redirects only within the same host reqwest forwards custom headers on a follow, so a redirect to another host would carry the x-api-key credential to a server it was never meant for. Keep same-host hops for proxies that normalize the path or upgrade the scheme. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: warn when the catalog api key travels in cleartext The x-api-key pair is a real storage credential: over plain http to a non-loopback host the network can read it. Warn at client build so the operator sees it; https and loopback stay quiet, and the key still sends since an http catalog is a valid deployment. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: require the same origin on catalog redirects A same-host hop can still downgrade https to http and put the api key on the wire; matching scheme, host, and port keeps the credential on the listener it was issued for. * lance: bound catalog redirect hops so loops fail fast The custom redirect policy drops reqwest's default 10-hop limit, so a catalog that redirects back to itself would hold each call open until the request timeout. Keep the bound alongside the origin check. * lance: gofmt the redirect helpers --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
DuckDB Lance Integration Test
Reads SeaweedFS Lance tables from DuckDB's lance core extension, the
counterpart of the DuckDB Iceberg tests in ../catalog/.
What this one proves that the others do not
The LanceDB and Spark suites go through the catalog. DuckDB does not: it reaches the data over S3 with no namespace involved. That exercises the other half of the design — a table bucket's layout is a valid Lance dataset directory, so a table stays readable when the catalog is not in the path.
What it does
TestDuckDBLance:
- Starts a
weed minicluster with S3 and the Lance Namespace enabled. - Creates a table bucket declared
LANCE, then declares a table through the namespace and writes a dataset into it with pylance. - Runs
duckdb_lance_ops.sqlinduckdb/duckdb:latest.
| Step | What it proves |
|---|---|
__lance_scan(s3://…) |
DuckDB reads a table this catalog created |
DESCRIBE |
the schema survived, vector column included |
WHERE id < 5 |
the filter path |
lance_vector_search |
vector search over data behind SeaweedFS |
a .lance path |
the replacement scan works on a suffixed path |
| a suffix-less path | and does not see one without the suffix |
The .lance suffix
DuckDB's replacement scan — SELECT * FROM 's3://…' — recognises a Lance dataset
by a .lance path suffix. Tables created through this catalog deliberately have
none: the catalog entry is the dataset directory, table names may not contain a
dot, and a suffix would leak into ARNs and policies.
So from DuckDB, a table in a SeaweedFS table bucket is read with
__lance_scan('s3://bucket/namespace/table') rather than the bare FROM 's3://…'
form. The test asserts both halves, so if the extension ever recognises a
suffix-less directory the test fails and tells us to update the documentation.
Credentials
CREATE SECRET seaweedfs (
TYPE lance,
ACCESS_KEY_ID '…', SECRET_ACCESS_KEY '…',
REGION 'us-east-1',
ENDPOINT 'http://seaweed:8333',
ALLOW_HTTP true,
VIRTUAL_HOSTED_STYLE_REQUEST false
);
Those are object_store's key names, the same as everywhere else Lance touches storage.
Running it
cd test/s3tables/catalog_duckdb_lance
(cd ../../../weed && go build .) # the harness runs this binary
go test -run TestDuckDBLance -v -timeout 30m .
Skipped without Docker, in -short mode, and if the DuckDB image cannot load the
extension.