mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-26 15:00:48 +01:00
Chris LuandDevin
d15396fc4d
* s3tables: scope the resolved anonymous identity instead of defaulting to allow Catalog authorization fell back to the open default whenever the request resolved to the configured anonymous identity, ignoring its grant. The fallback now applies only when no principal was resolved (zero-config), so anonymous requests are checked against the identity's configured actions like any other principal. The S3 Tables API and the Iceberg and Lance catalogs share this handler, so all three are covered. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: authenticate the catalog clients in the s3tables suites Anonymous calls into the Lance and Iceberg catalogs are now scoped by the anonymous identity's grant, so the suites that used to rely on the anonymous pass authenticate like their real deployments do: x-api-key for the Lance probes and REST drivers, client-credentials OAuth for pyiceberg. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * worker(lance): authenticate namespace calls with the storage credential Anonymous catalog calls are now scoped by the anonymous grant, so a worker that holds an S3 credential sends it as x-api-key on the namespace requests instead of relying on the anonymous pass. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: authenticate the pushdown catalog connection in catalog_lancedb Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: follow catalog redirects only within the same host reqwest forwards custom headers on a follow, so a redirect to another host would carry the x-api-key credential to a server it was never meant for. Keep same-host hops for proxies that normalize the path or upgrade the scheme. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: warn when the catalog api key travels in cleartext The x-api-key pair is a real storage credential: over plain http to a non-loopback host the network can read it. Warn at client build so the operator sees it; https and loopback stay quiet, and the key still sends since an http catalog is a valid deployment. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: require the same origin on catalog redirects A same-host hop can still downgrade https to http and put the api key on the wire; matching scheme, host, and port keeps the credential on the listener it was issued for. * lance: bound catalog redirect hops so loops fail fast The custom redirect policy drops reqwest's default 10-hop limit, so a catalog that redirects back to itself would hold each call open until the request timeout. Keep the bound alongside the origin check. * lance: gofmt the redirect helpers --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
LanceDB Integration Test
Drives the SeaweedFS Lance Namespace with LanceDB, the way
catalog_spark, catalog_trino and catalog_clickhouse drive the Iceberg REST
catalog with their engines.
Why a real client
Every serious bug in this catalog so far looked correct to a request written by hand: a deregister that deleted the dataset, an S3 door that refused every Lance file, a namespace that listed tables it would then deny. A hand-built HTTP test checks the shape of a response. A client checks whether the response is usable — that the location it hands back, the storage options beside it and the layout rules on the S3 door all line up at once.
LanceDB connects with connect_namespace("rest", ...), which speaks the routes
this catalog implements, so what runs here is the protocol rather than our own
idea of it.
What it does
TestLanceDBNamespace:
- Starts a
weed minicluster with S3 and the Lance Namespace enabled. - Creates a table bucket declared
LANCE, so the catalog refuses tables of any other format in it. - Builds
Dockerfile.client(LanceDB, pylance, lance-namespace) and runslancedb_ops.pyagainst the namespace.
Inside the container:
| Step | What it proves |
|---|---|
| seed a table | the namespace's location and credentials are enough to write |
table_names |
the catalog is browsable through LanceDB |
open_table |
LanceDB resolves a table through the catalog and reads it |
| schema check | the vector column survived the round trip |
create_index |
an IVF_PQ index builds, and its files land through the S3 door's layout rules |
search(...) |
approximate search over that index, on data behind SeaweedFS |
where("id < 5") |
so does the scan path, not only the index |
create_table |
LanceDB declares through the namespace and writes the data itself |
create_table with pushdown |
either the client falls back and the table is complete, or the catalog refuses with the spec's Unsupported and leaves nothing behind |
direct lance.dataset(uri) |
the catalog stays optional; the dataset opens without it |
The seeding is pylance rather than LanceDB, because the namespace records where a table lives and does not carry its data. That split is the design, not a limitation of the test.
Running it
cd test/s3tables/catalog_lancedb
(cd ../../../weed && go build .) # the harness runs this binary
go test -run TestLanceDBNamespace -v -timeout 30m .
Skipped without Docker, and in -short mode. The first run builds the client
image, which takes a few minutes; later runs reuse it.