mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-26 23:10:44 +01:00
Chris LuandDevin
d15396fc4d
* s3tables: scope the resolved anonymous identity instead of defaulting to allow Catalog authorization fell back to the open default whenever the request resolved to the configured anonymous identity, ignoring its grant. The fallback now applies only when no principal was resolved (zero-config), so anonymous requests are checked against the identity's configured actions like any other principal. The S3 Tables API and the Iceberg and Lance catalogs share this handler, so all three are covered. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: authenticate the catalog clients in the s3tables suites Anonymous calls into the Lance and Iceberg catalogs are now scoped by the anonymous identity's grant, so the suites that used to rely on the anonymous pass authenticate like their real deployments do: x-api-key for the Lance probes and REST drivers, client-credentials OAuth for pyiceberg. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * worker(lance): authenticate namespace calls with the storage credential Anonymous catalog calls are now scoped by the anonymous grant, so a worker that holds an S3 credential sends it as x-api-key on the namespace requests instead of relying on the anonymous pass. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: authenticate the pushdown catalog connection in catalog_lancedb Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: follow catalog redirects only within the same host reqwest forwards custom headers on a follow, so a redirect to another host would carry the x-api-key credential to a server it was never meant for. Keep same-host hops for proxies that normalize the path or upgrade the scheme. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: warn when the catalog api key travels in cleartext The x-api-key pair is a real storage credential: over plain http to a non-loopback host the network can read it. Warn at client build so the operator sees it; https and loopback stay quiet, and the key still sends since an http catalog is a valid deployment. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: require the same origin on catalog redirects A same-host hop can still downgrade https to http and put the api key on the wire; matching scheme, host, and port keeps the credential on the listener it was issued for. * lance: bound catalog redirect hops so loops fail fast The custom redirect policy drops reqwest's default 10-hop limit, so a catalog that redirects back to itself would hold each call open until the request timeout. Keep the bound alongside the origin check. * lance: gofmt the redirect helpers --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Spark Lance Integration Test
Drives the SeaweedFS Lance Namespace with Spark, through the Lance Spark
connector's DSV2 catalog. The Lance counterpart of catalog_spark, which does
the same for the Iceberg REST catalog.
Why Spark
Spark is the engine most likely to be pointed at a lakehouse, and it reaches the
catalog over the same Lance Namespace routes as every other client. Between this
and catalog_lancedb, the catalog is exercised by the two clients people
actually use, rather than only by the protocol's reference implementation.
What it does
TestSparkLanceNamespace:
- Starts a
weed minicluster with S3 and the Lance Namespace enabled. - Creates a table bucket declared
LANCE. - Runs
spark_lance_ops.pyinside the stockapache/spark:3.5.1image, with the connector pulled from Maven at submit time.
Inside Spark:
| Step | What it proves |
|---|---|
CREATE NAMESPACE / SHOW NAMESPACES |
the bucket is the first level of the path, and the catalog is writable |
CREATE TABLE ... USING lance |
the connector declares through the namespace and writes the data itself |
INSERT / SELECT count(*) |
a write lands and reads back through the catalog |
| schema check | the vector column survived the round trip |
WHERE id >= 2 |
the filter path, not only a full scan |
a second INSERT |
the commit a store that cannot order commits fails on |
Configuration
The connector's catalog properties. The host names below are placeholders — the
test passes dynamically allocated host.docker.internal ports, and a real
deployment uses whatever address the gateway answers on:
spark.sql.catalog.lance org.lance.spark.LanceNamespaceSparkCatalog
spark.sql.catalog.lance.impl rest
spark.sql.catalog.lance.uri http://seaweed:9101
spark.sql.catalog.lance.headers.x-api-key <access-key>:<secret-key>
spark.sql.catalog.lance.storage.aws_endpoint http://seaweed:8333
spark.sql.catalog.lance.storage.allow_http true
spark.sql.catalog.lance.storage.aws_access_key_id …
spark.sql.catalog.lance.storage.aws_secret_access_key …
Anything under storage. is handed to lance as object_store options, so those
are object_store's key names rather than Spark's s3a ones. Credentials belong
there because a gateway without STS vends none — see the note in
../catalog_lancedb/README.md.
Running it
cd test/s3tables/catalog_spark_lance
(cd ../../../weed && go build .) # the harness runs this binary
go test -run TestSparkLanceNamespace -v -timeout 40m .
Skipped without Docker, and in -short mode. The first run downloads the Spark
image and the connector bundle, which is a few hundred megabytes; later runs
reuse both.