Chris Lu
3466e74391
The counterpart of catalog_spark, which does this for the Iceberg REST
catalog. Spark is the engine most likely to be pointed at a lakehouse,
and it reaches the Lance catalog through the connector's DSV2 catalog -
org.lance.spark.LanceNamespaceSparkCatalog with impl=rest - over the same
routes every other client uses.
SHOW NAMESPACES -> ['`sparklance-lcephd80`.ml']
SHOW TABLES -> ['sparklance-lcephd80$ml$embeddings']
count -> 3
filtered -> [(2, 'two'), (3, 'three')]
count after a second commit -> 4
The second insert is there on purpose: a store that cannot order commits
fails on the second one, not the first.
Two things the run settled that were guesses beforehand. CREATE TABLE
works, because the connector declares through the namespace and writes the
data itself rather than pushing Arrow at the server. And SHOW TABLES
returns the namespace's own identifiers - bucket, namespace and name
joined by the delimiter - not bare Spark table names.
Credentials go under the catalog's storage.* prefix, which is handed to
lance as object_store options; a gateway without STS vends none, the same
trap the LanceDB suite documents.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
Spark Lance Integration Test
Drives the SeaweedFS Lance Namespace with Spark, through the Lance Spark
connector's DSV2 catalog. The Lance counterpart of catalog_spark, which does
the same for the Iceberg REST catalog.
Why Spark
Spark is the engine most likely to be pointed at a lakehouse, and it reaches the
catalog over the same Lance Namespace routes as every other client. Between this
and catalog_lancedb, the catalog is exercised by the two clients people
actually use, rather than only by the protocol's reference implementation.
What it does
TestSparkLanceNamespace:
- Starts a
weed minicluster with S3 and the Lance Namespace enabled. - Creates a table bucket declared
LANCE. - Runs
spark_lance_ops.pyinside the stockapache/spark:3.5.1image, with the connector pulled from Maven at submit time.
Inside Spark:
| Step | What it proves |
|---|---|
CREATE NAMESPACE / SHOW NAMESPACES |
the bucket is the first level of the path, and the catalog is writable |
CREATE TABLE ... USING lance |
the connector declares through the namespace and writes the data itself |
INSERT / SELECT count(*) |
a write lands and reads back through the catalog |
| schema check | the vector column survived the round trip |
WHERE id >= 2 |
the filter path, not only a full scan |
a second INSERT |
the commit a store that cannot order commits fails on |
Configuration
The connector's catalog properties, for reference:
spark.sql.catalog.lance org.lance.spark.LanceNamespaceSparkCatalog
spark.sql.catalog.lance.impl rest
spark.sql.catalog.lance.uri http://seaweed:9101
spark.sql.catalog.lance.storage.aws_endpoint http://seaweed:8333
spark.sql.catalog.lance.storage.allow_http true
spark.sql.catalog.lance.storage.aws_access_key_id …
spark.sql.catalog.lance.storage.aws_secret_access_key …
Anything under storage. is handed to lance as object_store options, so those
are object_store's key names rather than Spark's s3a ones. Credentials belong
there because a gateway without STS vends none — see the note in
../catalog_lancedb/README.md.
Running it
cd test/s3tables/catalog_spark_lance
(cd ../../../weed && go build .) # the harness runs this binary
go test -run TestSparkLanceNamespace -v -timeout 40m .
Skipped without Docker, and in -short mode. The first run downloads the Spark
image and the connector bundle, which is a few hundred megabytes; later runs
reuse both.