mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-11 17:10:40 +02:00

Files

T

Chris Lu 34bb444f33 test: drive the Lance namespace with Spark (#10864 )

* test: drive the Lance namespace with Spark

The counterpart of catalog_spark, which does this for the Iceberg REST
catalog. Spark is the engine most likely to be pointed at a lakehouse,
and it reaches the Lance catalog through the connector's DSV2 catalog -
org.lance.spark.LanceNamespaceSparkCatalog with impl=rest - over the same
routes every other client uses.

    SHOW NAMESPACES -> ['`sparklance-lcephd80`.ml']
    SHOW TABLES -> ['sparklance-lcephd80$ml$embeddings']
    count -> 3
    filtered -> [(2, 'two'), (3, 'three')]
    count after a second commit -> 4

The second insert is there on purpose: a store that cannot order commits
fails on the second one, not the first.

Two things the run settled that were guesses beforehand. CREATE TABLE
works, because the connector declares through the namespace and writes the
data itself rather than pushing Arrow at the server. And SHOW TABLES
returns the namespace's own identifiers - bucket, namespace and name
joined by the delimiter - not bare Spark table names.

Credentials go under the catalog's storage.* prefix, which is handed to
lance as object_store options; a gateway without STS vends none, the same
trap the LanceDB suite documents.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* test: verify the Lance table bucket was actually created

weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame Spark for a bucket that was never made.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* test: bound the Docker probe

An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* test: keep the ivy cache under the user's cache directory

It is mounted into a container running as root, so a shared temp path lets
another local user pre-create it and choose what Spark loads.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* test: assert the vector column's type, not only its name

A column that came back as array<double> or array<string> would still be
called vector and still pass.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* test: read the dataset off its location for real

The catalog being optional is the property that lets duckdb and pandas read
these tables; it was asserted in a comment and printed, never exercised.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* test: do not persist credentials in the Spark Lance checkout

The job only uploads a log on failure; nothing in it pushes.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* test: say the hosts in the README are placeholders

The suite passes dynamically allocated host.docker.internal ports.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

2026-08-21 23:49:09 -07:00

README.md

test: drive the Lance namespace with Spark (#10864 )

2026-08-21 23:49:09 -07:00

spark_lance_ops.py

test: drive the Lance namespace with Spark (#10864 )

2026-08-21 23:49:09 -07:00

spark_lance_test.go

test: drive the Lance namespace with Spark (#10864 )

2026-08-21 23:49:09 -07:00

README.md

Spark Lance Integration Test

Drives the SeaweedFS Lance Namespace with Spark, through the Lance Spark connector's DSV2 catalog. The Lance counterpart of catalog_spark, which does the same for the Iceberg REST catalog.

Why Spark

Spark is the engine most likely to be pointed at a lakehouse, and it reaches the catalog over the same Lance Namespace routes as every other client. Between this and catalog_lancedb, the catalog is exercised by the two clients people actually use, rather than only by the protocol's reference implementation.

What it does

TestSparkLanceNamespace:

Starts a weed mini cluster with S3 and the Lance Namespace enabled.
Creates a table bucket declared LANCE.
Runs spark_lance_ops.py inside the stock apache/spark:3.5.1 image, with the connector pulled from Maven at submit time.

Inside Spark:

Step	What it proves
`CREATE NAMESPACE` / `SHOW NAMESPACES`	the bucket is the first level of the path, and the catalog is writable
`CREATE TABLE ... USING lance`	the connector declares through the namespace and writes the data itself
`INSERT` / `SELECT count(*)`	a write lands and reads back through the catalog
schema check	the vector column survived the round trip
`WHERE id >= 2`	the filter path, not only a full scan
a second `INSERT`	the commit a store that cannot order commits fails on

Configuration

The connector's catalog properties. The host names below are placeholders — the test passes dynamically allocated host.docker.internal ports, and a real deployment uses whatever address the gateway answers on:

spark.sql.catalog.lance                        org.lance.spark.LanceNamespaceSparkCatalog
spark.sql.catalog.lance.impl                   rest
spark.sql.catalog.lance.uri                    http://seaweed:9101
spark.sql.catalog.lance.storage.aws_endpoint   http://seaweed:8333
spark.sql.catalog.lance.storage.allow_http     true
spark.sql.catalog.lance.storage.aws_access_key_id      …
spark.sql.catalog.lance.storage.aws_secret_access_key  …

Anything under storage. is handed to lance as object_store options, so those are object_store's key names rather than Spark's s3a ones. Credentials belong there because a gateway without STS vends none — see the note in ../catalog_lancedb/README.md.

Running it

cd test/s3tables/catalog_spark_lance
(cd ../../../weed && go build .)     # the harness runs this binary
go test -run TestSparkLanceNamespace -v -timeout 40m .

Skipped without Docker, and in -short mode. The first run downloads the Spark image and the connector bundle, which is a few hundred megabytes; later runs reuse both.