Add DuckDB Lance Integration

DuckDB reads Lance tables out of a table bucket over S3 with no catalog
involved, which is the property that keeps the data readable by anything
that does not speak the namespace protocol.

Records the one place the layout costs us: DuckDB's replacement scan
recognises a dataset by a .lance path suffix, which tables created through
this catalog deliberately do not have, so __lance_scan is the way in. Also
notes that the documented ATTACH ... (TYPE lance) form fails in 1.5.5.
Chris Lu
2026-08-21 13:04:56 -07:00
parent 489ebb7dd4
commit 97785b128f
4 changed files with 79 additions and 0 deletions
+76
@@ -0,0 +1,76 @@
# DuckDB Lance Integration
DuckDB's [`lance` core extension](https://duckdb.org/docs/stable/core_extensions/lance.html)
reads Lance tables out of a SeaweedFS table bucket **over S3, without the
catalog**. That works because a table bucket's layout is a valid Lance dataset
directory — see [[SeaweedFS Lance Catalog]].
Verified against DuckDB 1.5.5 by the integration suite in
`test/s3tables/catalog_duckdb_lance/`.
## Setup
```sql
INSTALL lance;
LOAD lance;
CREATE SECRET seaweedfs (
TYPE lance,
ACCESS_KEY_ID '',
SECRET_ACCESS_KEY '',
REGION 'us-east-1',
ENDPOINT 'http://localhost:8333',
ALLOW_HTTP true,
VIRTUAL_HOSTED_STYLE_REQUEST false
);
```
Those are object_store's key names, the same ones the namespace vends in
`storage_options`.
## Reading a table
A table created through the catalog lives at `s3://<bucket>/<namespace>/<table>`:
```sql
SELECT count(*) FROM __lance_scan('s3://vectors/ml/embeddings');
SELECT id, title
FROM __lance_scan('s3://vectors/ml/embeddings')
WHERE id < 5;
SELECT id
FROM lance_vector_search('s3://vectors/ml/embeddings', 'vector',
[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0], k := 3);
```
## Why `__lance_scan` and not `FROM 's3://…'`
DuckDB's replacement scan — the ergonomic `SELECT * FROM 's3://…'` form —
recognises a Lance dataset **by a `.lance` path suffix**. Tables created through
this catalog do not have one, deliberately: the catalog entry *is* the dataset
directory, a table name may not contain a dot, and a suffix would leak into ARNs
and bucket policies.
```sql
-- a dataset written to a .lance path: recognised
SELECT count(*) FROM 's3://vectors/ml/exported.lance';
-- a table created through the catalog: not recognised
SELECT count(*) FROM 's3://vectors/ml/embeddings'; -- Catalog Error
SELECT count(*) FROM __lance_scan('s3://vectors/ml/embeddings'); -- works
```
## Known limitation
`ATTACH 's3://bucket/namespace' AS ns (TYPE lance)` — the directory-namespace
form in the extension's own documentation — fails in DuckDB 1.5.5 with
`Cannot launch in-memory database in read-only mode`, whatever options are
passed. Use the scan functions above.
## See also
- [[SeaweedFS Lance Catalog]]
- [[LanceDB Integration]]
- [[Spark Lance Integration]]
- [[DuckDB Iceberg Integration]]
+1
@@ -108,4 +108,5 @@ lance.dataset("s3://vectors/ml/embeddings", storage_options=opts)
- [[SeaweedFS Lance Catalog]]
- [[Spark Lance Integration]]
- [[DuckDB Lance Integration]]
- [[Lance Maintenance Worker]] — compaction, index optimization, version cleanup
+1
@@ -118,6 +118,7 @@ Table buckets show the format they hold and the endpoint that serves them. For a
- [[LanceDB Integration]]
- [[Spark Lance Integration]]
- [[DuckDB Lance Integration]]
- [[Lance Maintenance Worker]]
- [[S3 Table Bucket]]
- [[SeaweedFS Iceberg Catalog]]
+1
@@ -127,6 +127,7 @@
### Lance Integrations
* [[LanceDB Integration]]
* [[Spark Lance Integration]]
* [[DuckDB Lance Integration]]
### S3 Authentication & IAM
* [[S3 Configuration]] - Start Here