diff --git a/DuckDB-Lance-Integration.md b/DuckDB-Lance-Integration.md new file mode 100644 index 0000000..0a8e74c --- /dev/null +++ b/DuckDB-Lance-Integration.md @@ -0,0 +1,76 @@ +# DuckDB Lance Integration + +DuckDB's [`lance` core extension](https://duckdb.org/docs/stable/core_extensions/lance.html) +reads Lance tables out of a SeaweedFS table bucket **over S3, without the +catalog**. That works because a table bucket's layout is a valid Lance dataset +directory — see [[SeaweedFS Lance Catalog]]. + +Verified against DuckDB 1.5.5 by the integration suite in +`test/s3tables/catalog_duckdb_lance/`. + +## Setup + +```sql +INSTALL lance; +LOAD lance; + +CREATE SECRET seaweedfs ( + TYPE lance, + ACCESS_KEY_ID '…', + SECRET_ACCESS_KEY '…', + REGION 'us-east-1', + ENDPOINT 'http://localhost:8333', + ALLOW_HTTP true, + VIRTUAL_HOSTED_STYLE_REQUEST false +); +``` + +Those are object_store's key names, the same ones the namespace vends in +`storage_options`. + +## Reading a table + +A table created through the catalog lives at `s3:////`: + +```sql +SELECT count(*) FROM __lance_scan('s3://vectors/ml/embeddings'); + +SELECT id, title +FROM __lance_scan('s3://vectors/ml/embeddings') +WHERE id < 5; + +SELECT id +FROM lance_vector_search('s3://vectors/ml/embeddings', 'vector', + [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0], k := 3); +``` + +## Why `__lance_scan` and not `FROM 's3://…'` + +DuckDB's replacement scan — the ergonomic `SELECT * FROM 's3://…'` form — +recognises a Lance dataset **by a `.lance` path suffix**. Tables created through +this catalog do not have one, deliberately: the catalog entry *is* the dataset +directory, a table name may not contain a dot, and a suffix would leak into ARNs +and bucket policies. + +```sql +-- a dataset written to a .lance path: recognised +SELECT count(*) FROM 's3://vectors/ml/exported.lance'; + +-- a table created through the catalog: not recognised +SELECT count(*) FROM 's3://vectors/ml/embeddings'; -- Catalog Error +SELECT count(*) FROM __lance_scan('s3://vectors/ml/embeddings'); -- works +``` + +## Known limitation + +`ATTACH 's3://bucket/namespace' AS ns (TYPE lance)` — the directory-namespace +form in the extension's own documentation — fails in DuckDB 1.5.5 with +`Cannot launch in-memory database in read-only mode`, whatever options are +passed. Use the scan functions above. + +## See also + +- [[SeaweedFS Lance Catalog]] +- [[LanceDB Integration]] +- [[Spark Lance Integration]] +- [[DuckDB Iceberg Integration]] diff --git a/LanceDB-Integration.md b/LanceDB-Integration.md index 8b829e3..99e8c91 100644 --- a/LanceDB-Integration.md +++ b/LanceDB-Integration.md @@ -108,4 +108,5 @@ lance.dataset("s3://vectors/ml/embeddings", storage_options=opts) - [[SeaweedFS Lance Catalog]] - [[Spark Lance Integration]] +- [[DuckDB Lance Integration]] - [[Lance Maintenance Worker]] — compaction, index optimization, version cleanup diff --git a/SeaweedFS-Lance-Catalog.md b/SeaweedFS-Lance-Catalog.md index 4c0730d..b506c15 100644 --- a/SeaweedFS-Lance-Catalog.md +++ b/SeaweedFS-Lance-Catalog.md @@ -118,6 +118,7 @@ Table buckets show the format they hold and the endpoint that serves them. For a - [[LanceDB Integration]] - [[Spark Lance Integration]] +- [[DuckDB Lance Integration]] - [[Lance Maintenance Worker]] - [[S3 Table Bucket]] - [[SeaweedFS Iceberg Catalog]] diff --git a/_Sidebar.md b/_Sidebar.md index 1f3dfcb..426394b 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -127,6 +127,7 @@ ### Lance Integrations * [[LanceDB Integration]] * [[Spark Lance Integration]] +* [[DuckDB Lance Integration]] ### S3 Authentication & IAM * [[S3 Configuration]] - Start Here