From 8537b257810bb187c4212b0d38cff8619ffcbf56 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Fri, 10 Apr 2026 11:20:37 -0700 Subject: [PATCH] Add DuckDB Iceberg Integration wiki page Document how to connect DuckDB to SeaweedFS Iceberg REST Catalog using OAuth2 authentication. Examples match the patterns validated in the integration test suite (TestDuckDBOAuthIntegration). Also add a cross-link from the main Iceberg Catalog page. --- DuckDB-Iceberg-Integration.md | 158 ++++++++++++++++++++++++++++++++++ SeaweedFS-Iceberg-Catalog.md | 6 ++ 2 files changed, 164 insertions(+) create mode 100644 DuckDB-Iceberg-Integration.md diff --git a/DuckDB-Iceberg-Integration.md b/DuckDB-Iceberg-Integration.md new file mode 100644 index 0000000..442f37a --- /dev/null +++ b/DuckDB-Iceberg-Integration.md @@ -0,0 +1,158 @@ +# DuckDB Iceberg Integration + +DuckDB can query Iceberg tables stored in SeaweedFS using the [Iceberg extension](https://duckdb.org/docs/extensions/iceberg.html) and the built-in Iceberg REST Catalog. + +## Prerequisites + +- SeaweedFS running with the Iceberg REST Catalog enabled (port `8181` by default) +- A table bucket created via `weed shell` or the S3 Tables API +- DuckDB v1.1.0+ with the Iceberg extension + +## Quick Start + +### 1. Start SeaweedFS + +```bash +weed mini +``` + +This starts SeaweedFS with the Iceberg REST Catalog on `http://localhost:8181`. + +### 2. Create a Table Bucket + +Using `weed shell`: + +```bash +weed shell +> s3tables.bucket -create -name my-catalog +``` + +### 3. Connect DuckDB + +```sql +INSTALL iceberg; +LOAD iceberg; + +-- Create an Iceberg catalog secret (connects to the REST Catalog) +CREATE SECRET iceberg_secret ( + TYPE ICEBERG, + ENDPOINT 'http://localhost:8181', + CLIENT_ID 'your-access-key', + CLIENT_SECRET 'your-secret-key' +); + +-- Create an S3 secret (for reading data files) +CREATE SECRET s3_secret ( + TYPE S3, + KEY_ID 'your-access-key', + SECRET 'your-secret-key', + ENDPOINT 'localhost:8333', + URL_STYLE 'path', + USE_SSL false +); +``` + +## Authentication + +SeaweedFS supports OAuth2 `client_credentials` flow for the Iceberg REST Catalog, which is what DuckDB uses. Your S3 access key and secret key are used as the `CLIENT_ID` and `CLIENT_SECRET`. + +When DuckDB creates an Iceberg secret, it automatically: +1. Posts to `POST /v1/oauth/tokens` with your credentials +2. Receives a bearer token +3. Uses the bearer token for all subsequent catalog requests + +### With IAM Credentials + +If SeaweedFS is configured with IAM (`-s3.config`), you must provide valid credentials: + +```sql +CREATE SECRET ( + TYPE ICEBERG, + ENDPOINT 'http://localhost:8181', + CLIENT_ID 'AKIAIOSFODNN7EXAMPLE', + CLIENT_SECRET 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY' +); +``` + +### Anonymous Access (Development) + +When SeaweedFS runs without IAM configuration (e.g., `weed mini` with no `-s3.config`), you can connect without credentials. DuckDB still requires the `CLIENT_ID` and `CLIENT_SECRET` fields, but any non-empty values will work: + +```sql +CREATE SECRET ( + TYPE ICEBERG, + ENDPOINT 'http://localhost:8181', + CLIENT_ID 'admin', + CLIENT_SECRET 'admin' +); +``` + +## Scoping Secrets to a Catalog + +Use the `SCOPE` parameter to bind a secret to a specific table bucket: + +```sql +CREATE SECRET iceberg_secret ( + TYPE ICEBERG, + ENDPOINT 'http://localhost:8181', + CLIENT_ID 'your-access-key', + CLIENT_SECRET 'your-secret-key', + SCOPE 's3://my-catalog/' +); + +CREATE SECRET s3_secret ( + TYPE S3, + KEY_ID 'your-access-key', + SECRET 'your-secret-key', + ENDPOINT 'localhost:8333', + URL_STYLE 'path', + USE_SSL false, + SCOPE 's3://my-catalog/' +); +``` + +## Querying Tables + +Once secrets are configured, you can query Iceberg tables: + +```sql +-- Scan an Iceberg table by its S3 path +SELECT * FROM iceberg_scan('s3://my-catalog/my-namespace/my-table'); + +-- With metadata path +SELECT * FROM iceberg_scan('s3://my-catalog/my-namespace/my-table/metadata/v1.metadata.json'); +``` + +## Configuration Reference + +| Parameter | Description | Default | +|-----------|-------------|---------| +| Iceberg REST port | `--port.iceberg` (standalone) or `-s3.port.iceberg` (mini) | `8181` | +| S3 port | `--port` (standalone) or `-s3.port` (mini) | `8333` | +| Disable Iceberg | Set port to `0` | Enabled | + +## Troubleshooting + +### "HTTP NotFound_404" on `/v1/oauth/tokens` + +Upgrade SeaweedFS. The OAuth2 token endpoint was added to support DuckDB's authentication flow. + +### "access denied" when creating secrets + +Ensure your `CLIENT_ID` and `CLIENT_SECRET` match a valid IAM identity configured in SeaweedFS. Check your `-s3.config` file. + +### DuckDB can connect but cannot read data + +Make sure you have an S3 secret configured in addition to the Iceberg secret. The Iceberg secret handles catalog operations (listing namespaces, table metadata), while the S3 secret is needed for reading the actual Parquet data files. + +```sql +-- Both secrets are needed: +-- 1. ICEBERG secret -> talks to the catalog API on port 8181 +-- 2. S3 secret -> reads data files from S3 on port 8333 +``` + +## See Also + +- [[SeaweedFS Iceberg Catalog]] - Architecture and Spark/Trino integration +- [[S3 Table Bucket]] - Managing table buckets +- [[S3 Tables Security]] - IAM policies for table access diff --git a/SeaweedFS-Iceberg-Catalog.md b/SeaweedFS-Iceberg-Catalog.md index 6827d4d..1e02395 100644 --- a/SeaweedFS-Iceberg-Catalog.md +++ b/SeaweedFS-Iceberg-Catalog.md @@ -150,6 +150,12 @@ SeaweedFS treats the Table Bucket as the root for that catalog instance. You can --- +## DuckDB Integration + +DuckDB connects via the Iceberg extension using OAuth2 authentication. See the dedicated [[DuckDB Iceberg Integration]] page for setup instructions. + +--- + ## Anonymous Access (Zero Config) If SeaweedFS is running in **Zero Configuration** mode (no `-s3.config` or `-iam.config` provided), the Iceberg Catalog allows anonymous access by default. This is useful for local development and testing.