mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-31 01:10:41 +01:00
Add DuckDB Iceberg Integration wiki page
Document how to connect DuckDB to SeaweedFS Iceberg REST Catalog using OAuth2 authentication. Examples match the patterns validated in the integration test suite (TestDuckDBOAuthIntegration). Also add a cross-link from the main Iceberg Catalog page.
1 parent
1e6c3aed2c
commit
8537b25781
2 files changed
+164
No files matched your search
@@ -0,0 +1,158 @@
|
||||
# DuckDB Iceberg Integration
|
||||
|
||||
DuckDB can query Iceberg tables stored in SeaweedFS using the [Iceberg extension](https://duckdb.org/docs/extensions/iceberg.html) and the built-in Iceberg REST Catalog.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- SeaweedFS running with the Iceberg REST Catalog enabled (port `8181` by default)
|
||||
- A table bucket created via `weed shell` or the S3 Tables API
|
||||
- DuckDB v1.1.0+ with the Iceberg extension
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Start SeaweedFS
|
||||
|
||||
```bash
|
||||
weed mini
|
||||
```
|
||||
|
||||
This starts SeaweedFS with the Iceberg REST Catalog on `http://localhost:8181`.
|
||||
|
||||
### 2. Create a Table Bucket
|
||||
|
||||
Using `weed shell`:
|
||||
|
||||
```bash
|
||||
weed shell
|
||||
> s3tables.bucket -create -name my-catalog
|
||||
```
|
||||
|
||||
### 3. Connect DuckDB
|
||||
|
||||
```sql
|
||||
INSTALL iceberg;
|
||||
LOAD iceberg;
|
||||
|
||||
-- Create an Iceberg catalog secret (connects to the REST Catalog)
|
||||
CREATE SECRET iceberg_secret (
|
||||
TYPE ICEBERG,
|
||||
ENDPOINT 'http://localhost:8181',
|
||||
CLIENT_ID 'your-access-key',
|
||||
CLIENT_SECRET 'your-secret-key'
|
||||
);
|
||||
|
||||
-- Create an S3 secret (for reading data files)
|
||||
CREATE SECRET s3_secret (
|
||||
TYPE S3,
|
||||
KEY_ID 'your-access-key',
|
||||
SECRET 'your-secret-key',
|
||||
ENDPOINT 'localhost:8333',
|
||||
URL_STYLE 'path',
|
||||
USE_SSL false
|
||||
);
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
SeaweedFS supports OAuth2 `client_credentials` flow for the Iceberg REST Catalog, which is what DuckDB uses. Your S3 access key and secret key are used as the `CLIENT_ID` and `CLIENT_SECRET`.
|
||||
|
||||
When DuckDB creates an Iceberg secret, it automatically:
|
||||
1. Posts to `POST /v1/oauth/tokens` with your credentials
|
||||
2. Receives a bearer token
|
||||
3. Uses the bearer token for all subsequent catalog requests
|
||||
|
||||
### With IAM Credentials
|
||||
|
||||
If SeaweedFS is configured with IAM (`-s3.config`), you must provide valid credentials:
|
||||
|
||||
```sql
|
||||
CREATE SECRET (
|
||||
TYPE ICEBERG,
|
||||
ENDPOINT 'http://localhost:8181',
|
||||
CLIENT_ID 'AKIAIOSFODNN7EXAMPLE',
|
||||
CLIENT_SECRET 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'
|
||||
);
|
||||
```
|
||||
|
||||
### Anonymous Access (Development)
|
||||
|
||||
When SeaweedFS runs without IAM configuration (e.g., `weed mini` with no `-s3.config`), you can connect without credentials. DuckDB still requires the `CLIENT_ID` and `CLIENT_SECRET` fields, but any non-empty values will work:
|
||||
|
||||
```sql
|
||||
CREATE SECRET (
|
||||
TYPE ICEBERG,
|
||||
ENDPOINT 'http://localhost:8181',
|
||||
CLIENT_ID 'admin',
|
||||
CLIENT_SECRET 'admin'
|
||||
);
|
||||
```
|
||||
|
||||
## Scoping Secrets to a Catalog
|
||||
|
||||
Use the `SCOPE` parameter to bind a secret to a specific table bucket:
|
||||
|
||||
```sql
|
||||
CREATE SECRET iceberg_secret (
|
||||
TYPE ICEBERG,
|
||||
ENDPOINT 'http://localhost:8181',
|
||||
CLIENT_ID 'your-access-key',
|
||||
CLIENT_SECRET 'your-secret-key',
|
||||
SCOPE 's3://my-catalog/'
|
||||
);
|
||||
|
||||
CREATE SECRET s3_secret (
|
||||
TYPE S3,
|
||||
KEY_ID 'your-access-key',
|
||||
SECRET 'your-secret-key',
|
||||
ENDPOINT 'localhost:8333',
|
||||
URL_STYLE 'path',
|
||||
USE_SSL false,
|
||||
SCOPE 's3://my-catalog/'
|
||||
);
|
||||
```
|
||||
|
||||
## Querying Tables
|
||||
|
||||
Once secrets are configured, you can query Iceberg tables:
|
||||
|
||||
```sql
|
||||
-- Scan an Iceberg table by its S3 path
|
||||
SELECT * FROM iceberg_scan('s3://my-catalog/my-namespace/my-table');
|
||||
|
||||
-- With metadata path
|
||||
SELECT * FROM iceberg_scan('s3://my-catalog/my-namespace/my-table/metadata/v1.metadata.json');
|
||||
```
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
| Parameter | Description | Default |
|
||||
|-----------|-------------|---------|
|
||||
| Iceberg REST port | `--port.iceberg` (standalone) or `-s3.port.iceberg` (mini) | `8181` |
|
||||
| S3 port | `--port` (standalone) or `-s3.port` (mini) | `8333` |
|
||||
| Disable Iceberg | Set port to `0` | Enabled |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "HTTP NotFound_404" on `/v1/oauth/tokens`
|
||||
|
||||
Upgrade SeaweedFS. The OAuth2 token endpoint was added to support DuckDB's authentication flow.
|
||||
|
||||
### "access denied" when creating secrets
|
||||
|
||||
Ensure your `CLIENT_ID` and `CLIENT_SECRET` match a valid IAM identity configured in SeaweedFS. Check your `-s3.config` file.
|
||||
|
||||
### DuckDB can connect but cannot read data
|
||||
|
||||
Make sure you have an S3 secret configured in addition to the Iceberg secret. The Iceberg secret handles catalog operations (listing namespaces, table metadata), while the S3 secret is needed for reading the actual Parquet data files.
|
||||
|
||||
```sql
|
||||
-- Both secrets are needed:
|
||||
-- 1. ICEBERG secret -> talks to the catalog API on port 8181
|
||||
-- 2. S3 secret -> reads data files from S3 on port 8333
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [[SeaweedFS Iceberg Catalog]] - Architecture and Spark/Trino integration
|
||||
- [[S3 Table Bucket]] - Managing table buckets
|
||||
- [[S3 Tables Security]] - IAM policies for table access
|
||||
@@ -150,6 +150,12 @@ SeaweedFS treats the Table Bucket as the root for that catalog instance. You can
|
||||
|
||||
---
|
||||
|
||||
## DuckDB Integration
|
||||
|
||||
DuckDB connects via the Iceberg extension using OAuth2 authentication. See the dedicated [[DuckDB Iceberg Integration]] page for setup instructions.
|
||||
|
||||
---
|
||||
|
||||
## Anonymous Access (Zero Config)
|
||||
|
||||
If SeaweedFS is running in **Zero Configuration** mode (no `-s3.config` or `-iam.config` provided), the Iceberg Catalog allows anonymous access by default. This is useful for local development and testing.
|
||||
|
||||
Reference in new issue
Block a user