diff --git a/Dremio-Iceberg-Integration.md b/Dremio-Iceberg-Integration.md index 7863448..0a70102 100644 --- a/Dremio-Iceberg-Integration.md +++ b/Dremio-Iceberg-Integration.md @@ -6,9 +6,8 @@ This page reflects the integration verified by the [Dremio OSS 25.2.0 catalog te ## Prerequisites -- SeaweedFS running with the Iceberg REST Catalog enabled (port `8181` by default) and the S3 API on `8333` -- A table bucket created via `weed shell` or the S3 Tables API - Dremio OSS 25.2.0 or later with the experimental REST catalog plugin enabled +- SeaweedFS started as shown in [Setup](#setup) below In Dremio OSS 25.2 the Iceberg REST catalog source is gated behind a support key. Pass it to the JVM via `DREMIO_JAVA_EXTRA_OPTS`: @@ -16,6 +15,20 @@ In Dremio OSS 25.2 the Iceberg REST catalog source is gated behind a support key DREMIO_JAVA_EXTRA_OPTS="-Ddremio.debug.sysopt.plugins.restcatalog.enabled=true" ``` +## Setup + +Start `weed mini` with credentials and a pre-created table bucket via environment variables: + +```bash +export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE +export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +export S3_TABLE_BUCKET=my-table-bucket + +weed mini -dir ~/data +``` + +This brings up the Iceberg REST Catalog on `http://localhost:8181`, the S3 endpoint on `http://localhost:8333`, an admin S3 identity using the AWS env vars (used as the Dremio source's S3 credentials below), and the table bucket `my-table-bucket` pre-created. + ## Configuration Iceberg REST catalog sources in Dremio are not configured through `dremio.conf`. After Dremio is up, register the source by `POST`ing to `/api/v3/catalog`: @@ -49,9 +62,9 @@ curl -X POST http://localhost:9047/api/v3/catalog \ {"name": "fs.s3a.create.file-status-check", "value": "false"} ], "secretPropertyList": [ - {"name": "fs.s3a.access.key", "value": "YOUR_ACCESS_KEY"}, - {"name": "fs.s3a.secret.key", "value": "YOUR_SECRET_KEY"}, - {"name": "credential", "value": "YOUR_ACCESS_KEY:YOUR_SECRET_KEY"} + {"name": "fs.s3a.access.key", "value": "AKIAIOSFODNN7EXAMPLE"}, + {"name": "fs.s3a.secret.key", "value": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"}, + {"name": "credential", "value": "AKIAIOSFODNN7EXAMPLE:wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"} ] } }' diff --git a/DuckDB-Iceberg-Integration.md b/DuckDB-Iceberg-Integration.md index 442f37a..5d15764 100644 --- a/DuckDB-Iceberg-Integration.md +++ b/DuckDB-Iceberg-Integration.md @@ -10,42 +10,46 @@ DuckDB can query Iceberg tables stored in SeaweedFS using the [Iceberg extension ## Quick Start -### 1. Start SeaweedFS +### 1. Start SeaweedFS with credentials and a table bucket + +`weed mini` is fully configurable via environment variables — credentials become an admin S3 identity, and `S3_TABLE_BUCKET` pre-creates the Iceberg table bucket. No IAM config file or `weed shell` step needed: ```bash -weed mini +export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE +export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +export S3_TABLE_BUCKET=my-catalog + +weed mini -dir ~/data ``` -This starts SeaweedFS with the Iceberg REST Catalog on `http://localhost:8181`. +This brings up: +- Iceberg REST Catalog on `http://localhost:8181` +- S3 endpoint on `http://localhost:8333` +- An admin S3 identity using the AWS env vars above (used as DuckDB's `CLIENT_ID` / `CLIENT_SECRET`) +- Table bucket `my-catalog` pre-created -### 2. Create a Table Bucket +`S3_TABLE_BUCKET` (or the `-tableBucket` flag) accepts a comma-separated list (`my-catalog,other-catalog`); existing buckets are left alone. `S3_BUCKET` / `-bucket` does the same for plain (non-table) buckets. -Using `weed shell`: - -```bash -weed shell -> s3tables.bucket -create -name my-catalog -``` - -### 3. Connect DuckDB +### 2. Connect DuckDB ```sql INSTALL iceberg; LOAD iceberg; --- Create an Iceberg catalog secret (connects to the REST Catalog) +-- Iceberg catalog secret (OAuth2 against the REST Catalog). +-- CLIENT_ID / CLIENT_SECRET must match the AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY exported above. CREATE SECRET iceberg_secret ( TYPE ICEBERG, ENDPOINT 'http://localhost:8181', - CLIENT_ID 'your-access-key', - CLIENT_SECRET 'your-secret-key' + CLIENT_ID 'AKIAIOSFODNN7EXAMPLE', + CLIENT_SECRET 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY' ); --- Create an S3 secret (for reading data files) +-- S3 secret for reading data files. Same credentials. CREATE SECRET s3_secret ( TYPE S3, - KEY_ID 'your-access-key', - SECRET 'your-secret-key', + KEY_ID 'AKIAIOSFODNN7EXAMPLE', + SECRET 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY', ENDPOINT 'localhost:8333', URL_STYLE 'path', USE_SSL false @@ -63,16 +67,7 @@ When DuckDB creates an Iceberg secret, it automatically: ### With IAM Credentials -If SeaweedFS is configured with IAM (`-s3.config`), you must provide valid credentials: - -```sql -CREATE SECRET ( - TYPE ICEBERG, - ENDPOINT 'http://localhost:8181', - CLIENT_ID 'AKIAIOSFODNN7EXAMPLE', - CLIENT_SECRET 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY' -); -``` +When `weed mini` is started with `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` exported (or with `-s3.config=iam.json` for richer multi-identity setups), the `CLIENT_ID` / `CLIENT_SECRET` passed to `CREATE SECRET` must match a registered access key / secret key pair. The catalog rejects unknown credentials with `401 invalid_client`. ### Anonymous Access (Development) diff --git a/RisingWave-Iceberg-Integration.md b/RisingWave-Iceberg-Integration.md index 33c5060..039ce0f 100644 --- a/RisingWave-Iceberg-Integration.md +++ b/RisingWave-Iceberg-Integration.md @@ -4,9 +4,22 @@ RisingWave can read from and write to SeaweedFS Iceberg tables using the `iceber ## Prerequisites -- SeaweedFS running with the Iceberg REST Catalog enabled (port `8181` by default) -- A table bucket created via `weed shell` or the S3 Tables API - RisingWave v2.5.0+ with Iceberg connector support +- SeaweedFS started as shown in [Setup](#setup) below + +## Setup + +Start `weed mini` with credentials and a pre-created table bucket via environment variables: + +```bash +export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE +export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +export S3_TABLE_BUCKET=my-table-bucket + +weed mini -dir ~/data +``` + +This brings up the Iceberg REST Catalog on `http://localhost:8181`, the S3 endpoint on `http://localhost:8333`, an admin S3 identity using the AWS env vars (used as RisingWave's `s3.access.key` / `s3.secret.key` below), and the table bucket `my-table-bucket` pre-created. ## Reading from Iceberg (Source) @@ -23,8 +36,8 @@ CREATE SOURCE my_source WITH ( warehouse.path = 's3://my-table-bucket', s3.endpoint = 'http://localhost:8333', s3.region = 'us-east-1', - s3.access.key = 'YOUR_ACCESS_KEY', - s3.secret.key = 'YOUR_SECRET_KEY', + s3.access.key = 'AKIAIOSFODNN7EXAMPLE', + s3.secret.key = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY', s3.path.style.access = 'true', catalog.rest.sigv4_enabled = 'true', catalog.rest.signing_region = 'us-east-1', @@ -60,8 +73,8 @@ WITH ( warehouse.path = 's3://my-table-bucket', s3.endpoint = 'http://localhost:8333', s3.region = 'us-east-1', - s3.access.key = 'YOUR_ACCESS_KEY', - s3.secret.key = 'YOUR_SECRET_KEY', + s3.access.key = 'AKIAIOSFODNN7EXAMPLE', + s3.secret.key = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY', s3.path.style.access = 'true', catalog.rest.sigv4_enabled = 'true', catalog.rest.signing_region = 'us-east-1', @@ -93,8 +106,8 @@ WITH ( warehouse.path = 's3://my-table-bucket', s3.endpoint = 'http://localhost:8333', s3.region = 'us-east-1', - s3.access.key = 'YOUR_ACCESS_KEY', - s3.secret.key = 'YOUR_SECRET_KEY', + s3.access.key = 'AKIAIOSFODNN7EXAMPLE', + s3.secret.key = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY', s3.path.style.access = 'true', catalog.rest.sigv4_enabled = 'true', catalog.rest.signing_region = 'us-east-1', diff --git a/Spark-Iceberg-Integration.md b/Spark-Iceberg-Integration.md index dfcdc9b..8f05db7 100644 --- a/Spark-Iceberg-Integration.md +++ b/Spark-Iceberg-Integration.md @@ -4,14 +4,27 @@ Apache Spark connects to SeaweedFS Iceberg tables using the Iceberg Spark runtim ## Prerequisites -- SeaweedFS running with the Iceberg REST Catalog enabled (port `8181` by default) -- A table bucket created via `weed shell` or the S3 Tables API - Spark 3.5+ with Iceberg packages +- SeaweedFS started as shown in [Setup](#setup) below Required packages: - `org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.2` (match your Spark version) - `org.apache.iceberg:iceberg-aws-bundle:1.5.2` +## Setup + +Start `weed mini` with credentials and a pre-created table bucket via environment variables: + +```bash +export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE +export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +export S3_TABLE_BUCKET=my-table-bucket + +weed mini -dir ~/data +``` + +This brings up the Iceberg REST Catalog on `http://localhost:8181`, the S3 endpoint on `http://localhost:8333`, an admin S3 identity using the AWS env vars (used as Spark's SigV4 credentials below), and the table bucket `my-table-bucket` pre-created. + ## Configuration ### PySpark @@ -34,15 +47,15 @@ spark = (SparkSession.builder .config("spark.sql.catalog.iceberg.rest.auth.type", "sigv4") .config("spark.sql.catalog.iceberg.rest.sigv4-enabled", "true") .config("spark.sql.catalog.iceberg.rest.signing-name", "s3") - .config("spark.sql.catalog.iceberg.rest.access-key-id", "YOUR_ACCESS_KEY") - .config("spark.sql.catalog.iceberg.rest.secret-access-key", "YOUR_SECRET_KEY") + .config("spark.sql.catalog.iceberg.rest.access-key-id", "AKIAIOSFODNN7EXAMPLE") + .config("spark.sql.catalog.iceberg.rest.secret-access-key", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY") # S3 FileIO .config("spark.sql.catalog.iceberg.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") .config("spark.sql.catalog.iceberg.s3.endpoint", "http://localhost:8333") .config("spark.sql.catalog.iceberg.s3.region", "us-east-1") - .config("spark.sql.catalog.iceberg.s3.access-key", "YOUR_ACCESS_KEY") - .config("spark.sql.catalog.iceberg.s3.secret-key", "YOUR_SECRET_KEY") + .config("spark.sql.catalog.iceberg.s3.access-key", "AKIAIOSFODNN7EXAMPLE") + .config("spark.sql.catalog.iceberg.s3.secret-key", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY") .config("spark.sql.catalog.iceberg.s3.path-style-access", "true") .getOrCreate() ) @@ -61,8 +74,8 @@ spark-sql \ --conf "spark.sql.catalog.iceberg.io-impl=org.apache.iceberg.aws.s3.S3FileIO" \ --conf "spark.sql.catalog.iceberg.s3.endpoint=http://localhost:8333" \ --conf "spark.sql.catalog.iceberg.s3.path-style-access=true" \ - --conf "spark.sql.catalog.iceberg.s3.access-key=YOUR_ACCESS_KEY" \ - --conf "spark.sql.catalog.iceberg.s3.secret-key=YOUR_SECRET_KEY" \ + --conf "spark.sql.catalog.iceberg.s3.access-key=AKIAIOSFODNN7EXAMPLE" \ + --conf "spark.sql.catalog.iceberg.s3.secret-key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \ --conf "spark.sql.catalog.iceberg.rest.sigv4-enabled=true" \ --conf "spark.sql.catalog.iceberg.rest.signing-name=s3" ``` diff --git a/Trino-Iceberg-Integration.md b/Trino-Iceberg-Integration.md index 308c518..38527f1 100644 --- a/Trino-Iceberg-Integration.md +++ b/Trino-Iceberg-Integration.md @@ -4,9 +4,22 @@ Trino connects to SeaweedFS Iceberg tables using the `iceberg` connector with th ## Prerequisites -- SeaweedFS running with the Iceberg REST Catalog enabled (port `8181` by default) -- A table bucket created via `weed shell` or the S3 Tables API - Trino 4xx+ with the Iceberg connector +- SeaweedFS started as shown in [Setup](#setup) below + +## Setup + +Start `weed mini` with credentials and a pre-created table bucket via environment variables: + +```bash +export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE +export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +export S3_TABLE_BUCKET=my-table-bucket + +weed mini -dir ~/data +``` + +This brings up the Iceberg REST Catalog on `http://localhost:8181`, the S3 endpoint on `http://localhost:8333`, an admin S3 identity using the AWS env vars (used as Trino's SigV4 credentials below), and the table bucket `my-table-bucket` pre-created. ## Configuration @@ -30,8 +43,8 @@ fs.native-s3.enabled=true s3.endpoint=http://localhost:8333 s3.path-style-access=true s3.signer-type=AwsS3V4Signer -s3.aws-access-key=YOUR_ACCESS_KEY -s3.aws-secret-key=YOUR_SECRET_KEY +s3.aws-access-key=AKIAIOSFODNN7EXAMPLE +s3.aws-secret-key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY s3.region=us-east-1 ```