diff --git a/Lakekeeper-Iceberg-Integration.md b/Lakekeeper-Iceberg-Integration.md new file mode 100644 index 0000000..9b6708b --- /dev/null +++ b/Lakekeeper-Iceberg-Integration.md @@ -0,0 +1,139 @@ +# Lakekeeper Iceberg Integration + +[Lakekeeper](https://lakekeeper.io/) is an open-source Iceberg catalog that can use SeaweedFS as its storage backend via S3 Tables and STS-vended credentials. + +## Architecture + +In a Lakekeeper setup with SeaweedFS: + +1. **Lakekeeper** acts as the Iceberg catalog, managing namespace and table metadata +2. **SeaweedFS** provides S3-compatible storage for data files (Parquet) and metadata +3. **STS (Security Token Service)** issues temporary credentials that Lakekeeper vends to clients + +This architecture supports credential vending — Lakekeeper assumes an IAM role via STS and passes short-lived credentials to query engines, avoiding the need to distribute long-lived secrets. + +## Prerequisites + +- SeaweedFS running with IAM and STS enabled +- A table bucket created via `weed shell` or the S3 Tables API +- Lakekeeper configured to use SeaweedFS as its storage + +## SeaweedFS IAM Configuration + +Lakekeeper requires STS support for credential vending. Configure SeaweedFS with an IAM config that includes STS settings and an assumable role: + +```json +{ + "identities": [ + { + "name": "admin", + "credentials": [ + { + "accessKey": "admin", + "secretKey": "admin" + } + ], + "actions": ["Admin", "Read", "List", "Tagging", "Write"] + } + ], + "sts": { + "tokenDuration": "12h", + "maxSessionLength": "24h", + "issuer": "seaweedfs-sts", + "signingKey": "BASE64_ENCODED_SIGNING_KEY" + }, + "roles": [ + { + "roleName": "LakekeeperVendedRole", + "roleArn": "arn:aws:iam::000000000000:role/LakekeeperVendedRole", + "trustPolicy": { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": "*", + "Action": "sts:AssumeRole" + } + ] + }, + "attachedPolicies": ["FullAccess"] + } + ], + "policies": [ + { + "name": "FullAccess", + "document": { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": "*", + "Resource": "*" + } + ] + } + } + ] +} +``` + +Start SeaweedFS with IAM enabled: + +```bash +weed mini \ + -s3.config /path/to/iam_config.json \ + -s3.iam.config /path/to/iam_config.json \ + -s3.iam.readOnly=false +``` + +## STS Credential Vending + +Lakekeeper uses STS `AssumeRole` to obtain temporary credentials for accessing SeaweedFS S3: + +``` +POST http://localhost:8333/?Action=AssumeRole + &RoleArn=arn:aws:iam::000000000000:role/LakekeeperVendedRole + &RoleSessionName=lakekeeper-session + &Version=2011-06-15 +``` + +The response includes temporary `AccessKeyId`, `SecretAccessKey`, and `SessionToken` that Lakekeeper vends to query engines. + +## S3 Tables Operations + +Lakekeeper interacts with SeaweedFS via the S3 Tables REST API using SigV4 signing with the `s3tables` service name: + +``` +# Create a table bucket +PUT /buckets +Content-Type: application/x-amz-json-1.1 +{"name": "iceberg-tables"} + +# Create a namespace +PUT /namespaces/{bucketARN} +{"namespace": ["my_namespace"]} + +# Create a table +PUT /tables/{bucketARN}/{namespace} +{"name": "my_table", "format": "ICEBERG"} +``` + +All requests must be signed with SigV4 using the `s3tables` service name and the appropriate region. + +## Key Configuration Parameters + +| Parameter | Value | +|-----------|-------| +| S3 endpoint | `http://localhost:8333` | +| STS endpoint | `http://localhost:8333` (same as S3) | +| Region | `us-east-1` | +| SigV4 service (S3 Tables) | `s3tables` | +| SigV4 service (S3 data) | `s3` | +| Role ARN | `arn:aws:iam::000000000000:role/LakekeeperVendedRole` | + +## See Also + +- [[SeaweedFS Iceberg Catalog]] - Architecture and concepts +- [[S3 Tables Security]] - IAM policies for table access +- [[S3 Table Bucket]] - Managing table buckets +- [[STS Integration Tests|Amazon-IAM-API]] - STS API details diff --git a/RisingWave-Iceberg-Integration.md b/RisingWave-Iceberg-Integration.md new file mode 100644 index 0000000..33c5060 --- /dev/null +++ b/RisingWave-Iceberg-Integration.md @@ -0,0 +1,134 @@ +# RisingWave Iceberg Integration + +RisingWave can read from and write to SeaweedFS Iceberg tables using the `iceberg` connector with the REST catalog type. + +## Prerequisites + +- SeaweedFS running with the Iceberg REST Catalog enabled (port `8181` by default) +- A table bucket created via `weed shell` or the S3 Tables API +- RisingWave v2.5.0+ with Iceberg connector support + +## Reading from Iceberg (Source) + +Create a source to read an existing Iceberg table: + +```sql +CREATE SOURCE my_source WITH ( + connector = 'iceberg', + catalog.type = 'rest', + catalog.uri = 'http://localhost:8181', + catalog.name = 'default', + database.name = 'my_namespace', + table.name = 'my_table', + warehouse.path = 's3://my-table-bucket', + s3.endpoint = 'http://localhost:8333', + s3.region = 'us-east-1', + s3.access.key = 'YOUR_ACCESS_KEY', + s3.secret.key = 'YOUR_SECRET_KEY', + s3.path.style.access = 'true', + catalog.rest.sigv4_enabled = 'true', + catalog.rest.signing_region = 'us-east-1', + catalog.rest.signing_name = 's3' +); +``` + +Query the source: + +```sql +SELECT * FROM my_source ORDER BY id; +``` + +## Writing to Iceberg (Sink) + +### Append-Only Sink + +Stream data from a RisingWave table to an Iceberg table in append-only mode: + +```sql +-- Create a RisingWave table +CREATE TABLE events (id INT, event VARCHAR); + +-- Create an append-only Iceberg sink +CREATE SINK events_sink FROM events +WITH ( + connector = 'iceberg', + catalog.type = 'rest', + catalog.uri = 'http://localhost:8181', + catalog.name = 'default', + database.name = 'my_namespace', + table.name = 'events', + warehouse.path = 's3://my-table-bucket', + s3.endpoint = 'http://localhost:8333', + s3.region = 'us-east-1', + s3.access.key = 'YOUR_ACCESS_KEY', + s3.secret.key = 'YOUR_SECRET_KEY', + s3.path.style.access = 'true', + catalog.rest.sigv4_enabled = 'true', + catalog.rest.signing_region = 'us-east-1', + catalog.rest.signing_name = 's3', + type = 'append-only', + force_append_only = 'true' +); + +-- Insert data (will be streamed to Iceberg) +INSERT INTO events VALUES (1, 'click'), (2, 'view'); +FLUSH; +``` + +### Upsert Sink + +For tables with a primary key, use upsert mode to propagate updates and deletes: + +```sql +CREATE TABLE users (id INT PRIMARY KEY, name VARCHAR); + +CREATE SINK users_sink FROM users +WITH ( + connector = 'iceberg', + catalog.type = 'rest', + catalog.uri = 'http://localhost:8181', + catalog.name = 'default', + database.name = 'my_namespace', + table.name = 'users', + warehouse.path = 's3://my-table-bucket', + s3.endpoint = 'http://localhost:8333', + s3.region = 'us-east-1', + s3.access.key = 'YOUR_ACCESS_KEY', + s3.secret.key = 'YOUR_SECRET_KEY', + s3.path.style.access = 'true', + catalog.rest.sigv4_enabled = 'true', + catalog.rest.signing_region = 'us-east-1', + catalog.rest.signing_name = 's3', + type = 'upsert', + primary_key = 'id' +); + +-- These operations are streamed to Iceberg +INSERT INTO users VALUES (1, 'Alice'), (2, 'Bob'); +FLUSH; +UPDATE users SET name = 'Charles' WHERE id = 1; +DELETE FROM users WHERE id = 2; +FLUSH; +``` + +## Configuration Reference + +| Parameter | Description | +|-----------|-------------| +| `catalog.type` | Must be `rest` | +| `catalog.uri` | Iceberg REST Catalog URL (default `http://localhost:8181`) | +| `catalog.name` | Catalog name (use `default`) | +| `database.name` | Iceberg namespace name | +| `table.name` | Iceberg table name | +| `warehouse.path` | S3 path to the table bucket (e.g., `s3://my-bucket`) | +| `s3.endpoint` | SeaweedFS S3 endpoint (default `http://localhost:8333`) | +| `s3.path.style.access` | Must be `true` for SeaweedFS | +| `catalog.rest.sigv4_enabled` | Enable SigV4 auth (`true` when IAM is configured) | +| `catalog.rest.signing_name` | Signing service name (`s3`) | +| `type` | Sink type: `append-only` or `upsert` | +| `primary_key` | Required for upsert sinks | + +## See Also + +- [[SeaweedFS Iceberg Catalog]] - Architecture and concepts +- [[S3 Table Bucket]] - Managing table buckets diff --git a/SeaweedFS-Iceberg-Catalog.md b/SeaweedFS-Iceberg-Catalog.md index 1e02395..f18d264 100644 --- a/SeaweedFS-Iceberg-Catalog.md +++ b/SeaweedFS-Iceberg-Catalog.md @@ -1,23 +1,57 @@ # SeaweedFS Iceberg Catalog -SeaweedFS provides a built-in Iceberg REST Catalog that can be used with popular analytics engines like Apache Spark and Trino. +SeaweedFS provides a built-in Iceberg REST Catalog that can be used with popular analytics engines. ## Architecture The SeaweedFS S3 Tables feature implements the **Iceberg REST Catalog API**. This allows clients to talk directly to SeaweedFS to manage Iceberg namespaces and tables, while the underlying data files (Parquet, Avro, Metadata JSON) are stored in SeaweedFS S3 buckets. -- **Endpoint**: The Iceberg REST API is available on the S3 port (default `8333`) under `/v1/`. -- **Authentication**: Uses AWS Signature Version 4 (SigV4) with the `s3tables` service name. +- **Iceberg REST Catalog**: Available on a dedicated port (default `8181`) +- **S3 Data Access**: Available on the S3 port (default `8333`) +- **Authentication**: SigV4 (Spark, Trino, RisingWave) or OAuth2 (DuckDB) ## Catalog and Bucket Relationship In SeaweedFS, an **Iceberg Catalog** corresponds 1:1 with a **Table Bucket**. -- When you configure a client (Spark/Trino) with a URI like `http://localhost:8333/v1/my-catalog/`, SeaweedFS maps requests to the bucket named `my-catalog`. -- If no catalog/prefix is provided in the URL (e.g., `http://localhost:8333/v1/`), it defaults to using a bucket named `warehouse`. +- When you configure a client with a URI prefix like `http://localhost:8181/v1/my-catalog/`, SeaweedFS maps requests to the bucket named `my-catalog`. +- If no catalog/prefix is provided in the URL (e.g., `http://localhost:8181/v1/`), it defaults to using a bucket named `warehouse`. This architecture allows you to manage multiple independent Iceberg catalogs on the same SeaweedFS cluster simply by creating multiple buckets. +## Quick Start + +### 1. Start SeaweedFS + +```bash +weed mini +``` + +This starts: +- S3 API on port `8333` +- Iceberg REST Catalog on port `8181` + +### 2. Create a Table Bucket + +```bash +weed shell +> s3tables.bucket -create -name my-catalog +``` + +### 3. Connect Your Query Engine + +See the integration guide for your engine below. + +## Client Integrations + +| Engine | Auth Method | Guide | +|--------|------------|-------| +| **Apache Spark** | SigV4 | [[Spark Iceberg Integration]] | +| **Trino** | SigV4 | [[Trino Iceberg Integration]] | +| **DuckDB** | OAuth2 | [[DuckDB Iceberg Integration]] | +| **RisingWave** | SigV4 | [[RisingWave Iceberg Integration]] | +| **Lakekeeper** | STS + SigV4 | [[Lakekeeper Iceberg Integration]] | + ## Metadata Storage SeaweedFS stores Iceberg metadata using a hybrid approach to maximize performance and compatibility: @@ -39,11 +73,13 @@ Table metadata follows the standard Iceberg V2 specification: ## Authentication and Authorization -Security is managed using the standard **AWS Signature Version 4 (SigV4)** protocol, integrated with SeaweedFS's IAM system. +### Authentication Methods -### Authentication -- Clients must sign requests using the `s3tables` service name (not `s3`). -- SeaweedFS validates the signature using the access key and secret key provided in the client configuration. +SeaweedFS supports two authentication methods for the Iceberg REST Catalog: + +**SigV4 (Spark, Trino, RisingWave)** — Clients sign each request using AWS Signature Version 4. This is the standard method used by most Iceberg-compatible engines. + +**OAuth2 (DuckDB)** — Clients exchange S3 credentials for a bearer token via `POST /v1/oauth/tokens` using the `client_credentials` grant type. The S3 access key is used as `client_id` and the secret key as `client_secret`. ### Authorization (IAM) Permissions are managed via **S3 Bucket Policies** applied to the Table Bucket. @@ -66,115 +102,22 @@ Permissions are managed via **S3 Bucket Policies** applied to the Table Bucket. } ``` -### Signing Details -- **Service Name**: `s3tables` -- **Region**: Defaults to `us-east-1` (configurable) -- **Endpoint**: The S3 API port (default `8333`) +### Anonymous Access (Development) ---- +If SeaweedFS is running without IAM configuration (e.g., `weed mini` with no `-s3.config`), the Iceberg Catalog allows anonymous access by default. This is useful for local development and testing. See each integration page for anonymous configuration details. -## Apache Spark Integration +## Configuration Reference -To connect Spark to SeaweedFS S3 Tables, you need the Iceberg Spark runtime and the AWS bundle for S3 support. +| Parameter | CLI Flag | Default | +|-----------|----------|---------| +| Iceberg REST port | `-s3.port.iceberg` (mini) / `--port.iceberg` (standalone) | `8181` | +| S3 port | `-s3.port` (mini) / `--port` (standalone) | `8333` | +| Disable Iceberg | Set port to `0` | Enabled | +| IAM config | `-s3.config` | None (anonymous) | -### Prerequisites +## See Also -Required packages: -- `org.apache.iceberg:iceberg-spark-runtime-3.5_2.12` (or matching your Spark version) -- `org.apache.iceberg:iceberg-aws-bundle` - -### Configuration - -Use the `rest` catalog type and configure SigV4 signing. - -**Example `spark-sql` command:** - -```bash -spark-sql \ - --packages "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.2,org.apache.iceberg:iceberg-aws-bundle:1.7.2" \ - --conf "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" \ - --conf "spark.sql.defaultCatalog=iceberg" \ - --conf "spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog" \ - --conf "spark.sql.catalog.iceberg.type=rest" \ - --conf "spark.sql.catalog.iceberg.uri=http://localhost:8333" \ - --conf "spark.sql.catalog.iceberg.warehouse=s3tablescatalog/my-table-bucket" \ - --conf "spark.sql.catalog.iceberg.io-impl=org.apache.iceberg.aws.s3.S3FileIO" \ - --conf "spark.sql.catalog.iceberg.s3.endpoint=http://localhost:8333" \ - --conf "spark.sql.catalog.iceberg.s3.path-style-access=true" \ - --conf "spark.sql.catalog.iceberg.s3.access-key-id=YOUR_ACCESS_KEY" \ - --conf "spark.sql.catalog.iceberg.s3.secret-access-key=YOUR_SECRET_KEY" \ - --conf "spark.sql.catalog.iceberg.rest.sigv4-enabled=true" \ - --conf "spark.sql.catalog.iceberg.rest.signing-name=s3tables" -``` - -> [!NOTE] -> The `warehouse` property should be in the format `s3tablescatalog/`. - ---- - -## Trino Integration - -Trino connects via the `iceberg` connector with the `rest` catalog type. - -### Configuration (`etc/catalog/iceberg.properties`) - -```properties -connector.name=iceberg -iceberg.catalog.type=rest -iceberg.rest-catalog.uri=http://localhost:8333 -iceberg.rest-catalog.warehouse=s3tablescatalog/my-table-bucket - -# Enable SigV4 authentication for the REST catalog -iceberg.rest-catalog.security=SIGV4 -iceberg.rest-catalog.signing-name=s3tables - -# S3 FileIO Configuration -fs.native-s3.enabled=true -s3.endpoint=http://localhost:8333 -s3.path-style-access=true -s3.signer-type=AwsS3V4Signer -s3.aws-access-key=YOUR_ACCESS_KEY -s3.aws-secret-key=YOUR_SECRET_KEY -s3.region=us-east-1 -``` - ---- - -## Notes on Table Buckets - -When configuring the catalog, the `warehouse` location typically points to a specific **Table Bucket**. -- **Spark**: `--conf "spark.sql.catalog.iceberg.warehouse=s3tablescatalog/my-table-bucket"` -- **Trino**: `iceberg.rest-catalog.prefix=my-table-bucket` (optional, can also be part of warehouse path) - -SeaweedFS treats the Table Bucket as the root for that catalog instance. You can have multiple catalogs pointing to different table buckets. - ---- - -## DuckDB Integration - -DuckDB connects via the Iceberg extension using OAuth2 authentication. See the dedicated [[DuckDB Iceberg Integration]] page for setup instructions. - ---- - -## Anonymous Access (Zero Config) - -If SeaweedFS is running in **Zero Configuration** mode (no `-s3.config` or `-iam.config` provided), the Iceberg Catalog allows anonymous access by default. This is useful for local development and testing. - -### Spark Configuration for Anonymous Access - -To connect without credentials, disable SigV4 signing in the Spark configuration: - -```bash -spark-sql \ - --packages "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.2,org.apache.iceberg:iceberg-aws-bundle:1.7.2" \ - --conf "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" \ - --conf "spark.sql.defaultCatalog=iceberg" \ - --conf "spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog" \ - --conf "spark.sql.catalog.iceberg.type=rest" \ - --conf "spark.sql.catalog.iceberg.uri=http://localhost:8333" \ - --conf "spark.sql.catalog.iceberg.warehouse=s3tablescatalog/my-table-bucket" \ - --conf "spark.sql.catalog.iceberg.io-impl=org.apache.iceberg.aws.s3.S3FileIO" \ - --conf "spark.sql.catalog.iceberg.s3.endpoint=http://localhost:8333" \ - --conf "spark.sql.catalog.iceberg.s3.path-style-access=true" \ - --conf "spark.sql.catalog.iceberg.rest.sigv4-enabled=false" -``` +- [[S3 Table Bucket]] - Creating and managing table buckets +- [[S3 Tables Security]] - IAM policies for table access +- [[S3 Table Bucket Commands]] - `weed shell` commands +- [[Iceberg Table Maintenance]] - Compaction and cleanup diff --git a/Spark-Iceberg-Integration.md b/Spark-Iceberg-Integration.md new file mode 100644 index 0000000..dfcdc9b --- /dev/null +++ b/Spark-Iceberg-Integration.md @@ -0,0 +1,143 @@ +# Spark Iceberg Integration + +Apache Spark connects to SeaweedFS Iceberg tables using the Iceberg Spark runtime with the `rest` catalog type and SigV4 authentication. + +## Prerequisites + +- SeaweedFS running with the Iceberg REST Catalog enabled (port `8181` by default) +- A table bucket created via `weed shell` or the S3 Tables API +- Spark 3.5+ with Iceberg packages + +Required packages: +- `org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.2` (match your Spark version) +- `org.apache.iceberg:iceberg-aws-bundle:1.5.2` + +## Configuration + +### PySpark + +```python +from pyspark.sql import SparkSession + +spark = (SparkSession.builder + .appName("SeaweedFS Iceberg") + .config("spark.jars.packages", + "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.2," + "org.apache.iceberg:iceberg-aws-bundle:1.5.2") + .config("spark.sql.extensions", + "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") + .config("spark.sql.catalog.iceberg", + "org.apache.iceberg.spark.SparkCatalog") + .config("spark.sql.catalog.iceberg.type", "rest") + .config("spark.sql.catalog.iceberg.uri", "http://localhost:8181") + # SigV4 authentication + .config("spark.sql.catalog.iceberg.rest.auth.type", "sigv4") + .config("spark.sql.catalog.iceberg.rest.sigv4-enabled", "true") + .config("spark.sql.catalog.iceberg.rest.signing-name", "s3") + .config("spark.sql.catalog.iceberg.rest.access-key-id", "YOUR_ACCESS_KEY") + .config("spark.sql.catalog.iceberg.rest.secret-access-key", "YOUR_SECRET_KEY") + # S3 FileIO + .config("spark.sql.catalog.iceberg.io-impl", + "org.apache.iceberg.aws.s3.S3FileIO") + .config("spark.sql.catalog.iceberg.s3.endpoint", "http://localhost:8333") + .config("spark.sql.catalog.iceberg.s3.region", "us-east-1") + .config("spark.sql.catalog.iceberg.s3.access-key", "YOUR_ACCESS_KEY") + .config("spark.sql.catalog.iceberg.s3.secret-key", "YOUR_SECRET_KEY") + .config("spark.sql.catalog.iceberg.s3.path-style-access", "true") + .getOrCreate() +) +``` + +### spark-sql CLI + +```bash +spark-sql \ + --packages "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.2,org.apache.iceberg:iceberg-aws-bundle:1.5.2" \ + --conf "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" \ + --conf "spark.sql.defaultCatalog=iceberg" \ + --conf "spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog" \ + --conf "spark.sql.catalog.iceberg.type=rest" \ + --conf "spark.sql.catalog.iceberg.uri=http://localhost:8181" \ + --conf "spark.sql.catalog.iceberg.io-impl=org.apache.iceberg.aws.s3.S3FileIO" \ + --conf "spark.sql.catalog.iceberg.s3.endpoint=http://localhost:8333" \ + --conf "spark.sql.catalog.iceberg.s3.path-style-access=true" \ + --conf "spark.sql.catalog.iceberg.s3.access-key=YOUR_ACCESS_KEY" \ + --conf "spark.sql.catalog.iceberg.s3.secret-key=YOUR_SECRET_KEY" \ + --conf "spark.sql.catalog.iceberg.rest.sigv4-enabled=true" \ + --conf "spark.sql.catalog.iceberg.rest.signing-name=s3" +``` + +## Example SQL + +### Namespace and Table Operations + +```sql +-- Create a namespace +CREATE NAMESPACE iceberg.my_namespace; + +-- Create a table +CREATE TABLE iceberg.my_namespace.users ( + id INT, + name STRING, + age INT +) USING iceberg; + +-- Insert data +INSERT INTO iceberg.my_namespace.users VALUES + (1, 'Alice', 30), + (2, 'Bob', 25), + (3, 'Charlie', 35); + +-- Query +SELECT * FROM iceberg.my_namespace.users; +SELECT COUNT(*) FROM iceberg.my_namespace.users; + +-- Update and delete +UPDATE iceberg.my_namespace.users SET age = 31 WHERE id = 1; +DELETE FROM iceberg.my_namespace.users WHERE id = 3; +``` + +### Multi-Level Namespaces + +```sql +CREATE NAMESPACE iceberg.analytics; +CREATE NAMESPACE iceberg.analytics.web; + +CREATE TABLE iceberg.analytics.web.pageviews ( + id INT, + url STRING, + ts TIMESTAMP +) USING iceberg; +``` + +### Time Travel + +```sql +-- Query a table at a specific point in time +SELECT COUNT(*) FROM iceberg.my_namespace.users + TIMESTAMP AS OF '2024-01-15 10:30:00'; +``` + +## Anonymous Access + +When SeaweedFS runs without IAM, disable SigV4: + +```python +spark = (SparkSession.builder + .config("spark.sql.catalog.iceberg.type", "rest") + .config("spark.sql.catalog.iceberg.uri", "http://localhost:8181") + .config("spark.sql.catalog.iceberg.rest.sigv4-enabled", "false") + .config("spark.sql.catalog.iceberg.io-impl", + "org.apache.iceberg.aws.s3.S3FileIO") + .config("spark.sql.catalog.iceberg.s3.endpoint", "http://localhost:8333") + .config("spark.sql.catalog.iceberg.s3.path-style-access", "true") + # ... other standard configs + .getOrCreate() +) +``` + +## See Also + +- [[SeaweedFS Iceberg Catalog]] - Architecture and concepts +- [[S3 Table Bucket]] - Managing table buckets +- [[run Spark on SeaweedFS]] - Spark with SeaweedFS HDFS connector (non-Iceberg) diff --git a/Trino-Iceberg-Integration.md b/Trino-Iceberg-Integration.md new file mode 100644 index 0000000..e586acc --- /dev/null +++ b/Trino-Iceberg-Integration.md @@ -0,0 +1,115 @@ +# Trino Iceberg Integration + +Trino connects to SeaweedFS Iceberg tables using the `iceberg` connector with the `rest` catalog type and SigV4 authentication. + +## Prerequisites + +- SeaweedFS running with the Iceberg REST Catalog enabled (port `8181` by default) +- A table bucket created via `weed shell` or the S3 Tables API +- Trino 4xx+ with the Iceberg connector + +## Configuration + +Create a catalog properties file at `etc/catalog/iceberg.properties`: + +```properties +connector.name=iceberg +iceberg.catalog.type=rest +iceberg.rest-catalog.uri=http://localhost:8181 +iceberg.rest-catalog.warehouse=s3://my-table-bucket + +# File format +iceberg.file-format=PARQUET +iceberg.unique-table-location=true + +# SigV4 authentication for the REST catalog +iceberg.rest-catalog.security=SIGV4 + +# S3 FileIO configuration +fs.native-s3.enabled=true +s3.endpoint=http://localhost:8333 +s3.path-style-access=true +s3.signer-type=AwsS3V4Signer +s3.aws-access-key=YOUR_ACCESS_KEY +s3.aws-secret-key=YOUR_SECRET_KEY +s3.region=us-east-1 +``` + +### Multi-Level Namespaces + +To use nested namespaces (e.g., `db.schema`), add: + +```properties +iceberg.rest-catalog.nested-namespace-enabled=true +``` + +## Example SQL + +### Schema Operations + +```sql +-- Create a schema (maps to an Iceberg namespace) +CREATE SCHEMA IF NOT EXISTS iceberg.my_namespace; + +-- List schemas +SHOW SCHEMAS FROM iceberg; +``` + +### Table Operations + +```sql +-- Create a table +CREATE TABLE iceberg.my_namespace.events ( + id INTEGER, + event VARCHAR, + ts TIMESTAMP(6) +) WITH ( + format = 'PARQUET' +); + +-- Insert data +INSERT INTO iceberg.my_namespace.events VALUES + (1, 'click', TIMESTAMP '2024-01-15 10:30:00'), + (2, 'view', TIMESTAMP '2024-01-15 11:00:00'); + +-- Query +SELECT * FROM iceberg.my_namespace.events; + +-- Inspect data files +SELECT file_path FROM iceberg.my_namespace."events$files" LIMIT 5; +``` + +### Multi-Level Namespace Example + +```sql +CREATE SCHEMA IF NOT EXISTS iceberg."analytics.web"; + +CREATE TABLE iceberg."analytics.web".pageviews ( + id INTEGER, + url VARCHAR, + ts TIMESTAMP(6) +) WITH ( + format = 'PARQUET' +); +``` + +## Anonymous Access + +When SeaweedFS runs without IAM, remove the SigV4 and credential properties: + +```properties +connector.name=iceberg +iceberg.catalog.type=rest +iceberg.rest-catalog.uri=http://localhost:8181 +iceberg.rest-catalog.warehouse=s3://my-table-bucket + +fs.native-s3.enabled=true +s3.endpoint=http://localhost:8333 +s3.path-style-access=true +s3.region=us-east-1 +``` + +## See Also + +- [[SeaweedFS Iceberg Catalog]] - Architecture and concepts +- [[S3 Table Bucket]] - Managing table buckets