mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-14 10:30:45 +02:00
Reorganize Iceberg integration wiki pages
Restructure the Iceberg Catalog wiki into a hub page with dedicated integration guides for each query engine: - Trino Iceberg Integration (new) - Spark Iceberg Integration (new) - RisingWave Iceberg Integration (new) - Lakekeeper Iceberg Integration (new) - DuckDB Iceberg Integration (existing) The main SeaweedFS Iceberg Catalog page now focuses on architecture, auth concepts, and links to per-engine guides. All examples are derived from the CI integration test configurations.
Chris Lu
@@ -0,0 +1,139 @@
|
||||
# Lakekeeper Iceberg Integration
|
||||
|
||||
[Lakekeeper](https://lakekeeper.io/) is an open-source Iceberg catalog that can use SeaweedFS as its storage backend via S3 Tables and STS-vended credentials.
|
||||
|
||||
## Architecture
|
||||
|
||||
In a Lakekeeper setup with SeaweedFS:
|
||||
|
||||
1. **Lakekeeper** acts as the Iceberg catalog, managing namespace and table metadata
|
||||
2. **SeaweedFS** provides S3-compatible storage for data files (Parquet) and metadata
|
||||
3. **STS (Security Token Service)** issues temporary credentials that Lakekeeper vends to clients
|
||||
|
||||
This architecture supports credential vending — Lakekeeper assumes an IAM role via STS and passes short-lived credentials to query engines, avoiding the need to distribute long-lived secrets.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- SeaweedFS running with IAM and STS enabled
|
||||
- A table bucket created via `weed shell` or the S3 Tables API
|
||||
- Lakekeeper configured to use SeaweedFS as its storage
|
||||
|
||||
## SeaweedFS IAM Configuration
|
||||
|
||||
Lakekeeper requires STS support for credential vending. Configure SeaweedFS with an IAM config that includes STS settings and an assumable role:
|
||||
|
||||
```json
|
||||
{
|
||||
"identities": [
|
||||
{
|
||||
"name": "admin",
|
||||
"credentials": [
|
||||
{
|
||||
"accessKey": "admin",
|
||||
"secretKey": "admin"
|
||||
}
|
||||
],
|
||||
"actions": ["Admin", "Read", "List", "Tagging", "Write"]
|
||||
}
|
||||
],
|
||||
"sts": {
|
||||
"tokenDuration": "12h",
|
||||
"maxSessionLength": "24h",
|
||||
"issuer": "seaweedfs-sts",
|
||||
"signingKey": "BASE64_ENCODED_SIGNING_KEY"
|
||||
},
|
||||
"roles": [
|
||||
{
|
||||
"roleName": "LakekeeperVendedRole",
|
||||
"roleArn": "arn:aws:iam::000000000000:role/LakekeeperVendedRole",
|
||||
"trustPolicy": {
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Principal": "*",
|
||||
"Action": "sts:AssumeRole"
|
||||
}
|
||||
]
|
||||
},
|
||||
"attachedPolicies": ["FullAccess"]
|
||||
}
|
||||
],
|
||||
"policies": [
|
||||
{
|
||||
"name": "FullAccess",
|
||||
"document": {
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": "*",
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Start SeaweedFS with IAM enabled:
|
||||
|
||||
```bash
|
||||
weed mini \
|
||||
-s3.config /path/to/iam_config.json \
|
||||
-s3.iam.config /path/to/iam_config.json \
|
||||
-s3.iam.readOnly=false
|
||||
```
|
||||
|
||||
## STS Credential Vending
|
||||
|
||||
Lakekeeper uses STS `AssumeRole` to obtain temporary credentials for accessing SeaweedFS S3:
|
||||
|
||||
```
|
||||
POST http://localhost:8333/?Action=AssumeRole
|
||||
&RoleArn=arn:aws:iam::000000000000:role/LakekeeperVendedRole
|
||||
&RoleSessionName=lakekeeper-session
|
||||
&Version=2011-06-15
|
||||
```
|
||||
|
||||
The response includes temporary `AccessKeyId`, `SecretAccessKey`, and `SessionToken` that Lakekeeper vends to query engines.
|
||||
|
||||
## S3 Tables Operations
|
||||
|
||||
Lakekeeper interacts with SeaweedFS via the S3 Tables REST API using SigV4 signing with the `s3tables` service name:
|
||||
|
||||
```
|
||||
# Create a table bucket
|
||||
PUT /buckets
|
||||
Content-Type: application/x-amz-json-1.1
|
||||
{"name": "iceberg-tables"}
|
||||
|
||||
# Create a namespace
|
||||
PUT /namespaces/{bucketARN}
|
||||
{"namespace": ["my_namespace"]}
|
||||
|
||||
# Create a table
|
||||
PUT /tables/{bucketARN}/{namespace}
|
||||
{"name": "my_table", "format": "ICEBERG"}
|
||||
```
|
||||
|
||||
All requests must be signed with SigV4 using the `s3tables` service name and the appropriate region.
|
||||
|
||||
## Key Configuration Parameters
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| S3 endpoint | `http://localhost:8333` |
|
||||
| STS endpoint | `http://localhost:8333` (same as S3) |
|
||||
| Region | `us-east-1` |
|
||||
| SigV4 service (S3 Tables) | `s3tables` |
|
||||
| SigV4 service (S3 data) | `s3` |
|
||||
| Role ARN | `arn:aws:iam::000000000000:role/LakekeeperVendedRole` |
|
||||
|
||||
## See Also
|
||||
|
||||
- [[SeaweedFS Iceberg Catalog]] - Architecture and concepts
|
||||
- [[S3 Tables Security]] - IAM policies for table access
|
||||
- [[S3 Table Bucket]] - Managing table buckets
|
||||
- [[STS Integration Tests|Amazon-IAM-API]] - STS API details
|
||||
@@ -0,0 +1,134 @@
|
||||
# RisingWave Iceberg Integration
|
||||
|
||||
RisingWave can read from and write to SeaweedFS Iceberg tables using the `iceberg` connector with the REST catalog type.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- SeaweedFS running with the Iceberg REST Catalog enabled (port `8181` by default)
|
||||
- A table bucket created via `weed shell` or the S3 Tables API
|
||||
- RisingWave v2.5.0+ with Iceberg connector support
|
||||
|
||||
## Reading from Iceberg (Source)
|
||||
|
||||
Create a source to read an existing Iceberg table:
|
||||
|
||||
```sql
|
||||
CREATE SOURCE my_source WITH (
|
||||
connector = 'iceberg',
|
||||
catalog.type = 'rest',
|
||||
catalog.uri = 'http://localhost:8181',
|
||||
catalog.name = 'default',
|
||||
database.name = 'my_namespace',
|
||||
table.name = 'my_table',
|
||||
warehouse.path = 's3://my-table-bucket',
|
||||
s3.endpoint = 'http://localhost:8333',
|
||||
s3.region = 'us-east-1',
|
||||
s3.access.key = 'YOUR_ACCESS_KEY',
|
||||
s3.secret.key = 'YOUR_SECRET_KEY',
|
||||
s3.path.style.access = 'true',
|
||||
catalog.rest.sigv4_enabled = 'true',
|
||||
catalog.rest.signing_region = 'us-east-1',
|
||||
catalog.rest.signing_name = 's3'
|
||||
);
|
||||
```
|
||||
|
||||
Query the source:
|
||||
|
||||
```sql
|
||||
SELECT * FROM my_source ORDER BY id;
|
||||
```
|
||||
|
||||
## Writing to Iceberg (Sink)
|
||||
|
||||
### Append-Only Sink
|
||||
|
||||
Stream data from a RisingWave table to an Iceberg table in append-only mode:
|
||||
|
||||
```sql
|
||||
-- Create a RisingWave table
|
||||
CREATE TABLE events (id INT, event VARCHAR);
|
||||
|
||||
-- Create an append-only Iceberg sink
|
||||
CREATE SINK events_sink FROM events
|
||||
WITH (
|
||||
connector = 'iceberg',
|
||||
catalog.type = 'rest',
|
||||
catalog.uri = 'http://localhost:8181',
|
||||
catalog.name = 'default',
|
||||
database.name = 'my_namespace',
|
||||
table.name = 'events',
|
||||
warehouse.path = 's3://my-table-bucket',
|
||||
s3.endpoint = 'http://localhost:8333',
|
||||
s3.region = 'us-east-1',
|
||||
s3.access.key = 'YOUR_ACCESS_KEY',
|
||||
s3.secret.key = 'YOUR_SECRET_KEY',
|
||||
s3.path.style.access = 'true',
|
||||
catalog.rest.sigv4_enabled = 'true',
|
||||
catalog.rest.signing_region = 'us-east-1',
|
||||
catalog.rest.signing_name = 's3',
|
||||
type = 'append-only',
|
||||
force_append_only = 'true'
|
||||
);
|
||||
|
||||
-- Insert data (will be streamed to Iceberg)
|
||||
INSERT INTO events VALUES (1, 'click'), (2, 'view');
|
||||
FLUSH;
|
||||
```
|
||||
|
||||
### Upsert Sink
|
||||
|
||||
For tables with a primary key, use upsert mode to propagate updates and deletes:
|
||||
|
||||
```sql
|
||||
CREATE TABLE users (id INT PRIMARY KEY, name VARCHAR);
|
||||
|
||||
CREATE SINK users_sink FROM users
|
||||
WITH (
|
||||
connector = 'iceberg',
|
||||
catalog.type = 'rest',
|
||||
catalog.uri = 'http://localhost:8181',
|
||||
catalog.name = 'default',
|
||||
database.name = 'my_namespace',
|
||||
table.name = 'users',
|
||||
warehouse.path = 's3://my-table-bucket',
|
||||
s3.endpoint = 'http://localhost:8333',
|
||||
s3.region = 'us-east-1',
|
||||
s3.access.key = 'YOUR_ACCESS_KEY',
|
||||
s3.secret.key = 'YOUR_SECRET_KEY',
|
||||
s3.path.style.access = 'true',
|
||||
catalog.rest.sigv4_enabled = 'true',
|
||||
catalog.rest.signing_region = 'us-east-1',
|
||||
catalog.rest.signing_name = 's3',
|
||||
type = 'upsert',
|
||||
primary_key = 'id'
|
||||
);
|
||||
|
||||
-- These operations are streamed to Iceberg
|
||||
INSERT INTO users VALUES (1, 'Alice'), (2, 'Bob');
|
||||
FLUSH;
|
||||
UPDATE users SET name = 'Charles' WHERE id = 1;
|
||||
DELETE FROM users WHERE id = 2;
|
||||
FLUSH;
|
||||
```
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
| Parameter | Description |
|
||||
|-----------|-------------|
|
||||
| `catalog.type` | Must be `rest` |
|
||||
| `catalog.uri` | Iceberg REST Catalog URL (default `http://localhost:8181`) |
|
||||
| `catalog.name` | Catalog name (use `default`) |
|
||||
| `database.name` | Iceberg namespace name |
|
||||
| `table.name` | Iceberg table name |
|
||||
| `warehouse.path` | S3 path to the table bucket (e.g., `s3://my-bucket`) |
|
||||
| `s3.endpoint` | SeaweedFS S3 endpoint (default `http://localhost:8333`) |
|
||||
| `s3.path.style.access` | Must be `true` for SeaweedFS |
|
||||
| `catalog.rest.sigv4_enabled` | Enable SigV4 auth (`true` when IAM is configured) |
|
||||
| `catalog.rest.signing_name` | Signing service name (`s3`) |
|
||||
| `type` | Sink type: `append-only` or `upsert` |
|
||||
| `primary_key` | Required for upsert sinks |
|
||||
|
||||
## See Also
|
||||
|
||||
- [[SeaweedFS Iceberg Catalog]] - Architecture and concepts
|
||||
- [[S3 Table Bucket]] - Managing table buckets
|
||||
+59
-116
@@ -1,23 +1,57 @@
|
||||
# SeaweedFS Iceberg Catalog
|
||||
|
||||
SeaweedFS provides a built-in Iceberg REST Catalog that can be used with popular analytics engines like Apache Spark and Trino.
|
||||
SeaweedFS provides a built-in Iceberg REST Catalog that can be used with popular analytics engines.
|
||||
|
||||
## Architecture
|
||||
|
||||
The SeaweedFS S3 Tables feature implements the **Iceberg REST Catalog API**. This allows clients to talk directly to SeaweedFS to manage Iceberg namespaces and tables, while the underlying data files (Parquet, Avro, Metadata JSON) are stored in SeaweedFS S3 buckets.
|
||||
|
||||
- **Endpoint**: The Iceberg REST API is available on the S3 port (default `8333`) under `/v1/`.
|
||||
- **Authentication**: Uses AWS Signature Version 4 (SigV4) with the `s3tables` service name.
|
||||
- **Iceberg REST Catalog**: Available on a dedicated port (default `8181`)
|
||||
- **S3 Data Access**: Available on the S3 port (default `8333`)
|
||||
- **Authentication**: SigV4 (Spark, Trino, RisingWave) or OAuth2 (DuckDB)
|
||||
|
||||
## Catalog and Bucket Relationship
|
||||
|
||||
In SeaweedFS, an **Iceberg Catalog** corresponds 1:1 with a **Table Bucket**.
|
||||
|
||||
- When you configure a client (Spark/Trino) with a URI like `http://localhost:8333/v1/my-catalog/`, SeaweedFS maps requests to the bucket named `my-catalog`.
|
||||
- If no catalog/prefix is provided in the URL (e.g., `http://localhost:8333/v1/`), it defaults to using a bucket named `warehouse`.
|
||||
- When you configure a client with a URI prefix like `http://localhost:8181/v1/my-catalog/`, SeaweedFS maps requests to the bucket named `my-catalog`.
|
||||
- If no catalog/prefix is provided in the URL (e.g., `http://localhost:8181/v1/`), it defaults to using a bucket named `warehouse`.
|
||||
|
||||
This architecture allows you to manage multiple independent Iceberg catalogs on the same SeaweedFS cluster simply by creating multiple buckets.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Start SeaweedFS
|
||||
|
||||
```bash
|
||||
weed mini
|
||||
```
|
||||
|
||||
This starts:
|
||||
- S3 API on port `8333`
|
||||
- Iceberg REST Catalog on port `8181`
|
||||
|
||||
### 2. Create a Table Bucket
|
||||
|
||||
```bash
|
||||
weed shell
|
||||
> s3tables.bucket -create -name my-catalog
|
||||
```
|
||||
|
||||
### 3. Connect Your Query Engine
|
||||
|
||||
See the integration guide for your engine below.
|
||||
|
||||
## Client Integrations
|
||||
|
||||
| Engine | Auth Method | Guide |
|
||||
|--------|------------|-------|
|
||||
| **Apache Spark** | SigV4 | [[Spark Iceberg Integration]] |
|
||||
| **Trino** | SigV4 | [[Trino Iceberg Integration]] |
|
||||
| **DuckDB** | OAuth2 | [[DuckDB Iceberg Integration]] |
|
||||
| **RisingWave** | SigV4 | [[RisingWave Iceberg Integration]] |
|
||||
| **Lakekeeper** | STS + SigV4 | [[Lakekeeper Iceberg Integration]] |
|
||||
|
||||
## Metadata Storage
|
||||
|
||||
SeaweedFS stores Iceberg metadata using a hybrid approach to maximize performance and compatibility:
|
||||
@@ -39,11 +73,13 @@ Table metadata follows the standard Iceberg V2 specification:
|
||||
|
||||
## Authentication and Authorization
|
||||
|
||||
Security is managed using the standard **AWS Signature Version 4 (SigV4)** protocol, integrated with SeaweedFS's IAM system.
|
||||
### Authentication Methods
|
||||
|
||||
### Authentication
|
||||
- Clients must sign requests using the `s3tables` service name (not `s3`).
|
||||
- SeaweedFS validates the signature using the access key and secret key provided in the client configuration.
|
||||
SeaweedFS supports two authentication methods for the Iceberg REST Catalog:
|
||||
|
||||
**SigV4 (Spark, Trino, RisingWave)** — Clients sign each request using AWS Signature Version 4. This is the standard method used by most Iceberg-compatible engines.
|
||||
|
||||
**OAuth2 (DuckDB)** — Clients exchange S3 credentials for a bearer token via `POST /v1/oauth/tokens` using the `client_credentials` grant type. The S3 access key is used as `client_id` and the secret key as `client_secret`.
|
||||
|
||||
### Authorization (IAM)
|
||||
Permissions are managed via **S3 Bucket Policies** applied to the Table Bucket.
|
||||
@@ -66,115 +102,22 @@ Permissions are managed via **S3 Bucket Policies** applied to the Table Bucket.
|
||||
}
|
||||
```
|
||||
|
||||
### Signing Details
|
||||
- **Service Name**: `s3tables`
|
||||
- **Region**: Defaults to `us-east-1` (configurable)
|
||||
- **Endpoint**: The S3 API port (default `8333`)
|
||||
### Anonymous Access (Development)
|
||||
|
||||
---
|
||||
If SeaweedFS is running without IAM configuration (e.g., `weed mini` with no `-s3.config`), the Iceberg Catalog allows anonymous access by default. This is useful for local development and testing. See each integration page for anonymous configuration details.
|
||||
|
||||
## Apache Spark Integration
|
||||
## Configuration Reference
|
||||
|
||||
To connect Spark to SeaweedFS S3 Tables, you need the Iceberg Spark runtime and the AWS bundle for S3 support.
|
||||
| Parameter | CLI Flag | Default |
|
||||
|-----------|----------|---------|
|
||||
| Iceberg REST port | `-s3.port.iceberg` (mini) / `--port.iceberg` (standalone) | `8181` |
|
||||
| S3 port | `-s3.port` (mini) / `--port` (standalone) | `8333` |
|
||||
| Disable Iceberg | Set port to `0` | Enabled |
|
||||
| IAM config | `-s3.config` | None (anonymous) |
|
||||
|
||||
### Prerequisites
|
||||
## See Also
|
||||
|
||||
Required packages:
|
||||
- `org.apache.iceberg:iceberg-spark-runtime-3.5_2.12` (or matching your Spark version)
|
||||
- `org.apache.iceberg:iceberg-aws-bundle`
|
||||
|
||||
### Configuration
|
||||
|
||||
Use the `rest` catalog type and configure SigV4 signing.
|
||||
|
||||
**Example `spark-sql` command:**
|
||||
|
||||
```bash
|
||||
spark-sql \
|
||||
--packages "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.2,org.apache.iceberg:iceberg-aws-bundle:1.7.2" \
|
||||
--conf "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" \
|
||||
--conf "spark.sql.defaultCatalog=iceberg" \
|
||||
--conf "spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog" \
|
||||
--conf "spark.sql.catalog.iceberg.type=rest" \
|
||||
--conf "spark.sql.catalog.iceberg.uri=http://localhost:8333" \
|
||||
--conf "spark.sql.catalog.iceberg.warehouse=s3tablescatalog/my-table-bucket" \
|
||||
--conf "spark.sql.catalog.iceberg.io-impl=org.apache.iceberg.aws.s3.S3FileIO" \
|
||||
--conf "spark.sql.catalog.iceberg.s3.endpoint=http://localhost:8333" \
|
||||
--conf "spark.sql.catalog.iceberg.s3.path-style-access=true" \
|
||||
--conf "spark.sql.catalog.iceberg.s3.access-key-id=YOUR_ACCESS_KEY" \
|
||||
--conf "spark.sql.catalog.iceberg.s3.secret-access-key=YOUR_SECRET_KEY" \
|
||||
--conf "spark.sql.catalog.iceberg.rest.sigv4-enabled=true" \
|
||||
--conf "spark.sql.catalog.iceberg.rest.signing-name=s3tables"
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> The `warehouse` property should be in the format `s3tablescatalog/<table-bucket-name>`.
|
||||
|
||||
---
|
||||
|
||||
## Trino Integration
|
||||
|
||||
Trino connects via the `iceberg` connector with the `rest` catalog type.
|
||||
|
||||
### Configuration (`etc/catalog/iceberg.properties`)
|
||||
|
||||
```properties
|
||||
connector.name=iceberg
|
||||
iceberg.catalog.type=rest
|
||||
iceberg.rest-catalog.uri=http://localhost:8333
|
||||
iceberg.rest-catalog.warehouse=s3tablescatalog/my-table-bucket
|
||||
|
||||
# Enable SigV4 authentication for the REST catalog
|
||||
iceberg.rest-catalog.security=SIGV4
|
||||
iceberg.rest-catalog.signing-name=s3tables
|
||||
|
||||
# S3 FileIO Configuration
|
||||
fs.native-s3.enabled=true
|
||||
s3.endpoint=http://localhost:8333
|
||||
s3.path-style-access=true
|
||||
s3.signer-type=AwsS3V4Signer
|
||||
s3.aws-access-key=YOUR_ACCESS_KEY
|
||||
s3.aws-secret-key=YOUR_SECRET_KEY
|
||||
s3.region=us-east-1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Notes on Table Buckets
|
||||
|
||||
When configuring the catalog, the `warehouse` location typically points to a specific **Table Bucket**.
|
||||
- **Spark**: `--conf "spark.sql.catalog.iceberg.warehouse=s3tablescatalog/my-table-bucket"`
|
||||
- **Trino**: `iceberg.rest-catalog.prefix=my-table-bucket` (optional, can also be part of warehouse path)
|
||||
|
||||
SeaweedFS treats the Table Bucket as the root for that catalog instance. You can have multiple catalogs pointing to different table buckets.
|
||||
|
||||
---
|
||||
|
||||
## DuckDB Integration
|
||||
|
||||
DuckDB connects via the Iceberg extension using OAuth2 authentication. See the dedicated [[DuckDB Iceberg Integration]] page for setup instructions.
|
||||
|
||||
---
|
||||
|
||||
## Anonymous Access (Zero Config)
|
||||
|
||||
If SeaweedFS is running in **Zero Configuration** mode (no `-s3.config` or `-iam.config` provided), the Iceberg Catalog allows anonymous access by default. This is useful for local development and testing.
|
||||
|
||||
### Spark Configuration for Anonymous Access
|
||||
|
||||
To connect without credentials, disable SigV4 signing in the Spark configuration:
|
||||
|
||||
```bash
|
||||
spark-sql \
|
||||
--packages "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.2,org.apache.iceberg:iceberg-aws-bundle:1.7.2" \
|
||||
--conf "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" \
|
||||
--conf "spark.sql.defaultCatalog=iceberg" \
|
||||
--conf "spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog" \
|
||||
--conf "spark.sql.catalog.iceberg.type=rest" \
|
||||
--conf "spark.sql.catalog.iceberg.uri=http://localhost:8333" \
|
||||
--conf "spark.sql.catalog.iceberg.warehouse=s3tablescatalog/my-table-bucket" \
|
||||
--conf "spark.sql.catalog.iceberg.io-impl=org.apache.iceberg.aws.s3.S3FileIO" \
|
||||
--conf "spark.sql.catalog.iceberg.s3.endpoint=http://localhost:8333" \
|
||||
--conf "spark.sql.catalog.iceberg.s3.path-style-access=true" \
|
||||
--conf "spark.sql.catalog.iceberg.rest.sigv4-enabled=false"
|
||||
```
|
||||
- [[S3 Table Bucket]] - Creating and managing table buckets
|
||||
- [[S3 Tables Security]] - IAM policies for table access
|
||||
- [[S3 Table Bucket Commands]] - `weed shell` commands
|
||||
- [[Iceberg Table Maintenance]] - Compaction and cleanup
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
# Spark Iceberg Integration
|
||||
|
||||
Apache Spark connects to SeaweedFS Iceberg tables using the Iceberg Spark runtime with the `rest` catalog type and SigV4 authentication.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- SeaweedFS running with the Iceberg REST Catalog enabled (port `8181` by default)
|
||||
- A table bucket created via `weed shell` or the S3 Tables API
|
||||
- Spark 3.5+ with Iceberg packages
|
||||
|
||||
Required packages:
|
||||
- `org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.2` (match your Spark version)
|
||||
- `org.apache.iceberg:iceberg-aws-bundle:1.5.2`
|
||||
|
||||
## Configuration
|
||||
|
||||
### PySpark
|
||||
|
||||
```python
|
||||
from pyspark.sql import SparkSession
|
||||
|
||||
spark = (SparkSession.builder
|
||||
.appName("SeaweedFS Iceberg")
|
||||
.config("spark.jars.packages",
|
||||
"org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.2,"
|
||||
"org.apache.iceberg:iceberg-aws-bundle:1.5.2")
|
||||
.config("spark.sql.extensions",
|
||||
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
|
||||
.config("spark.sql.catalog.iceberg",
|
||||
"org.apache.iceberg.spark.SparkCatalog")
|
||||
.config("spark.sql.catalog.iceberg.type", "rest")
|
||||
.config("spark.sql.catalog.iceberg.uri", "http://localhost:8181")
|
||||
# SigV4 authentication
|
||||
.config("spark.sql.catalog.iceberg.rest.auth.type", "sigv4")
|
||||
.config("spark.sql.catalog.iceberg.rest.sigv4-enabled", "true")
|
||||
.config("spark.sql.catalog.iceberg.rest.signing-name", "s3")
|
||||
.config("spark.sql.catalog.iceberg.rest.access-key-id", "YOUR_ACCESS_KEY")
|
||||
.config("spark.sql.catalog.iceberg.rest.secret-access-key", "YOUR_SECRET_KEY")
|
||||
# S3 FileIO
|
||||
.config("spark.sql.catalog.iceberg.io-impl",
|
||||
"org.apache.iceberg.aws.s3.S3FileIO")
|
||||
.config("spark.sql.catalog.iceberg.s3.endpoint", "http://localhost:8333")
|
||||
.config("spark.sql.catalog.iceberg.s3.region", "us-east-1")
|
||||
.config("spark.sql.catalog.iceberg.s3.access-key", "YOUR_ACCESS_KEY")
|
||||
.config("spark.sql.catalog.iceberg.s3.secret-key", "YOUR_SECRET_KEY")
|
||||
.config("spark.sql.catalog.iceberg.s3.path-style-access", "true")
|
||||
.getOrCreate()
|
||||
)
|
||||
```
|
||||
|
||||
### spark-sql CLI
|
||||
|
||||
```bash
|
||||
spark-sql \
|
||||
--packages "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.2,org.apache.iceberg:iceberg-aws-bundle:1.5.2" \
|
||||
--conf "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" \
|
||||
--conf "spark.sql.defaultCatalog=iceberg" \
|
||||
--conf "spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog" \
|
||||
--conf "spark.sql.catalog.iceberg.type=rest" \
|
||||
--conf "spark.sql.catalog.iceberg.uri=http://localhost:8181" \
|
||||
--conf "spark.sql.catalog.iceberg.io-impl=org.apache.iceberg.aws.s3.S3FileIO" \
|
||||
--conf "spark.sql.catalog.iceberg.s3.endpoint=http://localhost:8333" \
|
||||
--conf "spark.sql.catalog.iceberg.s3.path-style-access=true" \
|
||||
--conf "spark.sql.catalog.iceberg.s3.access-key=YOUR_ACCESS_KEY" \
|
||||
--conf "spark.sql.catalog.iceberg.s3.secret-key=YOUR_SECRET_KEY" \
|
||||
--conf "spark.sql.catalog.iceberg.rest.sigv4-enabled=true" \
|
||||
--conf "spark.sql.catalog.iceberg.rest.signing-name=s3"
|
||||
```
|
||||
|
||||
## Example SQL
|
||||
|
||||
### Namespace and Table Operations
|
||||
|
||||
```sql
|
||||
-- Create a namespace
|
||||
CREATE NAMESPACE iceberg.my_namespace;
|
||||
|
||||
-- Create a table
|
||||
CREATE TABLE iceberg.my_namespace.users (
|
||||
id INT,
|
||||
name STRING,
|
||||
age INT
|
||||
) USING iceberg;
|
||||
|
||||
-- Insert data
|
||||
INSERT INTO iceberg.my_namespace.users VALUES
|
||||
(1, 'Alice', 30),
|
||||
(2, 'Bob', 25),
|
||||
(3, 'Charlie', 35);
|
||||
|
||||
-- Query
|
||||
SELECT * FROM iceberg.my_namespace.users;
|
||||
SELECT COUNT(*) FROM iceberg.my_namespace.users;
|
||||
|
||||
-- Update and delete
|
||||
UPDATE iceberg.my_namespace.users SET age = 31 WHERE id = 1;
|
||||
DELETE FROM iceberg.my_namespace.users WHERE id = 3;
|
||||
```
|
||||
|
||||
### Multi-Level Namespaces
|
||||
|
||||
```sql
|
||||
CREATE NAMESPACE iceberg.analytics;
|
||||
CREATE NAMESPACE iceberg.analytics.web;
|
||||
|
||||
CREATE TABLE iceberg.analytics.web.pageviews (
|
||||
id INT,
|
||||
url STRING,
|
||||
ts TIMESTAMP
|
||||
) USING iceberg;
|
||||
```
|
||||
|
||||
### Time Travel
|
||||
|
||||
```sql
|
||||
-- Query a table at a specific point in time
|
||||
SELECT COUNT(*) FROM iceberg.my_namespace.users
|
||||
TIMESTAMP AS OF '2024-01-15 10:30:00';
|
||||
```
|
||||
|
||||
## Anonymous Access
|
||||
|
||||
When SeaweedFS runs without IAM, disable SigV4:
|
||||
|
||||
```python
|
||||
spark = (SparkSession.builder
|
||||
.config("spark.sql.catalog.iceberg.type", "rest")
|
||||
.config("spark.sql.catalog.iceberg.uri", "http://localhost:8181")
|
||||
.config("spark.sql.catalog.iceberg.rest.sigv4-enabled", "false")
|
||||
.config("spark.sql.catalog.iceberg.io-impl",
|
||||
"org.apache.iceberg.aws.s3.S3FileIO")
|
||||
.config("spark.sql.catalog.iceberg.s3.endpoint", "http://localhost:8333")
|
||||
.config("spark.sql.catalog.iceberg.s3.path-style-access", "true")
|
||||
# ... other standard configs
|
||||
.getOrCreate()
|
||||
)
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [[SeaweedFS Iceberg Catalog]] - Architecture and concepts
|
||||
- [[S3 Table Bucket]] - Managing table buckets
|
||||
- [[run Spark on SeaweedFS]] - Spark with SeaweedFS HDFS connector (non-Iceberg)
|
||||
@@ -0,0 +1,115 @@
|
||||
# Trino Iceberg Integration
|
||||
|
||||
Trino connects to SeaweedFS Iceberg tables using the `iceberg` connector with the `rest` catalog type and SigV4 authentication.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- SeaweedFS running with the Iceberg REST Catalog enabled (port `8181` by default)
|
||||
- A table bucket created via `weed shell` or the S3 Tables API
|
||||
- Trino 4xx+ with the Iceberg connector
|
||||
|
||||
## Configuration
|
||||
|
||||
Create a catalog properties file at `etc/catalog/iceberg.properties`:
|
||||
|
||||
```properties
|
||||
connector.name=iceberg
|
||||
iceberg.catalog.type=rest
|
||||
iceberg.rest-catalog.uri=http://localhost:8181
|
||||
iceberg.rest-catalog.warehouse=s3://my-table-bucket
|
||||
|
||||
# File format
|
||||
iceberg.file-format=PARQUET
|
||||
iceberg.unique-table-location=true
|
||||
|
||||
# SigV4 authentication for the REST catalog
|
||||
iceberg.rest-catalog.security=SIGV4
|
||||
|
||||
# S3 FileIO configuration
|
||||
fs.native-s3.enabled=true
|
||||
s3.endpoint=http://localhost:8333
|
||||
s3.path-style-access=true
|
||||
s3.signer-type=AwsS3V4Signer
|
||||
s3.aws-access-key=YOUR_ACCESS_KEY
|
||||
s3.aws-secret-key=YOUR_SECRET_KEY
|
||||
s3.region=us-east-1
|
||||
```
|
||||
|
||||
### Multi-Level Namespaces
|
||||
|
||||
To use nested namespaces (e.g., `db.schema`), add:
|
||||
|
||||
```properties
|
||||
iceberg.rest-catalog.nested-namespace-enabled=true
|
||||
```
|
||||
|
||||
## Example SQL
|
||||
|
||||
### Schema Operations
|
||||
|
||||
```sql
|
||||
-- Create a schema (maps to an Iceberg namespace)
|
||||
CREATE SCHEMA IF NOT EXISTS iceberg.my_namespace;
|
||||
|
||||
-- List schemas
|
||||
SHOW SCHEMAS FROM iceberg;
|
||||
```
|
||||
|
||||
### Table Operations
|
||||
|
||||
```sql
|
||||
-- Create a table
|
||||
CREATE TABLE iceberg.my_namespace.events (
|
||||
id INTEGER,
|
||||
event VARCHAR,
|
||||
ts TIMESTAMP(6)
|
||||
) WITH (
|
||||
format = 'PARQUET'
|
||||
);
|
||||
|
||||
-- Insert data
|
||||
INSERT INTO iceberg.my_namespace.events VALUES
|
||||
(1, 'click', TIMESTAMP '2024-01-15 10:30:00'),
|
||||
(2, 'view', TIMESTAMP '2024-01-15 11:00:00');
|
||||
|
||||
-- Query
|
||||
SELECT * FROM iceberg.my_namespace.events;
|
||||
|
||||
-- Inspect data files
|
||||
SELECT file_path FROM iceberg.my_namespace."events$files" LIMIT 5;
|
||||
```
|
||||
|
||||
### Multi-Level Namespace Example
|
||||
|
||||
```sql
|
||||
CREATE SCHEMA IF NOT EXISTS iceberg."analytics.web";
|
||||
|
||||
CREATE TABLE iceberg."analytics.web".pageviews (
|
||||
id INTEGER,
|
||||
url VARCHAR,
|
||||
ts TIMESTAMP(6)
|
||||
) WITH (
|
||||
format = 'PARQUET'
|
||||
);
|
||||
```
|
||||
|
||||
## Anonymous Access
|
||||
|
||||
When SeaweedFS runs without IAM, remove the SigV4 and credential properties:
|
||||
|
||||
```properties
|
||||
connector.name=iceberg
|
||||
iceberg.catalog.type=rest
|
||||
iceberg.rest-catalog.uri=http://localhost:8181
|
||||
iceberg.rest-catalog.warehouse=s3://my-table-bucket
|
||||
|
||||
fs.native-s3.enabled=true
|
||||
s3.endpoint=http://localhost:8333
|
||||
s3.path-style-access=true
|
||||
s3.region=us-east-1
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [[SeaweedFS Iceberg Catalog]] - Architecture and concepts
|
||||
- [[S3 Table Bucket]] - Managing table buckets
|
||||
Reference in New Issue
Block a user