From 91fa250ce52e221c6a471dcc5f98283d20fddbca Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sun, 8 Feb 2026 19:37:21 -0800 Subject: [PATCH] add table bucket wiki --- S3-Iceberg-Integration.md | 88 ++++++++++++++++++++++++++ S3-Table-Bucket.md | 2 + S3-Tables-Security.md | 130 ++++++++++++++++++++++++++++++++++++++ _Sidebar.md | 2 + 4 files changed, 222 insertions(+) create mode 100644 S3-Iceberg-Integration.md create mode 100644 S3-Tables-Security.md diff --git a/S3-Iceberg-Integration.md b/S3-Iceberg-Integration.md new file mode 100644 index 0000000..117d436 --- /dev/null +++ b/S3-Iceberg-Integration.md @@ -0,0 +1,88 @@ +# S3 Iceberg Integration + +SeaweedFS provides a built-in Iceberg REST Catalog that can be used with popular analytics engines like Apache Spark and Trino. + +## Architecture + +The SeaweedFS S3 Tables feature implements the **Iceberg REST Catalog API**. This allows clients to talk directly to SeaweedFS to manage Iceberg namespaces and tables, while the underlying data files (Parquet, Avro, Metadata JSON) are stored in SeaweedFS S3 buckets. + +- **Endpoint**: The Iceberg REST API is available on the S3 port (default `8333`) under `/v1/`. +- **Authentication**: Uses AWS Signature Version 4 (SigV4) with the `s3tables` service name. + +--- + +## Apache Spark Integration + +To connect Spark to SeaweedFS S3 Tables, you need the Iceberg Spark runtime and the AWS bundle for S3 support. + +### Prerequisites + +Required packages: +- `org.apache.iceberg:iceberg-spark-runtime-3.5_2.12` (or matching your Spark version) +- `org.apache.iceberg:iceberg-aws-bundle` + +### Configuration + +Use the `rest` catalog type and configure SigV4 signing. + +**Example `spark-sql` command:** + +```bash +spark-sql \ + --packages "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.2,org.apache.iceberg:iceberg-aws-bundle:1.7.2" \ + --conf "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" \ + --conf "spark.sql.defaultCatalog=iceberg" \ + --conf "spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog" \ + --conf "spark.sql.catalog.iceberg.type=rest" \ + --conf "spark.sql.catalog.iceberg.uri=http://localhost:8333" \ + --conf "spark.sql.catalog.iceberg.warehouse=s3tablescatalog/my-table-bucket" \ + --conf "spark.sql.catalog.iceberg.io-impl=org.apache.iceberg.aws.s3.S3FileIO" \ + --conf "spark.sql.catalog.iceberg.s3.endpoint=http://localhost:8333" \ + --conf "spark.sql.catalog.iceberg.s3.path-style-access=true" \ + --conf "spark.sql.catalog.iceberg.s3.access-key-id=YOUR_ACCESS_KEY" \ + --conf "spark.sql.catalog.iceberg.s3.secret-access-key=YOUR_SECRET_KEY" \ + --conf "spark.sql.catalog.iceberg.rest.sigv4-enabled=true" \ + --conf "spark.sql.catalog.iceberg.rest.signing-name=s3tables" \ + --conf "spark.sql.catalog.iceberg.rest.signing-region=us-east-1" +``` + +> [!NOTE] +> The `warehouse` property should be in the format `s3tablescatalog/`. + +--- + +## Trino Integration + +Trino connects via the `iceberg` connector with the `rest` catalog type. + +### Configuration (`etc/catalog/iceberg.properties`) + +```properties +connector.name=iceberg +iceberg.catalog.type=rest +iceberg.rest-catalog.uri=http://localhost:8333 +iceberg.rest-catalog.warehouse=s3tablescatalog/my-table-bucket + +# Enable SigV4 authentication for the REST catalog +iceberg.rest-catalog.security=SIGV4 +iceberg.rest-catalog.signing-name=s3tables +iceberg.rest-catalog.signing-region=us-east-1 + +# S3 FileIO Configuration +fs.native-s3.enabled=true +s3.endpoint=http://localhost:8333 +s3.path-style-access=true +s3.signer-type=AwsS3V4Signer +s3.aws-access-key=YOUR_ACCESS_KEY +s3.aws-secret-key=YOUR_SECRET_KEY +``` + +--- + +## Notes on Table Buckets + +When configuring the catalog, the `warehouse` location typically points to a specific **Table Bucket**. +- **Spark**: `--conf "spark.sql.catalog.iceberg.warehouse=s3tablescatalog/my-table-bucket"` +- **Trino**: `iceberg.rest-catalog.prefix=my-table-bucket` (optional, can also be part of warehouse path) + +SeaweedFS treats the Table Bucket as the root for that catalog instance. You can have multiple catalogs pointing to different table buckets. diff --git a/S3-Table-Bucket.md b/S3-Table-Bucket.md index 3fd6e3c..c97d4ff 100644 --- a/S3-Table-Bucket.md +++ b/S3-Table-Bucket.md @@ -17,3 +17,5 @@ SeaweedFS is adding support for Amazon S3 Tables, which provides a way to manage ## Documentation - [S3 Table Bucket Commands](S3-Table-Bucket-Commands) - Examples of how to manage table buckets, namespaces, and tables. +- [S3 Iceberg Integration](S3-Iceberg-Integration) - Usage with Apache Spark and Trino. +- [S3 Tables Security](S3-Tables-Security) - IAM permissions and policy reference. diff --git a/S3-Tables-Security.md b/S3-Tables-Security.md new file mode 100644 index 0000000..042938b --- /dev/null +++ b/S3-Tables-Security.md @@ -0,0 +1,130 @@ +# S3 Tables Security + +SeaweedFS S3 Tables implementation supports fine-grained access control using IAM-style policies. These policies can be attached to **Table Buckets** to control access to namespaces and tables. + +## Permission Model + +Permissions are defined using JSON policy documents similar to AWS IAM. Policies are attached to the **Table Bucket** using the `PutTableBucketPolicy` API. + +### Policy Structure + +A policy document consists of a list of statements: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": "*", + "Action": "s3tables:CreateTable", + "Resource": "*" + } + ] +} +``` + +- **Effect**: `Allow` or `Deny`. Deny always takes precedence. +- **Principal**: The user or role (e.g., `*`, `arn:aws:iam::123456789012:user/admin`). +- **Action**: The API operation to allow/deny (supports wildcards). +- **Resource**: The resource ARN (supports wildcards). +- **Condition**: (Optional) extra conditions for the rule. + +--- + +## Supported Actions + +The following actions are supported in policies. Actions can be specified with the `s3tables:` prefix or without it (for some contexts). + +### Table Bucket Operations +- `s3tables:CreateTableBucket` +- `s3tables:DeleteTableBucket` +- `s3tables:GetTableBucket` +- `s3tables:ListTableBuckets` +- `s3tables:PutTableBucketPolicy` +- `s3tables:GetTableBucketPolicy` +- `s3tables:DeleteTableBucketPolicy` + +### Namespace Operations +- `s3tables:CreateNamespace` +- `s3tables:DeleteNamespace` +- `s3tables:GetNamespace` +- `s3tables:ListNamespaces` + +### Table Operations +- `s3tables:CreateTable` +- `s3tables:DeleteTable` +- `s3tables:GetTable` +- `s3tables:ListTables` +- `s3tables:PutTablePolicy` +- `s3tables:GetTablePolicy` +- `s3tables:DeleteTablePolicy` + +--- + +## Example Policies + +### 1. Read-Only Access +Allow listing and getting tables, but no modifications. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": "*", + "Action": [ + "s3tables:Get*", + "s3tables:List*" + ], + "Resource": "*" + } + ] +} +``` + +### 2. Admin Access for Specific Namespace +Allow full access only to tables within the `finance` namespace. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": "arn:aws:iam::123456789012:user/finance-admin", + "Action": "s3tables:*", + "Resource": "*", + "Condition": { + "StringEquals": { + "s3tables:namespace": "finance" + } + } + } + ] +} +``` + +### 3. Deny Deletion +Allow everything except deleting tables. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": "*", + "Action": "s3tables:*", + "Resource": "*" + }, + { + "Effect": "Deny", + "Principal": "*", + "Action": "s3tables:DeleteTable", + "Resource": "*" + } + ] +} +``` diff --git a/_Sidebar.md b/_Sidebar.md index d4ba785..45ec6ad 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -84,6 +84,8 @@ ### S3 Table Bucket * [[S3 Table Bucket]] +* [[S3 Iceberg Integration]] +* [[S3 Tables Security]] ### S3 Authentication & IAM * [[S3 Configuration]] - Start Here