add table bucket wiki

Chris Lu committed 2026-02-08 19:37:21 -08:00
1 parent 7890523a74
commit 91fa250ce5
4 files changed
+222

No files matched your search

+88
@@ -0,0 +1,88 @@
# S3 Iceberg Integration
SeaweedFS provides a built-in Iceberg REST Catalog that can be used with popular analytics engines like Apache Spark and Trino.
## Architecture
The SeaweedFS S3 Tables feature implements the **Iceberg REST Catalog API**. This allows clients to talk directly to SeaweedFS to manage Iceberg namespaces and tables, while the underlying data files (Parquet, Avro, Metadata JSON) are stored in SeaweedFS S3 buckets.
- **Endpoint**: The Iceberg REST API is available on the S3 port (default `8333`) under `/v1/`.
- **Authentication**: Uses AWS Signature Version 4 (SigV4) with the `s3tables` service name.
---
## Apache Spark Integration
To connect Spark to SeaweedFS S3 Tables, you need the Iceberg Spark runtime and the AWS bundle for S3 support.
### Prerequisites
Required packages:
- `org.apache.iceberg:iceberg-spark-runtime-3.5_2.12` (or matching your Spark version)
- `org.apache.iceberg:iceberg-aws-bundle`
### Configuration
Use the `rest` catalog type and configure SigV4 signing.
**Example `spark-sql` command:**
```bash
spark-sql \
--packages "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.2,org.apache.iceberg:iceberg-aws-bundle:1.7.2" \
--conf "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" \
--conf "spark.sql.defaultCatalog=iceberg" \
--conf "spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog" \
--conf "spark.sql.catalog.iceberg.type=rest" \
--conf "spark.sql.catalog.iceberg.uri=http://localhost:8333" \
--conf "spark.sql.catalog.iceberg.warehouse=s3tablescatalog/my-table-bucket" \
--conf "spark.sql.catalog.iceberg.io-impl=org.apache.iceberg.aws.s3.S3FileIO" \
--conf "spark.sql.catalog.iceberg.s3.endpoint=http://localhost:8333" \
--conf "spark.sql.catalog.iceberg.s3.path-style-access=true" \
--conf "spark.sql.catalog.iceberg.s3.access-key-id=YOUR_ACCESS_KEY" \
--conf "spark.sql.catalog.iceberg.s3.secret-access-key=YOUR_SECRET_KEY" \
--conf "spark.sql.catalog.iceberg.rest.sigv4-enabled=true" \
--conf "spark.sql.catalog.iceberg.rest.signing-name=s3tables" \
--conf "spark.sql.catalog.iceberg.rest.signing-region=us-east-1"
```
> [!NOTE]
> The `warehouse` property should be in the format `s3tablescatalog/<table-bucket-name>`.
---
## Trino Integration
Trino connects via the `iceberg` connector with the `rest` catalog type.
### Configuration (`etc/catalog/iceberg.properties`)
```properties
connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http://localhost:8333
iceberg.rest-catalog.warehouse=s3tablescatalog/my-table-bucket
# Enable SigV4 authentication for the REST catalog
iceberg.rest-catalog.security=SIGV4
iceberg.rest-catalog.signing-name=s3tables
iceberg.rest-catalog.signing-region=us-east-1
# S3 FileIO Configuration
fs.native-s3.enabled=true
s3.endpoint=http://localhost:8333
s3.path-style-access=true
s3.signer-type=AwsS3V4Signer
s3.aws-access-key=YOUR_ACCESS_KEY
s3.aws-secret-key=YOUR_SECRET_KEY
```
---
## Notes on Table Buckets
When configuring the catalog, the `warehouse` location typically points to a specific **Table Bucket**.
- **Spark**: `--conf "spark.sql.catalog.iceberg.warehouse=s3tablescatalog/my-table-bucket"`
- **Trino**: `iceberg.rest-catalog.prefix=my-table-bucket` (optional, can also be part of warehouse path)
SeaweedFS treats the Table Bucket as the root for that catalog instance. You can have multiple catalogs pointing to different table buckets.
+2
@@ -17,3 +17,5 @@ SeaweedFS is adding support for Amazon S3 Tables, which provides a way to manage
## Documentation
- [S3 Table Bucket Commands](S3-Table-Bucket-Commands) - Examples of how to manage table buckets, namespaces, and tables.
- [S3 Iceberg Integration](S3-Iceberg-Integration) - Usage with Apache Spark and Trino.
- [S3 Tables Security](S3-Tables-Security) - IAM permissions and policy reference.
+130
@@ -0,0 +1,130 @@
# S3 Tables Security
SeaweedFS S3 Tables implementation supports fine-grained access control using IAM-style policies. These policies can be attached to **Table Buckets** to control access to namespaces and tables.
## Permission Model
Permissions are defined using JSON policy documents similar to AWS IAM. Policies are attached to the **Table Bucket** using the `PutTableBucketPolicy` API.
### Policy Structure
A policy document consists of a list of statements:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3tables:CreateTable",
"Resource": "*"
}
]
}
```
- **Effect**: `Allow` or `Deny`. Deny always takes precedence.
- **Principal**: The user or role (e.g., `*`, `arn:aws:iam::123456789012:user/admin`).
- **Action**: The API operation to allow/deny (supports wildcards).
- **Resource**: The resource ARN (supports wildcards).
- **Condition**: (Optional) extra conditions for the rule.
---
## Supported Actions
The following actions are supported in policies. Actions can be specified with the `s3tables:` prefix or without it (for some contexts).
### Table Bucket Operations
- `s3tables:CreateTableBucket`
- `s3tables:DeleteTableBucket`
- `s3tables:GetTableBucket`
- `s3tables:ListTableBuckets`
- `s3tables:PutTableBucketPolicy`
- `s3tables:GetTableBucketPolicy`
- `s3tables:DeleteTableBucketPolicy`
### Namespace Operations
- `s3tables:CreateNamespace`
- `s3tables:DeleteNamespace`
- `s3tables:GetNamespace`
- `s3tables:ListNamespaces`
### Table Operations
- `s3tables:CreateTable`
- `s3tables:DeleteTable`
- `s3tables:GetTable`
- `s3tables:ListTables`
- `s3tables:PutTablePolicy`
- `s3tables:GetTablePolicy`
- `s3tables:DeleteTablePolicy`
---
## Example Policies
### 1. Read-Only Access
Allow listing and getting tables, but no modifications.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": [
"s3tables:Get*",
"s3tables:List*"
],
"Resource": "*"
}
]
}
```
### 2. Admin Access for Specific Namespace
Allow full access only to tables within the `finance` namespace.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "arn:aws:iam::123456789012:user/finance-admin",
"Action": "s3tables:*",
"Resource": "*",
"Condition": {
"StringEquals": {
"s3tables:namespace": "finance"
}
}
}
]
}
```
### 3. Deny Deletion
Allow everything except deleting tables.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3tables:*",
"Resource": "*"
},
{
"Effect": "Deny",
"Principal": "*",
"Action": "s3tables:DeleteTable",
"Resource": "*"
}
]
}
```
+2
@@ -84,6 +84,8 @@
### S3 Table Bucket
* [[S3 Table Bucket]]
* [[S3 Iceberg Integration]]
* [[S3 Tables Security]]
### S3 Authentication & IAM
* [[S3 Configuration]] - Start Here