mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-20 21:14:25 +02:00
add table bucket wiki
1 parent
7890523a74
commit
91fa250ce5
4 files changed
+222
No files matched your search
@@ -0,0 +1,88 @@
|
||||
# S3 Iceberg Integration
|
||||
|
||||
SeaweedFS provides a built-in Iceberg REST Catalog that can be used with popular analytics engines like Apache Spark and Trino.
|
||||
|
||||
## Architecture
|
||||
|
||||
The SeaweedFS S3 Tables feature implements the **Iceberg REST Catalog API**. This allows clients to talk directly to SeaweedFS to manage Iceberg namespaces and tables, while the underlying data files (Parquet, Avro, Metadata JSON) are stored in SeaweedFS S3 buckets.
|
||||
|
||||
- **Endpoint**: The Iceberg REST API is available on the S3 port (default `8333`) under `/v1/`.
|
||||
- **Authentication**: Uses AWS Signature Version 4 (SigV4) with the `s3tables` service name.
|
||||
|
||||
---
|
||||
|
||||
## Apache Spark Integration
|
||||
|
||||
To connect Spark to SeaweedFS S3 Tables, you need the Iceberg Spark runtime and the AWS bundle for S3 support.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Required packages:
|
||||
- `org.apache.iceberg:iceberg-spark-runtime-3.5_2.12` (or matching your Spark version)
|
||||
- `org.apache.iceberg:iceberg-aws-bundle`
|
||||
|
||||
### Configuration
|
||||
|
||||
Use the `rest` catalog type and configure SigV4 signing.
|
||||
|
||||
**Example `spark-sql` command:**
|
||||
|
||||
```bash
|
||||
spark-sql \
|
||||
--packages "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.2,org.apache.iceberg:iceberg-aws-bundle:1.7.2" \
|
||||
--conf "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" \
|
||||
--conf "spark.sql.defaultCatalog=iceberg" \
|
||||
--conf "spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog" \
|
||||
--conf "spark.sql.catalog.iceberg.type=rest" \
|
||||
--conf "spark.sql.catalog.iceberg.uri=http://localhost:8333" \
|
||||
--conf "spark.sql.catalog.iceberg.warehouse=s3tablescatalog/my-table-bucket" \
|
||||
--conf "spark.sql.catalog.iceberg.io-impl=org.apache.iceberg.aws.s3.S3FileIO" \
|
||||
--conf "spark.sql.catalog.iceberg.s3.endpoint=http://localhost:8333" \
|
||||
--conf "spark.sql.catalog.iceberg.s3.path-style-access=true" \
|
||||
--conf "spark.sql.catalog.iceberg.s3.access-key-id=YOUR_ACCESS_KEY" \
|
||||
--conf "spark.sql.catalog.iceberg.s3.secret-access-key=YOUR_SECRET_KEY" \
|
||||
--conf "spark.sql.catalog.iceberg.rest.sigv4-enabled=true" \
|
||||
--conf "spark.sql.catalog.iceberg.rest.signing-name=s3tables" \
|
||||
--conf "spark.sql.catalog.iceberg.rest.signing-region=us-east-1"
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> The `warehouse` property should be in the format `s3tablescatalog/<table-bucket-name>`.
|
||||
|
||||
---
|
||||
|
||||
## Trino Integration
|
||||
|
||||
Trino connects via the `iceberg` connector with the `rest` catalog type.
|
||||
|
||||
### Configuration (`etc/catalog/iceberg.properties`)
|
||||
|
||||
```properties
|
||||
connector.name=iceberg
|
||||
iceberg.catalog.type=rest
|
||||
iceberg.rest-catalog.uri=http://localhost:8333
|
||||
iceberg.rest-catalog.warehouse=s3tablescatalog/my-table-bucket
|
||||
|
||||
# Enable SigV4 authentication for the REST catalog
|
||||
iceberg.rest-catalog.security=SIGV4
|
||||
iceberg.rest-catalog.signing-name=s3tables
|
||||
iceberg.rest-catalog.signing-region=us-east-1
|
||||
|
||||
# S3 FileIO Configuration
|
||||
fs.native-s3.enabled=true
|
||||
s3.endpoint=http://localhost:8333
|
||||
s3.path-style-access=true
|
||||
s3.signer-type=AwsS3V4Signer
|
||||
s3.aws-access-key=YOUR_ACCESS_KEY
|
||||
s3.aws-secret-key=YOUR_SECRET_KEY
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Notes on Table Buckets
|
||||
|
||||
When configuring the catalog, the `warehouse` location typically points to a specific **Table Bucket**.
|
||||
- **Spark**: `--conf "spark.sql.catalog.iceberg.warehouse=s3tablescatalog/my-table-bucket"`
|
||||
- **Trino**: `iceberg.rest-catalog.prefix=my-table-bucket` (optional, can also be part of warehouse path)
|
||||
|
||||
SeaweedFS treats the Table Bucket as the root for that catalog instance. You can have multiple catalogs pointing to different table buckets.
|
||||
@@ -17,3 +17,5 @@ SeaweedFS is adding support for Amazon S3 Tables, which provides a way to manage
|
||||
## Documentation
|
||||
|
||||
- [S3 Table Bucket Commands](S3-Table-Bucket-Commands) - Examples of how to manage table buckets, namespaces, and tables.
|
||||
- [S3 Iceberg Integration](S3-Iceberg-Integration) - Usage with Apache Spark and Trino.
|
||||
- [S3 Tables Security](S3-Tables-Security) - IAM permissions and policy reference.
|
||||
@@ -0,0 +1,130 @@
|
||||
# S3 Tables Security
|
||||
|
||||
SeaweedFS S3 Tables implementation supports fine-grained access control using IAM-style policies. These policies can be attached to **Table Buckets** to control access to namespaces and tables.
|
||||
|
||||
## Permission Model
|
||||
|
||||
Permissions are defined using JSON policy documents similar to AWS IAM. Policies are attached to the **Table Bucket** using the `PutTableBucketPolicy` API.
|
||||
|
||||
### Policy Structure
|
||||
|
||||
A policy document consists of a list of statements:
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Principal": "*",
|
||||
"Action": "s3tables:CreateTable",
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- **Effect**: `Allow` or `Deny`. Deny always takes precedence.
|
||||
- **Principal**: The user or role (e.g., `*`, `arn:aws:iam::123456789012:user/admin`).
|
||||
- **Action**: The API operation to allow/deny (supports wildcards).
|
||||
- **Resource**: The resource ARN (supports wildcards).
|
||||
- **Condition**: (Optional) extra conditions for the rule.
|
||||
|
||||
---
|
||||
|
||||
## Supported Actions
|
||||
|
||||
The following actions are supported in policies. Actions can be specified with the `s3tables:` prefix or without it (for some contexts).
|
||||
|
||||
### Table Bucket Operations
|
||||
- `s3tables:CreateTableBucket`
|
||||
- `s3tables:DeleteTableBucket`
|
||||
- `s3tables:GetTableBucket`
|
||||
- `s3tables:ListTableBuckets`
|
||||
- `s3tables:PutTableBucketPolicy`
|
||||
- `s3tables:GetTableBucketPolicy`
|
||||
- `s3tables:DeleteTableBucketPolicy`
|
||||
|
||||
### Namespace Operations
|
||||
- `s3tables:CreateNamespace`
|
||||
- `s3tables:DeleteNamespace`
|
||||
- `s3tables:GetNamespace`
|
||||
- `s3tables:ListNamespaces`
|
||||
|
||||
### Table Operations
|
||||
- `s3tables:CreateTable`
|
||||
- `s3tables:DeleteTable`
|
||||
- `s3tables:GetTable`
|
||||
- `s3tables:ListTables`
|
||||
- `s3tables:PutTablePolicy`
|
||||
- `s3tables:GetTablePolicy`
|
||||
- `s3tables:DeleteTablePolicy`
|
||||
|
||||
---
|
||||
|
||||
## Example Policies
|
||||
|
||||
### 1. Read-Only Access
|
||||
Allow listing and getting tables, but no modifications.
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Principal": "*",
|
||||
"Action": [
|
||||
"s3tables:Get*",
|
||||
"s3tables:List*"
|
||||
],
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Admin Access for Specific Namespace
|
||||
Allow full access only to tables within the `finance` namespace.
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Principal": "arn:aws:iam::123456789012:user/finance-admin",
|
||||
"Action": "s3tables:*",
|
||||
"Resource": "*",
|
||||
"Condition": {
|
||||
"StringEquals": {
|
||||
"s3tables:namespace": "finance"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Deny Deletion
|
||||
Allow everything except deleting tables.
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Principal": "*",
|
||||
"Action": "s3tables:*",
|
||||
"Resource": "*"
|
||||
},
|
||||
{
|
||||
"Effect": "Deny",
|
||||
"Principal": "*",
|
||||
"Action": "s3tables:DeleteTable",
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
+2
@@ -84,6 +84,8 @@
|
||||
|
||||
### S3 Table Bucket
|
||||
* [[S3 Table Bucket]]
|
||||
* [[S3 Iceberg Integration]]
|
||||
* [[S3 Tables Security]]
|
||||
|
||||
### S3 Authentication & IAM
|
||||
* [[S3 Configuration]] - Start Here
|
||||
|
||||
Reference in new issue
Block a user