more details about catalog

Chris Lu
2026-02-11 21:58:38 -08:00
parent 691bf39355
commit 82a66794c6
+62
@@ -9,6 +9,68 @@ The SeaweedFS S3 Tables feature implements the **Iceberg REST Catalog API**. Thi
- **Endpoint**: The Iceberg REST API is available on the S3 port (default `8333`) under `/v1/`.
- **Authentication**: Uses AWS Signature Version 4 (SigV4) with the `s3tables` service name.
## Catalog and Bucket Relationship
In SeaweedFS, an **Iceberg Catalog** corresponds 1:1 with a **Table Bucket**.
- When you configure a client (Spark/Trino) with a URI like `http://localhost:8333/v1/my-catalog/`, SeaweedFS maps requests to the bucket named `my-catalog`.
- If no catalog/prefix is provided in the URL (e.g., `http://localhost:8333/v1/`), it defaults to using a bucket named `warehouse`.
This architecture allows you to manage multiple independent Iceberg catalogs on the same SeaweedFS cluster simply by creating multiple buckets.
## Metadata Storage
SeaweedFS stores Iceberg metadata using a hybrid approach to maximize performance and compatibility:
### Namespaces
Namespace metadata (creation time, properties) is stored as **Extended Attributes (xattrs)** on the directory corresponding to the namespace in the Filer.
- This ensures lightweight namespace operations.
- The directory structure in the Filer mirrors the namespace hierarchy.
### Tables
Table metadata follows the standard Iceberg V2 specification:
- **Metadata Location**: Stored in the `metadata/` subdirectory of the table.
- **Data Location**: Stored in the `data/` subdirectory.
- **Format**:
- `vN.metadata.json`: The table metadata file.
- `snap-*.avro`: Snapshot manifest lists.
- `*.avro`: Manifest files.
- `*.parquet`: Data files.
## Authentication and Authorization
Security is managed using the standard **AWS Signature Version 4 (SigV4)** protocol, integrated with SeaweedFS's IAM system.
### Authentication
- Clients must sign requests using the `s3tables` service name (not `s3`).
- SeaweedFS validates the signature using the access key and secret key provided in the client configuration.
### Authorization (IAM)
Permissions are managed via **S3 Bucket Policies** applied to the Table Bucket.
- You can define granular permissions for `CreateNamespace`, `CreateTable`, `WriteTable`, etc.
- Example Policy to allow read-only access:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3tables:ListNamespaces",
"s3tables:GetTable",
"s3tables:ListTables"
],
"Resource": "arn:aws:s3tables:region:account:bucket/my-catalog/*"
}
]
}
```
### Signing Details
- **Service Name**: `s3tables`
- **Region**: Defaults to `us-east-1` (configurable)
- **Endpoint**: The S3 API port (default `8333`)
---
## Apache Spark Integration