From 82a66794c6a026abb7a4ecc27052d15935ca0985 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Wed, 11 Feb 2026 21:58:38 -0800 Subject: [PATCH] more details about catalog --- SeaweedFS-Iceberg-Catalog.md | 62 ++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/SeaweedFS-Iceberg-Catalog.md b/SeaweedFS-Iceberg-Catalog.md index aa8c96b..668fbf3 100644 --- a/SeaweedFS-Iceberg-Catalog.md +++ b/SeaweedFS-Iceberg-Catalog.md @@ -9,6 +9,68 @@ The SeaweedFS S3 Tables feature implements the **Iceberg REST Catalog API**. Thi - **Endpoint**: The Iceberg REST API is available on the S3 port (default `8333`) under `/v1/`. - **Authentication**: Uses AWS Signature Version 4 (SigV4) with the `s3tables` service name. +## Catalog and Bucket Relationship + +In SeaweedFS, an **Iceberg Catalog** corresponds 1:1 with a **Table Bucket**. + +- When you configure a client (Spark/Trino) with a URI like `http://localhost:8333/v1/my-catalog/`, SeaweedFS maps requests to the bucket named `my-catalog`. +- If no catalog/prefix is provided in the URL (e.g., `http://localhost:8333/v1/`), it defaults to using a bucket named `warehouse`. + +This architecture allows you to manage multiple independent Iceberg catalogs on the same SeaweedFS cluster simply by creating multiple buckets. + +## Metadata Storage + +SeaweedFS stores Iceberg metadata using a hybrid approach to maximize performance and compatibility: + +### Namespaces +Namespace metadata (creation time, properties) is stored as **Extended Attributes (xattrs)** on the directory corresponding to the namespace in the Filer. +- This ensures lightweight namespace operations. +- The directory structure in the Filer mirrors the namespace hierarchy. + +### Tables +Table metadata follows the standard Iceberg V2 specification: +- **Metadata Location**: Stored in the `metadata/` subdirectory of the table. +- **Data Location**: Stored in the `data/` subdirectory. +- **Format**: + - `vN.metadata.json`: The table metadata file. + - `snap-*.avro`: Snapshot manifest lists. + - `*.avro`: Manifest files. + - `*.parquet`: Data files. + +## Authentication and Authorization + +Security is managed using the standard **AWS Signature Version 4 (SigV4)** protocol, integrated with SeaweedFS's IAM system. + +### Authentication +- Clients must sign requests using the `s3tables` service name (not `s3`). +- SeaweedFS validates the signature using the access key and secret key provided in the client configuration. + +### Authorization (IAM) +Permissions are managed via **S3 Bucket Policies** applied to the Table Bucket. +- You can define granular permissions for `CreateNamespace`, `CreateTable`, `WriteTable`, etc. +- Example Policy to allow read-only access: + ```json + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3tables:ListNamespaces", + "s3tables:GetTable", + "s3tables:ListTables" + ], + "Resource": "arn:aws:s3tables:region:account:bucket/my-catalog/*" + } + ] + } + ``` + +### Signing Details +- **Service Name**: `s3tables` +- **Region**: Defaults to `us-east-1` (configurable) +- **Endpoint**: The S3 API port (default `8333`) + --- ## Apache Spark Integration