mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-20 13:30:46 +02:00
more details about catalog
@@ -9,6 +9,68 @@ The SeaweedFS S3 Tables feature implements the **Iceberg REST Catalog API**. Thi
|
||||
- **Endpoint**: The Iceberg REST API is available on the S3 port (default `8333`) under `/v1/`.
|
||||
- **Authentication**: Uses AWS Signature Version 4 (SigV4) with the `s3tables` service name.
|
||||
|
||||
## Catalog and Bucket Relationship
|
||||
|
||||
In SeaweedFS, an **Iceberg Catalog** corresponds 1:1 with a **Table Bucket**.
|
||||
|
||||
- When you configure a client (Spark/Trino) with a URI like `http://localhost:8333/v1/my-catalog/`, SeaweedFS maps requests to the bucket named `my-catalog`.
|
||||
- If no catalog/prefix is provided in the URL (e.g., `http://localhost:8333/v1/`), it defaults to using a bucket named `warehouse`.
|
||||
|
||||
This architecture allows you to manage multiple independent Iceberg catalogs on the same SeaweedFS cluster simply by creating multiple buckets.
|
||||
|
||||
## Metadata Storage
|
||||
|
||||
SeaweedFS stores Iceberg metadata using a hybrid approach to maximize performance and compatibility:
|
||||
|
||||
### Namespaces
|
||||
Namespace metadata (creation time, properties) is stored as **Extended Attributes (xattrs)** on the directory corresponding to the namespace in the Filer.
|
||||
- This ensures lightweight namespace operations.
|
||||
- The directory structure in the Filer mirrors the namespace hierarchy.
|
||||
|
||||
### Tables
|
||||
Table metadata follows the standard Iceberg V2 specification:
|
||||
- **Metadata Location**: Stored in the `metadata/` subdirectory of the table.
|
||||
- **Data Location**: Stored in the `data/` subdirectory.
|
||||
- **Format**:
|
||||
- `vN.metadata.json`: The table metadata file.
|
||||
- `snap-*.avro`: Snapshot manifest lists.
|
||||
- `*.avro`: Manifest files.
|
||||
- `*.parquet`: Data files.
|
||||
|
||||
## Authentication and Authorization
|
||||
|
||||
Security is managed using the standard **AWS Signature Version 4 (SigV4)** protocol, integrated with SeaweedFS's IAM system.
|
||||
|
||||
### Authentication
|
||||
- Clients must sign requests using the `s3tables` service name (not `s3`).
|
||||
- SeaweedFS validates the signature using the access key and secret key provided in the client configuration.
|
||||
|
||||
### Authorization (IAM)
|
||||
Permissions are managed via **S3 Bucket Policies** applied to the Table Bucket.
|
||||
- You can define granular permissions for `CreateNamespace`, `CreateTable`, `WriteTable`, etc.
|
||||
- Example Policy to allow read-only access:
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3tables:ListNamespaces",
|
||||
"s3tables:GetTable",
|
||||
"s3tables:ListTables"
|
||||
],
|
||||
"Resource": "arn:aws:s3tables:region:account:bucket/my-catalog/*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Signing Details
|
||||
- **Service Name**: `s3tables`
|
||||
- **Region**: Defaults to `us-east-1` (configurable)
|
||||
- **Endpoint**: The S3 API port (default `8333`)
|
||||
|
||||
---
|
||||
|
||||
## Apache Spark Integration
|
||||
|
||||
Reference in New Issue
Block a user