mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-18 03:58:32 +02:00
more details about catalog
1 parent
691bf39355
commit
82a66794c6
1 file changed
+62
@@ -9,6 +9,68 @@ The SeaweedFS S3 Tables feature implements the **Iceberg REST Catalog API**. Thi
|
|||||||
- **Endpoint**: The Iceberg REST API is available on the S3 port (default `8333`) under `/v1/`.
|
- **Endpoint**: The Iceberg REST API is available on the S3 port (default `8333`) under `/v1/`.
|
||||||
- **Authentication**: Uses AWS Signature Version 4 (SigV4) with the `s3tables` service name.
|
- **Authentication**: Uses AWS Signature Version 4 (SigV4) with the `s3tables` service name.
|
||||||
|
|
||||||
|
## Catalog and Bucket Relationship
|
||||||
|
|
||||||
|
In SeaweedFS, an **Iceberg Catalog** corresponds 1:1 with a **Table Bucket**.
|
||||||
|
|
||||||
|
- When you configure a client (Spark/Trino) with a URI like `http://localhost:8333/v1/my-catalog/`, SeaweedFS maps requests to the bucket named `my-catalog`.
|
||||||
|
- If no catalog/prefix is provided in the URL (e.g., `http://localhost:8333/v1/`), it defaults to using a bucket named `warehouse`.
|
||||||
|
|
||||||
|
This architecture allows you to manage multiple independent Iceberg catalogs on the same SeaweedFS cluster simply by creating multiple buckets.
|
||||||
|
|
||||||
|
## Metadata Storage
|
||||||
|
|
||||||
|
SeaweedFS stores Iceberg metadata using a hybrid approach to maximize performance and compatibility:
|
||||||
|
|
||||||
|
### Namespaces
|
||||||
|
Namespace metadata (creation time, properties) is stored as **Extended Attributes (xattrs)** on the directory corresponding to the namespace in the Filer.
|
||||||
|
- This ensures lightweight namespace operations.
|
||||||
|
- The directory structure in the Filer mirrors the namespace hierarchy.
|
||||||
|
|
||||||
|
### Tables
|
||||||
|
Table metadata follows the standard Iceberg V2 specification:
|
||||||
|
- **Metadata Location**: Stored in the `metadata/` subdirectory of the table.
|
||||||
|
- **Data Location**: Stored in the `data/` subdirectory.
|
||||||
|
- **Format**:
|
||||||
|
- `vN.metadata.json`: The table metadata file.
|
||||||
|
- `snap-*.avro`: Snapshot manifest lists.
|
||||||
|
- `*.avro`: Manifest files.
|
||||||
|
- `*.parquet`: Data files.
|
||||||
|
|
||||||
|
## Authentication and Authorization
|
||||||
|
|
||||||
|
Security is managed using the standard **AWS Signature Version 4 (SigV4)** protocol, integrated with SeaweedFS's IAM system.
|
||||||
|
|
||||||
|
### Authentication
|
||||||
|
- Clients must sign requests using the `s3tables` service name (not `s3`).
|
||||||
|
- SeaweedFS validates the signature using the access key and secret key provided in the client configuration.
|
||||||
|
|
||||||
|
### Authorization (IAM)
|
||||||
|
Permissions are managed via **S3 Bucket Policies** applied to the Table Bucket.
|
||||||
|
- You can define granular permissions for `CreateNamespace`, `CreateTable`, `WriteTable`, etc.
|
||||||
|
- Example Policy to allow read-only access:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"Version": "2012-10-17",
|
||||||
|
"Statement": [
|
||||||
|
{
|
||||||
|
"Effect": "Allow",
|
||||||
|
"Action": [
|
||||||
|
"s3tables:ListNamespaces",
|
||||||
|
"s3tables:GetTable",
|
||||||
|
"s3tables:ListTables"
|
||||||
|
],
|
||||||
|
"Resource": "arn:aws:s3tables:region:account:bucket/my-catalog/*"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Signing Details
|
||||||
|
- **Service Name**: `s3tables`
|
||||||
|
- **Region**: Defaults to `us-east-1` (configurable)
|
||||||
|
- **Endpoint**: The S3 API port (default `8333`)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Apache Spark Integration
|
## Apache Spark Integration
|
||||||
|
|||||||
Reference in new issue
Block a user