diff --git a/S3-Table-Bucket.md b/S3-Table-Bucket.md index 59c427a..9d289c2 100644 --- a/S3-Table-Bucket.md +++ b/S3-Table-Bucket.md @@ -1,21 +1,129 @@ # S3 Tables -> [!IMPORTANT] -> **Work in Progress** -> This feature is currently under development. - ## Introduction -SeaweedFS is adding support for Amazon S3 Tables, which provides a way to manage structured datasets using the Apache Iceberg table format. S3 Tables combine the scalability of S3 with the performance and features of dedicated table formats. +SeaweedFS supports **Amazon S3 Tables**, providing a dedicated interface for managing structured datasets using the **Apache Iceberg** table format. Unlike standard S3 buckets that store unstructured objects, S3 Table Buckets are optimized for analytics workloads, offering a hierarchical structure of **Namespaces** and **Tables**. + +This feature implements the **Iceberg REST Catalog API**, allowing direct integration with analytics engines like Apache Spark, Trino, and Flink without needing an external catalog service. ## Key Features -- **Apache Iceberg Support**: Native support for Iceberg tables. -- **Table Buckets**: Dedicated buckets for storing your tables. -- **Namespaces**: Logical grouping of tables within a table bucket. +- **Dedicated Management**: Separate API namespace (`s3tables`) for managing table buckets, distinct from standard S3 bucket operations. +- **Hierarchical Structure**: Organizes data into a `Bucket -> Namespace -> Table` hierarchy, scalable for large data lakes. +- **Built-in Iceberg Support**: Native validation for Iceberg file layouts, ensuring data integrity. +- **Granular Security**: Apply IAM policies at the Bucket, Namespace, or Table level for fine-grained access control. +- **Performance Optimized**: Designed for high-throughput analytics workloads. + +## Table Buckets vs. Normal Buckets + +| Feature | S3 Table Bucket | Standard S3 Bucket | +| :--- | :--- | :--- | +| **Primary Use Case** | Structured Data Lake (Iceberg Tables) | General Object Storage (Files, Media, Backups) | +| **Data Structure** | Hierarchical: `Namespace` -> `Table` | Flat or Directory-like: `Prefix` / `Object` | +| **API** | S3 Tables API & Iceberg REST Catalog | Standard S3 API | +| **Validation** | **Strict Iceberg Layout** (enforced on write) | None (accepts any file content) | +| **Versioning** | Managed via Iceberg Snapshots | S3 Object Versioning | +| **Access Control** | Policies per Table/Namespace | Bucket Policies & ACLs | + +## Architecture & File Layout + +S3 Table Buckets enforce a strict directory structure to maintain compatibility with the Iceberg specification. This ensures that all data stored in a table bucket is valid and queryable. + +### Logical Hierarchy + +```mermaid +graph TD + TB[Table Bucket] --> NS1[Namespace: sales] + TB --> NS2[Namespace: marketing] + NS1 --> T1[Table: orders] + NS1 --> T2[Table: customers] + NS2 --> T3[Table: campaigns] +``` + +### Physical File Layout + +Inside each **Table**, SeaweedFS enforces the following file layout. Writes that do not conform to this structure will be rejected. + +```mermaid +graph TD + Table[Table Root] --> Metadata[metadata/] + Table --> Data[data/] + + subgraph Metadata Files + Metadata --> M1[v1.metadata.json] + Metadata --> M2[snap-2938-1-u57d.avro] + Metadata --> M3[u57d-m0.avro] + end + + subgraph Data Files + Data --> P1[00000-0-u57d.parquet] + Data --> P2[partition=2024/00001-0-u57d.orc] + end +``` + +### Valid File Paths + +* **Metadata Directory** (`metadata/`): + * `v*.metadata.json`: Table metadata files. + * `snap-*.avro`: Snapshot manifest lists. + * `*.avro`: Manifest files. + * `version-hint.text`: Pointer to the latest version. + * `*.stats`: Trino/Iceberg stats files. + +* **Data Directory** (`data/`): + * `*.parquet`, `*.orc`, `*.avro`: Data files containing the actual table rows. + * Supports partition subdirectories (e.g., `data/year=2024/month=01/file.parquet`). + +## Workflows + +The following diagrams illustrate how clients (like Spark or Trino) interact with SeaweedFS S3 Tables. + +### Write (Commit) Workflow + +When writing data to a table, the client first uploads the data files and then commits the transaction via the REST Catalog API. + +```mermaid +sequenceDiagram + participant Client + participant S3 as SeaweedFS (S3) + participant Catalog as SeaweedFS (Catalog) + + Note over Client, S3: 1. Write Data Files + Client->>S3: PUT /bucket/ns/table/data/file.parquet + S3-->>Client: 200 OK (Validated) + + Note over Client, Catalog: 2. Commit Transaction + Client->>Catalog: POST /v1/.../tables/my_table (UpdateTable) + Catalog->>Catalog: Load Current Metadata + Catalog->>Catalog: Validate Requirements (Optimistic Locking) + Catalog->>Catalog: Apply Updates & Write vN+1.metadata.json + Catalog-->>Client: 200 OK (New Metadata Location) +``` + +### Read (Load) Workflow + +To read a table, the client first asks the catalog for the current metadata location, then reads the metadata to discover which data files to scan. + +```mermaid +sequenceDiagram + participant Client + participant Catalog as SeaweedFS (Catalog) + participant S3 as SeaweedFS (S3) + + Note over Client, Catalog: 1. Load Table Metadata + Client->>Catalog: GET /v1/.../tables/my_table + Catalog-->>Client: 200 OK (Metadata Location & Schema) + + Note over Client, S3: 2. Read Metadata & Data + Client->>S3: GET /bucket/ns/table/metadata/vN.metadata.json + S3-->>Client: Metadata JSON + Client->>Client: Parse Metadata & Snapshots + Client->>S3: GET /bucket/ns/table/data/file.parquet + S3-->>Client: Data Content +``` ## Documentation -- [S3 Table Bucket Commands](S3-Table-Bucket-Commands) - Examples of how to manage table buckets, namespaces, and tables. +- [S3 Table Bucket Commands](S3-Table-Bucket-Commands) - Examples of how to create and manage table buckets. - [SeaweedFS Iceberg Catalog](SeaweedFS-Iceberg-Catalog) - Usage with Apache Spark and Trino. - [S3 Tables Security](S3-Tables-Security) - IAM permissions and policy reference.