mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-18 17:55:26 +01:00
adding table bucket
1 parent
f9907f4326
commit
645c4cf45b
3 files changed
+299
No files matched your search
@@ -0,0 +1,277 @@
|
||||
SeaweedFS supports the S3 Tables API, providing a way to manage table-based data structures with AWS S3 compatibility. This guide provides commands to test the S3 Tables features in SeaweedFS.
|
||||
|
||||
## Configuration
|
||||
|
||||
### Prerequisites
|
||||
|
||||
For AWS CLI and curl commands, configure your environment variables:
|
||||
|
||||
```bash
|
||||
export S3_ENDPOINT=http://localhost:8333
|
||||
export AWS_ACCESS_KEY_ID=any
|
||||
export AWS_SECRET_ACCESS_KEY=any
|
||||
export AWS_DEFAULT_REGION=us-east-1
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> The `s3tables` service is a relatively new AWS service. Ensure your AWS CLI is up to date (v2.17+ recommended). If `aws s3tables` is not available, use the `curl` commands provided below.
|
||||
|
||||
---
|
||||
|
||||
## 1. Table Bucket Operations
|
||||
|
||||
### Create Table Bucket
|
||||
**AWS CLI:**
|
||||
```bash
|
||||
aws s3tables create-table-bucket --endpoint-url $S3_ENDPOINT --name my-test-bucket
|
||||
```
|
||||
|
||||
**curl:**
|
||||
```bash
|
||||
curl -X POST $S3_ENDPOINT/ \
|
||||
-H "X-Amz-Target: S3Tables.CreateTableBucket" \
|
||||
-H "Content-Type: application/x-amz-json-1.1" \
|
||||
-d '{"name": "my-test-bucket"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### List Table Buckets
|
||||
**AWS CLI:**
|
||||
```bash
|
||||
aws s3tables list-table-buckets --endpoint-url $S3_ENDPOINT
|
||||
```
|
||||
|
||||
**curl:**
|
||||
```bash
|
||||
curl -X POST $S3_ENDPOINT/ \
|
||||
-H "X-Amz-Target: S3Tables.ListTableBuckets" \
|
||||
-H "Content-Type: application/x-amz-json-1.1" \
|
||||
-d '{}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Get Table Bucket
|
||||
**AWS CLI:**
|
||||
```bash
|
||||
aws s3tables get-table-bucket --endpoint-url $S3_ENDPOINT --table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket
|
||||
```
|
||||
|
||||
**curl:**
|
||||
```bash
|
||||
curl -X POST $S3_ENDPOINT/ \
|
||||
-H "X-Amz-Target: S3Tables.GetTableBucket" \
|
||||
-H "Content-Type: application/x-amz-json-1.1" \
|
||||
-d '{"tableBucketARN": "arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Namespace Operations
|
||||
|
||||
### Create Namespace
|
||||
**AWS CLI:**
|
||||
```bash
|
||||
aws s3tables create-namespace --endpoint-url $S3_ENDPOINT --table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket --namespace my_namespace
|
||||
```
|
||||
|
||||
**curl:**
|
||||
```bash
|
||||
curl -X POST $S3_ENDPOINT/ \
|
||||
-H "X-Amz-Target: S3Tables.CreateNamespace" \
|
||||
-H "Content-Type: application/x-amz-json-1.1" \
|
||||
-d '{
|
||||
"tableBucketARN": "arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket",
|
||||
"namespace": ["my_namespace"]
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### List Namespaces
|
||||
**AWS CLI:**
|
||||
```bash
|
||||
aws s3tables list-namespaces --endpoint-url $S3_ENDPOINT --table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket
|
||||
```
|
||||
|
||||
**curl:**
|
||||
```bash
|
||||
curl -X POST $S3_ENDPOINT/ \
|
||||
-H "X-Amz-Target: S3Tables.ListNamespaces" \
|
||||
-H "Content-Type: application/x-amz-json-1.1" \
|
||||
-d '{"tableBucketARN": "arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Table Operations
|
||||
|
||||
### Create Table (Iceberg)
|
||||
**AWS CLI:**
|
||||
```bash
|
||||
aws s3tables create-table --endpoint-url $S3_ENDPOINT --table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket --namespace my_namespace --name my_table --format ICEBERG
|
||||
```
|
||||
|
||||
**curl:**
|
||||
```bash
|
||||
curl -X POST $S3_ENDPOINT/ \
|
||||
-H "X-Amz-Target: S3Tables.CreateTable" \
|
||||
-H "Content-Type: application/x-amz-json-1.1" \
|
||||
-d '{
|
||||
"tableBucketARN": "arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket",
|
||||
"namespace": ["my_namespace"],
|
||||
"name": "my_table",
|
||||
"format": "ICEBERG"
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### List Tables
|
||||
**AWS CLI:**
|
||||
```bash
|
||||
aws s3tables list-tables --endpoint-url $S3_ENDPOINT --table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket --namespace my_namespace
|
||||
```
|
||||
|
||||
**curl:**
|
||||
```bash
|
||||
curl -X POST $S3_ENDPOINT/ \
|
||||
-H "X-Amz-Target: S3Tables.ListTables" \
|
||||
-H "Content-Type: application/x-amz-json-1.1" \
|
||||
-d '{
|
||||
"tableBucketARN": "arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket",
|
||||
"namespace": ["my_namespace"]
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Policy and Tagging
|
||||
|
||||
### Put Table Bucket Policy
|
||||
**AWS CLI:**
|
||||
```bash
|
||||
aws s3tables put-table-bucket-policy --endpoint-url $S3_ENDPOINT \
|
||||
--table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket \
|
||||
--resource-policy "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"s3tables:*\",\"Resource\":\"*\"}]}"
|
||||
```
|
||||
|
||||
**curl:**
|
||||
```bash
|
||||
curl -X POST $S3_ENDPOINT/ \
|
||||
-H "X-Amz-Target: S3Tables.PutTableBucketPolicy" \
|
||||
-H "Content-Type: application/x-amz-json-1.1" \
|
||||
-d '{
|
||||
"tableBucketARN": "arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket",
|
||||
"resourcePolicy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"s3tables:*\",\"Resource\":\"*\"}]}"
|
||||
}'
|
||||
```
|
||||
|
||||
### Get Table Bucket Policy
|
||||
**AWS CLI:**
|
||||
```bash
|
||||
aws s3tables get-table-bucket-policy --endpoint-url $S3_ENDPOINT \
|
||||
--table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket
|
||||
```
|
||||
|
||||
**curl:**
|
||||
```bash
|
||||
curl -X POST $S3_ENDPOINT/ \
|
||||
-H "X-Amz-Target: S3Tables.GetTableBucketPolicy" \
|
||||
-H "Content-Type: application/x-amz-json-1.1" \
|
||||
-d '{"tableBucketARN": "arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket"}'
|
||||
```
|
||||
|
||||
### Tag Resource
|
||||
**AWS CLI:**
|
||||
```bash
|
||||
aws s3tables tag-resource --endpoint-url $S3_ENDPOINT \
|
||||
--resource-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket \
|
||||
--tags Environment=test,Owner=admin
|
||||
```
|
||||
|
||||
**curl:**
|
||||
```bash
|
||||
curl -X POST $S3_ENDPOINT/ \
|
||||
-H "X-Amz-Target: S3Tables.TagResource" \
|
||||
-H "Content-Type: application/x-amz-json-1.1" \
|
||||
-d '{
|
||||
"resourceArn": "arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket",
|
||||
"tags": {"Environment": "test", "Owner": "admin"}
|
||||
}'
|
||||
```
|
||||
|
||||
### List Tags
|
||||
**AWS CLI:**
|
||||
```bash
|
||||
aws s3tables list-tags-for-resource --endpoint-url $S3_ENDPOINT \
|
||||
--resource-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket
|
||||
```
|
||||
|
||||
**curl:**
|
||||
```bash
|
||||
curl -X POST $S3_ENDPOINT/ \
|
||||
-H "X-Amz-Target: S3Tables.ListTagsForResource" \
|
||||
-H "Content-Type: application/x-amz-json-1.1" \
|
||||
-d '{"resourceArn": "arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cleanup
|
||||
|
||||
### Delete Table
|
||||
**AWS CLI:**
|
||||
```bash
|
||||
aws s3tables delete-table --endpoint-url $S3_ENDPOINT \
|
||||
--table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket \
|
||||
--namespace my_namespace \
|
||||
--name my_table
|
||||
```
|
||||
|
||||
**curl:**
|
||||
```bash
|
||||
curl -X POST $S3_ENDPOINT/ \
|
||||
-H "X-Amz-Target: S3Tables.DeleteTable" \
|
||||
-H "Content-Type: application/x-amz-json-1.1" \
|
||||
-d '{
|
||||
"tableBucketARN": "arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket",
|
||||
"namespace": ["my_namespace"],
|
||||
"name": "my_table"
|
||||
}'
|
||||
```
|
||||
|
||||
### Delete Namespace
|
||||
**AWS CLI:**
|
||||
```bash
|
||||
aws s3tables delete-namespace --endpoint-url $S3_ENDPOINT \
|
||||
--table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket \
|
||||
--namespace my_namespace
|
||||
```
|
||||
|
||||
**curl:**
|
||||
```bash
|
||||
curl -X POST $S3_ENDPOINT/ \
|
||||
-H "X-Amz-Target: S3Tables.DeleteNamespace" \
|
||||
-H "Content-Type: application/x-amz-json-1.1" \
|
||||
-d '{
|
||||
"tableBucketARN": "arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket",
|
||||
"namespace": ["my_namespace"]
|
||||
}'
|
||||
```
|
||||
|
||||
### Delete Table Bucket
|
||||
**AWS CLI:**
|
||||
```bash
|
||||
aws s3tables delete-table-bucket --endpoint-url $S3_ENDPOINT \
|
||||
--table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket
|
||||
```
|
||||
|
||||
**curl:**
|
||||
```bash
|
||||
curl -X POST $S3_ENDPOINT/ \
|
||||
-H "X-Amz-Target: S3Tables.DeleteTableBucket" \
|
||||
-H "Content-Type: application/x-amz-json-1.1" \
|
||||
-d '{"tableBucketARN": "arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket"}'
|
||||
```
|
||||
@@ -0,0 +1,19 @@
|
||||
# S3 Tables
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Work in Progress**
|
||||
> This feature is currently under development.
|
||||
|
||||
## Introduction
|
||||
|
||||
SeaweedFS is adding support for Amazon S3 Tables, which provides a way to manage structured datasets using the Apache Iceberg table format. S3 Tables combine the scalability of S3 with the performance and features of dedicated table formats.
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Apache Iceberg Support**: Native support for Iceberg tables.
|
||||
- **Table Buckets**: Dedicated buckets for storing your tables.
|
||||
- **Namespaces**: Logical grouping of tables within a table bucket.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [S3 Table Bucket Commands](S3-Table-Bucket-Commands) - Examples of how to manage table buckets, namespaces, and tables.
|
||||
+3
@@ -82,6 +82,9 @@
|
||||
* [[S3 Nginx Proxy]]
|
||||
* [[Docker Compose for S3]]
|
||||
|
||||
### S3 Table Bucket
|
||||
* [[S3 Table Bucket]]
|
||||
|
||||
### S3 Authentication & IAM
|
||||
* [[S3 Configuration]] - Start Here
|
||||
* [[S3 Credentials]] (`-s3.config`)
|
||||
|
||||
Reference in new issue
Block a user