From e772edbb9f54ee5ea431b54ecac13a5574a7ad46 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Thu, 26 Mar 2026 12:52:03 -0700 Subject: [PATCH] s3 APIs --- Amazon-S3-API.md | 667 +++++++++++++--------------------------- S3-Policy-Conditions.md | 327 ++++++++++++++++++++ _Sidebar.md | 1 + 3 files changed, 536 insertions(+), 459 deletions(-) create mode 100644 S3-Policy-Conditions.md diff --git a/Amazon-S3-API.md b/Amazon-S3-API.md index e082458..9a84b1a 100644 --- a/Amazon-S3-API.md +++ b/Amazon-S3-API.md @@ -1,14 +1,13 @@ To be compatible with Amazon S3 API, a separate "weed s3" command is provided. This provides much faster access when reading or writing files, compared to operating files on the cloud. -# How it works? -`weed s3` will start a stateless gateway server to bridge the Amazon S3 API to SeaweedFS Filer. -For convenience, `weed server -s3` will start a master, a volume server, a filer, and the S3 gateway. And `weed filer -s3` can start a filer and the S3 gateway together also. +# How It Works -Each bucket is stored in one collection, and mapped to folder `/buckets/` by default. +`weed s3` starts a stateless gateway server to bridge the Amazon S3 API to SeaweedFS Filer. +For convenience, `weed server -s3` starts a master, a volume server, a filer, and the S3 gateway. `weed filer -s3` can start a filer and the S3 gateway together also. -A bucket can be deleted efficiently by deleting the whole collection. +Each bucket is stored in one collection, and mapped to folder `/buckets/` by default. A bucket can be deleted efficiently by deleting the whole collection. -From version 3.51 name of collection is `filerGroup_bucketname` or `bucketname` if filer group is not set. +From version 3.51 the collection name is `filerGroup_bucketname` or `bucketname` if filer group is not set. ## Support Many Buckets @@ -28,467 +27,217 @@ This will add 1 physical volume when existing volumes are full. If using replica fs.configure -locationPrefix=/buckets/ -replication=001 -volumeGrowthCount=2 -apply ``` -See https://github.com/seaweedfs/seaweedfs/wiki/Path-Specific-Configuration - - -# Supported APIs -Currently, the following APIs are supported. - -Some additional endpoints might be (partially), supported but are not in this list. -To be sure, you can look at the function defined in the files `weed/s3api/s3api_*_handlers_*.go`. - - -``` -// Object operations -* PutObject -* GetObject -* HeadObject -* CopyObject -* DeleteObject -* ListObjectsV2 -* ListObjectsV1 -* DeleteMultipleObjects -* GetObjectAttributes -* PostPolicy - -// Object Tagging -* GetObjectTagging -* PutObjectTagging -* DeleteObjectTagging - -// User Metadata -* PutObject (with x-amz-meta-* headers) -* GetObject / HeadObject (returns x-amz-meta-* headers) -* CopyObject (with metadata directive) - -// Server-Side Encryption -* PutObject (with SSE-KMS, SSE-C, SSE-S3) -* GetObject (with automatic decryption) -* HeadObject (with encryption metadata) -* CopyObject (with encryption/decryption) -* Multipart uploads with encryption -* Bucket default encryption - -// Conditional Operations -* All object operations support conditional headers: - - If-Match - - If-None-Match - - If-Modified-Since - - If-Unmodified-Since - -// Bucket operations -* PutBucket -* DeleteBucket -* HeadBucket -* ListBuckets -* PutBucketLifecycleConfiguration (partially, only for TTL) -* GetBucketLifecycleConfiguration (partially, only for TTL) -* DeleteBucketLifecycleConfiguration (partially, only for TTL) -* GetBucketCors -* PutBucketCors -* DeleteBucketCors - -// Multipart upload operations -* NewMultipartUpload -* CompleteMultipartUpload -* AbortMultipartUpload -* ListMultipartUploads -* PutObjectPart -* CopyObjectPart -* ListObjectParts - -// Object Versioning -* PutBucketVersioning -* GetBucketVersioning -* ListObjectVersions -* GetObject (with version ID) -* PutObject (with versioning) -* DeleteObject (with version ID) -* CopyObject (with version ID) -* RestoreObject (partial) - -// Object Lock and Retention -* GetObjectLockConfiguration -* PutObjectLockConfiguration -* GetObjectRetention -* PutObjectRetention -* GetObjectLegalHold -* PutObjectLegalHold -* BypassGovernanceRetention (via x-amz-bypass-governance-retention header) - -// Bucket Policies -* PutBucketPolicy -* GetBucketPolicy -* DeleteBucketPolicy -* Supported Conditions: - - s3:ExistingObjectTag -``` - -Not included: - - -# User Metadata (Custom Attributes) - -SeaweedFS supports S3 user-defined metadata via `x-amz-meta-*` headers. This allows you to attach custom key-value pairs to objects. - -## Setting User Metadata - -```bash -# Set environment variables -export S3_ENDPOINT=http://localhost:8333 - -# Using AWS CLI -aws --endpoint-url $S3_ENDPOINT s3 cp myfile.txt s3://mybucket/myfile.txt \ - --metadata "expire=2025-12-01,author=john,project=demo" - -# Using curl -curl -X PUT \ - -H "x-amz-meta-expire: 2025-12-01" \ - -H "x-amz-meta-author: john" \ - --data-binary @myfile.txt \ - "$S3_ENDPOINT/mybucket/myfile.txt" -``` - -## Reading User Metadata - -User metadata is returned in response headers when you GET or HEAD an object: - -```bash -# Using AWS CLI -aws --endpoint-url $S3_ENDPOINT s3api head-object --bucket mybucket --key myfile.txt - -# Using curl -curl -I "$S3_ENDPOINT/mybucket/myfile.txt" -# Response includes: -# x-amz-meta-expire: 2025-12-01 -# x-amz-meta-author: john -``` - -## Updating User Metadata - -To update metadata, use CopyObject with `x-amz-metadata-directive: REPLACE`: - -```bash -aws --endpoint-url $S3_ENDPOINT s3 cp s3://mybucket/myfile.txt s3://mybucket/myfile.txt \ - --metadata "expire=2026-01-01" \ - --metadata-directive REPLACE -``` - -## Limits - -Metadata keys are case-insensitive and stored in canonical format (e.g., `x-amz-meta-My-Key` becomes `X-Amz-Meta-My-Key`). - -# Feature difference - -| Feature | SeaweedFS | Amazon S3 | -| -------------------------------------- | --------- | --------- | -| Multi byte ranges reads | Yes | No | -| DeleteObject deletes a folder | Yes | No | -| same path for both a file and a folder | No | Yes | -| allows more than "/" as a delimiter | No | Yes | -| Object Versioning | Yes | Yes | -| MFA Delete for versioning | No | Yes | -| Server-Side Encryption (SSE-KMS) | Yes | Yes | -| Server-Side Encryption (SSE-C) | Yes | Yes | -| Server-Side Encryption (SSE-S3) | Yes | Yes | -| KMS Providers (Multi-cloud) | Yes | No | -| Conditional Headers (All operations) | Yes | Yes | -| Range requests with SSE-KMS | Yes | Yes | -| Range requests with SSE-C | Yes | Yes | -| Range requests with SSE-S3 | Yes | Yes | - -## Empty folders - -SeaweedFS has directories while AWS S3 only has objects with "fake" directories. In AWS S3, if the last file is deleted in a directory, the directory will disappear also. - -To be consistent with AWS S3, SeaweedFS automatically cleans up empty folders asynchronously after file deletions. - -# Server-Side Encryption - -Need encryption at rest? SeaweedFS speaks the same SSE dialects as Amazon S3, so your existing tools and SDKs just work. You can choose from three options: - -- **[SSE-KMS](Server-Side-Encryption-SSE-KMS)**: Use an external KMS (AWS KMS, Google Cloud KMS, OpenBao/Vault) -- **[SSE-C](Server-Side-Encryption-SSE-C)**: Bring your own keys for maximum control -- **SSE-S3**: Let SeaweedFS manage keys (explicit `AES256` header or bucket default encryption) - -All encryption types support: -- Automatic encryption/decryption -- Bucket default encryption -- Multipart upload encryption -- Cross-encryption copy operations -- AWS S3 compatibility - -For detailed setup guides and examples, see: -- **[Server-Side Encryption Overview](Server-Side-Encryption)** -- **[SSE-KMS Guide](Server-Side-Encryption-SSE-KMS)** -- **[SSE-C Guide](Server-Side-Encryption-SSE-C)** - -## Quick Examples - -```bash -# SSE-KMS (Key Management Service) -aws --endpoint-url $S3_ENDPOINT s3 cp file.txt s3://mybucket/kms-encrypted.txt --server-side-encryption aws:kms --ssekms-key-id alias/my-key - -# SSE-C (Customer-provided keys) -aws --endpoint-url $S3_ENDPOINT s3 cp file.txt s3://mybucket/customer-encrypted.txt --sse-c AES256 --sse-c-key fileb://my-key.bin - -# SSE-S3 (Server-managed) -aws --endpoint-url $S3_ENDPOINT s3 cp file.txt s3://mybucket/server-encrypted.txt --server-side-encryption AES256 -``` - -# S3 Conditional Operations - -SeaweedFS supports AWS S3-compatible conditional headers for safe concurrent operations and efficient caching: - -- **If-Match**: Execute only if ETag matches (optimistic locking) -- **If-None-Match**: Execute only if ETag doesn't match (prevent overwrites, caching) -- **If-Modified-Since**: Execute only if modified after date (conditional downloads) -- **If-Unmodified-Since**: Execute only if not modified after date (safe updates) - -Conditional operations enable: -- **Optimistic concurrency control**: Prevent lost updates -- **Efficient caching**: Reduce bandwidth with 304 Not Modified -- **Atomic operations**: Operations only proceed when safe -- **Data integrity**: Prevent accidental overwrites - -For detailed usage patterns and examples, see **[S3 Conditional Operations](S3-Conditional-Operations)**. - -## Quick Examples - -```bash -# Get current ETag -ETAG=$(aws --endpoint-url $S3_ENDPOINT s3api head-object --bucket mybucket --key file.txt --query ETag --output text) - -# Conditional update (optimistic locking) -curl -X PUT -H "If-Match: $ETAG" -d "updated content" "$S3_ENDPOINT/mybucket/file.txt" - -# Conditional download (caching) -curl -H "If-None-Match: $ETAG" "$S3_ENDPOINT/mybucket/file.txt" -# Returns 304 Not Modified if unchanged - -# Prevent overwrite (atomic create) -curl -X PUT -H "If-None-Match: *" -d "new content" "$S3_ENDPOINT/mybucket/newfile.txt" -``` - -# S3 Authentication - -> **For a complete overview of S3 configuration options and priority, see [[S3 Configuration]].** - -By default, the SeaweedFS S3 gateway starts in **"Allow All" mode** if no credentials have been configured. This means all S3 operations are permitted without authentication. - -As soon as **at least one identity** is configured (via config file, env vars, or shell), the gateway **automatically transitions** to "Authentication Required" mode. - -| Method | Option | Documentation | -|--------|--------|---------------| -| **Allow All (Default)** | (No config) | See [[S3 Credentials]] | -| **Static Configuration** | `-s3.config=config.json` | [[S3 Credentials]] | -| **Dynamic Configuration** | `s3.configure` in weed shell | See below | -| **Admin UI** | Web interface | [[Admin UI]] | -| **OIDC/JWT (Web Identity)** | `-s3.iam.config=iam.json` | [[OIDC Integration]] | - -## Dynamic Configuration - -Example command: -```bash -s3.configure -access_key=any -secret_key=any -buckets=bucket1 -user=me -actions=Read,Write,List,Tagging,Admin -apply -``` - -Output from above `s3.configure` command: -```json -{ - "identities": [ - { - "name": "me", - "credentials": [ - { - "accessKey": "any", - "secretKey": "any" - } - ], - "actions": [ - "Read:bucket1", - "Write:bucket1", - "List:bucket1", - "Tagging:bucket1", - "Admin:bucket1" - ] - } - ] -} -``` - -## Static Configuration - -To enable credential based access, create a config.json file similar to the example below, and specify it via `weed s3 -config=config.json`. The config file can be re-read on the HUP signal without restarting the main process via `pkill -HUP weed` - -You just need to create a user with all "Admin", "Read", "Write", "List", "Tagging" actions. -You can create as many users as needed. Each user can have multiple credentials. - -* The "Admin" action is needed to list, create, and delete buckets. -* The "Write" action allows uploading files to all buckets. -* The "WriteAcp" action allows writing the access control list (ACL) from all buckets. -* The "Read" action allows reading files from all buckets. -* The "ReadAcp" action allows reading the access control list (ACL) from all buckets. -* The "List" action allows listing files from all buckets. -* The "Tagging" action allows tagging files from all buckets. -* The "Write:" action allows uploading files within a bucket, e.g., "Write:bucket1". -* The "WriteAcp:" action allows writing ACL within a bucket, e.g., "Write:bucket1". -* The "Read:" action allows reading files within a bucket, e.g., "Read:bucket2". -* The "ReadAcp:" action allows reading ACL within a bucket, e.g., "Read:bucket2". -* The "List:" action allows listing files within a bucket, e.g., "List:bucket2". -* The "Tagging:" action allows tagging files within a bucket, e.g., "Tagging:bucket2". - -### Public access (with anonymous download) - -For public access, you can configure an identity with name "anonymous", usually with just "Read" action, or access to specific buckets. - -```json -{ - "identities": [ - { - "name": "anonymous", - "actions": [ - "Read" - ] - }, - { - "name": "some_name", - "credentials": [ - { - "accessKey": "some_access_key1", - "secretKey": "some_secret_key1" - } - ], - "actions": [ - "Admin", - "Read", - "ReadAcp", - "List", - "Tagging", - "Write", - "WriteAcp" - ] - }, - { - "name": "some_read_only_user", - "credentials": [ - { - "accessKey": "some_access_key2", - "secretKey": "some_secret_key2" - } - ], - "actions": [ - "Read", - "List" - ] - }, - { - "name": "some_normal_user", - "credentials": [ - { - "accessKey": "some_access_key3", - "secretKey": "some_secret_key3" - } - ], - "actions": [ - "Read", - "List", - "Tagging", - "Write" - ] - }, - { - "name": "user_limited_by_bucket", - "credentials": [ - { - "accessKey": "some_access_key4", - "secretKey": "some_secret_key4" - } - ], - "actions": [ - "Read:bucket1", - "Read:bucket2", - "Read:bucket3", - "Write:bucket1" - ] - } - ] -} -``` - -### Actions with wildcard - -Wildcard is partially supported for prefix lookup. The following example actions are allowed: - -``` -Read -Read:bucket -Read:bucket_prefix* -Read:bucket/* -Read:bucket/a/b/* -``` - -## Presigned URL - -Presigned URL is supported. See [[AWS-CLI-with-SeaweedFS#presigned-url]] for example. - - -# S3 Object Versioning - -SeaweedFS supports S3 object versioning, which allows you to keep multiple variants of an object in the same bucket. This provides data protection against accidental deletion or modification. - -For detailed information about object versioning, see the dedicated [[S3-Object-Versioning]] page. - - -# S3 Cross-Origin Resource Sharing (CORS) - -SeaweedFS supports S3-compatible Cross-Origin Resource Sharing (CORS) configuration, allowing web applications to make cross-origin requests to your S3 buckets. CORS is essential for web applications that need to access resources from different domains. - -For detailed information about CORS configuration, see the dedicated [[S3-CORS]] page. - - -# Reverse Proxy Support - -SeaweedFS S3 API supports deployment behind reverse proxies with full AWS Signature v4 authentication compatibility. This includes support for: - -- **X-Forwarded-Host**: Preserves original host header for signature verification -- **X-Forwarded-Port**: Automatically combines with X-Forwarded-Host for non-standard ports -- **X-Forwarded-Prefix**: Handles URL path prefix stripping by reverse proxies -- **Standard forwarded headers**: X-Forwarded-For, X-Forwarded-Proto, etc. - -## Path Prefix Handling - -When using reverse proxies that strip URL prefixes (e.g., `/s3/`, `/api/s3/`), SeaweedFS automatically handles signature verification for both the original prefixed path and the stripped path. This ensures seamless operation with: - -- API gateways -- Multi-tenant deployments -- Subpath hosting scenarios - -For detailed configuration examples and setup instructions, see the dedicated [[S3-Nginx-Proxy]] page. - +See [[Path Specific Configuration]] ## Multiple S3 Nodes -If you need to setup multiple S3 nodes, you can just start multiple s3 instances pointing to a filer. - -Usually you would also want to have multiple filers. The easiest way is to run filer together with a S3. +You can start multiple S3 instances pointing to a filer for horizontal scaling. The easiest way is to run filer together with S3: ```bash weed filer -s3 ``` -# Authentication with Filer +# Supported APIs -You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as -explained in [[Security-Configuration]] - -controlled by the `grpc.*` configuration in `security.toml`. +The following tables list all standard AWS S3 API operations and their support status in SeaweedFS, using official AWS API operation names. -**Starting with version 2.84, it is also possible to authenticate the HTTP -operations between the S3-API-Proxy and the Filer (especially -uploading new files).** This is configured by setting -`jwt.filer_signing.key` and `jwt.filer_signing.read.key` in -`security.toml`. +For IAM action names used in bucket policies (e.g., `s3:ListBucketMultipartUploads`), see [[S3 Bucket Policies]]. -With both configurations (gRPC and JWT), it is possible to have Filer -and S3 communicate in fully authenticated fashion; so Filer will reject -any unauthenticated communication. +## Bucket Operations + +| API Operation | Supported | Notes | +|---|---|---| +| CreateBucket | Yes | | +| DeleteBucket | Yes | Deletes the entire collection | +| HeadBucket | Yes | | +| ListBuckets | Yes | | +| ListDirectoryBuckets | No | S3 Express One Zone | + +## Bucket Configuration + +| API Operation | Supported | Notes | +|---|---|---| +| GetBucketAcl | Yes | | +| PutBucketAcl | Yes | | +| GetBucketCors | Yes | See [[S3 CORS]] | +| PutBucketCors | Yes | | +| DeleteBucketCors | Yes | | +| GetBucketEncryption | Yes | See [[Server-Side Encryption]] | +| PutBucketEncryption | Yes | | +| DeleteBucketEncryption | Yes | | +| GetBucketLifecycleConfiguration | Partial | TTL-based expiration only | +| PutBucketLifecycleConfiguration | Partial | TTL-based expiration only | +| DeleteBucketLifecycle | Partial | TTL-based expiration only | +| GetBucketLocation | Yes | Returns default location | +| GetBucketOwnershipControls | Yes | | +| PutBucketOwnershipControls | Yes | | +| DeleteBucketOwnershipControls | Yes | | +| GetBucketPolicy | Yes | See [[S3 Bucket Policies]] | +| PutBucketPolicy | Yes | | +| DeleteBucketPolicy | Yes | | +| GetBucketRequestPayment | Yes | Always returns BucketOwner | +| GetBucketTagging | Yes | | +| PutBucketTagging | Yes | | +| DeleteBucketTagging | Yes | | +| GetBucketVersioning | Yes | See [[S3 Object Versioning]] | +| PutBucketVersioning | Yes | | +| GetObjectLockConfiguration | Yes | See [[S3 Object Lock and Retention]] | +| PutObjectLockConfiguration | Yes | | +| GetPublicAccessBlock | Yes | | +| PutPublicAccessBlock | Yes | | +| DeletePublicAccessBlock | Yes | | +| GetBucketAccelerateConfiguration | No | | +| PutBucketAccelerateConfiguration | No | | +| GetBucketAnalyticsConfiguration | No | | +| PutBucketAnalyticsConfiguration | No | | +| DeleteBucketAnalyticsConfiguration | No | | +| ListBucketAnalyticsConfigurations | No | | +| GetBucketIntelligentTieringConfiguration | No | | +| PutBucketIntelligentTieringConfiguration | No | | +| DeleteBucketIntelligentTieringConfiguration | No | | +| ListBucketIntelligentTieringConfigurations | No | | +| GetBucketInventoryConfiguration | No | | +| PutBucketInventoryConfiguration | No | | +| DeleteBucketInventoryConfiguration | No | | +| ListBucketInventoryConfigurations | No | | +| GetBucketLogging | No | | +| PutBucketLogging | No | | +| GetBucketMetricsConfiguration | No | | +| PutBucketMetricsConfiguration | No | | +| DeleteBucketMetricsConfiguration | No | | +| ListBucketMetricsConfigurations | No | | +| GetBucketNotificationConfiguration | No | | +| PutBucketNotificationConfiguration | No | | +| GetBucketPolicyStatus | No | | +| GetBucketReplication | No | | +| PutBucketReplication | No | | +| DeleteBucketReplication | No | | +| PutBucketRequestPayment | No | | +| GetBucketWebsite | No | | +| PutBucketWebsite | No | | +| DeleteBucketWebsite | No | | + +## Object Operations + +| API Operation | Supported | Notes | +|---|---|---| +| CopyObject | Yes | Supports version ID, encryption, metadata directive | +| DeleteObject | Yes | Supports version ID; also deletes folders | +| DeleteObjects | Yes | Multi-object delete | +| GetObject | Yes | Supports range requests, conditional headers, SSE, user metadata | +| GetObjectAcl | Yes | | +| GetObjectAttributes | Yes | | +| GetObjectLegalHold | Yes | See [[S3 Object Lock and Retention]] | +| GetObjectRetention | Yes | | +| GetObjectTagging | Yes | | +| HeadObject | Yes | Returns user metadata, encryption info | +| ListObjects | Yes | V1 (legacy) | +| ListObjectsV2 | Yes | | +| ListObjectVersions | Yes | See [[S3 Object Versioning]] | +| PostObject | Yes | Browser-based form uploads via POST policy | +| PutObject | Yes | Supports SSE, user metadata (`x-amz-meta-*`), conditional headers | +| PutObjectAcl | Yes | | +| PutObjectLegalHold | Yes | | +| PutObjectRetention | Yes | | +| PutObjectTagging | Yes | | +| DeleteObjectTagging | Yes | | +| GetObjectTorrent | No | | +| RenameObject | No | | +| RestoreObject | No | | +| SelectObjectContent | No | | +| WriteGetObjectResponse | No | S3 Object Lambda | + +## Multipart Upload Operations + +| API Operation | Supported | Notes | +|---|---|---| +| CreateMultipartUpload | Yes | IAM action: `s3:CreateMultipartUpload` | +| UploadPart | Yes | IAM action: `s3:UploadPart` | +| UploadPartCopy | Yes | | +| CompleteMultipartUpload | Yes | IAM action: `s3:CompleteMultipartUpload` | +| AbortMultipartUpload | Yes | IAM action: `s3:AbortMultipartUpload` | +| ListMultipartUploads | Yes | IAM action: `s3:ListBucketMultipartUploads` | +| ListParts | Yes | IAM action: `s3:ListMultipartUploadParts` | + +All multipart operations are implicitly allowed when `s3:PutObject` is granted. See [[S3 Bucket Policies#multipart-upload-permission-inheritance]]. + +## Additional APIs + +SeaweedFS also implements these related AWS service APIs on the same S3 endpoint: + +| Service | Operations | Documentation | +|---|---|---| +| Security Token Service (STS) | AssumeRole, AssumeRoleWithWebIdentity, AssumeRoleWithLDAPIdentity | [[OIDC Integration]] | +| IAM API | User, policy, access key, and group management | [[Amazon IAM API]], [[AWS IAM CLI]] | +| S3 Tables | Table bucket, namespace, and table management (Apache Iceberg) | [[S3 Table Bucket]] | + +# Feature Differences + +| Feature | SeaweedFS | Amazon S3 | +|---|---|---| +| Multi byte ranges reads | Yes | No | +| DeleteObject deletes a folder | Yes | No | +| Same path for both a file and a folder | No | Yes | +| Allows more than "/" as a delimiter | No | Yes | +| Object Versioning | Yes | Yes | +| MFA Delete for versioning | No | Yes | +| Server-Side Encryption (SSE-KMS) | Yes | Yes | +| Server-Side Encryption (SSE-C) | Yes | Yes | +| Server-Side Encryption (SSE-S3) | Yes | Yes | +| KMS Providers (Multi-cloud) | Yes | No | +| Conditional Headers (All operations) | Yes | Yes | +| Range requests with SSE | Yes | Yes | +| Presigned URLs | Yes | Yes | + +## Empty Folders + +SeaweedFS has directories while AWS S3 only has objects with "fake" directories. In AWS S3, if the last file is deleted in a directory, the directory will disappear also. + +To be consistent with AWS S3, SeaweedFS automatically cleans up empty folders asynchronously after file deletions. + +# Related Pages + +**Configuration & Authentication** +- [[S3 Configuration]] — Overview of all configuration options +- [[S3 Credentials]] — Access key and identity management +- [[OIDC Integration]] — OIDC, STS, and role-based access +- [[Amazon IAM API]] / [[AWS IAM CLI]] — IAM API operations +- [[Security Configuration]] — mTLS and JWT between S3 gateway and filer + +**Security & Policies** +- [[S3 Bucket Policies]] — IAM-style policies with condition operators +- [[S3 Policy Conditions]] — Condition key reference +- [[S3 Policy Variables]] — Policy variable substitution +- [[S3 Object Lock and Retention]] — Object lock, retention, legal hold +- [[S3 Object Versioning]] — Version management + +**Encryption** +- [[Server-Side Encryption]] — SSE overview +- [[Server-Side Encryption SSE-KMS]] — AWS KMS, Google Cloud KMS, OpenBao/Vault +- [[Server-Side Encryption SSE-C]] — Customer-provided keys + +**Features** +- [[S3 Conditional Operations]] — If-Match, If-None-Match, If-Modified-Since, If-Unmodified-Since +- [[S3 CORS]] — Cross-Origin Resource Sharing +- [[S3 Bucket Quota]] — Storage quota management +- [[S3 Rate Limiting]] — Request rate limiting +- [[S3 API Audit log]] — Audit logging + +**S3 Tables & Iceberg** +- [[S3 Table Bucket]] — S3 Tables API +- [[SeaweedFS Iceberg Catalog]] — Apache Iceberg REST catalog +- [[Iceberg Table Maintenance]] — Table maintenance operations +- [[S3 Tables Security]] — Table bucket security + +**Deployment & Operations** +- [[S3 Nginx Proxy]] — Reverse proxy and X-Forwarded-* header support +- [[Docker Compose for S3]] — Docker deployment +- [[S3 API Benchmark]] — Performance benchmarking +- [[S3 API FAQ]] — Frequently asked questions + +**Client Tools** +- [[AWS CLI with SeaweedFS]] — AWS CLI usage, presigned URLs +- [[s3cmd with SeaweedFS]] +- [[rclone with SeaweedFS]] +- [[restic with SeaweedFS]] +- [[nodejs with Seaweed S3]] diff --git a/S3-Policy-Conditions.md b/S3-Policy-Conditions.md new file mode 100644 index 0000000..ecf73ca --- /dev/null +++ b/S3-Policy-Conditions.md @@ -0,0 +1,327 @@ +# S3 Policy Conditions + +SeaweedFS supports AWS S3-compatible policy conditions for fine-grained access control based on request context such as source IP, transport security, time, and request headers. + +## Overview + +Conditions allow you to restrict when a policy statement applies. They are used in bucket policies and IAM policies to enforce rules like "allow access only from a specific IP range" or "require HTTPS". + +A condition block has the form: + +```json +{ + "Condition": { + "": { + "": ["", ""] + } + } +} +``` + +Multiple operators within a `Condition` block are ANDed together. Multiple values for a single key are ORed. + +## Supported Condition Operators + +### String Operators + +| Operator | Description | +|----------|-------------| +| `StringEquals` | Exact case-sensitive match | +| `StringNotEquals` | Negated exact match | +| `StringLike` | Case-sensitive match with `*` and `?` wildcards | +| `StringNotLike` | Negated wildcard match | + +### Numeric Operators + +| Operator | Description | +|----------|-------------| +| `NumericEquals` | Equal | +| `NumericNotEquals` | Not equal | +| `NumericLessThan` | Less than | +| `NumericLessThanEquals` | Less than or equal | +| `NumericGreaterThan` | Greater than | +| `NumericGreaterThanEquals` | Greater than or equal | + +### Date Operators + +Date values must be in RFC 3339 format (e.g., `2025-01-01T00:00:00Z`). + +| Operator | Description | +|----------|-------------| +| `DateEquals` | Exact date match | +| `DateNotEquals` | Negated date match | +| `DateLessThan` | Before date | +| `DateLessThanEquals` | Before or at date | +| `DateGreaterThan` | After date | +| `DateGreaterThanEquals` | After or at date | + +### IP Address Operators + +Values can be individual IPs or CIDR ranges (e.g., `192.168.1.0/24`). + +| Operator | Description | +|----------|-------------| +| `IpAddress` | Source IP is within the specified range | +| `NotIpAddress` | Source IP is not within the specified range | + +### Other Operators + +| Operator | Description | +|----------|-------------| +| `Bool` | Boolean match (`"true"` or `"false"`) | +| `ArnEquals` | Exact ARN match | +| `ArnLike` | ARN match with wildcards | +| `Null` | Check whether a key is present (`"true"` = key absent, `"false"` = key present) | + +### Set Operators + +Prefix any string operator with `ForAnyValue:` or `ForAllValues:` for multi-valued keys: + +- `ForAnyValue:StringEquals` - at least one value matches +- `ForAllValues:StringEquals` - all values match + +## Supported Condition Keys + +### AWS Global Context Keys + +| Key | Type | Description | +|-----|------|-------------| +| `aws:SourceIp` | IP | Client IP address (supports CIDR) | +| `aws:SecureTransport` | Bool | `true` if the request was sent over HTTPS | +| `aws:CurrentTime` | Date | Current time in RFC 3339 format | +| `aws:UserAgent` | String | Client User-Agent header | +| `aws:Referer` | String | HTTP Referer header | +| `aws:username` | String | Username from principal ARN | +| `aws:userid` | String | User ID from principal ARN | +| `aws:PrincipalAccount` | String | Account ID from principal ARN | +| `aws:principaltype` | String | Principal type (`IAMUser`, `IAMRole`, `AssumedRole`) | +| `aws:PrincipalArn` | String | Full principal ARN | +| `aws:FederatedProvider` | String | Federated identity provider | +| `aws:PrincipalServiceName` | String | Service principal name | + +### S3-Specific Keys + +| Key | Type | Description | +|-----|------|-------------| +| `s3:prefix` | String | Prefix parameter from list operations | +| `s3:delimiter` | String | Delimiter parameter from list operations | +| `s3:max-keys` | Numeric | Max keys parameter from list operations | +| `s3:ExistingObjectTag/` | String | Value of an existing object tag | +| `s3:RequestMethod` | String | HTTP method (GET, PUT, POST, DELETE) | +| `s3:authType` | String | Auth type (`REST-HEADER` or `REST-QUERY-STRING`) | + +| `s3:x-amz-*` | String | Any `x-amz-*` request header (e.g., `s3:x-amz-server-side-encryption`) | + +### Identity Provider Keys + +| Key | Type | Description | +|-----|------|-------------| +| `jwt:preferred_username` | String | Username from JWT token | +| `jwt:sub` | String | Subject from JWT token | +| `jwt:iss` | String | Issuer from JWT token | +| `jwt:aud` | String | Audience from JWT token | +| `oidc:sub` | String | Subject from OIDC token | +| `oidc:aud` | String | Audience from OIDC token | +| `oidc:iss` | String | Issuer from OIDC token | +| `oidc:` | String | Any custom OIDC claim (e.g., `oidc:roles`, `oidc:groups`) | +| `saml:username` | String | Username from SAML assertion | +| `saml:sub` | String | Subject from SAML assertion | +| `saml:aud` | String | Audience from SAML assertion | +| `saml:iss` | String | Issuer from SAML assertion | +| `ldap:username` | String | Username from LDAP | +| `ldap:dn` | String | Distinguished name from LDAP | +| `ldap:` | String | Any LDAP attribute | + +## Source IP Handling + +When SeaweedFS runs behind a reverse proxy, the `aws:SourceIp` condition key is resolved using the following logic: + +1. If the direct connection is from a **private/loopback IP** (RFC 1918, IPv6 ULA), forwarding headers are trusted: + - `X-Forwarded-For` is checked for the rightmost non-private IP + - `X-Real-IP` is used as a fallback +2. If the direct connection is from a **public IP**, forwarding headers are ignored to prevent spoofing +3. Final fallback is always `RemoteAddr` + +This ensures correct behavior whether SeaweedFS is accessed directly or through a load balancer. + +## Examples + +### Restrict Access by IP Address + +Allow reads only from your office network: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": "*", + "Action": "s3:GetObject", + "Resource": "arn:aws:s3:::my-bucket/*", + "Condition": { + "IpAddress": { + "aws:SourceIp": ["10.0.0.0/8", "192.168.1.0/24"] + } + } + } + ] +} +``` + +Deny access from a specific IP range: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Deny", + "Principal": "*", + "Action": "s3:*", + "Resource": "arn:aws:s3:::my-bucket/*", + "Condition": { + "NotIpAddress": { + "aws:SourceIp": "203.0.113.0/24" + } + } + } + ] +} +``` + +### Require HTTPS + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Deny", + "Principal": "*", + "Action": "s3:*", + "Resource": "arn:aws:s3:::my-bucket/*", + "Condition": { + "Bool": { + "aws:SecureTransport": "false" + } + } + } + ] +} +``` + +### Time-Based Access + +Allow access only during a specific window: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": "*", + "Action": "s3:GetObject", + "Resource": "arn:aws:s3:::my-bucket/*", + "Condition": { + "DateGreaterThan": { + "aws:CurrentTime": "2025-01-01T00:00:00Z" + }, + "DateLessThan": { + "aws:CurrentTime": "2025-12-31T23:59:59Z" + } + } + } + ] +} +``` + +### Require Encryption Header + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Deny", + "Principal": "*", + "Action": "s3:PutObject", + "Resource": "arn:aws:s3:::my-bucket/*", + "Condition": { + "StringNotEquals": { + "s3:x-amz-server-side-encryption": "AES256" + } + } + } + ] +} +``` + +### Tag-Based Access Control + +Allow deletion only if the object is tagged as deletable: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": "*", + "Action": "s3:DeleteObject", + "Resource": "arn:aws:s3:::my-bucket/*", + "Condition": { + "StringEquals": { + "s3:ExistingObjectTag/status": "deletable" + } + } + } + ] +} +``` + +### Combining Multiple Conditions + +Multiple operators are ANDed — all must be true: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": "*", + "Action": "s3:GetObject", + "Resource": "arn:aws:s3:::my-bucket/*", + "Condition": { + "IpAddress": { + "aws:SourceIp": "192.168.1.0/24" + }, + "Bool": { + "aws:SecureTransport": "true" + }, + "DateGreaterThan": { + "aws:CurrentTime": "2025-01-01T00:00:00Z" + } + } + } + ] +} +``` + +## Policy Evaluation Logic + +Conditions follow AWS-compatible evaluation order: + +1. **Explicit Deny** — if any statement with a matching condition denies, access is denied +2. **Explicit Allow** — if a statement with a matching condition allows, access is allowed +3. **Default Deny** — if no statement matches, access is denied + +## See Also + +- [[S3 Bucket Policies]] — Managing bucket policies +- [[S3 Policy Variables]] — Dynamic policy variables +- [[S3 Credentials]] — S3 authentication +- [[Amazon IAM API]] — IAM API support +- [[OIDC Integration]] — JWT-based authentication diff --git a/_Sidebar.md b/_Sidebar.md index 1c94c98..7afb795 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -97,6 +97,7 @@ * [[S3 Credentials]] (`-s3.config`) * [[OIDC Integration]] (`-s3.iam.config`) * [[S3 Policy Variables]] +* [[S3 Policy Conditions]] * [[S3 Bucket Policies]] * [[Amazon IAM API]] * [[AWS IAM CLI]]