diff --git a/Amazon-S3-API.md b/Amazon-S3-API.md index 9a84b1a..acdb939 100644 --- a/Amazon-S3-API.md +++ b/Amazon-S3-API.md @@ -65,9 +65,9 @@ For IAM action names used in bucket policies (e.g., `s3:ListBucketMultipartUploa | GetBucketEncryption | Yes | See [[Server-Side Encryption]] | | PutBucketEncryption | Yes | | | DeleteBucketEncryption | Yes | | -| GetBucketLifecycleConfiguration | Partial | TTL-based expiration only | -| PutBucketLifecycleConfiguration | Partial | TTL-based expiration only | -| DeleteBucketLifecycle | Partial | TTL-based expiration only | +| GetBucketLifecycleConfiguration | Yes | See [[S3 Lifecycle]] | +| PutBucketLifecycleConfiguration | Yes | Transition rules not supported | +| DeleteBucketLifecycle | Yes | | | GetBucketLocation | Yes | Returns default location | | GetBucketOwnershipControls | Yes | | | PutBucketOwnershipControls | Yes | | diff --git a/S3-Lifecycle.md b/S3-Lifecycle.md new file mode 100644 index 0000000..251eac2 --- /dev/null +++ b/S3-Lifecycle.md @@ -0,0 +1,146 @@ +# S3 Lifecycle Configuration + +SeaweedFS supports S3 bucket lifecycle configuration for automated object expiration, non-current version cleanup, delete marker removal, and incomplete multipart upload abortion. + +## Supported Features + +| Feature | Status | Notes | +|---|---|---| +| `Expiration.Days` | Supported | Fast path via TTL + RocksDB compaction filter | +| `Expiration.Date` | Supported | Evaluated by lifecycle worker at scan time | +| `ExpiredObjectDeleteMarker` | Supported | Removes delete markers that are the sole remaining version | +| `NoncurrentVersionExpiration.NoncurrentDays` | Supported | Requires bucket versioning enabled | +| `NoncurrentVersionExpiration.NewerNoncurrentVersions` | Supported | Keep N newest non-current versions | +| `AbortIncompleteMultipartUpload.DaysAfterInitiation` | Supported | Per-rule prefix scoping | +| `Filter.Prefix` | Supported | | +| `Filter.Tag` | Supported | Evaluated at scan time | +| `Filter.And` (Prefix + Tags + Size) | Supported | Evaluated at scan time | +| `Filter.ObjectSizeGreaterThan` | Supported | Evaluated at scan time | +| `Filter.ObjectSizeLessThan` | Supported | Evaluated at scan time | +| `Transition` | Not supported | Requires storage class tiers | +| `NoncurrentVersionTransition` | Not supported | Requires storage class tiers | + +## API Endpoints + +``` +PUT /{bucket}?lifecycle # PutBucketLifecycleConfiguration +GET /{bucket}?lifecycle # GetBucketLifecycleConfiguration +DELETE /{bucket}?lifecycle # DeleteBucketLifecycle +``` + +## Example: Terraform + +```hcl +resource "aws_s3_bucket_lifecycle_configuration" "example" { + bucket = aws_s3_bucket.example.id + + rule { + id = "expire-logs" + status = "Enabled" + + filter { + prefix = "logs/" + } + + expiration { + days = 30 + } + + noncurrent_version_expiration { + noncurrent_days = 7 + newer_noncurrent_versions = 2 + } + + abort_incomplete_multipart_upload { + days_after_initiation = 7 + } + } +} +``` + +## Example: AWS CLI + +```bash +# Set lifecycle configuration +aws s3api put-bucket-lifecycle-configuration \ + --endpoint-url http://localhost:8333 \ + --bucket my-bucket \ + --lifecycle-configuration '{ + "Rules": [ + { + "ID": "expire-old", + "Status": "Enabled", + "Filter": { "Prefix": "" }, + "Expiration": { "Days": 90 }, + "NoncurrentVersionExpiration": { + "NoncurrentDays": 30 + }, + "AbortIncompleteMultipartUpload": { + "DaysAfterInitiation": 7 + } + } + ] + }' + +# Get lifecycle configuration +aws s3api get-bucket-lifecycle-configuration \ + --endpoint-url http://localhost:8333 \ + --bucket my-bucket + +# Delete lifecycle configuration +aws s3api delete-bucket-lifecycle \ + --endpoint-url http://localhost:8333 \ + --bucket my-bucket +``` + +## How It Works + +### Two-Tier Expiration Architecture + +**Tier 1 — TTL fast path:** Simple `Expiration.Days` rules with prefix-only filters are translated into TTL entries in `filer.conf`. The RocksDB compaction filter automatically removes expired entries during normal compaction at zero additional cost. This is the most efficient path for the common case. + +**Tier 2 — Scan-time evaluation:** Rules with tag filters, size filters, date-based expiration, non-current version expiration, and delete marker cleanup are evaluated by the lifecycle plugin worker. The worker periodically scans buckets and evaluates each object against the stored lifecycle XML configuration. + +### Which rules use which tier? + +| Rule Type | Tier | Why | +|---|---|---| +| `Expiration.Days` (prefix only) | TTL fast path | Can be expressed as per-entry TTL | +| `Expiration.Days` (with tags/size) | Worker scan | TTL can't express tag/size constraints | +| `Expiration.Date` | Worker scan | Absolute date, not relative TTL | +| `NoncurrentVersionExpiration` | Worker scan | Requires version enumeration | +| `ExpiredObjectDeleteMarker` | Worker scan | Requires version counting | +| `AbortIncompleteMultipartUpload` | Worker scan | Scans `.uploads` directory | + +### Lifecycle Worker + +The lifecycle plugin worker runs as part of the SeaweedFS plugin system. It: + +1. **Detects** buckets with lifecycle rules (from stored lifecycle XML or filer.conf TTLs) +2. **Scans** bucket contents and evaluates lifecycle rules against each object +3. **Executes** actions: delete expired objects, remove old non-current versions, clean up delete markers, abort stale multipart uploads + +Worker configuration options: + +| Setting | Default | Description | +|---|---|---| +| `batch_size` | 1000 | Entries per filer listing page | +| `max_deletes_per_bucket` | 10000 | Max expired objects to delete per run | +| `dry_run` | false | Detect but don't delete | +| `delete_marker_cleanup` | true | Remove expired delete markers | +| `abort_mpu_days` | 7 | Fallback for buckets without lifecycle XML MPU rules | +| `bucket_filter` | (all) | Wildcard pattern to scope lifecycle to specific buckets | + +## Versioning Integration + +Lifecycle rules interact with [S3 Object Versioning](S3-Object-Versioning): + +- **`NoncurrentVersionExpiration`** only applies to versioned buckets. Non-current versions are deleted after `NoncurrentDays` days since they were superseded. `NewerNoncurrentVersions` retains the N newest non-current versions. +- **`ExpiredObjectDeleteMarker`** removes delete markers that are the sole remaining version of an object (no non-current versions behind them). +- **`Expiration.Days`** on a versioned bucket creates a delete marker when the current version expires; it does not permanently delete the object. + +## Limitations + +- **Transition rules** (`Transition`, `NoncurrentVersionTransition`) are not supported. SeaweedFS does not have S3-equivalent storage class tiers. +- **Detection interval**: The lifecycle worker scans on a configurable interval (default 5 minutes). Objects may persist slightly beyond their configured expiration until the next scan. +- **TTL fast path** applies only to `Expiration.Days` rules with prefix-only filters. Rules with tag or size constraints are always evaluated at scan time. diff --git a/_Sidebar.md b/_Sidebar.md index 7afb795..bbbb0cd 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -74,6 +74,7 @@ ### AWS S3 API * [[Amazon S3 API]] * [[Supported APIs vs Minio]] +* [[S3 Lifecycle]] * [[S3 Conditional Operations]] * [[S3 CORS]] * [[S3 Object Lock and Retention]]