From ab03887e37350dcb430476eafbd914864836b8b0 Mon Sep 17 00:00:00 2001 From: chrislusf Date: Sun, 13 Jul 2025 20:15:01 -0700 Subject: [PATCH] add object locking and retention --- Amazon-S3-API.md | 160 ++++++++++++ S3-API-FAQ.md | 62 +++++ S3-Object-Lock-and-Retention.md | 424 ++++++++++++++++++++++++++++++++ _Sidebar.md | 1 + 4 files changed, 647 insertions(+) create mode 100644 S3-Object-Lock-and-Retention.md diff --git a/Amazon-S3-API.md b/Amazon-S3-API.md index d08f558..7443ba7 100644 --- a/Amazon-S3-API.md +++ b/Amazon-S3-API.md @@ -72,6 +72,25 @@ To be sure, you can look at the function defined in the files `weed/s3api/s3api_ * PutObjectPart * CopyObjectPart * ListObjectParts + +// Object Versioning +* PutBucketVersioning +* GetBucketVersioning +* ListObjectVersions +* GetObject (with version ID) +* PutObject (with versioning) +* DeleteObject (with version ID) +* CopyObject (with version ID) +* RestoreObject (partial) + +// Object Lock and Retention +* GetObjectLockConfiguration +* PutObjectLockConfiguration +* GetObjectRetention +* PutObjectRetention +* GetObjectLegalHold +* PutObjectLegalHold +* BypassGovernanceRetention (via x-amz-bypass-governance-retention header) ``` Not included: @@ -85,6 +104,8 @@ Not included: | DeleteObject deletes a folder | Yes | No | | same path for both a file and a folder | No | Yes | | allows more than "/" as a delimiter | No | Yes | +| Object Versioning | Yes | Yes | +| MFA Delete for versioning | No | Yes | ## Empty folders @@ -245,6 +266,145 @@ Read:bucket/a/b/* Presigned URL is supported. See [[AWS-CLI-with-SeaweedFS#presigned-url]] for example. +# S3 Object Versioning + +SeaweedFS supports S3 object versioning, which allows you to keep multiple variants of an object in the same bucket. This provides data protection against accidental deletion or modification. + +## Enable Versioning + +To enable versioning on a bucket, use the `PutBucketVersioning` API: + +```bash +aws s3api put-bucket-versioning \ + --bucket my-bucket \ + --versioning-configuration Status=Enabled +``` + +## Check Versioning Status + +To check the versioning status of a bucket: + +```bash +aws s3api get-bucket-versioning --bucket my-bucket +``` + +Response: +```json +{ + "Status": "Enabled" +} +``` + +## Suspend Versioning + +To suspend versioning (not disable completely): + +```bash +aws s3api put-bucket-versioning \ + --bucket my-bucket \ + --versioning-configuration Status=Suspended +``` + +## List Object Versions + +To list all versions of objects in a bucket: + +```bash +aws s3api list-object-versions --bucket my-bucket +``` + +Response includes both object versions and delete markers: +```json +{ + "Versions": [ + { + "Key": "example.txt", + "VersionId": "v_1234567890abcdef", + "IsLatest": true, + "LastModified": "2023-12-01T10:00:00Z", + "ETag": "\"abcdef1234567890\"", + "Size": 1024 + } + ], + "DeleteMarkers": [ + { + "Key": "deleted-file.txt", + "VersionId": "v_fedcba0987654321", + "IsLatest": true, + "LastModified": "2023-12-01T11:00:00Z" + } + ] +} +``` + +## Access Specific Versions + +### Get a specific version of an object: +```bash +aws s3api get-object \ + --bucket my-bucket \ + --key example.txt \ + --version-id v_1234567890abcdef \ + output.txt +``` + +### Copy a specific version: +```bash +aws s3api copy-object \ + --copy-source my-bucket/example.txt?versionId=v_1234567890abcdef \ + --bucket my-bucket \ + --key example-copy.txt +``` + +### Delete a specific version: +```bash +aws s3api delete-object \ + --bucket my-bucket \ + --key example.txt \ + --version-id v_1234567890abcdef +``` + +## Versioning Behavior + +### When Versioning is Enabled: +- **PUT Object**: Creates a new version with a unique version ID +- **GET Object**: Returns the latest version (unless version ID is specified) +- **DELETE Object**: Creates a delete marker (soft delete) +- **DELETE Object with version ID**: Permanently deletes that specific version + +### When Versioning is Suspended: +- **PUT Object**: Overwrites the object with version ID "null" +- **GET Object**: Returns the current version +- **DELETE Object**: Permanently deletes the object + +## Storage Layout + +SeaweedFS stores versioned objects using the following structure: +``` +/buckets/my-bucket/ +├── example.txt # Current version (if versioning suspended) +└── example.txt.versions/ + ├── v_1234567890abcdef # Version 1 + ├── v_fedcba0987654321 # Version 2 + └── v_abcdef1234567890 # Version 3 (latest) +``` + +## Limitations + +- **Version ID Format**: SeaweedFS generates version IDs in the format `v_<32-char-hex>` +- **Restore Operations**: Only partial support for `RestoreObject` API +- **Lifecycle Policies**: Version-specific lifecycle rules are not fully implemented +- **MFA Delete**: Not currently supported + +## Best Practices + +1. **Enable versioning before storing important data** to ensure all versions are captured +2. **Monitor storage usage** as versioning can increase storage consumption +3. **Implement lifecycle policies** to automatically clean up old versions +4. **Use version-specific operations** when you need to access historical data +5. **Consider performance impact** when listing many versions of objects + + ## Multiple S3 Nodes If you need to setup multiple S3 nodes, you can just start multiple s3 instances pointing to a filer. diff --git a/S3-API-FAQ.md b/S3-API-FAQ.md index c1ca16e..728c6d6 100644 --- a/S3-API-FAQ.md +++ b/S3-API-FAQ.md @@ -88,4 +88,66 @@ aws --endpoint-url http://127.0.0.1:8333 s3api put-bucket-lifecycle-configuratio Note that you don't need to add the part `/buckets/$BUCKET_NAME` in the configurations "Filter.Prefix" (contrary to using `fs.configure`, this is taken care of for you in the S3 API. +## Does SeaweedFS support S3 object versioning? + +Yes, SeaweedFS supports S3 object versioning. You can enable versioning on a bucket using the `PutBucketVersioning` API. When enabled, SeaweedFS will store multiple versions of an object in the same bucket, providing data protection against accidental deletion or modification. + +Key features supported: +- Enable/suspend versioning on buckets +- List all versions of objects +- Get, copy, and delete specific versions +- Delete markers for soft deletion + +For detailed documentation and examples, see [[Amazon S3 API#s3-object-versioning]]. + +## How does versioning affect storage usage? + +When versioning is enabled, each uploaded object creates a new version instead of overwriting the existing one. This means: +- Storage usage will increase as you accumulate versions +- All versions are preserved until explicitly deleted +- Delete operations create delete markers (soft delete) rather than immediately removing data + +To manage storage growth, you should: +- Monitor storage usage regularly +- Implement lifecycle policies to automatically clean up old versions +- Use version-specific deletions for permanent removal when needed + +## Does SeaweedFS support S3 Object Lock? + +Yes! SeaweedFS provides comprehensive support for S3 Object Lock features, including: + +### Object Lock Features +- **Governance Mode**: Objects can be deleted/modified by users with `s3:BypassGovernanceRetention` permission +- **Compliance Mode**: Objects cannot be deleted/modified by any user until retention expires +- **Legal Hold**: Objects cannot be deleted/modified until legal hold is explicitly removed + +### Supported APIs +- `GetObjectLockConfiguration` / `PutObjectLockConfiguration` (bucket-level) +- `GetObjectRetention` / `PutObjectRetention` (object-level) +- `GetObjectLegalHold` / `PutObjectLegalHold` (object-level) +- Governance bypass via `x-amz-bypass-governance-retention` header + +### Requirements +- Object Lock must be enabled when creating the bucket (cannot be added later) +- Versioning is automatically enabled and required for Object Lock +- Compatible with standard AWS S3 tools and SDKs + +For complete documentation, examples, and best practices, see [[S3 Object Lock and Retention]]. + +## What's the difference between Governance and Compliance modes? + +**Governance Mode**: +- Designed for internal governance and compliance requirements +- Can be bypassed by users with proper permissions (`s3:BypassGovernanceRetention`) +- Admin users can always bypass governance retention +- Suitable for testing and development environments + +**Compliance Mode**: +- Designed for regulatory compliance (SEC, FINRA, etc.) +- Cannot be bypassed by any user, including root/admin +- Provides the highest level of data protection +- Suitable for production environments with strict compliance requirements + +Both modes prevent accidental deletion and provide audit trails for compliance purposes. + diff --git a/S3-Object-Lock-and-Retention.md b/S3-Object-Lock-and-Retention.md new file mode 100644 index 0000000..6af1ebb --- /dev/null +++ b/S3-Object-Lock-and-Retention.md @@ -0,0 +1,424 @@ +# S3 Object Lock and Retention + +SeaweedFS provides comprehensive support for Amazon S3 Object Lock functionality, including object versioning, retention policies, legal holds, and WORM (Write Once Read Many) compliance features. + +## Overview + +Object Lock is a feature that allows you to store objects using a WORM (Write Once Read Many) model. It helps prevent objects from being deleted or overwritten for a fixed amount of time or indefinitely. Object Lock works in conjunction with versioning and is enabled at the bucket level. + +## Prerequisites + +- **Versioning Required**: Object Lock can only be enabled on buckets that have versioning enabled +- **Immutable Setting**: Object Lock can only be enabled when creating a bucket, not on existing buckets +- **S3 API**: All Object Lock operations are available through the S3 API + +## Key Features + +### 1. Retention Modes + +#### Governance Mode +- Objects are protected from deletion or modification by most users +- Users with `s3:BypassGovernanceRetention` permission can override protection +- Admin users can always bypass governance retention +- Suitable for internal compliance and governance requirements + +#### Compliance Mode +- Objects cannot be deleted or modified by any user, including the root user +- Retention periods cannot be shortened +- Provides the highest level of protection +- Suitable for regulatory compliance requirements + +### 2. Legal Hold +- Independent protection mechanism that can be applied to any object +- When enabled, objects cannot be deleted or modified until the legal hold is removed +- Can be used alongside retention policies +- Useful for litigation, investigation, or audit requirements + + + +## API Support + +SeaweedFS implements the complete S3 Object Lock API: + +### Bucket-Level Operations +- `GET /?object-lock` - Get bucket Object Lock configuration +- `PUT /?object-lock` - Set bucket Object Lock configuration + +### Object-Level Operations +- `GET /{object}?retention` - Get object retention settings +- `PUT /{object}?retention` - Set object retention settings +- `GET /{object}?legal-hold` - Get object legal hold status +- `PUT /{object}?legal-hold` - Set object legal hold status + +### Versioning Operations +- `GET /?versioning` - Get bucket versioning status +- `PUT /?versioning` - Set bucket versioning status +- `GET /?versions` - List object versions + +## Setup and Configuration + +### 1. Enable Object Lock on Bucket Creation + +Object Lock must be enabled when creating a bucket: + +```bash +# Create bucket with Object Lock enabled +aws s3api create-bucket \ + --bucket my-secure-bucket \ + --object-lock-enabled-for-bucket + +# Enable versioning (automatically done with Object Lock) +aws s3api put-bucket-versioning \ + --bucket my-secure-bucket \ + --versioning-configuration Status=Enabled +``` + +### 2. Configure Governance Bypass Permissions + +Grant users permission to bypass governance retention: + +```json +{ + "identities": [ + { + "name": "compliance-admin", + "credentials": [ + { + "accessKey": "admin123", + "secretKey": "secret123" + } + ], + "actions": [ + "Read:my-secure-bucket/*", + "Write:my-secure-bucket/*", + "BypassGovernanceRetention:my-secure-bucket/*" + ] + } + ] +} +``` + +## Usage Examples + +### 1. Object Retention + +#### Set Governance Retention +```bash +# Set 30-day governance retention +aws s3api put-object-retention \ + --bucket my-secure-bucket \ + --key important-document.pdf \ + --retention '{ + "Mode": "GOVERNANCE", + "RetainUntilDate": "2024-12-31T23:59:59Z" + }' +``` + +#### Set Compliance Retention +```bash +# Set 7-year compliance retention +aws s3api put-object-retention \ + --bucket my-secure-bucket \ + --key regulatory-record.json \ + --retention '{ + "Mode": "COMPLIANCE", + "RetainUntilDate": "2031-01-01T00:00:00Z" + }' +``` + +#### Get Object Retention +```bash +aws s3api get-object-retention \ + --bucket my-secure-bucket \ + --key important-document.pdf +``` + +### 2. Legal Hold + +#### Apply Legal Hold +```bash +aws s3api put-object-legal-hold \ + --bucket my-secure-bucket \ + --key investigation-file.doc \ + --legal-hold Status=ON +``` + +#### Remove Legal Hold +```bash +aws s3api put-object-legal-hold \ + --bucket my-secure-bucket \ + --key investigation-file.doc \ + --legal-hold Status=OFF +``` + +#### Check Legal Hold Status +```bash +aws s3api get-object-legal-hold \ + --bucket my-secure-bucket \ + --key investigation-file.doc +``` + +### 3. Governance Bypass + +#### Delete Object with Governance Bypass +```bash +# User with bypass permission can delete governance-protected objects +aws s3api delete-object \ + --bucket my-secure-bucket \ + --key document.pdf \ + --bypass-governance-retention +``` + +#### Bulk Delete with Governance Bypass +```bash +# Delete multiple objects with governance bypass +aws s3api delete-objects \ + --bucket my-secure-bucket \ + --delete file://delete-objects.json \ + --bypass-governance-retention +``` + +### 4. Object Versioning + +#### List Object Versions +```bash +aws s3api list-object-versions \ + --bucket my-secure-bucket \ + --prefix documents/ +``` + +#### Get Specific Version +```bash +aws s3api get-object \ + --bucket my-secure-bucket \ + --key document.pdf \ + --version-id "3/L4kqtJlcpXroDTDmpUMLUo" +``` + +#### Delete Specific Version +```bash +aws s3api delete-object \ + --bucket my-secure-bucket \ + --key document.pdf \ + --version-id "3/L4kqtJlcpXroDTDmpUMLUo" +``` + +## Permission System + +### IAM Actions + +SeaweedFS supports the following S3 IAM actions for Object Lock in identity-based permissions: + +- `s3:GetObjectRetention` - Get object retention settings +- `s3:PutObjectRetention` - Set object retention settings +- `s3:GetObjectLegalHold` - Get object legal hold status +- `s3:PutObjectLegalHold` - Set object legal hold status +- `s3:BypassGovernanceRetention` - Bypass governance retention +- `s3:GetBucketObjectLockConfiguration` - Get bucket Object Lock configuration +- `s3:PutBucketObjectLockConfiguration` - Set bucket Object Lock configuration + +### Identity-Based Permissions + +Configure user permissions in `identities.json`: + +```json +{ + "identities": [ + { + "name": "read-only-user", + "credentials": [{"accessKey": "ro123", "secretKey": "secret123"}], + "actions": [ + "Read:my-secure-bucket/*", + "GetObjectRetention:my-secure-bucket/*", + "GetObjectLegalHold:my-secure-bucket/*" + ] + }, + { + "name": "compliance-admin", + "credentials": [{"accessKey": "admin123", "secretKey": "secret123"}], + "actions": [ + "Read:my-secure-bucket/*", + "Write:my-secure-bucket/*", + "Admin:my-secure-bucket/*", + "BypassGovernanceRetention:my-secure-bucket/*" + ] + } + ] +} +``` + +## Error Handling + +### Common Error Scenarios + +1. **Object Lock Not Enabled** + - Error: `InvalidRequest` + - Solution: Enable Object Lock when creating the bucket + +2. **Versioning Not Enabled** + - Error: `InvalidRequest` + - Solution: Enable versioning before configuring Object Lock + +3. **Governance Bypass Not Permitted** + - Error: `AccessDenied` + - Solution: Grant `s3:BypassGovernanceRetention` permission + +4. **Compliance Mode Cannot Be Bypassed** + - Error: `AccessDenied` + - Solution: Wait for retention period to expire + +5. **Object Under Legal Hold** + - Error: `AccessDenied` + - Solution: Remove legal hold before deleting/modifying + +### Error Response Examples + +```xml + + + AccessDenied + Object is under COMPLIANCE mode retention and cannot be deleted + /my-secure-bucket/protected-file.pdf + 12345 + +``` + +## WORM Compliance + +SeaweedFS Object Lock provides WORM (Write Once Read Many) compliance: + +### Features +- **Immutable Objects**: Once written, objects cannot be modified +- **Retention Enforcement**: Objects cannot be deleted before retention expiry +- **Legal Hold Support**: Additional protection for litigation/investigation +- **Audit Trail**: All access attempts are logged for compliance + +### Use Cases +- **Regulatory Compliance**: Financial records, healthcare data, legal documents +- **Data Archiving**: Long-term storage with guaranteed integrity +- **Backup Protection**: Prevent accidental or malicious deletion +- **Audit Requirements**: Maintain evidence for legal proceedings + +## Best Practices + +### 1. Retention Planning +- **Start with Governance**: Use governance mode for testing and development +- **Compliance for Production**: Use compliance mode for regulatory requirements +- **Gradual Implementation**: Start with short retention periods and increase gradually + +### 2. Permission Management +- **Principle of Least Privilege**: Grant minimum required permissions +- **Separate Roles**: Different permissions for different use cases +- **Regular Audits**: Review and update permissions regularly + +### 3. Monitoring and Alerting +- **Monitor Bypass Attempts**: Track governance bypass usage +- **Retention Expiry Alerts**: Set up alerts for expiring retention periods +- **Failed Access Attempts**: Monitor and investigate access denials + +### 4. Backup and Recovery +- **Multiple Copies**: Maintain backups of Object Lock configurations +- **Cross-Region Replication**: Replicate to different regions for disaster recovery +- **Version Management**: Keep track of object versions and their retention status + +## Performance Considerations + +### Storage Efficiency +- **Versioning Overhead**: Each version consumes storage space +- **Retention Metadata**: Minimal overhead for Object Lock metadata +- **Cleanup Strategies**: Plan for version cleanup after retention expires + +### Access Patterns +- **Read-Heavy Workloads**: Object Lock is optimized for read operations +- **Write-Once Pattern**: Best suited for write-once, read-many scenarios +- **Batch Operations**: Use batch operations for bulk retention management + +## Limitations + +### Current Limitations +- **Bucket-Level Only**: Object Lock must be enabled during bucket creation +- **No Modification**: Existing buckets cannot be converted to Object Lock +- **Retention Limits**: Maximum 100 years retention period +- **Version Limits**: Consider version proliferation in high-write scenarios + +### AWS Compatibility +SeaweedFS Object Lock is fully compatible with AWS S3 Object Lock: +- Same API endpoints and parameters +- Identical error responses +- Compatible with AWS CLI and SDKs +- Consistent behavior across governance and compliance modes + +## Troubleshooting + +### Common Issues + +1. **Cannot Enable Object Lock** + - Check if versioning is enabled + - Ensure bucket is being created (not existing) + - Verify permissions for bucket operations + +2. **Retention Settings Not Applied** + - Verify Object Lock is enabled on bucket + - Check retention date format (ISO 8601) + - Ensure retention mode is valid (GOVERNANCE/COMPLIANCE) + +3. **Governance Bypass Fails** + - Verify user has `s3:BypassGovernanceRetention` permission + - Check if bypass header is included in request + - Ensure object is in GOVERNANCE mode (not COMPLIANCE) + +### Debug Commands + +```bash +# Check bucket Object Lock configuration +aws s3api get-object-lock-configuration --bucket my-secure-bucket + +# Check bucket versioning status +aws s3api get-bucket-versioning --bucket my-secure-bucket + +# Check object retention +aws s3api get-object-retention --bucket my-secure-bucket --key file.pdf + +# Check legal hold status +aws s3api get-object-legal-hold --bucket my-secure-bucket --key file.pdf +``` + +## Migration Guide + +### From Non-Object Lock Buckets + +1. **Create New Bucket**: Create new bucket with Object Lock enabled +2. **Copy Data**: Use `aws s3 sync` to copy data to new bucket +3. **Apply Retention**: Set retention policies on copied objects +4. **Update Applications**: Update applications to use new bucket +5. **Cleanup**: Remove old bucket after validation + +### From Other S3 Providers + +Object Lock configurations can be migrated using standard S3 API calls: + +```bash +# Export current configuration +aws s3api get-object-lock-configuration --bucket source-bucket > config.json + +# Apply to SeaweedFS bucket +aws s3api put-object-lock-configuration \ + --bucket target-bucket \ + --object-lock-configuration file://config.json +``` + + + +## Related Documentation + +- [S3 API FAQ](S3-API-FAQ.md) +- [Amazon S3 API](Amazon-S3-API.md) +- [Security Configuration](Security-Configuration.md) +- [Production Setup](Production-Setup.md) + +## Support + +For issues or questions about Object Lock features: +- GitHub Issues: [SeaweedFS Issues](https://github.com/seaweedfs/seaweedfs/issues) +- Community Forum: [SeaweedFS Discussions](https://github.com/seaweedfs/seaweedfs/discussions) +- Documentation: [SeaweedFS Wiki](https://github.com/seaweedfs/seaweedfs/wiki) \ No newline at end of file diff --git a/_Sidebar.md b/_Sidebar.md index d9762d2..f5382b8 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -67,6 +67,7 @@ ### AWS S3 API * [[Amazon S3 API]] +* [[S3 Object Lock and Retention]] * [[AWS CLI with SeaweedFS]] * [[s3cmd with SeaweedFS]] * [[rclone with SeaweedFS]]