diff --git a/Amazon-S3-API.md b/Amazon-S3-API.md index 7443ba7..3ab5c71 100644 --- a/Amazon-S3-API.md +++ b/Amazon-S3-API.md @@ -63,6 +63,9 @@ To be sure, you can look at the function defined in the files `weed/s3api/s3api_ * PutBucketLifecycleConfiguration (partially, only for TTL) * GetBucketLifecycleConfiguration (partially, only for TTL) * DeleteBucketLifecycleConfiguration (partially, only for TTL) +* GetBucketCors +* PutBucketCors +* DeleteBucketCors // Multipart upload operations * NewMultipartUpload @@ -270,139 +273,14 @@ Presigned URL is supported. See [[AWS-CLI-with-SeaweedFS#presigned-url]] for exa SeaweedFS supports S3 object versioning, which allows you to keep multiple variants of an object in the same bucket. This provides data protection against accidental deletion or modification. -## Enable Versioning +For detailed information about object versioning, see the dedicated [[S3-Object-Versioning]] page. -To enable versioning on a bucket, use the `PutBucketVersioning` API: -```bash -aws s3api put-bucket-versioning \ - --bucket my-bucket \ - --versioning-configuration Status=Enabled -``` +# S3 Cross-Origin Resource Sharing (CORS) -## Check Versioning Status +SeaweedFS supports S3-compatible Cross-Origin Resource Sharing (CORS) configuration, allowing web applications to make cross-origin requests to your S3 buckets. CORS is essential for web applications that need to access resources from different domains. -To check the versioning status of a bucket: - -```bash -aws s3api get-bucket-versioning --bucket my-bucket -``` - -Response: -```json -{ - "Status": "Enabled" -} -``` - -## Suspend Versioning - -To suspend versioning (not disable completely): - -```bash -aws s3api put-bucket-versioning \ - --bucket my-bucket \ - --versioning-configuration Status=Suspended -``` - -## List Object Versions - -To list all versions of objects in a bucket: - -```bash -aws s3api list-object-versions --bucket my-bucket -``` - -Response includes both object versions and delete markers: -```json -{ - "Versions": [ - { - "Key": "example.txt", - "VersionId": "v_1234567890abcdef", - "IsLatest": true, - "LastModified": "2023-12-01T10:00:00Z", - "ETag": "\"abcdef1234567890\"", - "Size": 1024 - } - ], - "DeleteMarkers": [ - { - "Key": "deleted-file.txt", - "VersionId": "v_fedcba0987654321", - "IsLatest": true, - "LastModified": "2023-12-01T11:00:00Z" - } - ] -} -``` - -## Access Specific Versions - -### Get a specific version of an object: -```bash -aws s3api get-object \ - --bucket my-bucket \ - --key example.txt \ - --version-id v_1234567890abcdef \ - output.txt -``` - -### Copy a specific version: -```bash -aws s3api copy-object \ - --copy-source my-bucket/example.txt?versionId=v_1234567890abcdef \ - --bucket my-bucket \ - --key example-copy.txt -``` - -### Delete a specific version: -```bash -aws s3api delete-object \ - --bucket my-bucket \ - --key example.txt \ - --version-id v_1234567890abcdef -``` - -## Versioning Behavior - -### When Versioning is Enabled: -- **PUT Object**: Creates a new version with a unique version ID -- **GET Object**: Returns the latest version (unless version ID is specified) -- **DELETE Object**: Creates a delete marker (soft delete) -- **DELETE Object with version ID**: Permanently deletes that specific version - -### When Versioning is Suspended: -- **PUT Object**: Overwrites the object with version ID "null" -- **GET Object**: Returns the current version -- **DELETE Object**: Permanently deletes the object - -## Storage Layout - -SeaweedFS stores versioned objects using the following structure: -``` -/buckets/my-bucket/ -├── example.txt # Current version (if versioning suspended) -└── example.txt.versions/ - ├── v_1234567890abcdef # Version 1 - ├── v_fedcba0987654321 # Version 2 - └── v_abcdef1234567890 # Version 3 (latest) -``` - -## Limitations - -- **Version ID Format**: SeaweedFS generates version IDs in the format `v_<32-char-hex>` -- **Restore Operations**: Only partial support for `RestoreObject` API -- **Lifecycle Policies**: Version-specific lifecycle rules are not fully implemented -- **MFA Delete**: Not currently supported - -## Best Practices - -1. **Enable versioning before storing important data** to ensure all versions are captured -2. **Monitor storage usage** as versioning can increase storage consumption -3. **Implement lifecycle policies** to automatically clean up old versions -4. **Use version-specific operations** when you need to access historical data -5. **Consider performance impact** when listing many versions of objects +For detailed information about CORS configuration, see the dedicated [[S3-CORS]] page. ## Multiple S3 Nodes diff --git a/S3-CORS.md b/S3-CORS.md new file mode 100644 index 0000000..f50789f --- /dev/null +++ b/S3-CORS.md @@ -0,0 +1,393 @@ +# S3 Cross-Origin Resource Sharing (CORS) + +SeaweedFS supports S3-compatible Cross-Origin Resource Sharing (CORS) configuration, allowing web applications to make cross-origin requests to your S3 buckets. CORS is essential for web applications that need to access resources from different domains. + +## Overview + +CORS defines a way for web applications running at one domain to access resources at another domain. When a web application tries to access your S3 bucket from a different domain, the browser will first send a preflight OPTIONS request to check if the cross-origin request is allowed. + +SeaweedFS handles CORS through: +- **Bucket-level CORS configuration**: Each bucket can have its own CORS rules +- **Persistent storage**: CORS configurations are stored in bucket metadata +- **Automatic header handling**: CORS middleware automatically applies appropriate headers +- **Preflight request support**: Proper handling of OPTIONS requests + +## CORS Configuration + +### Basic CORS Rule Structure + +A CORS configuration consists of one or more CORS rules. Each rule defines: + +```xml + + + rule-id + http://example.com + GET + POST + Content-Type + ETag + 3600 + + +``` + +### CORS Rule Elements + +- **ID** (optional): A unique identifier for the rule +- **AllowedOrigin** (required): Specifies which origins are allowed to access the bucket +- **AllowedMethod** (required): HTTP methods that are allowed (GET, PUT, POST, DELETE, HEAD) +- **AllowedHeader** (optional): Headers that are allowed in the actual request +- **ExposeHeader** (optional): Headers that browsers can access from the response +- **MaxAgeSeconds** (optional): How long browsers can cache the preflight response + +## Managing CORS Configuration + +### Set CORS Configuration + +Use the `PutBucketCors` API to set CORS configuration for a bucket: + +```bash +aws s3api put-bucket-cors \ + --bucket my-bucket \ + --cors-configuration file://cors-config.json +``` + +Example `cors-config.json`: +```json +{ + "CORSRules": [ + { + "ID": "allow-all-origins", + "AllowedOrigins": ["*"], + "AllowedMethods": ["GET", "POST", "PUT", "DELETE", "HEAD"], + "AllowedHeaders": ["*"], + "ExposeHeaders": ["ETag", "x-amz-version-id"], + "MaxAgeSeconds": 3600 + } + ] +} +``` + +### Get CORS Configuration + +Retrieve the current CORS configuration for a bucket: + +```bash +aws s3api get-bucket-cors --bucket my-bucket +``` + +Response: +```json +{ + "CORSRules": [ + { + "ID": "allow-all-origins", + "AllowedOrigins": ["*"], + "AllowedMethods": ["GET", "POST", "PUT", "DELETE", "HEAD"], + "AllowedHeaders": ["*"], + "ExposeHeaders": ["ETag", "x-amz-version-id"], + "MaxAgeSeconds": 3600 + } + ] +} +``` + +### Delete CORS Configuration + +Remove CORS configuration from a bucket: + +```bash +aws s3api delete-bucket-cors --bucket my-bucket +``` + +## CORS Rule Examples + +### Example 1: Allow Specific Domain + +```json +{ + "CORSRules": [ + { + "ID": "allow-example-domain", + "AllowedOrigins": ["https://example.com"], + "AllowedMethods": ["GET", "POST"], + "AllowedHeaders": ["Content-Type", "Authorization"], + "ExposeHeaders": ["ETag"], + "MaxAgeSeconds": 1800 + } + ] +} +``` + +### Example 2: Allow Multiple Domains + +```json +{ + "CORSRules": [ + { + "ID": "allow-multiple-domains", + "AllowedOrigins": [ + "https://app.example.com", + "https://staging.example.com", + "https://localhost:3000" + ], + "AllowedMethods": ["GET", "PUT", "POST", "DELETE"], + "AllowedHeaders": ["*"], + "ExposeHeaders": ["ETag", "x-amz-version-id"], + "MaxAgeSeconds": 3600 + } + ] +} +``` + +### Example 3: Wildcard Domain Support + +```json +{ + "CORSRules": [ + { + "ID": "allow-subdomain-wildcard", + "AllowedOrigins": ["https://*.example.com"], + "AllowedMethods": ["GET", "POST"], + "AllowedHeaders": ["Content-Type"], + "MaxAgeSeconds": 1800 + } + ] +} +``` + +### Example 4: Multiple Rules + +```json +{ + "CORSRules": [ + { + "ID": "read-only-rule", + "AllowedOrigins": ["*"], + "AllowedMethods": ["GET", "HEAD"], + "AllowedHeaders": ["*"], + "MaxAgeSeconds": 3600 + }, + { + "ID": "write-rule", + "AllowedOrigins": ["https://admin.example.com"], + "AllowedMethods": ["PUT", "POST", "DELETE"], + "AllowedHeaders": ["Content-Type", "Authorization"], + "ExposeHeaders": ["ETag"], + "MaxAgeSeconds": 1800 + } + ] +} +``` + +## Web Application Integration + +### JavaScript Example + +```javascript +// Configure your S3 client +const AWS = require('aws-sdk'); +const s3 = new AWS.S3({ + accessKeyId: 'your-access-key', + secretAccessKey: 'your-secret-key', + endpoint: 'http://localhost:8333', + s3ForcePathStyle: true, + region: 'us-east-1' +}); + +// Upload file from web application +const uploadFile = async (file, bucket, key) => { + const params = { + Bucket: bucket, + Key: key, + Body: file, + ContentType: file.type + }; + + try { + const result = await s3.upload(params).promise(); + console.log('Upload successful:', result); + return result; + } catch (error) { + console.error('Upload failed:', error); + throw error; + } +}; + +// Download file from web application +const downloadFile = async (bucket, key) => { + const params = { + Bucket: bucket, + Key: key + }; + + try { + const result = await s3.getObject(params).promise(); + return result.Body; + } catch (error) { + console.error('Download failed:', error); + throw error; + } +}; +``` + +### HTML Upload Form Example + +```html + + + + S3 Upload Example + + + + + + + + +``` + +## CORS Rule Evaluation + +SeaweedFS evaluates CORS rules in the following order: + +1. **Rule Matching**: The first rule that matches the request origin is used +2. **Origin Validation**: Check if the request origin matches any `AllowedOrigin` +3. **Method Validation**: For preflight requests, validate the requested method +4. **Header Validation**: For preflight requests, validate all requested headers +5. **Response Building**: Build appropriate CORS headers based on the matched rule + +### Preflight Request Handling + +For preflight requests (OPTIONS method), SeaweedFS: + +1. Checks if the origin matches an allowed origin +2. Validates the requested method against allowed methods +3. Validates all requested headers against allowed headers +4. Returns appropriate CORS headers if all validations pass +5. Returns 403 Forbidden if any validation fails + +### Actual Request Handling + +For actual requests (GET, POST, PUT, DELETE, etc.), SeaweedFS: + +1. Checks if the origin matches an allowed origin +2. Validates the request method against allowed methods +3. Applies appropriate CORS headers to the response +4. Continues with normal request processing + +## Performance and Caching + +### CORS Configuration Caching + +- CORS configurations are cached in memory for 5 minutes +- Cache is automatically invalidated when configuration changes +- Multiple S3 nodes share the same cached configuration + +### Browser Caching + +- Use `MaxAgeSeconds` to control how long browsers cache preflight responses +- Longer cache times reduce preflight requests but delay configuration changes +- Recommended values: 1800-3600 seconds (30-60 minutes) + +## Security Considerations + +### Origin Validation + +- Never use `*` for `AllowedOrigin` in production unless absolutely necessary +- Specify exact domains or use specific wildcard patterns +- Validate all origins against your application's requirements + +### Method Restrictions + +- Only allow necessary HTTP methods +- Restrict write operations (PUT, POST, DELETE) to trusted origins +- Consider separate rules for read-only vs. write operations + +### Header Security + +- Avoid using `*` for `AllowedHeaders` in production +- Only allow headers that your application actually needs +- Be cautious with authorization headers + +## Troubleshooting + +### Common Issues + +1. **CORS policy error**: Check that your origin is listed in `AllowedOrigins` +2. **Method not allowed**: Ensure the HTTP method is in `AllowedMethods` +3. **Header blocked**: Add required headers to `AllowedHeaders` +4. **Preflight failure**: Verify all preflight requirements are met + +### Debugging Tips + +- Use browser developer tools to inspect CORS headers +- Check server logs for CORS-related errors +- Test with simple requests first, then add complexity +- Verify bucket-level CORS configuration is correct + +### Testing CORS Configuration + +```bash +# Test preflight request +curl -X OPTIONS \ + -H "Origin: https://example.com" \ + -H "Access-Control-Request-Method: GET" \ + -H "Access-Control-Request-Headers: Content-Type" \ + http://localhost:8333/my-bucket/test-object + +# Test actual request +curl -X GET \ + -H "Origin: https://example.com" \ + http://localhost:8333/my-bucket/test-object +``` + +## Limitations + +- Maximum 100 CORS rules per bucket +- Wildcard support is limited to `*` character +- Complex regex patterns in origins are not supported +- CORS configuration is per-bucket, not per-object + +## Best Practices + +1. **Specify exact origins** instead of using wildcards when possible +2. **Use appropriate MaxAgeSeconds** to balance performance and flexibility +3. **Implement proper error handling** in your web applications +4. **Test CORS configuration thoroughly** before deploying to production +5. **Monitor CORS usage** and adjust rules as needed +6. **Keep CORS rules simple** and well-documented +7. **Use separate rules** for different access patterns (read vs. write) \ No newline at end of file diff --git a/S3-Object-Versioning.md b/S3-Object-Versioning.md new file mode 100644 index 0000000..24780b4 --- /dev/null +++ b/S3-Object-Versioning.md @@ -0,0 +1,137 @@ +# S3 Object Versioning + +SeaweedFS supports S3 object versioning, which allows you to keep multiple variants of an object in the same bucket. This provides data protection against accidental deletion or modification. + +## Enable Versioning + +To enable versioning on a bucket, use the `PutBucketVersioning` API: + +```bash +aws s3api put-bucket-versioning \ + --bucket my-bucket \ + --versioning-configuration Status=Enabled +``` + +## Check Versioning Status + +To check the versioning status of a bucket: + +```bash +aws s3api get-bucket-versioning --bucket my-bucket +``` + +Response: +```json +{ + "Status": "Enabled" +} +``` + +## Suspend Versioning + +To suspend versioning (not disable completely): + +```bash +aws s3api put-bucket-versioning \ + --bucket my-bucket \ + --versioning-configuration Status=Suspended +``` + +## List Object Versions + +To list all versions of objects in a bucket: + +```bash +aws s3api list-object-versions --bucket my-bucket +``` + +Response includes both object versions and delete markers: +```json +{ + "Versions": [ + { + "Key": "example.txt", + "VersionId": "v_1234567890abcdef", + "IsLatest": true, + "LastModified": "2023-12-01T10:00:00Z", + "ETag": "\"abcdef1234567890\"", + "Size": 1024 + } + ], + "DeleteMarkers": [ + { + "Key": "deleted-file.txt", + "VersionId": "v_fedcba0987654321", + "IsLatest": true, + "LastModified": "2023-12-01T11:00:00Z" + } + ] +} +``` + +## Access Specific Versions + +### Get a specific version of an object: +```bash +aws s3api get-object \ + --bucket my-bucket \ + --key example.txt \ + --version-id v_1234567890abcdef \ + output.txt +``` + +### Copy a specific version: +```bash +aws s3api copy-object \ + --copy-source my-bucket/example.txt?versionId=v_1234567890abcdef \ + --bucket my-bucket \ + --key example-copy.txt +``` + +### Delete a specific version: +```bash +aws s3api delete-object \ + --bucket my-bucket \ + --key example.txt \ + --version-id v_1234567890abcdef +``` + +## Versioning Behavior + +### When Versioning is Enabled: +- **PUT Object**: Creates a new version with a unique version ID +- **GET Object**: Returns the latest version (unless version ID is specified) +- **DELETE Object**: Creates a delete marker (soft delete) +- **DELETE Object with version ID**: Permanently deletes that specific version + +### When Versioning is Suspended: +- **PUT Object**: Overwrites the object with version ID "null" +- **GET Object**: Returns the current version +- **DELETE Object**: Permanently deletes the object + +## Storage Layout + +SeaweedFS stores versioned objects using the following structure: +``` +/buckets/my-bucket/ +├── example.txt # Current version (if versioning suspended) +└── example.txt.versions/ + ├── v_1234567890abcdef # Version 1 + ├── v_fedcba0987654321 # Version 2 + └── v_abcdef1234567890 # Version 3 (latest) +``` + +## Limitations + +- **Version ID Format**: SeaweedFS generates version IDs in the format `v_<32-char-hex>` +- **Restore Operations**: Only partial support for `RestoreObject` API +- **Lifecycle Policies**: Version-specific lifecycle rules are not fully implemented +- **MFA Delete**: Not currently supported + +## Best Practices + +1. **Enable versioning before storing important data** to ensure all versions are captured +2. **Monitor storage usage** as versioning can increase storage consumption +3. **Implement lifecycle policies** to automatically clean up old versions +4. **Use version-specific operations** when you need to access historical data +5. **Consider performance impact** when listing many versions of objects \ No newline at end of file