From 6503963c27c510a69fbf4a31bdb11b35894485bd Mon Sep 17 00:00:00 2001 From: chrislusf Date: Wed, 23 Jul 2025 11:45:04 -0700 Subject: [PATCH] add s3 credentials --- Environment-Variables.md | 53 ++------ S3-Credentials.md | 257 +++++++++++++++++++++++++++++++++++++++ _Sidebar.md | 1 + 3 files changed, 271 insertions(+), 40 deletions(-) create mode 100644 S3-Credentials.md diff --git a/Environment-Variables.md b/Environment-Variables.md index 45b1518..49e4672 100644 --- a/Environment-Variables.md +++ b/Environment-Variables.md @@ -14,50 +14,28 @@ weed master For `v`, `logtostderr`, `stderrthreshold`, `vmoudle`, `options`, `logdir`, `alsologtostderr`, `log_backtrace_at` , and `config_dir` you have to use `WEED_` as prefix for environment variable like this `WEED_CONFIG_DIR=/tmp` # S3 Admin Credentials -For S3 API server, you can use standard AWS environment variables to set default admin credentials: - -```shell -export AWS_ACCESS_KEY_ID=your_access_key -export AWS_SECRET_ACCESS_KEY=your_secret_key -weed s3 -filer=localhost:8888 -``` - -These environment variables will supplement existing configuration by adding admin credentials when: -- Both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are set -- No identity with the same access key already exists - -**Configuration Priority:** -1. **File or Filer configuration** is loaded first (if provided) -2. **Environment variables** are added second, supplementing the existing configuration -3. Environment variables are **skipped** if an access key conflict exists - -This approach ensures that: -- Existing S3 configuration files (with `admin_access` etc.) are preserved -- Environment variables provide additional admin access without conflicts -- AWS standard environment variables work seamlessly alongside existing setups - -The admin identity from environment variables will have: -- Name: `admin-` followed by the access key (or first 8 characters if longer than 8) -- Access Key: value of `AWS_ACCESS_KEY_ID` -- Secret Key: value of `AWS_SECRET_ACCESS_KEY` -- Permissions: Full admin access to all S3 operations - -This follows the standard AWS credential convention, making it compatible with existing AWS tooling and workflows. +For S3 API server authentication, see the dedicated **[S3 Credentials](S3-Credentials)** page which covers: +- Configuration file setup (highest priority) +- Filer configuration (medium priority) +- Environment variables as fallback (lowest priority) +- AWS standard environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) +- Complete authentication examples and troubleshooting # Docker -This is useful for using docker and docker compose -You have to override entrypoint to `weed` because defautl [entrypoint](https://github.com/seaweedfs/seaweedfs/blob/master/docker/entrypoint.sh) use default values for `volumeSizeLimitMB`, `volumePreallocate`, `mdir`, `dir`, and `max` and setting environment variables won't change these values. -## Docker +You can set environment variables easily in Docker: ```shell -docker run --entrypoint weed -it -e IP_BIND=0.0.0.0 -e MDIR=/tmp -e PORT=5000 -e VOLUMEPREALLOCATE=true chrislusf/seaweedfs:3.45 master +docker run --name master -d -p 9333:9333 -p 19333:19333 \ + -e MDIR="/data" -e PORT="9333" \ + chrislusf/seaweedfs:latest \ + master ``` -## Docker Compose +## Docker Compose with Environment Variables ```yaml version: '3.9' services: master: - image: chrislusf/seaweedfs:3.45 + image: chrislusf/seaweedfs:latest ports: - 9333:9333 - 19333:19333 @@ -69,12 +47,7 @@ services: # or `VOLUMEPREALLOCATE:` entrypoint: weed command: master -``` -## Docker Compose with S3 Admin Credentials -```yaml -version: '3.9' -services: filer: image: chrislusf/seaweedfs:latest ports: diff --git a/S3-Credentials.md b/S3-Credentials.md new file mode 100644 index 0000000..30c175e --- /dev/null +++ b/S3-Credentials.md @@ -0,0 +1,257 @@ +# S3 Credentials + +SeaweedFS S3 API supports multiple authentication methods with a clear priority system. This page explains how to configure S3 credentials for your SeaweedFS setup. + +## Authentication Methods + +### 1. Configuration File (Highest Priority) + +Create a JSON configuration file and use the `-config` option: + +```json +{ + "identities": [ + { + "name": "admin_user", + "credentials": [ + { + "accessKey": "admin_access_key", + "secretKey": "admin_secret_key" + } + ], + "actions": ["Admin", "Read", "Write"] + }, + { + "name": "read_only_user", + "credentials": [ + { + "accessKey": "readonly_access_key", + "secretKey": "readonly_secret_key" + } + ], + "actions": ["Read"] + } + ] +} +``` + +Start S3 server with config file: +```bash +weed s3 -config=/path/to/s3.json -filer=localhost:8888 +``` + +### 2. Filer Configuration (Medium Priority) + +Store configuration in the filer using the credential manager. This allows dynamic configuration updates without restarting the S3 server. + +### 3. Environment Variables (Fallback) + +Use AWS standard environment variables as a fallback when no other configuration is available: + +```bash +export AWS_ACCESS_KEY_ID=your_access_key +export AWS_SECRET_ACCESS_KEY=your_secret_key +weed s3 -filer=localhost:8888 +``` + +**Important**: Environment variables are only used when: +- No `-config` option is provided +- No configuration is available from the filer +- Both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are set + +## Priority System + +SeaweedFS uses the following priority order for S3 credentials: + +1. **Configuration File** (if `-config` option is provided) +2. **Filer Configuration** (if available and no config file) +3. **Environment Variables** (fallback only) + +Higher priority methods completely override lower priority methods - there is no merging or supplementing. + +## Configuration Examples + +### Production Setup +```bash +# Use dedicated configuration file +weed s3 -config=/etc/seaweedfs/s3.json -filer=filer1:8888,filer2:8888 +``` + +### Development Setup +```bash +# Use environment variables for quick setup +export AWS_ACCESS_KEY_ID=dev_access_key +export AWS_SECRET_ACCESS_KEY=dev_secret_key +weed s3 -filer=localhost:8888 +``` + +### Docker Compose +```yaml +version: '3.9' +services: + s3: + image: chrislusf/seaweedfs:latest + ports: + - 8333:8333 + environment: + AWS_ACCESS_KEY_ID: s3admin + AWS_SECRET_ACCESS_KEY: s3secret + entrypoint: weed + command: s3 -filer=filer:8888 + depends_on: + - filer +``` + +## Credential Features + +### Actions +Identities can have different permission levels: +- `Admin`: Full access to all S3 operations +- `Read`: Read-only access +- `Write`: Read and write access +- `Read_ACP`: Read access control permissions +- `Write_ACP`: Write access control permissions + +### Multiple Credentials +Each identity can have multiple access key/secret key pairs: + +```json +{ + "name": "multi_key_user", + "credentials": [ + { + "accessKey": "key1", + "secretKey": "secret1" + }, + { + "accessKey": "key2", + "secretKey": "secret2" + } + ], + "actions": ["Read", "Write"] +} +``` + +### Account Management +Identities can be associated with accounts for better organization and cross-account access control. + +## Anonymous Access + +By default, if no credentials are configured, SeaweedFS allows anonymous access to all S3 operations. To enable authentication: + +1. Configure at least one identity using any of the methods above +2. Authentication will be automatically enabled +3. All requests will require valid credentials + +## Configuration Reloading + +SeaweedFS supports different reloading mechanisms depending on which authentication method you use: + +| Configuration Method | Auto Reload | Manual Reload | Live Reload | +|---------------------|-------------|---------------|-------------| +| **Configuration File** (`-config` option) | ❌ No | ✅ SIGHUP | ❌ No | +| **Filer Configuration** (credential manager) | ✅ Yes | ✅ Yes | ✅ Yes | +| **Environment Variables** | ❌ No | ❌ No | ❌ No | + +### Static Configuration Files + +When using the `-config` option, you can reload the configuration by sending a SIGHUP signal: + +```bash +# Find the S3 server process ID +ps aux | grep "weed s3" + +# Send SIGHUP signal to reload configuration +kill -HUP + +# Or if using systemd +systemctl reload seaweedfs-s3 +``` + +The server will log the reload: +``` +I0723 12:34:56.789 s3api_server.go:98] Loaded 3 identities from config file /etc/seaweedfs/s3.json +``` + +### Filer-based Configuration + +Filer-based configurations automatically reload when changes are detected: + +```bash +# Changes are automatically applied +weed shell +> s3.configure -user=newuser -access_key=key123 -secret_key=secret123 -actions=Admin -apply +``` + +The server will automatically detect and apply changes: +``` +I0723 12:35:12.456 auth_credentials_subscribe.go:55] updated /etc/seaweedfs/iam/identity.json +``` + +### Environment Variables + +Environment variable changes require a complete restart of the S3 server: + +```bash +# Update environment variables +export AWS_ACCESS_KEY_ID=new_access_key +export AWS_SECRET_ACCESS_KEY=new_secret_key + +# Restart the S3 server +systemctl restart seaweedfs-s3 +``` + +### Verifying Configuration Reloads + +Monitor the logs to verify configuration updates: + +```bash +# Watch for reload messages +tail -f /var/log/seaweedfs/s3.log | grep -E "updated|Loaded.*identities" + +# Check current configuration via shell +weed shell +> s3.configure +``` + +## Troubleshooting + +### Common Issues + +**Environment variables not working:** +- Check that no `-config` option is provided +- Verify no configuration exists in the filer +- Ensure both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are set + +**Configuration file not loading:** +- Verify the file path is correct +- Check JSON syntax is valid +- Ensure the file is readable by the SeaweedFS process + +**Invalid credentials error:** +- Verify access key and secret key are correct +- Check that the identity has the required actions/permissions +- Ensure the credential store is properly configured + +### Debug Commands + +Check current configuration: +```bash +# View current identities (if using filer store) +weed shell +> s3.configure -list +``` + +Test credentials: +```bash +# Test with AWS CLI +aws --endpoint-url=http://localhost:8333 s3 ls +``` + +## Security Best Practices + +1. **Use Configuration Files in Production**: Environment variables are visible in process lists +2. **Rotate Credentials Regularly**: Update access keys and secret keys periodically +3. **Principle of Least Privilege**: Grant only the minimum required permissions +4. **Secure Storage**: Store configuration files with appropriate file permissions +5. **Monitor Access**: Enable audit logging to track S3 API usage \ No newline at end of file diff --git a/_Sidebar.md b/_Sidebar.md index f5382b8..e3f307d 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -66,6 +66,7 @@ * [[Gateway to Remote Object Storage]] ### AWS S3 API +* [[S3 Credentials]] * [[Amazon S3 API]] * [[S3 Object Lock and Retention]] * [[AWS CLI with SeaweedFS]]