Github-Mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-20 13:30:46 +02:00
Code Issues Packages Projects Releases Wiki Activity

Resolve merge conflicts: update Java client to 4.00 and remove hdfs2 references

Chris Lu
2025-11-26 18:27:21 -08:00
parent 3969eeaa76 7fb862b04b
commit fae68740b7
69 changed files with 6030 additions and 284 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AWS-CLI-with-SeaweedFS.md
+44 -1
@@ -19,6 +19,23 @@ $ aws configure set default.s3.signature_version s3v4
```
### Reverse proxy sub-path configuration
This is undefined behavior as AWS S3 servers always have sub-domains instead of sub-paths. \
Use this only if you can't create (sub-)domain and use other port!
AWS CLI appends sub-path before actual path so need to add `X-Forwarded-Prefix` header (set to `/s3` for example)
Example for Caddy web server
```
redir /s3 /s3/
handle_path /s3/* {
reverse_proxy localhost:8333 {
header_up X-Forwarded-Prefix /s3
}
}
```
### Execute commands
```
# list buckets
@@ -64,5 +81,31 @@ http://localhost:8333/newbucket/t.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Cre
# access the url
$ curl "http://localhost:8333/newbucket/t.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=some_access_key1%2F20200726%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200726T161749Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=e0cc153209e414ca8168661f57827aa03ab84e7041ef9270ff639bcc519d24f5"
```
## Server-Side Encryption with AWS CLI
### SSE-KMS
```bash
aws --endpoint-url http://localhost:8333 s3 cp file.txt s3://bucket/kms.txt \
--sse aws:kms \
--sse-kms-key-id "test-key-123"
```
### SSE-C
```bash
# Generate a 256-bit key
openssl rand -base64 32 > key.b64
aws --endpoint-url http://localhost:8333 s3 cp file.txt s3://bucket/ssec.txt \
--sse-c AES256 \
--sse-c-key fileb://key.b64
```
### SSE-S3 (Server-managed)
```bash
aws --endpoint-url http://localhost:8333 s3 cp file.txt s3://bucket/sse-s3.txt \
--sse AES256
```
## OIDC/JWT to S3
For Keycloak and other OIDC providers, you can obtain a JWT and access S3 directly (or use STS to assume a role). See [[Keycloak Integration]].
AWS-IAM-CLI.md
+7 -1
@@ -16,6 +16,13 @@ s3.configure -apply -user admin -access_key some_access_key1 -secret_key some_se
## Create S3 credentials
Make sure you are using the admin:
```bash
export AWS_ACCESS_KEY_ID=some_access_key1
export AWS_SECRET_ACCESS_KEY=some_secret_key1
```
Create user and access key
```
aws --endpoint http://127.0.0.1:8111 iam create-access-key --user-name Bob
@@ -42,7 +49,6 @@ echo '
"s3:List*"
],
"Resource": [
"arn:aws:s3:::EXAMPLE-BUCKET",
"arn:aws:s3:::EXAMPLE-BUCKET/*"
]
}
Actual-Users.md
+2
@@ -13,6 +13,8 @@ Add your name here if your company is using SeaweedFS.
* [techbay](http://www.qualebs.com/techbay/)
* [Imagineapp](http://www.imagineapp.co/)
* ![SANMS](https://i.imgur.com/JNyx3CC_d.jpg?maxwidth=640&shape=thumb&fidelity=high)
* 京东登月平台 (http://geek.csdn.net/news/detail/228285)
Admin-UI.md
+481
@@ -0,0 +1,481 @@
# This is still work in progress. Some features work, some not. Everything is subject to change.
# Weed Admin
The `weed admin` command starts a modern web-based administration interface for SeaweedFS cluster management.
## Overview
The admin interface provides a comprehensive web UI for managing SeaweedFS clusters, including:
- **Cluster topology visualization and monitoring**
- **Volume management and operations**
- **File browser and management**
- **System metrics and performance monitoring**
- **Configuration management**
The admin interface automatically discovers filers from the master servers and runs a gRPC server for worker connections on HTTP port + 10000.
## Usage
```bash
weed admin [options]
```
## Options
| Option | Default | Description |
|--------|---------|-------------|
| `-port` | 23646 | Admin server port |
| `-masters` | localhost:9333 | Comma-separated master servers |
| `-dataDir` | "" | Directory to store admin configuration and data files |
| `-adminUser` | admin | Admin interface username |
| `-adminPassword` | "" | Admin interface password (if empty, auth is disabled) |
## Examples
### Basic Usage
```bash
# Start admin interface on default port (23646)
weed admin -masters=localhost:9333
# Start with custom port and multiple masters
weed admin -port=8080 -masters="master1:9333,master2:9333"
# Start with specific data directory
weed admin -port=23646 -masters="localhost:9333" -dataDir="/var/lib/seaweedfs-admin"
# Start with home directory expansion
weed admin -port=23646 -masters="localhost:9333" -dataDir="~/seaweedfs-admin"
```
### With Authentication
```bash
# Enable authentication
weed admin -adminUser=admin -adminPassword=secret123 -masters="localhost:9333"
```
### Production Deployment
```bash
# Production setup with data persistence and authentication
weed admin \
-port=23646 \
-masters="master1:9333,master2:9333,master3:9333" \
-dataDir="/var/lib/seaweedfs-admin" \
-adminUser=admin \
-adminPassword=strongpassword123
```
## Data Directory
The data directory (`-dataDir`) is used to persist admin configuration data:
- **If specified**: Configuration and data are persisted to disk
- **If not specified**: All configuration is kept in memory only
- **Path expansion**: Supports tilde (`~`) expansion for home directory
- **Auto-creation**: Directory is automatically created if it doesn't exist
- **Format**: Configuration files are stored in JSON format for easy editing
## Security and Authentication
### Authentication
- **Disabled by default**: If `-adminPassword` is not set, no authentication is required
- **Session-based**: When enabled, uses secure session management with auto-generated session keys
- **User credentials**: Login with `-adminUser` and `-adminPassword`
### TLS/HTTPS Configuration
The admin server reads TLS configuration from `security.toml`:
```toml
[https.admin]
cert = "/etc/ssl/admin.crt"
key = "/etc/ssl/admin.key"
ca = "/etc/ssl/ca.crt" # optional, for mutual TLS
```
- **HTTPS**: Automatically enabled if `https.admin.key` is configured
- **Mutual TLS**: Enabled if `https.admin.ca` is configured
- **Certificate loading**: Certificates are loaded from the security configuration
### Security Best Practices
1. **Set strong passwords**: Use strong `-adminPassword` for production
2. **Configure TLS**: Use HTTPS for production deployments
3. **Firewall rules**: Restrict admin interface access to authorized networks
4. **Regular updates**: Keep SeaweedFS updated for security patches
## Worker Communication
The admin server also runs a gRPC server for worker connections:
- **Port**: HTTP port + 10000 (e.g., if admin runs on 23646, gRPC runs on 33646)
- **Purpose**: Handles worker connections and task distribution
- **TLS**: Uses `[grpc.admin]` configuration from `security.toml`
- **Fallback**: Workers fall back to insecure connections if TLS is unavailable
## Configuration File
The admin server reads configuration from `security.toml` in the following order:
1. Current directory (`.`)
2. `$HOME/.seaweedfs/`
3. `/usr/local/etc/seaweedfs/`
4. `/etc/seaweedfs/`
### Generate Example Configuration
```bash
# Generate example security.toml
weed scaffold -config=security
```
## Features
### Automatic Service Discovery
- **Master discovery**: Connects to specified master servers
- **Filer discovery**: Automatically discovers filers from masters
- **Health monitoring**: Monitors cluster health and status
### Web Interface
The admin interface provides a comprehensive web-based management console with the following sections:
#### Dashboard
- **Cluster Overview**: Real-time cluster status and health metrics
- **System Statistics**: Total volumes, files, size, and volume size limits
- **Node Status**: Master, filer, volume server, and message broker status
- **Data Centers**: Geographic distribution of storage nodes
#### Object Store Management
- **S3 Buckets**: View, create, delete, and manage S3-compatible buckets
- **Bucket Details**: Quota management and configuration
- **User Management**: Create and manage S3 API users with permissions
- **Access Keys**: Generate and manage access/secret key pairs
- **Policies**: Manage bucket policies and user permissions
#### File Browser
- **Directory Navigation**: Browse filesystem hierarchy through web interface
- **File Operations**: Upload, download, delete, and manage files
- **File Properties**: View file metadata, permissions, and storage details
- **Bulk Operations**: Multi-select for batch operations
#### Cluster Management
- **Master Servers**: View master node status, leadership, and connectivity
- **Filer Servers**: Monitor filer instances and metadata operations
- **Volume Servers**: Track storage nodes, capacity, and health status
- **Volume Management**: View volume distribution, replication, and status
- **Collections**: Monitor data collections and their volume allocation
#### Message Queue Management
- **Brokers**: View message queue broker status and configuration
- **Topics**: Manage topics, partitions, and message retention
- **Subscribers**: Monitor subscriber connections and consumer lag
- **Topic Details**: View message statistics and partition distribution
### API Endpoints
The admin interface provides RESTful API endpoints for:
- Cluster status and topology
- Volume management
- File operations
- System metrics
- Configuration management
## Troubleshooting
### Common Issues
1. **No filers discovered**:
- Check master server connectivity
- Verify master addresses are correct
- Ensure masters are running and accessible
2. **Authentication not working**:
- Verify `-adminPassword` is set correctly
- Check session cookie settings
- Clear browser cache and cookies
3. **TLS/HTTPS issues**:
- Verify certificate paths in `security.toml`
- Check certificate validity and permissions
- Ensure certificates are in PEM format
4. **Worker connections failing**:
- Check if gRPC port (HTTP port + 10000) is accessible
- Verify TLS configuration for worker connections
- Check firewall rules for gRPC port
### Debug Information
Enable debug logging for detailed troubleshooting:
```bash
# Run with verbose logging
weed admin -v=4 -masters="localhost:9333"
```
## Related Commands
- [`weed master`](https://github.com/seaweedfs/seaweedfs/wiki/Master-Server): Start master servers
- [`weed filer`](https://github.com/seaweedfs/seaweedfs/wiki/Filer-Server): Start filer servers
- [`weed scaffold`](https://github.com/seaweedfs/seaweedfs/wiki/Scaffold): Generate configuration files
## Admin UI Pages Reference
### Accessing the Admin Interface
Once the admin server is running, access the web interface at:
```
http://localhost:23646
```
Or with custom port:
```
http://localhost:PORT
```
### Dashboard (/)
The main dashboard provides a comprehensive overview of your SeaweedFS cluster:
**Key Metrics**:
- Total volumes, files, and storage size
- Volume size limit configuration
- Cluster health status
**Cluster Topology**:
- Master server status and leader election
- Filer server instances and connections
- Volume server distribution and capacity
- Message broker status (if enabled)
- Data center geographic distribution
**Real-time Updates**: The dashboard automatically refreshes to show current cluster status.
### Object Store Management
#### S3 Buckets (/object-store/buckets)
Manage S3-compatible storage buckets:
**Features**:
- View all buckets with creation dates and sizes
- Create new buckets with Object Lock support
- Delete buckets (with confirmation)
- Set bucket quotas and limits
- Export bucket list to CSV
**Bucket Operations**:
- **Create Bucket**: Specify name and optional Object Lock enablement
- **Delete Bucket**: Remove empty buckets with confirmation dialog
- **Quota Management**: Set storage limits per bucket
- **Bucket Details**: View detailed statistics and configuration
#### User Management (/object-store/users)
Manage S3 API users and their permissions:
**User Operations**:
- Create new users with email and permissions
- Edit existing user permissions and details
- Delete users with confirmation
- Generate and manage access keys
- View user activity and permissions
**Permission Types**:
- **Admin**: Full access to all buckets and operations
- **Read**: Read access to specified buckets
- **Write**: Write access to specified buckets
- **List**: List bucket contents
- **Tagging**: Manage object tags
- **Object Lock Permissions**:
- **BypassGovernanceRetention**: Override governance retention
- **GetObjectRetention**: Read object retention settings
- **PutObjectRetention**: Modify object retention
- **GetObjectLegalHold**: Read legal hold status
- **PutObjectLegalHold**: Set legal hold
- **GetBucketObjectLockConfiguration**: Read bucket Object Lock config
- **PutBucketObjectLockConfiguration**: Modify bucket Object Lock config
**Access Key Management**:
- Generate new access/secret key pairs
- View existing access keys (secret keys are masked)
- Delete unused access keys
- Copy keys to clipboard
#### Policies (/object-store/policies)
Manage bucket policies and access control:
**Policy Operations**:
- Create JSON-based bucket policies
- Edit existing policies with syntax validation
- Delete policies
- Validate policy syntax before saving
- View policy effects and permissions
### File Browser (/files)
Web-based file system interface:
**Navigation**:
- Browse directory hierarchy
- Navigate with breadcrumb navigation
- Search files and folders
- Sort by name, size, or modification date
**File Operations**:
- **Upload**: Single or multiple file upload with progress tracking
- **Download**: Direct file download or streaming
- **Delete**: Remove files and folders with confirmation
- **Create Folders**: New directory creation
- **Rename**: File and folder renaming
- **Copy/Move**: File management operations
**Advanced Features**:
- Drag-and-drop file upload
- Bulk selection for batch operations
- File property viewing (size, permissions, metadata)
- Preview for supported file types
- Export file listings to CSV
### Cluster Management
#### Master Servers (/cluster/masters)
Monitor master server cluster:
**Information Displayed**:
- Master server addresses and ports
- Leader election status
- Connection health and response times
- Configuration synchronization status
- Cluster membership changes
#### Filer Servers (/cluster/filers)
Track filer instances:
**Monitoring**:
- Filer server addresses and health
- Metadata store backend status
- Connected clients and operations
- Performance metrics and response times
#### Volume Servers (/cluster/volume-servers)
Manage storage nodes:
**Server Information**:
- Server addresses and capacity
- Free space and utilization
- Active volume counts
- Data center and rack assignment
- Health status and connectivity
#### Volume Management (/cluster/volumes)
Detailed volume tracking:
**Volume Details**:
- Volume ID and size information
- Replication status and factor
- Read/write statistics
- Storage location and server mapping
- Collection assignment
**Operations**:
- View volume distribution across servers
- Monitor replication health
- Track volume growth and utilization
#### Collections (/cluster/collections)
Monitor data collections:
**Collection Information**:
- Collection names and volume counts
- Replication configuration
- Storage distribution
- Growth patterns and capacity planning
### Message Queue Management
#### Brokers (/mq/brokers)
Monitor message queue brokers:
**Broker Status**:
- Broker addresses and health
- Topic assignment and leadership
- Connection counts and throughput
- Configuration and settings
#### Topics (/mq/topics)
Manage message queue topics:
**Topic Operations**:
- Create new topics with partition configuration
- View topic statistics and message counts
- Manage topic retention policies
- Monitor consumer lag and throughput
#### Topic Details (/mq/topics/{namespace}/{topic})
Detailed topic information:
**Statistics**:
- Message production and consumption rates
- Partition distribution and leadership
- Subscriber connections and lag
- Storage utilization per partition
**Configuration Options**:
- Task scheduling parameters
- Worker connection settings
- Retry policies and timeouts
- Resource allocation limits
## Navigation and UI Features
### Responsive Design
- Mobile-friendly interface
- Collapsible sidebar navigation
- Responsive tables and charts
- Touch-friendly controls
### Real-time Updates
- Live cluster status monitoring
- Automatic page refresh for dynamic content
- WebSocket connections for real-time data
- Progress indicators for long-running operations
### Security Features
- Session-based authentication
- CSRF protection
- Secure cookie handling
- TLS/HTTPS support
### Accessibility
- Keyboard navigation support
- Screen reader compatibility
- High contrast mode support
- Semantic HTML structure
## See Also
- [SeaweedFS Architecture](https://github.com/seaweedfs/seaweedfs/wiki/SeaweedFS-Architecture)
- [Security Configuration](https://github.com/seaweedfs/seaweedfs/wiki/Security-Configuration)
- [S3 Object Lock and Retention](S3-Object-Lock-and-Retention.md)
- [S3 API FAQ](S3-API-FAQ.md)
Amazon-S3-API.md
+158 -6
@@ -4,7 +4,7 @@ To be compatible with Amazon S3 API, a separate "weed s3" command is provided. T
`weed s3` will start a stateless gateway server to bridge the Amazon S3 API to SeaweedFS Filer.
For convenience, `weed server -s3` will start a master, a volume server, a filer, and the S3 gateway. And `weed filer -s3` can start a filer and the S3 gateway together also.
Each bucket is stored in one collection, and mapped to folder /buckets/<bucket_name> by default.
Each bucket is stored in one collection, and mapped to folder `/buckets/<bucket_name>` by default.
A bucket can be deleted efficiently by deleting the whole collection.
@@ -32,7 +32,11 @@ See https://github.com/seaweedfs/seaweedfs/wiki/Path-Specific-Configuration
# Supported APIs
Currently, the following APIs are supported.
Currently, the following APIs are supported.
Some additional endpoints might be (partially), supported but are not in this list.
To be sure, you can look at the function defined in the files `weed/s3api/s3api_*_handlers_*.go`.
```
// Object operations
@@ -51,11 +55,32 @@ Currently, the following APIs are supported.
* PutObjectTagging
* DeleteObjectTagging
// Server-Side Encryption (NEW)
* PutObject (with SSE-KMS, SSE-C, SSE-S3)
* GetObject (with automatic decryption)
* HeadObject (with encryption metadata)
* CopyObject (with encryption/decryption)
* Multipart uploads with encryption
* Bucket default encryption
// Conditional Operations (NEW)
* All object operations support conditional headers:
- If-Match
- If-None-Match
- If-Modified-Since
- If-Unmodified-Since
// Bucket operations
* PutBucket
* DeleteBucket
* HeadBucket
* ListBuckets
* PutBucketLifecycleConfiguration (partially, only for TTL)
* GetBucketLifecycleConfiguration (partially, only for TTL)
* DeleteBucketLifecycleConfiguration (partially, only for TTL)
* GetBucketCors
* PutBucketCors
* DeleteBucketCors
// Multipart upload operations
* NewMultipartUpload
@@ -65,6 +90,25 @@ Currently, the following APIs are supported.
* PutObjectPart
* CopyObjectPart
* ListObjectParts
// Object Versioning
* PutBucketVersioning
* GetBucketVersioning
* ListObjectVersions
* GetObject (with version ID)
* PutObject (with versioning)
* DeleteObject (with version ID)
* CopyObject (with version ID)
* RestoreObject (partial)
// Object Lock and Retention
* GetObjectLockConfiguration
* PutObjectLockConfiguration
* GetObjectRetention
* PutObjectRetention
* GetObjectLegalHold
* PutObjectLegalHold
* BypassGovernanceRetention (via x-amz-bypass-governance-retention header)
```
Not included:
@@ -78,22 +122,96 @@ Not included:
| DeleteObject deletes a folder | Yes | No |
| same path for both a file and a folder | No | Yes |
| allows more than "/" as a delimiter | No | Yes |
| Object Versioning | Yes | Yes |
| MFA Delete for versioning | No | Yes |
| Server-Side Encryption (SSE-KMS) | Yes | Yes |
| Server-Side Encryption (SSE-C) | Yes | Yes |
| Server-Side Encryption (SSE-S3) | Yes | Yes |
| KMS Providers (Multi-cloud) | Yes | No |
| Conditional Headers (All operations) | Yes | Yes |
| Range requests with SSE-KMS | Yes | Yes |
| Range requests with SSE-C | Yes | Yes |
| Range requests with SSE-S3 | Yes | Yes |
## Empty folders
SeaweedFS has directories while AWS S3 only has objects with "fake" directories. In AWS S3, if the last file is deleted in a directory, the directory will disappear also.
To be consistent with AWS S3, SeaweedFS tries to skip empty folders while listing. You can use `weed s3 -allowEmptyFolder` to toggle this behavior.
To be consistent with AWS S3, SeaweedFS tries to check whether the folder is empty after each deletion. You can use `weed s3 -allowEmptyFolder` to toggle this behavior.
This is not so ideal. Another approach is to list current directory when deleting a file, which will slow down quite a bit especially when deleting multiple files. SeaweedFS did not take this approach.
# Server-Side Encryption
The last approach, which is most efficient, is to maintain counters for each folder, and drop the folder as soon as it becomes empty. This is implemented in [[Cloud Monitoring]].
Need encryption at rest? SeaweedFS speaks the same SSE dialects as Amazon S3, so your existing tools and SDKs just work. You can choose from three options:
- **[SSE-KMS](Server-Side-Encryption-SSE-KMS)**: Use an external KMS (AWS KMS, Google Cloud KMS, OpenBao/Vault)
- **[SSE-C](Server-Side-Encryption-SSE-C)**: Bring your own keys for maximum control
- **SSE-S3**: Let SeaweedFS manage keys (explicit `AES256` header or bucket default encryption)
All encryption types support:
- Automatic encryption/decryption
- Bucket default encryption
- Multipart upload encryption
- Cross-encryption copy operations
- AWS S3 compatibility
For detailed setup guides and examples, see:
- **[Server-Side Encryption Overview](Server-Side-Encryption)**
- **[SSE-KMS Guide](Server-Side-Encryption-SSE-KMS)**
- **[SSE-C Guide](Server-Side-Encryption-SSE-C)**
## Quick Examples
```bash
# SSE-KMS (Key Management Service)
aws s3 cp file.txt s3://mybucket/kms-encrypted.txt --server-side-encryption aws:kms --ssekms-key-id alias/my-key
# SSE-C (Customer-provided keys)
aws s3 cp file.txt s3://mybucket/customer-encrypted.txt --sse-c AES256 --sse-c-key fileb://my-key.bin
# SSE-S3 (Server-managed)
aws s3 cp file.txt s3://mybucket/server-encrypted.txt --server-side-encryption AES256
```
# S3 Conditional Operations
SeaweedFS supports AWS S3-compatible conditional headers for safe concurrent operations and efficient caching:
- **If-Match**: Execute only if ETag matches (optimistic locking)
- **If-None-Match**: Execute only if ETag doesn't match (prevent overwrites, caching)
- **If-Modified-Since**: Execute only if modified after date (conditional downloads)
- **If-Unmodified-Since**: Execute only if not modified after date (safe updates)
Conditional operations enable:
- **Optimistic concurrency control**: Prevent lost updates
- **Efficient caching**: Reduce bandwidth with 304 Not Modified
- **Atomic operations**: Operations only proceed when safe
- **Data integrity**: Prevent accidental overwrites
For detailed usage patterns and examples, see **[S3 Conditional Operations](S3-Conditional-Operations)**.
## Quick Examples
```bash
# Get current ETag
ETAG=$(aws s3api head-object --bucket mybucket --key file.txt --query ETag --output text)
# Conditional update (optimistic locking)
curl -X PUT -H "If-Match: $ETAG" -d "updated content" "http://localhost:8333/mybucket/file.txt"
# Conditional download (caching)
curl -H "If-None-Match: $ETAG" "http://localhost:8333/mybucket/file.txt"
# Returns 304 Not Modified if unchanged
# Prevent overwrite (atomic create)
curl -X PUT -H "If-None-Match: *" -d "new content" "http://localhost:8333/mybucket/newfile.txt"
```
# S3 Authentication
By default, the access key and secret key to access `weed s3` is not authenticated. To enable credential based access, you can choose static or dynamic configuration:
* **Dynamic Configuration**: setup auth with `s3.configure` in `weed shell`
* **Static Configuration**: create a config.json file similar to the example below, and specify it via `weed s3 -config=config.json`
* **OIDC/JWT (Web Identity)**: for Keycloak and other OpenID providers, see [[Keycloak Integration]] for STS and JWT to S3 usage
## Dynamic Configuration
@@ -128,7 +246,7 @@ Output from above `s3.configure` command:
## Static Configuration
To enable credential based access, create a config.json file similar to the example below, and specify it via `weed s3 -config=config.json`. The config file can be re-read on the HUB signal without restarting the main process via `pkill -HUP weed`
To enable credential based access, create a config.json file similar to the example below, and specify it via `weed s3 -config=config.json`. The config file can be re-read on the HUP signal without restarting the main process via `pkill -HUP weed`
You just need to create a user with all "Admin", "Read", "Write", "List", "Tagging" actions.
You can create as many users as needed. Each user can have multiple credentials.
@@ -242,6 +360,40 @@ Read:bucket/a/b/*
Presigned URL is supported. See [[AWS-CLI-with-SeaweedFS#presigned-url]] for example.
# S3 Object Versioning
SeaweedFS supports S3 object versioning, which allows you to keep multiple variants of an object in the same bucket. This provides data protection against accidental deletion or modification.
For detailed information about object versioning, see the dedicated [[S3-Object-Versioning]] page.
# S3 Cross-Origin Resource Sharing (CORS)
SeaweedFS supports S3-compatible Cross-Origin Resource Sharing (CORS) configuration, allowing web applications to make cross-origin requests to your S3 buckets. CORS is essential for web applications that need to access resources from different domains.
For detailed information about CORS configuration, see the dedicated [[S3-CORS]] page.
# Reverse Proxy Support
SeaweedFS S3 API supports deployment behind reverse proxies with full AWS Signature v4 authentication compatibility. This includes support for:
- **X-Forwarded-Host**: Preserves original host header for signature verification
- **X-Forwarded-Port**: Automatically combines with X-Forwarded-Host for non-standard ports
- **X-Forwarded-Prefix**: Handles URL path prefix stripping by reverse proxies
- **Standard forwarded headers**: X-Forwarded-For, X-Forwarded-Proto, etc.
## Path Prefix Handling
When using reverse proxies that strip URL prefixes (e.g., `/s3/`, `/api/s3/`), SeaweedFS automatically handles signature verification for both the original prefixed path and the stripped path. This ensures seamless operation with:
- API gateways
- Multi-tenant deployments
- Subpath hosting scenarios
For detailed configuration examples and setup instructions, see the dedicated [[S3-Nginx-Proxy]] page.
## Multiple S3 Nodes
If you need to setup multiple S3 nodes, you can just start multiple s3 instances pointing to a filer.
Client-Libraries.md
+2
@@ -14,6 +14,7 @@ Name | Author | Language
[@trubavuong/seaweedfs](https://www.npmjs.com/package/@trubavuong/seaweedfs) | Vuong Tru | Javascript
[@trubavuong/fastify-seaweedfs](https://www.npmjs.com/package/@trubavuong/fastify-seaweedfs) | Vuong Tru | Javascript
[seaweedfs client plug-in for egg.js](https://github.com/aliwalker/egg-seaweed-client) | aliwalker | Javascript
[seaweedTS - Typescript library for seaweedfs](https://www.npmjs.com/package/seaweedts) | Nathan Nesbitt | Typescript
[Weedo](https://github.com/ginuerzh/weedo) | Ginuerzh | Go
[goseaweed](https://github.com/tnextday/goseaweed) | tnextday | Go
[goseaweedfs](https://github.com/linxGnu/goseaweedfs) | linxGnu | Go
@@ -25,6 +26,7 @@ Name | Author | Language
[Python-weed](https://github.com/darkdarkfruit/python-weed) | Darkdarkfruit | Python
[Django-weed](https://github.com/ProstoKSI/django-weed) | ProstoKSI | Python
[Pyweed](https://github.com/utek/pyweed) | Utek | Python
[Aioseaweedfs](https://code.pobblelabs.org/fossil/aioseaweed) | davestgermain | Python
[Scala SeaweedFS client](https://github.com/chiradip/WeedFsScalaClient) | Chiradip | Scala
[Seaweedrb](https://github.com/jguest/seaweedrb) | John Guest | Ruby
[SeaweedFs.NET](https://github.com/piechpatrick/SeaweedFs.Client) | piechpatrick | C#
Cloud-Drive-Benefits.md
+12 -12
@@ -1,17 +1,17 @@
# Introduction
> NOTE: SeaweedFS provides **two mechanisms** to use cloud storage:
> 1) **SeaweedFS Cloud Drive** (**<== You are here**)
> - in this case, you can **mount** an S3 bucket to the Seaweedfs file system (in the filer), and access the remote files
> through SeaweedFS. Effectively, SeaweedFS caches the files from the cloud storage.
> - In this mode, the file structure in cloud store is exactly matching the SeaweedFS structure - so every
> file in SeaweedFS will also become a file in the cloud storage provider.
> - This is useful in case you want to use the files inside the cloud provider's infrastructure.
> - However, this does **not support file encryption** in any way (obviously), as the files are put to Cloud Storage as is.
> 2) [Tiered Storage with Cloud Tier](https://github.com/seaweedfs/seaweedfs/wiki/Cloud-Tier)
> - In this mode, seaweedFS **moves full volume files to the cloud storage provider**, so files which are 1 GB (in our case) big.
> - This mode supports [Filer Data Encryption](https://github.com/seaweedfs/seaweedfs/wiki/Filer-Data-Encryption) transparently.
> - The chunk files uploaded to the cloud provider are not usable outside of SeaweedFS.
SeaweedFS provides **two mechanisms** to use cloud storage:
1) **SeaweedFS Cloud Drive** (**<== You are here**)
- in this case, you can **mount** an S3 bucket to the Seaweedfs file system (in the filer), and access the remote files
through SeaweedFS. Effectively, SeaweedFS caches the files from the cloud storage.
- In this mode, the file structure in cloud store is exactly matching the SeaweedFS structure - so every
file in SeaweedFS will also become a file in the cloud storage provider.
- This is useful in case you want to use the files inside the cloud provider's infrastructure.
- However, this does **not support file encryption** in any way (obviously), as the files are put to Cloud Storage as is.
2) [Tiered Storage with Cloud Tier](https://github.com/seaweedfs/seaweedfs/wiki/Cloud-Tier)
- In this mode, seaweedFS **moves full volume files to the cloud storage provider**, so files which are 1 GB (in our case) big.
- This mode supports [Filer Data Encryption](https://github.com/seaweedfs/seaweedfs/wiki/Filer-Data-Encryption) transparently.
- The chunk files uploaded to the cloud provider are not usable outside of SeaweedFS.
## Cloud is not for everyone
Cloud-Drive-Quick-Setup.md
+1 -1
@@ -101,7 +101,7 @@ These cache or uncache jobs can vary wildly. Here are some examples:
# uncache file size older than 3600 seconds
> remote.uncache -dir=/buckets/bucket1 -maxAge=3600
# uncache file size more than 10240 bytes
> remote.cache -dir=/buckets/bucket1 -minSize=10240
> remote.uncache -dir=/buckets/bucket1 -minSize=10240
```
Cloud-Monitoring.md
+2 -70
@@ -1,73 +1,5 @@
Seaweed Cloud Monitoring is a service provided by seaweedfs.com. It is an attempt to help SeaweedFS to grow organically,
Seaweed Cloud Monitoring was a service provided by seaweedfs.com to attempt to help SeaweedFS to grow organically,
* Iterate faster without upgrading the whole cluster.
* Help SeaweedFS admin to manage the clusters via internet.
# Architecture
```
Browser => Seaweed Cloud <==> Seaweed Agent <==> SeaweedFS cluster
```
The Seaweed Agent is a simple docker image `chrislusf/seaweed_agent`. Just need to run the docker image, setting the filer address and a password. To upgrade, just pull the latest image and restart.
The docker image should be able to run for recent SeaweedFS clusters since 2.58 or so. There are no specific requirement for the SeaweedFS cluster.
It subscribes to metadata changes from a SeaweedFS cluster, and build up statistics. When it starts, it may take some time to process existing metadata logs, if there are a lot of files or a lot of updates.
When accessing the monitoring URL, the Seaweed Cloud would proxy the requests to the agent, get the statistics, and render to the web page. There are no data persisted on the Seaweed Cloud. (It is running on a free-tier web server.)
## Features
Currently it has very basic features and all in beta.
Existing features:
* Near real time directory statistics on disk usage, file count and directory count.
* Set quota for any directory.
* Automatically delete empty folders in buckets (under `/buckets`), to be more compatible with AWS S3 object store.
![](CloudMonitoring.png)
# Start Seaweed Agent
To try it out, run these with docker. It should print out a URL which you can visit.
```
docker run --pull always -ti chrislusf/seaweed_agent swagent -filer 192.168.2.11:8888 -password=your_password
# or set the password via environment variables
docker run --pull always -e SEAWEED_PASSWORD=your_password -ti chrislusf/seaweed_agent swagent -filer 192.168.2.11:8888
```
The Seaweed Agent is still constantly evolving. So better always use the latest version.
# Get Cloud Monitoring URL
The container instance should have similar output, which you can find the Cloud Monitoring URL:
```
docker run --pull always -e SEAWEED_PASSWORD=abc -ti chrislusf/seaweed_agent swagent -filer 192.168.2.10:8888
2021/09/14 05:35:41 connecting to grpc.seaweedfs.com:4772
2021/09/14 05:35:41 read /etc/seaweedfs/filer.conf: ReadEntry: filer: no entry is found in filer store
2021/09/14 05:35:41 tracking 192.168.2.10:8888: 1970-01-01 00:00:00 +0000 UTC
2021/09/14 05:35:42 connected to grpc.seaweedfs.com:4772
---
Free Monitoring Data URL:
https://cloud.seaweedfs.com/ui/593HZ8CV9VF5976A8S0UABCDEFGHIJKL
---
Filer version: 30GB 2.67
```
## Find the URL through `weed shell`
Now when starting `weed shell`, it should also print out the Cloud Monitoring URL:
```
$ weed shell
master: localhost:9333 filer: localhost:8888
I0913 22:50:51 96301 masterclient.go:96] adminShell masterClient Connecting to master localhost:9333
---
Free Monitoring Data URL:
https://cloud.seaweedfs.com/ui/593HZ8CV9VF5976A8S0UABCDEFGHIJKL
---
>
```
**This service has now been deprecated and is no longer usable**
Cloud-Tier.md
+4
@@ -76,12 +76,16 @@ Multiple S3 buckets are supported. Usually you just need to configure one backen
aws_secret_access_key = "" # if empty, loads from the shared credentials file (~/.aws/credentials).
region = "us-west-2"
bucket = "one_bucket_two" # an existing bucket
force_path_style = "false" # The default value is true. If you are using Alibaba Cloud OSS (path style is no longer supported), please set it to false.
storage_class = "STANDARD_IA"
```
### Rclone
> NOTE: Since version 3.60 the rclone backend is removed
> in standard builds to reduce size for common use cases. Use the `_full` weed binary.
Here's an example of a configuration for an Rclone storage backend:
```
Components.md
+4 -10
@@ -1,12 +1,10 @@
# SeaweedFS Components
SeaweedFS comprises 3 main conceptual components. The master service and the volume service together provide a distributed object store, with user-configurable replication and redundancy. The optional filer and S3 service are additional layers on top of the object store. Each of these services may run as one or separate instances, on various actual servers.
SeaweedFS comprises 3 main components. The master service and the volume service together provide a distributed object store, with user-configurable replication and redundancy. The optional filer and S3 service are additional layers on top of the object store. Each of these services may run as one or separate processes, on one or more operating systems.
## Master service
The essential master service works smart, not hard.
It represents a cluster of 1 (or 3 or more servers) that own a consistent view of the entire SeaweedFS cluster and communicate it to all participating nodes, through a Leader elected via Raft protocol.
It represents a cluster of 1 (or 3 or n/2+1) that own a consistent view of the entire SeaweedFS cluster and communicate it to all participating nodes, through a leader elected via [Raft](https://raft.github.io/) protocol.
The number of servers in the master service must always be odd, to ensure that a majority consensus can be formed. You're best off keeping this number down, a small number of stable servers is better than a large pool of flakey boxes. 1 or 3 is typical.
@@ -18,17 +16,13 @@ If the leader is unavailable, the raft consensus protocol ensures that a new lea
## Volume service
The volume service is where the hard work is done.
It packs many objects (files and chunks of files) efficiently into larger individual volumes, which can be arbitrarily large blocks on disk. Redundancy and replication of data is managed at the volume level, not on a per-object level.
Each volume server sends periodic heartbeats with status and volume information back to the leader, via a master.
## Filer service
The optional filer service does heavy lifting so you don't have to.
It organizes SeaweedFS volumes and objects, into user-visible paths (like URLs or file systems) over HTTP or UNIX FUSE mounts.
It organizes SeaweedFS volumes and objects into user-visible paths (like URLs or file systems) over HTTP or UNIX FUSE mounts.
Filer provides a convenient and common abstraction that can be used to provide normal looking filesystems, or web APIs for down/uploads, to existing applications without modification.
@@ -38,7 +32,7 @@ This optional service provides AWS style S3 buckets, similar to the filer servic
## Volume Concept
The volume in SeaweedFS means a single actual file consists of many small files. When master starts, it configures the volume file size, default to 30GB. At the beginning, there are 8 volumes created.
In SeaweedFS, a volume is a single file consisting of many small files. When a master server starts, it sets the volume file maximum size to 30GB (see: `-volumeSizeLimitMB`). At volume server initialization, it will create 8 of these volumes (see: `-max`).
Each volume has its own TTL and replication.
Data-Backup.md
+18 -11
@@ -1,28 +1,30 @@
## A Simple Backup Strategy
The most important thing for storage is to keep data safe and not losing them. It gives you a comfort of thought that your data is safely backup.
The most important thing for storage is to keep data safe and not lose it. It gives you a comfort of thought that your data is safely backed up.
However, we do not want to always copy the whole data files over. We want to do it incrementally.
However, we do not always want to copy the all the data files over. We want to do it incrementally.
"weed backup" command is your friend.
`weed backup` command is your friend.
Run "weed backup" command on any machine that have enough disk spaces. Assuming we want to backup volume 5.
Run `weed backup` command on any machine that has enough disk space. Assuming we want to backup volume 5:
weed backup -server=master:port -dir=. -volumeId=5
```bash
weed backup -server=master:port -dir=. -volumeId=5
```
If local volume 5 does not exist, it will be created. All remote needle entries are fetched and compared to local needle entries. The delta is calculated and local missing files are fetched from the volume server.
If local volume 5 does not exist, it will be created. All remote needle entries are fetched and compared to local needle entries. The delta is calculated, and local missing files are fetched from the volume server.
If you specify `-volumeId=87`, but volume 87 does not exist, it's ok. No files will be created locally. This gives the opportunity that you can create a backup script simply looping from 1 to 100. All existing volumes will be backed up. The non-existing volumes can also be backed up when they are created remotely.
If you specify `-volumeId=87`, but volume 87 does not exist, it's ok. No files will be created locally. This gives the opportunity for you to create a backup script simply looping from 1 to 100. All existing volumes will be backed up. The non-existing volumes can also be backed up when they are created remotely.
The backup scripts is just one command, not a continuous running service though. High Availability servers will be added later.
The backup scripts are just one command, not a continuous running service though. High Availability servers will be added later.
## How to create a mirror of a cluster
### To Start
- Pause operations on cluster you want to backup. This is to avoid a mismatch in the two data sets(the volumes and the Filer metadata) you will be moving separately and combining in the backup cluster.
- Pause operations on cluster you want to back up. This is to avoid a mismatch in the two data sets (the volumes and the Filer metadata) you will be moving separately and combining in the backup cluster.
- Install SeaweedFS on a new machine/cluster of machines. Use the same version of SeaweedFS if you can to avoid any issues that could otherwise come up.
- *Do not start your volume servers yet!!* This is to avoid SeaweedFS creating it's own volumes(we will be using the volumes backed up from the current operational cluster).
- *Do not start your volume servers yet!!* This is to avoid SeaweedFS creating it's own volumes (we will be using the volumes backed up from the current operational cluster).
### Prepare the New Cluster and Backup Your Data
@@ -32,18 +34,23 @@ The backup scripts is just one command, not a continuous running service though.
### Backup the Metadata
run `fs.meta.save` on the cluster you are pulling from and save the output. This can look like:
Run `fs.meta.save` on the cluster you are pulling from and save the output. This can look like:
```sh
# You will need permission to create a file in the destination directory
# I recommend changing the file name because the default naming convention is not very readable BUT it does show the date the file was created which can be good information to store and know.
fs.meta.save -o=[yourlocaldir]/[yourfilename].meta
```
then download the file on the Filer machine you are using in the backup cluster. This can look like:
```sh
# This tool requires that the remote machine be accessable via SSH and that you have the password
scp [hostmachineusername]@[hostmachineip]:/remote_directory/file /local/directory
```
run `fs.meta.load` in the backup cluster on the Filer:
```sh
fs.meta.load [filepath/filename.meta]
```
Docker-Compose-for-S3.md
+45
@@ -0,0 +1,45 @@
To use SeaweedFS S3 API, here is the simplest form:
```
services:
seaweedfs-s3:
image: chrislusf/seaweedfs
container_name: seaweedfs-s3
ports:
- "8333:8333"
entrypoint: /bin/sh -c
command: |
"echo '{
\"identities\": [
{
\"name\": \"anonymous\",
\"actions\": [
\"Read\"
]
},
{
\"name\": \"some_admin_user\",
\"credentials\": [
{
\"accessKey\": \"some_access_key1\",
\"secretKey\": \"some_secret_key1\"
}
],
\"actions\": [
\"Admin\",
\"Read\",
\"List\",
\"Tagging\",
\"Write\"
]
}
]
}' > /etc/seaweedfs/config.json && \
weed server -s3 -s3.config /etc/seaweedfs/config.json"
restart: unless-stopped
```
## Notes
- To enable advanced IAM (STS, OIDC providers), mount a JSON and add `-iam.config=/etc/seaweedfs/iam.json` to the S3 command. See [[Keycloak Integration]].
- To use SSE-KMS (with OpenBao/Vault, AWS KMS, GCP KMS), mount KMS config (e.g. `s3_kms.json`) and start with `-config=/etc/seaweedfs/s3_kms.json`. See [[Server-Side-Encryption-SSE-KMS]].
- SSE-S3 and bucket default encryption work without external KMS; see [[Server-Side-Encryption]].
Environment-Variables.md
+236 -8
@@ -13,20 +13,49 @@ weed master
# Weed prefix
For `v`, `logtostderr`, `stderrthreshold`, `vmoudle`, `options`, `logdir`, `alsologtostderr`, `log_backtrace_at` , and `config_dir` you have to use `WEED_` as prefix for environment variable like this `WEED_CONFIG_DIR=/tmp`
# Docker
This is useful for using docker and docker compose
You have to override entrypoint to `weed` because defautl [entrypoint](https://github.com/seaweedfs/seaweedfs/blob/master/docker/entrypoint.sh) use default values for `volumeSizeLimitMB`, `volumePreallocate`, `mdir`, `dir`, and `max` and setting environment variables won't change these values.
## Docker
```shell
docker run --entrypoint weed -it -e IP_BIND=0.0.0.0 -e MDIR=/tmp -e PORT=5000 -e VOLUMEPREALLOCATE=true chrislusf/seaweedfs:3.45 master
## Configuration File Settings
For configuration file settings (like filer stores, replication settings, etc.), you must use the `WEED_` prefix with dots (`.`) replaced by underscores (`_`).
For example, the `filer.toml` configuration:
```toml
[redis2]
enabled = true
address = "localhost:6379"
password = "secret"
database = 0
```
## Docker Compose
Becomes these environment variables:
```shell
WEED_REDIS2_ENABLED=true
WEED_REDIS2_ADDRESS=localhost:6379
WEED_REDIS2_PASSWORD=secret
WEED_REDIS2_DATABASE=0
```
# S3 Admin Credentials
For S3 API server authentication, see the dedicated **[S3 Credentials](S3-Credentials)** page which covers:
- Configuration file setup (highest priority)
- Filer configuration (medium priority)
- Environment variables as fallback (lowest priority)
- AWS standard environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
- Complete authentication examples and troubleshooting
# Docker
You can set environment variables easily in Docker:
```shell
docker run --name master -d -p 9333:9333 -p 19333:19333 \
-e MDIR="/data" -e PORT="9333" \
chrislusf/seaweedfs:latest \
master
```
## Docker Compose with Environment Variables
```yaml
version: '3.9'
services:
master:
image: chrislusf/seaweedfs:3.45
image: chrislusf/seaweedfs:latest
ports:
- 9333:9333
- 19333:19333
@@ -38,4 +67,203 @@ services:
# or `VOLUMEPREALLOCATE:`
entrypoint: weed
command: master
filer:
image: chrislusf/seaweedfs:latest
ports:
- 8888:8888
environment:
# ... other filer environment variables
entrypoint: weed
command: filer -master=master:9333
s3:
image: chrislusf/seaweedfs:latest
ports:
- 8333:8333
environment:
AWS_ACCESS_KEY_ID: s3admin
AWS_SECRET_ACCESS_KEY: s3secret
entrypoint: weed
command: s3 -filer=filer:8888
depends_on:
- filer
```
# Filer Metadata Store Configuration
The filer supports multiple metadata storage backends. You can configure them using environment variables instead of a `filer.toml` file.
## Redis Configuration
### Basic Redis (`redis2`)
```yaml
version: '3.9'
services:
redis:
image: redis:7-alpine
ports:
- 6379:6379
command: redis-server --requirepass your_password
master:
image: chrislusf/seaweedfs:latest
ports:
- 9333:9333
command: master
volume:
image: chrislusf/seaweedfs:latest
ports:
- 8080:8080
command: volume -mserver=master:9333
depends_on:
- master
filer:
image: chrislusf/seaweedfs:latest
ports:
- 8888:8888
environment:
# Enable Redis as metadata store
- WEED_REDIS2_ENABLED=true
- WEED_REDIS2_ADDRESS=redis:6379
- WEED_REDIS2_PASSWORD=your_password
- WEED_REDIS2_DATABASE=0
# Optional: TLS configuration
- WEED_REDIS2_ENABLE_TLS=false
# Disable default leveldb2
- WEED_LEVELDB2_ENABLED=false
command: filer -master=master:9333
depends_on:
- master
- volume
- redis
```
### Redis Sentinel
```shell
WEED_REDIS2_SENTINEL_ENABLED=true
WEED_REDIS2_SENTINEL_ADDRESSES=sentinel1:26379,sentinel2:26379,sentinel3:26379
WEED_REDIS2_SENTINEL_MASTERNAME=mymaster
WEED_REDIS2_SENTINEL_USERNAME=
WEED_REDIS2_SENTINEL_PASSWORD=secret
WEED_REDIS2_SENTINEL_DATABASE=0
WEED_LEVELDB2_ENABLED=false
```
### Redis Cluster
```shell
WEED_REDIS_CLUSTER2_ENABLED=true
WEED_REDIS_CLUSTER2_ADDRESSES=redis1:6379,redis2:6379,redis3:6379
WEED_REDIS_CLUSTER2_PASSWORD=secret
WEED_REDIS_CLUSTER2_READONLY=false
WEED_REDIS_CLUSTER2_ROUTEBYLATENCY=false
WEED_LEVELDB2_ENABLED=false
```
## MySQL/MariaDB Configuration
```yaml
version: '3.9'
services:
mysql:
image: mysql:8
ports:
- 3306:3306
environment:
- MYSQL_ROOT_PASSWORD=secret
- MYSQL_DATABASE=seaweedfs
- MYSQL_USER=seaweedfs
- MYSQL_PASSWORD=secret
master:
image: chrislusf/seaweedfs:latest
ports:
- 9333:9333
command: master
volume:
image: chrislusf/seaweedfs:latest
ports:
- 8080:8080
command: volume -mserver=master:9333
depends_on:
- master
filer:
image: chrislusf/seaweedfs:latest
ports:
- 8888:8888
environment:
# MySQL configuration
- WEED_MYSQL_ENABLED=true
- WEED_MYSQL_HOSTNAME=mysql
- WEED_MYSQL_PORT=3306
- WEED_MYSQL_DATABASE=seaweedfs
- WEED_MYSQL_USERNAME=seaweedfs
- WEED_MYSQL_PASSWORD=secret
- WEED_MYSQL_CONNECTION_MAX_IDLE=5
- WEED_MYSQL_CONNECTION_MAX_OPEN=75
- WEED_MYSQL_CONNECTION_MAX_LIFETIME_SECONDS=600
- WEED_MYSQL_INTERPOLATEPARAMS=true
# Disable default leveldb2
- WEED_LEVELDB2_ENABLED=false
command: filer -master=master:9333
depends_on:
- master
- volume
- mysql
```
## PostgreSQL Configuration
```shell
WEED_POSTGRES_ENABLED=true
WEED_POSTGRES_HOSTNAME=postgres
WEED_POSTGRES_PORT=5432
WEED_POSTGRES_DATABASE=seaweedfs
WEED_POSTGRES_USERNAME=seaweedfs
WEED_POSTGRES_PASSWORD=secret
WEED_POSTGRES_SSLMODE=disable
WEED_POSTGRES_CONNECTION_MAX_IDLE=5
WEED_POSTGRES_CONNECTION_MAX_OPEN=75
WEED_POSTGRES_CONNECTION_MAX_LIFETIME_SECONDS=600
WEED_LEVELDB2_ENABLED=false
```
## MongoDB Configuration
```shell
WEED_MONGODB_ENABLED=true
WEED_MONGODB_URI=mongodb://mongodb:27017
WEED_MONGODB_DATABASE=seaweedfs
WEED_MONGODB_USERNAME=seaweedfs
WEED_MONGODB_PASSWORD=secret
WEED_LEVELDB2_ENABLED=false
```
## Etcd Configuration
```shell
WEED_ETCD_ENABLED=true
WEED_ETCD_SERVERS=etcd1:2379,etcd2:2379,etcd3:2379
WEED_ETCD_KEY_PREFIX=seaweedfs.
WEED_ETCD_TIMEOUT=3s
WEED_LEVELDB2_ENABLED=false
```
## Important Notes
1. **Only one store can be enabled**: Make sure to disable the default `leveldb2` store when using an external metadata store:
```shell
WEED_LEVELDB2_ENABLED=false
```
2. **Available stores**: To see all available filer stores and their configuration options, run:
```shell
weed scaffold -config=filer
```
3. **Data migration**: Changing stores doesn't automatically migrate existing data. Apply these configurations to new installations or migrate data manually.
4. **Array values**: For configuration options that accept arrays (like Redis cluster addresses), use comma-separated values:
```shell
WEED_REDIS_CLUSTER2_ADDRESSES=host1:6379,host2:6379,host3:6379
```
Erasure-Coding-for-warm-storage.md
+9 -1
@@ -47,7 +47,7 @@ The scripts have 3 steps related to erasure coding.
### Erasure Encode Sealed Data
`ec.encode` command will find volumes that are almost full and has been stale for a period of time.
The default command is `ec.encode -fullPercent=95 -quietFor=1h`. It will find volumes at least 95% of the maximum volume size, which is usually 30GB, and have no updates for 1 hour.
The default command is `ec.encode -fullPercent=95 -quietFor=1h -rebalance`. It will find volumes at least 95% of the maximum volume size, which is usually 30GB, and have no updates for 1 hour. Once the encoding is completed, EC shards are re-balanced; see [EC data balancing](#ec-data-balancing) below.
Note that if you have any collections, i.e. because you're using s3 where every bucket is a collection you explicitly need to specify the collection for the ec.encode command for them to be erasure coded `ec.encode -collection="collection" -fullPercent=95 -quietFor=1h`.
@@ -67,6 +67,14 @@ With servers added or removed, some data shards may not be laid out optimally. F
The default command is `ec.balance -force`. It will try to spread the data shards evenly to minimize the data shard loss risk.
EC shard re-balancing happens in three steps:
1. Duplicate EC shards for the same volume + server are deleted.
2. EC shards are balanced across volumes for all racks.
3. EC shards are balanced across volumes for individual racks.
Destination volume/racks for EC shards are selected based on capacity, favoring those with less preexisting shards in order to ensure an uniform distribution. Additionally, EC shards selection obey default [replica placement settings](Replication#the-meaning-of-replication-type) for the master server.
## How the read works?
When all data shards are online, the read for one file key are assigned to one volume server (A) that has at least one data shard for the volume. Server A will read its copy of index file, and locate the volume server (B), and read from server B for the file key.
Error-reporting-to-sentry.md
+15
@@ -0,0 +1,15 @@
# Overview
Since [probably 3.65](https://github.com/seaweedfs/seaweedfs/commit/953f5713490343f3cb4d36eb17bae9e8716068b4) release will be possible to send errors to sentry.
To enable reporting some environment variables must be set
* SENTRY_DSN (from project settings)
* SENTRY_ENVIRONMENT (production, prod, test or whatever is used in organization)
* SENTRY_RELEASE (weed@3.64)
No other settings involved at this moment.
SENTRY_DSN is the only required option and could be obtained from project settings.
SENTRY_ENVIRONMENT and SENTRY_RELEASE may be empty or not set at all.
Assign it to what match your organization's flow.
FAQ.md
+1 -1
@@ -168,7 +168,7 @@ If you **must** have custom gRPC ports, you can specify a custom gRPC port when
For example:
```
weed master -port=9333 -port.grpc=9444
weed volume -port=8080 -port.grpc=8444 -msever=localhost:9333.9444
weed volume -port=8080 -port.grpc=8444 -mserver=localhost:9333.9444
weed filer -port=8888 -port.grpc=9999 -master=localhost:9333.9444
weed shell -filer=localhst:8888.9999 -master=localhost:9333.9444
weed mount -dir=mm -filer=localhst:8888.9999
FUSE-Mount.md
+13
@@ -50,6 +50,12 @@ mount -t weed fuse /mnt -o "filer='192.168.0.1:8888,192.168.0.2:8888',filer.path
Now you can operate the SeaweedFS files, browsing or modifying directories and files, in local file system.
To unmount, just shut it down the "weed mount".
#### Mount with fstab
```
weed#fuse /mnt fuse _netdev,filer='192.168.0.1:8888',filer.path=/ 0 0
```
#### Mount outside of a SeaweedFS cluster
In addition to connecting to filer server, `weed mount` also directly connects to volume servers directly for better performance.
@@ -310,4 +316,11 @@ The issue is with samba.conf. If you see NT_STATUS_ACCESS_DENIED error, try to a
Size: total number of volumes * volume size limit
Used: (Logical Total Size of files - Logical Deleted File Size) * replication = physical disk size taken
Available = Size - Used
```
### Can't mount as non-root user ###
From https://github.com/seaweedfs/seaweedfs/issues/877
```
Workaround is to use Linux capabilities on weed executable.
setcap cap_net_raw,cap_net_admin,cap_dac_override+eip /usr/local/bin/weed
After that you need to use "weed mount" with option allowOthers=false
```
File-Operations-Quick-Reference.md
+172
@@ -0,0 +1,172 @@
# File Operations Quick Reference
This page provides a quick reference for common file operations in SeaweedFS using the HTTP API.
## Basic File Operations
### Upload a File
**Small files (direct upload):**
```bash
# Upload with PUT
curl -X PUT "http://localhost:8888/path/to/file.txt" -d "file content"
# Upload with POST (multipart)
curl -X POST "http://localhost:8888/path/to/file.txt" -F "file=@localfile.txt"
```
### Download a File
```bash
# Get file content
curl "http://localhost:8888/path/to/file.txt"
# Download to local file
curl "http://localhost:8888/path/to/file.txt" -o localfile.txt
```
### Delete a File
```bash
# Delete single file
curl -X DELETE "http://localhost:8888/path/to/file.txt"
# Delete directory (recursive)
curl -X DELETE "http://localhost:8888/path/to/directory/?recursive=true"
```
## Advanced File Operations
### Move/Rename Files
```bash
# Move file to new location
curl -X POST "http://localhost:8888/new/path/file.txt?mv.from=/old/path/file.txt"
# Rename file in same directory
curl -X POST "http://localhost:8888/path/to/newname.txt?mv.from=/path/to/oldname.txt"
```
### Copy Files ⭐ **New Feature**
```bash
# Copy file (preserves original)
curl -X POST "http://localhost:8888/backup/file.txt?cp.from=/original/file.txt"
# Copy with automatic name resolution
curl -X POST "http://localhost:8888/backup/?cp.from=/original/file.txt"
# Result: /backup/file.txt
```
### List Directory Contents
```bash
# List directory
curl -H "Accept: application/json" "http://localhost:8888/path/to/directory/?pretty=y"
# List with pagination
curl -H "Accept: application/json" "http://localhost:8888/path/?limit=10&lastFileName=somefile.txt"
```
### Create Directory
```bash
# Create empty directory
curl -X POST "http://localhost:8888/path/to/new/directory/"
```
## File Metadata Operations
### Get File Information
```bash
# Get file metadata
curl -I "http://localhost:8888/path/to/file.txt"
```
### File Attributes and Tagging
```bash
# Set custom attributes
curl -X PUT "http://localhost:8888/path/to/file.txt" \
-H "Seaweed-Custom-Attribute: value" \
-F "file=@localfile.txt"
# Set TTL (time to live)
curl -X POST "http://localhost:8888/path/to/file.txt?ttl=3600" \
-F "file=@localfile.txt"
```
## Response Codes
| HTTP Code | Operation | Meaning |
| --------- | --------- | ------- |
| 200 OK | GET | File retrieved successfully |
| 201 Created | POST/PUT | File uploaded successfully |
| 204 No Content | DELETE, COPY, MOVE | Operation completed successfully |
| 400 Bad Request | Any | Invalid parameters or unsupported operation |
| 404 Not Found | GET, DELETE | File or directory not found |
| 409 Conflict | POST/PUT | File already exists (when using exclusive creation) |
## Operation Comparison
| Operation | Source File | Use Case | Performance |
| --------- | ----------- | -------- | ----------- |
| **Upload** | External → SeaweedFS | Add new files | Network dependent |
| **Download** | SeaweedFS → External | Retrieve files | Network dependent |
| **Move** (`mv.from`) | Deleted | Rename/relocate | Very fast (metadata only) |
| **Copy** (`cp.from`) | Preserved | Backup/duplicate | Depends on file size |
| **Delete** | Deleted | Remove files | Fast |
## Best Practices
### When to Use Copy vs Move
**Use Copy (`cp.from`) when:**
- Creating backups before modifications
- Duplicating configuration templates
- Staging files for testing
- Need to preserve original file
**Use Move (`mv.from`) when:**
- Renaming files
- Reorganizing file structure
- Moving files between directories
- Don't need original file
### Performance Tips
1. **Batch operations**: Group multiple operations when possible
2. **Small files**: Use direct PUT for files < 1MB
3. **Large files**: POST with multipart automatically chunks files
4. **Copy operations**: Server-side copy is much faster than download + upload
5. **Directory operations**: Plan directory structure to minimize reorganization
### Error Handling
```bash
# Check operation status
if curl -X POST "http://localhost:8888/dest?cp.from=/src" -w "%{http_code}" -o /dev/null -s | grep -q "204"; then
echo "Copy successful"
else
echo "Copy failed"
fi
```
## Related Documentation
- [Filer Server API](Filer-Server-API.md) - Complete API reference
- [Filer Commands and Operations](Filer-Commands-and-Operations.md) - Command-line tools
- [Getting Started](Getting-Started.md) - Installation and setup
- [FAQ](FAQ.md) - Common questions and answers
## Examples Repository
For more complex examples and use cases, see:
- [SeaweedFS Examples](https://github.com/seaweedfs/seaweedfs/tree/master/examples) (if available)
- Community-contributed scripts and tools
- Language-specific client libraries
---
💡 **Tip**: Use the `?pretty=y` parameter with JSON responses to get formatted output for easier reading during development and testing.
Filer-Cassandra-Setup.md
+3 -2
@@ -18,10 +18,11 @@ create keyspace seaweedfs WITH replication = {
use seaweedfs;
CREATE TABLE filemeta (
dirhash bigint,
directory varchar,
name varchar,
meta blob,
PRIMARY KEY (directory, name)
PRIMARY KEY ((dirhash, directory), name)
) WITH CLUSTERING ORDER BY (name ASC);
```
@@ -33,7 +34,7 @@ Try run ```weed filer -h``` to see an example filer.toml file. The file should b
Here is the shortest example for Cassandra
```bash
[cassandra]
[cassandra2]
enabled = true
keyspace="seaweedfs"
hosts=[
Filer-Commands-and-Operations.md
+30
@@ -23,6 +23,36 @@ The above `weed copy` command is very efficient. It will contact the master serv
This put very little loads on filer and the master server. Data is only transmitted between the local machine and the volume server.
## Copy files within Filer
SeaweedFS also supports copying files within the filer using the HTTP API. This is useful for creating backups, duplicates, or templates without downloading and re-uploading files.
### HTTP API Copy
```bash
# Copy a file to a new location
curl -X POST 'http://localhost:8888/path/to/destination?cp.from=/path/to/source'
# Copy with automatic name resolution
curl -X POST 'http://localhost:8888/backup/?cp.from=/important/config.json'
# Creates: /backup/config.json
```
**Features:**
- **Efficient**: Server-side copy without client data transfer
- **Independent chunks**: Creates new chunk copies (not shared references)
- **Atomic operation**: Either succeeds completely or fails with no partial state
- **Preserves metadata**: File attributes, timestamps, and permissions maintained
**Comparison of copy methods:**
| Method | Use Case | Data Transfer | Performance |
| ------ | -------- | ------------- | ----------- |
| `weed filer.copy` | Local files → Filer | Client → Volume Server | Good for initial uploads |
| `cp.from` HTTP API | File → File within Filer | Volume Server → Volume Server | Excellent for server-side copies |
For more details, see the [Filer Server API documentation](Filer-Server-API.md#copy-files).
## Register a file on Filer
As mentioned above, the (path, fileId, fileSize) can be registered on filer with this gRPC call.
Filer-JWT-Use.md
+6
@@ -1,3 +1,7 @@
## Introduction
See [the security overview](https://github.com/seaweedfs/seaweedfs/wiki/Security-Overview#securing-filer-http-with-jwt) for a more complete description.
## How to get a valid JWT
The Filer won't generate JWTs, you will have to configure another service to create a JWT with the same signing key set in the `security.toml` file described on the Security Configuration wiki page. When generating the JWT, ensure that the timeout in seconds matches the timeout set in the `security.toml` file.
@@ -9,3 +13,5 @@ The Filer supports receiving a JWT in three different forms:
1. Through the `Authorization: Bearer <token>` header
2. Via the request's query parameters: http://localhost:8888/buckets/all?jwt=token
3. In an HTTP-only cookie named AT(Access Token)
Filer-Server-API.md
+125 -2
@@ -36,7 +36,8 @@ You can append to any HTTP API with &pretty=y to see a formatted json output.
| ttl | time to live, examples, 3m: 3 minutes, 4h: 4 hours, 5d: 5 days, 6w: 6 weeks, 7M: 7 months, 8y: 8 years | empty |
| maxMB | max chunk size | empty |
| mode | file mode | 0660 |
| op | file operation, currently only support "append" | empty |
| offset | incompatible with `op=append`. Defines the number of bytes from the file beginning to insert the uploaded chunk | empty |
| op | file operation, currently only support "append", incompatible with `offset=`. | empty |
| skipCheckParentDir | Ensuring parent directory exists cost one metadata API call. Skipping this can reduce network latency. | false |
| header: `Content-Type` | used for auto compression | empty |
| header: `Content-Disposition` | used as response content-disposition | empty |
@@ -48,7 +49,6 @@ You can append to any HTTP API with &pretty=y to see a formatted json output.
| resolveManifest | resolve manifest chunks | false |
### notice
* It is recommended to add retries when writing to Filer.
* `AutoChunking` is not supported for method `PUT`. If the file length is greater than 256MB, only the leading 256MB in the `PUT` request will be saved.
* When appending to a file, each append will create one chunk and added to the file metadata. If there are too many small appends, there could be too many chunks. So try to keep each append size reasonably big.
Examples:
@@ -236,6 +236,129 @@ Notice that the tag names follow http header key convention, with the first char
| ---- | -- | -- |
| mv.from | move from one file or directory to another location | Required field |
### Copy files
SeaweedFS supports efficient file copying using the `cp.from` parameter. This operation creates a complete copy of a file while preserving the original file.
#### Basic Usage
```bash
# Copy a file to the same directory with a new name
> curl -X POST 'http://localhost:8888/documents/report_backup.pdf?cp.from=/documents/report.pdf'
# Copy a file to a different directory
> curl -X POST 'http://localhost:8888/backup/important.txt?cp.from=/projects/important.txt'
# Copy with automatic name resolution (uses source filename)
> curl -X POST 'http://localhost:8888/backup/?cp.from=/projects/important.txt'
# Creates: /backup/important.txt
```
#### Parameters
| POST Parameter | Description | Default |
| ---- | -- | -- |
| cp.from | Source file path to copy from. Must be a valid file path. | Required field |
#### How Copy Works
**Small Files (< chunk size):**
- Content is stored directly in the filer metadata
- Copy operation duplicates the content bytes
- Very fast, only metadata operation required
**Large Files (chunked):**
- File data is stored as chunks on volume servers
- Copy operation reads data from source chunks and writes to new chunks
- Creates independent chunk copies (not shared references)
- Preserves file integrity and allows independent deletion
#### Examples
**Copy a configuration file:**
```bash
curl -X POST 'http://localhost:8888/config/app.conf.backup?cp.from=/config/app.conf'
```
**Copy a large media file:**
```bash
curl -X POST 'http://localhost:8888/media/backup/video.mp4?cp.from=/media/original/video.mp4'
```
**Copy with path resolution:**
```bash
# If destination ends with /, uses source filename
curl -X POST 'http://localhost:8888/backup/?cp.from=/important/data.json'
# Result: /backup/data.json
```
#### Response Codes
| HTTP Code | Description |
| ---- | -- |
| 204 No Content | Copy operation completed successfully |
| 400 Bad Request | Invalid source path, missing cp.from parameter, or directory copy attempt |
| 404 Not Found | Source file does not exist |
| 500 Internal Server Error | Volume server error or chunk copy failure |
#### Performance Characteristics
- **Small files**: Near-instant (metadata only)
- **Large files**: Proportional to file size (requires data transfer)
- **Network efficient**: Direct volume-to-volume transfer when possible
- **Atomic operation**: Either completes fully or fails with no partial state
#### Limitations
- **Directory copying**: Not supported (returns 400 error)
- **Cross-cluster copying**: Limited to same SeaweedFS cluster
- **Concurrent access**: Source file should not be modified during copy
#### Error Examples
**Attempting to copy a directory:**
```bash
curl -X POST 'http://localhost:8888/new_folder/?cp.from=/existing_folder/'
# Returns: 400 Bad Request - "directory copying not yet supported"
```
**Source file not found:**
```bash
curl -X POST 'http://localhost:8888/copy.txt?cp.from=/nonexistent.txt'
# Returns: 400 Bad Request - "failed to get src entry"
```
**Missing cp.from parameter:**
```bash
curl -X POST 'http://localhost:8888/copy.txt'
# Returns: Normal file upload behavior (not a copy operation)
```
#### Comparison with Move Operation
| Operation | Source File | Use Case | Speed |
| ---- | ---- | ---- | ---- |
| `mv.from` | Deleted | Rename/relocate files | Very fast (metadata only) |
| `cp.from` | Preserved | Backup/duplicate files | Depends on file size |
#### Best Practices
1. **Backup workflows**: Use copy for creating backups before modifications
2. **Template files**: Copy configuration templates to create new instances
3. **Data migration**: Copy files before cross-cluster transfers
4. **Testing**: Copy production files to staging environments
#### Shell Command Equivalent
The SeaweedFS copy operation is similar to:
```bash
# SeaweedFS copy
curl -X POST 'http://localhost:8888/dst?cp.from=/src'
# Unix equivalent
cp /src /dst
```
Note: Directory copying is not currently supported. Only individual files can be copied.
### Create an empty folder
Folders usually are created automatically when uploading a file. To create an empty file, you can use this:
```
Filer-Store-Replication.md
+2 -2
@@ -14,7 +14,7 @@ So when a filer was added or removed from the cluster, there is no configuration
What's more, adding a fresh new filer will automatically synchronize the metadata with other filers. Resuming a paused filer will also resume the metadata synchronization from when it was stopped. Everything will/should just work!
Note however that using multiple replicated embedded filers is only eventually consistent. So using this setup behind a load-balancer might not work as expected.
Note however that using multiple replicated embedded filers is only eventually consistent. So using this setup behind a load-balancer might not work as expected, although this can be solved with load balancer configuration, such as hashing connections based on the `uploadId` GET query, or temporary sticky sessions to ensure the objects requested are available.
# Metadata synchronization
@@ -27,7 +27,7 @@ Knowing all the peers, one filer will keep its own metadata updated:
This is tightly related to FUSE Mount, which streams filer meta data changes from one filer. When using multiple filers but without peer file metadata updates, a FUSE mount can only see the changes applied to the connected filer.
So aggregating metadata updates form its peers is required when the filers are using either shared or dedicated filer stores.
So aggregating metadata updates from its peers is required when the filers are using either shared or dedicated filer stores.
```
FUSE mount <----> filer1 -- filer2
Filer-Stores.md
+12 -2
@@ -28,14 +28,17 @@ The Filer Store persists all file metadata and directory information.
| ElasticSearch| O(logN)| unlimited | Distributed, Fast || Yes| | Scalable, Searchable. Need to manually build. |
| HBase | O(logN)| unlimited | Distributed, Fast | | Native| | Scalable |
| TiKV | O(logN)| unlimited | Distributed, Fast |Atomic|Yes| Yes| Scalable. High Availability. Need to manually build. |
| Tarantool | O(logN)| unlimited | Local or Distributed, Fast ||Yes|| Scalable. High Availability. Need to manually build. |
#### Switching between different Stores
It is easy to switch between different filer stores.
It is easy to switch between different filer stores.
This is useful if you started with embedded DB and want to move to distributed one or in reverse.
For example:
```sh
# first save current filer meta data
# make sure you don't create or modify files in filer before this to maintain consistency of metadata
# save current filer meta data to local disk
$ weed shell
> fs.cd /
@@ -45,6 +48,7 @@ total 65 directories, 292 files
meta data for http://localhost:8888/ is saved to localhost-8888-20190417-005421.meta
> exit
# you can turn off old filer if you have metadata saved to file
# now switch to a new filer, and load the previously saved metadata
$ weed shell
> fs.meta.load localhost-8888-20190417-005421.meta
@@ -52,6 +56,12 @@ $ weed shell
total 65 directories, 292 files
localhost-8888-20190417-005421.meta is loaded to http://localhost:8888/
# optionally, you can use concurrency to speed up the meta load process
# -concurrency=N specifies the number of parallel meta load operations to filer
# caution: this may not work correctly with some databases like Redis
# $ weed shell
# > fs.meta.load -concurrency=1 localhost-8888-20190417-005421.meta
```
### Extending Storage Options
Gateway-to-Remote-Object-Storage.md
+1 -1
@@ -43,7 +43,7 @@ It is OK to pause it, and resume.
It is also OK to change the `-createBucketAt=xxx` value to a different one, since it only affects new bucket creation.
```
$ weed filer.remote.sync -createBucketAt=cloud1
$ weed filer.remote.gateway -createBucketAt=cloud1
synchronize /buckets, default new bucket creation in cloud1 ...
```
Getting-Started.md
+3 -5
@@ -85,7 +85,7 @@ docker run -p 8080:8080 -p 18080:18080 --name volume --link master chrislusf/sea
But with Compose it's easiest.
To startup just run:
```
docker-compose -f docker/seaweedfs-compose.yml -p seaweedfs up
docker compose -f docker/seaweedfs-compose.yml -p seaweedfs up
```
## Using SeaweedFS in docker
@@ -141,16 +141,14 @@ You can use docker volumes to persist data:
```bash
# start our weed server daemonized
docker run --name weed -d -p 9333:9333 -p 8080:8080 -p 18080:8080 \
-v seaweedvolume:/data chrislusf/seaweedfs server -dir="/data" \
-publicIp="$(curl -s cydev.ru/ip)"
-v seaweedvolume:/data chrislusf/seaweedfs server -dir="/data"
```
Alternatively, you can mount a directory on the host machine into the container:
```bash
# start our weed server daemonized
docker run --name weed -d -p 9333:9333 -p 8080:8080 -p 18080:8080 \
-v /opt/weedfs/data:/data chrislusf/seaweedfs server -dir="/data" \
-publicIp="$(curl -s cydev.ru/ip)"
-v /opt/weedfs/data:/data chrislusf/seaweedfs server -dir="/data"
```
Note that according to [Docker's documentation](https://docs.docker.com/storage/volumes/), volumes are the preferred mechanism for persisting data.
Independent-Benchmarks.md
+3
@@ -24,3 +24,6 @@ https://github.com/bingoohuang/blog/issues/57
# Benchmark SeaweedFS as a GlusterFS replacement
https://github.com/seaweedfs/seaweedfs/wiki/Benchmark-SeaweedFS-as-a-GlusterFS-replacement
# Benchmark Object Stores
https://it-notes.dragas.net/2025/11/06/self-hosting-your-mastodon-media-with-seaweedfs/
Kafka-to-Kafka-Gateway-to-SMQ-to-SQL.md
+54
@@ -0,0 +1,54 @@
# Kafka client → Kafka gateway → SMQ → SQL
Bring your existing Kafka clients. Point them at the SeaweedFS Kafka gateway. Messages flow into Seaweed Message Queue (SMQ) for streaming, while SeaweedFS persists them into Parquet for SQL analytics.
> See the end-to-end picture: [[Structured Data Lake with SMQ and SQL]].
## Why use the Kafka gateway
- Keep your Kafka tooling and clients
- Scale stateless brokers and storage independently
- Get streaming + Parquet-based analytics without changing producers
## Architecture
```
Kafka Clients <=> SeaweedFS Kafka Gateway <=> SMQ Brokers => Subscribers
\
+--> SeaweedFS (Parquet) => SQL Engines
```
The gateway speaks the Kafka protocol to clients and maps topics/partitions, offsets, and consumer groups to SMQ semantics.
## What stays the same
- Kafka client libraries and tooling (producers/consumers)
- Topic/partition concepts
- Consumer groups and offsets
## What you gain
- Durable Parquet storage for batch analytics
- One pipeline for both streaming and SQL
- Simple, scalable operations (stateless brokers, disaggregated storage)
## Getting started
1) Start SMQ and the Kafka gateway (example ports):
```bash
weed mq.broker -port=17777 -master=localhost:9333
weed mq.agent -port=16777 -broker=localhost:17777
weed mq.kafka -port=19092 -broker=localhost:17777
```
2) Point your Kafka producer/consumer at `localhost:19092`.
3) Query the resulting Parquet data with your SQL engine of choice (Trino, Spark, DuckDB, etc.).
## Next steps
- Central concepts: [[Structured Data Lake with SMQ and SQL]]
- SMQ overview: [[Seaweed Message Queue]]
Keycloak-Integration.md
+187
@@ -0,0 +1,187 @@
# Keycloak Integration (OIDC) with SeaweedFS S3 Gateway
This guide shows how to integrate Keycloak (OpenID Connect) with SeaweedFS S3 Gateway using the advanced IAM and STS configuration. It supports both:
- Direct OIDC authentication to S3 with Bearer tokens
- OIDC to STS role assumption using trust policies and role mapping
## Prerequisites
- A running Keycloak server and a realm (e.g. `seaweedfs`)
- A Keycloak client (e.g. `seaweedfs-s3`) created in that realm
- SeaweedFS with advanced IAM enabled via `-iam.config`
## Step 1: Configure Keycloak
1) Create a client
- Client ID: `seaweedfs-s3`
- Access Type: public or confidential (confidential requires a client secret)
- Standard Flow: enabled (for browser login flows)
2) Add role/group claims to tokens
- Add a mapper of type "Group Membership" or "User Realm Role" that puts roles/groups into a top-level claim:
- Claim name: `groups` (recommended) or `roles`
- Add to ID token: true
- Add to Access token: true
3) Confirm OIDC discovery
- Open your realm discovery document and note the issuer and certs endpoints (Keycloak Quarkus defaults):
- Issuer: `https://KEYCLOAK/realms/<realm>`
- JWKS (certs): `https://KEYCLOAK/realms/<realm>/protocol/openid-connect/certs`
- UserInfo: `https://KEYCLOAK/realms/<realm>/protocol/openid-connect/userinfo`
Note: For older Keycloak distributions, issuer may include `/auth` in the path.
## Step 2: Prepare SeaweedFS IAM config
Create an IAM configuration JSON file (e.g. `/etc/seaweed/iam_keycloak.json`) and reference it with `weed s3 -iam.config=...`.
Minimal example with Keycloak OIDC provider, role mapping, roles, and policies:
```json
{
"sts": {
"tokenDuration": "1h",
"maxSessionLength": "12h",
"issuer": "seaweedfs-sts",
"signingKey": "c2Vhd2VlZGZzLXNpZ25pbmcta2V5LTMyLWNoYXJzLWxvbmc="
},
"providers": [
{
"name": "keycloak",
"type": "oidc",
"enabled": true,
"config": {
"issuer": "https://KEYCLOAK/realms/seaweedfs",
"clientId": "seaweedfs-s3",
"clientSecret": "<optional-if-confidential>",
"jwksUri": "https://KEYCLOAK/realms/seaweedfs/protocol/openid-connect/certs",
"userInfoUri": "https://KEYCLOAK/realms/seaweedfs/protocol/openid-connect/userinfo",
"scopes": ["openid", "profile", "email", "roles", "groups"],
"roleMapping": {
"rules": [
{ "claim": "groups", "value": "admins", "role": "arn:aws:iam::role/S3AdminRole" },
{ "claim": "groups", "value": "developers", "role": "arn:aws:iam::role/S3WriteRole" }
],
"defaultRole": "arn:aws:iam::role/S3ReadOnlyRole"
}
}
}
],
"policies": [
{
"name": "S3ReadOnlyPolicy",
"document": {
"Version": "2012-10-17",
"Statement": [
{ "Effect": "Allow", "Action": ["s3:List*", "s3:Get*"], "Resource": ["*"] }
]
}
},
{
"name": "S3WritePolicy",
"document": {
"Version": "2012-10-17",
"Statement": [
{ "Effect": "Allow", "Action": ["s3:List*", "s3:Get*", "s3:Put*", "s3:DeleteObject"], "Resource": ["*"] }
]
}
},
{
"name": "S3AdminPolicy",
"document": {
"Version": "2012-10-17",
"Statement": [
{ "Effect": "Allow", "Action": ["s3:*"] , "Resource": ["*"] }
]
}
}
],
"roles": [
{
"roleName": "S3ReadOnlyRole",
"roleArn": "arn:aws:iam::role/S3ReadOnlyRole",
"attachedPolicies": ["S3ReadOnlyPolicy"],
"trustPolicy": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Federated": "*" },
"Action": ["sts:AssumeRoleWithWebIdentity"],
"Condition": {
"StringEquals": { "seaweed:Issuer": "https://KEYCLOAK/realms/seaweedfs" }
}
}
]
}
},
{
"roleName": "S3WriteRole",
"roleArn": "arn:aws:iam::role/S3WriteRole",
"attachedPolicies": ["S3WritePolicy"],
"trustPolicy": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Federated": "*" },
"Action": ["sts:AssumeRoleWithWebIdentity"],
"Condition": {
"StringEquals": { "seaweed:Issuer": "https://KEYCLOAK/realms/seaweedfs" }
}
}
]
}
},
{
"roleName": "S3AdminRole",
"roleArn": "arn:aws:iam::role/S3AdminRole",
"attachedPolicies": ["S3AdminPolicy"],
"trustPolicy": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Federated": "*" },
"Action": ["sts:AssumeRoleWithWebIdentity"],
"Condition": {
"StringEquals": { "seaweed:Issuer": "https://KEYCLOAK/realms/seaweedfs" }
}
}
]
}
}
]
}
```
Notes:
- Set `signingKey` to a strong random secret (base64-encoded 32+ bytes). All S3 gateway instances must share the same STS `issuer` and `signingKey`.
- Explicit `jwksUri` and `userInfoUri` are recommended for Keycloak.
- Ensure your Keycloak mappers populate a top-level `groups` (or `roles`) claim.
## Step 3: Start the S3 Gateway
```bash
weed s3 -filer=filer:8888 -port=8333 -iam.config=/etc/seaweed/iam_keycloak.json
```
For multi-instance deployments, use the same IAM config on each instance.
## Using It
- Direct OIDC to S3 (Bearer): obtain a Keycloak access or ID token for client `seaweedfs-s3` and call S3 with:
```bash
curl -H "Authorization: Bearer $KEYCLOAK_TOKEN" http://s3-gateway:8333/
```
- Role selection: SeaweedFS maps OIDC claims via `roleMapping`. With the example above, users in `admins` get `S3AdminRole`, `developers` get `S3WriteRole`, others default to `S3ReadOnlyRole`.
## Troubleshooting
- Invalid token: verify token `iss` equals the configured provider `issuer` and `aud` or `azp` equals the client ID.
- JWKS errors: ensure `jwksUri` is reachable from the S3 gateway. For Keycloak, use the `.../protocol/openid-connect/certs` endpoint.
- No roles applied: confirm Keycloak mapper emits `groups` (or `roles`) at top-level in the token, and adjust `roleMapping` rules accordingly.
- Trust policy denied: ensure `seaweed:Issuer` in the trust policy matches your Keycloak realm issuer exactly.
Kubernetes-Backups-and-Recovery-with-K8up.md
+149 -69
@@ -60,9 +60,9 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034
5. Install k8up
```bash
repo add k8up-io https://k8up-io.github.io/k8up
helm repo add k8up-io https://k8up-io.github.io/k8up
helm repo update
kubectl apply -f https://github.com/k8up-io/k8up/releases/download/k8up-4.4.3/k8up-crd.yaml
helm install k8up k8up-io/k8up
```
@@ -84,11 +84,12 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034
export PATH=$PATH:$HOME/minio-binaries/
```
2. Clone repo and cd to the helm dir
2. Download SeaweedFS, unzip, then cd to the helm dir
```bash
git clone https://github.com/seaweedfs/seaweedfs
cd seaweedfs/k8s/charts/seaweedfs
wget https://github.com/seaweedfs/seaweedfs/archive/refs/tags/3.77.zip
unzip 3.77.zip
cd seaweedfs-3.77/k8s/charts/seaweedfs
```
3. Create a minimal values file for the Seaweedfs deployment which adds annotations for K8up.
@@ -96,85 +97,129 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034
```bash
/bin/cat << EOF > test-values.yaml
master:
enabled: true
data:
type: "persistentVolumeClaim"
size: "5Gi"
size: "10G"
storageClass: "local-path"
annotations:
"k8up.io/backup": "true"
livenessProbe:
periodSeconds: 5
readinessProbe:
periodSeconds: 5
volume:
enabled: true
readMode: proxy
dataDirs:
- name: data
type: "persistentVolumeClaim"
size: "10G"
storageClass: "local-path"
annotations:
"k8up.io/backup": "true"
maxVolumes: 0
idx: {}
livenessProbe:
periodSeconds: 5
readinessProbe:
periodSeconds: 5
filer:
enabled: true
encryptVolumeData: true
enablePVC: true
storage: 10Gi
defaultReplicaPlacement: "000"
data:
type: "persistentVolumeClaim"
size: "5Gi"
size: "10G"
storageClass: "local-path"
annotations:
"k8up.io/backup": "true"
filer:
enablePVC: true
storage: 5Gi
data:
type: "persistentVolumeClaim"
size: "5Gi"
storageClass: "local-path"
annotations:
s3:
enabled: true
enableAuth: true
port: 8333
httpsPort: 0
allowEmptyFolder: false
domainName: ""
enableAuth: false
skipAuthSecretCreation: false
auditLogConfig: {}
createBuckets:
- name: shared
anonymousRead: false
livenessProbe:
periodSeconds: 5
readinessProbe:
periodSeconds: 5
s3:
enabled: false
cosi:
enabled: false
EOF
```
4. Deploy via Helm
4. Deploy via Helm (takes longer on slow drives)
```bash
helm template . -f test-values.yaml > manifests.yaml
kubectl apply -f manifests.yaml
helm install seaweedfs . -f test-values.yaml --wait
```
5. Expose the S3 endpoint
5. Expose the filer service via a LoadBalancer (servicelb in k3s). This will let us view the filer's file browser UI as well as reach the S3 endpoint during the demo.
```bash
/bin/cat << EOF > service.yaml
apiVersion: v1
kind: Service
metadata:
name: swfs-s3-nodeport
labels:
"app.kubernetes.io/name": "seaweedfs"
app.kubernetes.io/component: filer
app.kubernetes.io/instance: seaweedfs
app.kubernetes.io/name: seaweedfs
name: seaweedfs-filer-lb
spec:
type: NodePort
ports:
- port: 8333
nodePort: 30000
- name: swfs-filer
port: 8888
protocol: TCP
targetPort: 8888
- name: swfs-filer-grpc
port: 18888
protocol: TCP
targetPort: 18888
- name: swfs-s3
port: 8333
protocol: TCP
targetPort: 8333
- name: metrics
port: 9327
protocol: TCP
targetPort: 9327
selector:
"app.kubernetes.io/name": "seaweedfs"
app.kubernetes.io/component: filer
app.kubernetes.io/name: seaweedfs
type: LoadBalancer
EOF
```
6. Export your NodeIP as an env var
6. Export your LoadBalancer IP address as an env var
```bash
export NODE_IP=""
```
7. Create an alias for your server using your S3 CLI tool:
- You can find the `admin_access_key_id` and `admin_secret_access_key` values in the secret `seaweedfs-s3-secret`
```bash
mc alias set seaweedfs http://$NODE_IP:30000 $admin_access_key_id $admin_secret_access_key
mc alias set seaweedfs http://$NODE_IP:8333 $admin_access_key_id $admin_secret_access_key
```
8. Create a bucket that will hold our demo data
```bash
mc mb seaweedfs s3://backups
mc mb seaweedfs/backups
```
8. Add some data to the bucket
@@ -182,19 +227,27 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034
```bash
mc cp ./some-file seaweedfs/backups/
```
9. Verify its there
```bash
mc ls seaweedfs/backups
```
10. Open the Web UI at http://$NODE_IP:8888 in a browser to view or add more data.
<h2 id="configure-scheduled-backups-of-seaweedf3-to-b2">Configure scheduled backups of SeaweedFS to B2</h2>
1. Create a secret containing your external S3 credentials
- You will need to get these from your provider (Backblaze, Wasabi etc..):
```bash
export ACCESS_KEY_ID=$(echo -n "" | base64)
export ACCESS_SECRET_KEY=$(echo -n "" |base64)
```
```bash
/bin/cat << EOF > backblaze-secret.yaml
apiVersion: v1
@@ -209,15 +262,15 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034
kubectl apply -f backblaze-secret.yaml
```
2. Create a secret containing a random password for restic
- Generate a password and base64 encode it.
- Generate a password.
```bash
export RESTIC_PASS=$(openssl rand -base64 32)
export RESTIC_PASS=""
```
- Create a secret manifest
```bash
@@ -227,11 +280,11 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034
metadata:
name: restic-repo
type: Opaque
data:
stringData:
"password": "$RESTIC_PASS"
EOF
```
- Create the secret
```bash
@@ -245,9 +298,9 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034
```bash
export BACKUP_S3_URL=""
export BACKUP_S3_BUCKET=""
```
```
- Create a manifest for the backup
- Create a manifest for the backup
```bash
/bin/cat << EOF > backup.yaml
@@ -282,7 +335,7 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034
EOF
```
- Create the backup
- Create the backup and let it run
```bash
kubectl apply -f backup.yaml
@@ -290,12 +343,14 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034
<h2 id="restore-seaweedfs-from-b2-backups">Restore SeaweedFS from B2 backups</h2>
1. Uninstall SeaweedFS and delete your scheduled backup
1. Uninstall SeaweedFS, delete the PVCs, secrets, and scheduled backup
```bash
k delete -f manifests.yaml
kubectl delete -f backup.yaml
helm uninstall seaweedfs
kubectl delete pvc data-default-seaweedfs-master-0
kubectl delete pvc data-filer-seaweedfs-filer-0
kubectl delete pvc data-seaweedfs-volume-0
```
2. Create PVCs to hold our restored data
@@ -345,7 +400,7 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034
storage: 10Gi
EOF
```
- Create the PVCs
```bash
@@ -357,7 +412,7 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034
```bash
# the password used in your restic-repo secret
export RESTIC_PASSWORD=""
# Your S3 credentials
export AWS_ACCESS_KEY_ID=""
export AWS_SECRET_ACCESS_KEY=""
@@ -379,9 +434,9 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034
3 snapshots
```
4. Use the K8up CLI or a declarative setup to restore data to the PVC. You will need to do this for each PVC that needs to be restored
- Example manifest for a S3-to-PVC restore job which uses the restic snapshots shown above.
4. Use the K8up CLI or a declarative setup to restore data to the PVC. You will need to do this for each PVC that needs to be restored
- Example manifest for a S3-to-PVC restore job which uses the restic snapshots shown above.
```bash
/bin/cat << EOF > s3-to-pvc.yaml
@@ -461,7 +516,7 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034
```bash
kubectl apply -f s3-to-pvc.yaml
```
5. Re-deploy Seaweedfs from the existing PVCs
- Create a manifest that targets the PVCs we created
@@ -469,45 +524,70 @@ PR ref: https://github.com/seaweedfs/seaweedfs/pull/5034
```bash
/bin/cat << EOF > restore-values.yaml
master:
enabled: true
data:
type: "existingClaim"
claimName: "swfs-master-data"
livenessProbe:
periodSeconds: 5
readinessProbe:
periodSeconds: 5
volume:
data:
type: "existingClaim"
claimName: "swfs-volume-data"
enabled: true
readMode: proxy
dataDirs:
- name: data
type: "existingClaim"
claimName: "swfs-volume-data"
maxVolumes: 0
idx: {}
livenessProbe:
periodSeconds: 5
readinessProbe:
periodSeconds: 5
filer:
enabled: true
encryptVolumeData: true
enablePVC: true
storage: 10Gi
defaultReplicaPlacement: "000"
data:
type: "existingClaim"
claimName: "swfs-filer-data"
s3:
enabled: true
enableAuth: false
port: 8333
httpsPort: 0
allowEmptyFolder: false
domainName: ""
enableAuth: false
skipAuthSecretCreation: false
auditLogConfig: {}
livenessProbe:
periodSeconds: 5
readinessProbe:
periodSeconds: 5
s3:
enabled: false
cosi:
enabled: false
EOF
```
- Deploy via Helm
```bash
helm template . -f restore-values.yaml > manifests.yaml
kubectl apply -f manifests.yaml
helm install seaweedfs . -f restore-values.yaml --wait
```
7. Update your alias for your server:
- get the `admin_access_key_id` and `admin_secret_access_key` from the secret `seaweedfs-s3-secret`
```bash
mc alias set seaweedfs http://$NODE_IP:30000 $admin_access_key_id $admin_secret_access_key
```
- View for your data:
```bash
Master-Server-API.md
+1 -1
@@ -235,7 +235,7 @@ curl "http://localhost:9333/dir/status?pretty=y"
### Check Volume Status
```
curl localhost:9333/vol/status?pretty=y
curl "localhost:9333/vol/status?pretty=y"
{
"Version": "30GB 2.24 ",
"Volumes": {
Optimization.md
+1
@@ -50,6 +50,7 @@ copy_1 = 7 # create 1 x 7 = 7 actual volumes
copy_2 = 6 # create 2 x 6 = 12 actual volumes
copy_3 = 3 # create 3 x 3 = 9 actual volumes
copy_other = 1 # create n x 1 = n actual volumes
threshold = 0.9 # create threshold
```
## Increase concurrent reads
Path-Specific-Configuration.md
+2
@@ -99,6 +99,8 @@ Usage of fs.configure:
assign writes with this ttl
-volumeGrowthCount int
the number of physical volumes to add if no writable volumes
-worm
write-once-read-many, written files are readonly
```
PostgreSQL-compatible-Server-weed-db.md
+106
@@ -0,0 +1,106 @@
# PostgreSQL-compatible Server (weed db)
SeaweedFS exposes a PostgreSQL wire protocol server so you can connect with any PostgreSQL client (psql, JDBC/ODBC, BI tools) and run SQL over Seaweed Message Queue (SMQ) topics. It is read-only and designed for analytics: query live streams and Parquet-backed history without ETL.
## What you get
- PostgreSQL wire protocol compatibility (psql, JDBC/ODBC, BI tools)
- Query live + historical data (hybrid scanner: brokers + Parquet files)
- Simple ops: stateless server, MD5 auth, optional TLS
- Works alongside pub/sub and Kafka ingestion — one structured data lake
## Start the server
```bash
# Development (no auth)
weed db
# Production (MD5 auth)
weed db -auth=md5 -users='{"admin":"secret","analyst":"readonly"}' -host=0.0.0.0 -port=5432
# Credentials from file (recommended)
echo '{"admin":"secret","analyst":"readonly"}' > /etc/seaweedfs/users.json
weed db -auth=md5 -users="@/etc/seaweedfs/users.json"
# With TLS
auth_users='@/etc/seaweedfs/users.json'
weed db -auth=md5 -users="$auth_users" -tls-cert=/etc/ssl/server.crt -tls-key=/etc/ssl/server.key
```
Key options:
- `-auth`: `trust` (dev), `password`, `md5` (recommended)
- `-users`: inline JSON or `@/path/to/users.json`
- `-master`: SeaweedFS master (comma-separated for HA)
- `-max-connections`, `-idle-timeout`, `-database`
- `-tls-cert`, `-tls-key` for encrypted connections
## Connect with clients
```bash
# psql
PGPASSWORD=secret psql -h localhost -p 5432 -U admin -d default
psql "postgresql://admin:secret@localhost:5432/default"
```
```java
// JDBC
String url = "jdbc:postgresql://localhost:5432/default";
Connection conn = DriverManager.getConnection(url, "admin", "secret");
```
```python
# psycopg2
import psycopg2
conn = psycopg2.connect(host="localhost", port=5432, user="admin", password="secret", database="default")
```
BI tools: use the native PostgreSQL connector (host, port 5432, database `default`, user/password).
## SQL you can run
Schema/navigation:
```sql
SHOW DATABASES; -- MQ namespaces
USE my_namespace; -- switch database
SHOW TABLES; -- topics in namespace
DESCRIBE topic_name; -- schema
```
Queries (read-only):
```sql
SELECT * FROM events LIMIT 10;
SELECT * FROM events WHERE _ts >= '2025-01-01' LIMIT 1000;
SELECT COUNT(*) FROM events; -- fast-path optimized
SELECT MIN(timestamp), MAX(timestamp) FROM events; -- fast-path
-- System columns available on all tables
SELECT _ts, _key, _source, * FROM events LIMIT 5;
```
Notes:
- `_ts` supports convenient string-to-timestamp parsing in WHERE clauses
- Live + Parquet results are merged transparently
## Limitations (current)
- Read-only: no INSERT/UPDATE/DELETE, no DDL (CREATE/DROP/ALTER)
- Not supported yet: `JOIN`, `ORDER BY`, `GROUP BY`, `HAVING`, window functions, subqueries
## Operations and scaling
- Stateless server; run multiple `weed db` instances behind a load balancer
- Use MD5 auth and TLS in production
- Tune `-max-connections` and `-idle-timeout`
## How this fits your data lake
- Producers (Kafka or pub/sub) write schematized messages into SMQ
- SeaweedFS persists to Parquet; brokers serve live data
- PostgreSQL-compatible server exposes SQL over both — one data lake
## Related
- [[Structured Data Lake with SMQ and SQL]]
- [[SQL Queries on Message Queue]]
- [[SQL Quick Reference]]
- [[Seaweed Message Queue]]
Production-Setup.md
+26 -24
@@ -2,16 +2,16 @@ A robust production setup requires more configuration -- and care -- than the
[[Getting Started]] guide. There are multiple layers of components. Please follow
the steps to set them up from bottom up, one by one.
* Setup object storage
* Setup Masters
* Set up object storage
* Set up Masters
* Add volume servers
* Setup file storage
* Set up file storage
* Choose filer store
* Setup Filer
And then, choose the component you want to setup
* Setup S3
* Setup FUSE mount
And then, choose the component you want to set up
* Set up S3
* Set up FUSE mount
* Cluster Maintenance
## Prerequisites
@@ -25,12 +25,12 @@ Make sure the ports are open. By default
| Filer | 8888 | 18888 |
| S3 | 8333 | |
If you have multi-homed servers (many IP addresses and interfaces), you need
to ensure SeaweedFS uses the correct IP, for cluster communication. Append
If you have multi-homed servers (many IP addresses and interfaces),
ensure SeaweedFS uses the correct IP for cluster communication. Append
`-ip=xx.xx.xx.xx` to specify the appropriate address.
If you wish to use a different IP address for user-facing services, then
set `-publicIp=yy.yy.yy.yy` as well.
set `-ip.bind=yy.yy.yy.yy` as well.
### For single node setup
@@ -46,17 +46,17 @@ empty spaces, you need to reduce the volume size.
weed server -filer -s3 -ip=xx.xx.xx.xx -volume.max=0 -master.volumeSizeLimitMB=1024
```
# Setup object storage
# Set up object storage
## Setup Masters
## Set up Masters
### One master is fine
If there are 2 machines, it is not possible to achieve consensus. Just do not bother to setup multiple masters.
If there are 2 machines, it is not possible to achieve consensus. Just do not bother to set up multiple masters.
Even for large clusters, it is totally fine to have one single master. The load on master is very light. It is unlikely to go down. You can always just restart it since it only has soft states collected from volume servers.
### Setup masters
### Set up masters
OK. Your CTO just wants multiple masters. To do so, see [[Failover Master Server]] for details.
@@ -71,7 +71,7 @@ weed master -mdir=/data/seaweedfs/master -peers=ip1:9333,ip2:9333,ip3:9333 -ip=i
Additional notes:
* Depending on the available disk space on each volume server, the master may need
to reduce maximum volume size, e.g., add `-volumeSizeLimitMB=1024`. This will ensure
each volume has several volumes. On the note, you can't change `volumeSizeLimitMB` later.
each volume server has several volumes. On the note, you can't change `volumeSizeLimitMB` later.
* Since it is for production, you may also want to add `-metrics.address=<Prometheus gateway address>`. See [[System Metrics]].
@@ -110,17 +110,18 @@ Additional notes:
* If the disk space is huge and there will be a lot of volumes, configure `-index=leveldb` to reduce memory load.
* For busy volume servers, `-compactionMBps` can help to throttle the background jobs, e.g., compaction, balancing, encoding/decoding,etc.
* After adding volume servers, there will not be data rebalancing. It is generally not a good idea to actively rebalance data, which cost network bandwidth and slows down other servers. Data are written to new servers after new volumes are created on them. You can use `weed shell` and run `volume.balance -force` to manually balance them.
* Multiple volume servers on the same physical host count as separate servers for replication purposes. So if you have two physical hosts with multiple volume servers each, replication `001` (one replica in the same rack) does not guarantee that each copy will be stored on different physical hosts.
## Check the object store setup
Now the object store setup is completed. You can visit `http://<master>:9333/` to check it around.
* Make sure the Free volume count is not zero.
* Try to assign some file ids to trigger a volume allocation.
* Ensure the Free volume count is not zero.
* Try to assign some file IDs to trigger a volume allocation.
If you only use SeaweedFS object store, that is all.
# Setup file storage
# Set up file storage
## Choose filer store
@@ -128,7 +129,7 @@ If currently only one filer is needed, just use one filer with default filer sto
You can always migrate to other scalable filer store by export and import the filer meta data. See [[Filer Stores]]
Run `weed scaffold -config=filer` to generate an example `filer.toml` file.
Run `weed scaffold -config=filer` to generate an example `filer.toml` file. This file choose `leveldb2` as the filer store by default which stores file meta in local on disk. `leveldb2` only support one filer.
The filer store to choose depends on your requirements, your existing data stores, etc.
@@ -140,7 +141,7 @@ weed filer -ip=xxx.xxx.xxx.xxx -master=ip1:9333,ip2:9333,ip3:9333 -dataCenter=dc
```
Additional notes:
* Both `weed filer` and `weed master` has option `-defaultReplicaPlacement`. `weed master` uses it for the object store, while `weed filer` uses it for files. The `weed filer` default setting is "000", and overwrites the one `weed master` has.
* Both `weed filer` and `weed master` has option `-defaultReplicaPlacement`. `weed master` uses it for the object store, while `weed filer` uses it for files. The `weed filer` setting will default to the value for `weed master`.
* `-encryptVolumeData` option is when you need to encrypt the data on volume servers. See [[Filer Data Encryption]]
## Setup multiple filers
@@ -156,24 +157,25 @@ Follow [[Amazon S3 API]] to generate a json config file, to assign accessKey and
Start s3 together with filer. This avoids the setup for s3 to support multiple filers.
```
weed filer -s3 -s3.config=<config.json> -port=8333
weed filer -s3 -s3.config=<config.json> -s3.port=8333
```
The endpoint is `http://<s3_server_host>:8333`.
## Setup FUSE mount
## Set up FUSE mount
Run
`weed mount -filer=<filer_host:filer_port> -chunkCacheCountLimit=xxx -chunkSizeLimitMB=4`
`weed mount -filer=<filer_host:filer_port> -cacheCapacityMB=xxx -chunkSizeLimitMB=4 -dir=mount_point_dir`
* `-chunkCacheCountLimit` means how many entries cached in memory, default to 1000. With default `-chunkSizeLimitMB` set to 4, it may take up to 4x1000 MB memory. If all files are bigger than 4MB.
* `-cacheCapacityMB` means file chunk read cache capacity in MB with tiered cache(memory + disk), default 0 which means chunk cache for read is disabled.
* `-chunkSizeLimitMB` local write buffer size, also chunk large file, default 2 MB.
* `-replication` is the replication level for each file. It overwrites replication settings on both filer and master.
* `-volumeServerAccess=[direct|publicUrl|filerProxy]` is used if master, volume server, and filer are inside a cluster, but `weed mount` is outside of the cluster. With this option set to `filerProxy`, only filer needs to be exposed to outside. All read write access to volume servers will be proxied by filer.
## Cluster Maintenance
In a cluster, volume servers can go down. But automatic rebalancing will be problematic. It can cause unexpected busy network activities. For example, the heartbeat of a volume server may come and go, causing unnecessary busy system. Current recommended strategy is to keep existing data readonly, and automatically add new writable volumes.
In a cluster, volume servers can go down. But automatic rebalancing will be problematic. It can cause unexpected busy network activities. For example, the heartbeat of a volume server may come and go, causing an unnecessarily busy system. Currently recommended strategy is to keep the existing data readonly, and automatically add new writable volumes.
There are `volume.balance` and `volume.fix.replication` commands in `weed shell`. You can configure them to run during off hours.
Pub-Sub-to-SMQ-to-SQL.md
+67
@@ -0,0 +1,67 @@
# Pub/Sub to SMQ to SQL: From Streams to Tables
Seaweed Message Queue (SMQ) bridges your live pub/sub streams and your analytical world. You publish structured messages, SMQ brokers stream them in real time, and at the same time SeaweedFS persists them into Parquet files so you can query with SQL engines later.
> New here? See the bigger picture: [[Structured Data Lake with SMQ and SQL]].
## Why this matters
- Real-time and batch in one pipeline: stream to subscribers now, query with SQL later.
- Store once, use twice: messages land in Parquet (columnar, compressed), great for analytics.
- Stateless brokers, disaggregated storage: scale brokers independently of storage.
## Architecture at a glance
```
Publishers => SMQ Agent (gRPC) => SMQ Brokers => Subscribers
\
+--> SeaweedFS (Parquet) => SQL Engines
```
- **Publish**: Send structured messages with a schema.
- **Stream**: Subscribers process messages with consumer groups and offsets.
- **Persist**: Messages are compacted/organized into Parquet files in SeaweedFS.
- **Query**: Point your SQL engines to the Parquet location.
## What you publish
Messages are structured records defined by a schema. SMQ validates these on publish and keeps ordering guarantees per key while enabling high concurrency via a sliding window.
## What you query
Parquet files written by SMQ are queryable by your favorite SQL engines:
- Trino/Presto
- Spark SQL
- DuckDB
- ClickHouse (via file table engines)
Point them to the Parquet path in SeaweedFS and query away.
## Quick start
1) Start a broker and an agent:
```bash
weed mq.broker -port=17777 -master=localhost:9333
weed mq.agent -port=16777 -broker=localhost:17777
```
2) Define a schema and publish:
```go
type MyEvent struct {
Key []byte
UserId int64
Action string
}
```
3) Subscribe in real time. Your consumers get the stream; your data team gets Parquet.
## Where to go next
- Central concepts: [[Structured Data Lake with SMQ and SQL]]
- Messaging basics: [[Seaweed Message Queue]]
Replication.md
+15 -13
@@ -83,17 +83,7 @@ No change to Submitting, Reading, and Deleting files.
*Note: This subject to change.*
Value | Meaning
---|---
000 | no replication, just one copy
001 | replicate once on the same rack
010 | replicate once on a different rack in the same data center
100 | replicate once on a different data center
200 | replicate twice on two other different data center
110 | replicate once on a different rack, and once on a different data center
... | ...
So if the replication type is xyz
The replication type is defined by a 3 digit string as follows:
Column | Meaning
---|---
@@ -101,9 +91,21 @@ Column | Meaning
**y** | number of replica in other racks in the same data center
**z** | number of replica in other servers in the same rack
x,y,z each can be 0, 1, or 2. So there are 9 possible replication types, and can be easily extended.
Each replication type will physically create x+y+z+1 copies of volume data files.
The replication string represents the additional number of copies of data that will be maintained by the seaweed cluster beyond the original data volume. Data remains readable as long as at least on replica is accessible. However, writes are permitted only if the number of replicas is achievable based on the replication configuration string. The max value that can be specified is '255'. The total number of copies of this volume will be the sum of the digits + 1. The original data is not counted.
Here are some possible replication configuration examples and what they mean:
Value | Meaning
---|---
000 | no replication, just one copy
001 | replicate once on the same rack
010 | replicate once on a different rack in the same data center
052 | replicate to five different racks within the same data center and two different volumes within the same rack
100 | replicate once on a different data center
200 | replicate twice on two other different data center
110 | replicate once on a different rack within the same data center and once on a different data center
... | ...
255 | replicate twice on two different data centers, five different racks and 5 different volumes servers with respect to where the original volume exists.
## Allocate File Key on specific data center
Run-Blob-Storage-on-Public-Internet.md
+2 -2
@@ -14,7 +14,7 @@ We will use 2 servers. Server 1 will host master, 2x volumes (2 disks, one volum
# todo: use 2 step build process, copy over weed binary to fresh container (do not need curl and tar at runtime)
FROM alpine
RUN apk update && apk add wget tar
RUN wget https://github.com/seaweedfs/seaweedfs/releases/download/3.59/linux_amd64_large_disk.tar.gz
RUN wget https://github.com/seaweedfs/seaweedfs/releases/download/3.80/linux_amd64_large_disk.tar.gz
RUN tar -xf linux_amd64_large_disk.tar.gz
RUN chmod +x weed
RUN mv weed /usr/bin/
@@ -144,7 +144,7 @@ services:
I haven't tested this, but this is how I would go about making it HA:
- Run master server on 3 or 5 physical servers
- Run caddy as a sidecar on every server that runs a master
- Install caddy from own Dockerfile, adding [redis plugin](github.com/gamalan/caddy-tlsredis) for distributed SSL cert storage and distributed locks for SSL cert issuing.
- Install caddy from own Dockerfile, adding [redis plugin](https://github.com/gamalan/caddy-tlsredis) for distributed SSL cert storage and distributed locks for SSL cert issuing.
- Run redis in high-availability mode, for example by [following this docker swarm guide](https://medium.com/@emmano3h/redis-high-availability-with-docker-swarm-2142a4d80b49). Caddy redis plugin probably doesn't allow multiple IP addresses, so might have to add `haproxy` sidecar to every caddy sidecar as well to load balance to redis cluster.
- Update `command` of volume servers docker-compose file to add all master server IPs
S3-API-FAQ.md
+204 -1
@@ -43,7 +43,7 @@ Having separate LevelDB instance, or separate SQL tables, will help to isolate t
Due to the semantics of the S3 API, empty directories (aka prefixes) aren't shown. However, an entry is still stored in the filer metadata store. When workload access patterns create many unique directories and then remove all the objects inside those directories, the filer metadata store can grow unbounded with orphaned directories. These directories are visible in the filer metadata store itself, but not using the S3 API.
If the filer argument `-allowEmptyFolder=false` is set, the orphaned directories are cleaned up during list requests for non bucket-level directories. Normally this works well, but if the workload never performs a list operation, the orphaned directories may never be cleaned up. To force cleanup, simply list an existing, non bucket-level directory.
If the filer argument `-s3.allowEmptyFolder=false` is set, the orphaned directories are cleaned up during list requests for non bucket-level directories. Normally this works well, but if the workload never performs a list operation, the orphaned directories may never be cleaned up. To force cleanup, simply list an existing, non bucket-level directory.
Example using rclone:
@@ -52,3 +52,206 @@ rclone lsf seaweedfs:my-bucket/dir
```
If the directory `dir` exists in `my-bucket`, the orphaned metadata will be cleaned up. Note that due to slight API usage differences, `rclone ls` does not trigger cleanup, but `rclone lsf` will.
## Setting TTL
It is possible to set a TTL for a specific directory using the S3 API. They are set using [`PutBucketLifecycleConfiguration`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html).
Example of a JSON configuration is below. It is an equivalent of calling the following with `weed shell`.
```
fs.configure -locationPrefix /buckets/f341868e-baff-4e20-896a-08bc148e32f9/my-directory-whose-files-will-expire-in-20-days -ttl 20d -apply
```
:
```
{
"Rules": [
{
"Status": "Enabled",
"Filter": {
"Prefix": "my-directory-whose-files-will-expire-in-20-days"
},
"Expiration": {
"Days": 20
}
}
]
}
```
Save that in a `.json` file and call it, for example, via the `aws` cli:
```
BUCKET_NAME=f341868e-baff-4e20-896a-08bc148e32f9
aws --endpoint-url http://127.0.0.1:8333 s3api put-bucket-lifecycle-configuration --bucket $BUCKET_NAME --lifecycle-configuration "file://lifecycle_policy.json"
```
Note that you don't need to add the part `/buckets/$BUCKET_NAME` in the configurations "Filter.Prefix" (contrary to using `fs.configure`, this is taken care of for you in the S3 API.
## Does SeaweedFS support S3 object versioning?
Yes, SeaweedFS supports S3 object versioning. You can enable versioning on a bucket using the `PutBucketVersioning` API. When enabled, SeaweedFS will store multiple versions of an object in the same bucket, providing data protection against accidental deletion or modification.
Key features supported:
- Enable/suspend versioning on buckets
- List all versions of objects
- Get, copy, and delete specific versions
- Delete markers for soft deletion
For detailed documentation and examples, see [[Amazon S3 API#s3-object-versioning]].
## How does versioning affect storage usage?
When versioning is enabled, each uploaded object creates a new version instead of overwriting the existing one. This means:
- Storage usage will increase as you accumulate versions
- All versions are preserved until explicitly deleted
- Delete operations create delete markers (soft delete) rather than immediately removing data
To manage storage growth, you should:
- Monitor storage usage regularly
- Implement lifecycle policies to automatically clean up old versions
- Use version-specific deletions for permanent removal when needed
## Does SeaweedFS support encrypted range requests?
Yes. Range requests work just fine with encrypted objects across all SSE modes:
- **SSE-KMS**: Supported
- **SSE-C**: Supported
- **SSE-S3**: Supported
## Does SeaweedFS support bucket default encryption?
Yes. You can set a bucket-level default encryption policy using the standard S3 bucket encryption API. Uploads without explicit encryption headers will follow the bucket policy. This applies to SSE-KMS and SSE-S3.
For setup guides, see [[Server-Side-Encryption]].
## Does SeaweedFS support S3 Object Lock?
Yes! SeaweedFS provides comprehensive support for S3 Object Lock features, including:
### Object Lock Features
- **Governance Mode**: Objects can be deleted/modified by users with `s3:BypassGovernanceRetention` permission
- **Compliance Mode**: Objects cannot be deleted/modified by any user until retention expires
- **Legal Hold**: Objects cannot be deleted/modified until legal hold is explicitly removed
### Supported APIs
- `GetObjectLockConfiguration` / `PutObjectLockConfiguration` (bucket-level)
- `GetObjectRetention` / `PutObjectRetention` (object-level)
- `GetObjectLegalHold` / `PutObjectLegalHold` (object-level)
- Governance bypass via `x-amz-bypass-governance-retention` header
### Requirements
- Object Lock must be enabled when creating the bucket (cannot be added later)
- Versioning is automatically enabled and required for Object Lock
- Compatible with standard AWS S3 tools and SDKs
For complete documentation, examples, and best practices, see [[S3 Object Lock and Retention]].
## What's the difference between Governance and Compliance modes?
**Governance Mode**:
- Designed for internal governance and compliance requirements
- Can be bypassed by users with proper permissions (`s3:BypassGovernanceRetention`)
- Admin users can always bypass governance retention
- Suitable for testing and development environments
**Compliance Mode**:
- Designed for regulatory compliance (SEC, FINRA, etc.)
- Cannot be bypassed by any user, including root/admin
- Provides the highest level of data protection
- Suitable for production environments with strict compliance requirements
Both modes prevent accidental deletion and provide audit trails for compliance purposes.
## S3 authentication fails when using reverse proxy
### Symptom
When accessing SeaweedFS S3 API through a reverse proxy, you might encounter signature verification errors such as:
- `SignatureDoesNotMatch` errors
- Authentication failures for presigned URLs
- Inconsistent behavior between direct access and proxied access
### Common Causes and Solutions
**1. Missing X-Forwarded-Host header**
The reverse proxy must set the `X-Forwarded-Host` header to preserve the original host information for signature calculation.
```nginx
proxy_set_header X-Forwarded-Host $host;
```
**2. URL path prefix stripping without X-Forwarded-Prefix**
If your reverse proxy strips URL prefixes (e.g., `/s3/bucket/object` → `/bucket/object`), you must set the `X-Forwarded-Prefix` header:
```nginx
# For /s3/ subpath
location /s3/ {
proxy_set_header X-Forwarded-Prefix /s3;
rewrite ^/s3/(.*) /$1 break;
proxy_pass http://seaweedfs;
}
```
**3. Request buffering enabled**
Nginx request buffering can interfere with chunked transfer encoding and signature verification:
```nginx
proxy_request_buffering off;
proxy_buffering off;
```
**4. Missing or incorrect forwarded headers**
Ensure all necessary headers are forwarded:
```nginx
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $remote_addr;
```
**Note**: SeaweedFS automatically combines `X-Forwarded-Host` and `X-Forwarded-Port` for signature verification, omitting standard ports (80 for HTTP, 443 for HTTPS).
### Testing Your Configuration
You can test your reverse proxy configuration using AWS CLI:
```bash
# Test basic bucket listing
aws s3 ls --endpoint-url https://yourdomain.com/s3
# Test presigned URL generation and access
aws s3 presign s3://test-bucket/test-object --endpoint-url https://yourdomain.com/s3
```
For detailed configuration examples, see the [[S3-Nginx-Proxy]] documentation.
## TLS error: "client sent an HTTP request to an HTTPS server"
SeaweedFS has two separate communication layers:
- **gRPC (Control Plane)**: Metadata operations - configured via `[grpc.*]`
- **HTTP/HTTPS (Data Plane)**: File data uploads/downloads - configured via `[https.*]`
If you see this error when S3 API communicates with a TLS-enabled Filer, enable HTTPS client mode:
```toml
[https.filer]
cert = "/path/to/filer.crt"
key = "/path/to/filer.key"
ca = "/path/to/ca.crt"
[https.client]
enabled = true
cert = "/path/to/client.crt"
key = "/path/to/client.key"
ca = "/path/to/ca.crt"
```
The `[https.filer]` makes the Filer accept HTTPS, while `[https.client]` with `enabled = true` makes clients (S3 API, mount, etc.) use HTTPS for data operations. See [[Security-Overview]] for architecture details and [[Security-Configuration]] for configuration reference.
S3-CORS.md
+431
@@ -0,0 +1,431 @@
# S3 Cross-Origin Resource Sharing (CORS)
SeaweedFS supports S3-compatible Cross-Origin Resource Sharing (CORS) configuration, allowing web applications to make cross-origin requests to your S3 buckets. CORS is essential for web applications that need to access resources from different domains.
## Overview
CORS defines a way for web applications running at one domain to access resources at another domain. When a web application tries to access your S3 bucket from a different domain, the browser will first send a preflight OPTIONS request to check if the cross-origin request is allowed.
SeaweedFS handles CORS through:
- **Global CORS configuration**: Server-wide default CORS settings
- **Bucket-level CORS configuration**: Each bucket can have its own CORS rules
- **Persistent storage**: CORS configurations are stored in bucket metadata
- **Automatic header handling**: CORS middleware automatically applies appropriate headers
- **Preflight request support**: Proper handling of OPTIONS requests
## Quick Start: Global CORS Configuration
The simplest way to enable CORS for all buckets is using the `-s3.allowedOrigins` parameter:
```bash
# Allow all origins (useful for development)
weed server -s3 -s3.allowedOrigins=*
# Allow specific origins
weed server -s3 -s3.allowedOrigins=https://app.example.com,https://admin.example.com
# Docker Compose example
services:
seaweedfs:
image: chrislusf/seaweedfs:latest
command: "server -s3 -s3.allowedOrigins=*"
```
This global configuration:
- Works immediately without additional setup
- Applies to all buckets by default
- Can be overridden by bucket-level CORS configuration
- Supports GET, PUT, POST, DELETE, and HEAD methods
- Allows all headers (*)
### CORS Configuration Priority
SeaweedFS uses the following priority order:
1. **Bucket-level CORS** (if configured via `aws s3api put-bucket-cors`) - highest priority
2. **Global CORS** (from `-s3.allowedOrigins` parameter) - fallback if no bucket config
3. **No CORS** (if neither is configured) - no CORS headers applied
This means you can set a permissive global default and override it with stricter rules for specific buckets.
## Advanced: Bucket-Level CORS Configuration
### Basic CORS Rule Structure
A CORS configuration consists of one or more CORS rules. Each rule defines:
```xml
<CORSConfiguration>
<CORSRule>
<ID>rule-id</ID>
<AllowedOrigin>http://example.com</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedHeader>Content-Type</AllowedHeader>
<ExposeHeader>ETag</ExposeHeader>
<MaxAgeSeconds>3600</MaxAgeSeconds>
</CORSRule>
</CORSConfiguration>
```
### CORS Rule Elements
- **ID** (optional): A unique identifier for the rule
- **AllowedOrigin** (required): Specifies which origins are allowed to access the bucket
- **AllowedMethod** (required): HTTP methods that are allowed (GET, PUT, POST, DELETE, HEAD)
- **Note**: Do NOT include `OPTIONS` in AllowedMethods - it is automatically handled for preflight requests
- **AllowedHeader** (optional): Headers that are allowed in the actual request
- **ExposeHeader** (optional): Headers that browsers can access from the response
- **MaxAgeSeconds** (optional): How long browsers can cache the preflight response
## Managing CORS Configuration
### Set CORS Configuration
Use the `PutBucketCors` API to set CORS configuration for a bucket:
```bash
aws s3api put-bucket-cors \
--bucket my-bucket \
--cors-configuration file://cors-config.json
```
Example `cors-config.json`:
```json
{
"CORSRules": [
{
"ID": "allow-all-origins",
"AllowedOrigins": ["*"],
"AllowedMethods": ["GET", "POST", "PUT", "DELETE", "HEAD"],
"AllowedHeaders": ["*"],
"ExposeHeaders": ["ETag", "x-amz-version-id"],
"MaxAgeSeconds": 3600
}
]
}
```
### Get CORS Configuration
Retrieve the current CORS configuration for a bucket:
```bash
aws s3api get-bucket-cors --bucket my-bucket
```
Response:
```json
{
"CORSRules": [
{
"ID": "allow-all-origins",
"AllowedOrigins": ["*"],
"AllowedMethods": ["GET", "POST", "PUT", "DELETE", "HEAD"],
"AllowedHeaders": ["*"],
"ExposeHeaders": ["ETag", "x-amz-version-id"],
"MaxAgeSeconds": 3600
}
]
}
```
### Delete CORS Configuration
Remove CORS configuration from a bucket:
```bash
aws s3api delete-bucket-cors --bucket my-bucket
```
## CORS Rule Examples
### Example 1: Allow Specific Domain
```json
{
"CORSRules": [
{
"ID": "allow-example-domain",
"AllowedOrigins": ["https://example.com"],
"AllowedMethods": ["GET", "POST"],
"AllowedHeaders": ["Content-Type", "Authorization"],
"ExposeHeaders": ["ETag"],
"MaxAgeSeconds": 1800
}
]
}
```
### Example 2: Allow Multiple Domains
```json
{
"CORSRules": [
{
"ID": "allow-multiple-domains",
"AllowedOrigins": [
"https://app.example.com",
"https://staging.example.com",
"https://localhost:3000"
],
"AllowedMethods": ["GET", "PUT", "POST", "DELETE"],
"AllowedHeaders": ["*"],
"ExposeHeaders": ["ETag", "x-amz-version-id"],
"MaxAgeSeconds": 3600
}
]
}
```
### Example 3: Wildcard Domain Support
```json
{
"CORSRules": [
{
"ID": "allow-subdomain-wildcard",
"AllowedOrigins": ["https://*.example.com"],
"AllowedMethods": ["GET", "POST"],
"AllowedHeaders": ["Content-Type"],
"MaxAgeSeconds": 1800
}
]
}
```
### Example 4: Multiple Rules
```json
{
"CORSRules": [
{
"ID": "read-only-rule",
"AllowedOrigins": ["*"],
"AllowedMethods": ["GET", "HEAD"],
"AllowedHeaders": ["*"],
"MaxAgeSeconds": 3600
},
{
"ID": "write-rule",
"AllowedOrigins": ["https://admin.example.com"],
"AllowedMethods": ["PUT", "POST", "DELETE"],
"AllowedHeaders": ["Content-Type", "Authorization"],
"ExposeHeaders": ["ETag"],
"MaxAgeSeconds": 1800
}
]
}
```
## Web Application Integration
### JavaScript Example
```javascript
// Configure your S3 client
const AWS = require('aws-sdk');
const s3 = new AWS.S3({
accessKeyId: 'your-access-key',
secretAccessKey: 'your-secret-key',
endpoint: 'http://localhost:8333',
s3ForcePathStyle: true,
region: 'us-east-1'
});
// Upload file from web application
const uploadFile = async (file, bucket, key) => {
const params = {
Bucket: bucket,
Key: key,
Body: file,
ContentType: file.type
};
try {
const result = await s3.upload(params).promise();
console.log('Upload successful:', result);
return result;
} catch (error) {
console.error('Upload failed:', error);
throw error;
}
};
// Download file from web application
const downloadFile = async (bucket, key) => {
const params = {
Bucket: bucket,
Key: key
};
try {
const result = await s3.getObject(params).promise();
return result.Body;
} catch (error) {
console.error('Download failed:', error);
throw error;
}
};
```
### HTML Upload Form Example
```html
<!DOCTYPE html>
<html>
<head>
<title>S3 Upload Example</title>
</head>
<body>
<input type="file" id="fileInput" />
<button onclick="uploadFile()">Upload</button>
<script>
async function uploadFile() {
const fileInput = document.getElementById('fileInput');
const file = fileInput.files[0];
if (!file) {
alert('Please select a file');
return;
}
const formData = new FormData();
formData.append('file', file);
try {
const response = await fetch('http://localhost:8333/my-bucket/' + file.name, {
method: 'PUT',
body: file,
headers: {
'Content-Type': file.type
}
});
if (response.ok) {
alert('Upload successful!');
} else {
alert('Upload failed: ' + response.statusText);
}
} catch (error) {
alert('Upload error: ' + error.message);
}
}
</script>
</body>
</html>
```
## CORS Rule Evaluation
SeaweedFS evaluates CORS rules in the following order:
1. **Rule Matching**: The first rule that matches the request origin is used
2. **Origin Validation**: Check if the request origin matches any `AllowedOrigin`
3. **Method Validation**: For preflight requests, validate the requested method
4. **Header Validation**: For preflight requests, validate all requested headers
5. **Response Building**: Build appropriate CORS headers based on the matched rule
### Preflight Request Handling
For preflight requests (OPTIONS method), SeaweedFS:
1. Checks if the origin matches an allowed origin
2. Validates the requested method against allowed methods
3. Validates all requested headers against allowed headers
4. Returns appropriate CORS headers if all validations pass
5. Returns 403 Forbidden if any validation fails
### Actual Request Handling
For actual requests (GET, POST, PUT, DELETE, etc.), SeaweedFS:
1. Checks if the origin matches an allowed origin
2. Validates the request method against allowed methods
3. Applies appropriate CORS headers to the response
4. Continues with normal request processing
## Performance and Caching
### CORS Configuration Caching
- CORS configurations are cached in memory for 5 minutes
- Cache is automatically invalidated when configuration changes
- Multiple S3 nodes share the same cached configuration
### Browser Caching
- Use `MaxAgeSeconds` to control how long browsers cache preflight responses
- Longer cache times reduce preflight requests but delay configuration changes
- Recommended values: 1800-3600 seconds (30-60 minutes)
## Security Considerations
### Origin Validation
- Never use `*` for `AllowedOrigin` in production unless absolutely necessary
- Specify exact domains or use specific wildcard patterns
- Validate all origins against your application's requirements
### Method Restrictions
- Only allow necessary HTTP methods
- Restrict write operations (PUT, POST, DELETE) to trusted origins
- Consider separate rules for read-only vs. write operations
### Header Security
- Avoid using `*` for `AllowedHeaders` in production
- Only allow headers that your application actually needs
- Be cautious with authorization headers
## Troubleshooting
### Common Issues
1. **Invalid HTTP method: OPTIONS**: Do NOT include `OPTIONS` in `AllowedMethods` - it is automatically handled for CORS preflight requests. Only use: GET, PUT, POST, DELETE, HEAD
2. **CORS policy error**: Check that your origin is listed in `AllowedOrigins`
3. **Method not allowed**: Ensure the HTTP method is in `AllowedMethods`
4. **Header blocked**: Add required headers to `AllowedHeaders`
5. **Preflight failure**: Verify all preflight requirements are met
### Debugging Tips
- Use browser developer tools to inspect CORS headers
- Check server logs for CORS-related errors
- Test with simple requests first, then add complexity
- Verify bucket-level CORS configuration is correct
### Testing CORS Configuration
```bash
# Test preflight request
curl -X OPTIONS \
-H "Origin: https://example.com" \
-H "Access-Control-Request-Method: GET" \
-H "Access-Control-Request-Headers: Content-Type" \
http://localhost:8333/my-bucket/test-object
# Test actual request
curl -X GET \
-H "Origin: https://example.com" \
http://localhost:8333/my-bucket/test-object
```
## Limitations
- Maximum 100 CORS rules per bucket
- Wildcard support is limited to `*` character
- Complex regex patterns in origins are not supported
- CORS configuration is per-bucket, not per-object
## Best Practices
1. **Specify exact origins** instead of using wildcards when possible
2. **Use appropriate MaxAgeSeconds** to balance performance and flexibility
3. **Implement proper error handling** in your web applications
4. **Test CORS configuration thoroughly** before deploying to production
5. **Monitor CORS usage** and adjust rules as needed
6. **Keep CORS rules simple** and well-documented
7. **Use separate rules** for different access patterns (read vs. write)
S3-Conditional-Operations.md
+84
@@ -0,0 +1,84 @@
# S3 Conditional Operations
SeaweedFS supports AWS S3-compatible conditional headers for safe concurrent access patterns, optimistic locking, and efficient conditional operations.
## Supported Conditional Headers
| Header | Applies To | Condition | Use Case |
|--------|------------|-----------|----------|
| **If-Match** | GET, PUT, COPY | ETag matches | Ensure object hasn't changed |
| **If-None-Match** | GET, PUT, COPY | ETag doesn't match | Prevent overwrites, caching |
| **If-Modified-Since** | GET, COPY | Modified after date | Conditional downloads |
| **If-Unmodified-Since** | GET, PUT, COPY | Not modified after date | Safe modifications |
## HTTP Examples
### Conditional GET (Caching)
```bash
# First, get the object and note its ETag
curl -I "http://localhost:8333/mybucket/myfile.txt"
# Response: ETag: "d41d8cd98f00b204e9800998ecf8427e"
# Conditional GET - only download if object changed
curl -H "If-None-Match: \"d41d8cd98f00b204e9800998ecf8427e\"" \
"http://localhost:8333/mybucket/myfile.txt"
```
### Conditional PUT (Optimistic Locking)
```bash
# Safe update - only modify if object hasn't changed
curl -X PUT -H "If-Match: \"d41d8cd98f00b204e9800998ecf8427e\"" \
-H "Content-Type: text/plain" \
--data "Updated content" \
"http://localhost:8333/mybucket/myfile.txt"
# Prevent overwrites - only create if object doesn't exist
curl -X PUT -H "If-None-Match: *" \
-H "Content-Type: text/plain" \
--data "New file content" \
"http://localhost:8333/mybucket/newfile.txt"
```
### If-Modified-Since
```bash
# Only download if modified after specific date
curl -H "If-Modified-Since: Wed, 15 Jan 2024 10:00:00 GMT" \
"http://localhost:8333/mybucket/log-file.txt"
```
### If-Unmodified-Since
```bash
# Update only if not modified since last read
curl -X PUT \
-H "If-Unmodified-Since: Wed, 15 Jan 2024 10:00:00 GMT" \
-H "Content-Type: text/plain" \
--data "Updated content" \
"http://localhost:8333/mybucket/document.txt"
```
### Copy Operations
```bash
# Copy only if source object hasn't changed
curl -X PUT \
-H "x-amz-copy-source: /source-bucket/source-file.txt" \
-H "x-amz-copy-source-if-match: \"source-etag\"" \
"http://localhost:8333/dest-bucket/dest-file.txt"
```
## HTTP Status Codes
| Status Code | Condition | Meaning |
|-------------|-----------|---------|
| **200 OK** | Condition met | Operation succeeded |
| **304 Not Modified** | If-None-Match matched | Object unchanged (GET only) |
| **412 Precondition Failed** | Condition not met | Operation blocked by condition |
## Related Documentation
- [Amazon S3 API](Amazon-S3-API.md)
- [Server-Side Encryption](Server-Side-Encryption.md)
S3-Credentials.md
+445
@@ -0,0 +1,445 @@
# S3 Credentials
SeaweedFS S3 API supports multiple authentication methods with a clear priority system. This page explains how to configure S3 credentials for your SeaweedFS setup.
## Authentication Methods
### 1. Configuration File (Highest Priority)
Create a JSON configuration file and use the `-config` option:
```json
{
"identities": [
{
"name": "admin_user",
"credentials": [
{
"accessKey": "admin_access_key",
"secretKey": "admin_secret_key"
}
],
"actions": ["Admin", "Read", "Write"]
},
{
"name": "read_only_user",
"credentials": [
{
"accessKey": "readonly_access_key",
"secretKey": "readonly_secret_key"
}
],
"actions": ["Read"]
}
]
}
```
Start S3 server with config file:
```bash
weed s3 -config=/path/to/s3.json -filer=localhost:8888
```
### 2. Filer Configuration (Medium Priority)
Store configuration in the filer using the credential manager. This allows dynamic configuration updates without restarting the S3 server.
### 3. Admin UI (Web Interface)
Use the SeaweedFS Admin UI to manage S3 credentials through a web interface:
```bash
# Start the admin interface (separate from filer)
weed admin -masters=localhost:9333
# Access the admin UI (default port 23646)
http://localhost:23646
```
Navigate to **Object Store → Users** (`/object-store/users`) to:
- **Create Users**: Add new S3 users with email and permissions
- **Edit Permissions**: Modify existing user access levels
- **Manage Access Keys**: Generate and delete access key pairs
- **View User Details**: Check user activity and current permissions
The Admin UI stores credentials in the filer using the same filer configuration method, so changes are automatically synchronized across all S3 servers connected to the same filer.
### 4. Environment Variables (Fallback)
Use AWS standard environment variables as a fallback when no other configuration is available:
```bash
export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
weed s3 -filer=localhost:8888
```
**Important**: Environment variables are only used when:
- No `-config` option is provided
- No configuration is available from the filer
- Both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are set
## Priority System
SeaweedFS uses the following priority order for S3 credentials:
1. **Configuration File** (if `-config` option is provided)
2. **Filer Configuration** (if available and no config file)
3. **Admin UI** (web interface that stores in filer configuration)
4. **Environment Variables** (fallback only)
Higher priority methods completely override lower priority methods - there is no merging or supplementing.
**Important**: Admin UI and Filer Configuration both use the same underlying storage (filer), so they have the same effective priority. The Admin UI provides a user-friendly interface for managing what is stored as filer configuration.
## Configuration Examples
### Production Setup
```bash
# Use dedicated configuration file
weed s3 -config=/etc/seaweedfs/s3.json -filer=filer1:8888,filer2:8888
```
### Development Setup
```bash
# Use environment variables for quick setup
export AWS_ACCESS_KEY_ID=dev_access_key
export AWS_SECRET_ACCESS_KEY=dev_secret_key
weed s3 -filer=localhost:8888
```
### Docker Compose
```yaml
version: '3.9'
services:
s3:
image: chrislusf/seaweedfs:latest
ports:
- 8333:8333
environment:
AWS_ACCESS_KEY_ID: s3admin
AWS_SECRET_ACCESS_KEY: s3secret
entrypoint: weed
command: s3 -filer=filer:8888
depends_on:
- filer
```
## Credential Features
### Actions
Identities can have different permission levels:
- `Admin`: Full access to all S3 operations
- `Read`: Read-only access
- `Write`: Read and write access
- `Read_ACP`: Read access control permissions
- `Write_ACP`: Write access control permissions
### Multiple Credentials
Each identity can have multiple access key/secret key pairs:
```json
{
"name": "multi_key_user",
"credentials": [
{
"accessKey": "key1",
"secretKey": "secret1"
},
{
"accessKey": "key2",
"secretKey": "secret2"
}
],
"actions": ["Read", "Write"]
}
```
### Account Management
Identities can be associated with accounts for better organization and cross-account access control.
## Bucket-Specific Permissions
SeaweedFS supports restricting user access to specific buckets using bucket-scoped actions. This allows you to create users who have full access to one bucket but no access to other buckets.
### Single Bucket Full Access
To create a user with full access to only one specific bucket, use bucket-scoped actions:
```json
{
"identities": [
{
"name": "bucket1_user",
"credentials": [
{
"accessKey": "bucket1_access_key",
"secretKey": "bucket1_secret_key"
}
],
"actions": [
"Read:mybucket",
"Write:mybucket",
"List:mybucket",
"Tagging:mybucket",
"Admin:mybucket"
]
}
]
}
```
This user can:
- ✅ Read, write, list, and tag objects in `mybucket`
- ✅ Create and delete objects in `mybucket`
- ✅ Manage bucket settings for `mybucket`
- ❌ Access any other buckets
- ❌ Create new buckets (requires global Admin action)
### Bucket-Specific Actions
Actions can be scoped to specific buckets using the format `Action:BucketName`:
| Action Format | Description | Example |
|--------------|-------------|---------|
| `Read:bucket1` | Read access to bucket1 only | Get objects from bucket1 |
| `Write:bucket1` | Write access to bucket1 only | Put/delete objects in bucket1 |
| `List:bucket1` | List access to bucket1 only | List objects in bucket1 |
| `Admin:bucket1` | Admin access to bucket1 only | Bucket management for bucket1 |
| `Tagging:bucket1` | Tagging access to bucket1 only | Manage object tags in bucket1 |
### Multiple Bucket Access
Users can have access to multiple specific buckets:
```json
{
"name": "multi_bucket_user",
"credentials": [{"accessKey": "key", "secretKey": "secret"}],
"actions": [
"Read:bucket1",
"Write:bucket1",
"List:bucket1",
"Read:bucket2",
"List:bucket2"
]
}
```
This user has:
- Full read/write access to `bucket1`
- Read-only access to `bucket2`
- No access to any other buckets
### Wildcard Support
SeaweedFS supports wildcard patterns for flexible bucket access:
```json
{
"name": "prefix_user",
"credentials": [{"accessKey": "key", "secretKey": "secret"}],
"actions": [
"Read:user-*",
"Write:user-*",
"List:user-*"
]
}
```
This user can access all buckets starting with `user-` (like `user-data`, `user-logs`, etc.).
### Object-Level Permissions
You can restrict access to specific paths within a bucket:
```json
{
"name": "path_limited_user",
"credentials": [{"accessKey": "key", "secretKey": "secret"}],
"actions": [
"Read:mybucket/uploads/*",
"Write:mybucket/uploads/*",
"List:mybucket"
]
}
```
This user can:
- Only read/write objects under `mybucket/uploads/` path
- List the bucket to see the directory structure
- Cannot access objects in other paths within the bucket
### Configuration Methods
Bucket-specific permissions work with all configuration methods:
#### Dynamic Configuration (weed shell)
```bash
# Create user with access to specific bucket
s3.configure -access_key=bucket1user -secret_key=bucket1pass -buckets=mybucket -user=bucket1_user -actions=Read,Write,List,Tagging,Admin -apply
```
#### Static Configuration File
Use the JSON examples shown above in your configuration file.
#### Admin UI
1. Navigate to **Object Store → Users**
2. Create a new user
3. In the permissions section, specify bucket-scoped actions like `Read:mybucket`
#### Environment Variables
Environment variables create global admin access and cannot be scoped to specific buckets.
### Best Practices
1. **Principle of Least Privilege**: Grant only the minimum permissions needed
2. **Use Specific Bucket Names**: Avoid wildcards unless necessary for flexibility
3. **Separate Users for Different Buckets**: Create dedicated users for each bucket or application
4. **Test Permissions**: Verify users can only access intended buckets
5. **Monitor Access**: Use audit logs to track bucket access patterns
### Troubleshooting
**User can access other buckets:**
- Verify no global actions (`Read`, `Write`, `Admin`) are granted
- Check for wildcard patterns that might be too broad
- Ensure bucket names in actions match exactly
**User cannot access intended bucket:**
- Verify bucket name spelling in actions
- Check that all required actions are granted (e.g., `List` for listing objects)
- Test with AWS CLI: `aws --endpoint-url=http://localhost:8333 s3 ls s3://mybucket`
## Anonymous Access
By default, if no credentials are configured, SeaweedFS allows anonymous access to all S3 operations. To enable authentication:
1. Configure at least one identity using any of the methods above
2. Authentication will be automatically enabled
3. All requests will require valid credentials
## Configuration Reloading
SeaweedFS supports different reloading mechanisms depending on which authentication method you use:
| Configuration Method | Auto Reload | Manual Reload | Live Reload |
|---------------------|-------------|---------------|-------------|
| **Configuration File** (`-config` option) | ❌ No | ✅ SIGHUP | ❌ No |
| **Filer Configuration** (credential manager) | ✅ Yes | ✅ Yes | ✅ Yes |
| **Admin UI** (web interface) | ✅ Yes | ✅ Yes | ✅ Yes |
| **Environment Variables** | ❌ No | ❌ No | ❌ No |
### Static Configuration Files
When using the `-config` option, you can reload the configuration by sending a SIGHUP signal:
```bash
# Find the S3 server process ID
ps aux | grep "weed s3"
# Send SIGHUP signal to reload configuration
kill -HUP <seaweedfs_s3_pid>
# Or if using systemd
systemctl reload seaweedfs-s3
```
The server will log the reload:
```
I0723 12:34:56.789 s3api_server.go:98] Loaded 3 identities from config file /etc/seaweedfs/s3.json
```
### Filer-based Configuration
Filer-based configurations automatically reload when changes are detected:
```bash
# Changes are automatically applied
weed shell
> s3.configure -user=newuser -access_key=key123 -secret_key=secret123 -actions=Admin -apply
```
The server will automatically detect and apply changes:
```
I0723 12:35:12.456 auth_credentials_subscribe.go:55] updated /etc/seaweedfs/iam/identity.json
```
### Admin UI Configuration
Admin UI changes are automatically applied in real-time since they use the same filer-based storage:
1. **Access Admin UI**: Navigate to `http://localhost:23646`
2. **Go to Users**: Click **Object Store → Users**
3. **Make Changes**: Create, edit, or delete users through the web interface
4. **Automatic Sync**: Changes are immediately applied to all connected S3 servers
The server will show the same automatic detection messages as filer-based configuration since they share the same underlying storage mechanism.
### Environment Variables
Environment variable changes require a complete restart of the S3 server:
```bash
# Update environment variables
export AWS_ACCESS_KEY_ID=new_access_key
export AWS_SECRET_ACCESS_KEY=new_secret_key
# Restart the S3 server
systemctl restart seaweedfs-s3
```
### Verifying Configuration Reloads
Monitor the logs to verify configuration updates:
```bash
# Watch for reload messages
tail -f /var/log/seaweedfs/s3.log | grep -E "updated|Loaded.*identities"
# Check current configuration via shell
weed shell
> s3.configure
```
## Troubleshooting
### Common Issues
**Environment variables not working:**
- Check that no `-config` option is provided
- Verify no configuration exists in the filer
- Ensure both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are set
**Configuration file not loading:**
- Verify the file path is correct
- Check JSON syntax is valid
- Ensure the file is readable by the SeaweedFS process
**Invalid credentials error:**
- Verify access key and secret key are correct
- Check that the identity has the required actions/permissions
- Ensure the credential store is properly configured
### Debug Commands
Check current configuration:
```bash
# View current identities (if using filer store)
weed shell
> s3.configure -list
```
Test credentials:
```bash
# Test with AWS CLI
aws --endpoint-url=http://localhost:8333 s3 ls
```
## Security Best Practices
1. **Use Configuration Files in Production**: Environment variables are visible in process lists
2. **Rotate Credentials Regularly**: Update access keys and secret keys periodically
3. **Principle of Least Privilege**: Grant only the minimum required permissions
4. **Secure Storage**: Store configuration files with appropriate file permissions
5. **Monitor Access**: Enable audit logging to track S3 API usage
S3-Nginx-Proxy.md
+106 -2
@@ -2,10 +2,54 @@ It's a common concept to put a proxy in front of S3 that handles requests. Nginx
For virtual-hosted style URL buckets, you'll need to add a [wildcard DNS record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) for your S3 subdomain.
Make sure the config sets the `X-Forwarded-Host` and optionally the `X-Forwarded-Port` if you are using a non-standard port. SeaweedFS will automatically combine these headers to reconstruct the correct host information for signature verification.
## Reverse Proxy with URL Path Prefixes
SeaweedFS S3 API supports the `X-Forwarded-Prefix` header for scenarios where a reverse proxy strips URL path prefixes before forwarding requests. This is common when hosting the S3 API under a subpath like `/s3/` or `/api/s3/`.
### How X-Forwarded-Prefix Works
When a reverse proxy strips a URL prefix:
1. **Client request**: `https://example.com/s3/my-bucket/my-object`
2. **Proxy strips prefix** and forwards: `https://backend:8333/my-bucket/my-object`
3. **Proxy adds header**: `X-Forwarded-Prefix: /s3`
SeaweedFS will:
1. First attempt signature verification using the **original path** (`/s3/my-bucket/my-object`)
2. Fall back to verification using the **stripped path** (`/my-bucket/my-object`) if the first attempt fails
This ensures both regular S3 requests and presigned URLs work correctly with reverse proxies that strip prefixes.
### Example Use Cases
- **API Gateway**: `/api/s3/bucket/object` → `/bucket/object`
- **Multi-tenant setup**: `/tenant1/s3/bucket/object` → `/bucket/object`
- **Subpath hosting**: `/storage/s3/bucket/object` → `/bucket/object`
### Important Notes
- The `X-Forwarded-Prefix` header should contain the stripped prefix (e.g., `/s3`)
- `X-Forwarded-Port` is automatically combined with `X-Forwarded-Host` for non-standard ports
- Standard ports (80 for HTTP, 443 for HTTPS) are omitted from the host header automatically
- Both regular S3 authentication and presigned URLs are supported
- This feature works with all S3 operations that require signature verification
Additionally, make sure that `proxy_request_buffering` is `off` (default is `on`), as the proxy will buffer the request, and send the request to the backend as a whole instead of chunked, and again the signature computed by the client side will be different as it would have taken into account the `Transfer-Encoding: chunked` header that is dropped by the proxy when it buffers.
### Example Nginx config
#### Standard Configuration (without URL prefix stripping)
```
upstream seaweedfs { server localhost:8333 fail_timeout=0; keepalive 20;}
upstream seaweedfs {
# Hash on uploadId query string in the GET request create consistency for multipart uploads,
# only necessary when using local embedded filer store (leveldb)
hash $arg_uploadId consistent;
server localhost:8333 fail_timeout=0;
keepalive 20;
}
## Also you can use unix domain socket instead for better performance:
# upstream seaweedfs { server unix:/tmp/seaweedfs-s3-8333.sock; keepalive 20;}
@@ -17,7 +61,7 @@ server {
# The regex will support path style as well as virtual-hosted style bucket URLs
# path style: http://s3.yourdomain.com/mybucket
# virtual-hosted style: http://mybucket.s3.yourdomain.com
server_name ~^(?:(?<bucket>[^.]+)\.)s3\.yourdomain\.com;
server_name ~^(?:(?<bucket>[^.]+)\.)?s3\.yourdomain\.com;
ignore_invalid_headers off;
client_max_body_size 0;
@@ -26,11 +70,15 @@ server {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_request_buffering off;
chunked_transfer_encoding off;
# If bucket subdomain is not empty,
@@ -43,6 +91,62 @@ server {
proxy_pass http://seaweedfs;
}
ssl on;
ssl_certificate /{path_to_ssl_cert}/cert.pem;
ssl_certificate_key /{path_to_ssl_cert}/key.pem;
}
```
#### Configuration with URL Prefix Stripping (X-Forwarded-Prefix)
For scenarios where you need to host SeaweedFS S3 API under a subpath:
```
upstream seaweedfs {
hash $arg_uploadId consistent;
server localhost:8333 fail_timeout=0;
keepalive 20;
}
server {
listen 443 ssl;
server_name yourdomain.com;
ignore_invalid_headers off;
client_max_body_size 0;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_request_buffering off;
chunked_transfer_encoding off;
# S3 API under /s3/ subpath
location /s3/ {
# Set the X-Forwarded-Prefix header to the stripped prefix
proxy_set_header X-Forwarded-Prefix /s3;
# Strip the /s3 prefix before forwarding to backend
rewrite ^/s3/(.*) /$1 break;
proxy_pass http://seaweedfs;
}
# Alternative: S3 API under /api/s3/ subpath
location /api/s3/ {
proxy_set_header X-Forwarded-Prefix /api/s3;
rewrite ^/api/s3/(.*) /$1 break;
proxy_pass http://seaweedfs;
}
ssl on;
ssl_certificate /{path_to_ssl_cert}/cert.pem;
ssl_certificate_key /{path_to_ssl_cert}/key.pem;
S3-Object-Lock-and-Retention.md
+424
@@ -0,0 +1,424 @@
# S3 Object Lock and Retention
SeaweedFS provides comprehensive support for Amazon S3 Object Lock functionality, including object versioning, retention policies, legal holds, and WORM (Write Once Read Many) compliance features.
## Overview
Object Lock is a feature that allows you to store objects using a WORM (Write Once Read Many) model. It helps prevent objects from being deleted or overwritten for a fixed amount of time or indefinitely. Object Lock works in conjunction with versioning and is enabled at the bucket level.
## Prerequisites
- **Versioning Required**: Object Lock can only be enabled on buckets that have versioning enabled
- **Immutable Setting**: Object Lock can only be enabled when creating a bucket, not on existing buckets
- **S3 API**: All Object Lock operations are available through the S3 API
## Key Features
### 1. Retention Modes
#### Governance Mode
- Objects are protected from deletion or modification by most users
- Users with `s3:BypassGovernanceRetention` permission can override protection
- Admin users can always bypass governance retention
- Suitable for internal compliance and governance requirements
#### Compliance Mode
- Objects cannot be deleted or modified by any user, including the root user
- Retention periods cannot be shortened
- Provides the highest level of protection
- Suitable for regulatory compliance requirements
### 2. Legal Hold
- Independent protection mechanism that can be applied to any object
- When enabled, objects cannot be deleted or modified until the legal hold is removed
- Can be used alongside retention policies
- Useful for litigation, investigation, or audit requirements
## API Support
SeaweedFS implements the complete S3 Object Lock API:
### Bucket-Level Operations
- `GET /?object-lock` - Get bucket Object Lock configuration
- `PUT /?object-lock` - Set bucket Object Lock configuration
### Object-Level Operations
- `GET /{object}?retention` - Get object retention settings
- `PUT /{object}?retention` - Set object retention settings
- `GET /{object}?legal-hold` - Get object legal hold status
- `PUT /{object}?legal-hold` - Set object legal hold status
### Versioning Operations
- `GET /?versioning` - Get bucket versioning status
- `PUT /?versioning` - Set bucket versioning status
- `GET /?versions` - List object versions
## Setup and Configuration
### 1. Enable Object Lock on Bucket Creation
Object Lock must be enabled when creating a bucket:
```bash
# Create bucket with Object Lock enabled
aws s3api create-bucket \
--bucket my-secure-bucket \
--object-lock-enabled-for-bucket
# Enable versioning (automatically done with Object Lock)
aws s3api put-bucket-versioning \
--bucket my-secure-bucket \
--versioning-configuration Status=Enabled
```
### 2. Configure Governance Bypass Permissions
Grant users permission to bypass governance retention:
```json
{
"identities": [
{
"name": "compliance-admin",
"credentials": [
{
"accessKey": "admin123",
"secretKey": "secret123"
}
],
"actions": [
"Read:my-secure-bucket/*",
"Write:my-secure-bucket/*",
"BypassGovernanceRetention:my-secure-bucket/*"
]
}
]
}
```
## Usage Examples
### 1. Object Retention
#### Set Governance Retention
```bash
# Set 30-day governance retention
aws s3api put-object-retention \
--bucket my-secure-bucket \
--key important-document.pdf \
--retention '{
"Mode": "GOVERNANCE",
"RetainUntilDate": "2024-12-31T23:59:59Z"
}'
```
#### Set Compliance Retention
```bash
# Set 7-year compliance retention
aws s3api put-object-retention \
--bucket my-secure-bucket \
--key regulatory-record.json \
--retention '{
"Mode": "COMPLIANCE",
"RetainUntilDate": "2031-01-01T00:00:00Z"
}'
```
#### Get Object Retention
```bash
aws s3api get-object-retention \
--bucket my-secure-bucket \
--key important-document.pdf
```
### 2. Legal Hold
#### Apply Legal Hold
```bash
aws s3api put-object-legal-hold \
--bucket my-secure-bucket \
--key investigation-file.doc \
--legal-hold Status=ON
```
#### Remove Legal Hold
```bash
aws s3api put-object-legal-hold \
--bucket my-secure-bucket \
--key investigation-file.doc \
--legal-hold Status=OFF
```
#### Check Legal Hold Status
```bash
aws s3api get-object-legal-hold \
--bucket my-secure-bucket \
--key investigation-file.doc
```
### 3. Governance Bypass
#### Delete Object with Governance Bypass
```bash
# User with bypass permission can delete governance-protected objects
aws s3api delete-object \
--bucket my-secure-bucket \
--key document.pdf \
--bypass-governance-retention
```
#### Bulk Delete with Governance Bypass
```bash
# Delete multiple objects with governance bypass
aws s3api delete-objects \
--bucket my-secure-bucket \
--delete file://delete-objects.json \
--bypass-governance-retention
```
### 4. Object Versioning
#### List Object Versions
```bash
aws s3api list-object-versions \
--bucket my-secure-bucket \
--prefix documents/
```
#### Get Specific Version
```bash
aws s3api get-object \
--bucket my-secure-bucket \
--key document.pdf \
--version-id "3/L4kqtJlcpXroDTDmpUMLUo"
```
#### Delete Specific Version
```bash
aws s3api delete-object \
--bucket my-secure-bucket \
--key document.pdf \
--version-id "3/L4kqtJlcpXroDTDmpUMLUo"
```
## Permission System
### IAM Actions
SeaweedFS supports the following S3 IAM actions for Object Lock in identity-based permissions:
- `s3:GetObjectRetention` - Get object retention settings
- `s3:PutObjectRetention` - Set object retention settings
- `s3:GetObjectLegalHold` - Get object legal hold status
- `s3:PutObjectLegalHold` - Set object legal hold status
- `s3:BypassGovernanceRetention` - Bypass governance retention
- `s3:GetBucketObjectLockConfiguration` - Get bucket Object Lock configuration
- `s3:PutBucketObjectLockConfiguration` - Set bucket Object Lock configuration
### Identity-Based Permissions
Configure user permissions in `identities.json`:
```json
{
"identities": [
{
"name": "read-only-user",
"credentials": [{"accessKey": "ro123", "secretKey": "secret123"}],
"actions": [
"Read:my-secure-bucket/*",
"GetObjectRetention:my-secure-bucket/*",
"GetObjectLegalHold:my-secure-bucket/*"
]
},
{
"name": "compliance-admin",
"credentials": [{"accessKey": "admin123", "secretKey": "secret123"}],
"actions": [
"Read:my-secure-bucket/*",
"Write:my-secure-bucket/*",
"Admin:my-secure-bucket/*",
"BypassGovernanceRetention:my-secure-bucket/*"
]
}
]
}
```
## Error Handling
### Common Error Scenarios
1. **Object Lock Not Enabled**
- Error: `InvalidRequest`
- Solution: Enable Object Lock when creating the bucket
2. **Versioning Not Enabled**
- Error: `InvalidRequest`
- Solution: Enable versioning before configuring Object Lock
3. **Governance Bypass Not Permitted**
- Error: `AccessDenied`
- Solution: Grant `s3:BypassGovernanceRetention` permission
4. **Compliance Mode Cannot Be Bypassed**
- Error: `AccessDenied`
- Solution: Wait for retention period to expire
5. **Object Under Legal Hold**
- Error: `AccessDenied`
- Solution: Remove legal hold before deleting/modifying
### Error Response Examples
```xml
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AccessDenied</Code>
<Message>Object is under COMPLIANCE mode retention and cannot be deleted</Message>
<Resource>/my-secure-bucket/protected-file.pdf</Resource>
<RequestId>12345</RequestId>
</Error>
```
## WORM Compliance
SeaweedFS Object Lock provides WORM (Write Once Read Many) compliance:
### Features
- **Immutable Objects**: Once written, objects cannot be modified
- **Retention Enforcement**: Objects cannot be deleted before retention expiry
- **Legal Hold Support**: Additional protection for litigation/investigation
- **Audit Trail**: All access attempts are logged for compliance
### Use Cases
- **Regulatory Compliance**: Financial records, healthcare data, legal documents
- **Data Archiving**: Long-term storage with guaranteed integrity
- **Backup Protection**: Prevent accidental or malicious deletion
- **Audit Requirements**: Maintain evidence for legal proceedings
## Best Practices
### 1. Retention Planning
- **Start with Governance**: Use governance mode for testing and development
- **Compliance for Production**: Use compliance mode for regulatory requirements
- **Gradual Implementation**: Start with short retention periods and increase gradually
### 2. Permission Management
- **Principle of Least Privilege**: Grant minimum required permissions
- **Separate Roles**: Different permissions for different use cases
- **Regular Audits**: Review and update permissions regularly
### 3. Monitoring and Alerting
- **Monitor Bypass Attempts**: Track governance bypass usage
- **Retention Expiry Alerts**: Set up alerts for expiring retention periods
- **Failed Access Attempts**: Monitor and investigate access denials
### 4. Backup and Recovery
- **Multiple Copies**: Maintain backups of Object Lock configurations
- **Cross-Region Replication**: Replicate to different regions for disaster recovery
- **Version Management**: Keep track of object versions and their retention status
## Performance Considerations
### Storage Efficiency
- **Versioning Overhead**: Each version consumes storage space
- **Retention Metadata**: Minimal overhead for Object Lock metadata
- **Cleanup Strategies**: Plan for version cleanup after retention expires
### Access Patterns
- **Read-Heavy Workloads**: Object Lock is optimized for read operations
- **Write-Once Pattern**: Best suited for write-once, read-many scenarios
- **Batch Operations**: Use batch operations for bulk retention management
## Limitations
### Current Limitations
- **Bucket-Level Only**: Object Lock must be enabled during bucket creation
- **No Modification**: Existing buckets cannot be converted to Object Lock
- **Retention Limits**: Maximum 100 years retention period
- **Version Limits**: Consider version proliferation in high-write scenarios
### AWS Compatibility
SeaweedFS Object Lock is fully compatible with AWS S3 Object Lock:
- Same API endpoints and parameters
- Identical error responses
- Compatible with AWS CLI and SDKs
- Consistent behavior across governance and compliance modes
## Troubleshooting
### Common Issues
1. **Cannot Enable Object Lock**
- Check if versioning is enabled
- Ensure bucket is being created (not existing)
- Verify permissions for bucket operations
2. **Retention Settings Not Applied**
- Verify Object Lock is enabled on bucket
- Check retention date format (ISO 8601)
- Ensure retention mode is valid (GOVERNANCE/COMPLIANCE)
3. **Governance Bypass Fails**
- Verify user has `s3:BypassGovernanceRetention` permission
- Check if bypass header is included in request
- Ensure object is in GOVERNANCE mode (not COMPLIANCE)
### Debug Commands
```bash
# Check bucket Object Lock configuration
aws s3api get-object-lock-configuration --bucket my-secure-bucket
# Check bucket versioning status
aws s3api get-bucket-versioning --bucket my-secure-bucket
# Check object retention
aws s3api get-object-retention --bucket my-secure-bucket --key file.pdf
# Check legal hold status
aws s3api get-object-legal-hold --bucket my-secure-bucket --key file.pdf
```
## Migration Guide
### From Non-Object Lock Buckets
1. **Create New Bucket**: Create new bucket with Object Lock enabled
2. **Copy Data**: Use `aws s3 sync` to copy data to new bucket
3. **Apply Retention**: Set retention policies on copied objects
4. **Update Applications**: Update applications to use new bucket
5. **Cleanup**: Remove old bucket after validation
### From Other S3 Providers
Object Lock configurations can be migrated using standard S3 API calls:
```bash
# Export current configuration
aws s3api get-object-lock-configuration --bucket source-bucket > config.json
# Apply to SeaweedFS bucket
aws s3api put-object-lock-configuration \
--bucket target-bucket \
--object-lock-configuration file://config.json
```
## Related Documentation
- [S3 API FAQ](S3-API-FAQ.md)
- [Amazon S3 API](Amazon-S3-API.md)
- [Security Configuration](Security-Configuration.md)
- [Production Setup](Production-Setup.md)
## Support
For issues or questions about Object Lock features:
- GitHub Issues: [SeaweedFS Issues](https://github.com/seaweedfs/seaweedfs/issues)
- Community Forum: [SeaweedFS Discussions](https://github.com/seaweedfs/seaweedfs/discussions)
- Documentation: [SeaweedFS Wiki](https://github.com/seaweedfs/seaweedfs/wiki)
S3-Object-Versioning.md
+137
@@ -0,0 +1,137 @@
# S3 Object Versioning
SeaweedFS supports S3 object versioning, which allows you to keep multiple variants of an object in the same bucket. This provides data protection against accidental deletion or modification.
## Enable Versioning
To enable versioning on a bucket, use the `PutBucketVersioning` API:
```bash
aws s3api put-bucket-versioning \
--bucket my-bucket \
--versioning-configuration Status=Enabled
```
## Check Versioning Status
To check the versioning status of a bucket:
```bash
aws s3api get-bucket-versioning --bucket my-bucket
```
Response:
```json
{
"Status": "Enabled"
}
```
## Suspend Versioning
To suspend versioning (not disable completely):
```bash
aws s3api put-bucket-versioning \
--bucket my-bucket \
--versioning-configuration Status=Suspended
```
## List Object Versions
To list all versions of objects in a bucket:
```bash
aws s3api list-object-versions --bucket my-bucket
```
Response includes both object versions and delete markers:
```json
{
"Versions": [
{
"Key": "example.txt",
"VersionId": "v_1234567890abcdef",
"IsLatest": true,
"LastModified": "2023-12-01T10:00:00Z",
"ETag": "\"abcdef1234567890\"",
"Size": 1024
}
],
"DeleteMarkers": [
{
"Key": "deleted-file.txt",
"VersionId": "v_fedcba0987654321",
"IsLatest": true,
"LastModified": "2023-12-01T11:00:00Z"
}
]
}
```
## Access Specific Versions
### Get a specific version of an object:
```bash
aws s3api get-object \
--bucket my-bucket \
--key example.txt \
--version-id v_1234567890abcdef \
output.txt
```
### Copy a specific version:
```bash
aws s3api copy-object \
--copy-source my-bucket/example.txt?versionId=v_1234567890abcdef \
--bucket my-bucket \
--key example-copy.txt
```
### Delete a specific version:
```bash
aws s3api delete-object \
--bucket my-bucket \
--key example.txt \
--version-id v_1234567890abcdef
```
## Versioning Behavior
### When Versioning is Enabled:
- **PUT Object**: Creates a new version with a unique version ID
- **GET Object**: Returns the latest version (unless version ID is specified)
- **DELETE Object**: Creates a delete marker (soft delete)
- **DELETE Object with version ID**: Permanently deletes that specific version
### When Versioning is Suspended:
- **PUT Object**: Overwrites the object with version ID "null"
- **GET Object**: Returns the current version
- **DELETE Object**: Permanently deletes the object
## Storage Layout
SeaweedFS stores versioned objects using the following structure:
```
/buckets/my-bucket/
├── example.txt # Current version (if versioning suspended)
└── example.txt.versions/
├── v_1234567890abcdef # Version 1
├── v_fedcba0987654321 # Version 2
└── v_abcdef1234567890 # Version 3 (latest)
```
## Limitations
- **Version ID Format**: SeaweedFS generates version IDs in the format `v_<32-char-hex>`
- **Restore Operations**: Only partial support for `RestoreObject` API
- **Lifecycle Policies**: Version-specific lifecycle rules are not fully implemented
- **MFA Delete**: Not currently supported
## Best Practices
1. **Enable versioning before storing important data** to ensure all versions are captured
2. **Monitor storage usage** as versioning can increase storage consumption
3. **Implement lifecycle policies** to automatically clean up old versions
4. **Use version-specific operations** when you need to access historical data
5. **Consider performance impact** when listing many versions of objects
SQL-Queries-on-Message-Queue.md
+627
@@ -0,0 +1,627 @@
# SQL Queries on Message Queue
SeaweedFS provides a powerful SQL query engine that allows you to query Message Queue topics using standard SQL syntax. This feature enables analytics, reporting, and data exploration on your message data using familiar SQL tools and PostgreSQL-compatible clients.
## Overview
The SQL query engine bridges the gap between SeaweedFS's Message Queue and traditional SQL databases by providing:
- **PostgreSQL Wire Protocol Compatibility** - Use any PostgreSQL client, tool, or application
- **Real-time + Historical Data** - Query both live messages and archived Parquet data
- **Standard SQL Operations** - SELECT, aggregations, filtering, and schema operations
- **Multiple Interface Options** - Database server mode and interactive CLI
- **Secure Authentication** - MD5, password, and trust authentication methods
## Architecture
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ SQL Clients │ │ PostgreSQL │ │ SeaweedFS │
│ (psql, apps) │────│ Wire Protocol │────│ SQL Engine │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│
┌─────────────────┐
│ Hybrid Scanner │
│ • Live Messages │
│ • Parquet Files │
└─────────────────┘
```
## Quick Start
### 1. Start the Database Server
```bash
# Start PostgreSQL-compatible database server
weed db -auth=md5 -users='{"admin":"secret","analyst":"readonly"}'
# Or use a JSON file for credentials
echo '{"admin":"secret","analyst":"readonly"}' > users.json
weed db -auth=md5 -users="@users.json" -port=5432
```
### 2. Connect with psql
```bash
# Connect using psql
PGPASSWORD=secret psql -h localhost -p 5432 -U admin -d default
# Or with connection string
psql "postgresql://admin:secret@localhost:5432/default"
```
### 3. Start Querying
```sql
-- List available databases (MQ namespaces)
SHOW DATABASES;
-- Switch to a namespace
USE my_namespace;
-- List tables (MQ topics)
SHOW TABLES;
-- Query message data
SELECT * FROM user_events WHERE _ts > '2025-01-01' LIMIT 10;
-- Perform basic aggregations
SELECT COUNT(*) as total_events FROM user_events;
```
## Commands and Interfaces
### Database Server Mode (`weed db`)
Starts a PostgreSQL-compatible database server that accepts connections from any PostgreSQL client.
```bash
# Basic usage
weed db
# Production setup with MD5 authentication
weed db -auth=md5 -users="@/etc/seaweedfs/users.json" \
-host=0.0.0.0 -port=5432
# With TLS encryption
weed db -auth=md5 -users="@users.json" \
-tls-cert=/etc/ssl/server.crt \
-tls-key=/etc/ssl/server.key
# Custom configuration
weed db -auth=md5 -users='{"admin":"pass"}' \
-port=5433 \
-master=master1:9333 \
-max-connections=200 \
-idle-timeout=2h
```
**Options:**
- `-auth`: Authentication method (`trust`, `password`, `md5`)
- `-users`: User credentials (JSON format or file)
- `-host`: Database server host (default: `localhost`)
- `-port`: Database server port (default: `5432`)
- `-master`: SeaweedFS master server address
- `-database`: Default database name
- `-max-connections`: Maximum concurrent connections
- `-idle-timeout`: Connection idle timeout
- `-tls-cert`, `-tls-key`: TLS certificate and key files
### Interactive CLI Mode (`weed sql`)
Provides an interactive SQL shell for quick queries and exploration.
```bash
# Start interactive SQL shell
weed sql -master=localhost:9333
# Connect to specific namespace
weed sql -master=localhost:9333 -namespace=analytics
# Execute single query
weed sql -master=localhost:9333 -exec="SHOW TABLES"
```
## Authentication and Security
### Authentication Methods
1. **`trust`** (Default - Development Only)
- No authentication required
- Suitable for local development only
2. **`md5`** (Recommended for Production)
- Secure MD5 hash + salt authentication
- Compatible with all PostgreSQL clients
- Safe even without TLS
3. **`password`** (Development + TLS Only)
- Clear text password transmission
- Requires TLS encryption for production use
### Credential Formats
#### JSON Format (Inline)
```bash
weed db -auth=md5 -users='{"admin":"secret","analyst":"readonly"}'
# Supports any special characters
weed db -auth=md5 -users='{"admin":"pass;with;semicolons","user":"pass:with:colons"}'
```
#### File Format (Recommended)
```bash
# Create users file
cat > /etc/seaweedfs/users.json << EOF
{
"admin": "strong_password_123!",
"analyst": "readonly_user_456",
"developer": "dev_access_789"
}
EOF
# Use file-based credentials
weed db -auth=md5 -users="@/etc/seaweedfs/users.json"
```
### TLS Encryption
```bash
# Generate certificates (example)
openssl req -x509 -newkey rsa:2048 -keyout server.key -out server.crt -days 365 -nodes
# Start with TLS
weed db -auth=md5 -users="@users.json" \
-tls-cert=server.crt -tls-key=server.key
```
## SQL Operations
### Schema Operations
```sql
-- List all databases (MQ namespaces)
SHOW DATABASES;
-- Switch database context
USE namespace_name;
-- List tables in current database (MQ topics)
SHOW TABLES;
-- Describe table schema
DESCRIBE table_name;
SHOW COLUMNS FROM table_name;
-- Note: CREATE TABLE, DROP TABLE and ALTER TABLE are not supported
```
### Data Queries
```sql
-- Basic SELECT
SELECT * FROM user_events LIMIT 10;
-- Filtering with WHERE clauses
SELECT * FROM user_events
WHERE event_type = 'login'
AND _ts > '2025-01-01';
-- Basic aggregations (limited support)
SELECT COUNT(*) FROM user_events;
SELECT MIN(timestamp), MAX(timestamp) FROM user_events;
SELECT SUM(value_column), AVG(value_column) FROM user_events;
-- System columns (available on all topics)
SELECT
_ts, -- Message timestamp (formatted, supports string parsing in WHERE)
_key, -- Message key
_source, -- Data source (parquet file or live)
*
FROM user_events;
```
**Note:** The `_ts` system column supports automatic parsing of timestamp strings in WHERE clauses. Supported formats include `'2025-01-01'`, `'2025-01-01T15:30:00Z'`, `'2025-01-01 15:30:00'`, etc. Other timestamp columns require exact value matching.
### Supported WHERE Clause Operations
```sql
-- Comparison operators
SELECT * FROM user_events WHERE user_id = 123;
SELECT * FROM user_events WHERE _ts > '2025-01-01';
SELECT * FROM user_events WHERE value <= 100;
-- Pattern matching
SELECT * FROM user_events WHERE event_type LIKE 'login%';
-- IN clause
SELECT * FROM user_events WHERE event_type IN ('login', 'logout', 'signup');
-- Combining conditions
SELECT * FROM user_events
WHERE event_type = 'purchase'
AND _ts > '2025-01-01'
AND value > 50;
```
### INTERVAL Expressions and Timestamp Arithmetic
```sql
-- Basic INTERVAL expressions
SELECT INTERVAL '1 hour';
SELECT INTERVAL '30 minutes';
SELECT INTERVAL '24 hours';
SELECT INTERVAL '7 days';
-- Timestamp arithmetic with INTERVAL
SELECT NOW() - INTERVAL '1 hour' as one_hour_ago;
SELECT CURRENT_TIMESTAMP - INTERVAL '24 hours' as yesterday;
-- Using INTERVAL in WHERE clauses for time-based filtering
SELECT * FROM user_events
WHERE _ts >= NOW() - INTERVAL '1 hour';
SELECT * FROM user_events
WHERE _ts >= CURRENT_TIMESTAMP - INTERVAL '24 hours'
AND _ts < CURRENT_TIMESTAMP - INTERVAL '1 hour';
-- Supported INTERVAL units
SELECT NOW() - INTERVAL '1 second';
SELECT NOW() - INTERVAL '5 minutes';
SELECT NOW() - INTERVAL '2 hours';
SELECT NOW() - INTERVAL '3 days';
SELECT NOW() - INTERVAL '1 week';
```
### Limitations
**Not Supported:**
- `ORDER BY` clauses
- `GROUP BY` clauses
- `HAVING` clauses
- `JOIN` operations
- `CREATE TABLE` statements
- `DROP TABLE` statements
- `ALTER TABLE` statements
- Complex aggregations with grouping
- Window functions
- Subqueries
## Client Examples
### Python (psycopg2)
```python
import psycopg2
import pandas as pd
# Connect to SeaweedFS SQL server
conn = psycopg2.connect(
host="localhost",
port=5432,
user="admin",
password="secret",
database="default"
)
# Query data
query = """
SELECT event_type, user_id, _ts
FROM user_events
WHERE _ts > '2025-01-01'
LIMIT 100
"""
# Use pandas for easy data analysis
df = pd.read_sql(query, conn)
print(df)
# Close connection
conn.close()
```
### Java JDBC
```java
import java.sql.*;
public class SeaweedFSQuery {
public static void main(String[] args) throws SQLException {
String url = "jdbc:postgresql://localhost:5432/default";
Connection conn = DriverManager.getConnection(url, "admin", "secret");
String query = "SELECT * FROM user_events LIMIT 10";
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
while (rs.next()) {
System.out.println("User ID: " + rs.getInt("user_id"));
System.out.println("Event: " + rs.getString("event_type"));
System.out.println("Timestamp: " + rs.getTimestamp("timestamp"));
}
conn.close();
}
}
```
### Go (lib/pq)
```go
package main
import (
"database/sql"
"fmt"
"log"
_ "github.com/lib/pq"
)
func main() {
connStr := "host=localhost port=5432 user=admin password=secret dbname=default sslmode=disable"
db, err := sql.Open("postgres", connStr)
if err != nil {
log.Fatal(err)
}
defer db.Close()
query := "SELECT event_type, user_id, _ts FROM user_events WHERE _ts > '2025-01-01' LIMIT 100"
rows, err := db.Query(query)
if err != nil {
log.Fatal(err)
}
defer rows.Close()
for rows.Next() {
var eventType string
var userID int
var timestamp string
if err := rows.Scan(&eventType, &userID, &timestamp); err != nil {
log.Fatal(err)
}
fmt.Printf("Event: %s, User: %d, Time: %s\n", eventType, userID, timestamp)
}
}
```
### Node.js (pg)
```javascript
const { Client } = require('pg');
const client = new Client({
host: 'localhost',
port: 5432,
user: 'admin',
password: 'secret',
database: 'default'
});
async function queryData() {
await client.connect();
const query = `
SELECT event_type, user_id, timestamp
FROM user_events
WHERE timestamp > $1
LIMIT 100
`;
const result = await client.query(query, ['2024-01-01']);
console.log('Recent Events:');
result.rows.forEach(row => {
console.log(`${row.event_type} by user ${row.user_id} at ${row.timestamp}`);
});
await client.end();
}
queryData().catch(console.error);
```
## Data Sources and Architecture
### Hybrid Message Scanner
The SQL engine uses a hybrid approach to query both real-time and historical data:
1. **Live Messages** - Queries unflushed messages directly from MQ brokers
2. **Parquet Files** - Queries archived/flushed messages from Parquet storage
3. **Seamless Integration** - Results are merged to provide complete data view
### System Columns
Every topic automatically includes system columns:
- `_ts` - Message timestamp (formatted timestamp)
- `_key` - Message partition key
- `_source` - Data source identifier (parquet file path or "live")
### Schema Evolution
The SQL engine supports backward-compatible schema evolution:
- New columns can be added to existing topics
- Old queries continue to work with new data
- Missing columns return NULL values
## Performance Optimization
### Fast Path Aggregations
The engine optimizes common aggregation queries:
```sql
-- These queries use optimized fast paths
SELECT COUNT(*) FROM user_events;
SELECT MIN(timestamp) FROM user_events;
SELECT MAX(timestamp) FROM user_events;
-- Add WHERE clauses for more specific queries
SELECT COUNT(*) FROM user_events WHERE event_type = 'login';
```
### Query Best Practices
1. **Use time-based filtering** for large datasets:
```sql
SELECT * FROM events WHERE _ts >= '2025-01-01' AND _ts < '2025-02-01';
```
2. **Limit result sets** for exploration:
```sql
SELECT * FROM events WHERE _ts > '2025-01-01' LIMIT 1000;
```
3. **Use appropriate indexes** on frequently queried columns (when supported)
4. **Leverage system columns** for debugging:
```sql
SELECT _source, _ts FROM events LIMIT 100;
```
## BI Tool Integration
### Apache Superset
```python
# Database URI for Apache Superset
postgresql://admin:secret@localhost:5432/default
```
### Grafana
```yaml
# Grafana datasource configuration
apiVersion: 1
datasources:
- name: SeaweedFS
type: postgres
url: localhost:5432
database: default
user: admin
password: secret
sslmode: disable
```
### Tableau
Use the PostgreSQL connector with:
- Server: `localhost`
- Port: `5432`
- Database: `default`
- Username: `admin`
- Password: `secret`
## Troubleshooting
### Common Issues
1. **"Database server not running"**
```bash
# Check if weed db is running
ps aux | grep "weed db"
# Check port availability
netstat -ln | grep 5432
```
2. **"Authentication failed"**
```bash
# Verify user credentials
cat users.json
# Test with trust authentication
weed db -auth=trust
```
3. **"No data returned"**
```sql
-- Check if topics exist
SHOW TABLES;
-- Check data sources
SELECT _source FROM topic_name LIMIT 100;
-- Check timestamp ranges
SELECT MIN(timestamp), MAX(timestamp) FROM topic_name;
```
4. **"Connection timeout"**
```bash
# Increase timeout settings
weed db -idle-timeout=24h -max-connections=50
# Check network connectivity
telnet localhost 5432
```
### Debug Mode
Enable debug logging for troubleshooting:
```bash
# Set debug mode
export GLOG_v=2
# Start with verbose logging
weed db -auth=md5 -users="@users.json" -v=2
```
### Performance Issues
1. **Slow queries on large datasets**:
- Add time-based WHERE clauses
- Use LIMIT for exploration
- Consider data partitioning strategies
2. **High memory usage**:
- Reduce concurrent connections
- Limit result set sizes
- Monitor broker memory usage
3. **Network timeouts**:
- Increase idle timeout settings
- Check network stability between components
- Use connection pooling in applications
## Production Deployment
### Recommended Setup
```bash
# Production database server
weed db \
-auth=md5 \
-users="@/etc/seaweedfs/users.json" \
-host=0.0.0.0 \
-port=5432 \
-master=master1:9333,master2:9333,master3:9333 \
-max-connections=100 \
-idle-timeout=1h \
-tls-cert=/etc/ssl/seaweedfs-db.crt \
-tls-key=/etc/ssl/seaweedfs-db.key
```
### High Availability
- Run multiple `weed db` instances behind a load balancer
- Use master server failover configuration
- Monitor connection health and query performance
- Implement connection pooling in client applications
### Security Checklist
- [ ] Use MD5 or password authentication (never trust in production)
- [ ] Enable TLS encryption for data in transit
- [ ] Store credentials in secure files with proper permissions
- [ ] Limit database server network access with firewalls
- [ ] Monitor authentication attempts and query patterns
- [ ] Regular security updates and credential rotation
## Related Documentation
- [Seaweed Message Queue](Seaweed-Message-Queue.md)
- [Security Configuration](Security-Configuration.md)
- [Production Setup](Production-Setup.md)
- [System Metrics](System-Metrics.md)
SQL-Quick-Reference.md
+233
@@ -0,0 +1,233 @@
# SQL Quick Reference
Quick reference guide for SeaweedFS SQL queries on Message Queue topics.
## Commands
### Start Database Server
```bash
# Basic (development)
weed db
# Production with MD5 auth
weed db -auth=md5 -users='{"admin":"secret"}' -host=0.0.0.0
# With TLS encryption
weed db -auth=md5 -users="@users.json" -tls-cert=server.crt -tls-key=server.key
```
### Interactive CLI
```bash
# Start SQL shell
weed sql
# Execute single query
weed sql -exec="SHOW TABLES"
```
## Authentication
### Credential Formats
```bash
# JSON inline
-users='{"user1":"pass1","user2":"pass2"}'
# JSON file
-users="@/path/to/users.json"
```
### Auth Methods
- `trust` - No auth (dev only)
- `md5` - Hash + salt (recommended)
- `password` - Clear text (TLS required)
## Client Connections
### psql
```bash
# Basic connection
psql -h localhost -p 5432 -U admin -d default
# With password
PGPASSWORD=secret psql -h localhost -p 5432 -U admin -d default
# Connection string
psql "postgresql://admin:secret@localhost:5432/default"
```
### Programming Languages
```python
# Python
import psycopg2
conn = psycopg2.connect(host="localhost", port=5432, user="admin", password="secret", database="default")
```
```java
// Java
String url = "jdbc:postgresql://localhost:5432/default";
Connection conn = DriverManager.getConnection(url, "admin", "secret");
```
```go
// Go
db, err := sql.Open("postgres", "host=localhost port=5432 user=admin password=secret dbname=default sslmode=disable")
```
## SQL Operations
### Schema Commands
```sql
SHOW DATABASES; -- List MQ namespaces
USE namespace_name; -- Switch database
SHOW TABLES; -- List MQ topics
DESCRIBE table_name; -- Show table schema
-- Note: CREATE TABLE, DROP TABLE and ALTER TABLE not supported
```
### Query Commands
```sql
-- Basic queries
SELECT * FROM events LIMIT 10;
SELECT * FROM events WHERE _ts > '2025-01-01';
SELECT * FROM events WHERE status IN ('active', 'completed');
SELECT COUNT(*) FROM events;
-- NULL checking operations
SELECT * FROM events WHERE status IS NULL;
SELECT * FROM events WHERE status IS NOT NULL;
SELECT * FROM events WHERE user_id IS NOT NULL AND status = 'active';
-- System columns (available on all tables)
SELECT _ts, _key, _source, * FROM events;
-- Aggregations (optimized)
SELECT COUNT(*) FROM events;
SELECT MIN(timestamp), MAX(timestamp) FROM events;
```
### Time-based Queries
```sql
-- Time filtering with system timestamp column (_ts)
-- Automatic string-to-timestamp conversion for _ts system column
SELECT * FROM events
WHERE _ts >= '2025-01-01'
AND _ts < '2025-02-01'
LIMIT 1000;
-- INTERVAL expressions and timestamp arithmetic
SELECT NOW() - INTERVAL '1 hour';
SELECT * FROM events WHERE _ts >= NOW() - INTERVAL '24 hours';
SELECT * FROM events WHERE _ts >= CURRENT_TIMESTAMP - INTERVAL '1 day';
-- BETWEEN clauses with INTERVAL arithmetic
SELECT * FROM events WHERE _ts BETWEEN NOW() - INTERVAL '1 week' AND NOW();
SELECT * FROM events WHERE _ts BETWEEN '2025-01-01' AND '2025-12-31';
-- Current time functions
SELECT NOW(), CURRENT_TIMESTAMP, CURRENT_DATE FROM events LIMIT 1;
```
## System Columns
Every topic includes these system columns:
- `_ts` - Message timestamp (formatted timestamp, supports automatic string-to-timestamp conversion in WHERE clauses)
- `_key` - Message partition key
- `_source` - Data source ("live" or parquet file path)
**Note:** The `_ts` column supports automatic parsing of timestamp strings in WHERE clauses. Formats supported: `'2025-01-01'`, `'2025-01-01T15:30:00Z'`, `'2025-01-01 15:30:00'`, etc.
## NULL Value Handling
### NULL Checking Operations
```sql
-- Check for NULL values
SELECT * FROM events WHERE description IS NULL;
-- Check for non-NULL values
SELECT * FROM events WHERE user_id IS NOT NULL;
-- Combine with other conditions
SELECT * FROM events
WHERE user_id IS NOT NULL
AND status = 'active'
AND _ts >= '2025-01-01';
-- Filter out records with missing data
SELECT * FROM events WHERE user_id IS NOT NULL AND description IS NOT NULL;
```
### NULL Value Semantics
- **Empty strings** are treated as valid values (not NULL)
- **Missing fields** in records are considered NULL
- **Boolean, numeric, and timestamp values** are never NULL once present
- **Bytes values** are treated as non-NULL even if empty
- NULL values are excluded from aggregate functions like `COUNT(column_name)`
## Troubleshooting
### Check Status
```sql
-- Verify tables exist
SHOW TABLES;
-- Check data sources
SELECT _source FROM table_name LIMIT 100;
-- Verify time range
SELECT MIN(timestamp), MAX(timestamp) FROM table_name;
```
### Common Issues
1. **No data**: Check `SHOW TABLES` and topic names
2. **Auth failed**: Verify credentials in users file
3. **Timeouts**: Increase `-idle-timeout` setting
4. **Slow queries**: Add WHERE clauses and LIMIT
5. **NULL filtering**: Use `IS NULL` / `IS NOT NULL` instead of `= NULL` / `!= NULL`
### Debug Mode
```bash
# Enable verbose logging
export GLOG_v=2
weed db -v=2 ...
```
## Performance Tips
1. **Filter by time** for large datasets:
```sql
WHERE _ts >= '2025-01-01' AND _ts < '2025-02-01'
```
2. **Use LIMIT** for exploration:
```sql
SELECT * FROM events WHERE _ts > '2025-01-01' LIMIT 1000
```
3. **Fast aggregations** (basic functions only):
```sql
SELECT COUNT(*) FROM events; -- Optimized
SELECT MIN(timestamp), MAX(timestamp) FROM events; -- Optimized
```
4. **Check data sources**:
```sql
SELECT _source, _ts FROM events LIMIT 100;
```
5. **Filter NULL values for cleaner results**:
```sql
SELECT * FROM events WHERE user_id IS NOT NULL LIMIT 100;
```
## Limitations
**Not Supported:**
- `ORDER BY`, `GROUP BY`, `HAVING` clauses
- `JOIN` operations between tables
- `CREATE TABLE`, `DROP TABLE`, `ALTER TABLE` statements
- Window functions and subqueries
- Complex aggregations with grouping
## Links
For detailed documentation, see [[SQL Queries on Message Queue]].
Seaweed-Message-Queue.md
+158
@@ -0,0 +1,158 @@
# Introduction
Seaweed Message Queue (SMQ) is a distributed messaging system built on top of SeaweedFS. It provides:
* Structured message : all messages need to have a schema
* Streamed publishing with async acknowledgements
* Messages are stored in Parquet files, for both streaming and batch reading
* Disaggregated storage
* Scalable stateless message brokers
# Architecture
The system consists of three main components:
1. **Message Queue Agent**: A gRPC server that provides a simplified interface for clients
2. **Message Queue Brokers**: Stateless brokers that handle message routing and storage
3. **SeaweedFS**: The underlying storage system that persists messages in Parquet format
```
Publishers => gRPC Publish APIs => Agent => Brokers => Agent => gRPC Subscribe APIs => Subscribers
^
|
v
SeaweedFS
```
The Agent can be run either on the server side, or as a sidecar on each client.
# Features
## Core Features
* Structured Messages: SeaweedFS is used to store unstructured data files, while Seaweed Message Queue is used to store structured messages
* Messages stored in SeaweedFS can be converted into Parquet files, saving disk space with more efficient columnar compression
* The messages in Parquet files can be streamed via Seaweed messaging brokers
* The Parquet files can be read in batches directly from SeaweedFS
## Publishing Features
* Messages published successfully are acknowledged asynchronously
* Partition-based message routing
* Schema validation for message structure
## Subscribing Features
* Message consume offsets are tracked and persisted on the server side
* Consumer APIs can process messages in parallel while still ensuring serial processing of messages with the same key
* Configurable sliding window for concurrent message processing
* Ability to start consuming from specific timestamps or offsets
# Usage Examples
## Starting the Services
1. Start a Message Queue Broker:
```bash
weed mq.broker -port=17777 -master=localhost:9333
```
2. Start a Message Queue Agent:
```bash
weed mq.agent -port=16777 -broker=localhost:17777
```
## Defining Message Schema
Messages in SMQ must have a defined schema. Here's an example of defining a message type:
```go
type MyRecord struct {
Key []byte
Field1 []byte
Field2 string
Field3 int32
Field4 int64
Field5 float32
Field6 float64
Field7 bool
}
func MyRecordType() *schema_pb.RecordType {
return schema.RecordTypeBegin().
WithField("key", schema.TypeBytes).
WithField("field1", schema.TypeBytes).
WithField("field2", schema.TypeString).
WithField("field3", schema.TypeInt32).
WithField("field4", schema.TypeInt64).
WithField("field5", schema.TypeFloat).
WithField("field6", schema.TypeDouble).
WithField("field7", schema.TypeBoolean).
RecordTypeEnd()
}
```
## Publishing Messages
```go
// Create a publish session
session, err := agent_client.NewPublishSession(
"localhost:16777", // agent address
schema.NewSchema("my_namespace", "my_topic", MyRecordType()),
6, // partition count
"publisher1", // client name
)
// Publish a message
myRecord := &MyRecord{
Key: []byte("key1"),
Field1: []byte("value1"),
Field2: "string value",
Field3: 123,
Field4: 456,
Field5: 1.23,
Field6: 4.56,
Field7: true,
}
err := session.PublishMessageRecord(myRecord.Key, myRecord.ToRecordValue())
```
## Subscribing to Messages
```go
// Create a subscribe session
session, err := agent_client.NewSubscribeSession(
"localhost:16777", // agent address
&agent_client.SubscribeOption{
ConsumerGroup: "my-group",
ConsumerGroupInstanceId: "consumer1",
Topic: topic.NewTopic("my_namespace", "topmy_topicic"),
OffsetType: schema_pb.OffsetType_RESUME_OR_EARLIEST,
MaxSubscribedPartitions: 3, // maximum number of partitions this consumer instance can subscribe
SlidingWindowSize: 16, // concurrently process up-to 16 messages with different message key
},
)
// Subscribe to messages
session.SubscribeMessageRecord(
func(key []byte, recordValue *schema_pb.RecordValue) {
record := FromRecordValue(recordValue)
fmt.Printf("Received: %+v\n", record)
},
func() {
fmt.Println("Subscription completed")
},
)
```
# Configuration
## Broker Configuration
* `-port`: gRPC server port (default: 17777)
* `-master`: comma-separated master servers
* `-filerGroup`: share metadata with other filers in the same group
* `-dataCenter`: prefer volumes in this data center
* `-rack`: prefer volumes in this rack
## Agent Configuration
* `-port`: gRPC server port (default: 16777)
* `-broker`: comma-separated message queue brokers
Security-Configuration.md
+72 -10
@@ -1,5 +1,7 @@
For an overview of SeaweedFS security architecture and communication layers, see [[Security-Overview]].
The first step is generating `security.toml` file via `weed scaffold -config=security`:
```
@@ -11,6 +13,11 @@ $ weed scaffold -config=security
# /etc/seaweedfs/security.toml
# this file is read by master, volume server, and filer
# comma separated origins allowed to make requests to the filer and s3 gateway.
# enter in this format: https://domain.com, or http://localhost:port
[cors.allowed_origins]
values = "*"
# this jwt signing key is read by master and volume server, and it is used for write operations:
# - the Master server generates the JWT, which can be used to write a certain file on a volume server
# - the Volume server validates the JWT on writing
@@ -26,6 +33,13 @@ expires_after_seconds = 10 # seconds
[access]
ui = false
# by default the filer UI is enabled. This can be a security risk if the filer is exposed to the public
# and the JWT for reads is not set. If you don't want the public to have access to the objects in your
# storage, and you haven't set the JWT for reads it is wise to disable access to directory metadata.
# This disables access to the Filer UI, and will no longer return directory metadata in GET requests.
[filer.expose_directory_metadata]
enabled = true
# this jwt signing key is read by master and volume server, and it is used for read operations:
# - the Master server generates the JWT, which can be used to read a certain file on a volume server
# - the Volume server validates the JWT on reading
@@ -51,49 +65,95 @@ expires_after_seconds = 10 # seconds
key = ""
expires_after_seconds = 10 # seconds
# all grpc tls authentications are mutual
# the values for the following ca, cert, and key are paths to the PERM files.
# the host name is not checked, so the PERM files can be shared.
# gRPC mTLS configuration
# All gRPC TLS authentications are mutual (mTLS)
# The values for ca, cert, and key are paths to the certificate/key files
# The host name is not checked, so the certificate files can be shared
[grpc]
ca = ""
# Set wildcard domain for enable TLS authentication by common names
allowed_wildcard_domain = "" # .mycompany.com
# Volume server gRPC options (server-side)
# Enables mTLS for incoming gRPC connections to volume server
[grpc.volume]
cert = ""
key = ""
allowed_commonNames = "" # comma-separated SSL certificate common names
# Master server gRPC options (server-side)
# Enables mTLS for incoming gRPC connections to master server
[grpc.master]
cert = ""
key = ""
allowed_commonNames = "" # comma-separated SSL certificate common names
# Filer server gRPC options (server-side)
# Enables mTLS for incoming gRPC connections to filer server
[grpc.filer]
cert = ""
key = ""
allowed_commonNames = "" # comma-separated SSL certificate common names
# S3 server gRPC options (server-side)
# Enables mTLS for incoming gRPC connections to S3 server
[grpc.s3]
cert = ""
key = ""
allowed_commonNames = "" # comma-separated SSL certificate common names
[grpc.msg_broker]
cert = ""
key = ""
allowed_commonNames = "" # comma-separated SSL certificate common names
# use this for any place needs a grpc client
# i.e., "weed backup|benchmark|filer.copy|filer.replicate|mount|s3|upload"
[grpc.msg_agent]
cert = ""
key = ""
allowed_commonNames = "" # comma-separated SSL certificate common names
# gRPC client configuration for outgoing gRPC connections
# Used by clients (S3, mount, backup, benchmark, filer.copy, filer.replicate, upload, etc.)
# when connecting to any gRPC server (master, volume, filer)
[grpc.client]
cert = ""
key = ""
# volume server https options
# Note: work in progress!
# this does not work with other clients, e.g., "weed filer|mount" etc, yet.
# HTTPS client configuration for outgoing HTTP connections
# Used by S3, mount, filer.copy, backup, and other clients when communicating with master/volume/filer
# Set enabled=true to use HTTPS instead of HTTP for data operations (separate from gRPC)
# If [https.filer] or [https.volume] are enabled on servers, clients must have [https.client] enabled=true
[https.client]
enabled = true
enabled = false # Set to true to enable HTTPS for all outgoing HTTP client connections
cert = "" # Client certificate for mTLS (optional if server doesn't require client cert)
key = "" # Client key for mTLS (optional if server doesn't require client cert)
ca = "" # CA certificate to verify server certificates (required when enabled=true)
# Volume server HTTPS options (server-side)
# Enables HTTPS for incoming HTTP connections to volume server
[https.volume]
cert = ""
key = ""
ca = ""
# Master server HTTPS options (server-side)
# Enables HTTPS for incoming HTTP connections to master server (web UI, HTTP API)
[https.master]
cert = ""
key = ""
ca = ""
# Filer server HTTPS options (server-side)
# Enables HTTPS for incoming HTTP connections to filer server (web UI, HTTP API)
[https.filer]
cert = ""
key = ""
ca = ""
# disable_tls_verify_client_cert = true|false (default: false)
# white list. It's checking request ip address.
[guard]
white_list = ""
```
The following command is what I used to generate the private key and certificate files, using https://github.com/square/certstrap. To compile this tool, you can run `go get github.com/square/certstrap` - or alternatively `brew install certstrap` if you are on Mac OS and use [Homebrew](https://brew.sh).
@@ -162,4 +222,6 @@ Java gRPC uses Netty's SslContext. From https://netty.io/wiki/sslcontextbuilder-
If you are using existing certificates: make sure they all have the **Extended Key Usage** 'TLS Web Server Authentication' AND 'TLS Web Client Authentication' set - as grpc uses them for both use-cases!
Else you will see those errors: `error reading server preface: remote error: tls: bad certificate`
Else you will see those errors: `error reading server preface: remote error: tls: bad certificate`
Choose your CA carefully when using existing certificates in an enterprise environment. Since Seaweed only checks the certificate against the CA and optionally validates the CN of the certificate against a whitelist, do not use the root CA of the company or any other CA you don't control, as this means someone else can generate a client certificate your cluster will accept as legitimate. Instead, generate an intermediate CA for each Seaweed cluster and use it to generate server and client certificates. This intermediate CA is the one you should use in `grpc.ca` and `https.*.ca` properties.
Security-Overview.md
+33
@@ -2,6 +2,39 @@
Since SeaweedFS is a distributed system with many volume servers, the volume servers have the risk of being changed without proper access control. We want to have the freedom to place a volume server anywhere we want, with the confidence that nobody can tamper the data.
## Understanding Communication Layers
SeaweedFS has two separate communication layers that must be secured independently:
### gRPC Communication (Control Plane)
Used for metadata operations and cluster coordination:
- S3 ↔ Filer: Getting bucket info, file metadata
- Filer ↔ Master: Getting volume assignments, cluster info
- Volume ↔ Master: Heartbeats, volume management
- **Configured via**: `[grpc.*]` sections in security.toml
### HTTP/HTTPS Communication (Data Plane)
Used for actual file data uploads/downloads:
- S3/Filer/Client → Volume Server: Uploading and downloading actual file content
- S3 → Filer: File data operations (when S3 acts as a proxy)
- **Configured via**: `[https.*]` sections in security.toml
### Configuration Sections Summary
| Configuration Block | Purpose |
|---------------------|---------|
| `[grpc.client]` | Used by clients (S3, mount, filer.copy, backup, etc.) to connect to any gRPC server |
| `[grpc.filer]`, `[grpc.master]`, `[grpc.volume]`, `[grpc.s3]` | Server-side gRPC mTLS - secures gRPC endpoints |
| `[https.client]` | Client-side HTTPS configuration - tells clients (S3/Filer/etc.) to use HTTPS for HTTP connections |
| `[https.volume]`, `[https.master]`, `[https.filer]` | Server-side HTTPS configuration - makes servers accept HTTPS connections |
**Important**:
- `[https.client]` with `enabled = true` is required for clients to use HTTPS when connecting to HTTPS-enabled servers
- If you enable `[https.filer]` on the Filer server, you must also enable `[https.client]` so that S3 API can connect via HTTPS
- Otherwise, you'll get "client sent an HTTP request to an HTTPS server" errors
## Security Features
We will address the volume servers first. The following items are not covered, yet:
- master server http REST services
Server-Side-Encryption-SSE-C.md
+137
@@ -0,0 +1,137 @@
# Server-Side Encryption with Customer-provided Keys (SSE-C)
With SSE-C, you bring your own keys and SeaweedFS does the heavy lifting. Your app sends the key with each request, and we encrypt/decrypt on the server side—without ever storing your key.
## Overview
SSE-C gives you client-side key management with server-side encryption:
- **Client provides**: AES-256 encryption key and MD5 hash
- **SeaweedFS handles**: Encryption/decryption operations transparently
- **Security**: Keys are never stored on the server
## Required Headers
For all SSE-C operations, include these headers:
```http
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: <base64-encoded-256-bit-key>
X-Amz-Server-Side-Encryption-Customer-Key-MD5: <md5-of-key>
```
## HTTP Examples
### Upload Encrypted Object
```bash
# Generate a 256-bit key
KEY=$(openssl rand -base64 32)
KEY_MD5=$(echo -n "$KEY" | base64 -d | md5sum | cut -d' ' -f1)
# Upload encrypted object
curl -X PUT "http://localhost:8333/bucket/encrypted-file.txt" \
-H "X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256" \
-H "X-Amz-Server-Side-Encryption-Customer-Key: $KEY" \
-H "X-Amz-Server-Side-Encryption-Customer-Key-MD5: $KEY_MD5" \
-H "Content-Type: text/plain" \
--data "This content will be encrypted"
```
### Download Encrypted Object
```bash
# Download and decrypt object (must use same key)
curl "http://localhost:8333/bucket/encrypted-file.txt" \
-H "X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256" \
-H "X-Amz-Server-Side-Encryption-Customer-Key: $KEY" \
-H "X-Amz-Server-Side-Encryption-Customer-Key-MD5: $KEY_MD5"
```
### Get Object Metadata
```bash
# Get metadata for encrypted object
curl -I "http://localhost:8333/bucket/encrypted-file.txt" \
-H "X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256" \
-H "X-Amz-Server-Side-Encryption-Customer-Key: $KEY" \
-H "X-Amz-Server-Side-Encryption-Customer-Key-MD5: $KEY_MD5"
```
### Copy Operations
```bash
# Copy encrypted object to new location (same key)
curl -X PUT "http://localhost:8333/bucket/copied-file.txt" \
-H "x-amz-copy-source: /bucket/encrypted-file.txt" \
-H "X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256" \
-H "X-Amz-Server-Side-Encryption-Customer-Key: $KEY" \
-H "X-Amz-Server-Side-Encryption-Customer-Key-MD5: $KEY_MD5" \
-H "x-amz-copy-source-server-side-encryption-customer-algorithm: AES256" \
-H "x-amz-copy-source-server-side-encryption-customer-key: $KEY" \
-H "x-amz-copy-source-server-side-encryption-customer-key-md5: $KEY_MD5"
# Copy encrypted object with different key
NEW_KEY=$(openssl rand -base64 32)
NEW_KEY_MD5=$(echo -n "$NEW_KEY" | base64 -d | md5sum | cut -d' ' -f1)
curl -X PUT "http://localhost:8333/bucket/reencrypted-file.txt" \
-H "x-amz-copy-source: /bucket/encrypted-file.txt" \
-H "X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256" \
-H "X-Amz-Server-Side-Encryption-Customer-Key: $NEW_KEY" \
-H "X-Amz-Server-Side-Encryption-Customer-Key-MD5: $NEW_KEY_MD5" \
-H "x-amz-copy-source-server-side-encryption-customer-algorithm: AES256" \
-H "x-amz-copy-source-server-side-encryption-customer-key: $KEY" \
-H "x-amz-copy-source-server-side-encryption-customer-key-md5: $KEY_MD5"
```
### AWS CLI Usage
```bash
# Upload with SSE-C
aws s3 cp file.txt s3://mybucket/file.txt \
--sse-c AES256 \
--sse-c-key fileb://customer-key.bin
# Download with SSE-C
aws s3 cp s3://mybucket/file.txt downloaded-file.txt \
--sse-c AES256 \
--sse-c-key fileb://customer-key.bin
# Copy with SSE-C (same key)
aws s3 cp s3://mybucket/file.txt s3://mybucket/file-copy.txt \
--sse-c AES256 \
--sse-c-key fileb://customer-key.bin \
--sse-c-copy-source AES256 \
--sse-c-copy-source-key fileb://customer-key.bin
```
## Error Codes
| Error | HTTP Status | Description |
|-------|-------------|-------------|
| `InvalidEncryptionAlgorithmError` | 400 | Algorithm must be "AES256" |
| `InvalidArgument` | 400 | Invalid key format or MD5 mismatch |
| `InvalidRequest` | 400 | Missing SSE-C headers |
## Common Issues
**Wrong algorithm:**
```bash
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES128 # Error!
```
**Invalid key length:**
```bash
X-Amz-Server-Side-Encryption-Customer-Key: dGVzdA== # Error! (too short)
```
**Missing key for encrypted object:**
```bash
curl http://localhost:8333/bucket/encrypted-file.txt # Error!
```
## Related Documentation
- [Server-Side Encryption](Server-Side-Encryption.md)
- [Amazon S3 API](Amazon-S3-API.md)
Server-Side-Encryption-SSE-KMS.md
+310
@@ -0,0 +1,310 @@
# SSE-KMS: Server-Side Encryption with Key Management Service
SeaweedFS works with your existing Key Management Service (KMS) so you can keep keys where they belong. This guide walks you through AWS KMS, Google Cloud KMS, and OpenBao/Vault. Azure Key Vault is also available as experimental (build tag `azurekms`).
## Supported KMS Providers
| Provider | Status | Use Cases |
|----------|--------|-----------|
| **AWS KMS** | Full support | AWS-centric deployments |
| **Google Cloud KMS** | Full support | GCP-centric deployments |
| **OpenBao/Vault** | Full support | Hybrid/on-premises environments |
| **Azure Key Vault** | Experimental (build tag `azurekms`) | Azure-centric deployments |
## Quick Start Guide
### 1. Configure SeaweedFS
Tell SeaweedFS about your KMS in the S3 config JSON file:
```json
{
"identities": [
{
"name": "admin",
"credentials": [{"accessKey": "admin", "secretKey": "password"}],
"actions": ["Admin", "Read", "Write"]
}
],
"kms": {
"default_provider": "openbao",
"providers": {
"openbao": {
"type": "openbao",
"address": "http://localhost:8200",
"token": "root-token",
"transit_path": "transit",
"cache_enabled": true,
"cache_ttl": "1h"
}
}
}
}
```
### 2. Start S3 with KMS Config
```bash
# Start S3 API with KMS and IAM configuration
weed s3 -config=s3_kms_config.json -port=8333
```
**Note:** The S3 config JSON file contains both KMS provider settings AND IAM-style access control (identities, credentials, permissions).
### 3. Test the Integration
```bash
# Upload object with SSE-KMS
aws s3 cp test-file.txt s3://mybucket/test-file.txt \
--server-side-encryption aws:kms \
--ssekms-key-id alias/my-key
```
---
## AWS KMS Integration
### Step 1: Create KMS Key
```bash
# Create customer-managed KMS key
aws kms create-key --description "SeaweedFS encryption key"
# Create key alias
aws kms create-alias \
--alias-name alias/seaweedfs-key \
--target-key-id <key-id-from-above>
```
### Step 2: Configure SeaweedFS
```json
{
"identities": [
{
"name": "admin",
"credentials": [{"accessKey": "admin", "secretKey": "password"}],
"actions": ["Admin", "Read", "Write"]
}
],
"kms": {
"default_provider": "aws-kms",
"providers": {
"aws-kms": {
"type": "aws-kms",
"region": "us-east-1",
"key_id": "alias/seaweedfs-key"
}
}
}
}
```
### Step 3: Test AWS KMS
```bash
# Upload with AWS KMS encryption
aws s3 cp file.txt s3://mybucket/file.txt \
--server-side-encryption aws:kms \
--ssekms-key-id alias/seaweedfs-key
```
---
## Google Cloud KMS Integration
### Step 1: Create KMS Resources
```bash
# Create key ring
gcloud kms keyrings create seaweedfs-keyring --location us-east1
# Create encryption key
gcloud kms keys create seaweedfs-key \
--keyring seaweedfs-keyring \
--location us-east1 \
--purpose encryption
```
### Step 2: Configure SeaweedFS
```json
{
"identities": [
{
"name": "admin",
"credentials": [{"accessKey": "admin", "secretKey": "password"}],
"actions": ["Admin", "Read", "Write"]
}
],
"kms": {
"default_provider": "gcp-kms",
"providers": {
"gcp-kms": {
"type": "gcp-kms",
"project_id": "my-project-id",
"location": "us-east1",
"key_ring": "seaweedfs-keyring",
"key_name": "seaweedfs-key",
"credentials_file": "/etc/seaweedfs/gcp-kms-key.json"
}
}
}
}
```
---
## OpenBao/Vault Integration
### Step 1: Setup OpenBao/Vault
```bash
# Start OpenBao in dev mode (for testing)
openbao server -dev -dev-root-token-id="root-token"
# Enable transit secrets engine
openbao secrets enable transit
# Create encryption key
openbao write -f transit/keys/seaweedfs-key
```
### Step 2: Configure SeaweedFS
```json
{
"identities": [
{
"name": "admin",
"credentials": [{"accessKey": "admin", "secretKey": "password"}],
"actions": ["Admin", "Read", "Write"]
}
],
"kms": {
"default_provider": "openbao",
"providers": {
"openbao": {
"type": "openbao",
"address": "http://localhost:8200",
"token": "root-token",
"transit_path": "transit",
"cache_enabled": true,
"cache_ttl": "1h"
}
}
}
}
```
### Step 3: Test OpenBao/Vault
```bash
# Upload with Vault encryption
aws s3 cp file.txt s3://mybucket/file.txt \
--server-side-encryption aws:kms \
--ssekms-key-id seaweedfs-key
```
---
## Azure Key Vault Integration (Experimental)
Azure Key Vault support exists behind the build tag `azurekms` and is considered experimental. To enable it, build SeaweedFS with the tag and configure the provider:
```bash
# Build with Azure KMS support (example)
go build -tags azurekms ./weed
```
```json
{
"kms": {
"providers": {
"azure": {
"type": "azure",
"vault_url": "https://<your-vault>.vault.azure.net/",
"tenant_id": "<tenant>",
"client_id": "<client>",
"client_secret": "<secret>",
"use_default_creds": false
}
}
}
}
```
---
## Multi-Provider Configuration
### Provider Selection Strategies
```json
{
"identities": [...],
"kms": {
"default_provider": "openbao",
"providers": {
"openbao": {
"type": "openbao",
"address": "https://vault.internal:8200",
"token": "root-token",
"transit_path": "transit"
},
"aws-kms": {
"type": "aws-kms",
"region": "us-east-1",
"key_id": "alias/seaweedfs-aws"
},
"gcp-kms": {
"type": "gcp-kms",
"project_id": "my-gcp-project",
"location": "global",
"key_ring": "seaweedfs-keyring",
"key_name": "seaweedfs-gcp-key",
"credentials_file": "/etc/seaweedfs/gcp-key.json"
}
},
"buckets": {
"financial-data": {"provider": "openbao"},
"ml-models": {"provider": "gcp-kms"},
"general-storage": {"provider": "aws-kms"}
}
}
}
```
## Security Best Practices
### 1. Key Management
- Grant minimal required KMS permissions
- Use resource-based policies where possible
- Implement proper key rotation policies
- Document key usage and ownership
### 2. Access Control
```json
{
"identities": [
{
"name": "admin",
"credentials": [{"accessKey": "admin", "secretKey": "password"}],
"actions": ["Admin", "Read", "Write"]
},
{
"name": "readonly",
"credentials": [{"accessKey": "readonly", "secretKey": "password"}],
"actions": ["Read"]
}
]
}
```
## Related Documentation
- **[Server-Side Encryption](Server-Side-Encryption.md)**: Overview of all SSE types
- **[SSE-C Guide](Server-Side-Encryption-SSE-C.md)**: Customer-provided keys
- **[Amazon S3 API](Amazon-S3-API.md)**: Main S3 API documentation
Server-Side-Encryption.md
+122
@@ -0,0 +1,122 @@
# Server-Side Encryption (SSE)
If you're using SeaweedFS with the S3 API, you can encrypt objects at rest without changing your apps. We support the same server-side encryption (SSE) options as Amazon S3, so you can pick the one that fits how you already manage keys.
## Overview
Use this quick guide to choose the right option:
| Encryption Type | Key Management | Use Case |
|------------------|----------------|----------|
| **SSE-KMS** | External KMS providers | Enterprise key management, audit trails |
| **SSE-C** | Customer-provided | Full customer control, regulatory compliance |
| **SSE-S3** | SeaweedFS-managed | Simple server-managed encryption, bucket defaults |
## Encryption Types
### SSE-KMS (Server-Side Encryption with Key Management Service)
- **Pick this if**: You already use a KMS and want strong audit trails
- **Keys live in**: External providers (AWS KMS, Google Cloud KMS, OpenBao/Vault, Azure Key Vault [experimental])
- **Why teams like it**: Centralized key management, detailed audit logs, per-bucket key assignment, optional Bucket Key optimization
- **Configuration**: Requires KMS provider setup in the S3 config
- **Documentation**: [SSE-KMS Guide](Server-Side-Encryption-SSE-KMS.md)
### SSE-C (Server-Side Encryption with Customer-Provided Keys)
- **Pick this if**: You want to bring your own keys and keep full control
- **Keys live in**: Your application (sent per request)
- **Why teams like it**: No key storage on the server; maximum control for compliance-heavy environments
- **Configuration**: Keys provided via HTTP headers
- **Documentation**: [SSE-C Guide](Server-Side-Encryption-SSE-C.md)
### SSE-S3 (Server-Managed Keys)
- **Pick this if**: You want simple, fully managed encryption with minimal setup
- **Keys live in**: SeaweedFS (we handle the key management for you)
- **Why teams like it**: Works with explicit `x-amz-server-side-encryption: AES256` and bucket default encryption; supports multipart uploads and range requests
- **Configuration**: Optional bucket-level default encryption via the standard S3 bucket encryption API
## Quick Start
### SSE-KMS (Enterprise)
```bash
# Configure KMS in s3 config file (see KMS Providers Integration guide)
# Then upload with KMS encryption
aws s3 cp file.txt s3://mybucket/file.txt \
--server-side-encryption aws:kms \
--ssekms-key-id test-key-123
```
### SSE-C (Customer Keys)
```bash
# Generate customer key
openssl rand 32 > customer-key.bin
# Upload with customer-provided key
aws s3 cp file.txt s3://mybucket/file.txt \
--sse-c AES256 \
--sse-c-key fileb://customer-key.bin
```
### SSE-S3 (Server-Managed)
```bash
# Explicit SSE-S3 on upload (or configure bucket default encryption)
aws s3 cp file.txt s3://mybucket/file.txt \
--server-side-encryption AES256
```
## Configuration
### Basic Setup
Configure KMS providers and IAM settings in your S3 config file:
```json
{
"identities": [
{
"name": "admin",
"credentials": [{"accessKey": "admin", "secretKey": "password"}],
"actions": ["Admin", "Read", "Write"]
}
],
"kms": {
"default_provider": "openbao",
"providers": {
"openbao": {
"type": "openbao",
"address": "http://localhost:8200",
"token": "root-token",
"transit_path": "transit"
}
}
}
}
```
**Note:** The S3 config JSON file contains both KMS encryption settings AND IAM-style access control (user identities, credentials, permissions).
### Start S3 API with Encryption Support
```bash
# Start with KMS config
weed s3 -config=s3_kms_config.json
```
## Features
**Supported Operations:**
- All standard S3 operations (PUT, GET, HEAD, COPY, DELETE)
- Multipart uploads with consistent encryption
- Cross-encryption copy operations
- Object metadata preservation
- Range requests for SSE-C, SSE-KMS, and SSE-S3
**AWS S3 Compatibility:**
- Identical API behavior and headers
- Compatible with all S3 clients and SDKs
- Same error codes and responses
## Implementation Notes
- **SSE-KMS**: Supports AWS KMS, Google Cloud KMS, OpenBao/Vault; Azure Key Vault is available behind the `azurekms` build tag (experimental)
- **SSE-C**: Full support with security best practices
- **SSE-S3**: Supported with SeaweedFS-managed keys and bucket default encryption
For hands-on setup guides and examples, see the individual encryption method docs linked above.
Server-Startup-Setup.md → Server-Startup-via-Systemd.md
+1
@@ -97,6 +97,7 @@ Type=simple
User=root
Group=root
ExecStartPre=/bin/sleep 30
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/weed filer
WorkingDirectory=/media/jmn/HDD/
Store-file-with-a-Time-To-Live.md
+4
@@ -72,3 +72,7 @@ For deploying to production, the TTL volume maximum size should be taken into co
It's recommended not to mix the TTL volumes and non TTL volumes in the same cluster. This is because the volume maximum size, default to 30GB, is configured on the volume master at the cluster level.
We could implement the configuration for max volume size for each TTL. However, it could get fairly verbose. Maybe later if it is strongly desired.
## Via S3 API
This can also be set via the S3 api using LifecycleConfiguration. See the section about that [here](https://github.com/seaweedfs/seaweedfs/wiki/S3-API-FAQ#setting-ttl).
Structured-Data-Lake-with-SMQ-and-SQL.md
+52
@@ -0,0 +1,52 @@
# Structured Data Lake with SMQ and SQL
SeaweedFS + Seaweed Message Queue (SMQ) gives you a unified pipeline: produce structured messages, process them in real time, and query the same data with SQL. Whether producers speak Kafka or a simple pub/sub gRPC API, they both write schematized messages into the same data lake.
## Core ideas
- Schematized messages can be queried directly with SQL (no ETL required)
- SMQ brokers are computation-only nodes and scale linearly with demand
- Structured data is written as messages and can be queried in real time
- Together, SeaweedFS + SMQ form a data lake for structured data (hot streams + Parquet)
## Ingestion paths
Two equivalent ways to ingest structured messages:
- Kafka clients → [[Kafka to Kafka Gateway to SMQ to SQL]]
- Pub/Sub clients → [[Pub-Sub to SMQ to SQL]]
Both end up with the same outcomes: live streams for subscribers and Parquet files in SeaweedFS for SQL engines.
## Architecture
```
Producers (Kafka or Pub/Sub) ==> SMQ Brokers ==> Subscribers (real-time)
\
+--> SeaweedFS (Parquet) ==> SQL Engines
```
## Querying the lake
Point your SQL engines at the Parquet paths:
- Trino/Presto
- Spark SQL
- DuckDB
- ClickHouse (file table engines)
Examples are available in the ingestion pages.
## Operate at scale
- Scale SMQ brokers horizontally; they are stateless computation nodes
- Storage is disaggregated (SeaweedFS) for durable, efficient Parquet files
## Learn more
- Messaging basics: [[Seaweed Message Queue]]
- PostgreSQL-compatible server: [[PostgreSQL-compatible Server weed db]]
- Kafka ingestion: [[Kafka to Kafka Gateway to SMQ to SQL]]
- Pub/Sub ingestion: [[Pub-Sub to SMQ to SQL]]
Super-Large-Directories.md
+2 -2
@@ -53,7 +53,7 @@ So for large directories configured in Redis, SeaweedFS skips this operation, so
The consequences are:
* The directory listing for this folder is not supported.
* The filer meta dada import and export for this folder is not supported. You can still do it for specific child folders though.
* The filer meta data import and export for this folder is not supported. You can still do it for specific child folders though.
* Once this is configured, it can not be changed back easily. You will need to write code to iterate all sub entries for that.
# How to configure it?
@@ -61,7 +61,7 @@ The consequences are:
In `filer.toml` for Cassandra/Redis, there is an option `superLargeDirectories`. For example, if you will have a lot of user data under `/home/users`
```
[cassandra]
[cassandra2]
...
superLargeDirectories = [
"/home/users",
TensorFlow-with-SeaweedFS.md
+1 -1
@@ -18,7 +18,7 @@ model.fit(train_dataset, ...)
```
# TensorFlow on SeaweedFS S3
[TensorFlow already supports S3](https://github.com/tensorflow/examples/blob/master/community/en/docs/deploy/s3.md)
[TensorFlow already supports S3](https://github.com/tensorflow/examples/blame/tflmm/v0.2.4/community/en/docs/deploy/s3.md) (Old Link)
Here is an adaption of it with unnecessary content removed.
Volume-Files-Structure.md
+1
@@ -0,0 +1 @@
![struc](https://github.com/seaweedfs/seaweedfs/assets/36403983/3f29f4ed-a085-4e5e-81fe-1581d1a8b3c7)
Volume-Management.md
+3 -3
@@ -1,4 +1,4 @@
When running large clusters, it is common to add more volume servers, or some volume servers are down, or some volume servers are replaced. These topology changes can cause missing volume replicas, or unbalanced number of volumes on volume servers.
When managing large clusters, it's common to add more volume servers, have some servers go down, or replace others. These changes can lead to missing volume replicas or an uneven distribution of volumes across the servers.
## Optimize volumes
See [[Optimization]] page on how to optimize for concurrent writes and concurrent reads.
@@ -14,7 +14,7 @@ scripts = """
ec.rebuild -force
ec.balance -force
volume.balance -force
volume.fix.replication
volume.fix.replication -force
"""
sleep_minutes = 17 # sleep minutes between each script execution
@@ -65,4 +65,4 @@ The balancing plan will try to evenly spread the number of writable and readonly
Run `weed shell` and `volume.mount -node <host>:<port> -volumeId <id>` to mount a volume file.
You can also execute locally on volume server `kill -s HUP $(pgrep -f "weed volume")` to mount for all new volume files.
To mount all new volume files you can send a hang-up signal to the volume server causing a reload with a command such as `pkill -HUP -f "weed volume"`.
Worker.md
+283
@@ -0,0 +1,283 @@
# This is still work in progress!
# Weed Worker
The `weed worker` command starts a maintenance worker that connects to an admin server to process cluster maintenance tasks.
## Overview
Workers are distributed maintenance agents that connect to the admin server to process various maintenance tasks such as:
- **Vacuum**: Reclaim disk space by removing deleted files
- **Erasure Coding**: Convert volumes to erasure-coded format for storage efficiency
- **Remote Upload**: Upload volumes to remote/cloud storage
- **Replication**: Fix replication issues and maintain data consistency
- **Balance**: Redistribute volumes across volume servers for load balancing
Workers automatically register with the admin server and receive tasks based on their capabilities and current load.
## Usage
```bash
weed worker [options]
```
## Options
| Option | Default | Description |
|--------|---------|-------------|
| `-admin` | localhost:23646 | Admin server address |
| `-capabilities` | vacuum,erasure_coding,balance | Comma-separated list of task types this worker can handle |
| `-maxConcurrent` | 2 | Maximum number of concurrent tasks |
| `-heartbeat` | 30s | Heartbeat interval to admin server |
| `-taskInterval` | 5s | Task request interval |
## Examples
### Basic Usage
```bash
# Start worker connecting to local admin server
weed worker -admin=localhost:23646
# Connect to remote admin server
weed worker -admin=admin.example.com:23646
# Start worker with custom admin server and port
weed worker -admin=192.168.1.100:8080
```
### Capability Configuration
```bash
# Worker that only handles vacuum tasks
weed worker -admin=localhost:23646 -capabilities=vacuum
# Worker that handles vacuum and replication tasks
weed worker -admin=localhost:23646 -capabilities=vacuum,replication
# Worker with all capabilities (default)
weed worker -admin=localhost:23646 -capabilities=vacuum,ec,remote,replication,balance
# Worker using capability aliases
weed worker -admin=localhost:23646 -capabilities=vacuum,ec,remote,replication
```
### Performance Tuning
```bash
# High-performance worker with more concurrent tasks
weed worker -admin=localhost:23646 -maxConcurrent=8
# More frequent task requests for busy clusters
weed worker -admin=localhost:23646 -taskInterval=2s
# Custom heartbeat interval
weed worker -admin=localhost:23646 -heartbeat=10s
```
## Task Capabilities
Workers can be configured to handle specific types of maintenance tasks:
### Available Task Types
| Capability | Description |
|------------|-------------|
| `vacuum` | Reclaim disk space by removing deleted files |
| `erasure_coding` | Convert volumes to erasure-coded format |
| `balance` | Redistribute volumes for load balancing |
## Worker Architecture
### Worker Lifecycle
1. **Registration**: Worker connects to admin server via gRPC
2. **Capabilities**: Worker reports its capabilities to admin
3. **Task Request**: Worker periodically requests tasks from admin
4. **Task Execution**: Worker processes assigned tasks
5. **Heartbeat**: Worker sends periodic heartbeats to admin
6. **Graceful Shutdown**: Worker completes current tasks before stopping
### Connection Details
- **Protocol**: gRPC connection to admin server
- **Port**: Admin HTTP port + 10000 (e.g., admin on 23646 → gRPC on 33646)
- **Security**: Supports TLS using `[grpc.worker]` configuration
- **Fallback**: Falls back to insecure connection if TLS unavailable
## Configuration
### Security Configuration
Workers read TLS configuration from `security.toml`:
```toml
[grpc.worker]
cert = "/etc/ssl/worker.crt"
key = "/etc/ssl/worker.key"
ca = "/etc/ssl/ca.crt"
```
### Worker Identification
- **Worker ID**: Automatically generated unique identifier
- **Address**: Worker's network address (auto-detected)
- **Capabilities**: Reported task capabilities
- **Status**: Current worker status (active, idle, busy)
## Task Processing
### Concurrent Task Handling
- **Max Concurrent**: Configurable via `-maxConcurrent` (default: 2)
- **Task Queue**: Workers maintain internal task queues
- **Load Balancing**: Admin distributes tasks based on worker load
- **Task Completion**: Workers report task completion status
### Task Request Cycle
1. Worker requests tasks from admin server
2. Admin assigns tasks based on worker capabilities and load
3. Worker processes tasks concurrently
4. Worker reports task completion/failure
5. Cycle repeats based on `-taskInterval`
## Monitoring and Status
### Worker Status
Workers report the following status information:
- **Worker ID**: Unique identifier
- **Current Load**: Number of active tasks
- **Capabilities**: Supported task types
- **Last Heartbeat**: Timestamp of last heartbeat
- **Tasks Completed**: Total completed tasks
- **Tasks Failed**: Total failed tasks
- **Uptime**: Worker uptime duration
### Health Monitoring
- **Heartbeat**: Periodic heartbeat to admin server
- **Task Timeout**: Tasks have configurable timeouts
- **Error Reporting**: Failed tasks are reported to admin
- **Automatic Retry**: Failed tasks may be retried
## Best Practices
### Deployment
1. **Multiple Workers**: Deploy multiple workers for redundancy
2. **Capability Specialization**: Consider specialized workers for specific tasks
3. **Resource Allocation**: Ensure adequate CPU and memory for concurrent tasks
4. **Network Connectivity**: Ensure reliable connection to admin server
### Performance
1. **Concurrent Tasks**: Tune `-maxConcurrent` based on available resources
2. **Task Interval**: Adjust `-taskInterval` based on cluster activity
3. **Heartbeat Frequency**: Balance between responsiveness and overhead
4. **Resource Monitoring**: Monitor worker resource usage
### Security
1. **TLS Configuration**: Use TLS for production deployments
2. **Network Security**: Secure communication between workers and admin
3. **Access Control**: Limit worker deployment to trusted systems
4. **Certificate Management**: Manage and rotate TLS certificates
## Troubleshooting
### Common Issues
1. **Cannot connect to admin server**:
- Verify admin server address and port
- Check network connectivity
- Ensure admin server is running
- Verify gRPC port (admin HTTP port + 10000)
2. **No tasks received**:
- Check worker capabilities match available tasks
- Verify worker registration with admin
- Check admin server logs for task assignment
- Ensure worker is not overloaded
3. **TLS connection failures**:
- Verify `security.toml` configuration
- Check certificate paths and permissions
- Ensure certificates are valid
- Check certificate compatibility
4. **Task execution failures**:
- Check worker logs for error details
- Verify worker has necessary permissions
- Check disk space and resources
- Ensure target volumes are accessible
### Debug Information
Enable debug logging:
```bash
# Run with verbose logging
weed worker -admin=localhost:23646 -v=4
```
### Worker Logs
Workers log important events:
- Connection status to admin server
- Task assignments and completion
- Error conditions and failures
- Heartbeat and health information
## Task-Specific Information
### Vacuum Tasks
- **Purpose**: Reclaim disk space from deleted files
- **Requirements**: Access to volume servers
- **Duration**: Varies based on volume size and deleted data
- **Impact**: Temporary increase in I/O during vacuum process
### Erasure Coding Tasks
- **Purpose**: Convert volumes to erasure-coded format
- **Requirements**: Multiple volume servers for redundancy
- **Duration**: Long-running, depends on volume size
- **Impact**: Reduces storage requirements but increases complexity
### Remote Upload Tasks
- **Purpose**: Upload volumes to remote/cloud storage
- **Requirements**: Cloud storage credentials and connectivity
- **Duration**: Depends on volume size and upload bandwidth
- **Impact**: Enables tiered storage and backup strategies
### Replication Tasks
- **Purpose**: Fix replication consistency issues
- **Requirements**: Access to master and volume servers
- **Duration**: Quick, depends on replication factor
- **Impact**: Ensures data consistency and availability
### Balance Tasks
- **Purpose**: Redistribute volumes across volume servers
- **Requirements**: Multiple volume servers
- **Duration**: Depends on data movement requirements
- **Impact**: Improves cluster load distribution
## Related Commands
- [`weed admin`](Weed-Admin.md): Start admin server that manages workers
- [`weed master`](https://github.com/seaweedfs/seaweedfs/wiki/Master-Server): Start master servers
- [`weed volume`](https://github.com/seaweedfs/seaweedfs/wiki/Volume-Server): Start volume servers
- [`weed scaffold`](https://github.com/seaweedfs/seaweedfs/wiki/Scaffold): Generate configuration files
## See Also
- [SeaweedFS Architecture](https://github.com/seaweedfs/seaweedfs/wiki/SeaweedFS-Architecture)
- [Maintenance Operations](https://github.com/seaweedfs/seaweedfs/wiki/Maintenance)
- [Security Configuration](https://github.com/seaweedfs/seaweedfs/wiki/Security-Configuration)
- [Erasure Coding](https://github.com/seaweedfs/seaweedfs/wiki/Erasure-Coding)
- [Remote Storage](https://github.com/seaweedfs/seaweedfs/wiki/Remote-Storage)
_Sidebar.md
+30 -2
@@ -18,12 +18,13 @@
* [[Store file with a Time To Live]]
* [[Failover Master Server]]
* [[Erasure coding for warm storage]]
* [[Server Startup Setup]]
* [[Server Startup via Systemd]]
* [[Environment Variables]]
### Filer
* [[Filer Setup]]
* [[Directories and Files]]
* [[File Operations Quick Reference]]
* [[Data Structure for Large Files]]
* [[Filer Data Encryption]]
* [[Filer Commands and Operations]]
@@ -37,6 +38,10 @@
* [[Choosing a Filer Store]]
* [[Customize Filer Store]]
### Management
* [[Admin UI]]
* [[Worker]]
### Advanced Filer Configurations
* [[Migrate to Filer Store]]
* [[Add New Filer Store|Customize Filer Store]]
@@ -48,6 +53,7 @@
### [[FUSE Mount]]
* [[FIO benchmark]]
* [[fstab]]
### [[WebDAV]]
@@ -61,7 +67,12 @@
* [[Gateway to Remote Object Storage]]
### AWS S3 API
* [[S3 Credentials]]
* [[Amazon S3 API]]
* [[S3 Conditional Operations]]
* [[S3 CORS]]
* [[S3 Object Lock and Retention]]
* [[S3 Object Versioning]]
* [[AWS CLI with SeaweedFS]]
* [[s3cmd with SeaweedFS]]
* [[rclone with SeaweedFS]]
@@ -72,10 +83,17 @@
* [[S3 Bucket Quota]]
* [[S3 API Audit log]]
* [[S3 Nginx Proxy]]
* [[Docker Compose for S3]]
### Server-Side Encryption
* [[Server-Side Encryption]]
* [[Server-Side Encryption SSE-KMS]]
* [[Server-Side Encryption SSE-C]]
### AWS IAM
* [[Amazon IAM API]]
* [[AWS IAM CLI]]
* [[Keycloak Integration]]
### Machine Learning
* [[TensorFlow with SeaweedFS]]
@@ -95,9 +113,18 @@
* [[Async Replication to Cloud]] [Deprecated]
* [[Kubernetes Backups and Recovery with K8up]]
### Messaging
### Metadata Change Events
* [[Filer Metadata Events]]
### Messaging
* [[Structured Data Lake with SMQ and SQL]]
* [[Seaweed Message Queue]]
* [[SQL Queries on Message Queue]]
* [[SQL Quick Reference]]
* [[PostgreSQL-compatible Server weed db]]
* [[Pub-Sub to SMQ to SQL]]
* [[Kafka to Kafka Gateway to SMQ to SQL]]
### Use Cases
* [[Use Cases]]
* [[Actual Users]]
@@ -117,6 +144,7 @@
* [[Cloud Monitoring]]
* [[Load Command Line Options from a file]]
* [[SRV Service Discovery]]
* [[Volume Files Structure]]
### Security
* [[Security Overview]]
fstab.md
+30
@@ -0,0 +1,30 @@
To mount SeaweedFS using `/etc/fstab` (such as on boot):
* Follow the directions in https://github.com/seaweedfs/seaweedfs/wiki/FUSE-Mount for setting up a SeaweedFS mount subtype for FUSE (hint: `cp weed /sbin/weed`)
* Install SeaweedFS as usual, making sure any specific settings (such as `security.toml`) are in the correct location to be read
* Add the fstab entry as described below
* But make sure that `systemd-fstab-generator(8)` is not used on your system
If you have a single filer server, this is the syntax you will use:
`fuse /path/to/mountpoint fuse.weed filer=localhost:8888,filer.path=/,defaults,_netdev 0 0`
If you have multiple filer servers, this is the syntax you will use:
`fuse /path/to/mountpoint fuse.weed filer='192.168.0.1:8888,192.168.0.2:8888',filer.path=/,defaults,_netdev 0 0`
Place the appropriate line into `/etc/fstab` and attempt to mount your filesystem. If you encounter any issues, run `weed mount` directly in verbose/debug mode to diagnose the issue, and switch back to fstab-style mounting once you have resolved the issues.
## Systemd
* No matter what systemd options (`nofail`, `x-systemd.device-timeout`, `x-systemd.mount-timeout`, etc.) you add to /etc/fstab, you won’t be able to make `systemd.mount(5)` handle the mount properly. You will always get an error when starting mount unit, even though the filesystem ends up being mounted.
* `x-systemd.automount` doesn’t work at all.
* The `_netdev` results in errors like:
```log
kernel: fuse: Unknown parameter '_netdev'
```
p.s. Not sure if it’s related. (systemd 255.6)
In general, if your system is deeply integrated with systemd, it is better to create `systemd.service(5)` to mount.
nodejs-with-Seaweed-S3.md
+5
@@ -20,6 +20,11 @@ const s3client = new S3Client({
region: `us-east-1`,
// force path style for compatibility reasons
forcePathStyle: true,
// dual stack endpoint is not supported by seaweed
useDualstackEndpoint: false,
// checksum validation should be disabled, overwise `x-amz-checksum` will be injected directly into files
responseChecksumValidation: `WHEN_REQUIRED`,
requestChecksumCalculation: 'WHEN_REQUIRED',
// credentials is mandatory and s3 authorization should be enabled with `s3.configure`
credentials: {
accessKeyId: `same as -accesskey`,
rclone-with-SeaweedFS.md
+17
@@ -19,6 +19,23 @@ chunk_size = 50Mi
force_path_style = true
```
### Reverse proxy sub-path configuration
This is undefined behavior as AWS S3 servers always have sub-domains instead of sub-paths. \
Use this only if you can't create (sub-)domain and use other port!
rclone (like AWS CLI) appends sub-path before actual path so need to add `X-Forwarded-Prefix` header (set to `/s3` for example)
Example for Caddy web server
```
redir /s3 /s3/
handle_path /s3/* {
reverse_proxy localhost:8333 {
header_up X-Forwarded-Prefix /s3
}
}
```
### Execute commands
copy files
restic-with-SeaweedFS.md
+25 -5
@@ -3,14 +3,31 @@ See https://restic.github.io
On mac: `brew install restic`
### Configuration
Set these environment variables. The key values do not matter.
### Weed Configuration
Set the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables in both your `weed` environment as well as your `restic` environment.
The key values do not matter _but they must be provided_, as `restic` expects (and effectively requires) non-anonymous access when communicating with an S3 bucket.
This may involve configuring your `weed` server similar to the following:
```bash
export AWS_ACCESS_KEY_ID=any-key-id
export AWS_SECRET_ACCESS_KEY=any-access-key
export AWS_ACCESS_KEY_ID="any-key-id"
export AWS_SECRET_ACCESS_KEY="any-access-key"
export WEED_S3_CONFIGURE_CMD="s3.configure \
-access_key \"$AWS_ACCESS_KEY_ID\" \
-secret_key \"$AWS_SECRET_ACCESS_KEY\" \
-user iam \
-actions Admin \
-apply"
echo "$WEED_S3_CONFIGURE_CMD" | weed shell
```
### Execute commands
### Execute Restic Commands
See: https://github.com/seaweedfs/seaweedfs/wiki/s3cmd-with-SeaweedFS
First, create the bucket:
```bash
s3cmd mb s3://resticbucket
@@ -18,6 +35,9 @@ s3cmd mb s3://resticbucket
Then, initialize the restic bucket and backup to it:
```console
$ export AWS_ACCESS_KEY_ID="any-key-id"
$ export AWS_SECRET_ACCESS_KEY="any-access-key"
$ restic -r s3:http://localhost:8333/resticbucket init
$ restic -r s3:http://localhost:8333/resticbucket backup /Users/chris/dev/gopath/bin/
s3cmd-with-SeaweedFS.md
+31 -12
@@ -30,11 +30,30 @@ Make sure the `.s3cfg` file has these values
# Setup endpoint
host_base = localhost:8333
host_bucket = localhost:8333
use_https = No
use_https = False
# Enable S3 v4 signature APIs
signature_v2 = False
```
### Reverse proxy sub-path configuration
This is undefined behavior as AWS S3 servers always have sub-domains instead of sub-paths. \
Use this only if you can't create (sub-)domain and use other port!
For s3cmd, `X-Forwarded-Host` is required to be `your.server/s3` but not for other utils
so it's better to use `signature_v2 = True` in s3cfg
Example for Caddy web server
```
redir /s3 /s3/
handle_path /s3/* {
reverse_proxy localhost:8333 {
header_up X-Forwarded-Host your.server/s3
header_up X-Forwarded-Prefix /s3
}
}
```
### Execute commands
```
$ s3cmd mb s3://newbucket
@@ -43,31 +62,31 @@ Bucket 's3://newbucket/' created
$ s3cmd ls s3://
2019-01-01 01:30 s3://newbucket
$ s3cmd put /etc/passwd s3://newbucket
$ s3cmd put /etc/motd s3://newbucket
WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
upload: '/etc/passwd' -> 's3://newbucket/passwd' [1 of 1]
upload: '/etc/motd' -> 's3://newbucket/motd' [1 of 1]
6804 of 6804 100% in 0s 87.93 kB/s done
$ s3cmd get s3://newbucket/passwd
download: 's3://newbucket/passwd' -> './passwd' [1 of 1]
$ s3cmd get s3://newbucket/motd
download: 's3://newbucket/motd' -> './motd' [1 of 1]
6804 of 6804 100% in 0s 595.33 kB/s done
# change the file
$ vi passwd
$ vi motd
$ s3cmd sync passwd s3://newbucket/
$ s3cmd sync motd s3://newbucket/
WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
upload: 'passwd' -> 's3://newbucket/passwd' [1 of 1]
upload: 'motd' -> 's3://newbucket/motd' [1 of 1]
22 of 22 100% in 0s 6.45 kB/s done
Done. Uploaded 22 bytes in 1.0 seconds, 22.00 B/s.
$ s3cmd ls s3://newbucket/
2019-01-01 01:32 22 s3://newbucket/passwd
2019-01-01 01:32 22 s3://newbucket/motd
$ s3cmd del s3://newbucket/passwd
delete: 's3://newbucket/passwd'
$ s3cmd del s3://newbucket/motd
delete: 's3://newbucket/motd'
$ s3cmd rb s3://newbucket
Bucket 's3://newbucket/' removed
```
```
weed-shell.md
+1 -1
@@ -120,7 +120,7 @@ volume 2480 fast-volume-1:8080 has 163827 entries, fast-volume-3:8080 missed 0 a
volume 2480 fast-volume-3:8080 has 163827 entries, fast-volume-1:8080 missed 0 and partially deleted 0 entries
```
2. `volume.fsck`
Search for files that are in the filler, but there are no chunks on the volume servers
Search for files that are in the filer, but there are no chunks on the volume servers
```
lock;volume.fsck -findMissingChunksInFiler -verifyNeedles -collection logs-data -volumeId 2480 -v
checking directory /buckets/logs-data/2023-02-13