mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-15 19:10:48 +02:00
Eliah RusinandChris Lu
cfa8afec92
* filer: guard the FoundationDB value size limit, not the transaction limit An entry's whole chunk list is one FoundationDB value, and FDB caps a value at 100,000 bytes while a transaction may reach 10MB. UpdateEntry checked the transaction limit, so every entry between the two limits passed the guard and was rejected by FDB itself with error 2103 (Value length exceeds limit). The failure surfaced inside the store rather than at the guard, so the S3 layer dropped the connection and clients saw a network fault instead of an error. Check the value limit in UpdateEntry and KvPut instead, after gzip and before the transaction, with an error that names the limit it hit. The removed transaction-size constant guarded nothing else: DeleteFolderChildren batches by entry count. Refs #11158 * filer: fold at 500 chunks in the foundationdb build Manifest packing is what keeps a large file's entry small, but it only ran once a flat chunk list reached 10000 chunks. A FoundationDB value stops at 100,000 bytes and an entry's whole chunk list is one value, which at ~100 bytes per chunk record is about 1000 chunks -- so on FDB the write always failed before packing could help: a 3.3 GiB PutObject at the default -maxMB=4 was already past the limit. FoundationDB support is its own build (`go build -tags foundationdb`, shipped as its own image), so the batch is a build-time choice and needs no negotiation at run time. The tagged build folds at 500, every other build keeps 10000 and is untouched. 500 is not arbitrary: a single fold level leaves (chunks/batch) manifest pointers plus up to (batch-1) unfolded chunks in the entry, so the reachable chunk count is highest when the two terms are near equal. For a 100,000-byte budget that optimum is 500, which holds an entry inside the limit up to ~250,000 chunks -- ~1 TB at -maxMB=4, against ~4 GB before. Larger files need nested packing, which no batch size substitutes for. One binary serves every role in that image, so the filer and each client that folds -- S3, mount, WebDAV, weed shell, filer.copy -- agree on the batch by construction. A binary built with the tag but pointed at another store folds earlier than that store requires, costing one manifest blob per 500 chunks and one read to resolve it. Fixes #11158 * filer: fold with rollback inside MaybeManifestize, not beside it A fold that fails midway has already uploaded manifest blobs for its earlier batches, and returns only the data chunks -- dropping the manifests it had separated out of the caller's list. Both were wrong in ways that mattered: - AppendToEntry assigned that shortened list straight to entry.Chunks and created the entry, so an append to an already-folded file whose fold failed lost every previously folded chunk. weed mount had the same shape. - cleanupChunks logged the error as "not good, but should be ok" and then returned it through a named result, failing the whole CreateEntry or UpdateEntry, while the blobs it had written stayed behind referenced by nothing. The S3 path was alone in handling this, through a private helper beside MaybeManifestize. A second entry point next to the one everything else calls just means the wrong one gets used, so the behaviour moves inside MaybeManifestize: on failure it returns inputChunks as it received them, and hands the blobs it saved to a deleteChunks callback. The filer, S3 and filer.copy pass their existing deleters -- filer.copy already cleans up this way after a failed upload -- and mount, WebDAV and weed shell pass nil, which reports the blobs rather than collecting them, as before. Each caller keeps its own error policy: the filer HTTP PUT path and filer.copy still fail the request, the rest still continue with the flat list, which is a correct entry. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>
FoundationDB Filer Store
This package provides a FoundationDB-based filer store for SeaweedFS, offering ACID transactions and horizontal scalability.
Features
- ACID Transactions: Strong consistency guarantees with full ACID properties
- Horizontal Scalability: Automatic data distribution across multiple nodes
- High Availability: Built-in fault tolerance and automatic failover
- Efficient Directory Operations: Optimized for large directory listings
- Key-Value Support: Full KV operations for metadata storage
- Compression: Automatic compression for large entry chunks
Installation
Prerequisites
- FoundationDB Server: Install and configure a FoundationDB cluster
- FoundationDB Client Libraries: Install libfdb_c client libraries
- Go Build Tags: Use the
foundationdbbuild tag when compiling
Building SeaweedFS with FoundationDB Support
go build -tags foundationdb -o weed
Configuration
Basic Configuration
Add the following to your filer.toml:
[foundationdb]
enabled = true
cluster_file = "/etc/foundationdb/fdb.cluster"
api_version = 740
timeout = "5s"
max_retry_delay = "1s"
directory_prefix = "seaweedfs"
Configuration Options
| Option | Description | Default | Required |
|---|---|---|---|
enabled |
Enable FoundationDB filer store | false |
Yes |
cluster_file |
Path to FDB cluster file | /etc/foundationdb/fdb.cluster |
Yes |
api_version |
FoundationDB API version | 740 |
No |
timeout |
Operation timeout duration | 5s |
No |
max_retry_delay |
Maximum retry delay | 1s |
No |
directory_prefix |
Directory prefix for organization | seaweedfs |
No |
batch_enabled |
Enable write batching (see Performance section) | false |
No |
batch_size |
Max operations per batch | 100 |
No |
batch_interval |
Max time before batch flush | 1ms |
No |
Path-Specific Configuration
For path-specific filer stores:
[foundationdb.backup]
enabled = true
cluster_file = "/etc/foundationdb/fdb.cluster"
directory_prefix = "seaweedfs_backup"
location = "/backup"
Environment Variables
Configure via environment variables:
export WEED_FOUNDATIONDB_ENABLED=true
export WEED_FOUNDATIONDB_CLUSTER_FILE=/etc/foundationdb/fdb.cluster
export WEED_FOUNDATIONDB_API_VERSION=740
export WEED_FOUNDATIONDB_TIMEOUT=5s
export WEED_FOUNDATIONDB_MAX_RETRY_DELAY=1s
export WEED_FOUNDATIONDB_DIRECTORY_PREFIX=seaweedfs
FoundationDB Cluster Setup
Single Node (Development)
# Start FoundationDB server
foundationdb start
# Initialize database
fdbcli --exec 'configure new single ssd'
Multi-Node Cluster (Production)
- Install FoundationDB on all nodes
- Configure cluster file (
/etc/foundationdb/fdb.cluster) - Initialize cluster:
fdbcli --exec 'configure new double ssd'
Docker Setup
Use the provided docker-compose.yml in test/foundationdb/:
cd test/foundationdb
make setup
Performance Considerations
Write Batching Configuration
By default, write batching is disabled (batch_enabled = false). Each write commits
immediately in its own transaction. This provides optimal latency for S3 PUT operations.
When to enable batching:
- High-throughput bulk ingestion workloads
- Scenarios where you can tolerate slightly higher per-operation latency
- Workloads with many concurrent small writes
Batching configuration options:
[foundationdb]
# Enable write batching (disabled by default for optimal S3 latency)
batch_enabled = true
# Maximum operations per batch
batch_size = 100
# Maximum time to wait before flushing a batch
batch_interval = "1ms"
Performance comparison:
- Batching disabled: Each S3 PUT commits immediately (~1-5ms per op depending on FDB latency)
- Batching enabled: Operations are grouped, reducing total commits but adding batch interval latency
Optimal Configuration
- API Version: Use the latest stable API version (720+)
- Directory Structure: Use logical directory prefixes to isolate different SeaweedFS instances
- Transaction Size: Keep transactions under 10MB (FDB limit)
- Concurrency: Use multiple client connections for parallel operations
Monitoring
Monitor FoundationDB cluster status:
fdbcli --exec 'status'
fdbcli --exec 'status details'
Scaling
FoundationDB automatically handles:
- Data distribution across nodes
- Load balancing
- Automatic failover
- Storage node addition/removal
Testing
Unit Tests
cd weed/filer/foundationdb
go test -tags foundationdb -v
Integration Tests
cd test/foundationdb
make test
End-to-End Tests
cd test/foundationdb
make test-e2e
Troubleshooting
Common Issues
-
Connection Failures:
- Verify cluster file path
- Check FoundationDB server status
- Validate network connectivity
-
Transaction Conflicts:
- Reduce transaction scope
- Implement retry logic
- Check for concurrent operations
-
Performance Issues:
- Monitor cluster health
- Check data distribution
- Optimize directory structure
Debug Information
Enable verbose logging:
weed -v=2 server -filer
Check FoundationDB status:
fdbcli --exec 'status details'
Security
Network Security
- Configure TLS for FoundationDB connections
- Use firewall rules to restrict access
- Monitor connection attempts
Data Encryption
- Enable encryption at rest in FoundationDB
- Use encrypted connections
- Implement proper key management
Limitations
- Maximum transaction size: 10MB
- Single transaction timeout: configurable (default 5s)
- API version compatibility required
- Requires FoundationDB cluster setup
Support
For issues specific to the FoundationDB filer store:
- Check FoundationDB cluster status
- Verify configuration settings
- Review SeaweedFS logs with verbose output
- Test with minimal reproduction case
For FoundationDB-specific issues, consult the FoundationDB documentation.