* volume growth: spread writable volumes of a layout over servers and disks Volume growth picked every server, and the volume server picked every disk, independently: the server weighted by free slots, the disk by the lowest total volume count. A grow request therefore stacks several writable volumes of one collection on some disks while most disks get none (a Poisson spread: with 160 replicas on 479 disks about 20 disks get two or three), and their writes share those disks. Data centers and racks are still picked weighted by free slots. Inside a rack the server is now one of those holding the fewest writable volumes of the layout per disk, weighted by free slots among them; this covers the main server, same-rack replicas and replicas walked into other racks and data centers. On the volume server the disk is the one with the fewest writable volumes of the collection, then the lowest volume count as before. A grow request registers each volume before placing the next, so a request smaller than the number of disks lands on distinct disks. Fixes #12746 * volume growth: scan the layout load once per grow request Placing each volume rescanned all writable volumes of the layout under its read lock. RecordAssign takes that lock exclusively on every assign, so with thousands of writables (0.8 ms for 10000) every volume of a grow request stalled assigns once more. GrowByCountAndType now scans once and findAndGrow adds every volume it creates to the counts. Per volume, placement costs about 130-160 us on 480 servers whatever the number of writables, against 120 us before the spread change; the single scan is 0.17 ms for 2000 writables and 0.8 ms for 10000. * volume growth: address review of the spread placement - Count only disks that report volume slots in a server's load. An unavailable disk, or one low on space with no volumes, is still listed in DiskTags with no slots; counting it understated the load of a server with failed disks and piled volumes onto its remaining ones. - On the volume server, judge a volume full by its .dat size, which the master compares with the limit, not by ContentSize. - A server that refuses an allocation ranks last for the rest of the grow request, and the request retries the volume up to 3 times. The least loaded server is picked first deterministically, so without this one server that cannot fit a preallocated volume stopped the layout from growing. - Scan the layout load after warmup, so a grow request issued right after a leader change sees the writable volumes the returning servers report. * volume growth: keep the request-local placement load ahead of the request fields The JSON request fields stay together at the end of VolumeGrowOption, and the comment now says the load lives for the whole grow request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): place a new volume on the disk with the fewest writable volumes of its collection Store::add_volume picked the disk with the most free slots, so the Rust volume server ignored the collection spread rule the Go server now applies. find_free_location ranks disks by the writable volumes of the collection first (not read-only, .dat below the volume size limit), then by free slots. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume growth: leave a refusing server out of every placement tier A server that refused an allocation only ranked last among the servers of its rack, so the weighted data center and rack picks could still choose its rack on every retry while other racks had room. It now offers no free slots for the rest of the grow request, which takes its slots out of the rack and data center weights too. If only refusing servers are left, they are tried again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume growth: require the full same-rack count only in the main rack The data center check asked every rack for SameRackCount+1 free servers, but only the main rack holds the same-rack copies; each other rack takes one. With a refused server left out of its rack, the check rejected a feasible placement, the retry put the refused server back, and it was picked again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
SeaweedFS Volume Server (Rust)
A drop-in replacement for the SeaweedFS Go volume server, rewritten in Rust. It uses binary-compatible storage formats (.dat, .idx, .vif) and speaks the same HTTP and gRPC protocols, so it works with an unmodified Go master server.
Building
Requires Rust 1.91.1+ (2024 edition), matching rust-version in Cargo.toml.
The patch release matters: 1.91.0 does not build. The edition itself only needs
1.85; the higher floor comes from the dependency tree — chiefly the AWS SDK — so
it moves with those crates. CI builds on the latest stable.
cd seaweed-volume
cargo build --release
The binary is produced at target/release/seaweed-volume.
Running
Start a Go master server first, then point the Rust volume server at it:
# Minimal
seaweed-volume --port 8080 --master localhost:9333 --dir /data/vol1 --max 7
# Multiple data directories
seaweed-volume --port 8080 --master localhost:9333 \
--dir /mnt/ssd1,/mnt/ssd2 --max 100,100 --disk ssd
# With datacenter/rack topology
seaweed-volume --port 8080 --master localhost:9333 --dir /data/vol1 --max 7 \
--dataCenter dc1 --rack rack1
# With JWT authentication
seaweed-volume --port 8080 --master localhost:9333 --dir /data/vol1 --max 7 \
--securityFile /etc/seaweedfs/security.toml
# With TLS (configured in security.toml via [https.volume] and [grpc.volume] sections)
seaweed-volume --port 8080 --master localhost:9333 --dir /data/vol1 --max 7 \
--securityFile /etc/seaweedfs/security.toml
Common flags
| Flag | Default | Description |
|---|---|---|
--port |
8080 |
HTTP listen port |
--port.grpc |
port+10000 |
gRPC listen port |
--master |
localhost:9333 |
Comma-separated master server addresses |
--dir |
/tmp |
Comma-separated data directories |
--max |
8 |
Max volumes per directory (comma-separated) |
--ip |
auto-detect | Server IP / identifier |
--ip.bind |
same as --ip |
Bind address |
--dataCenter |
Datacenter name | |
--rack |
Rack name | |
--disk |
Disk type tag: hdd, ssd, or custom |
|
--index |
memory |
Needle map type: memory, leveldb, leveldbMedium, leveldbLarge |
--readMode |
proxy |
Non-local read mode: local, proxy, redirect |
--fileSizeLimitMB |
256 |
Max upload file size |
--minFreeSpace |
1 (percent) |
Min free disk space before marking volumes read-only |
--securityFile |
Path to security.toml for JWT keys and TLS certs |
|
--metricsPort |
0 (disabled) |
Prometheus metrics endpoint port |
--whiteList |
Comma-separated IPs with write permission | |
--preStopSeconds |
10 |
Graceful drain period before shutdown |
--compactionMBps |
0 (unlimited) |
Compaction I/O rate limit |
--compactionMaxParallelCommits |
0 (unlimited) |
Max concurrent vacuum commits |
--pprof |
false |
Enable pprof HTTP handlers |
Set RUST_LOG=debug (or trace, info, warn) for log level control.
Durable (fsync=true) uploads always go through per-volume batching workers
that share one .dat/.idx sync per batch, matching the Go volume server;
non-fsync writes take the synchronous path.
Features
- Binary compatible -- reads and writes the same
.dat/.idx/.viffiles as the Go server; seamless migration with no data conversion. - HTTP + gRPC -- full implementation of the volume server HTTP API and all gRPC RPCs including streaming operations (copy, tail, incremental copy, vacuum).
- Master heartbeat -- bidirectional streaming heartbeat with the Go master server; volume and EC shard registration, leader failover, graceful shutdown deregistration.
- JWT authentication -- signing key configuration via
security.tomlwith token source precedence (query > header > cookie), file_id claims validation, and separate read/write keys. - TLS -- HTTPS for the HTTP API and mTLS for gRPC, configured through
security.toml. - Erasure coding -- Reed-Solomon EC shard management: mount/unmount, read, rebuild, copy, delete, and shard-to-volume reconstruction.
- S3 remote storage --
FetchAndWriteNeedlereads from any S3-compatible backend (AWS, MinIO, Wasabi, Backblaze, etc.) and writes locally. SupportsVolumeTierMoveDatToRemote/FromRemotefor tiered storage. - Needle map backends -- in-memory HashMap, LevelDB (via
rusty-leveldb), or redb (pure Rust disk-backed) needle maps. - Image processing -- on-the-fly resize/crop, JPEG EXIF orientation auto-fix, WebP support.
- Streaming reads -- large files (>1MB) are streamed via
spawn_blockingto avoid blocking the async runtime. - Auto-compression -- compressible file types (text, JSON, CSS, JS, SVG, etc.) are gzip-compressed on upload.
- Prometheus metrics -- counters, histograms, and gauges exported at a dedicated metrics port; optional push gateway support.
- Graceful shutdown -- SIGINT/SIGTERM handling with configurable
preStopSecondsdrain period.
Testing
Rust unit tests
cd seaweed-volume
cargo test
Go integration tests
The Go test suite can target either the Go or Rust volume server via the VOLUME_SERVER_IMPL environment variable:
# Run all HTTP + gRPC integration tests against the Rust server
VOLUME_SERVER_IMPL=rust go test -v -count=1 -timeout 1200s \
./test/volume_server/grpc/... ./test/volume_server/http/...
# Run a single test
VOLUME_SERVER_IMPL=rust go test -v -count=1 -timeout 60s \
-run "TestName" ./test/volume_server/http/...
# Run S3 remote storage tests
VOLUME_SERVER_IMPL=rust go test -v -count=1 -timeout 180s \
-run "TestFetchAndWriteNeedle" ./test/volume_server/grpc/...
Load testing
A load test harness is available at test/volume_server/loadtest/. See that directory for usage instructions and scenarios.
Architecture
The server runs three listeners concurrently:
- HTTP (Axum 0.7) -- admin and public routers for file upload/download, status, and stats endpoints.
- gRPC (Tonic 0.12) -- all
VolumeServerRPCs from the SeaweedFS protobuf definition. - Metrics (optional) -- Prometheus scrape endpoint on a separate port.
Key source modules:
| Path | Description |
|---|---|
src/main.rs |
Entry point, server startup, signal handling |
src/config.rs |
CLI parsing and configuration resolution |
src/server/volume_server.rs |
HTTP router setup and middleware |
src/server/handlers.rs |
HTTP request handlers (read, write, delete, status) |
src/server/grpc_server.rs |
gRPC service implementation |
src/server/heartbeat.rs |
Master heartbeat loop |
src/storage/volume.rs |
Volume read/write/delete logic |
src/storage/needle.rs |
Needle (file entry) serialization |
src/storage/store.rs |
Multi-volume store management |
src/security.rs |
JWT validation and IP whitelist guard |
src/remote_storage/ |
S3 remote storage backend |
See DEV_PLAN.md for the full development history and feature checklist.