Files
seaweedfs/seaweed-volume
Chris LuandClaude Opus 5.5 9cd183b04f volume: fsync a needle delete that asks for it (#12736)
* volume: fsync a needle delete that asks for it

A delete now takes fsync like a write: the tombstone is synced before the
needle map drops the needle, and the .idx row before the delete is
acknowledged. A failed .dat sync takes the tombstone back off the .dat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: honor fsync=true on DELETE and forward it to replicas

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): fsync a needle delete that asks for it

Mirrors the Go volume server: DELETE with fsync=true flushes the tombstone
before the index drops the needle and the .idx before the ack, and a failed
.dat flush takes the tombstone back off the .dat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: let a durable delete through on a full volume

A queued delete hit the batch worker's volume size check, which a
tombstone append is exempt from inline, so fsync=true deletes failed once
the volume reached its hard limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: sync a durable delete of an already-deleted needle

An earlier delete without fsync may have dropped the needle with its
tombstone still unsynced, so the inline path syncs before acking anyway,
as the batch worker already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: refuse a batch request that races the worker stopping

stopWorker can close the batch channel between startWorker handing it out
and the send, which panicked. The request now falls back to the inline path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): sync a durable delete of an already-deleted needle

Mirrors the Go volume server: an earlier delete without fsync may have
dropped the needle with its tombstone still unsynced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): run a durable delete on a blocking thread

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep syncDelete's signature for existing callers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: refuse a write a delete cancelled before it appends

A write cancelled while still queued behind the volume locks used to
append, publish and then tombstone itself without a sync, after its
delete may already have been acknowledged as durable. A crash could
keep the row and lose the tombstone. Refuse it before the append.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): refuse a write a delete cancelled before it appends

Mirrors the Go change: a write cancelled while still queued behind the
volume locks no longer appends, publishes and tombstones itself without
a sync after its delete may already be durable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): drop the write guards for a durable delete's flushes

The durable delete held the write lease and the io write guard across
both fsyncs, so reads of the volume waited on them. Append and publish
the tombstone under the guards and flush with only the io read side, as
write_synced_run and the Go path already do. do_delete_request keeps its
signature.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stop taking writes when a durable delete fails to index

The needle map drops the needle before it appends the .idx row, so after
a failed append a durable retry found nothing to delete and acked, and a
later write could bury the unindexed tombstone so a reload brought the
needle back. Fence the volume the way a failed durable write does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): stop taking writes when a durable delete fails to index

Rust counterpart of the Go change: the needle map drops the needle
before it appends the .idx row, so a failed append left a durable retry
nothing to delete, and a later write could bury the unindexed tombstone.
publish_delete_request now fences the volume for a durable delete, as
publish_write does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stop taking writes when any delete fails to index

A non-durable delete whose .idx append failed left the volume writable with
the needle already dropped from the map, so a later durable retry found
nothing to delete and acknowledged without the missing row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): stop taking writes when any delete fails to index

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the fsync flag on the existing delete functions

DeleteVolumeNeedle, deleteNeedle2 and syncDelete take fsync instead of
gaining *WithFsync variants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): take the fsync flag on the existing delete functions

delete_volume_needle and delete_needle take fsync instead of gaining
*_with_fsync variants; the durable path and the deletable checks fold
back into delete_needle, and the cancelled-write check mirrors Go's
isCancelledWrite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:18:29 -08:00
..

SeaweedFS Volume Server (Rust)

A drop-in replacement for the SeaweedFS Go volume server, rewritten in Rust. It uses binary-compatible storage formats (.dat, .idx, .vif) and speaks the same HTTP and gRPC protocols, so it works with an unmodified Go master server.

Building

Requires Rust 1.91.1+ (2024 edition), matching rust-version in Cargo.toml. The patch release matters: 1.91.0 does not build. The edition itself only needs 1.85; the higher floor comes from the dependency tree — chiefly the AWS SDK — so it moves with those crates. CI builds on the latest stable.

cd seaweed-volume
cargo build --release

The binary is produced at target/release/seaweed-volume.

Running

Start a Go master server first, then point the Rust volume server at it:

# Minimal
seaweed-volume --port 8080 --master localhost:9333 --dir /data/vol1 --max 7

# Multiple data directories
seaweed-volume --port 8080 --master localhost:9333 \
  --dir /mnt/ssd1,/mnt/ssd2 --max 100,100 --disk ssd

# With datacenter/rack topology
seaweed-volume --port 8080 --master localhost:9333 --dir /data/vol1 --max 7 \
  --dataCenter dc1 --rack rack1

# With JWT authentication
seaweed-volume --port 8080 --master localhost:9333 --dir /data/vol1 --max 7 \
  --securityFile /etc/seaweedfs/security.toml

# With TLS (configured in security.toml via [https.volume] and [grpc.volume] sections)
seaweed-volume --port 8080 --master localhost:9333 --dir /data/vol1 --max 7 \
  --securityFile /etc/seaweedfs/security.toml

Common flags

Flag Default Description
--port 8080 HTTP listen port
--port.grpc port+10000 gRPC listen port
--master localhost:9333 Comma-separated master server addresses
--dir /tmp Comma-separated data directories
--max 8 Max volumes per directory (comma-separated)
--ip auto-detect Server IP / identifier
--ip.bind same as --ip Bind address
--dataCenter Datacenter name
--rack Rack name
--disk Disk type tag: hdd, ssd, or custom
--index memory Needle map type: memory, leveldb, leveldbMedium, leveldbLarge
--readMode proxy Non-local read mode: local, proxy, redirect
--fileSizeLimitMB 256 Max upload file size
--minFreeSpace 1 (percent) Min free disk space before marking volumes read-only
--securityFile Path to security.toml for JWT keys and TLS certs
--metricsPort 0 (disabled) Prometheus metrics endpoint port
--whiteList Comma-separated IPs with write permission
--preStopSeconds 10 Graceful drain period before shutdown
--compactionMBps 0 (unlimited) Compaction I/O rate limit
--compactionMaxParallelCommits 0 (unlimited) Max concurrent vacuum commits
--pprof false Enable pprof HTTP handlers

Set RUST_LOG=debug (or trace, info, warn) for log level control.

Durable (fsync=true) uploads always go through per-volume batching workers that share one .dat/.idx sync per batch, matching the Go volume server; non-fsync writes take the synchronous path.

Features

  • Binary compatible -- reads and writes the same .dat/.idx/.vif files as the Go server; seamless migration with no data conversion.
  • HTTP + gRPC -- full implementation of the volume server HTTP API and all gRPC RPCs including streaming operations (copy, tail, incremental copy, vacuum).
  • Master heartbeat -- bidirectional streaming heartbeat with the Go master server; volume and EC shard registration, leader failover, graceful shutdown deregistration.
  • JWT authentication -- signing key configuration via security.toml with token source precedence (query > header > cookie), file_id claims validation, and separate read/write keys.
  • TLS -- HTTPS for the HTTP API and mTLS for gRPC, configured through security.toml.
  • Erasure coding -- Reed-Solomon EC shard management: mount/unmount, read, rebuild, copy, delete, and shard-to-volume reconstruction.
  • S3 remote storage -- FetchAndWriteNeedle reads from any S3-compatible backend (AWS, MinIO, Wasabi, Backblaze, etc.) and writes locally. Supports VolumeTierMoveDatToRemote/FromRemote for tiered storage.
  • Needle map backends -- in-memory HashMap, LevelDB (via rusty-leveldb), or redb (pure Rust disk-backed) needle maps.
  • Image processing -- on-the-fly resize/crop, JPEG EXIF orientation auto-fix, WebP support.
  • Streaming reads -- large files (>1MB) are streamed via spawn_blocking to avoid blocking the async runtime.
  • Auto-compression -- compressible file types (text, JSON, CSS, JS, SVG, etc.) are gzip-compressed on upload.
  • Prometheus metrics -- counters, histograms, and gauges exported at a dedicated metrics port; optional push gateway support.
  • Graceful shutdown -- SIGINT/SIGTERM handling with configurable preStopSeconds drain period.

Testing

Rust unit tests

cd seaweed-volume
cargo test

Go integration tests

The Go test suite can target either the Go or Rust volume server via the VOLUME_SERVER_IMPL environment variable:

# Run all HTTP + gRPC integration tests against the Rust server
VOLUME_SERVER_IMPL=rust go test -v -count=1 -timeout 1200s \
  ./test/volume_server/grpc/... ./test/volume_server/http/...

# Run a single test
VOLUME_SERVER_IMPL=rust go test -v -count=1 -timeout 60s \
  -run "TestName" ./test/volume_server/http/...

# Run S3 remote storage tests
VOLUME_SERVER_IMPL=rust go test -v -count=1 -timeout 180s \
  -run "TestFetchAndWriteNeedle" ./test/volume_server/grpc/...

Load testing

A load test harness is available at test/volume_server/loadtest/. See that directory for usage instructions and scenarios.

Architecture

The server runs three listeners concurrently:

  • HTTP (Axum 0.7) -- admin and public routers for file upload/download, status, and stats endpoints.
  • gRPC (Tonic 0.12) -- all VolumeServer RPCs from the SeaweedFS protobuf definition.
  • Metrics (optional) -- Prometheus scrape endpoint on a separate port.

Key source modules:

Path Description
src/main.rs Entry point, server startup, signal handling
src/config.rs CLI parsing and configuration resolution
src/server/volume_server.rs HTTP router setup and middleware
src/server/handlers.rs HTTP request handlers (read, write, delete, status)
src/server/grpc_server.rs gRPC service implementation
src/server/heartbeat.rs Master heartbeat loop
src/storage/volume.rs Volume read/write/delete logic
src/storage/needle.rs Needle (file entry) serialization
src/storage/store.rs Multi-volume store management
src/security.rs JWT validation and IP whitelist guard
src/remote_storage/ S3 remote storage backend

See DEV_PLAN.md for the full development history and feature checklist.