Files
seaweedfs/seaweed-volume/Cargo.toml
T
985706cc32 volume: check a needle's CRC the first time a range of it is read (#12467)
* volume: check a needle's CRC the first time a range of it is read

A ranged read never checked the CRC, so a replica with a damaged needle
served the damaged bytes with 206 and the reader never tried another
copy. The first ranged read of a needle now streams the whole needle
through the existing CRC check, with the usual pooled page buffers,
before writing any byte. A mismatch fails the read, so the reader falls
back to another replica. Needles that pass go into a bounded LRU, keyed
by volume, compaction revision, id, append time and checksum, so later
ranges of the same needle stream as before.

Fixes #5959 for ranged reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: key verified ranges by volume load, share concurrent checks

From review: the cache key read v.SuperBlock.CompactionRevision without
the data file lock, racing a compaction commit that reloads the super
block, and a remounted copy of a volume inherited the old copy's
verified entries. Key entries by a load generation instead, an atomic
counter that changes on every load, including the reload in
CommitCompact, and cache a result only if no reload happened during
the check. Concurrent first ranges of a needle now share one check
through singleflight instead of each reading the whole needle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep hot needles verified, leave the io error streak to the range read

From review: Contains does not refresh an entry's place in the LRU, so
even a constantly read needle was evicted after 64k other needles were
verified; look it up with Get instead. The verification pass also
cleared the io error streak before the range read ran, so a range read
failing with EIO after a passing check never built up a streak; the
pass into io.Discard now leaves the streak to the range read, like the
meta read before a streamed GET.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume (Rust): check a needle's CRC the first time a range of it is read

Mirror of the Go change: a range served from the data file now checks the
whole needle's CRC before any byte of it is sent, so a damaged replica
answers 500 and the reader tries another copy. Needles that pass go into a
bounded LRU keyed by the volume's load generation, needle id, append time
and checksum; concurrent first ranges of a needle share one check, and a
check that passes leaves the io error streak to the range read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume (Rust): a range over the whole needle checks its CRC

Go reads a range that covers the whole needle through the checked full-read
path. The Rust volume server read it from the data file unchecked, so
"bytes=0-" served a damaged needle with 206. Check the slice's CRC before
answering, as the full read does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check the covered blocks of every range against CRCs recorded at the first check

The first ranged read of a needle now records the CRC of each block
(64 KiB to 1 MiB, at most 64 blocks below 64 MiB) while it checks the
whole needle. Later ranges read the block-aligned span around them and
write a block's bytes only once it matches, so damage that appears
after the first check is not served.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume (Rust): check the covered blocks of every range against CRCs recorded at the first check

verify_checksum now records a CRC per block (same sizes as Go's
rangeCheckBlockSize) and returns them from the cache; read_checked_range
reads the block-aligned span of a range and answers 500 if a block no
longer matches, so damage after the first check is not served.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-07 10:53:07 -08:00

177 lines
5.1 KiB
TOML

[package]
name = "weed-volume"
version = "0.1.0"
edition = "2024"
# The edition needs 1.85; the dependency tree needs more. Verified with
# `cargo +1.91.1 check --all-targets` (1.90 fails on the AWS SDK).
rust-version = "1.91.1"
description = "SeaweedFS Volume Server — Rust implementation"
[lib]
name = "seaweed_volume"
[[bin]]
name = "weed-volume"
path = "src/main.rs"
[features]
# Default: 5-byte offsets (8TB max volume size), matching production Go builds (-tags 5BytesOffset).
# Disable with --no-default-features for 4-byte offsets (32GB max volume size).
default = ["5bytes"]
5bytes = []
# Unstable redb cursor bulk-load for full_rebuild. Off in production.
# Pulls redb's experimental_cursor (and therefore experimental-api-5).
redb-experimental-cursor = ["redb/experimental_cursor"]
[lints.clippy]
# Protobuf message literals keep `..Default::default()` on purpose: it is
# what lets a proto gain a field without touching every constructor.
needless_update = "allow"
# Every `unsafe` block states its precondition, right above the block.
undocumented_unsafe_blocks = "warn"
[dependencies]
# Helpers the Rust plugin workers (seaweed-worker) need as well. A path
# dependency because the two trees are separate cargo workspaces with no
# common root manifest.
seaweed-common = { path = "../seaweed-common" }
# Async runtime
tokio = { version = "1", features = ["full"] }
tokio-stream = { version = "0.1", features = ["net"] }
tokio-io-timeout = "1"
# gRPC + protobuf
tonic = { version = "0.14", features = ["tls-aws-lc"] }
tonic-prost = "0.14"
tonic-reflection = "0.14"
prost = "0.14"
prost-types = "0.14"
# HTTP server
axum = { version = "0.8", features = ["multipart"] }
http-body = "1"
httpdate = "1"
hyper = { version = "1", features = ["full"] }
hyper-util = { version = "0.1", features = ["tokio", "service", "server-auto", "http1", "http2"] }
tower = { version = "0.5", features = ["util"] }
tower-http = { version = "0.6", features = ["cors", "trace"] }
# CLI
clap = { version = "4", features = ["derive"] }
# Metrics
prometheus = { version = "0.13", default-features = false, features = ["process"] }
# JWT
jsonwebtoken = { version = "10", features = ["rust_crypto"] }
# TLS
rustls = "0.23"
tokio-rustls = "0.26"
rustls-pemfile = "2"
# LevelDB (via RocksDB for better Rust support)
# Using rusty-leveldb for pure Rust LevelDB
rusty-leveldb = "3"
# Disk-backed needle map (alternative to in-memory HashMap)
redb = "4"
# Reed-Solomon erasure coding
reed-solomon-erasure = "6"
# Logging
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
# Config
toml = "0.8"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
serde_urlencoded = "0.7"
# CRC32 — using Castagnoli polynomial (CRC32-C), matching Go's crc32.Castagnoli
crc32c = "0.6"
crc32fast = "1"
# xxhash64 - must match Go's cespare/xxhash for the heartbeat volume digest
xxhash-rust = { version = "0.8", features = ["xxh64"] }
# Memory-mapped files
memmap2 = "0.9"
# UUID
uuid = { version = "1", features = ["v4"] }
# HTTP client (for proxying, remote fetch)
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "stream", "multipart", "json"] }
# Content hashing
md-5 = "0.10"
base64 = "0.22"
# Compression
flate2 = "1"
# Image processing
image = { version = "0.25", default-features = false, features = ["png", "jpeg", "gif", "webp"] }
kamadak-exif = "0.5"
# Multipart form-data parsing
multer = "3"
# MIME type guessing from file extensions
mime_guess = "2"
# Misc
bytes = "1"
rand = "0.10"
chrono = "0.4"
hex = "0.4"
parking_lot = "0.12"
dashmap = "6"
lru = "0.16"
thiserror = "1"
anyhow = "1"
async-trait = "0.1"
futures = "0.3"
async-stream = "0.3"
x509-parser = "0.16"
# Disk space checking
sysinfo = "0.31"
libc = "0.2"
# AWS S3 SDK (for remote storage backends)
aws-config = { version = "1", features = ["behavior-version-latest"] }
aws-sdk-s3 = { version = "1.125.0", default-features = false, features = ["sigv4a", "http-1x", "default-https-client", "rt-tokio"] }
aws-credential-types = "1"
aws-types = "1"
# pprof is Unix-only (requires libc/nix APIs not available on Windows)
[target.'cfg(unix)'.dependencies]
pprof = { version = "0.15", features = ["prost-codec"] }
# GetDiskFreeSpaceExW for per-path disk capacity on Windows (0.61.2 already
# in the tree via tempfile/mio, so this unifies rather than adding a version).
[target.'cfg(windows)'.dependencies]
windows-sys = { version = "0.61", features = ["Win32_Storage_FileSystem"] }
[dev-dependencies]
tempfile = "3"
# Paused-time tests drive the write-queue batch window deterministically.
tokio = { version = "1", features = ["full", "test-util"] }
# Already a transitive dependency of aws-sdk-s3 at a single locked version;
# needed directly only for the canned HttpClient in remote_storage::s3 tests.
aws-smithy-runtime-api = "1"
[build-dependencies]
tonic-prost-build = "0.14"
# Ships protoc with the build so neither CI nor a developer needs a system
# install, and so the version is pinned rather than whatever the platform's
# package manager happens to carry.
protoc-bin-vendored = "3"
[patch.crates-io]
reed-solomon-erasure = { path = "vendor/reed-solomon-erasure" }