mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-07 20:47:46 +01:00
985706cc32
* volume: check a needle's CRC the first time a range of it is read A ranged read never checked the CRC, so a replica with a damaged needle served the damaged bytes with 206 and the reader never tried another copy. The first ranged read of a needle now streams the whole needle through the existing CRC check, with the usual pooled page buffers, before writing any byte. A mismatch fails the read, so the reader falls back to another replica. Needles that pass go into a bounded LRU, keyed by volume, compaction revision, id, append time and checksum, so later ranges of the same needle stream as before. Fixes #5959 for ranged reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: key verified ranges by volume load, share concurrent checks From review: the cache key read v.SuperBlock.CompactionRevision without the data file lock, racing a compaction commit that reloads the super block, and a remounted copy of a volume inherited the old copy's verified entries. Key entries by a load generation instead, an atomic counter that changes on every load, including the reload in CommitCompact, and cache a result only if no reload happened during the check. Concurrent first ranges of a needle now share one check through singleflight instead of each reading the whole needle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: keep hot needles verified, leave the io error streak to the range read From review: Contains does not refresh an entry's place in the LRU, so even a constantly read needle was evicted after 64k other needles were verified; look it up with Get instead. The verification pass also cleared the io error streak before the range read ran, so a range read failing with EIO after a passing check never built up a streak; the pass into io.Discard now leaves the streak to the range read, like the meta read before a streamed GET. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume (Rust): check a needle's CRC the first time a range of it is read Mirror of the Go change: a range served from the data file now checks the whole needle's CRC before any byte of it is sent, so a damaged replica answers 500 and the reader tries another copy. Needles that pass go into a bounded LRU keyed by the volume's load generation, needle id, append time and checksum; concurrent first ranges of a needle share one check, and a check that passes leaves the io error streak to the range read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume (Rust): a range over the whole needle checks its CRC Go reads a range that covers the whole needle through the checked full-read path. The Rust volume server read it from the data file unchecked, so "bytes=0-" served a damaged needle with 206. Check the slice's CRC before answering, as the full read does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: check the covered blocks of every range against CRCs recorded at the first check The first ranged read of a needle now records the CRC of each block (64 KiB to 1 MiB, at most 64 blocks below 64 MiB) while it checks the whole needle. Later ranges read the block-aligned span around them and write a block's bytes only once it matches, so damage that appears after the first check is not served. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume (Rust): check the covered blocks of every range against CRCs recorded at the first check verify_checksum now records a CRC per block (same sizes as Go's rangeCheckBlockSize) and returns them from the cache; read_checked_range reads the block-aligned span of a range and answers 500 if a block no longer matches, so damage after the first check is not served. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>