volume: check a needle's CRC the first time a range of it is read (#12467)

* volume: check a needle's CRC the first time a range of it is read

A ranged read never checked the CRC, so a replica with a damaged needle
served the damaged bytes with 206 and the reader never tried another
copy. The first ranged read of a needle now streams the whole needle
through the existing CRC check, with the usual pooled page buffers,
before writing any byte. A mismatch fails the read, so the reader falls
back to another replica. Needles that pass go into a bounded LRU, keyed
by volume, compaction revision, id, append time and checksum, so later
ranges of the same needle stream as before.

Fixes #5959 for ranged reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: key verified ranges by volume load, share concurrent checks

From review: the cache key read v.SuperBlock.CompactionRevision without
the data file lock, racing a compaction commit that reloads the super
block, and a remounted copy of a volume inherited the old copy's
verified entries. Key entries by a load generation instead, an atomic
counter that changes on every load, including the reload in
CommitCompact, and cache a result only if no reload happened during
the check. Concurrent first ranges of a needle now share one check
through singleflight instead of each reading the whole needle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep hot needles verified, leave the io error streak to the range read

From review: Contains does not refresh an entry's place in the LRU, so
even a constantly read needle was evicted after 64k other needles were
verified; look it up with Get instead. The verification pass also
cleared the io error streak before the range read ran, so a range read
failing with EIO after a passing check never built up a streak; the
pass into io.Discard now leaves the streak to the range read, like the
meta read before a streamed GET.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume (Rust): check a needle's CRC the first time a range of it is read

Mirror of the Go change: a range served from the data file now checks the
whole needle's CRC before any byte of it is sent, so a damaged replica
answers 500 and the reader tries another copy. Needles that pass go into a
bounded LRU keyed by the volume's load generation, needle id, append time
and checksum; concurrent first ranges of a needle share one check, and a
check that passes leaves the io error streak to the range read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume (Rust): a range over the whole needle checks its CRC

Go reads a range that covers the whole needle through the checked full-read
path. The Rust volume server read it from the data file unchecked, so
"bytes=0-" served a damaged needle with 206. Check the slice's CRC before
answering, as the full read does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check the covered blocks of every range against CRCs recorded at the first check

The first ranged read of a needle now records the CRC of each block
(64 KiB to 1 MiB, at most 64 blocks below 64 MiB) while it checks the
whole needle. Later ranges read the block-aligned span around them and
write a block's bytes only once it matches, so damage that appears
after the first check is not served.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume (Rust): check the covered blocks of every range against CRCs recorded at the first check

verify_checksum now records a CRC per block (same sizes as Go's
rangeCheckBlockSize) and returns them from the cache; read_checked_range
reads the block-aligned span of a range and answers 500 if a block no
longer matches, so damage after the first check is not served.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

seaweed-volume/src/server/handlers.rs

@@ -2008,8 +2008,9 @@ async fn range_from_source_response(
 ) -> Response {
     let range_str = range_str.to_string();
     let tracking = track_download.then(|| state.clone());
+    let chunk_size = streaming_chunk_size(state.read_buffer_size_bytes, info.data_size as usize);
     tokio::task::spawn_blocking(move || {
-        handle_range_request_from_source(&range_str, info, response_headers, tracking)
+        handle_range_request_from_source(&range_str, info, response_headers, tracking, chunk_size)
     })
     .await
     .unwrap_or_else(|e| {
@@ -2353,6 +2354,7 @@ fn handle_range_request_from_source(
     info: crate::storage::volume::NeedleStreamInfo,
     mut headers: HeaderMap,
     state: Option<Arc<VolumeServerState>>,
+    chunk_size: usize,
 ) -> Response {
     let total = info.data_size as i64;
     let ranges = match parse_range_header(range_str, total) {
@@ -2376,21 +2378,57 @@ fn handle_range_request_from_source(
         return (StatusCode::OK, headers).into_response();
     }
 
+    let block_crcs = if ranges.iter().any(|r| r.start != 0 || r.length != total) {
+        match info.verify_checksum(chunk_size) {
+            Ok(crcs) => Some(crcs),
+            Err(err) => {
+                return (
+                    StatusCode::INTERNAL_SERVER_ERROR,
+                    format!("range read error: {}", err),
+                )
+                    .into_response();
+            }
+        }
+    } else {
+        None
+    };
+
     let read_slice = |start: i64, length: i64| -> Result<Vec<u8>, std::io::Error> {
         if length <= 0 {
             return Ok(Vec::new());
         }
-        let mut buf = vec![0u8; length as usize];
-        match info
-            .source
-            .read_exact_at(&mut buf, info.data_file_offset + start as u64)
-        {
+        let read = match &block_crcs {
+            Some(crcs) if length != total => {
+                info.read_checked_range(crcs, start as u64, length as u64)
+            }
+            _ => {
+                let mut buf = vec![0u8; length as usize];
+                info.source
+                    .read_exact_at(&mut buf, info.data_file_offset + start as u64)
+                    .map(|()| buf)
+            }
+        };
+        match read {
+            Ok(buf) if length == total && !info.checksum_matches(&buf) => {
+                metrics::HANDLER_COUNTER
+                    .with_label_values(&[metrics::ERROR_CRC])
+                    .inc();
+                Err(std::io::Error::new(
+                    std::io::ErrorKind::InvalidData,
+                    format!(
+                        "needle data checksum mismatch for needle {}",
+                        info.needle_id
+                    ),
+                ))
+            }
             // A finished range read clears the streak a failed meta read
             // counted, like readNeedleDataInto's terminal report.
-            Ok(()) => {
+            Ok(buf) => {
                 info.io_errors.check_read_write_error(None);
                 Ok(buf)
             }
+            // A block that fails its CRC leaves the streak alone, as Go's.
+            Err(e) if e.kind() == std::io::ErrorKind::InvalidData => Err(e),
             Err(e) => {
                 info.io_errors.check_read_write_error(Some(&e));
                 Err(e)
@@ -7013,13 +7051,145 @@ mod tests {
             io_unavailable: Arc::default(),
             io_errors: io_errors.clone(),
             needle_id: NeedleId(1),
-            checksum: 0,
+            checksum: CRC::new(&data).0,
+            append_at_ns: 0,
+            load_generation: u64::MAX,
         };
-        let resp = handle_range_request_from_source("bytes=10-19", info, HeaderMap::new(), None);
+        // The checksum pass leaves the streak to the range read.
+        info.verify_checksum(64).unwrap();
+        assert_eq!(io_errors.get_io_error_state().1, 2);
+        let resp =
+            handle_range_request_from_source("bytes=10-19", info, HeaderMap::new(), None, 64);
         assert_eq!(resp.status(), StatusCode::PARTIAL_CONTENT);
         assert_eq!(io_errors.get_io_error_state().1, 0);
     }
 
+    /// A range carries no checksum of its own, so the first ranged read of a
+    /// needle checks the whole needle's CRC and sends none of the range on a
+    /// mismatch; the reader can then fall back to another replica. A verified
+    /// needle is checked again once the volume reloads.
+    #[tokio::test]
+    async fn test_range_read_checks_the_needle_checksum() {
+        let tmp = tempfile::TempDir::new().unwrap();
+        let state = volume_test_state(&tmp);
+        let payload =
+            |seed: u32| -> Vec<u8> { (0..3000u32).map(|i| ((i * 7 + seed) % 251) as u8).collect() };
+        let damage = |data: &[u8]| {
+            let dat = tmp.path().join("1.dat");
+            let mut bytes = std::fs::read(&dat).unwrap();
+            let at = bytes.windows(data.len()).position(|w| w == data).unwrap();
+            bytes[at + 10] ^= 0xff;
+            std::fs::write(&dat, bytes).unwrap();
+        };
+
+        let intact = payload(1);
+        let path = put_test_needle(&state, 0x6e7a_0c01, &intact);
+        let (status, _, body) =
+            send_read(&state, Method::GET, &path, Some(b"bytes=1000-1499")).await;
+        assert_eq!(status, StatusCode::PARTIAL_CONTENT);
+        assert_eq!(body, &intact[1000..1500]);
+
+        // Damage a byte outside the range: the range itself is intact, but
+        // the needle is not, so this replica must not serve it.
+        let damaged = payload(2);
+        let path = put_test_needle(&state, 0x6e7a_0c02, &damaged);
+        damage(&damaged);
+        for range in [&b"bytes=1000-1499"[..], b"bytes=0-", b"bytes=0-1,1000-1001"] {
+            let (status, _, body) = send_read(&state, Method::GET, &path, Some(range)).await;
+            assert_eq!(status, StatusCode::INTERNAL_SERVER_ERROR, "{range:?}");
+            assert!(
+                !body.windows(2).any(|w| w == &damaged[1000..1002]),
+                "{range:?}"
+            );
+        }
+
+        // Concurrent first ranges of a needle all see the outcome of the check.
+        for (seed, damage_it) in [(3, false), (4, true)] {
+            let data = payload(seed);
+            let path = put_test_needle(&state, 0x6e7a_0c00 + seed as u64, &data);
+            if damage_it {
+                damage(&data);
+            }
+            let reads = (0..8).map(|_| send_read(&state, Method::GET, &path, Some(b"bytes=10-19")));
+            for (status, _, _) in futures::future::join_all(reads).await {
+                assert_eq!(status == StatusCode::PARTIAL_CONTENT, !damage_it, "{seed}");
+            }
+        }
+
+        // A verified needle is checked again after a reload, as a compaction
+        // commit does; until then later ranges skip the check.
+        let data = payload(5);
+        let path = put_test_needle(&state, 0x6e7a_0c05, &data);
+        let (status, _, _) = send_read(&state, Method::GET, &path, Some(b"bytes=10-19")).await;
+        assert_eq!(status, StatusCode::PARTIAL_CONTENT);
+        let (_, v) = state
+            .store
+            .read()
+            .unwrap()
+            .find_volume(VolumeId(1))
+            .unwrap();
+        v.compact_by_index(0, 0, |_| true).unwrap();
+        v.commit_compact().unwrap();
+        damage(&data);
+        let (status, _, _) = send_read(&state, Method::GET, &path, Some(b"bytes=10-19")).await;
+        assert_eq!(status, StatusCode::INTERNAL_SERVER_ERROR);
+
+        // Damage that appears after a needle passed its check is caught by
+        // the block it falls in, while ranges over intact blocks still read.
+        let block = 64 * 1024;
+        let data: Vec<u8> = (0..5 * block - 100)
+            .map(|i| (i * 7 + i / 251) as u8)
+            .collect();
+        let path = put_test_needle(&state, 0x6e7a_0c06, &data);
+        let range = |from: usize, to: usize| format!("bytes={}-{}", from, to - 1).into_bytes();
+        let damaged_ranges = [
+            (3 * block + 10, 3 * block + 110),
+            (block - 50, 4 * block - 50),
+        ];
+        for (from, to) in damaged_ranges {
+            let (status, _, body) =
+                send_read(&state, Method::GET, &path, Some(&range(from, to))).await;
+            assert_eq!(status, StatusCode::PARTIAL_CONTENT);
+            assert_eq!(body, &data[from..to]);
+        }
+        let dat = tmp.path().join("1.dat");
+        let mut bytes = std::fs::read(&dat).unwrap();
+        let at = bytes.windows(64).position(|w| w == &data[..64]).unwrap();
+        bytes[at + 3 * block + 500] ^= 0xff;
+        std::fs::write(&dat, bytes).unwrap();
+        for (from, to) in damaged_ranges {
+            let (status, _, _) =
+                send_read(&state, Method::GET, &path, Some(&range(from, to))).await;
+            assert_eq!(status, StatusCode::INTERNAL_SERVER_ERROR, "{from}-{to}");
+        }
+        for (from, to) in [
+            (10, 110),
+            (2 * block + 1, 3 * block),
+            (4 * block + 5, data.len()),
+        ] {
+            let (status, _, body) =
+                send_read(&state, Method::GET, &path, Some(&range(from, to))).await;
+            assert_eq!(status, StatusCode::PARTIAL_CONTENT, "{from}-{to}");
+            assert_eq!(body, &data[from..to]);
+        }
+    }
+
+    #[test]
+    fn test_range_check_block_size() {
+        use crate::storage::volume::range_check_block_size;
+        let block = 64 * 1024;
+        let mib = 1024 * 1024;
+        for (size, want) in [
+            (0, block),
+            (64 * block, block),
+            (64 * block + 1, 2 * block),
+            (64 * mib, mib),
+            (1 << 34, mib),
+        ] {
+            assert_eq!(range_check_block_size(size), want, "{size}");
+        }
+    }
+
     /// Range is read as Go reads it: a byte its parseRange cannot parse is a
     /// 416, Unicode whitespace around a range is trimmed, an empty value is no
     /// range, and HEAD ignores it.