weed/server: verify data shards against bitrot sidecar before ec decode (#12422) (#12427)

* weed/server: verify data shards against bitrot sidecar before ec decode (#12422)

* ec.decode: verify data shards against the mounted volume's bitrot protection

The decode looked up only the generation-0 .ecsum beside the data base and
each location's data directory, so it missed a sidecar kept with the index
and ignored the generation the volume actually serves.

Take the protection the volume's per-disk runtimes resolved instead (On, else
Invalid, else Off), and test a corrupt block past the first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: check for live entries before verifying the shards

A volume whose needles are all deleted gets its shards purged on the
no-live-entries status; a failing checksum must not keep it around.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: verify data shards against the bitrot protection before a decode

Mirrors the Go volume server: the decode takes the protection the volume's
per-disk runtimes resolved and refuses a data shard that fails its .ecsum
checksum, after the no-live-entries check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: a decode refuses a data shard that fails its .ecsum checksum

Runs against both volume server implementations: the refused decode keeps
every shard and the sidecar, and the needle stays readable through EC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: reload the bitrot sidecar before verifying the shards

A rebuild's backfill writes the sidecar without reloading the mounted
runtimes, which then still report no protection; resolve it again across
the EC metadata directories, as a shard mount does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: reload the bitrot sidecar before verifying the shards

Mirrors the Go volume server: the decode job resolves the sidecar again on
every runtime of the volume before choosing its protection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: a decode finds a sidecar written after the shards were mounted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read through the EC shards after the refused decode

The source volume stayed mounted, so the final read never touched the
shards it is meant to check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: refuse when a sidecar reload reports another layout

ReloadBitrotSidecar logged and dropped the layout-mismatch error, so the
runtime stayed Off and the decode skipped the data-shard check. It now
returns that error and the decode refuses on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a decode when a sidecar reload reports another layout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: a decode refuses a late sidecar that records another layout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: refuse when an unmount left no runtime to resolve the sidecar

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a decode when an unmount left no runtime to resolve the sidecar

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

seaweed-volume/src/storage/erasure_coding/ec_bitrot.rs

@@ -581,6 +581,51 @@ pub fn verify_shard_blocks(
     Ok(mismatched)
 }
 
+/// Checks the data shards a decode reads against the volume's bitrot
+/// protection: a corrupt shard copied into the .dat is lost for good once the
+/// caller deletes the parity that could repair it. Mirrors Go's
+/// `VerifyEcVolumeDataShards`.
+pub fn verify_ec_volume_data_shards(
+    prot: Option<&EcBitrotProtection>,
+    status: BitrotStatus,
+    shard_paths: &[String],
+) -> io::Result<()> {
+    let prot = match (status, prot) {
+        (BitrotStatus::Off, _) => return Ok(()),
+        (BitrotStatus::On, Some(prot)) => prot,
+        _ => {
+            return Err(io::Error::new(
+                io::ErrorKind::InvalidData,
+                "bitrot sidecar is malformed or unverifiable",
+            ));
+        }
+    };
+    for (shard_id, path) in shard_paths.iter().enumerate() {
+        let Some(entry) = shard_checksums(prot, shard_id as u32) else {
+            continue;
+        };
+        let mismatched =
+            verify_shard_file_blocks(path, entry, prot.block_size as i64).map_err(|e| {
+                io::Error::new(
+                    e.kind(),
+                    format!("verify data shard {} {}: {}", shard_id, path, e),
+                )
+            })?;
+        if !mismatched.is_empty() {
+            return Err(io::Error::new(
+                io::ErrorKind::InvalidData,
+                format!(
+                    "data shard {} {} fails its bitrot checksum in {} blocks",
+                    shard_id,
+                    path,
+                    mismatched.len()
+                ),
+            ));
+        }
+    }
+    Ok(())
+}
+
 /// Builds the `EcShardConfig` proto for the given layout. The bitrot sidecar
 /// carries its own top-level encode_uuid, so the nested config leaves it empty.
 pub fn ec_shard_config(data_shards: u32, parity_shards: u32, block_size: i64) -> EcShardConfig {

seaweed-volume/src/storage/erasure_coding/ec_volume.rs

@@ -238,8 +238,8 @@ pub struct EcVolume {
     /// no encode identity, so provenance is the only signal that a loaded
     /// manifest may describe a different encode run. Same purpose as
     /// `ecx_actual_dir`. Empty when no sidecar was found. Always committed in
-    /// the same step as `bitrot`/`bitrot_status`, so a reload whose `Err` gets
-    /// swallowed by `reload_bitrot_sidecar` cannot leave this describing a
+    /// the same step as `bitrot`/`bitrot_status`, so a reload whose `Err` a
+    /// caller ignores cannot leave this describing a
     /// sidecar other than the one those two fields actually hold.
     pub(crate) bitrot_source_dir: String,
 
@@ -802,14 +802,17 @@ impl EcVolume {
     /// shard delivery can bring the manifest with it, and the receive path only
     /// writes the file, so without this the in-memory volume keeps the
     /// protection state it resolved at mount (off) until a remount.
-    pub fn reload_bitrot_sidecar(&mut self, additional_dirs: &[String]) {
-        if let Err(e) = self.load_active_bitrot_sidecar(additional_dirs) {
+    /// Returns the layout-mismatch error a mount would fail on.
+    pub fn reload_bitrot_sidecar(&mut self, additional_dirs: &[String]) -> io::Result<()> {
+        let result = self.load_active_bitrot_sidecar(additional_dirs);
+        if let Err(e) = &result {
             tracing::warn!(
                 volume_id = self.volume_id.0,
                 error = %e,
                 "reload bitrot sidecar",
             );
         }
+        result
     }
 
     /// Load the generation-0 checksum sidecar into `self.bitrot`/`self.bitrot_status`.
@@ -888,8 +891,8 @@ impl EcVolume {
             self.parity_shards as usize,
         );
         // Provenance is committed together with the sidecar it describes, so a
-        // swallowed reload (`reload_bitrot_sidecar` logs and discards the `Err`
-        // above) cannot leave them disagreeing: the geometry-mismatch return
+        // reload whose `Err` above a caller ignores cannot leave them
+        // disagreeing: the geometry-mismatch return
         // happens before this point, so `bitrot`/`bitrot_status` below and
         // `bitrot_source_dir` here always describe the same load attempt.
         // Only a path that actually exists is a real source: `path` falls back
@@ -934,6 +937,31 @@ impl EcVolume {
         (self.bitrot.clone(), self.bitrot_status)
     }
 
+    /// The protection of a volume's per-disk runtimes: the first `On`, else
+    /// `Invalid`, else `Off`. After a restart only the runtime whose disk holds
+    /// the sidecar resolves it. Mirrors Go's `SelectBitrotProtection`.
+    pub(crate) fn select_bitrot_protection(
+        volumes: &[&EcVolume],
+    ) -> (
+        Option<crate::pb::volume_server_pb::EcBitrotProtection>,
+        crate::storage::erasure_coding::ec_bitrot::BitrotStatus,
+    ) {
+        use crate::storage::erasure_coding::ec_bitrot::BitrotStatus;
+        let mut invalid = false;
+        for v in volumes {
+            match v.bitrot_protection() {
+                (prot, BitrotStatus::On) => return (prot, BitrotStatus::On),
+                (_, BitrotStatus::Invalid) => invalid = true,
+                _ => {}
+            }
+        }
+        if invalid {
+            (None, BitrotStatus::Invalid)
+        } else {
+            (None, BitrotStatus::Off)
+        }
+    }
+
     /// Read-only EC bitrot checksum scrub of the LOCAL shards of this volume's
     /// active generation.
     ///