ec: scrub a volume through its parity data (#11006)

* Introduce a new `READS` scrub mode. `READS` performs a full volume scrub but, unlike `FULL`, it will attempt to reconstruct data for missing/damaged shard intervals from other shards in the cluster when necessary. The goal of this check is to ensure that EC volume contents _are readable by Seaweed_ even on a degraded storage state, by exercising parity data which is not read in `FULL` mode. This is useful not only to validate data is user-readable, but also to detect potential parity shard issues which may be difficult to pinpoint otherwise - particularly for older volumes lacking sidecar data, and hence unaffected by `CHECKSUM` scrubs. For regular volumes, this operation is equivalent to `FULL`. Example: ``` > ec.shard.unmount --volumeId=1 --shardId=0,3,11 --delete --apply Live shard topology for volume ID 1 (14 shards): 0@10.200.18.89:9001 1@10.200.18.89:9002 2@10.200.18.89:9003 3@10.200.18.89:9004 4@10.200.18.89:9005 5@10.200.18.89:9006 6@10.200.18.89:9007 7@10.200.18.89:9008 8@10.200.18.89:9009 9@10.200.18.89:9013 10@10.200.18.89:9010 11@10.200.18.89:9011 12@10.200.18.89:9012 13@10.200.18.89:9020 Will unmount + delete 3 shard(s): 0@10.200.18.89:9001 3@10.200.18.89:9004 11@10.200.18.89:9011 Unmounting shard 0@10.200.18.89:9001 for volume ID 1... Deleting shard 0@10.200.18.89:9001 for volume ID 1... Unmounting shard 3@10.200.18.89:9004 for volume ID 1... Deleting shard 3@10.200.18.89:9004 for volume ID 1... Unmounting shard 11@10.200.18.89:9011 for volume ID 1... Deleting shard 11@10.200.18.89:9011 for volume ID 1... All done! > ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=full using FULL mode Scrubbing 10.200.18.89:9002 (1/1)... Scrubbed 6 EC files and 1 volumes on 1 nodes Got scrub failures on 1 EC volumes and 1 EC shards :( Affected volumes: 10.200.18.89:9002:1 Affected shards: 10.200.18.89:9002:1:0 > ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=reads using READS mode Scrubbing 10.200.18.89:9002 (1/1)... Scrubbed 6 EC files and 1 volumes on 1 nodes ``` * ec: report the shards a READS scrub had to rebuild A READS scrub that recovers an interval was recording nothing, so a volume missing three shards came back clean and nobody repaired it. The unreadable shard is now recorded before the rebuild is attempted: READS reports the same broken shards as FULL and differs only in whether the needles themselves failed, which is the signal worth having - shards are gone, data is still there. forceDeletedNeedlesCheck now applies to READS as well, in the shell and in the RPC guard: it runs the same needle walk as FULL. Regenerated the proto instead of hand-editing it, so the pancis typo (which protoc-gen-go-grpc emits into eight other files here) and the header whitespace stay as generated. Mirrors into the Rust volume server, which also now honors force_deleted_needles_check rather than hardcoding it off. Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq * ec: answer a deleted needle from a READS rebuild as deleted #11020 gave the Rust recovery a deleted flag alongside its bytes, and it answers a deleted needle with no bytes at all. The READS scrub appended that empty answer, which does not compile against the new signature and, once it did, would leave the needle short and report the size mismatch as damage. Zero-fill the interval instead, the way the direct read beside it already does: the assembled needle then reaches read_bytes as the delete-state mismatch the walk already tolerates. Go takes the same branch off the flag its recovery returns, rather than discarding it. Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq --------- Co-authored-by: Lisandro Pin <lisandro.pin@proton.ch>
2026-11-06 12:07:51 +01:00 · 2026-08-28 16:42:34 -07:00
1 parent ba5b14b457
commit fcc2ea61d3
@@ -38,6 +38,7 @@ fn scrub_mode_label(mode: i32) -> &'static str {
        2 => "FULL",
        3 => "LOCAL",
        4 => "CHECKSUM",
+        5 => "READS",
        _ => "UNKNOWN",
    }
 }
@@ -4097,7 +4098,7 @@ impl VolumeServer for VolumeGrpcService {
        // Validate mode
        let mode = req.mode;
        match mode {
-            1 | 2 | 3 => {} // INDEX=1, FULL=2, LOCAL=3
+            1 | 2 | 3 | 5 => {} // INDEX=1, FULL=2, LOCAL=3, READS=5 (FULL for regular volumes)
            _ => {
                return Err(Status::invalid_argument(format!(
                    "unsupported volume scrub mode {}",
@@ -4197,7 +4198,7 @@ impl VolumeServer for VolumeGrpcService {
        // Validate mode
        let mode = req.mode;
        match mode {
-            1 | 2 | 3 | 4 => {} // INDEX=1, FULL=2, LOCAL=3, CHECKSUM=4
+            1 | 2 | 3 | 4 | 5 => {} // INDEX=1, FULL=2, LOCAL=3, CHECKSUM=4, READS=5
            _ => {
                return Err(Status::invalid_argument(format!(
                    "unsupported EC volume scrub mode {}",
@@ -4206,6 +4207,14 @@ impl VolumeServer for VolumeGrpcService {
            }
        }

+        // Only the modes that walk needles can be strict about deleted ones.
+        let force_deleted_needles_check = req.force_deleted_needles_check;
+        if force_deleted_needles_check && mode != 2 && mode != 5 {
+            return Err(Status::invalid_argument(
+                "deleted needle checks are only supported for FULL and READS scrubs",
+            ));
+        }
+
        // Collect the volume ids under a brief lock, then release it: FULL (mode 2)
        // reads remote shards and must not hold the !Send store guard across .await.
        let vids: Vec<VolumeId> = {
@@ -4247,8 +4256,8 @@ impl VolumeServer for VolumeGrpcService {
                        }
                    }
                }
-                2 => {
-                    // FULL: Go-parity per-needle local+remote walk, PLUS a TEMPORARY
+                2 | 5 => {
+                    // FULL/READS: Go-parity per-needle local+remote walk, PLUS a TEMPORARY
                    // local Reed-Solomon parity check. The needle walk only reads
                    // DATA-shard intervals of LIVE needles, so on its own it can't
                    // catch silent bitrot in a PARITY shard or an unwalked cold
@@ -4280,8 +4289,13 @@ impl VolumeServer for VolumeGrpcService {

                    // (1) Per-needle local+remote walk (Go ScrubEcVolume parity).
                    let (files, mut shard_infos, mut errs) =
-                        crate::server::store_ec::scrub_ec_volume_distributed(&self.state, vid, false)
-                            .await;
+                        crate::server::store_ec::scrub_ec_volume_distributed(
+                            &self.state,
+                            vid,
+                            force_deleted_needles_check,
+                            mode == 5,
+                        )
+                        .await;
                    total_files += files as u64; // count comes from the needle walk only

                    // (2) Local parity check, gated on all-shards-local. Blocking RS
@@ -246,7 +246,9 @@ pub async fn read_ec_shard_needle_distributed(
 /// without decoding (so genuine shard faults are reported rather than healed).
 /// Mirrors Go's `Store.ScrubEcVolume`. Returns (rows walked, broken shards,
 /// errors). `force_deleted_needles_check` disables the benign delete-state
-/// size-mismatch suppression.
+/// size-mismatch suppression. `recover_unreadable` (READS mode) rebuilds an
+/// unreadable interval from the surviving shards: the same shards are reported
+/// broken, but only needles parity can no longer recover become errors.
 ///
 /// Shard locations are refreshed once up front. Each needle is then processed via
 /// `scrub_snapshot_under_lock` + lock-drop + no-reconstruct `read_remote_ec_shard_interval`,
@@ -255,6 +257,7 @@ pub async fn scrub_ec_volume_distributed(
    state: &Arc<VolumeServerState>,
    vid: VolumeId,
    force_deleted_needles_check: bool,
+    recover_unreadable: bool,
 ) -> (i64, Vec<crate::pb::volume_server_pb::EcShardInfo>, Vec<String>) {
    // Phase A — under the Store read lock, run the index scrub and grab the
    // paths/scalars + shard-location staleness; release the lock before any await.
@@ -395,8 +398,9 @@ pub async fn scrub_ec_volume_distributed(
            }
        };

-        // Read each interval local-then-remote WITHOUT reconstructing: we verify
-        // the shards are valid, we do not heal them. Locations refreshed above.
+        // Read each interval local-then-remote. Neither read decodes: the point is to
+        // find shards that are themselves broken, not to heal around them. READS then
+        // rebuilds what it could not read. Locations refreshed above.
        let n_intervals = snapshot.intervals.len();
        let mut data: Vec<u8> = Vec::with_capacity(snapshot.actual_size);
        for (i, res) in snapshot.intervals.iter().enumerate() {
@@ -426,11 +430,9 @@ pub async fn scrub_ec_volume_distributed(
                        // -> the delete-state suppression (mirrors Go's pre-zeroed buffer).
                        Ok((_, true)) => data.resize(data.len() + *ssize, 0),
                        Ok((buf, false)) => data.extend_from_slice(&buf),
-                        Err(_) => {
-                            errs.push(format!(
-                                "failed to read EC shard {} for needle {} on volume {} (interval {}/{})",
-                                shard_id, id.0, vid.0, i + 1, n_intervals
-                            ));
+                        Err(read_err) => {
+                            // The shard is broken whether or not the needle survives it,
+                            // so report it either way.
                            broken_shards.insert(
                                *shard_id,
                                crate::pb::volume_server_pb::EcShardInfo {
@@ -441,7 +443,41 @@ pub async fn scrub_ec_volume_distributed(
                                    ..Default::default()
                                },
                            );
-                            break;
+                            if !recover_unreadable {
+                                errs.push(format!(
+                                    "failed to read EC shard {} for needle {} on volume {} (interval {}/{}): {}",
+                                    shard_id, id.0, vid.0, i + 1, n_intervals, read_err
+                                ));
+                                break;
+                            }
+                            match recover_one_remote_ec_shard_interval(
+                                state,
+                                vid,
+                                id,
+                                *shard_id,
+                                *shard_offset,
+                                *ssize,
+                                &locations,
+                                data_shards,
+                                total_shards - data_shards,
+                                snapshot.encode_ts_ns,
+                            )
+                            .await
+                            {
+                                // Same as the direct read above: a holder reporting the
+                                // needle deleted is authoritative and answers with no
+                                // bytes, so zero-fill and let the delete-state
+                                // suppression have it.
+                                Ok((_, true)) => data.resize(data.len() + *ssize, 0),
+                                Ok((buf, false)) => data.extend_from_slice(&buf),
+                                Err(e) => {
+                                    errs.push(format!(
+                                        "failed to recover EC shard {} for needle {} on volume {} (interval {}/{}): {}",
+                                        shard_id, id.0, vid.0, i + 1, n_intervals, e
+                                    ));
+                                    break;
+                                }
+                            }
                        }
                    }
                }