fix(ec): correct EC FULL scrub for deleted needles, shard-location cache, and parity coverage (#10152)

* fix(ec): correct EC FULL scrub for deleted needles + shard-location cache

Addresses review findings on the EC FULL distributed scrub:
- Remote EC reads now thread Go's (bytes, is_deleted) contract. A runtime EC
  delete keeps the .ecx size positive (the delete lives in .ecj/memory), so the
  raw-index walk verifies the needle, and its header interval is usually remote;
  the peer answers is_deleted with no payload. The scrub zero-fills that interval
  (so the needle reaches read_bytes -> SizeMismatch{0} -> the delete-state
  suppression), the serving direct read short-circuits to not-found, and
  reconstruction EXCLUDES the shard instead of feeding zeros into Reed-Solomon.
- The walk skips size.is_deleted() (not just is_tombstone), so a -originalSize
  .ecx entry (pre-encode delete) can't yield empty intervals or panic parse_header.
- Restore Go's < data_shards completeness guard (per-volume, custom-ratio aware)
  and per-shard merge in the location cache instead of clobber-with-partial.
- Abort the scrub with an error on mid-scan unmount instead of a false-CLEAN.
- Hoist the refreshed location map once instead of cloning it per needle.

Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo

* feat(scrub): keep RS parity check in EC FULL until CHECKSUM lands

The per-needle FULL walk only reads live data-shard intervals, so it can't catch
bitrot in a parity shard or an unwalked cold region. Run verify_ec_shards
alongside the walk, gated on all-shards-local (single-node EC), via spawn_blocking.
A deliberate temporary divergence from Go FULL; moves to mode 4 (CHECKSUM) once
the .ecsum subsystem lands.

Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
This commit is contained in:
Chris Lu authored and GitHub committed 2026-06-30 10:18:31 -07:00
1 parent a85318111c
commit 8f2a2abae4
4 files changed
+266 -72

No files matched your search

@@ -207,13 +207,12 @@ pub fn rebuild_ec_files(
Ok(())
}
/// Verify EC shards by computing parity against the existing data and identifying corrupted shards.
///
/// No longer wired to a scrub mode: FULL is now Go's per-needle local+remote walk
/// (`store_ec::scrub_ec_volume_distributed`). Retained as a standalone Reed-Solomon
/// parity check — the only parity/cold-region verification available until the
/// `.ecsum` bitrot CHECKSUM path lands.
#[allow(dead_code)]
/// Reed-Solomon parity check over the locally-held shards: recompute parity from
/// the data shards and flag any shard whose bytes disagree. Wired into FULL
/// (scrub mode 2) as a TEMPORARY local parity/cold-region check — the per-needle
/// FULL walk only reads live data-shard intervals, so on its own it can't catch
/// bitrot in a parity shard or an unwalked region. Move to mode 4 (CHECKSUM) and
/// drop it from mode 2 once the `.ecsum` subsystem lands.
pub fn verify_ec_shards(
dir: &str,
collection: &str,