feat(scrub): EC FULL scrub — distributed local+remote needle walk (#10149)

* feat(ec): add scrub_ec_volume_distributed (FULL EC scrub, local+remote)

Ports Go's Store.ScrubEcVolume: walk the raw .ecx, verify every needle across
local AND remote shards without decoding (report faults, don't heal), with the
#10130 deleted-needle size-mismatch suppression gated on a force flag. Reuses
the read path's lock-drop + no-reconstruct read_remote_ec_shard_interval so no
!Send store guard is held across an .await.

Walks the unmasked index (scrub_snapshot_under_lock locates from the raw
(offset, size), not locate_needle) so logically-deleted-but-present needles are
still byte-verified, matching Go. Refreshes shard locations once up front and
hard-fails on a master-lookup error rather than retrying per needle.

Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo

* feat(scrub): dispatch EC FULL (mode 2) to the distributed needle walk

FULL ran a local-only Reed-Solomon parity check; route it to the per-needle
local+remote walk instead, mirroring Go. The handler collects vids under a brief
lock then releases it: FULL self-locks per needle (it awaits remote reads),
INDEX/LOCAL re-acquire a brief lock. verify_ec_shards is retained but no longer
wired to a mode.

Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
This commit is contained in:
Chris Lu authored and GitHub committed 2026-06-30 03:29:46 -07:00
1 parent 53087cb237
commit d18b85ef61
3 files changed
+277 -87

No files matched your search

@@ -208,6 +208,12 @@ pub fn rebuild_ec_files(
}
/// Verify EC shards by computing parity against the existing data and identifying corrupted shards.
///
/// No longer wired to a scrub mode: FULL is now Go's per-needle local+remote walk
/// (`store_ec::scrub_ec_volume_distributed`). Retained as a standalone Reed-Solomon
/// parity check — the only parity/cold-region verification available until the
/// `.ecsum` bitrot CHECKSUM path lands.
#[allow(dead_code)]
pub fn verify_ec_shards(
dir: &str,
collection: &str,