mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-06 03:57:48 +01:00
fix(ec): correct EC FULL scrub for deleted needles, shard-location cache, and parity coverage (#10152)
* fix(ec): correct EC FULL scrub for deleted needles + shard-location cache
Addresses review findings on the EC FULL distributed scrub:
- Remote EC reads now thread Go's (bytes, is_deleted) contract. A runtime EC
delete keeps the .ecx size positive (the delete lives in .ecj/memory), so the
raw-index walk verifies the needle, and its header interval is usually remote;
the peer answers is_deleted with no payload. The scrub zero-fills that interval
(so the needle reaches read_bytes -> SizeMismatch{0} -> the delete-state
suppression), the serving direct read short-circuits to not-found, and
reconstruction EXCLUDES the shard instead of feeding zeros into Reed-Solomon.
- The walk skips size.is_deleted() (not just is_tombstone), so a -originalSize
.ecx entry (pre-encode delete) can't yield empty intervals or panic parse_header.
- Restore Go's < data_shards completeness guard (per-volume, custom-ratio aware)
and per-shard merge in the location cache instead of clobber-with-partial.
- Abort the scrub with an error on mid-scan unmount instead of a false-CLEAN.
- Hoist the refreshed location map once instead of cloning it per needle.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(scrub): keep RS parity check in EC FULL until CHECKSUM lands
The per-needle FULL walk only reads live data-shard intervals, so it can't catch
bitrot in a parity shard or an unwalked cold region. Run verify_ec_shards
alongside the walk, gated on all-shards-local (single-node EC), via spawn_blocking.
A deliberate temporary divergence from Go FULL; moves to mode 4 (CHECKSUM) once
the .ecsum subsystem lands.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
This commit is contained in:
1 parent
a85318111c
commit
8f2a2abae4
4 files changed
+266
-72
No files matched your search
@@ -207,13 +207,12 @@ pub fn rebuild_ec_files(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Verify EC shards by computing parity against the existing data and identifying corrupted shards.
|
||||
///
|
||||
/// No longer wired to a scrub mode: FULL is now Go's per-needle local+remote walk
|
||||
/// (`store_ec::scrub_ec_volume_distributed`). Retained as a standalone Reed-Solomon
|
||||
/// parity check — the only parity/cold-region verification available until the
|
||||
/// `.ecsum` bitrot CHECKSUM path lands.
|
||||
#[allow(dead_code)]
|
||||
/// Reed-Solomon parity check over the locally-held shards: recompute parity from
|
||||
/// the data shards and flag any shard whose bytes disagree. Wired into FULL
|
||||
/// (scrub mode 2) as a TEMPORARY local parity/cold-region check — the per-needle
|
||||
/// FULL walk only reads live data-shard intervals, so on its own it can't catch
|
||||
/// bitrot in a parity shard or an unwalked region. Move to mode 4 (CHECKSUM) and
|
||||
/// drop it from mode 2 once the `.ecsum` subsystem lands.
|
||||
pub fn verify_ec_shards(
|
||||
dir: &str,
|
||||
collection: &str,
|
||||
|
||||
Reference in new issue
Block a user