fix(storage): prune partial EC shards when sibling disk has healthy .dat (#9478) (#9480)

* fix(storage): prune partial EC shards when sibling disk has healthy .dat (#9478)

handleFoundEcxFile only checks for .dat in the same disk location as the
EC shards. In a multi-disk volume server an interrupted encode can leave
.ec?? + .ecx on disk B while the source .dat still lives on disk A: the
per-disk loader sees no .dat next to .ecx, mistakes the leftover for a
distributed-EC layout, and mounts the partial shards. The volume server
then heartbeats both a regular replica and an EC shard for the same vid
and the master keeps both.

Sweep the store after per-disk loading and before the cross-disk
reconcile to delete partial EC files when a healthy .dat for the same
(collection, vid) exists on a sibling disk. Push DeletedEcShardsChan for
every pruned shard so master forgets the new-shard message the per-disk
pass already emitted, instead of waiting for the next periodic heartbeat.

* fix(seaweed-volume): mirror prune of partial EC with sibling .dat (#9478)

Rust port of the same Store-level prune added to weed/storage. The
per-disk EC loader in disk_location.rs only checks for .dat in the same
disk as the EC shards, so an interrupted encode that leaves .ec?? + .ecx
on disk B while the source .dat sits on disk A is mounted as if it were
a distributed-EC layout. The volume server then heartbeats both a
regular replica and an EC shard for the same vid.

Sweep the store after per-disk loading and before the cross-disk
reconcile, dropping in-memory EcVolumes with fewer than DATA_SHARDS_COUNT
shards when a .dat for the same (collection, vid) exists on a sibling
disk, and remove all on-disk EC artefacts for them. The Rust heartbeat
path already diff-emits deletes from the next ec_volumes snapshot, so no
explicit delete-channel push is needed here.

Tests cover both the issue 9478 layout and a distributed-EC layout with
no .dat anywhere on the store, which must be left alone.

* fix(storage): validate sibling .dat size before deleting partial EC (#9478)

The earlier prune deleted partial EC files whenever any .dat for the
same vid existed on a sibling disk — including a zero-byte shell. A
shell is no more useful than the partial shard it would replace, and
the partial shard might still combine with shards on other servers
in a recoverable distributed-EC layout. Wiping it based on a corrupt
sibling .dat is data loss masquerading as cleanup.

Tighten the check: when the EC's .vif recorded a non-zero source size
in datFileSize, require the sibling .dat to be at least that many
bytes; otherwise fall back to "at least a superblock". The .vif value
is what the encoder wrote at the moment the source was sealed, so a
sibling .dat smaller than that is provably truncated. Carry the size
through indexDatOwners alongside the location.

The Rust port had the same gap and an additional bug behind it:
EcVolume::new wasn't reading datFileSize from .vif, so the safety
check always fell back to the superblock floor. Wire datFileSize
through. The existing shard-size calculation in
LocateEcShardNeedleInterval already uses dat_file_size when non-zero,
so populating it also matches Go's behaviour there.

Tests cover the truncated-sibling case in both ports.
This commit is contained in:
Chris Lu authored and GitHub committed 2026-05-13 09:25:10 -07:00
1 parent 3f1eaf9724
commit de28c4df61
8 files changed
+909 -9

No files matched your search

@@ -120,10 +120,15 @@ impl EcVolume {
shards.push(None);
}
// Read expire_at_sec and version from .vif if present (matches Go's MaybeLoadVolumeInfo).
// Prefer the data dir; fall back to the idx dir for the
// cross-disk reconcile case (#9212 / #9244).
let (expire_at_sec, vif_version) = {
// Read expire_at_sec, version, and dat_file_size from .vif if
// present (matches Go's MaybeLoadVolumeInfo). Prefer the data
// dir; fall back to the idx dir for the cross-disk reconcile
// case (#9212 / #9244). `dat_file_size` is the source .dat
// size at encode time, used both by `locate_ec_shard_needle`
// for shard-size math and by the Store-level prune in
// `store_ec_reconcile.rs` to verify a sibling-disk .dat is
// plausibly the encoding source (#9478).
let (expire_at_sec, vif_version, vif_dat_file_size) = {
let vif_path = locate_vif_path(dir, dir_idx, collection, volume_id);
if let Ok(vif_content) = std::fs::read_to_string(&vif_path) {
if let Ok(vif_info) =
@@ -134,12 +139,12 @@ impl EcVolume {
} else {
Version::current()
};
(vif_info.expire_at_sec, ver)
(vif_info.expire_at_sec, ver, vif_info.dat_file_size)
} else {
(0, Version::current())
(0, Version::current(), 0)
}
} else {
(0, Version::current())
(0, Version::current(), 0)
}
};
@@ -150,7 +155,7 @@ impl EcVolume {
dir_idx: dir_idx.to_string(),
version: vif_version,
shards,
dat_file_size: 0,
dat_file_size: vif_dat_file_size,
data_shards,
parity_shards,
ecx_file: None,