ec.rebuild: remount shards the rebuild replaced, keep regenerated copies (#12193)

* volume: remount ec shards the rebuild replaced under a mounted handle

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec.rebuild: keep copied shards the rebuild regenerated in place

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: reopen regenerated ec shards in place instead of remounting

UnmountEcShards and MountEcShards send deltas through separate buffered
channels, so a remount can reach the master as add-then-delete and drop
the repaired shard from lookups until the next full EC heartbeat.
ReopenEcShard swaps the mounted shard's descriptor under the volume lock
instead: registration, disk type, and heartbeat state never flip, and
readers holding the old shard degrade to a clean error the parity path
already recovers from.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: reopen rebuilt ec shards in place, mirroring Go

volume_ec_shards_rebuild recreates a shard file under a mounted handle
whose descriptor still serves the detached inode (a mounted shard whose
file was deleted under it). DiskLocation::reopen_ec_shard swaps the
mounted descriptor for a fresh open, matching Go's ReopenEcShard — the
registration and heartbeat view never flip.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: fail ec rebuild when a regenerated shard cannot be reopened

* rust volume: fail ec shards rebuild when a rebuilt shard cannot be reopened

* volume: repoint mounted ec shards after rebuild across disks

A mounted shard whose file was deleted under the mount is "missing" to
the rebuild: the regenerated file lands on the rebuild location while
the handle keeps serving the detached inode on another disk. Reopening
the mounted disk then failed the whole RPC, so the caller never saw the
rebuilt ids and its deferred cleanup deleted the regenerated copies.

Repoint every mounted handle of the rebuilt volume instead of only this
run's outputs: reopen in place when the rebuilt file is on the mounted
disk, otherwise drop the stale handle and remount the shard from
whichever local disk holds the file via a delta-free load split out of
MountEcShards — a same-server move, so the master's registration never
flips. A shard that can be neither reopened nor remounted is left
unmounted rather than serving replaced bytes, and the repoint step no
longer fails the RPC, so RebuiltShardIds always reaches the caller.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: repoint mounted ec shards after rebuild across disks

Mirror the Go repoint loop: iterate every mounted shard of the rebuilt
volume, reopen in place when the rebuilt file is on the mounted disk,
and otherwise drop the stale handle and remount the shard from
whichever local disk holds it. The repoint no longer fails the RPC, so
rebuilt_shard_ids still reaches the caller and its copy cleanup keeps
the regenerated shards.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: still rebuild ecx and repoint mounts when no shard is missing

* master: keep a node in an ec shard's lookup while a disk still holds it

A rebuild that relocates a mounted shard between a node's disks reports
the move as a delete on the source disk plus an add on the destination.
Registering first is a no-op because shard lookups key on the node, then
the unregister dropped the node outright — and steady-state heartbeats
never restore it, so readers and later rebuilds lose the repaired copy.
Skip the delete while another disk on the node still holds the shard.

UnregisterDataNode now clears the node's own ec records before dropping
its registrations, so the retained-shard check cannot skip the teardown,
and each unregister gets a ShardsInfo copy since DeleteEcShard drains
the shared one.

* rust volume: fold the journal on a mounted index instead of rewriting it

The rebuild handler regenerated .ecx through File::create even while the
volume was mounted, truncating the index under readers that search the
open descriptor with its old size — a live lookup could hit EOF mid-scan.
Go folds the deletion journal in place on a mounted volume; do the same
through rebuild_ecx_from_journal and reserve the full regen for the
unmounted path, where nothing holds the file open to interrupt.

* volume, rust volume: report rebuilt shards on late failures, fix rust corrupt discovery

Two review findings the repoint rework left open:

- A RebuildEcxFile failure still returns a plain error after shards were
  regenerated, so the caller discards the response holding the rebuilt
  ids and the deferred cleanup deletes copies the rebuild regenerated
  in place. The Go handler now packs the ids into the gRPC status
  details and the shell pulls them back out on error; the Rust handler
  carries them in grpc-status-details-bin the same way.
- The Rust rebuild still counted a sidecar-corrupt shard as present, so
  it fed bad bytes into reconstruction and never regenerated the shard.
  Discovery now loads the generation-0 .ecsum the way Go does, verifies
  every present shard, and reclassifies mismatches (and zero-size
  residue) as rebuild targets written to a .rebuilding temp file,
  verified, and atomically renamed over the original only after every
  output passes the sidecar check. Honors unsafe_ignore_sidecar, added
  to VolumeEcShardsRebuildRequest to match the Go proto.

* volume: repoint sibling journals sharing a folded .ecj after rebuild

RebuildEcxFile unlinks the deletion journal and the mounted path only
reopens its own handle — a sibling mount journaling into the same file
keeps appending to the detached inode, and those deletes never reach a
path-based reader again. Reopen every mounted volume sharing the index
base before leaving the rebuild.

* rust volume: repoint sibling journals sharing a folded .ecj after rebuild

rebuild_ecx_from_journal replaces the journal and reopens only its own
handle; mounts on sibling disks sharing that .ecj keep the detached
handle, so their next appends are lost at restart. Adopt the folded
journal on every mounted runtime sharing the path before releasing the
store lock, clearing their in-memory set — every journaled id is now an
.ecx tombstone.

* ec: swap the deletion journal handle only after the new file opens

A failed reopen left ecjFile nil, so every later delete returned
"ec volume closed". Open first, then close and swap — mirroring the
Rust adopt_folded_journal ordering.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec.rebuild: fail the rebuild when a sibling journal cannot be repointed

A warning left the sibling mount journaling to a detached inode while
the RPC reported success. Propagate the error like the Rust handler's
adopt_folded_journal map_err.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec: keep deletes loud when the journal reopen cannot open the replacement

Reverts the swap-on-success ordering: a detached handle kept on failure
lets deletes append and sync to the removed inode, then vanish at
restart. Closing first also frees the descriptor before the open.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: drop the folded journal handle before reopening it

adopt_folded_journal and the post-fold recreate now clear ecj_file
before opening the replacement, so a failed reopen leaves deletes
erroring instead of appending to the detached inode. Mirrors Go's
ReopenDeletionJournal.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec: drop the cached journal size when its reopen fails

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

seaweed-volume/src/storage/disk_location.rs

@@ -995,6 +995,32 @@ impl DiskLocation {
         Ok(())
     }
 
+    /// Swap the mounted shard's descriptor for a fresh open of its on-disk
+    /// file, keeping the registration — and therefore the master's view —
+    /// untouched. Mirrors Go's `DiskLocation.ReopenEcShard`: the rebuild
+    /// recreates a shard file under a handle that still serves the detached
+    /// inode, and reopening repoints reads at the repaired file without an
+    /// unmount/remount heartbeat flip-flop. Readers holding a cloned fd keep
+    /// the old inode until they finish — the same contract eviction relies on.
+    pub fn reopen_ec_shard(&mut self, vid: VolumeId, shard_id: ShardId) -> Result<bool, VolumeError> {
+        let dir = self.directory.clone();
+        let Some(ec_vol) = self.ec_volumes.get_mut(&vid) else {
+            return Ok(false);
+        };
+        let Some(existing) = ec_vol
+            .shards
+            .get_mut(shard_id as usize)
+            .and_then(|s| s.as_mut())
+        else {
+            return Ok(false);
+        };
+        let mut new_shard = EcVolumeShard::new(&dir, &existing.collection, vid, shard_id);
+        new_shard.disk_type = existing.disk_type.clone();
+        new_shard.open().map_err(VolumeError::Io)?;
+        *existing = new_shard;
+        Ok(true)
+    }
+
     /// Unmount EC shards for a volume on this location.
     ///
     /// Only shards that are actually mounted decrement the per-shard
@@ -2359,4 +2385,53 @@ mod tests {
             );
         }
     }
+
+    /// Mirrors Go's TestReopenEcShardReadsReplacedFile: a shard file the
+    /// rebuild replaced under its mount keeps serving the old inode through
+    /// the open descriptor; reopen_ec_shard must repoint the mounted shard at
+    /// the new file without dropping its registration.
+    #[test]
+    fn test_reopen_ec_shard_reads_replaced_file() {
+        let tmp = TempDir::new().unwrap();
+        let dir = tmp.path().to_str().unwrap();
+        let mut loc = DiskLocation::new(
+            dir,
+            dir,
+            10,
+            DiskType::HardDrive,
+            MinFreeSpace::Percent(0.0),
+            Vec::new(),
+        )
+        .unwrap();
+
+        std::fs::write(format!("{}/pics_5.ecx", dir), [0u8; 16]).unwrap();
+        let shard_path = format!("{}/pics_5.ec00", dir);
+        std::fs::write(&shard_path, b"old bytes").unwrap();
+        loc.mount_ec_shards(VolumeId(5), "pics", &[0], "").unwrap();
+
+        let read_shard = |loc: &DiskLocation| -> Vec<u8> {
+            let ec_vol = loc.find_ec_volume(VolumeId(5)).expect("ec volume mounted");
+            let shard = ec_vol.shards[0].as_ref().expect("shard 0 mounted");
+            let mut buf = vec![0u8; shard.file_size() as usize];
+            let n = shard.read_at(&mut buf, 0).unwrap();
+            buf.truncate(n);
+            buf
+        };
+        assert_eq!(read_shard(&loc), b"old bytes");
+
+        // The rebuild writes a fresh file and renames it over the original:
+        // the mounted descriptor is now on the detached inode.
+        let rebuilt = format!("{}.rebuilding", shard_path);
+        std::fs::write(&rebuilt, b"new bytes").unwrap();
+        std::fs::rename(&rebuilt, &shard_path).unwrap();
+        assert_eq!(read_shard(&loc), b"old bytes", "mounted shard still reads the replaced inode");
+
+        assert!(loc.reopen_ec_shard(VolumeId(5), 0).unwrap());
+        assert_eq!(read_shard(&loc), b"new bytes", "reopened shard should read the regenerated file");
+
+        assert!(
+            !loc.reopen_ec_shard(VolumeId(5), 7).unwrap(),
+            "a shard that is not mounted should report false"
+        );
+    }
 }

seaweed-volume/src/storage/erasure_coding/ec_bitrot.rs

@@ -147,6 +147,40 @@ pub fn bitrot_sidecar_path(base: &str, generation: u32) -> String {
     }
 }
 
+/// Resolves the sidecar path for a generation, searching the data base, the
+/// index base, and any additional directories — mirroring Go's
+/// `findBitrotSidecar`, which handles split data/idx layouts and per-disk
+/// mirrors the same way shard/.vif lookups do. Returns "" when no candidate
+/// exists.
+pub fn find_bitrot_sidecar(
+    generation: u32,
+    data_base: &str,
+    index_base: &str,
+    additional_dirs: &[&str],
+) -> String {
+    let mut candidates = vec![
+        bitrot_sidecar_path(data_base, generation),
+        bitrot_sidecar_path(index_base, generation),
+    ];
+    let base = std::path::Path::new(data_base)
+        .file_name()
+        .map(|n| n.to_string_lossy().into_owned())
+        .unwrap_or_default();
+    for dir in additional_dirs {
+        if dir.is_empty() || base.is_empty() {
+            continue;
+        }
+        candidates.push(bitrot_sidecar_path(
+            &format!("{}/{}", dir, base),
+            generation,
+        ));
+    }
+    candidates
+        .into_iter()
+        .find(|c| !c.is_empty() && std::path::Path::new(c).exists())
+        .unwrap_or_default()
+}
+
 /// Remove the bitrot checksum sidecars for a base file name: the legacy
 /// `<base>.ecsum` (generation 0) and any versioned `<base>.ecsum.v<N>`.
 ///

seaweed-volume/src/storage/erasure_coding/ec_encoder.rs

@@ -5,6 +5,7 @@
 
 use std::fs::File;
 use std::io;
+use std::io::Write;
 
 use reed_solomon_erasure::galois_8::ReedSolomon;
 
@@ -146,16 +147,33 @@ pub fn uniform_block_size(dat_file_size: i64, data_shards: usize) -> i64 {
 /// `additional_dirs` lists sibling disk locations to search for existing shards when
 /// they are not found in `dir`. This is required on multi-disk volume servers where
 /// shards for the same volume may be spread across disks.
+/// Rebuild EC shard files from the surviving shards.
+///
+/// `missing_shard_ids` are shards absent everywhere: each is written at
+/// `dir/<base>.ecNN`. `corrupt_shard_paths` are present shards the caller
+/// reclassified as corrupt (a bitrot sidecar mismatch or zero-size residue):
+/// each is written to `<discovered path>.rebuilding` and atomically renamed
+/// over the corrupt original only after every output verifies, so a failed
+/// rebuild leaves the untouched original rather than a half-written file —
+/// mirrors Go's `generateMissingEcFiles`. When `protection` is the volume's
+/// generation-0 sidecar, every regenerated shard is verified against it
+/// before publish: a mismatch means the sidecar is stale, and the rebuild
+/// fails closed by removing its outputs instead of blessing wrong bytes.
+///
+/// Shards absent from `dir` are read from `additional_dirs` (sibling disks on
+/// the same server), mirroring Go's findShardFile.
 pub fn rebuild_ec_files(
     dir: &str,
     collection: &str,
     volume_id: VolumeId,
     missing_shard_ids: &[u32],
+    corrupt_shard_paths: &[(u32, String)],
     data_shards: usize,
     parity_shards: usize,
     additional_dirs: &[&str],
+    protection: Option<&EcBitrotProtection>,
 ) -> io::Result<()> {
-    if missing_shard_ids.is_empty() {
+    if missing_shard_ids.is_empty() && corrupt_shard_paths.is_empty() {
         return Ok(());
     }
 
@@ -163,6 +181,21 @@ pub fn rebuild_ec_files(
         .map_err(|e| io::Error::other(format!("reed-solomon init: {:?}", e)))?;
 
     let total_shards = data_shards + parity_shards;
+
+    // The rebuild set: genuinely-missing shards plus reclassified-corrupt
+    // ones. Everything else is a Reed-Solomon input.
+    let mut is_rebuild = vec![false; total_shards];
+    for &i in missing_shard_ids {
+        if (i as usize) < total_shards {
+            is_rebuild[i as usize] = true;
+        }
+    }
+    for &(i, _) in corrupt_shard_paths {
+        if (i as usize) < total_shards {
+            is_rebuild[i as usize] = true;
+        }
+    }
+
     let mut shards: Vec<EcVolumeShard> = (0..total_shards as u8)
         .map(|i| EcVolumeShard::new(dir, collection, volume_id, i))
         .collect();
@@ -172,7 +205,7 @@ pub fn rebuild_ec_files(
     // on the same volume server) before failing — mirrors Go's findShardFile logic.
     let mut shard_size = 0;
     for (i, shard) in shards.iter_mut().enumerate() {
-        if !missing_shard_ids.contains(&(i as u32)) {
+        if !is_rebuild[i] {
             if shard.open().is_ok() {
                 let size = shard.file_size();
                 if size > shard_size {
@@ -210,12 +243,72 @@ pub fn rebuild_ec_files(
         ));
     }
 
-    // Create the missing shards for writing
-    for i in missing_shard_ids {
-        if let Some(shard) = shards.get_mut(*i as usize) {
-            shard.create()?;
+    // Create output files: a genuinely-absent shard is written at its final
+    // path; a reclassified-corrupt shard goes to a temp file beside its
+    // discovered path and is renamed over the original after verification.
+    // (final_path, write_path, file) — write_path differs only for corrupt.
+    let mut outputs: Vec<Option<(String, String, Option<File>)>> =
+        (0..total_shards).map(|_| None).collect();
+    let create_output = |path: &str| -> io::Result<File> {
+        crate::storage::volume_open::open_volume_file(
+            std::fs::OpenOptions::new()
+                .read(true)
+                .write(true)
+                .create(true)
+                .truncate(true),
+            path,
+        )
+    };
+    for &i in missing_shard_ids {
+        let idx = i as usize;
+        if idx >= total_shards {
+            continue;
         }
+        let final_path = format!(
+            "{}.ec{:02}",
+            volume_file_name(dir, collection, volume_id),
+            i
+        );
+        outputs[idx] = Some((
+            final_path.clone(),
+            final_path.clone(),
+            Some(create_output(&final_path)?),
+        ));
     }
+    for &(i, ref corrupt_path) in corrupt_shard_paths {
+        let idx = i as usize;
+        if idx >= total_shards {
+            continue;
+        }
+        let write_path = format!("{}.rebuilding", corrupt_path);
+        outputs[idx] = Some((
+            corrupt_path.clone(),
+            write_path.clone(),
+            Some(create_output(&write_path)?),
+        ));
+    }
+
+    // Removes every generated output on a failed fail-closed rebuild: temp
+    // replacements AND genuinely-missing shards written at their final path,
+    // so no unverified bytes are published. A reclassified-corrupt shard's
+    // untouched original (at final_path, distinct from its write_path) is left
+    // in place. Mirrors Go's cleanupRebuildOutputs.
+    let cleanup_outputs = |mut outputs: Vec<Option<(String, String, Option<File>)>>| {
+        let write_paths: Vec<String> = outputs
+            .iter_mut()
+            .flatten()
+            .map(|slot| {
+                // Take the handle so it closes before the unlink — Windows
+                // refuses to remove an open file.
+                let _ = slot.2.take();
+                slot.1.clone()
+            })
+            .collect();
+        drop(outputs);
+        for p in write_paths {
+            let _ = std::fs::remove_file(p);
+        }
+    };
 
     let block_size = ERASURE_CODING_SMALL_BLOCK_SIZE;
     let mut remaining = shard_size;
@@ -232,10 +325,11 @@ pub fn rebuild_ec_files(
         // shard; treat it as an error rather than reconstructing over a
         // zero-padded tail and publishing the result as restored redundancy.
         for (i, shard) in shards.iter().enumerate() {
-            if !missing_shard_ids.contains(&(i as u32)) {
+            if !is_rebuild[i] {
                 let mut buf = vec![0u8; to_process];
                 let n = shard.read_at(&mut buf, offset)?;
                 if n != to_process {
+                    cleanup_outputs(std::mem::take(&mut outputs));
                     return Err(io::Error::new(
                         io::ErrorKind::UnexpectedEof,
                         format!(
@@ -249,14 +343,25 @@ pub fn rebuild_ec_files(
         }
 
         // Reconstruct missing shards
-        rs.reconstruct(&mut buffers)
-            .map_err(|e| io::Error::other(format!("reed-solomon reconstruct: {:?}", e)))?;
+        if let Err(e) = rs.reconstruct(&mut buffers) {
+            cleanup_outputs(std::mem::take(&mut outputs));
+            return Err(io::Error::other(format!(
+                "reed-solomon reconstruct: {:?}",
+                e
+            )));
+        }
 
-        // Write recovered data into the missing shards
-        for i in missing_shard_ids {
-            let idx = *i as usize;
-            if let Some(buf) = buffers[idx].take() {
-                shards[idx].write_all(&buf)?;
+        // Write recovered data into the rebuilt shards
+        for i in 0..total_shards {
+            if is_rebuild[i] {
+                if let Some(buf) = buffers[i].take() {
+                    if let Some(f) = outputs[i].as_mut().and_then(|slot| slot.2.as_mut()) {
+                        if let Err(e) = f.write_all(&buf) {
+                            cleanup_outputs(std::mem::take(&mut outputs));
+                            return Err(e);
+                        }
+                    }
+                }
             }
         }
 
@@ -264,7 +369,80 @@ pub fn rebuild_ec_files(
         remaining -= to_process as i64;
     }
 
-    // Close all shards
+    // Verify regenerated shards against the sidecar. Reed-Solomon is
+    // deterministic, so a regenerated shard that does NOT match the sidecar
+    // means the sidecar is wrong/stale (not the shard) — fail closed rather
+    // than publishing bytes we cannot trust. Mirrors Go's post-rebuild
+    // verification pass.
+    if let Some(prot) = protection {
+        for slot in outputs.iter_mut().flatten() {
+            if let Some(f) = slot.2.as_mut() {
+                if let Err(e) = f.sync_all() {
+                    cleanup_outputs(std::mem::take(&mut outputs));
+                    return Err(e);
+                }
+            }
+        }
+        for i in 0..total_shards {
+            let Some((_, ref write_path, _)) = outputs[i] else {
+                continue;
+            };
+            let Some(entry) = ec_bitrot::shard_checksums(prot, i as u32) else {
+                continue;
+            };
+            match ec_bitrot::verify_shard_file_blocks(write_path, entry, prot.block_size as i64) {
+                Err(e) => {
+                    cleanup_outputs(std::mem::take(&mut outputs));
+                    return Err(io::Error::new(
+                        e.kind(),
+                        format!("bitrot: verify regenerated shard {}: {}", i, e),
+                    ));
+                }
+                Ok(mismatched) if !mismatched.is_empty() => {
+                    cleanup_outputs(std::mem::take(&mut outputs));
+                    return Err(io::Error::new(
+                        io::ErrorKind::InvalidData,
+                        format!(
+                            "bitrot: regenerated shard {} does not match sidecar ({} blocks differ); sidecar likely stale — aborting",
+                            i,
+                            mismatched.len()
+                        ),
+                    ));
+                }
+                _ => {}
+            }
+        }
+    }
+
+    // Flush and close every output, then atomically move reclassified-corrupt
+    // rebuilds over their originals — the rename lands only after all writes
+    // and the sidecar verification above have succeeded.
+    for slot in outputs.iter_mut().flatten() {
+        if let Some(f) = slot.2.as_mut() {
+            if let Err(e) = f.sync_all() {
+                cleanup_outputs(std::mem::take(&mut outputs));
+                return Err(e);
+            }
+        }
+    }
+    drop(outputs);
+    for &(i, ref corrupt_path) in corrupt_shard_paths {
+        if (i as usize) >= total_shards {
+            continue;
+        }
+        let write_path = format!("{}.rebuilding", corrupt_path);
+        if let Err(e) = std::fs::rename(&write_path, corrupt_path) {
+            return Err(io::Error::new(
+                e.kind(),
+                format!(
+                    "bitrot: replace corrupt shard {} ({} -> {}): {}",
+                    i, write_path, corrupt_path, e
+                ),
+            ));
+        }
+    }
+
+    // Close input shards
     for shard in &mut shards {
         shard.close();
     }
@@ -1016,7 +1194,7 @@ mod tests {
 
         // Rebuild a different (genuinely missing) shard.
         std::fs::remove_file(format!("{}/1.ec07", dir)).unwrap();
-        let res = rebuild_ec_files(&dir, "", VolumeId(1), &[7], 10, 4, &[]);
+        let res = rebuild_ec_files(&dir, "", VolumeId(1), &[7], &[], 10, 4, &[], None);
         assert!(res.is_err(), "truncated input shard must abort the rebuild");
     }
 
@@ -1029,7 +1207,7 @@ mod tests {
 
         let dropped = format!("{}/1.ec07", dir);
         std::fs::remove_file(&dropped).unwrap();
-        rebuild_ec_files(&dir, "", VolumeId(1), &[7], 10, 4, &[]).unwrap();
+        rebuild_ec_files(&dir, "", VolumeId(1), &[7], &[], 10, 4, &[], None).unwrap();
         assert!(
             std::path::Path::new(&dropped).exists(),
             "rebuilt shard .ec07 should exist"
@@ -1072,7 +1250,7 @@ mod tests {
 
         // Without additional_dirs the rebuild must fail: shards 0, 4, 8 are
         // not in primary and shard 7 cannot be reconstructed without them.
-        let res = rebuild_ec_files(&primary, "", VolumeId(1), &[7], 10, 4, &[]);
+        let res = rebuild_ec_files(&primary, "", VolumeId(1), &[7], &[], 10, 4, &[], None);
         assert!(
             res.is_err(),
             "rebuild without additional_dirs must fail when input shards are on another disk"
@@ -1088,9 +1266,11 @@ mod tests {
             "",
             VolumeId(1),
             &[7],
+            &[],
             10,
             4,
             &[secondary.as_str()],
+            None,
         )
         .unwrap();
 

seaweed-volume/src/storage/erasure_coding/ec_volume.rs

@@ -1939,8 +1939,7 @@ impl EcVolume {
     /// `deleted_needles` instead). The rebuild is atomic with respect to
     /// the journal: if any individual write fails the .ecj file is left
     /// in place and the error is propagated so tombstones are not lost.
-    #[expect(dead_code, reason = "no caller yet; see the doc comment")]
-    fn rebuild_ecx_from_journal(&mut self) -> io::Result<()> {
+    pub(crate) fn rebuild_ecx_from_journal(&mut self) -> io::Result<()> {
         let ecj_path = self.ecj_file_name();
         if !std::path::Path::new(&ecj_path).exists() {
             return Ok(());
@@ -1973,8 +1972,11 @@ impl EcVolume {
 
         // Fold successful — drop and recreate the journal, clear the
         // in-memory deleted set (all of its contents are now materialized
-        // in .ecx), and reset the cached size.
+        // in .ecx), and reset the cached size. The old handle goes first:
+        // if the recreate fails, deletes must error rather than keep
+        // appending to the removed inode.
         fs::remove_file(&ecj_path)?;
+        self.ecj_file = None;
         self.ecj_file = Some(open_ecj_append(&ecj_path)?);
         self.ecj_file_size = 0;
         if let Ok(mut set) = self.deleted_needles.write() {
@@ -1984,6 +1986,26 @@ impl EcVolume {
         Ok(())
     }
 
+    /// Adopt a journal another runtime sharing this `.ecj` path folded under
+    /// the store lock: the file was replaced, so swap the detached handle for
+    /// a fresh one, reset the cached size, and drop the in-memory set — every
+    /// id it held is now an `.ecx` tombstone. Without this, later appends
+    /// land on the removed inode and the deletes vanish at restart.
+    pub(crate) fn adopt_folded_journal(&mut self) -> io::Result<()> {
+        let ecj_path = self.ecj_file_name();
+        // The old handle points at the replaced file: drop it before the
+        // reopen so a failure leaves deletes erroring, not appending to a
+        // detached inode. Mirrors Go's ReopenDeletionJournal.
+        self.ecj_file = None;
+        let reopened = open_ecj_append(&ecj_path)?;
+        self.ecj_file_size = reopened.metadata()?.len() as i64;
+        self.ecj_file = Some(reopened);
+        if let Ok(mut set) = self.deleted_needles.write() {
+            set.clear();
+        }
+        Ok(())
+    }
+
     // ---- Deletion journal ----
 
     /// Record a needle delete: append the id to the .ecj deletion journal