ec.rebuild: remount shards the rebuild replaced, keep regenerated copies (#12193)

* volume: remount ec shards the rebuild replaced under a mounted handle

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec.rebuild: keep copied shards the rebuild regenerated in place

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: reopen regenerated ec shards in place instead of remounting

UnmountEcShards and MountEcShards send deltas through separate buffered
channels, so a remount can reach the master as add-then-delete and drop
the repaired shard from lookups until the next full EC heartbeat.
ReopenEcShard swaps the mounted shard's descriptor under the volume lock
instead: registration, disk type, and heartbeat state never flip, and
readers holding the old shard degrade to a clean error the parity path
already recovers from.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: reopen rebuilt ec shards in place, mirroring Go

volume_ec_shards_rebuild recreates a shard file under a mounted handle
whose descriptor still serves the detached inode (a mounted shard whose
file was deleted under it). DiskLocation::reopen_ec_shard swaps the
mounted descriptor for a fresh open, matching Go's ReopenEcShard — the
registration and heartbeat view never flip.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: fail ec rebuild when a regenerated shard cannot be reopened

* rust volume: fail ec shards rebuild when a rebuilt shard cannot be reopened

* volume: repoint mounted ec shards after rebuild across disks

A mounted shard whose file was deleted under the mount is "missing" to
the rebuild: the regenerated file lands on the rebuild location while
the handle keeps serving the detached inode on another disk. Reopening
the mounted disk then failed the whole RPC, so the caller never saw the
rebuilt ids and its deferred cleanup deleted the regenerated copies.

Repoint every mounted handle of the rebuilt volume instead of only this
run's outputs: reopen in place when the rebuilt file is on the mounted
disk, otherwise drop the stale handle and remount the shard from
whichever local disk holds the file via a delta-free load split out of
MountEcShards — a same-server move, so the master's registration never
flips. A shard that can be neither reopened nor remounted is left
unmounted rather than serving replaced bytes, and the repoint step no
longer fails the RPC, so RebuiltShardIds always reaches the caller.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: repoint mounted ec shards after rebuild across disks

Mirror the Go repoint loop: iterate every mounted shard of the rebuilt
volume, reopen in place when the rebuilt file is on the mounted disk,
and otherwise drop the stale handle and remount the shard from
whichever local disk holds it. The repoint no longer fails the RPC, so
rebuilt_shard_ids still reaches the caller and its copy cleanup keeps
the regenerated shards.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: still rebuild ecx and repoint mounts when no shard is missing

* master: keep a node in an ec shard's lookup while a disk still holds it

A rebuild that relocates a mounted shard between a node's disks reports
the move as a delete on the source disk plus an add on the destination.
Registering first is a no-op because shard lookups key on the node, then
the unregister dropped the node outright — and steady-state heartbeats
never restore it, so readers and later rebuilds lose the repaired copy.
Skip the delete while another disk on the node still holds the shard.

UnregisterDataNode now clears the node's own ec records before dropping
its registrations, so the retained-shard check cannot skip the teardown,
and each unregister gets a ShardsInfo copy since DeleteEcShard drains
the shared one.

* rust volume: fold the journal on a mounted index instead of rewriting it

The rebuild handler regenerated .ecx through File::create even while the
volume was mounted, truncating the index under readers that search the
open descriptor with its old size — a live lookup could hit EOF mid-scan.
Go folds the deletion journal in place on a mounted volume; do the same
through rebuild_ecx_from_journal and reserve the full regen for the
unmounted path, where nothing holds the file open to interrupt.

* volume, rust volume: report rebuilt shards on late failures, fix rust corrupt discovery

Two review findings the repoint rework left open:

- A RebuildEcxFile failure still returns a plain error after shards were
  regenerated, so the caller discards the response holding the rebuilt
  ids and the deferred cleanup deletes copies the rebuild regenerated
  in place. The Go handler now packs the ids into the gRPC status
  details and the shell pulls them back out on error; the Rust handler
  carries them in grpc-status-details-bin the same way.
- The Rust rebuild still counted a sidecar-corrupt shard as present, so
  it fed bad bytes into reconstruction and never regenerated the shard.
  Discovery now loads the generation-0 .ecsum the way Go does, verifies
  every present shard, and reclassifies mismatches (and zero-size
  residue) as rebuild targets written to a .rebuilding temp file,
  verified, and atomically renamed over the original only after every
  output passes the sidecar check. Honors unsafe_ignore_sidecar, added
  to VolumeEcShardsRebuildRequest to match the Go proto.

* volume: repoint sibling journals sharing a folded .ecj after rebuild

RebuildEcxFile unlinks the deletion journal and the mounted path only
reopens its own handle — a sibling mount journaling into the same file
keeps appending to the detached inode, and those deletes never reach a
path-based reader again. Reopen every mounted volume sharing the index
base before leaving the rebuild.

* rust volume: repoint sibling journals sharing a folded .ecj after rebuild

rebuild_ecx_from_journal replaces the journal and reopens only its own
handle; mounts on sibling disks sharing that .ecj keep the detached
handle, so their next appends are lost at restart. Adopt the folded
journal on every mounted runtime sharing the path before releasing the
store lock, clearing their in-memory set — every journaled id is now an
.ecx tombstone.

* ec: swap the deletion journal handle only after the new file opens

A failed reopen left ecjFile nil, so every later delete returned
"ec volume closed". Open first, then close and swap — mirroring the
Rust adopt_folded_journal ordering.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec.rebuild: fail the rebuild when a sibling journal cannot be repointed

A warning left the sibling mount journaling to a detached inode while
the RPC reported success. Propagate the error like the Rust handler's
adopt_folded_journal map_err.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec: keep deletes loud when the journal reopen cannot open the replacement

Reverts the swap-on-success ordering: a detached handle kept on failure
lets deletes append and sync to the removed inode, then vanish at
restart. Closing first also frees the descriptor before the open.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: drop the folded journal handle before reopening it

adopt_folded_journal and the post-fold recreate now clear ecj_file
before opening the replacement, so a failed reopen leaves deletes
erroring instead of appending to the detached inode. Mirrors Go's
ReopenDeletionJournal.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec: drop the cached journal size when its reopen fails

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

weed/ec/ec_rebuild.go

@@ -14,6 +14,7 @@ import (
 	"github.com/seaweedfs/seaweedfs/weed/storage/needle"
 	"github.com/seaweedfs/seaweedfs/weed/storage/types"
 	"github.com/seaweedfs/seaweedfs/weed/util"
+	"google.golang.org/grpc/status"
 )
 
 type ecRebuilder struct {
@@ -265,8 +266,19 @@ func (erb *ecRebuilder) rebuildOneEcVolume(collection string, volumeId needle.Vo
 		if !erb.applyChanges || len(copiedShardIds) == 0 {
 			return
 		}
-		if derr := SourceServerDeleteEcShards(erb.env.GrpcDialOption, collection, volumeId, pb.NewServerAddressFromDataNode(rebuilder.Info), copiedShardIds); derr != nil {
-			erb.write("%s delete copied ec shards %s %d.%v: %v\n", rebuilder.Info.Id, collection, volumeId, copiedShardIds, derr)
+		var orphaned []erasure_coding.ShardId
+		for _, shardId := range copiedShardIds {
+			// A copy the rebuild regenerated in place is mounted output, not a
+			// working file: deleting it strands the only good replica.
+			if !slices.Contains(generatedShardIds, shardId) {
+				orphaned = append(orphaned, shardId)
+			}
+		}
+		if len(orphaned) == 0 {
+			return
+		}
+		if derr := SourceServerDeleteEcShards(erb.env.GrpcDialOption, collection, volumeId, pb.NewServerAddressFromDataNode(rebuilder.Info), orphaned); derr != nil {
+			erb.write("%s delete copied ec shards %s %d.%v: %v\n", rebuilder.Info.Id, collection, volumeId, orphaned, derr)
 		}
 	}()
 	if err != nil {
@@ -306,6 +318,18 @@ func (erb *ecRebuilder) generateMissingShards(collection string, volumeId needle
 		})
 		if rebuildErr == nil {
 			rebuiltShardIds = erasure_coding.Uint32ToShardIds(resp.RebuiltShardIds)
+			return nil
+		}
+		// A rebuild that fails after regenerating shards still reports their
+		// ids in the status details: without them the deferred cleanup treats
+		// every copied shard — including ones the rebuild regenerated in
+		// place — as an orphan and deletes the repaired files.
+		if st, ok := status.FromError(rebuildErr); ok {
+			for _, detail := range st.Details() {
+				if r, ok := detail.(*volume_server_pb.VolumeEcShardsRebuildResponse); ok {
+					rebuiltShardIds = erasure_coding.Uint32ToShardIds(r.RebuiltShardIds)
+				}
+			}
 		}
 		return rebuildErr
 	})

weed/server/volume_grpc_ec_rebuild_reopen_test.go

@@ -0,0 +1,185 @@
+package weed_server
+
+import (
+	"bytes"
+	"context"
+	"crypto/rand"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
+	"github.com/seaweedfs/seaweedfs/weed/stats"
+	"github.com/seaweedfs/seaweedfs/weed/storage"
+	"github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
+	"github.com/seaweedfs/seaweedfs/weed/storage/needle"
+	"github.com/seaweedfs/seaweedfs/weed/storage/types"
+	"github.com/seaweedfs/seaweedfs/weed/util"
+	"github.com/stretchr/testify/require"
+)
+
+// A corrupt shard that stays mounted while VolumeEcShardsRebuild regenerates it
+// in place must stop serving the pre-repair bytes: the handler reopens the
+// mounted shard so reads hit the regenerated file, not the detached inode.
+// Regression test for ec.rebuild leaving a replaced shard file open until
+// restart.
+func TestVolumeEcShardsRebuild_ReopensMountedCorruptShard(t *testing.T) {
+	if !erasure_coding.BitrotProtectionEnabled {
+		t.Skip("bitrot protection is compiled out")
+	}
+	dataDir := filepath.Join(t.TempDir(), "data")
+	require.NoError(t, os.MkdirAll(dataDir, 0o755))
+	store := storage.NewStore(nil, "localhost", 8080, 18080, "http://localhost:8080", "store-id",
+		[]string{dataDir}, []int32{100}, []util.MinFreeSpace{{}}, "",
+		storage.NeedleMapInMemory, []types.DiskType{types.HardDriveType}, nil, 3, stats.DefaultDiskIOProbeConfig())
+	done := make(chan struct{})
+	go func() {
+		for {
+			select {
+			case <-store.NewVolumesChan:
+			case <-store.NewEcShardsChan:
+			case <-store.DeletedVolumesChan:
+			case <-store.DeletedEcShardsChan:
+			case <-store.StateUpdateChan:
+			case <-done:
+				return
+			}
+		}
+	}()
+	t.Cleanup(func() {
+		store.Close()
+		close(done)
+	})
+	st, err := storage.NewState(t.TempDir())
+	require.NoError(t, err)
+	store.State = st
+	vs := &VolumeServer{store: store}
+
+	collection := "col"
+	vid := needle.VolumeId(42)
+	base := erasure_coding.EcShardFileName(collection, dataDir, int(vid))
+
+	dat := make([]byte, 7000)
+	_, err = rand.Read(dat)
+	require.NoError(t, err)
+	require.NoError(t, os.WriteFile(base+".dat", dat, 0o644))
+	prot, err := erasure_coding.WriteEcFiles(base, nil)
+	require.NoError(t, err)
+	require.NoError(t, erasure_coding.SaveBitrotSidecar(erasure_coding.BitrotSidecarPath(base, 0), prot))
+	require.NoError(t, os.WriteFile(base+".ecx", make([]byte, 16), 0o644))
+
+	require.NoError(t, store.MountEcShards(collection, vid, 0, ""))
+	readShard := func() []byte {
+		shard, found := store.Locations[0].FindEcShard(vid, 0)
+		require.True(t, found)
+		buf := make([]byte, shard.Size())
+		_, err := shard.ReadAt(buf, 0)
+		require.NoError(t, err)
+		return buf
+	}
+	good := readShard()
+
+	// Corrupt the file in place: the mounted descriptor now serves bad bytes.
+	corrupt := bytes.Repeat([]byte{0xBB}, 1024)
+	f, err := os.OpenFile(base+erasure_coding.ToExt(0), os.O_WRONLY, 0)
+	require.NoError(t, err)
+	_, err = f.WriteAt(corrupt, 0)
+	require.NoError(t, err)
+	require.NoError(t, f.Close())
+	require.True(t, bytes.Equal(corrupt, readShard()[:len(corrupt)]))
+
+	_, err = vs.VolumeEcShardsRebuild(context.Background(), &volume_server_pb.VolumeEcShardsRebuildRequest{
+		VolumeId:   uint32(vid),
+		Collection: collection,
+	})
+	require.NoError(t, err)
+
+	require.True(t, bytes.Equal(good, readShard()),
+		"mounted shard should serve the regenerated file after rebuild")
+}
+
+// A shard whose file is deleted under the mount is "missing" to the rebuild:
+// it regenerates on the rebuild disk while the mounted handle still serves
+// the detached inode on another disk. Reopening the mounted disk cannot fix
+// that, so the handler moves the registration to the disk holding the rebuilt
+// file — and still reports the rebuilt ids so the caller's copy cleanup keeps
+// them.
+func TestVolumeEcShardsRebuild_RelocatesMountedShardAcrossDisks(t *testing.T) {
+	if !erasure_coding.BitrotProtectionEnabled {
+		t.Skip("bitrot protection is compiled out")
+	}
+	tempDir := t.TempDir()
+	dirA := filepath.Join(tempDir, "a")
+	dirB := filepath.Join(tempDir, "b")
+	require.NoError(t, os.MkdirAll(dirA, 0o755))
+	require.NoError(t, os.MkdirAll(dirB, 0o755))
+	store := storage.NewStore(nil, "localhost", 8080, 18080, "http://localhost:8080", "store-id",
+		[]string{dirA, dirB}, []int32{100, 100}, []util.MinFreeSpace{{}, {}}, "",
+		storage.NeedleMapInMemory, []types.DiskType{types.HardDriveType, types.HardDriveType}, nil, 3, stats.DefaultDiskIOProbeConfig())
+	done := make(chan struct{})
+	go func() {
+		for {
+			select {
+			case <-store.NewVolumesChan:
+			case <-store.NewEcShardsChan:
+			case <-store.DeletedVolumesChan:
+			case <-store.DeletedEcShardsChan:
+			case <-store.StateUpdateChan:
+			case <-done:
+				return
+			}
+		}
+	}()
+	t.Cleanup(func() {
+		store.Close()
+		close(done)
+	})
+	st, err := storage.NewState(t.TempDir())
+	require.NoError(t, err)
+	store.State = st
+	vs := &VolumeServer{store: store}
+
+	collection := "col"
+	vid := needle.VolumeId(42)
+	baseA := erasure_coding.EcShardFileName(collection, dirA, int(vid))
+	baseB := erasure_coding.EcShardFileName(collection, dirB, int(vid))
+	shardName := erasure_coding.ToExt(0)
+
+	dat := make([]byte, 7000)
+	_, err = rand.Read(dat)
+	require.NoError(t, err)
+	require.NoError(t, os.WriteFile(baseA+".dat", dat, 0o644))
+	prot, err := erasure_coding.WriteEcFiles(baseA, nil)
+	require.NoError(t, err)
+	require.NoError(t, erasure_coding.SaveBitrotSidecar(erasure_coding.BitrotSidecarPath(baseA, 0), prot))
+	require.NoError(t, os.WriteFile(baseA+".ecx", make([]byte, 16), 0o644))
+
+	original, err := os.ReadFile(baseA + shardName)
+	require.NoError(t, err)
+
+	// Mount shard 0 from disk B, then delete the file under the mount: the
+	// handle keeps serving the detached inode while the disk reports the
+	// shard missing.
+	require.NoError(t, os.Rename(baseA+shardName, baseB+shardName))
+	require.NoError(t, store.MountEcShards(collection, vid, 0, ""))
+	diskId, _, found := store.FindEcShard(vid, 0)
+	require.True(t, found)
+	require.Equal(t, uint32(1), diskId, "shard 0 should be mounted on disk B")
+	require.NoError(t, os.Remove(baseB+shardName))
+
+	resp, err := vs.VolumeEcShardsRebuild(context.Background(), &volume_server_pb.VolumeEcShardsRebuildRequest{
+		VolumeId:   uint32(vid),
+		Collection: collection,
+	})
+	require.NoError(t, err)
+	require.Equal(t, []uint32{0}, resp.RebuiltShardIds,
+		"a mounted shard whose file was rebuilt on another disk must still be reported rebuilt")
+
+	diskId, shard, found := store.FindEcShard(vid, 0)
+	require.True(t, found)
+	require.Equal(t, uint32(0), diskId, "rebuilt shard should be mounted on the disk holding its file")
+	buf := make([]byte, shard.Size())
+	_, err = shard.ReadAt(buf, 0)
+	require.NoError(t, err)
+	require.True(t, bytes.Equal(original, buf), "relocated shard should serve the regenerated file")
+}

weed/server/volume_grpc_erasure_coding.go

@@ -186,6 +186,19 @@ func recordEcRebuild(result string, d time.Duration) {
 	stats.VolumeServerECRebuildCounter.WithLabelValues(result).Inc()
 }
 
+// ecRebuildErrorWithIds reports a rebuild failure that happened after shards
+// were regenerated on disk. The error still carries their ids so the caller's
+// deferred cleanup keeps the copies it regenerated in place — to the shell an
+// unreported regenerated copy is just an orphan to delete, and a plain gRPC
+// error would drop the response holding them.
+func ecRebuildErrorWithIds(rebuiltShardIds []uint32, err error) error {
+	st := status.New(codes.Internal, err.Error())
+	if withIds, derr := st.WithDetails(&volume_server_pb.VolumeEcShardsRebuildResponse{RebuiltShardIds: rebuiltShardIds}); derr == nil {
+		st = withIds
+	}
+	return st.Err()
+}
+
 // VolumeEcShardsRebuild generates the any of the missing .ec00 ~ .ec13 files
 func (vs *VolumeServer) VolumeEcShardsRebuild(ctx context.Context, req *volume_server_pb.VolumeEcShardsRebuildRequest) (*volume_server_pb.VolumeEcShardsRebuildResponse, error) {
 	if err := vs.checkGrpcAdminAuth(ctx); err != nil {
@@ -280,13 +293,58 @@ func (vs *VolumeServer) VolumeEcShardsRebuild(ctx context.Context, req *volume_s
 			err = ev.ReopenDeletionJournal()
 		}
 		unlockJournal()
+		if err == nil {
+			// Mounts on sibling disks can journal into the same .ecj — the
+			// fold unlinked it, so repoint their handles too or their next
+			// append lands on the detached inode.
+			for _, location := range vs.store.Locations {
+				if location == rebuildLocation {
+					continue
+				}
+				if sibling, ok := location.FindEcVolume(needle.VolumeId(req.VolumeId)); ok && sibling.EcIndexBaseFileName() == indexBaseFileName {
+					unlockSibling := sibling.LockDeletionJournal()
+					serr := sibling.ReopenDeletionJournal()
+					unlockSibling()
+					if serr != nil {
+						err = fmt.Errorf("reopen shared ec journal on %s: %v", location.Directory, serr)
+						break
+					}
+				}
+			}
+		}
 		if err != nil {
 			recordEcRebuild("failure", time.Since(start))
-			return nil, fmt.Errorf("RebuildEcxFile %s: %v", indexBaseFileName, err)
+			return nil, ecRebuildErrorWithIds(rebuiltShardIds, fmt.Errorf("RebuildEcxFile %s: %v", indexBaseFileName, err))
 		}
 	} else if err := erasure_coding.RebuildEcxFile(indexBaseFileName); err != nil {
 		recordEcRebuild("failure", time.Since(start))
-		return nil, fmt.Errorf("RebuildEcxFile %s: %v", indexBaseFileName, err)
+		return nil, ecRebuildErrorWithIds(rebuiltShardIds, fmt.Errorf("RebuildEcxFile %s: %v", indexBaseFileName, err))
+	}
+
+	// A regenerated shard that was mounted keeps serving its old file through
+	// the open descriptor — the rename left the mounted handle on the detached
+	// inode. Repoint every mounted handle of this volume at the file now on
+	// disk, not just this run's outputs: a shard an earlier rebuild wrote on
+	// another disk is "present" this run and would never re-enter
+	// generatedShardIds, yet its mounted handle is still stale.
+	for _, location := range vs.store.Locations {
+		for _, shardId := range location.MountedEcShardIds(needle.VolumeId(req.VolumeId)) {
+			if _, rerr := location.ReopenEcShard(needle.VolumeId(req.VolumeId), shardId); rerr == nil {
+				continue
+			}
+			// The mounted disk lost the file and the rebuild wrote it
+			// elsewhere — or the file is unreadable in place. Drop the stale
+			// handle and remount from whichever local disk holds the rebuilt
+			// file; both steps emit no heartbeat delta, so the master's
+			// registration never flips. A shard that cannot be remounted is
+			// left unmounted — the honest state — rather than serving
+			// replaced bytes. Reporting the rebuilt ids on this path keeps
+			// the caller's cleanup from deleting them as strays.
+			location.UnloadEcShard(needle.VolumeId(req.VolumeId), shardId)
+			if _, _, lerr := vs.store.LoadEcShardAcrossDisks(req.Collection, needle.VolumeId(req.VolumeId), shardId); lerr != nil {
+				glog.Warningf("rebuilt ec shard %d.%d left unmounted: %v", req.VolumeId, shardId, lerr)
+			}
+		}
 	}
 
 	recordEcRebuild("success", time.Since(start))

weed/storage/disk_location_ec.go

@@ -255,6 +255,59 @@ func (l *DiskLocation) loadEcShardWithIdxDir(collection string, vid needle.Volum
 	return ecVolume, nil
 }
 
+// ReopenEcShard swaps the mounted shard's descriptor for a fresh open of its
+// on-disk file while leaving the volume's registration — and therefore the
+// master's view — untouched. It exists for the in-place rebuild path:
+// RebuildEcFiles renames a regenerated shard over the file a mounted handle is
+// still serving, and reopening repoints reads at the repaired file without
+// the delete/add heartbeat flip-flop an unmount+remount would send.
+func (l *DiskLocation) ReopenEcShard(vid needle.VolumeId, shardId erasure_coding.ShardId) (bool, error) {
+	l.ecVolumesLock.Lock()
+	defer l.ecVolumesLock.Unlock()
+
+	ecVolume, found := l.ecVolumes[vid]
+	if !found {
+		return false, nil
+	}
+	var oldShard *erasure_coding.EcVolumeShard
+	position := -1
+	for i, s := range ecVolume.Shards {
+		if s.ShardId == shardId {
+			position = i
+			oldShard = s
+			break
+		}
+	}
+	if oldShard == nil {
+		return false, nil
+	}
+
+	newShard, err := erasure_coding.NewEcVolumeShard(oldShard.DiskType, l.Directory, oldShard.Collection, vid, shardId)
+	if err != nil {
+		return true, fmt.Errorf("reopen ec shard %d.%d: %w", vid, shardId, err)
+	}
+	ecVolume.Shards[position] = newShard
+
+	// Readers that already resolved the old shard keep their object; its Close
+	// turns their next ReadAt into a clean error recovered via parity, the same
+	// contract eviction relies on. Mount/Unmount balance out, so the shard
+	// gauge does not move.
+	oldShard.Unmount()
+	oldShard.Close()
+	return true, nil
+}
+
+// MountedEcShardIds returns the ids of shards this disk currently serves for
+// the volume, or nil when the volume is not mounted here.
+func (l *DiskLocation) MountedEcShardIds(vid needle.VolumeId) []erasure_coding.ShardId {
+	l.ecVolumesLock.RLock()
+	defer l.ecVolumesLock.RUnlock()
+	if ecVolume, found := l.ecVolumes[vid]; found {
+		return ecVolume.ShardIdList()
+	}
+	return nil
+}
+
 func (l *DiskLocation) UnloadEcShard(vid needle.VolumeId, shardId erasure_coding.ShardId) bool {
 
 	l.ecVolumesLock.Lock()

weed/storage/erasure_coding/ec_volume_delete.go

@@ -132,6 +132,7 @@ func (ev *EcVolume) ReopenDeletionJournal() error {
 	}
 	ecjFile, err := backend.OpenVolumeFile(ev.FileName(".ecj"), os.O_RDWR|os.O_CREATE)
 	if err != nil {
+		ev.ecjFileSize = 0
 		return fmt.Errorf("reopen ec volume journal %s: %w", ev.FileName(".ecj"), err)
 	}
 	ev.ecjFile = ecjFile

weed/storage/store_ec.go

@@ -253,6 +253,41 @@ func (s *Store) CollectErasureCodingHeartbeat() *master_pb.Heartbeat {
 }
 
 func (s *Store) MountEcShards(collection string, vid needle.VolumeId, shardId erasure_coding.ShardId, sourceDiskType string) error {
+	ecVolume, diskId, err := s.LoadEcShardAcrossDisks(collection, vid, shardId)
+	if err != nil {
+		return err
+	}
+	glog.V(0).Infof("MountEcShards %d.%d on disk ID %d", vid, shardId, diskId)
+
+	// Apply the orchestrator-supplied source disk type so the EC
+	// volume reports under it instead of the location's. Empty means
+	// "fall back to location's disk type" (#9423).
+	if sourceDiskType != "" {
+		ecVolume.SetDiskType(types.ToDiskType(sourceDiskType))
+	}
+
+	si := erasure_coding.NewShardsInfo()
+	si.Set(erasure_coding.NewShardInfo(shardId, erasure_coding.ShardSize(ecVolume.ShardSize())))
+	s.NewEcShardsChan <- &master_pb.VolumeEcShardInformationMessage{
+		Id:          uint32(vid),
+		Collection:  collection,
+		EcIndexBits: uint32(si.Bitmap()),
+		ShardSizes:  si.SizesInt64(),
+		DiskType:    string(ecVolume.DiskType()),
+		ExpireAtSec: ecVolume.ExpireAtSec,
+		DiskId:      uint32(diskId),
+		EncodeTsNs:  ecVolume.EncodeTsNs,
+	}
+	s.UpdateMaxVolumeIdSeen(vid)
+	return nil
+}
+
+// LoadEcShardAcrossDisks mounts shardId on whichever local disk holds its
+// file and returns the mounted volume plus the disk it landed on. It emits
+// no heartbeat delta — callers that change the master's view do that
+// themselves (MountEcShards), while same-server repoints that must not flip
+// the master's registration use it directly.
+func (s *Store) LoadEcShardAcrossDisks(collection string, vid needle.VolumeId, shardId erasure_coding.ShardId) (*erasure_coding.EcVolume, uint32, error) {
 	// The .ecx index file may live on a different disk than the one
 	// holding the .ec?? shard being mounted: ec.balance / ec.rebuild can
 	// place the .ecx on one local disk while later distributing shards
@@ -292,29 +327,7 @@ func (s *Store) MountEcShards(collection string, vid needle.VolumeId, shardId er
 		}
 		ecVolume, err := location.loadEcShardWithIdxDir(collection, vid, shardId, idxDir)
 		if err == nil {
-			glog.V(0).Infof("MountEcShards %d.%d on disk ID %d", vid, shardId, diskId)
-
-			// Apply the orchestrator-supplied source disk type so the EC
-			// volume reports under it instead of the location's. Empty means
-			// "fall back to location's disk type" (#9423).
-			if sourceDiskType != "" {
-				ecVolume.SetDiskType(types.ToDiskType(sourceDiskType))
-			}
-
-			si := erasure_coding.NewShardsInfo()
-			si.Set(erasure_coding.NewShardInfo(shardId, erasure_coding.ShardSize(ecVolume.ShardSize())))
-			s.NewEcShardsChan <- &master_pb.VolumeEcShardInformationMessage{
-				Id:          uint32(vid),
-				Collection:  collection,
-				EcIndexBits: uint32(si.Bitmap()),
-				ShardSizes:  si.SizesInt64(),
-				DiskType:    string(ecVolume.DiskType()),
-				ExpireAtSec: ecVolume.ExpireAtSec,
-				DiskId:      uint32(diskId),
-				EncodeTsNs:  ecVolume.EncodeTsNs,
-			}
-			s.UpdateMaxVolumeIdSeen(vid)
-			return nil
+			return ecVolume, uint32(diskId), nil
 		}
 		if errors.Is(err, os.ErrNotExist) {
 			// Shard or index not on this disk; another disk may own it.
@@ -329,9 +342,9 @@ func (s *Store) MountEcShards(collection string, vid needle.VolumeId, shardId er
 		// scanned for so the operator can tell "no .ecx anywhere" apart
 		// from "shard not on this server".
 		if !ecxFound {
-			return fmt.Errorf("MountEcShards %d.%d: no .ecx index found on any local disk", vid, shardId)
+			return nil, 0, fmt.Errorf("MountEcShards %d.%d: no .ecx index found on any local disk", vid, shardId)
 		}
-		return fmt.Errorf("MountEcShards %d.%d not found on disk", vid, shardId)
+		return nil, 0, fmt.Errorf("MountEcShards %d.%d not found on disk", vid, shardId)
 	}
 	// Some disks returned a real (non-ENOENT) error. Report them all so
 	// the caller can see whether the failures cluster around one disk
@@ -343,7 +356,7 @@ func (s *Store) MountEcShards(collection string, vid needle.VolumeId, shardId er
 		}
 		b = append(b, fmt.Sprintf("%s: %v", f.dir, f.err)...)
 	}
-	return fmt.Errorf("MountEcShards %d.%d load failures: %s", vid, shardId, string(b))
+	return nil, 0, fmt.Errorf("MountEcShards %d.%d load failures: %s", vid, shardId, string(b))
 }
 
 func (s *Store) UnmountEcShards(vid needle.VolumeId, shardId erasure_coding.ShardId, reqEncodeTsNs int64) error {

weed/storage/store_ec_rebuild_remount_test.go

@@ -0,0 +1,64 @@
+package storage
+
+import (
+	"bytes"
+	"os"
+	"testing"
+
+	"github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
+	"github.com/seaweedfs/seaweedfs/weed/storage/needle"
+	"github.com/stretchr/testify/require"
+)
+
+// A shard file the rebuild replaced under its mount keeps serving the old
+// inode through the open descriptor. ReopenEcShard must repoint the mounted
+// shard at the new file without dropping its registration.
+func TestReopenEcShardReadsReplacedFile(t *testing.T) {
+	store := newEcTargetTestStore(t, 1)
+	collection := "col"
+	vid := needle.VolumeId(42)
+	dataDir := store.Locations[0].Directory
+
+	writeEcShard0(t, dataDir, collection, vid)
+	writeEcIndex(t, dataDir, collection, vid)
+
+	shardPath := erasure_coding.EcShardFileName(collection, dataDir, int(vid)) + erasure_coding.ToExt(0)
+	content := make([]byte, calculateExpectedShardSize(1024*1024, 10))
+	for i := range content {
+		content[i] = 'A'
+	}
+	require.NoError(t, os.WriteFile(shardPath, content, 0o644))
+
+	require.NoError(t, store.MountEcShards(collection, vid, 0, ""))
+
+	readShard := func() []byte {
+		shard, found := store.Locations[0].FindEcShard(vid, 0)
+		require.True(t, found)
+		buf := make([]byte, len(content))
+		n, err := shard.ReadAt(buf, 0)
+		require.NoError(t, err)
+		require.Equal(t, len(content), n)
+		return buf
+	}
+	require.True(t, bytes.Equal(content, readShard()))
+
+	for i := range content {
+		content[i] = 'B'
+	}
+	rebuilt := shardPath + ".rebuilding"
+	require.NoError(t, os.WriteFile(rebuilt, content, 0o644))
+	require.NoError(t, os.Rename(rebuilt, shardPath))
+
+	stale := readShard()
+	require.False(t, bytes.Equal(content, stale), "mounted shard should still read the replaced inode")
+
+	reopened, err := store.Locations[0].ReopenEcShard(vid, 0)
+	require.NoError(t, err)
+	require.True(t, reopened, "shard is mounted; reopen should find it")
+
+	require.True(t, bytes.Equal(content, readShard()), "reopened shard should read the regenerated file")
+
+	reopened, err = store.Locations[0].ReopenEcShard(vid, 7)
+	require.NoError(t, err)
+	require.False(t, reopened, "a shard that is not mounted should report false")
+}

weed/topology/data_node_ec.go

@@ -140,6 +140,29 @@ func (dn *DataNode) HasEcShards(volumeId needle.VolumeId) (found bool) {
 	return
 }
 
+// HasEcShard reports whether any disk on this node still holds the shard.
+// Registration lookups key on the node alone, so a shard moving between its
+// disks must not drop the node from the lookup while a copy remains.
+func (dn *DataNode) HasEcShard(volumeId needle.VolumeId, shardId erasure_coding.ShardId) (found bool) {
+	dn.RLock()
+	defer dn.RUnlock()
+	for _, c := range dn.children {
+		disk := c.(*Disk)
+		disk.ecShardsLock.RLock()
+		for _, ev := range disk.ecShards[volumeId] {
+			if ev.ShardsInfo.Has(shardId) {
+				found = true
+				break
+			}
+		}
+		disk.ecShardsLock.RUnlock()
+		if found {
+			return true
+		}
+	}
+	return
+}
+
 // doUpdateEcShardsLocked rewrites disk.ecShards from actualByKey. Caller
 // MUST hold dn.Lock; each disk's ecShardsLock is taken internally around
 // the read-modify-write of its ecShards map.

weed/topology/data_node_ec_multi_disk_test.go

@@ -161,6 +161,76 @@ func TestEcShardsAfterRestartHeartbeat(t *testing.T) {
 	}
 }
 
+// TestEcShardRelocationAcrossDisksKeepsLookup covers a rebuild relocating a
+// mounted shard between a node's disks: the source disk reports a delete and
+// the destination an add, but lookups key on the node — registering first is
+// a no-op, so the unregister must not drop a node that still holds the shard
+// on another of its disks.
+func TestEcShardRelocationAcrossDisksKeepsLookup(t *testing.T) {
+	topo := NewTopology("weedfs", sequence.NewMemorySequencer(), 32*1024, 5, false)
+	dc := topo.GetOrCreateDataCenter("dc1")
+	rack := dc.GetOrCreateRack("rack1")
+	dn := rack.GetOrCreateDataNode("127.0.0.1", 34534, 0, "127.0.0.1", "", map[string]uint32{"": 100})
+
+	const vid = uint32(15)
+	listed := func(shardId erasure_coding.ShardId) bool {
+		locs, ok := topo.LookupEcShards(needle.VolumeId(vid))
+		if !ok {
+			return false
+		}
+		for _, n := range locs.Locations[shardId] {
+			if n.Id() == dn.Id() {
+				return true
+			}
+		}
+		return false
+	}
+
+	// shard 3 sits on disk 0, then ec.rebuild relocates it to disk 1.
+	topo.SyncDataNodeEcShards([]*master_pb.VolumeEcShardInformationMessage{
+		buildEcShardMessage(vid, "c", "", 0, []erasure_coding.ShardId{3}),
+	}, dn)
+	topo.SyncDataNodeEcShards([]*master_pb.VolumeEcShardInformationMessage{
+		buildEcShardMessage(vid, "c", "", 1, []erasure_coding.ShardId{3}),
+	}, dn)
+	if !listed(3) {
+		t.Fatalf("shard 3 moved disk0->disk1 dropped node %s from lookup", dn.Id())
+	}
+
+	// The same move reported as an incremental delta keeps the lookup too.
+	topo.IncrementalSyncDataNodeEcShards(
+		[]*master_pb.VolumeEcShardInformationMessage{buildEcShardMessage(vid, "c", "", 0, []erasure_coding.ShardId{5})},
+		[]*master_pb.VolumeEcShardInformationMessage{buildEcShardMessage(vid, "c", "", 1, []erasure_coding.ShardId{5})},
+		dn)
+	if !listed(5) {
+		t.Fatalf("shard 5 moved incrementally dropped node %s from lookup", dn.Id())
+	}
+
+	// A shard duplicated across two disks keeps the node listed when one
+	// copy goes away.
+	topo.SyncDataNodeEcShards([]*master_pb.VolumeEcShardInformationMessage{
+		buildEcShardMessage(vid, "c", "", 0, []erasure_coding.ShardId{7}),
+		buildEcShardMessage(vid, "c", "", 1, []erasure_coding.ShardId{3, 5, 7}),
+	}, dn)
+	topo.SyncDataNodeEcShards([]*master_pb.VolumeEcShardInformationMessage{
+		buildEcShardMessage(vid, "c", "", 1, []erasure_coding.ShardId{3, 5, 7}),
+	}, dn)
+	if !listed(7) {
+		t.Fatalf("shard 7 still held on disk 1 dropped node %s from lookup", dn.Id())
+	}
+
+	// A shard gone from every disk does leave the lookup.
+	topo.SyncDataNodeEcShards([]*master_pb.VolumeEcShardInformationMessage{
+		buildEcShardMessage(vid, "c", "", 1, []erasure_coding.ShardId{3, 5}),
+	}, dn)
+	if listed(7) {
+		t.Fatalf("shard 7 deleted on all disks still lists node %s", dn.Id())
+	}
+	if !listed(3) || !listed(5) {
+		t.Fatalf("shards 3 and 5 on disk 1 lost node %s", dn.Id())
+	}
+}
+
 func buildEcShardMessage(vid uint32, collection, diskType string, diskId uint32, shardIds []erasure_coding.ShardId) *master_pb.VolumeEcShardInformationMessage {
 	si := erasure_coding.NewShardsInfo()
 	for _, sid := range shardIds {

weed/topology/topology_ec.go

@@ -159,6 +159,9 @@ func (t *Topology) UnRegisterEcShards(ecvi *erasure_coding.EcVolumeInfo, dn *Dat
 		return
 	}
 	for _, shardId := range ecvi.ShardsInfo.Ids() {
+		if dn.HasEcShard(ecvi.VolumeId, shardId) {
+			continue
+		}
 		locations.DeleteShard(shardId, dn)
 	}
 }

weed/topology/topology_event_handling.go

@@ -5,7 +5,6 @@ import (
 	"time"
 
 	"github.com/seaweedfs/seaweedfs/weed/stats"
-	"github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
 	"github.com/seaweedfs/seaweedfs/weed/storage/types"
 	"google.golang.org/grpc"
 
@@ -123,9 +122,15 @@ func (t *Topology) UnRegisterDataNode(dn *DataNode) {
 		defer t.releaseResolvedQuarantine(v.Id)
 	}
 
-	// unregister ec shards when volume server disconnected
+	// unregister ec shards when volume server disconnected. The node's own
+	// records clear first so the still-held check cannot skip the drop, and
+	// each unregister gets a copy — DeleteEcShard drains the shared
+	// ShardsInfo.
 	for _, s := range dn.GetEcShards() {
-		t.UnRegisterEcShards(s, dn)
+		held := *s
+		held.ShardsInfo = s.ShardsInfo.Copy()
+		dn.DeleteEcShard(s)
+		t.UnRegisterEcShards(&held, dn)
 	}
 
 	negativeUsages := dn.GetDiskUsages().negative()
@@ -133,7 +138,6 @@ func (t *Topology) UnRegisterDataNode(dn *DataNode) {
 		dn.UpAdjustDiskUsageDelta(dt, du)
 	}
 	dn.DeltaUpdateVolumes([]storage.VolumeInfo{}, dn.GetVolumes())
-	dn.DeltaUpdateEcShards([]*erasure_coding.EcVolumeInfo{}, dn.GetEcShards())
 	t.unregisterDataNodeAddress(dn.ServerAddress(), dn)
 	if dn.Parent() != nil {
 		dn.Parent().UnlinkChildNode(dn.Id())