volume server: ec.decode verifies, cleans up and compacts like Go, off the runtime (#11547)

* volume server: ec.decode reads the .ecx from the index dir it was copied to

VolumeEcShardsCopy writes the .ecx/.ecj into the receiver's -dir.idx, so
with a split data/index dir the decode target has no .ecx beside its
shards. VolumeEcShardsToVolume sized the .dat from the right .ecx but
built the .idx from the data dir, failing with NotFound after the .dat
was already published. It now reads .ecx/.ecj from where the EC volume
opened them and writes the .idx beside the .dat, where Go leaves it.

The live-entry check and the .dat size also ignored deletions recorded
only in the .ecj, which Go folds into the .ecx (RebuildEcxFile) first:
a fully deleted volume was decoded instead of reported as having no live
entries, and deleted tail needles were copied into the .dat. Both now
treat journaled ids as deleted, without rewriting the sealed .ecx.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode keeps the decoded volume writable and reads every .ecj

The rebuilt .idx copied a journaled tail needle's .ecx row verbatim after
the .dat was cut short before it, so the mount saw a row past EOF and
marked the decoded volume read-only. Rows of deleted needles the .dat no
longer holds are now dropped, and each journaled needle still in the .dat
gets one tombstone instead of one per journal entry.

VolumeEcShardsCopy appends journals collected from other holders into
the idx dir, but the decode read only the .ecj beside the .ecx, which
sits in the data dir when this server generated the shards. It now
reads both, once, in bounded chunks via the loader EcVolume uses.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: test ec.decode drops a sealed .ecx tail tombstone

Covers the other half of the rule added in the previous commit: a tail
needle tombstoned in the .ecx itself (Go's RebuildEcxFile) is cut from
the .dat, and its row must not reach the rebuilt .idx either.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode runs its file I/O off the async runtime

VolumeEcShardsToVolume released the store lock before decoding, but read
the .ecx/.ecj, rebuilt the .dat and wrote the .idx inside the async
handler, parking a runtime worker for the length of a volume-sized copy.
The decode now runs in spawn_blocking on inputs snapshotted under the
store lock.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode checks the rebuilt .dat is complete

Go stats the decoded .dat before writing the .idx (VerifyDecodedDatFile)
and fails the decode when it is shorter than the extent the EC index
references, since the caller deletes the shards once the call returns.
The Rust handler returned success without that check. The rebuild
already fails on a short shard read, so this guards the published file
itself.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode drops the decoded volume's bitrot sidecars

Go removes <base>.ecsum and <base>.ecsum.v<N> beside the .dat and beside
the .ecx once the .idx is written, so a stale checksum sidecar cannot
pass for the protection of a later re-encode. The Rust handler left them
in place. Removal is best effort, as in Go.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode compacts the decoded volume

Go ends VolumeEcShardsToVolume with an offline CompactVolumeFiles, so the
decoded volume holds only live needles. The Rust decode left every needle
deleted through the .ecj in the .dat, tombstoned in the .idx, until a
later vacuum reclaimed it.

Store::compact_volume_files loads the unmounted volume, checks free space
the way the vacuum does (the estimate now lives in one helper), and runs
the vacuum's compact-by-index and commit. As in Go a failed compaction is
logged and the decode still succeeds, so the uncompacted .idx rules stay:
the tests that pin them now make the compaction fail.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode keeps deletes journaled while the .dat is written

The decode read the .ecj journals once, before rebuilding the .dat, so a
delete that reached the EC volume during the rebuild was left out of the
new .idx and the needle came back live. Each journal's read length is now
kept, and the bytes appended since are read just before the .idx is
written, after waiting out any journal append in flight (appends hold
the store write lock), so every delete acknowledged by then is in the
.idx. A delete after that point is still lost, as in Go.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Guard overlapping ec decode requests; serialize journal catch-up

volume_ec_shards_to_volume runs its decode in spawn_blocking, so a
dropped request leaves the job running and a retry would race it on the
temporary and final volume files. Claim the vid in a per-server
in-flight set until the blocking job finishes, and return Unavailable
to an overlapping request. The Go handler has the same exposure and
gets the same guard.

Journal appends hold the store write lock through their
sync-or-truncate, so holding a read lock across the catch-up read
guarantees every record it sees is committed: a rolled-back delete can
no longer leave a tombstone in the decoded index.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Reconcile the swap when offline compaction commit fails

A CommitCompact that fails after the .cpc marker may have renamed .dat
but not .idx. cleanup_compact refuses while the marker exists, so the
mismatched pair survived until a restart reconciled it — and the decode
caller treats the failure as non-fatal. Run reconcileCompactState on
commit failure so a decided swap rolls forward and orphan temps are
removed before the volume can mount.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Release the decode claim on panic

* volume: add ec_decodes_in_flight to the integration-test state literal

* volume server: hold the decode tail's lock through compaction

The catch_up read released before the rebuilt .idx was written and the
volume compacted, so a delete synced to .ecj in that window was durably
journaled yet absent from the published index — resurrecting the needle.
Rust now holds the store read lock from catch_up through compact, and Go
mirrors it by holding the volume's journal lock from the journal-
consuming index write through CompactVolumeFiles.

* volume server: serialize ec decode's tail per volume, not per store

Review follow-ups on the decode path:

- Rust: holding the store read lock from journal catch-up through the
  offline compaction stalled every writer on unrelated volumes for the
  whole rewrite. The new ec_decode_tail set marks the vid only while its
  .idx is published and .cpd/.cpx swapped; the two local .ecj append paths
  (VolumeEcBlobDelete, the distributed delete's local journal) wait on a
  Notify for that span — Go's per-volume ecjFileAccessLock semantics
  without the global stall. VolumeMount and the staged-adopt path are also
  held off while a decode claim is in flight so neither can race the swap.

- Rust: the initial journal read ran unlocked, so bytes a rolled-back
  append later truncated could be folded in as phantom tombstones. The
  first pass stays unlocked (a slow journal must not stall the store) and
  a rescan under the quiescing read lock re-reads only committed content;
  catch_up now rebuilds the id set when a regular journal shrank.

- Go: the decode resolved the compaction DiskLocation through
  FindEcVolume while holding the journal lock, inverting DestroyEcVolume's
  map->journal order into a deadlock. The lookup now happens first, and
  DestroyEcVolume/deleteEcVolumeById/DiskLocation.Close destroy outside
  the map lock.

- Go: RebuildEcxFile unlinks .ecj while the volume's ecjFile handle stays
  open, so later deletes could commit to a detached inode. Both call sites
  now fold under the journal lock and ReopenDeletionJournal repoints the
  handle at the live path, working on the volume's resolved .ecx dir
  (EcIndexBaseFileName) rather than the configured index dir.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: fence EC remounts behind the destroy tombstone

DestroyEcVolume, deleteEcVolumeById, and the collection-delete sweep now
remove the EcVolume from ecVolumes before destroying it off-lock, so a
concurrent remount could re-open shard files that the in-flight destroy
then unlinks — registering a detached fd.

Each destroy records a per-vid tombstone channel in a new
ecVolumesDestroying map before dropping the map entry and closes it when
Destroy returns. The tombstone intentionally survives as the vid's
destroy generation: loadEcShardWithIdxDir compares it before and after
opening the shard, so a destroy that both started and finished inside the
open window is still detected. A mismatch drops the just-opened shard
(releasing its fd and mount gauge) and retries after the destroy
completes; a successful mount clears the stale tombstone.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: rescan the .ecj under the store lock only after a rollback

The decode's second journal pass ran a full rescan under the store read
lock on every decode, stalling unrelated writers for the length of the
scan. Bump a process-wide epoch whenever a failed append truncates its
uncommitted tail; an unchanged epoch between the unlocked read and the
quiesced pass proves every id folded in was committed, so catch_up()
suffices. catch_up() also treats a journal that was read but has since
disappeared as shrunk to zero, so its earlier ids cannot linger.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: check the decode tail under the store write lock on delete

A blob delete waited for the publishing tail before taking the store
write lock, so a decode that claimed the tail while the delete was
parked behind the decoder's read lock could still see the journal append
land after the rebuilt .idx — an acknowledged delete the mount would
miss. Test tail membership under the write lock instead, retrying after
the wait; journal_delete_local reports WouldBlock for the same recheck
on the distributed path.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: claim the vid for mount and staged adoption, per volume

VolumeMount and the staged .copying adoption held the
ec_decodes_in_flight set lock through slow file renames and mounts,
stalling every unrelated volume's decode, mount, and adoption. Take the
per-volume claim instead — the same exclusion against a racing decode
for this vid, released when the call returns.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: fail the decode when a compaction commit marker survives

CompactVolumeFiles' caller logged a compaction error and went on to
delete the EC shards. When the commit marker (.cpc) is still on disk the
.dat/.idx swap was decided but could not be reconciled, so the mounted
pair may be mismatched — report the failure instead so the shards are
kept and the caller can retry.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: gate the parked-delete test on the held write lock

The releaser thread and the spawned delete raced for the store write
lock; on a slow runner the delete could acquire it first and commit
before the tail was ever claimed, failing !delete.is_finished() on the
Windows unit-test job. Spawn the delete only after the thread reports
the lock held.

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

weed/server/volume_grpc_erasure_coding.go

@@ -268,7 +268,23 @@ func (vs *VolumeServer) VolumeEcShardsRebuild(ctx context.Context, req *volume_s
 	if !util.FileExists(indexBaseFileName+".ecx") && rebuildLocation.IdxDirectory != rebuildLocation.Directory {
 		indexBaseFileName = path.Join(rebuildLocation.Directory, baseFileName)
 	}
-	if err := erasure_coding.RebuildEcxFile(indexBaseFileName); err != nil {
+	if ev, found := rebuildLocation.FindEcVolume(needle.VolumeId(req.VolumeId)); found {
+		// A mounted volume appends runtime deletes through ev.ecjFile, so the
+		// fold runs under its journal lock on the journal's own base, and the
+		// handle is repointed: otherwise RebuildEcxFile's unlink strands later
+		// appends on the detached inode.
+		indexBaseFileName = ev.EcIndexBaseFileName()
+		unlockJournal := ev.LockDeletionJournal()
+		err := erasure_coding.RebuildEcxFile(indexBaseFileName)
+		if err == nil {
+			err = ev.ReopenDeletionJournal()
+		}
+		unlockJournal()
+		if err != nil {
+			recordEcRebuild("failure", time.Since(start))
+			return nil, fmt.Errorf("RebuildEcxFile %s: %v", indexBaseFileName, err)
+		}
+	} else if err := erasure_coding.RebuildEcxFile(indexBaseFileName); err != nil {
 		recordEcRebuild("failure", time.Since(start))
 		return nil, fmt.Errorf("RebuildEcxFile %s: %v", indexBaseFileName, err)
 	}
@@ -1138,6 +1154,11 @@ func (vs *VolumeServer) VolumeEcShardsToVolume(ctx context.Context, req *volume_
 		return vs.adoptStagedVolume(req)
 	}
 
+	if _, loaded := vs.ecDecodesInFlight.LoadOrStore(req.VolumeId, struct{}{}); loaded {
+		return nil, status.Errorf(codes.Unavailable, "ec volume %d is already being decoded", req.VolumeId)
+	}
+	defer vs.ecDecodesInFlight.Delete(req.VolumeId)
+
 	// Collect all EC shards (NewEcVolume will load EC config from .vif into v.ECContext)
 	// Use MaxShardCount (32) to support custom EC ratios up to 32 total shards
 	tempShards := make([]string, erasure_coding.MaxShardCount)
@@ -1168,18 +1189,50 @@ func (vs *VolumeServer) VolumeEcShardsToVolume(ctx context.Context, req *volume_
 		}
 	}
 
-	dataBaseFileName, indexBaseFileName := v.DataBaseFileName(), v.IndexBaseFileName()
-	if !util.FileExists(indexBaseFileName + ".ecx") {
-		indexBaseFileName = dataBaseFileName
+	dataBaseFileName := v.DataBaseFileName()
+	// The fold and the index write must work on the same .ecj that runtime
+	// deletes append to through ecjFile — the volume's resolved index dir,
+	// which can differ from IndexBaseFileName when an .ecx copy exists in
+	// both the data and index directories.
+	indexBaseFileName := v.EcIndexBaseFileName()
+
+	// Resolve the offline-compaction location before taking the journal
+	// lock: FindEcVolume takes the ec-volume map lock, and acquiring it inside
+	// the journal lock inverts DestroyEcVolume's map->journal order — a decode
+	// holding the journal lock while waiting on the map deadlocks a destroy
+	// holding the map lock while waiting on the journal.
+	var volumeLocation *storage.DiskLocation
+	for _, location := range vs.store.Locations {
+		if candidate, found := location.FindEcVolume(needle.VolumeId(req.VolumeId)); found && candidate == v {
+			volumeLocation = location
+			break
+		}
+	}
+	if volumeLocation == nil {
+		return nil, fmt.Errorf("ec volume %d location not found for offline compaction", req.VolumeId)
 	}
 
 	// Merge .ecj deletions into .ecx so that HasLiveNeedles and FindDatFileSize
 	// see the full set of deleted needles. Without this, needles deleted after the
 	// last ecx rebuild would still appear live, causing the decoded .dat to include
 	// data that should be skipped and HasLiveNeedles to return a false positive.
+	//
+	// The journal lock is held across the fold: RebuildEcxFile unlinks .ecj,
+	// and a delete committed between its read and the unlink would land on the
+	// detached inode that ecjFile keeps open — synced, successful, and
+	// invisible to every path-based reader. ReopenDeletionJournal then points
+	// the handle back at a fresh journal, so deletes committed during the .dat
+	// rebuild stay durable and reach the index write below.
+	unlockJournal := v.LockDeletionJournal()
 	if err := erasure_coding.RebuildEcxFile(indexBaseFileName); err != nil {
+		unlockJournal()
 		return nil, fmt.Errorf("RebuildEcxFile %s: %v", indexBaseFileName, err)
 	}
+	if err := v.ReopenDeletionJournal(); err != nil {
+		unlockJournal()
+		return nil, fmt.Errorf("reopen deletion journal %s: %v", indexBaseFileName, err)
+	}
+	unlockJournal()
 
 	// If the EC index contains no live entries, decoding should be a no-op:
 	// just allow the caller to purge EC shards and do not generate an empty normal volume.
@@ -1215,6 +1268,11 @@ func (vs *VolumeServer) VolumeEcShardsToVolume(ctx context.Context, req *volume_
 		return nil, err
 	}
 
+	// Runtime deletes serialize on the volume's journal lock; holding it from
+	// the journal-consuming index write through the offline compaction keeps a
+	// committed delete from slipping past the rebuilt .idx.
+	defer v.LockDeletionJournal()()
+
 	// write .idx file from .ecx and .ecj files
 	if err := erasure_coding.WriteIdxFileFromEcIndex(indexBaseFileName); err != nil {
 		return nil, fmt.Errorf("WriteIdxFileFromEcIndex %s: %v", v.IndexBaseFileName(), err)
@@ -1227,17 +1285,6 @@ func (vs *VolumeServer) VolumeEcShardsToVolume(ctx context.Context, req *volume_
 		removeBitrotSidecars(indexBaseFileName)
 	}
 
-	var volumeLocation *storage.DiskLocation
-	for _, location := range vs.store.Locations {
-		if candidate, found := location.FindEcVolume(needle.VolumeId(req.VolumeId)); found && candidate == v {
-			volumeLocation = location
-			break
-		}
-	}
-	if volumeLocation == nil {
-		return nil, fmt.Errorf("ec volume %d location not found for offline compaction", req.VolumeId)
-	}
-
 	if err := vs.store.CompactVolumeFiles(
 		needle.VolumeId(req.VolumeId),
 		v.Collection,
@@ -1247,6 +1294,13 @@ func (vs *VolumeServer) VolumeEcShardsToVolume(ctx context.Context, req *volume_
 		0,
 		vs.compactionBytePerSecond,
 	); err != nil {
+		// Benign only when the swap never started or was settled: a surviving
+		// .cpc marker means the commit was decided but the renames could not
+		// be reconciled, so .dat/.idx may be a mismatched pair — fail the
+		// decode and let the caller keep the shards.
+		if util.FileExists(dataBaseFileName + ".cpc") {
+			return nil, fmt.Errorf("CompactVolumeFiles %s: %w", dataBaseFileName, err)
+		}
 		glog.Errorf("CompactVolumeFiles %s: %v", dataBaseFileName, err)
 	}
 

weed/server/volume_server.go

@@ -60,6 +60,11 @@ type VolumeServer struct {
 	fileSizeLimitBytes            int64
 	isHeartbeating                bool
 	stopChan                      chan bool
+
+	// Volumes with an EC decode in flight. A cancelled request does not
+	// stop the handler; this keeps a retry from racing it on the same
+	// volume files.
+	ecDecodesInFlight sync.Map
 }
 
 func NewVolumeServer(adminMux, publicMux *http.ServeMux, ip string,

weed/storage/disk_location.go

@@ -48,6 +48,11 @@ type DiskLocation struct {
 	// erasure coding
 	ecVolumes     map[needle.VolumeId]*erasure_coding.EcVolume
 	ecVolumesLock sync.RWMutex
+	// vids whose EcVolume was removed from ecVolumes but is still being
+	// destroyed off-lock; the channel closes when Destroy returns. A remount
+	// of the vid must wait for it, or the dying volume could unlink files
+	// the new one just opened.
+	ecVolumesDestroying map[needle.VolumeId]chan struct{}
 
 	ecShardNotifyHandler func(collection string, vid needle.VolumeId, shardId erasure_coding.ShardId, ecVolume *erasure_coding.EcVolume)
 
@@ -117,6 +122,7 @@ func NewDiskLocation(dir string, maxVolumeCount int32, minFreeSpace util.MinFree
 	}
 	location.volumes = make(map[needle.VolumeId]*Volume)
 	location.ecVolumes = make(map[needle.VolumeId]*erasure_coding.EcVolume)
+	location.ecVolumesDestroying = make(map[needle.VolumeId]chan struct{})
 	location.closeCh = make(chan struct{})
 	go func() {
 		location.CheckDiskSpace(config)
@@ -474,8 +480,12 @@ func (l *DiskLocation) DeleteCollectionFromDiskLocation(collection string) (dele
 	}()
 
 	go func() {
-		for _, v := range delEcVolsMap {
+		for k, v := range delEcVolsMap {
 			v.Destroy()
+			l.ecVolumesLock.Lock()
+			done := l.ecVolumesDestroying[k]
+			l.ecVolumesLock.Unlock()
+			close(done)
 		}
 		wg.Done()
 	}()
@@ -649,11 +659,20 @@ func (l *DiskLocation) Close() {
 	l.volumesLock.Unlock()
 
 	l.ecVolumesLock.Lock()
-	for _, ecVolume := range l.ecVolumes {
-		ecVolume.Close()
+	ecVolumes := make([]*erasure_coding.EcVolume, 0, len(l.ecVolumes))
+	for vid, ecVolume := range l.ecVolumes {
+		ecVolumes = append(ecVolumes, ecVolume)
+		delete(l.ecVolumes, vid)
 	}
 	l.ecVolumesLock.Unlock()
 
+	// Close outside the write lock: EcVolume.Close takes the deletion-journal
+	// lock, which a running ec.decode can hold — closing under the map lock
+	// would stall every EC lookup and invert the map->journal lock order.
+	for _, ecVolume := range ecVolumes {
+		ecVolume.Close()
+	}
+
 	close(l.closeCh)
 	return
 }

weed/storage/disk_location_ec.go

@@ -24,6 +24,17 @@ var (
 	re = regexp.MustCompile(`\.ec\d{2,3}`)
 )
 
+// markEcDestroying records a pending destroy for vid and returns the channel
+// closed when it finishes. Callers must hold ecVolumesLock for writing.
+func (l *DiskLocation) markEcDestroying(vid needle.VolumeId) chan struct{} {
+	if l.ecVolumesDestroying == nil {
+		l.ecVolumesDestroying = make(map[needle.VolumeId]chan struct{})
+	}
+	done := make(chan struct{})
+	l.ecVolumesDestroying[vid] = done
+	return done
+}
+
 func (l *DiskLocation) FindEcVolume(vid needle.VolumeId) (*erasure_coding.EcVolume, bool) {
 	l.ecVolumesLock.RLock()
 	defer l.ecVolumesLock.RUnlock()
@@ -37,13 +48,27 @@ func (l *DiskLocation) FindEcVolume(vid needle.VolumeId) (*erasure_coding.EcVolu
 
 func (l *DiskLocation) DestroyEcVolume(vid needle.VolumeId) {
 	l.ecVolumesLock.Lock()
-	defer l.ecVolumesLock.Unlock()
-
 	ecVolume, found := l.ecVolumes[vid]
+	var done chan struct{}
 	if found {
-		ecVolume.Destroy()
+		done = l.markEcDestroying(vid)
 		delete(l.ecVolumes, vid)
 	}
+	l.ecVolumesLock.Unlock()
+
+	if !found {
+		return
+	}
+	// Destroy outside the write lock: EcVolume.Destroy's Close waits on the
+	// deletion-journal lock, which a running ec.decode can hold — under the
+	// map lock that wait would stall every EC lookup and invert the
+	// map->journal lock order.
+	ecVolume.Destroy()
+
+	// The tombstone stays after close() as this vid's destroy generation: a
+	// remount compares it before and after opening files to spot a destroy
+	// that ran inside the window. The next successful mount clears it.
+	close(done)
 }
 
 // UnloadEcVolume drops the in-memory EcVolume for vid from this one disk without
@@ -141,17 +166,57 @@ func (l *DiskLocation) LoadEcShard(collection string, vid needle.VolumeId, shard
 // (issue #9212).
 func (l *DiskLocation) loadEcShardWithIdxDir(collection string, vid needle.VolumeId, shardId erasure_coding.ShardId, idxDir string) (*erasure_coding.EcVolume, error) {
 
-	ecVolumeShard, err := erasure_coding.NewEcVolumeShard(l.DiskType, l.Directory, collection, vid, shardId)
-	if err != nil {
-		if err == os.ErrNotExist {
-			return nil, os.ErrNotExist
+	// A destroy-in-progress or already-finished one leaves a tombstone; it
+	// stays in the map until a mount clears it, so comparing the channel
+	// before and after opening the shard detects any destroy that could have
+	// unlinked the files in between — including one that ran to completion
+	// inside the window.
+	var ecVolumeShard *erasure_coding.EcVolumeShard
+	for {
+		l.ecVolumesLock.Lock()
+		gen := l.ecVolumesDestroying[vid]
+		l.ecVolumesLock.Unlock()
+		if gen != nil {
+			select {
+			case <-gen:
+				// Destroy already finished; the closed tombstone is the
+				// generation token the post-open recheck compares against.
+			default:
+				// Destroy in flight: its unlinks could detach anything this
+				// mount opens, so wait it out and take a fresh generation.
+				<-gen
+				continue
+			}
 		}
-		return nil, fmt.Errorf("failed to create ec shard %d.%d: %w", vid, shardId, err)
+
+		var err error
+		ecVolumeShard, err = erasure_coding.NewEcVolumeShard(l.DiskType, l.Directory, collection, vid, shardId)
+		if err != nil {
+			if err == os.ErrNotExist {
+				return nil, os.ErrNotExist
+			}
+			return nil, fmt.Errorf("failed to create ec shard %d.%d: %w", vid, shardId, err)
+		}
+
+		l.ecVolumesLock.Lock()
+		if cur := l.ecVolumesDestroying[vid]; cur != gen {
+			// A destroy intervened while the shard was being opened; its
+			// unlink may have detached the file. Drop the handle, wait for
+			// the destroy, and retry on a clean slate.
+			l.ecVolumesLock.Unlock()
+			ecVolumeShard.Unmount() // release the gauge the constructor's Mount took
+			ecVolumeShard.Close()
+			if cur != nil {
+				<-cur
+			}
+			continue
+		}
+		break
 	}
-	l.ecVolumesLock.Lock()
 	defer l.ecVolumesLock.Unlock()
 	ecVolume, found := l.ecVolumes[vid]
 	if !found {
+		var err error
 		ecVolume, err = erasure_coding.NewEcVolume(l.DiskType, l.Directory, idxDir, collection, vid)
 		if err != nil {
 			// Wrap with %w so MountEcShards / startup reconcile can use
@@ -160,6 +225,9 @@ func (l *DiskLocation) loadEcShardWithIdxDir(collection string, vid needle.Volum
 			return nil, fmt.Errorf("failed to create ec volume %d: %w", vid, err)
 		}
 		l.ecVolumes[vid] = ecVolume
+		// Fresh generation for this vid: the next destroy sets a new
+		// tombstone rather than matching the cleared one.
+		delete(l.ecVolumesDestroying, vid)
 	}
 	added, err := ecVolume.AddEcVolumeShard(ecVolumeShard)
 	if err != nil {
@@ -389,16 +457,21 @@ func (l *DiskLocation) loadEcShardsWithIdxDir(shards []string, collection string
 }
 
 func (l *DiskLocation) deleteEcVolumeById(vid needle.VolumeId) (e error) {
-	// Add write lock since we're modifying the ecVolumes map
 	l.ecVolumesLock.Lock()
-	defer l.ecVolumesLock.Unlock()
-
 	ecVolume, ok := l.ecVolumes[vid]
+	var done chan struct{}
+	if ok {
+		done = l.markEcDestroying(vid)
+		delete(l.ecVolumes, vid)
+	}
+	l.ecVolumesLock.Unlock()
+
 	if !ok {
 		return
 	}
+	// Destroy outside the map lock — see DestroyEcVolume.
 	ecVolume.Destroy()
-	delete(l.ecVolumes, vid)
+	close(done)
 	return
 }
 
@@ -410,8 +483,11 @@ func (l *DiskLocation) unmountEcVolumeByCollection(collectionName string) map[ne
 		}
 	}
 
-	for k, _ := range deltaVols {
+	for k := range deltaVols {
 		delete(l.ecVolumes, k)
+		// Caller destroys these outside the lock; tombstone until then so a
+		// remount can't open files the destroy is about to unlink.
+		l.markEcDestroying(k)
 	}
 	return deltaVols
 }

weed/storage/disk_location_ec_test.go

@@ -847,6 +847,113 @@ func TestLoadEcShardDuplicateReleasesTheNewShard(t *testing.T) {
 	}
 }
 
+// TestLoadEcShardWaitsForPendingDestroy: DestroyEcVolume deletes the map entry
+// and destroys files outside ecVolumesLock, so a remount could otherwise open
+// shard files mid-unlink. The ecVolumesDestroying tombstone must hold the mount
+// until the destroy closes it, and the mount must then proceed and clear it.
+func TestLoadEcShardWaitsForPendingDestroy(t *testing.T) {
+	dir := t.TempDir()
+	diskLocation := NewDiskLocation(dir, 10, util.MinFreeSpace{}, dir, types.HardDriveType, nil, stats.DefaultDiskIOProbeConfig())
+	defer diskLocation.Close()
+
+	if err := os.WriteFile(filepath.Join(dir, "125.ec00"), []byte("shard bytes"), 0o644); err != nil {
+		t.Fatalf("seed .ec00: %v", err)
+	}
+	if err := os.WriteFile(filepath.Join(dir, "125.ecx"), make([]byte, 16), 0o644); err != nil {
+		t.Fatalf("seed .ecx: %v", err)
+	}
+
+	done := make(chan struct{})
+	diskLocation.ecVolumesLock.Lock()
+	diskLocation.ecVolumesDestroying[needle.VolumeId(125)] = done
+	diskLocation.ecVolumesLock.Unlock()
+
+	mounted := make(chan error, 1)
+	go func() {
+		_, err := diskLocation.LoadEcShard("", needle.VolumeId(125), erasure_coding.ShardId(0))
+		mounted <- err
+	}()
+
+	select {
+	case <-mounted:
+		t.Fatal("LoadEcShard returned while a destroy generation was still in flight")
+	case <-time.After(100 * time.Millisecond):
+	}
+
+	close(done)
+	select {
+	case err := <-mounted:
+		if err != nil {
+			t.Fatalf("LoadEcShard after destroy completion: %v", err)
+		}
+	case <-time.After(5 * time.Second):
+		t.Fatal("LoadEcShard did not proceed after the destroy finished")
+	}
+
+	if _, found := diskLocation.FindEcShard(needle.VolumeId(125), erasure_coding.ShardId(0)); !found {
+		t.Fatal("shard must be registered once the pending destroy is over")
+	}
+	diskLocation.ecVolumesLock.RLock()
+	_, tombstoneLeft := diskLocation.ecVolumesDestroying[needle.VolumeId(125)]
+	diskLocation.ecVolumesLock.RUnlock()
+	if tombstoneLeft {
+		t.Error("a successful remount must clear the vid's destroy tombstone")
+	}
+}
+
+// TestDestroyEcVolumeClosesTombstone: once DestroyEcVolume returns, the
+// tombstone is closed so waits unblock, and a fresh remount re-opens the
+// regenerated files rather than anything the destroy unlinked.
+func TestDestroyEcVolumeClosesTombstone(t *testing.T) {
+	dir := t.TempDir()
+	diskLocation := NewDiskLocation(dir, 10, util.MinFreeSpace{}, dir, types.HardDriveType, nil, stats.DefaultDiskIOProbeConfig())
+	defer diskLocation.Close()
+
+	seedShard := func() {
+		if err := os.WriteFile(filepath.Join(dir, "126.ec00"), []byte("shard bytes"), 0o644); err != nil {
+			t.Fatalf("seed .ec00: %v", err)
+		}
+		if err := os.WriteFile(filepath.Join(dir, "126.ecx"), make([]byte, 16), 0o644); err != nil {
+			t.Fatalf("seed .ecx: %v", err)
+		}
+	}
+	seedShard()
+
+	if _, err := diskLocation.LoadEcShard("", needle.VolumeId(126), erasure_coding.ShardId(0)); err != nil {
+		t.Fatalf("initial LoadEcShard: %v", err)
+	}
+	diskLocation.DestroyEcVolume(needle.VolumeId(126))
+
+	if _, found := diskLocation.FindEcVolume(needle.VolumeId(126)); found {
+		t.Fatal("destroyed volume must not stay in ecVolumes")
+	}
+	diskLocation.ecVolumesLock.RLock()
+	done, ok := diskLocation.ecVolumesDestroying[needle.VolumeId(126)]
+	diskLocation.ecVolumesLock.RUnlock()
+	if !ok {
+		t.Fatal("destroy must leave a tombstone behind for remounts to compare against")
+	}
+	select {
+	case <-done:
+	case <-time.After(5 * time.Second):
+		t.Fatal("tombstone must be closed once DestroyEcVolume returns")
+	}
+	if util.FileExists(filepath.Join(dir, "126.ec00")) {
+		t.Fatal("destroy must have removed the shard file")
+	}
+
+	seedShard()
+	if _, err := diskLocation.LoadEcShard("", needle.VolumeId(126), erasure_coding.ShardId(0)); err != nil {
+		t.Fatalf("remount after destroy: %v", err)
+	}
+	diskLocation.ecVolumesLock.RLock()
+	_, tombstoneLeft := diskLocation.ecVolumesDestroying[needle.VolumeId(126)]
+	diskLocation.ecVolumesLock.RUnlock()
+	if tombstoneLeft {
+		t.Error("remount must clear the stale destroy tombstone")
+	}
+}
+
 // TestLoadAllEcShardsSplitDirZeroSizedCleanup: the scan merges Directory and
 // IdxDirectory listings, so a stale zero-sized file in one directory and a
 // fresh same-named file in the other are different files behind one entry

weed/storage/erasure_coding/ec_volume.go

@@ -587,6 +587,15 @@ func (ev *EcVolume) IndexBaseFileName() string {
 	return EcShardFileName(ev.Collection, ev.dirIdx, int(ev.VolumeId))
 }
 
+// EcIndexBaseFileName returns the base path of the volume's live .ecx/.ecj
+// pair — where NewEcVolume actually found them (ecxActualDir), which can
+// differ from IndexBaseFileName when an .ecx copy exists in both the data
+// and index directories. Anything that must agree with the live journal
+// handle (ecjFile) uses this, not the configured index dir.
+func (ev *EcVolume) EcIndexBaseFileName() string {
+	return EcShardFileName(ev.Collection, ev.ecxActualDir, int(ev.VolumeId))
+}
+
 func (ev *EcVolume) ShardSize() uint64 {
 	if len(ev.Shards) > 0 {
 		return uint64(ev.Shards[0].Size())

weed/storage/erasure_coding/ec_volume_delete.go

@@ -6,6 +6,7 @@ import (
 	"os"
 
 	"github.com/seaweedfs/seaweedfs/weed/glog"
+	"github.com/seaweedfs/seaweedfs/weed/storage/backend"
 	"github.com/seaweedfs/seaweedfs/weed/storage/types"
 	"github.com/seaweedfs/seaweedfs/weed/util"
 )
@@ -110,6 +111,41 @@ func (ev *EcVolume) appendJournalLocked(b []byte) error {
 	return nil
 }
 
+// LockDeletionJournal serializes the caller against runtime .ecj appends
+// (DeleteNeedleFromEcx) until the returned func is called. Decode holds it
+// while consuming the journal into the rebuilt index so a committed delete
+// cannot slip past the publish.
+func (ev *EcVolume) LockDeletionJournal() func() {
+	ev.ecjFileAccessLock.Lock()
+	return ev.ecjFileAccessLock.Unlock
+}
+
+// ReopenDeletionJournal repoints ecjFile at the live .ecj path after
+// RebuildEcxFile unlinked it. Without this, later DeleteNeedleFromEcx
+// appends keep landing on the detached inode — synced, successful, and
+// invisible to every path-based reader. Caller must hold the journal lock
+// (see LockDeletionJournal).
+func (ev *EcVolume) ReopenDeletionJournal() error {
+	if ev.ecjFile != nil {
+		_ = ev.ecjFile.Close()
+		ev.ecjFile = nil
+	}
+	ecjFile, err := backend.OpenVolumeFile(ev.FileName(".ecj"), os.O_RDWR|os.O_CREATE)
+	if err != nil {
+		return fmt.Errorf("reopen ec volume journal %s: %w", ev.FileName(".ecj"), err)
+	}
+	ev.ecjFile = ecjFile
+	// A successful RebuildEcxFile leaves the path unlinked, so this is a
+	// fresh empty file — but track whatever is actually there so the
+	// rollback truncate in DeleteNeedleFromEcx can never wipe real records.
+	if fi, statErr := ecjFile.Stat(); statErr == nil {
+		ev.ecjFileSize = fi.Size()
+	} else {
+		ev.ecjFileSize = 0
+	}
+	return nil
+}
+
 func RebuildEcxFile(baseFileName string) error {
 
 	if !util.FileExists(baseFileName + ".ecj") {

weed/storage/store_vacuum.go

@@ -184,6 +184,12 @@ func (s *Store) CompactVolumeFiles(vid needle.VolumeId, collection string, locat
 	}
 
 	if err := tempVolume.CommitCompact(); err != nil {
+		// A failed commit may have swapped only one of .dat/.idx; reconcile
+		// rolls a decided swap forward or removes orphan temp files before
+		// this volume can mount with a mismatched pair.
+		if reconcileErr := tempVolume.reconcileCompactState(); reconcileErr != nil {
+			return fmt.Errorf("commit compact volume %d: %v (reconcile failed: %v)", vid, err, reconcileErr)
+		}
 		if cleanupErr := tempVolume.cleanupCompact(); cleanupErr != nil {
 			return fmt.Errorf("commit compact volume %d: %v (cleanup failed: %v)", vid, err, cleanupErr)
 		}