fix(replication): verify-before-destroy in VolumeCopy, check.disk, and over-replication trim (#9943)

* volume: verify before destroy in VolumeCopy and replication repair

Four data-safety fixes around copy/repair paths that could destroy or
resurrect data before verifying the source or survivors.

(a) VolumeCopy no longer deletes a pre-existing local replica up front.
The delete is deferred until ReadVolumeFileStatus on the source succeeds,
so a transient source outage (or a retry after one) can no longer wipe a
healthy destination replica. Gated on source readability only; size/count
comparisons are intentionally not used because they invert legitimately
after divergent vacuum/compaction. Mirrored in the Rust volume server.

(b) volume.check.disk no longer resurrects vacuumed-deleted needles. A
key present-and-live on the source but entirely absent on the target is
ambiguous: it may be a genuine missing write, or a needle deleted on the
target and then vacuumed (its index entry and any tombstone are gone). An
individual needle AppendAtNs has no monotonic relation to a vacuum
watermark, so the old cutoff heuristic could not tell them apart. Without
positive proof the absence is a missing write, the safe default is to NOT
push it back. Tradeoff: a real missing write may go unrepaired until a
tombstone-aware path exists, but we never raise back deleted data.

(c) Over-replication trim no longer resurrects needles or removes the
wrong replica. The pre-delete sync now runs read-only (divergence check
only) instead of writing the doomed replica's needles into the survivor.
pickOneReplicaToDelete only ever removes the smallest of multiple healthy
writable replicas; it refuses the trim when doing so would leave only
read-only/integrity-flagged survivors, since file_count>0 alone cannot
prove the survivor's .dat is readable.

(d) Incomplete-volume (.note) cleanup keeps the shared .vif when an .ecx
for the same vid coexists on the disk, so removing an interrupted regular
copy cannot strip a coexisting EC volume's info file. VolumeCopy now
surfaces .note write/remove errors instead of ignoring them. In the Rust
volume server (where a persisting note is actually reachable) the .note
check moves below the empty-stub sweep and EC validation, keeps the .vif
on EC coexistence, and the mount path fails when a .note still persists.

* shell: scope the over-replication writable-survivor guard to the trim path only

The writable-survivor guard (never trim down to a read-only survivor) lived
inside the shared pickOneReplicaToDelete, so it also gated the misplaced-volume
relocation via pickOneMisplacedVolume -- a misplaced read-only volume (e.g. a
full one) would silently stop being rebalanced. Extract pickSmallestReplica
for the relocation path (which deletes-and-recreates and must act on read-only
replicas), and keep the writable-survivor guard only in pickOneReplicaToDelete
used by the over-replication trim.

* seaweed-volume: recompute keep_vif after invalid-EC cleanup in the .note path

keep_vif used the pre-validation ecx_exists snapshot, so when the EC-validation
step above removed the invalid .ecx/shards, the .note cleanup still preserved a
now-orphaned .vif. Re-check .ecx existence at cleanup time, matching the Go
hasEcxFile re-check.

* shell: keep placement when picking an over-replication victim to delete

The trim picked the smallest writable replica without regard to placement, so
it could delete the only replica in a required failure domain (e.g. with "100"
and replicas dc1 + two in dc2, deleting dc1 leaves both survivors in dc2).
Prefer a writable replica whose removal still satisfies placement, falling back
to the smallest writable only when none does.

weed/shell/command_volume_check_disk.go

@@ -22,7 +22,6 @@ import (
 	"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
 	"github.com/seaweedfs/seaweedfs/weed/server/constants"
 	"github.com/seaweedfs/seaweedfs/weed/storage/needle_map"
-	"github.com/seaweedfs/seaweedfs/weed/storage/volume_replica"
 	"google.golang.org/grpc"
 )
 
@@ -44,6 +43,12 @@ type volumeCheckDisk struct {
 	syncDeletions      bool
 	fixReadOnly        bool
 	nonRepairThreshold float64
+	// resurrectMissingNeedles controls whether a needle present on the source
+	// but entirely absent on the target is pushed back. Default false: an
+	// absent needle is indistinguishable from a vacuumed delete, so the safe
+	// default never raises deleted data. No caller sets this true today; it is
+	// the seam for a future tombstone-aware repair path.
+	resurrectMissingNeedles bool
 
 	ewg *ErrorWaitGroup
 }
@@ -511,9 +516,8 @@ func (vcd *volumeCheckDisk) doVolumeCheckDisk(minuend, subtrahend *needle_map.Me
 	// hash join, can be more efficient
 	var missingNeedles []needle_map.NeedleValue
 	var partiallyDeletedNeedles []needle_map.NeedleValue
+	var skippedAbsentNeedles int
 	var counter int
-	doCutoffOfLastNeedle := true
-	cutoffFromAtNs := uint64(vcd.now.UnixNano())
 
 	minuend.DescendingVisit(func(minuendValue needle_map.NeedleValue) error {
 		counter++
@@ -521,27 +525,34 @@ func (vcd *volumeCheckDisk) doVolumeCheckDisk(minuend, subtrahend *needle_map.Me
 			if minuendValue.Size.IsDeleted() {
 				return nil
 			}
-			if doCutoffOfLastNeedle {
-				if needleMeta, err := volume_replica.ReadNeedleMeta(vcd.grpcDialOption(), pb.NewServerAddressFromDataNode(source.location.dataNode), source.info.Id, minuendValue); err == nil {
-					// needles older than the cutoff time are not missing yet
-					if needleMeta.AppendAtNs > cutoffFromAtNs {
-						return nil
-					}
-					doCutoffOfLastNeedle = false
-				}
+			// A key present-and-live on the source but entirely absent on the
+			// target is ambiguous: either a genuine missing write, or a needle
+			// that was deleted on the target and then vacuumed away (its index
+			// entry, including any tombstone, is gone after vacuum). An
+			// individual needle's AppendAtNs has no monotonic relation to a
+			// vacuum watermark, so it cannot distinguish the two. Without
+			// positive proof the absence is a missing write (rather than a
+			// vacuumed delete), the safe default is to NOT resurrect: a real
+			// missing write may go unrepaired until a tombstone-aware path
+			// exists, but we never raise back data the operator deleted.
+			if !vcd.resurrectMissingNeedles {
+				skippedAbsentNeedles++
+				return nil
 			}
 			missingNeedles = append(missingNeedles, minuendValue)
 		} else {
 			if minuendValue.Size.IsDeleted() && !subtrahendValue.Size.IsDeleted() {
 				partiallyDeletedNeedles = append(partiallyDeletedNeedles, minuendValue)
 			}
-			if doCutoffOfLastNeedle {
-				doCutoffOfLastNeedle = false
-			}
 		}
 		return nil
 	})
 
+	if skippedAbsentNeedles > 0 {
+		vcd.write("volume %d %s: not resurrecting %d needle(s) absent on %s (cannot prove they are missing writes vs vacuumed deletes)",
+			source.info.Id, source.location.dataNode.Id, skippedAbsentNeedles, target.location.dataNode.Id)
+	}
+
 	vcd.write("volume %d %s has %d entries, %s missed %d and partially deleted %d entries",
 		source.info.Id, source.location.dataNode.Id, counter, target.location.dataNode.Id, len(missingNeedles), len(partiallyDeletedNeedles))