shell: volume.check.disk — actionable verdict for diverged vacuumed replicas (#11197)

* shell: volume.check.disk — actionable verdict for diverged vacuumed replicas

When -resurrectMissingNeedles is gated off because both replicas have been
vacuumed (compaction revision > 0) — the normal state of any production
cluster — check.disk previously stopped at 'cannot prove they are missing
writes vs vacuumed deletes' and did nothing, leaving a diverged replica with
no repair path. volume.fix.replication does not catch it either: it only
acts when the replica COUNT is below the expected replication, never when
two replicas are both present but hold different live data.

Classify the divergence instead of dead-ending:

  liveDivergence() counts live (non-deleted) needles present on one replica's
  index but entirely absent from the other, in both directions. Tombstones
  are excluded, so vacuum asymmetry (a compacted replica that dropped deleted
  entries) is not mistaken for divergence.

  reportDivergenceVerdict() turns the count into an operator action:
    - one-sided (one replica has all the live data, the other has no unique
      live needles) -> print the exact safe repair:
        volume.copy -source <complete> -target <lagging> -volumeId <id>
      Re-copying the complete replica is safe precisely because the lagging
      side holds no unique live data; VolumeCopy's verify-before-destroy gate
      independently confirms the source holds the volume before deleting the
      target.
    - two-sided (split-brain, both sides have unique live data) -> warn and
      do NOT emit an auto repair; point to volume.fsck -findMissingChunksInFiler
      to confirm the 'missing' needles are orphans before converging.

Report-only: no data is modified and the resurrection safety gate is
untouched. This is what lets a 13-volume diverged cluster be diagnosed and
repaired in minutes instead of by hand-diffing every index.

Observed motivating case: home SeaweedFS 4.45 cluster, 13 010 cross-rack
volumes diverged after failed replicate writes (ReplicatedWrite MaxAttempts=1
fire-and-forget), 11 one-sided + 2 two-sided, all repaired via volume.copy.

* shell: volume.check.disk — address review nits on divergence verdict

- Use pb.NewServerAddressFromDataNode (dialable ip:port, Address with Id
  fallback) for the advertised volume.copy -source/-target instead of the
  logical node Id, which may not be dialable.
- Make the one-sided verdict tombstone-aware: when the lagging replica has
  been vacuumed, absent live needles may be valid deletions whose tombstones
  were dropped, so a whole-volume re-copy would resurrect them. The command
  is only advertised as safe when the lagging side is proven never-vacuumed
  (compaction revision 0 read under -resurrectMissingNeedles); otherwise a
  caveat is printed pointing at fsck/needle-level repair.
- Fix reversed copy direction when the source replica is the lagging one
  (must copy complete -> lagging in both cases).
- Test: real tombstone (negative size) with the correct 0/0 expectation and
  || assertion; verdict test now covers dialable address, corrected
  direction, and caveat on/off.

* shell: volume.check.disk — per-replica revision knowledge, no copy command for vacuumed lagging side

- Track srcRevKnown/tgtRevKnown separately: in unidirectional mode the
  target revision IS read, so a proven never-vacuumed target no longer
  gets a false resurrection warning (regression: bidi=false, target rev 0).
- A one-sided verdict now only emits the volume.copy command when the
  lagging side is proven never-vacuumed; when it is vacuumed (or unproven)
  the verdict refuses to print the destructive command and points at
  fsck/needle-level repair instead — an appended caveat next to a ready-to-
  paste copy command was still inviting the resurrection.
- deletionCaveat now returns the boolean safety decision.

* shell: volume.check.disk — preserve gRPC port, proven two-sided is not split-brain

- Emit the raw ServerAddress string (host:port.grpcPort) instead of
  String()/ToHttpAddress(), which drops the custom gRPC port and would
  make the suggested volume.copy dial the default port and fail.
- Two-sided divergence with both replicas proven never-vacuumed under
  -resurrectMissingNeedles is mutually missed writes, not split-brain:
  recommend re-running with -apply (in-place resurrection both
  directions) instead of the split-brain no-auto-repair warning.
- Tests: case F (proven two-sided -> -apply, no split-brain warning),
  case G (custom gRPC ports preserved in emitted addresses).

---------

Co-authored-by: timolow <tim@timolow.dev>

weed/shell/command_volume_check_disk.go

@@ -549,9 +549,127 @@ func (vcd *volumeCheckDisk) checkBoth(source, target *VolumeReplica, bidi bool)
 		return sourceHasChanges, targetHasChanges, errors.Join(errs...)
 	}
 
+	// When nothing was repaired — typically because resurrection is gated off
+	// since both replicas have been vacuumed (compaction revision > 0), which is
+	// the normal state of any production cluster — doVolumeCheckDisk only logged
+	// "cannot prove they are missing writes vs vacuumed deletes" and stopped.
+	// That dead-end leaves a diverged replica with no actionable path, so classify
+	// the divergence and print the exact repair. This is report-only: it changes
+	// no data and does not bypass the resurrection safety gate.
+	if !targetHasChanges && !sourceHasChanges {
+		sourceOnly, targetOnly := vcd.liveDivergence(sourceDB, targetDB)
+		if sourceOnly > 0 || targetOnly > 0 {
+			vcd.reportDivergenceVerdict(source, target, sourceOnly, targetOnly,
+				sourceRevision, targetRevision,
+				vcd.resurrectMissingNeedles && bidi, vcd.resurrectMissingNeedles, vcd.resurrectMissingNeedles)
+		}
+	}
+
 	return sourceHasChanges, targetHasChanges, nil
 }
 
+// liveDivergence counts live (non-deleted) needles present on a's index but
+// entirely absent from b's, and the reverse. It is used to classify a diverged
+// replica pair: one-sided (the lagging replica holds no unique live data, so a
+// re-copy of the complete side converges it — subject to the deletion caveat
+// reportDivergenceVerdict states, since the absent needles may be valid
+// deletions on a vacuumed replica) versus two-sided (true split-brain — do
+// not auto-repair). Tombstones are excluded, so vacuum asymmetry (a compacted
+// replica that has dropped deleted entries) does not create a false
+// difference.
+func (vcd *volumeCheckDisk) liveDivergence(a, b *needle_map.MemDb) (aOnly, bOnly int) {
+	a.DescendingVisit(func(v needle_map.NeedleValue) error {
+		if v.Size.IsDeleted() {
+			return nil
+		}
+		if _, found := b.Get(v.Key); !found {
+			aOnly++
+		}
+		return nil
+	})
+	b.DescendingVisit(func(v needle_map.NeedleValue) error {
+		if v.Size.IsDeleted() {
+			return nil
+		}
+		if _, found := a.Get(v.Key); !found {
+			bOnly++
+		}
+		return nil
+	})
+	return
+}
+
+// reportDivergenceVerdict prints the actionable outcome for a diverged replica
+// pair that check.disk could not repair in place. revSrc/revTgt are the
+// replicas' compaction revisions (0 = never vacuumed; >0 = at least one
+// vacuum dropped deleted entries from the index); srcRevKnown/tgtRevKnown
+// report whether each revision was actually read (target is read even in
+// unidirectional mode; source only under -bidirectional). resurrectFlag is
+// the command line's -resurrectMissingNeedles.
+//
+// The verdict is tombstone-aware: when the lagging replica has been vacuumed,
+// a needle that is live on the complete side but absent on the lagging side is
+// ambiguous — either a missing write, or a valid deletion whose tombstone the
+// lagging side's vacuum already dropped. A whole-volume re-copy converges the
+// divergence either way, but in the latter case it resurrects deleted data, so
+// the volume.copy command is only emitted when the lagging side is proven
+// never-vacuumed; otherwise the verdict points at a non-destructive
+// needle-level repair instead.
+func (vcd *volumeCheckDisk) reportDivergenceVerdict(source, target *VolumeReplica, sourceOnly, targetOnly int, revSrc, revTgt uint32, srcRevKnown, tgtRevKnown, resurrectFlag bool) {
+	// Raw string form, NOT String(): ServerAddress.String() drops the custom
+	// gRPC port suffix, and volume.copy's dialer accepts the host:port.grpcPort
+	// form (see checkDialable in weed/operation/volume_move).
+	srcAddr := string(pb.NewServerAddressFromDataNode(source.location.dataNode))
+	tgtAddr := string(pb.NewServerAddressFromDataNode(target.location.dataNode))
+	vid := source.info.Id
+	switch {
+	case sourceOnly > 0 && targetOnly == 0:
+		// target is lagging: it holds no unique live data.
+		safe := deletionCaveat(resurrectFlag, tgtRevKnown, revTgt == 0)
+		if safe {
+			vcd.write("volume %d: ONE-SIDED divergence — %s is missing %d live needle(s) that exist on %s; %s holds no unique live data. Safe repair (whole-volume re-copy, complete -> lagging; verify-before-destroy is enforced): volume.copy -source %s -target %s -volumeId %d",
+				vid, tgtAddr, sourceOnly, srcAddr, tgtAddr, srcAddr, tgtAddr, vid)
+		} else {
+			vcd.write("volume %d: ONE-SIDED divergence — %s is missing %d live needle(s) that exist on %s; %s holds no unique live data. Do NOT re-copy the whole volume: the absent needles may be valid deletions already vacuumed away on the lagging side, which a re-copy would resurrect. Restore only the confirmed-missing needles (volume.fsck -collection <c> -volumeId %d -findMissingChunksInFiler, then needle-level repair), or re-copy only after accepting that risk.",
+				vid, tgtAddr, sourceOnly, srcAddr, tgtAddr, vid)
+		}
+	case targetOnly > 0 && sourceOnly == 0:
+		// source is lagging: it holds no unique live data.
+		safe := deletionCaveat(resurrectFlag, srcRevKnown, revSrc == 0)
+		if safe {
+			vcd.write("volume %d: ONE-SIDED divergence — %s is missing %d live needle(s) that exist on %s; %s holds no unique live data. Safe repair (whole-volume re-copy, complete -> lagging; verify-before-destroy is enforced): volume.copy -source %s -target %s -volumeId %d",
+				vid, srcAddr, targetOnly, tgtAddr, srcAddr, tgtAddr, srcAddr, vid)
+		} else {
+			vcd.write("volume %d: ONE-SIDED divergence — %s is missing %d live needle(s) that exist on %s; %s holds no unique live data. Do NOT re-copy the whole volume: the absent needles may be valid deletions already vacuumed away on the lagging side, which a re-copy would resurrect. Restore only the confirmed-missing needles (volume.fsck -collection <c> -volumeId %d -findMissingChunksInFiler, then needle-level repair), or re-copy only after accepting that risk.",
+				vid, srcAddr, targetOnly, tgtAddr, srcAddr, vid)
+		}
+	case sourceOnly > 0 && targetOnly > 0:
+		if resurrectFlag && srcRevKnown && tgtRevKnown && revSrc == 0 && revTgt == 0 {
+			// Both replicas proven never-vacuumed: the mutually missing
+			// needles are provably missing writes on both sides (the same
+			// proof the resurrection gate uses), not split-brain. The two
+			// doVolumeCheckDisk passes above already queued them for
+			// in-place resurrection but this was a simulation run, so
+			// nothing was applied.
+			vcd.write("volume %d: TWO-SIDED divergence, both replicas never vacuumed — %s is missing %d live needle(s) that exist on %s AND %s is missing %d that exist on %s. These are mutually missed writes, not split-brain: re-run this command with -apply to resurrect them in place (both directions).",
+				vid, tgtAddr, sourceOnly, srcAddr, srcAddr, targetOnly, tgtAddr)
+		} else {
+			vcd.write("volume %d: TWO-SIDED (split-brain) divergence — %s has %d unique live needle(s) AND %s has %d. Do NOT auto-repair: each side may hold data the other lacks. Confirm orphans with volume.fsck -collection <c> -volumeId %d -findMissingChunksInFiler before re-copying the complete replica, or restore the missing needles manually.",
+				vid, srcAddr, sourceOnly, tgtAddr, targetOnly, vid)
+		}
+	}
+}
+
+// deletionCaveat reports whether a one-sided divergence's complete -> lagging
+// re-copy is safe: the lagging side is proven never-vacuumed (compaction
+// revision 0) and that proof was actually taken under the
+// -resurrectMissingNeedles flag, so its absent live needles are missing
+// writes by the same proof the resurrection gate uses — not vacuumed
+// deletions.
+func deletionCaveat(resurrectFlag, laggingRevKnown, laggingNeverVacuumed bool) bool {
+	return resurrectFlag && laggingRevKnown && laggingNeverVacuumed
+}
+
 func (vcd *volumeCheckDisk) doVolumeCheckDisk(minuend, subtrahend *needle_map.MemDb, source, target *VolumeReplica, resurrectAbsent bool, targetRevision uint32) (hasChanges bool, err error) {
 
 	// find missing keys