backup: resume incremental copies by index position instead of appendAtNs (#12228)

* volume server: add last-entry cursor fields to VolumeIncrementalCopy

* backup: resume incremental copies by index position instead of appendAtNs

appendAtNs is stamped by whichever replica received the write, so the
same needle carries a different timestamp on every replica. A backup
that resumes from a different replica than last time can append the
last needle again or skip needles written inside the clock gap.

Send the local copy's index entry count and last entry identity (key,
size, cookie) with VolumeIncrementalCopy. The replica verifies the same
record sits at that position and streams from the next index entry, so
the resume no longer depends on any replica's clock. When the boundary
record does not match, or the local copy is ahead, the replica returns
FailedPrecondition and the backup rebuilds the local copy from scratch.

Old servers ignore the new fields and keep the timestamp behavior.

* rust volume server: mirror positional resume in VolumeIncrementalCopy

* volume server: add entries_hash prefix checksum to VolumeIncrementalCopy

* backup: verify the whole index prefix, not just the boundary entry

* rust volume server: verify the index prefix hash in resume_by_last_entry

* backup: keep the source's compaction revision on rebuilt copies

* backup: diverge on a zero-offset boundary entry

A remote-tier tombstone lives in the .idx with no .dat record, so
readNeedleAt(0) parses the superblock and its error bypassed the
ErrIncrementalCopyDiverged mapping; the backup got a generic failure
instead of FailedPrecondition and never rebuilt.

* rust volume server: mirror zero-offset boundary divergence

* rust volume server: gate the empty-dat early return on entry_count

With a positional resume the Go handler still reaches ResumeByLastEntry,
so an empty replica reports divergence; the Rust early return answered a
clean empty stream and weed backup kept a stale copy.

* backup: hash the index entry's size field, not trailing offset bytes

In a 5BytesOffset build the offset occupies bytes 8-12 and the size
starts at byte 13, so the hash folded in one byte of the offset.
Matching records at different offsets then hashed differently and a
valid backup was reported diverged and rebuilt.

* backup: read the index in whole-entry chunks for the prefix hash

When entries are 17 bytes a 1 MiB read buffer does not end on an entry
boundary: the partial tail entry was skipped by the loop but still
counted as consumed, so every later chunk was hashed at a misaligned
position.

* rust volume server: hash the index entry's size field after the offset

Mirrors the Go hashIndexEntries change: with the 5bytes feature the
offset occupies bytes 8-12, so starting the size slice at byte 12
folded one offset byte into the hash and matching records at different
offsets compared diverged.

* rust volume server: read the index in whole-entry chunks for the prefix hash

Mirrors the Go change. With 17-byte entries a 1 MiB buffer does not
end on an entry boundary, so slicing the partial tail entry panicked
once a backup held 61681 entries.

* rust volume server: default the new incremental copy request fields in tests

weed/command/backup.go

@@ -5,6 +5,8 @@ import (
 	"fmt"
 
 	"google.golang.org/grpc"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
 
 	"github.com/seaweedfs/seaweedfs/weed/pb"
 
@@ -155,6 +157,7 @@ func backupFromLocation(volumeServer pb.ServerAddress, grpcDialOption grpc.DialO
 	datSize, _, _ := v.FileStat()
 
 	// If local volume is larger than remote, recreate it
+	rebuilt := false
 	if datSize > stats.TailOffset {
 		if err := v.Destroy(false, false, false); err != nil {
 			v.Close()
@@ -166,14 +169,39 @@ func backupFromLocation(volumeServer pb.ServerAddress, grpcDialOption grpc.DialO
 		if err != nil {
 			return fmt.Errorf("recreating volume: %w", err), false
 		}
+		rebuilt = true
 	}
 
 	// Perform the incremental backup
-	if err := v.IncrementalBackup(volumeServer, grpcDialOption); err != nil {
+	err = v.IncrementalBackup(volumeServer, grpcDialOption)
+	if err != nil && status.Code(err) == codes.FailedPrecondition {
+		// The local copy is not a prefix of this replica's records; rebuild it.
+		if err := v.Destroy(false, false, false); err != nil {
+			v.Close()
+			return fmt.Errorf("destroying volume: %w", err), false
+		}
+		v, err = storage.NewVolume(*s.dir, *s.dir, *s.collection, vid, storage.NeedleMapInMemory, replication, ttl, 0, ver, 0, 0)
+		if err != nil {
+			return fmt.Errorf("recreating volume: %w", err), false
+		}
+		err = v.IncrementalBackup(volumeServer, grpcDialOption)
+		rebuilt = true
+	}
+	if err != nil {
 		v.Close()
 		return fmt.Errorf("incremental backup: %w", err), false
 	}
 
+	// A rebuilt copy starts at compaction revision zero; adopt the source's
+	// revision or the next run compacts it in key order and diverges again.
+	if rebuilt {
+		v.SuperBlock.CompactionRevision = uint16(stats.CompactRevision)
+		if _, err = v.DataBackend.WriteAt(v.SuperBlock.Bytes(), 0); err != nil {
+			v.Close()
+			return fmt.Errorf("writing superblock: %w", err), false
+		}
+	}
+
 	v.Close()
 	return nil, false
 }