* volume server: add idx_content to VolumeIncrementalCopyResponse
A remote-tier delete is an index-only tombstone row with no .dat record,
so it cannot reach an incremental copy through the .dat stream. Carrying
the .idx tail verbatim lets the backup reproduce those rows at their
exact positions.
* backup: carry index-only tombstones through incremental copies
A remote-tier delete leaves an offset-0 tombstone in .idx with no .dat
record. The copy only streamed .dat bytes and regenerated .idx from
them, so the tombstone never reached the backup: the deleted needle
stayed live and every later incremental diverged and rebuilt again.
backupCursor and ResumeByLastEntry now accept an index-only tombstone
as a boundary, matched by key and size since there is no record to read
a cookie from. VolumeIncrementalCopy appends the .idx tail after the
.dat tail, and IncrementalBackup writes it at the index boundary,
truncates, and loads the new rows — so tombstones land at the exact
positions the replica has them.
* seaweed-volume: mirror index-only tombstone handling in incremental copy
resume_by_last_entry accepts an offset-0 tombstone boundary matched on
key and size, skips its missing cookie check, and resumes .dat streaming
at the next entry that has a record — index-only rows ride the new
idx_content stream instead of reporting divergence. The server captures
the .idx bound before the .dat one (a row is published only after its
bytes) and streams the tail after the data.
* backup: keep incremental copy bounds and streams consistent
Review fixes:
* Bound capture moved after the resume reads and reads .dat/.idx sizes
together (Go FileStat, Rust file_stat under one lock), so every row
resume saw is inside the index bound and each row's record is always
inside the data bound.
* A compaction commit changes CompactionRevision: the server now rejects
the copy when the revision moved between resume, file opens, or the
end of the stream, so a backup never mixes revisions. The .idx is
opened up front to pin the inode.
* An index tail that ends before the captured bound is an error, not a
silent success.
* IncrementalBackup rolls the .idx tail and .dat bytes back to the
pre-copy boundary when the stream fails, so a partial row cannot wedge
backupCursor and unapplied rows cannot pass for current ones.
* The index tail is replayed through a small indexTailReceiver interface,
covering the leveldb needle maps too, not just the in-memory map.
* backup: bound the resume scan to the captured index size
The resume and the stream bounds are now captured in one order:
revision, then the .dat/.idx pair, then the scan. ResumeByLastEntry and
resume_by_last_entry take the index bound and scan no further, so a row
appended while the copy is being set up can never enter the index
stream without its .dat bytes.
The index-tail replay stays on the in-memory map only: the leveldb
map's DoOffsetLoading writes to the .cpldb compaction database, not the
live one, so it cannot serve a live tail replay; other maps keep
regenerating the index from .dat as before.
* seaweed-volume: guard incremental copy total against a located offset past .dat
A corrupt index can place the resume offset beyond the captured .dat
size; has_data already refuses the reader, but total still evaluates
dat_size - s and underflows.
* backup: bound the incremental .dat tail by index-covered records
A record staged but not yet indexed can sit inside the captured .dat
bound while its row stays outside the index bound — the fsync gap
between staging and publishing is not covered by one stat. Streaming
those bytes leaves them unindexed on the backup, and the next run
appends the record again. Bound the .dat tail by the last record the
streamed rows cover on both producers, and cover the bound walk with
tests.
* backup: cap the .dat stream at the captured bound and roll back replay failures
sendFileContent could read past stopOffset on its last chunk, so a
needle appended after FileStat arrived without its index row and the
next copy appended at the wrong position. Each read is now limited to
the remaining captured bytes.
A failed truncateIndex or DoOffsetLoading after a successful stream
left the index tail on disk unapplied, which a later backupCursor could
mistake for applied rows; both now roll the tail back to the pre-copy
boundary like the stream-error path.
* backup: bound the incremental .dat tail by the maximum covered offset
A key-sorted .idx ends on the highest-key row, which need not be the
record physically last in .dat, so the last-row bound could cut off
live records. Take the end of the highest-offset row inside the bound
instead, and cover the case with a sorted-idx test in Go and Rust.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* backup: restore .dat and the needle map when index replay fails
A failed DoOffsetLoading left the received .dat bytes in place and could
leave partial mappings for rows the index rollback just removed, so the
next copy appended the same records again or read stale entries. Truncate
.dat back to the copy boundary and restore the touched mappings and
metrics from a pre-replay snapshot.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* backup: cover no index rows with the superblock bound
When no row inside the index bound names a record — an empty index or a
bound of index-only tombstones — the covered end is the superblock, not
the captured .dat size, so staged bytes are never streamed without a
covering row.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* seaweed-volume: apply the covered bound even when the index streams no rows
The bound was gated on idx_end > idx_start, so an empty captured index
skipped it entirely and a staged .dat tail still shipped. Gate on the
stream-index predicate instead, matching Go's streamIndex check.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* backup: propagate index coverage read errors
* backup: rebuild the needle map from the truncated index on replay failure
* seaweed-volume: propagate index coverage read errors
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: add last-entry cursor fields to VolumeIncrementalCopy
* backup: resume incremental copies by index position instead of appendAtNs
appendAtNs is stamped by whichever replica received the write, so the
same needle carries a different timestamp on every replica. A backup
that resumes from a different replica than last time can append the
last needle again or skip needles written inside the clock gap.
Send the local copy's index entry count and last entry identity (key,
size, cookie) with VolumeIncrementalCopy. The replica verifies the same
record sits at that position and streams from the next index entry, so
the resume no longer depends on any replica's clock. When the boundary
record does not match, or the local copy is ahead, the replica returns
FailedPrecondition and the backup rebuilds the local copy from scratch.
Old servers ignore the new fields and keep the timestamp behavior.
* rust volume server: mirror positional resume in VolumeIncrementalCopy
* volume server: add entries_hash prefix checksum to VolumeIncrementalCopy
* backup: verify the whole index prefix, not just the boundary entry
* rust volume server: verify the index prefix hash in resume_by_last_entry
* backup: keep the source's compaction revision on rebuilt copies
* backup: diverge on a zero-offset boundary entry
A remote-tier tombstone lives in the .idx with no .dat record, so
readNeedleAt(0) parses the superblock and its error bypassed the
ErrIncrementalCopyDiverged mapping; the backup got a generic failure
instead of FailedPrecondition and never rebuilt.
* rust volume server: mirror zero-offset boundary divergence
* rust volume server: gate the empty-dat early return on entry_count
With a positional resume the Go handler still reaches ResumeByLastEntry,
so an empty replica reports divergence; the Rust early return answered a
clean empty stream and weed backup kept a stale copy.
* backup: hash the index entry's size field, not trailing offset bytes
In a 5BytesOffset build the offset occupies bytes 8-12 and the size
starts at byte 13, so the hash folded in one byte of the offset.
Matching records at different offsets then hashed differently and a
valid backup was reported diverged and rebuilt.
* backup: read the index in whole-entry chunks for the prefix hash
When entries are 17 bytes a 1 MiB read buffer does not end on an entry
boundary: the partial tail entry was skipped by the loop but still
counted as consumed, so every later chunk was hashed at a misaligned
position.
* rust volume server: hash the index entry's size field after the offset
Mirrors the Go hashIndexEntries change: with the 5bytes feature the
offset occupies bytes 8-12, so starting the size slice at byte 12
folded one offset byte into the hash and matching records at different
offsets compared diverged.
* rust volume server: read the index in whole-entry chunks for the prefix hash
Mirrors the Go change. With 17-byte entries a 1 MiB buffer does not
end on an entry boundary, so slicing the partial tail entry panicked
once a backup held 61681 entries.
* rust volume server: default the new incremental copy request fields in tests
* fix 11400
* persist failed-recovery quarantine and harden rollback
- record the unavailable state in a .unavailable marker, fsync it, and
re-arm it on load so a restart cannot serve an unverified pair
- quarantine the volume so heartbeats stop advertising it
- block MarkVolumeWritable while unavailable, rechecked under noWriteLock
- fail every request of a failed batch, not only the succeeded ones
- restore the needle map and truncate .dat on inline fsync rollback failure
- add truncateIndex for the sorted-file needle map
- mirror the fail-closed semantics in the Rust volume server
* volume: erase rolled-back mappings instead of leaving tombstones
A rolled-back batch or failed inline write used Delete() to undo a
needle that did not exist beforehand, leaving a tombstoned map entry
whose stale offset makes the next write to that needle fail reading a
header that no longer exists. Add removeMapping/restoreMapping to the
mappers so recovery erases entries that were absent before the batch
and reinstates the exact prior offset/size for ones that were,
including tombstones. The index row still goes through Delete so a
replay forgets the needle.
* volume: gate bulk readers on unavailable and fsync the marker's dir
- fsync_dir(&self.dir) synced the volume dir's parent, not the dir
holding .unavailable; pass the marker path so the create survives
a host crash
- export UnavailableError and check it in ReadAllNeedles,
VolumeTailSender, VolumeIncrementalCopy, and IncrementalBackup so
replica-sync paths cannot stream or append data from an unverified
.dat/.idx pair; mirror on the Rust side via read_dat_slice,
read_all_needles, dat_scan_plan, and the incremental-copy handler
* volume: drop issue references from comments near touched code
* volume: stop active scans when the volume becomes unavailable
The stream entry-point checks ran once per RPC, so a volume quarantined
by a failed recovery mid-scan kept serving data. Recheck availability
per needle/chunk on the detached read paths: tail scan and heartbeat,
read-all, incremental copy, incremental backup writes, and the Rust
StreamingBody chunk reads. Rust incremental copy also rejects a
quarantined volume before sync_to_disk touches the backend.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
