* volume server: add idx_content to VolumeIncrementalCopyResponse
A remote-tier delete is an index-only tombstone row with no .dat record,
so it cannot reach an incremental copy through the .dat stream. Carrying
the .idx tail verbatim lets the backup reproduce those rows at their
exact positions.
* backup: carry index-only tombstones through incremental copies
A remote-tier delete leaves an offset-0 tombstone in .idx with no .dat
record. The copy only streamed .dat bytes and regenerated .idx from
them, so the tombstone never reached the backup: the deleted needle
stayed live and every later incremental diverged and rebuilt again.
backupCursor and ResumeByLastEntry now accept an index-only tombstone
as a boundary, matched by key and size since there is no record to read
a cookie from. VolumeIncrementalCopy appends the .idx tail after the
.dat tail, and IncrementalBackup writes it at the index boundary,
truncates, and loads the new rows — so tombstones land at the exact
positions the replica has them.
* seaweed-volume: mirror index-only tombstone handling in incremental copy
resume_by_last_entry accepts an offset-0 tombstone boundary matched on
key and size, skips its missing cookie check, and resumes .dat streaming
at the next entry that has a record — index-only rows ride the new
idx_content stream instead of reporting divergence. The server captures
the .idx bound before the .dat one (a row is published only after its
bytes) and streams the tail after the data.
* backup: keep incremental copy bounds and streams consistent
Review fixes:
* Bound capture moved after the resume reads and reads .dat/.idx sizes
together (Go FileStat, Rust file_stat under one lock), so every row
resume saw is inside the index bound and each row's record is always
inside the data bound.
* A compaction commit changes CompactionRevision: the server now rejects
the copy when the revision moved between resume, file opens, or the
end of the stream, so a backup never mixes revisions. The .idx is
opened up front to pin the inode.
* An index tail that ends before the captured bound is an error, not a
silent success.
* IncrementalBackup rolls the .idx tail and .dat bytes back to the
pre-copy boundary when the stream fails, so a partial row cannot wedge
backupCursor and unapplied rows cannot pass for current ones.
* The index tail is replayed through a small indexTailReceiver interface,
covering the leveldb needle maps too, not just the in-memory map.
* backup: bound the resume scan to the captured index size
The resume and the stream bounds are now captured in one order:
revision, then the .dat/.idx pair, then the scan. ResumeByLastEntry and
resume_by_last_entry take the index bound and scan no further, so a row
appended while the copy is being set up can never enter the index
stream without its .dat bytes.
The index-tail replay stays on the in-memory map only: the leveldb
map's DoOffsetLoading writes to the .cpldb compaction database, not the
live one, so it cannot serve a live tail replay; other maps keep
regenerating the index from .dat as before.
* seaweed-volume: guard incremental copy total against a located offset past .dat
A corrupt index can place the resume offset beyond the captured .dat
size; has_data already refuses the reader, but total still evaluates
dat_size - s and underflows.
* backup: bound the incremental .dat tail by index-covered records
A record staged but not yet indexed can sit inside the captured .dat
bound while its row stays outside the index bound — the fsync gap
between staging and publishing is not covered by one stat. Streaming
those bytes leaves them unindexed on the backup, and the next run
appends the record again. Bound the .dat tail by the last record the
streamed rows cover on both producers, and cover the bound walk with
tests.
* backup: cap the .dat stream at the captured bound and roll back replay failures
sendFileContent could read past stopOffset on its last chunk, so a
needle appended after FileStat arrived without its index row and the
next copy appended at the wrong position. Each read is now limited to
the remaining captured bytes.
A failed truncateIndex or DoOffsetLoading after a successful stream
left the index tail on disk unapplied, which a later backupCursor could
mistake for applied rows; both now roll the tail back to the pre-copy
boundary like the stream-error path.
* backup: bound the incremental .dat tail by the maximum covered offset
A key-sorted .idx ends on the highest-key row, which need not be the
record physically last in .dat, so the last-row bound could cut off
live records. Take the end of the highest-offset row inside the bound
instead, and cover the case with a sorted-idx test in Go and Rust.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* backup: restore .dat and the needle map when index replay fails
A failed DoOffsetLoading left the received .dat bytes in place and could
leave partial mappings for rows the index rollback just removed, so the
next copy appended the same records again or read stale entries. Truncate
.dat back to the copy boundary and restore the touched mappings and
metrics from a pre-replay snapshot.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* backup: cover no index rows with the superblock bound
When no row inside the index bound names a record — an empty index or a
bound of index-only tombstones — the covered end is the superblock, not
the captured .dat size, so staged bytes are never streamed without a
covering row.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* seaweed-volume: apply the covered bound even when the index streams no rows
The bound was gated on idx_end > idx_start, so an empty captured index
skipped it entirely and a staged .dat tail still shipped. Gate on the
stream-index predicate instead, matching Go's streamIndex check.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* backup: propagate index coverage read errors
* backup: rebuild the needle map from the truncated index on replay failure
* seaweed-volume: propagate index coverage read errors
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: count index-load metrics only for live-key removals
The in-memory and leveldb offset loaders counted a deletion for every
tombstone row replayed, including re-deletes of already-deleted keys
(runtime logDelete only counts a live removal), and added the returned
old size to the byte counters even when it was a tombstone sentinel —
uint64(-1) wraps DeletionByteCounter. The leveldb offset loader also
counted every index row as a file and its bytes unconditionally. Gate
the counters the same way the runtime paths do so a reload reports the
same numbers the runtime counters hold.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: derive index metrics from latest-row state in the metric loader
The leveldb metric loader walked the index in reverse and incremented
FileCounter for every row (inflating FileCount with tombstones) and
DeletionCounter for every superseded row plus every deleted-key row
(double counting both). Per key, the runtime deleted count is exactly
(valid rows) - (1 when the latest row is live), so count live-latest
rows via the same bloom filter and derive both deletion counters; this
matches the runtime logPut/logDelete counters exactly instead of
drifting with tombstone history.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* server: warn instead of failing volume copies on counter drift
checkCopyCounts compared the source's in-memory counters against the
target's fresh-load counters, but counters computed by different
loaders drifted apart for identical .idx data, so a good copy was
rejected (issue #11678). The file-size checks already gate integrity;
log the mismatch instead of failing the copy.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: assert MaxFileKey parity in the metric loader test
MaxFileKey derives from row order alone, so unlike the bloom-filtered
deletion counters it must always match the runtime counters exactly.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix 11400
* persist failed-recovery quarantine and harden rollback
- record the unavailable state in a .unavailable marker, fsync it, and
re-arm it on load so a restart cannot serve an unverified pair
- quarantine the volume so heartbeats stop advertising it
- block MarkVolumeWritable while unavailable, rechecked under noWriteLock
- fail every request of a failed batch, not only the succeeded ones
- restore the needle map and truncate .dat on inline fsync rollback failure
- add truncateIndex for the sorted-file needle map
- mirror the fail-closed semantics in the Rust volume server
* volume: erase rolled-back mappings instead of leaving tombstones
A rolled-back batch or failed inline write used Delete() to undo a
needle that did not exist beforehand, leaving a tombstoned map entry
whose stale offset makes the next write to that needle fail reading a
header that no longer exists. Add removeMapping/restoreMapping to the
mappers so recovery erases entries that were absent before the batch
and reinstates the exact prior offset/size for ones that were,
including tombstones. The index row still goes through Delete so a
replay forgets the needle.
* volume: gate bulk readers on unavailable and fsync the marker's dir
- fsync_dir(&self.dir) synced the volume dir's parent, not the dir
holding .unavailable; pass the marker path so the create survives
a host crash
- export UnavailableError and check it in ReadAllNeedles,
VolumeTailSender, VolumeIncrementalCopy, and IncrementalBackup so
replica-sync paths cannot stream or append data from an unverified
.dat/.idx pair; mirror on the Rust side via read_dat_slice,
read_all_needles, dat_scan_plan, and the incremental-copy handler
* volume: drop issue references from comments near touched code
* volume: stop active scans when the volume becomes unavailable
The stream entry-point checks ran once per RPC, so a volume quarantined
by a failed recovery mid-scan kept serving data. Recheck availability
per needle/chunk on the detached read paths: tail scan and heartbeat,
read-all, incremental copy, incremental backup writes, and the Rust
StreamingBody chunk reads. Rust incremental copy also rejects a
quarantined volume before sync_to_disk touches the backend.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(volume): keep vacuum running past dangling .idx entries
Vacuum compaction aborted entirely on the first .idx entry whose offset
pointed past the end of the .dat file, surfacing as `cannot hydrate
needle from file: EOF` and stalling progress on every other volume.
In both Go and Rust:
- During compaction, skip an unreadable needle and continue. The bytes
it pointed at were already unreachable via reads, so dropping the
index reference makes the post-vacuum volume consistent. Real EIO
still bails out so a disk fault is not silently papered over.
- At volume load, do a single linear scan of the .idx and confirm
every (offset + actual size) fits inside .dat. The pre-existing
integrity check only looked at the last 10 entries, so deeper
corruption (e.g. left over from a crashed batched write) went
undetected and only surfaced later as a vacuum EOF. A failure now
marks the volume read-only at load time so an operator can react.
Refs #8928
* fix(volume): only skip permanent-corruption needle reads during vacuum
Address PR review feedback (gemini-code-assist + coderabbit):
The original patch skipped any non-EIO read failure, which would silently
drop needles on transient errors — Windows hardware bad-sector errors
(ERROR_CRC etc.) never surface as syscall.EIO; tiered-storage network
timeouts and EROFS would also slip through and shrink the volume.
Switch to an explicit whitelist of permanent-corruption shapes:
- Add needle.ErrorCorrupted sentinel and wrap CRC and "index out of
range" errors with %w so callers can match via errors.Is.
- copyDataBasedOnIndexFile now skips only when the read failure is
io.EOF, io.ErrUnexpectedEOF, ErrorSizeMismatch, ErrorSizeInvalid,
or ErrorCorrupted. Anything else (real disk faults, environmental
errors, Windows hardware codes) aborts the compaction so an
operator notices.
- Mirror the same whitelist in the Rust volume server, matching on
io::ErrorKind::UnexpectedEof and the NeedleError corruption variants
(SizeMismatch, CrcMismatch, IndexOutOfRange, TailTooShort).
Also add `defer v.Close()` in TestVerifyIndexFitsInDat so Windows
t.TempDir() cleanup can release the .dat/.idx handles.
Refs #8928
* fix(volume): wrap entry-not-found size-mismatch with ErrorSizeMismatch
Address PR review: the fallback branch in ReadBytes returned an
unwrapped fmt.Errorf, so isSkippableNeedleReadError (and any caller
using errors.Is(..., ErrorSizeMismatch)) could not match it. Wrap
with %w so the whitelist applies, while leaving the existing direct
sentinel return for the OffsetSize==4 / offset<MaxPossibleVolumeSize
retry path unchanged so ReadData's `err == ErrorSizeMismatch` retry
still triggers.
Refs #8928
* fix(volume): integrate dangling-idx check into existing index load walk
Address PR review (gemini-code-assist, medium): the structural .idx
check used to do a second linear scan of the index file at every volume
load, doubling the disk-I/O cost on servers managing many volumes.
Track the largest (offset + actual size) seen during the existing
needle-map load walks (`LoadCompactNeedleMap`, `NewLevelDbNeedleMap`,
`NewSortedFileNeedleMap`'s `newNeedleMapMetricFromIndexFile`,
`DoOffsetLoading`) on a new `MaximumNeedleEnd` field on `mapMetric`,
exposed as `MaxNeedleEnd()` on the NeedleMapper interface.
`volume.load()` then compares `nm.MaxNeedleEnd()` to the .dat size
after the load is complete — pure numeric comparison, no extra I/O.
The standalone `verifyIndexFitsInDat` helper and its caller in
`CheckVolumeDataIntegrity` are removed; the test that used to drive
the helper directly now exercises the new path via
`LoadCompactNeedleMap`.
Mirror the same change in the Rust volume server: track
`max_needle_end` on `NeedleMapMetric`, expose via `max_needle_end()`
on `CompactNeedleMap`, `RedbNeedleMap`, and the `NeedleMap` enum.
The Rust load walk already happens in `load_from_idx` for both map
kinds, so the structural check becomes free.
Refs #8928
This is a follow-up fix to PR #7332 which partially addressed the issue.
The problem is that size=0 needles are in a gray area:
- IsValid() returns false for size=0 (because size must be > 0)
- IsDeleted() returns false for size=0 (because size must be < 0 or == TombstoneFileSize)
PR #7332 only fixed 2 places, but several other places still had the same bug:
1. needle_map_memory.go:doLoading - line 43 still used oldSize.IsValid()
2. needle_map_memory.go:DoOffsetLoading - used during vacuum and incremental loading
3. needle_map_leveldb.go:generateLevelDbFile - used when generating LevelDB needle maps
4. needle_map_leveldb.go:DoOffsetLoading - used during incremental loading for LevelDB
5. needle_map/compact_map.go:delete - couldn't delete size=0 entries because:
- The condition 'size > 0' failed for size=0
- Even if it passed, negating 0 gives 0 (not marking as deleted)
Changes:
- Changed size.IsValid() to !size.IsDeleted() in doLoading and DoOffsetLoading functions
- Fixed compact_map delete to use TombstoneFileSize for size=0 entries
Fixes#7293
- Modified `needle_map_memory.go` to include needles with size=0 during needle map loading
- Updated `volume_write.go` to handle size=0 needles in delete operations
* types packages is imported more than onece
* lazy-loading
* fix bugs
* fix bugs
* fix unit tests
* fix test error
* rename function
* unload ldb after initial startup
* Don't load ldb when starting volume server if ldbtimeout is set.
* remove uncessary unloadldb
* Update weed/command/server.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Update weed/command/volume.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: guol-fnst <goul-fnst@fujitsu.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* optimiz vacuuming volume
* fix bugx
* rename parameters
* fix conflict
* change copyDataBasedOnIndexFile to an instance method
* close needlemap
* optimiz commiting Vacuum volume for leveldb index
* fix bugs
* fix leveldb loading bugs
* refactor
* fix leveldb loading bug
* add leveldb recovery
* add test case for levelDB
* modify test case to cover all the new branches
* use one tmpNm instead of two instances
* refactor
* refactor
* move setWatermark to the end
* add test for watermark and updating leveldb
* fix error logic
* refactor, add test
* check nil before close needlemapeer
add test case
fix metric bug
* add tests, fix bugs
* adjust log level
remove wrong test case
refactor
* avoid duplicate updating metric for leveldb index
