* fix 11400
* persist failed-recovery quarantine and harden rollback
- record the unavailable state in a .unavailable marker, fsync it, and
re-arm it on load so a restart cannot serve an unverified pair
- quarantine the volume so heartbeats stop advertising it
- block MarkVolumeWritable while unavailable, rechecked under noWriteLock
- fail every request of a failed batch, not only the succeeded ones
- restore the needle map and truncate .dat on inline fsync rollback failure
- add truncateIndex for the sorted-file needle map
- mirror the fail-closed semantics in the Rust volume server
* volume: erase rolled-back mappings instead of leaving tombstones
A rolled-back batch or failed inline write used Delete() to undo a
needle that did not exist beforehand, leaving a tombstoned map entry
whose stale offset makes the next write to that needle fail reading a
header that no longer exists. Add removeMapping/restoreMapping to the
mappers so recovery erases entries that were absent before the batch
and reinstates the exact prior offset/size for ones that were,
including tombstones. The index row still goes through Delete so a
replay forgets the needle.
* volume: gate bulk readers on unavailable and fsync the marker's dir
- fsync_dir(&self.dir) synced the volume dir's parent, not the dir
holding .unavailable; pass the marker path so the create survives
a host crash
- export UnavailableError and check it in ReadAllNeedles,
VolumeTailSender, VolumeIncrementalCopy, and IncrementalBackup so
replica-sync paths cannot stream or append data from an unverified
.dat/.idx pair; mirror on the Rust side via read_dat_slice,
read_all_needles, dat_scan_plan, and the incremental-copy handler
* volume: drop issue references from comments near touched code
* volume: stop active scans when the volume becomes unavailable
The stream entry-point checks ran once per RPC, so a volume quarantined
by a failed recovery mid-scan kept serving data. Recheck availability
per needle/chunk on the detached read paths: tail scan and heartbeat,
read-all, incremental copy, incremental backup writes, and the Rust
StreamingBody chunk reads. Rust incremental copy also rejects a
quarantined volume before sync_to_disk touches the backend.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Allows to track overall R/W errors in real time through Prometheus.
Will follow up with a PR for Seaweed's REST API.
Co-authored-by: Lisandro Pin <lisandro.pin@proton.ch>
