152 Commits

Include renames

Author	SHA1	Message	Date
3 people	10b0f2b8ad	volume server: refuse the rest of a grouped run after a durable index failure (#11576) ... * volume server: refuse the rest of a grouped run after a durable index failure A durable write whose needle-map put fails stops the volume taking writes (#10825): sent on its own, the next write then fails read only before it appends. The grouped run from #11543 appends and syncs every entry before publishing any, then kept publishing the entries after the failed one and acked them once the shared .idx sync went through. When the failed put tore its .idx row, the rows appended after it land off alignment, so the next load parses them as garbage and the acked writes are gone. Once a durable entry fails to publish, refuse every later entry of the run with ReadOnly, as the per-needle path does. The entries before it stay acked; their rows go down with the run's one .idx sync. The refused records stay on the .dat unindexed, as the failed one does on its own. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: refuse a grouped entry staged as a cookie mismatch too After a durable entry in a grouped run fails to index, the entries after it are refused as they would be on their own. On its own an entry meets check_writable before its cookie check, so one staged as a cookie mismatch now gets the refusal too, instead of keeping its staging error. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: trim a torn .idx row back so the next stays aligned A failed write_index_entry can leave half a row in the .idx. With the writer appending at the tail, every row written after it lands off alignment and the next load parses them as garbage, so a write acked behind a torn row does not come back. Trim the file back to idx_file_offset on a failed append, in both needle maps, and cover it with a test that writes past a torn row and reloads. * volume server: refuse queued Go writes once a durable index update fails processBatch kept writing after a failed nm.Put, and the single-write path checked IsReadOnly only outside the volume lock. A durable write whose index update fails now marks the volume noWriteOrDelete, and each queued request is checked before it appends, so the ones after a failed durable entry are refused the way a lone write is. Deletes get the same noWriteOrDelete refusal a lone delete gets. * volume server: refuse appends while a torn .idx row cannot be trimmed When trimming back a half-written .idx row itself fails, the next append would land after the torn bytes and every later row would parse off alignment on load. Latch the map as torn and refuse appends until the trim succeeds, on both CompactNeedleMap and RedbNeedleMap; the same latch covers an orphan row that could not be trimmed after a failed redb commit. The .idx writer is now opened with write+append access so truncate_to (set_len) works on Windows, where an append-only handle cannot trim. * volume server: write .idx rows at idx_file_offset, not via append mode Rust's OpenOptions on Windows strips FILE_WRITE_DATA whenever append is set so the handle stays strictly append-only, which makes set_len fail - the torn-row trim could never succeed there. Open the .idx writer with plain write access and seek to idx_file_offset before each row, the same positioned-write model the Go server uses. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-10-03 21:04:30 +08:00
5 people	07da302da0	volume server: ec.decode verifies, cleans up and compacts like Go, off the runtime (#11547) ... * volume server: ec.decode reads the .ecx from the index dir it was copied to VolumeEcShardsCopy writes the .ecx/.ecj into the receiver's -dir.idx, so with a split data/index dir the decode target has no .ecx beside its shards. VolumeEcShardsToVolume sized the .dat from the right .ecx but built the .idx from the data dir, failing with NotFound after the .dat was already published. It now reads .ecx/.ecj from where the EC volume opened them and writes the .idx beside the .dat, where Go leaves it. The live-entry check and the .dat size also ignored deletions recorded only in the .ecj, which Go folds into the .ecx (RebuildEcxFile) first: a fully deleted volume was decoded instead of reported as having no live entries, and deleted tail needles were copied into the .dat. Both now treat journaled ids as deleted, without rewriting the sealed .ecx. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: ec.decode keeps the decoded volume writable and reads every .ecj The rebuilt .idx copied a journaled tail needle's .ecx row verbatim after the .dat was cut short before it, so the mount saw a row past EOF and marked the decoded volume read-only. Rows of deleted needles the .dat no longer holds are now dropped, and each journaled needle still in the .dat gets one tombstone instead of one per journal entry. VolumeEcShardsCopy appends journals collected from other holders into the idx dir, but the decode read only the .ecj beside the .ecx, which sits in the data dir when this server generated the shards. It now reads both, once, in bounded chunks via the loader EcVolume uses. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: test ec.decode drops a sealed .ecx tail tombstone Covers the other half of the rule added in the previous commit: a tail needle tombstoned in the .ecx itself (Go's RebuildEcxFile) is cut from the .dat, and its row must not reach the rebuilt .idx either. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: ec.decode runs its file I/O off the async runtime VolumeEcShardsToVolume released the store lock before decoding, but read the .ecx/.ecj, rebuilt the .dat and wrote the .idx inside the async handler, parking a runtime worker for the length of a volume-sized copy. The decode now runs in spawn_blocking on inputs snapshotted under the store lock. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: ec.decode checks the rebuilt .dat is complete Go stats the decoded .dat before writing the .idx (VerifyDecodedDatFile) and fails the decode when it is shorter than the extent the EC index references, since the caller deletes the shards once the call returns. The Rust handler returned success without that check. The rebuild already fails on a short shard read, so this guards the published file itself. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: ec.decode drops the decoded volume's bitrot sidecars Go removes <base>.ecsum and <base>.ecsum.v<N> beside the .dat and beside the .ecx once the .idx is written, so a stale checksum sidecar cannot pass for the protection of a later re-encode. The Rust handler left them in place. Removal is best effort, as in Go. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: ec.decode compacts the decoded volume Go ends VolumeEcShardsToVolume with an offline CompactVolumeFiles, so the decoded volume holds only live needles. The Rust decode left every needle deleted through the .ecj in the .dat, tombstoned in the .idx, until a later vacuum reclaimed it. Store::compact_volume_files loads the unmounted volume, checks free space the way the vacuum does (the estimate now lives in one helper), and runs the vacuum's compact-by-index and commit. As in Go a failed compaction is logged and the decode still succeeds, so the uncompacted .idx rules stay: the tests that pin them now make the compaction fail. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: ec.decode keeps deletes journaled while the .dat is written The decode read the .ecj journals once, before rebuilding the .dat, so a delete that reached the EC volume during the rebuild was left out of the new .idx and the needle came back live. Each journal's read length is now kept, and the bytes appended since are read just before the .idx is written, after waiting out any journal append in flight (appends hold the store write lock), so every delete acknowledged by then is in the .idx. A delete after that point is still lost, as in Go. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Guard overlapping ec decode requests; serialize journal catch-up volume_ec_shards_to_volume runs its decode in spawn_blocking, so a dropped request leaves the job running and a retry would race it on the temporary and final volume files. Claim the vid in a per-server in-flight set until the blocking job finishes, and return Unavailable to an overlapping request. The Go handler has the same exposure and gets the same guard. Journal appends hold the store write lock through their sync-or-truncate, so holding a read lock across the catch-up read guarantees every record it sees is committed: a rolled-back delete can no longer leave a tombstone in the decoded index. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Reconcile the swap when offline compaction commit fails A CommitCompact that fails after the .cpc marker may have renamed .dat but not .idx. cleanup_compact refuses while the marker exists, so the mismatched pair survived until a restart reconciled it — and the decode caller treats the failure as non-fatal. Run reconcileCompactState on commit failure so a decided swap rolls forward and orphan temps are removed before the volume can mount. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Release the decode claim on panic * volume: add ec_decodes_in_flight to the integration-test state literal * volume server: hold the decode tail's lock through compaction The catch_up read released before the rebuilt .idx was written and the volume compacted, so a delete synced to .ecj in that window was durably journaled yet absent from the published index — resurrecting the needle. Rust now holds the store read lock from catch_up through compact, and Go mirrors it by holding the volume's journal lock from the journal- consuming index write through CompactVolumeFiles. * volume server: serialize ec decode's tail per volume, not per store Review follow-ups on the decode path: - Rust: holding the store read lock from journal catch-up through the offline compaction stalled every writer on unrelated volumes for the whole rewrite. The new ec_decode_tail set marks the vid only while its .idx is published and .cpd/.cpx swapped; the two local .ecj append paths (VolumeEcBlobDelete, the distributed delete's local journal) wait on a Notify for that span — Go's per-volume ecjFileAccessLock semantics without the global stall. VolumeMount and the staged-adopt path are also held off while a decode claim is in flight so neither can race the swap. - Rust: the initial journal read ran unlocked, so bytes a rolled-back append later truncated could be folded in as phantom tombstones. The first pass stays unlocked (a slow journal must not stall the store) and a rescan under the quiescing read lock re-reads only committed content; catch_up now rebuilds the id set when a regular journal shrank. - Go: the decode resolved the compaction DiskLocation through FindEcVolume while holding the journal lock, inverting DestroyEcVolume's map->journal order into a deadlock. The lookup now happens first, and DestroyEcVolume/deleteEcVolumeById/DiskLocation.Close destroy outside the map lock. - Go: RebuildEcxFile unlinks .ecj while the volume's ecjFile handle stays open, so later deletes could commit to a detached inode. Both call sites now fold under the journal lock and ReopenDeletionJournal repoints the handle at the live path, working on the volume's resolved .ecx dir (EcIndexBaseFileName) rather than the configured index dir. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume server: fence EC remounts behind the destroy tombstone DestroyEcVolume, deleteEcVolumeById, and the collection-delete sweep now remove the EcVolume from ecVolumes before destroying it off-lock, so a concurrent remount could re-open shard files that the in-flight destroy then unlinks — registering a detached fd. Each destroy records a per-vid tombstone channel in a new ecVolumesDestroying map before dropping the map entry and closes it when Destroy returns. The tombstone intentionally survives as the vid's destroy generation: loadEcShardWithIdxDir compares it before and after opening the shard, so a destroy that both started and finished inside the open window is still detected. A mismatch drops the just-opened shard (releasing its fd and mount gauge) and retries after the destroy completes; a successful mount clears the stale tombstone. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume server: rescan the .ecj under the store lock only after a rollback The decode's second journal pass ran a full rescan under the store read lock on every decode, stalling unrelated writers for the length of the scan. Bump a process-wide epoch whenever a failed append truncates its uncommitted tail; an unchanged epoch between the unlocked read and the quiesced pass proves every id folded in was committed, so catch_up() suffices. catch_up() also treats a journal that was read but has since disappeared as shrunk to zero, so its earlier ids cannot linger. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume server: check the decode tail under the store write lock on delete A blob delete waited for the publishing tail before taking the store write lock, so a decode that claimed the tail while the delete was parked behind the decoder's read lock could still see the journal append land after the rebuilt .idx — an acknowledged delete the mount would miss. Test tail membership under the write lock instead, retrying after the wait; journal_delete_local reports WouldBlock for the same recheck on the distributed path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume server: claim the vid for mount and staged adoption, per volume VolumeMount and the staged .copying adoption held the ec_decodes_in_flight set lock through slow file renames and mounts, stalling every unrelated volume's decode, mount, and adoption. Take the per-volume claim instead — the same exclusion against a racing decode for this vid, released when the call returns. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume server: fail the decode when a compaction commit marker survives CompactVolumeFiles' caller logged a compaction error and went on to delete the EC shards. When the commit marker (.cpc) is still on disk the .dat/.idx swap was decided but could not be reconciled, so the mounted pair may be mismatched — report the failure instead so the shards are kept and the caller can retry. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume server: gate the parked-delete test on the held write lock The releaser thread and the spawned delete raced for the store write lock; on a slow runner the delete could acquire it first and commit before the tail was ever claimed, failing !delete.is_finished() on the Windows unit-test job. Spawn the delete only after the thread reports the lock held. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-03 14:55:15 +08:00
4 people	0ff7794c54	volume: compact an oversized .ecj at mount, safely (Rust + Go) (#11555) ... * volume: compact an oversized .ecj at mount, safely (Rust + Go) Restore the mount-time compaction dropped from #11408, Rust + Go parity. A journal already bloated by repeated shard copies is folded down to the id set it encodes. - Trigger after load when file_records > max(threshold, 4x distinct), with a 1 MiB floor so small journals are never rewritten. The set is written to .ecj.compact.tmp + fsync, the handle dropped, renamed, the directory fsynced and the append handle reopened. A failure before the rename keeps the original journal and handle; a failure after it fails the mount. - Go never compacts after a failed journal load; the set would be partial and the rewrite would drop the unread records. - A per-path registry (ecj_registry.rs / ecj_registry.go) counts EcVolume holders and out-of-band writers of each .ecj. Compaction runs only when this volume is the sole holder and no copy is writing; holders and writers wait while one runs. This covers shared -dir.idx journals and cross-disk reconcile, where another EcVolume may hold the same journal. - VolumeEcShardsCopy and EC index recovery register as writers around their .ecj append and partial-file cleanup. - Under the reservation, re-check that the file on disk is still the inode and size that was loaded. - Publish errors are classified where they happen; a failed rename plus a failed restore reports both errors. - Compaction runs after the .vif / bitrot checks, so a refused mount leaves the journal untouched. - The tmp is opened like other volume files, removed at mount if a crash left it, and listed in every EC index cleanup path. Failure paths are tested through the real mount via injectable fs steps (open_with / newEcVolumeWith), plus sibling holders, active copies, changed-after-load, stale tmp cleanup, refused mounts and the Go load-error guard. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: fail the mount when the compacted .ecj's directory cannot be synced The Rust mount synced the journal's directory after renaming the compacted file over it through the crate's best-effort fsync_dir, which returns Ok when the directory cannot be opened. A rename needs only write and search permission, so on a directory without read permission the replacement was published, never synced, and the mount went on taking deletes against it. Sync through a helper that propagates the open error, as Go's util.FsyncDir already does, so that case fails the mount like any other post-rename sync failure. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: test the no-compaction-after-failed-load rule through the Go mount The test for it handed compactEcjAfterLoad an artificial error on a volume that had loaded cleanly, so it would not notice NewEcVolume dropping the real load error on the way to compaction. Make the journal read one of the injectable ecjFsOps steps and fail it inside the real mount, after the first chunk, on a journal whose last entry is an id the first chunk does not hold. The mount must leave the file byte for byte as it was; a clean remount then compacts and keeps that id. The Rust mount fails outright on a load error, so it has no equivalent path. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: register ReceiveFile's .ecj writes with the journal registry ReceiveFile refuses a mounted EC volume only once, when the info message arrives, then creates the .ecj and streams chunks into it. A volume that mounted on that journal mid-stream could find a bloated prefix, pass the inode-and-size re-check and rename a compacted file over it; the rest of the stream then went to the unlinked inode and was lost. Register the path as a writer before the file is created, in both the Go and Rust handlers, and hold it until the file is closed and any partial copy removed, as the shard-copy and index-recovery appends already do. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: skip .ecj compaction when a writer ran since the journal was loaded Compaction checked only that no writer was active at the reservation, and that the file was still the loaded inode at the loaded size. A ReceiveFile truncates and refills the journal in place, so one that ran during the mount's load, or after it, and finished before the reservation could leave different ids at the same length; compaction then wrote the stale set over them. Give each path a write generation that every writer bumps as it starts. A holder records it, and whether a writer was active, when it registers, which is before it opens and loads the journal. It may compact only if no writer was active then and the generation has not moved. Same rule in Go and Rust; the journal read becomes an injectable step in Rust as it is in Go, so both test the in-place rewrite through the real mount. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: match the ReadOnly(VolumeId) variant in write_volume_needles #11543 matched VolumeError::ReadOnly as a unit variant in Store::write_volume_needles, and #11544 changed it to ReadOnly(VolumeId) in the same merge window. Each passed CI on its own, but master no longer compiles the Rust volume server. Carry the volume id through. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-10-03 09:08:19 +08:00
3 people	fa77cde7da	vacuum: check compaction space against live bytes, not volume size (#11524) ... * vacuum: size the compaction space check by live bytes, not volume size ensureCompactVolumeSpace required the volume's current .dat and .idx size as free space before compacting. That is the size of the garbage, not of what compaction writes, so on a disk that filled up until its volumes went read-only every compaction was refused, including all-garbage volumes that would compact to a superblock and an empty index. The sweep then retried every volume each cycle and reclaimed nothing (issue #11516). Estimate the output from what the needle map already tracks: live content bytes plus a per-needle framing upper bound behind a superblock, and one index entry per live needle. The estimate never exceeds the current volume size and preallocate still wins when larger. Volumes whose deleted sizes are unknown (.sdx converted back to .idx) keep the whole volume as the estimate. The disk probe moves behind a package variable so the tests can stand in for a full disk; the tests build real volumes instead of re-implementing the formula. * vacuum: space check reserves the index on top of preallocate, checks a separate index disk Review follow-ups: preallocate only stands in for the new .dat, so the rebuilt index is added on top of it; with separate index directories the data disk is checked for the .cpd and the index disk for the .cpx; and the estimates carry 1/16 headroom because counters rebuilt from an index file pass through a Bloom filter with a 0.1% false positive rate. Neither estimate exceeds the current file. * vacuum: split the space check by filesystem, not by directory name Two directories can sit on one filesystem and share its free space, so the data and index estimates are checked separately only when the index directory is on another device; otherwise the sum must fit. Unknown is treated as shared. * vacuum: ask the index directory for its share even when it looks like the same filesystem A volume mounted under the data directory's drive letter on Windows has the same volume name, so the identity check calls it shared. Checking the index directory for the index estimate as well costs one statfs and catches a full index mount either way. * vacuum: identify a Windows volume by its GUID, not its path prefix A volume can be reached through a drive letter and through a folder it is mounted on, so filepath.VolumeName says nothing about the free-space pool. Resolve each directory to its mount point and compare the volume GUIDs; when that fails the two are treated as shared. * vacuum: keep the framing and disk_space_low coverage the rebase displaced * rust volume: split the compaction space check across data and index disks Mirror the Go check: estimate the new .dat and rebuilt .idx separately — live content plus per-needle framing capped at the current file, with preallocate standing in for the data file when larger — and check each directory against its own filesystem's free space. Two directories on one filesystem are asked for the sum. * vacuum: tighten comments on the compaction space check Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-03 08:31:56 +08:00
4 people	35b090a4df	volume: merge .ecj as a set union on EC shard copy + index recovery (Rust+Go) (#11554) ... * volume: merge .ecj as a set union on EC shard copy + index recovery (Rust+Go) An EC volume's deletion journal is a set of needle ids, but shard copy and index recovery appended the peer's whole journal, doubling the file on every ec_balance round trip. Fold the peer's ids in as a union instead: only ids the local journal lacks are appended. - The journal is never replaced. A mounted EcVolume merges a peer's ids through its live handle under the lock deletes take (Go MergeJournal / Rust merge_journal), wherever its journal lives. - An unmounted journal gets only the missing ids appended while mounts are excluded; the delta is read outside the lock and re-read if the journal changed. - The source .ecj streams into memory as an id set: no staging files, chunked reads, memory proportional to distinct ids. - Go and Rust agree that a source journal exists when it sends a modified time or any bytes. A missing source stays a no-op. - Rust runs every merge in spawn_blocking and shares one receive/merge path between shard copy and index recovery. The decode path and the journal format are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: route .ecj merges to the runtime that holds the journal open Disks sharing one index directory all resolved as the journal's owner, so the last one won and a sibling's mounted runtime was skipped: the merge appended behind its open handle and the sibling kept serving the peer's deleted needles until remount. Callers now name the receiving disk by its data directory; the merge goes through that disk's runtime, else a sibling runtime whose journal is the target file. In Go the unmounted append now holds every disk's EC lock (in location order) while it rechecks for a mount, so a sibling mounting from this disk's index during the unlocked read is merged through instead. In Rust a mount that lands during the read is merged through directly and its added count returned, rather than discarded and reported as zero. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: sync merged .ecj records outside the disks' EC locks The unmounted merge held every disk's EC read lock across its fsync, so a slow sync on one disk held off mounts on all of them, along with the EC reads queued behind those mounts. Mounts only need to be excluded while the records are written: the write now happens under the locks and the fsync after they are released, since a later mount reads the written records from the page cache. A failed fsync rolls back only if nothing has mounted the journal or appended to it since the write. A merge through a mounted volume now keeps only that volume's disk locked across its fsync. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: roll back an unsynced .ecj merge through a volume mounted mid-sync If a volume mounted after the unmounted merge wrote its records but before the fsync failed, the rollback kept the records because the journal was now open, leaving ids in the volume's deleted set that may never reach disk; a retried merge then saw them and synced nothing. The rollback now goes through that volume the way its own failed journal fsync does: truncate back and drop the ids from the in-memory set, so a retry appends and syncs them again. It still keeps the records if the volume journaled since, as truncating would lose that delete. No fsync runs under the disk locks. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: decide .ecj merge rollback from the journal's actual length Two runtimes can hold one journal (cross-disk mounts). The rollback of an unsynced merge checked one runtime's cached ecjFileSize, which another runtime's appends leave stale, so it could truncate a delete that runtime had already synced. The rollback now holds every holder's journal lock and truncates only if the file's actual length is still the append's end, then updates each holder's size and deleted set. Otherwise later records follow the merged ones, so they stay and are rewritten in place and synced outside the locks, rather than left possibly not durable. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: keep unsynced .ecj merge ids out of mounted deleted sets When a merge's fsync failed, later records blocked the rollback, and the rewrite-and-sync failed as well, the merged ids stayed in every mounted volume's deleted set without being shown durable, so a retried merge saw them as present and synced nothing. They now leave those sets while the records stay in the file, matching DeleteNeedleFromEcx, which publishes an id only after its record syncs. The merge returns the error and a retry appends and syncs them again. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: publish merged .ecj ids to every holder of the journal Two runtimes can journal into the same file when disks share an index directory. The merge went through only the first holder, leaving a sibling's in-memory deleted set without the ids, so it could keep serving a needle the peer deleted until it remounted. Every holder of the journal now gets the merged ids, in Go and in the volume server. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Publish merged .ecj ids to the journal actually written mountedEcJournal prefers the receiving disk's own runtime for the vid, whose journal may live in its data directory while the copied records name a sibling's journal in the index directory. Publishing by the requested ecjPath then marked a holder of a different file deleted on records that file never persisted, resurrecting the needles on remount. Publish by the picked runtime's journal path instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-02 22:46:36 +08:00
3 people	d8f926cf46	volume server: stream needle chunks without the store lock (#11488) ... * volume server: read GET/HEAD needles off the store lock, and only once The GET/HEAD handler read the needle synchronously on the tokio worker while holding store.read(): first a stream-info read that loaded the whole record just to parse its meta, then, for every needle that was not streamed (small, compressed, chunk manifest, image ops), a second full read. For a tiered volume each read is an S3 GET under the store lock, and a writer queued behind it parks every other store reader. The regular-volume read now runs in spawn_blocking. Under the store guard it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat handle or the remote backend, offset, size); the guard is dropped before any needle data I/O. No data-file lease is held across the read either, since a writer waits for one while holding the store write lock. The index size decides the read, as in Go's readNeedle: a HEAD, a ranged read or a needle above the stream threshold reads only its header and meta tail (ReadNeedleMeta) and hands off to StreamingBody or the range path; everything else is read in full once, with its checksum verified. A compressed or manifest needle found by the meta read is then read in full once. The range-from-source read also moves to spawn_blocking. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: stream needle chunks without the store lock StreamingBody::poll_frame took store.read() and find_volume for every chunk to compare the volume's compaction revision, dup'd the source handle, and allocated a fresh chunk buffer. With -hasSlowRead=false the stream also holds a data-file read lease for its whole life, while a writer waits for that lease under store.write(): the next chunk's store.read() then waits for the writer and the writer for the stream. The per-chunk re-lookup was also wrong. The stream reads a handle opened at plan time, which pins the .dat inode the offset was resolved against; a vacuum commit renames a new file over .dat and leaves that inode untouched. The re-looked-up offset belongs to the new file but was read from the old inode, so a stream whose needle a vacuum moved ended in a checksum error. The pinned offset stays valid, so the check, and with it every store access, is dropped, along with the now unused re_lookup_needle_data_offset and the revision fields of the read plan. The source is shared as an Arc instead of dup'd per chunk, and the chunk buffer is a BytesMut that the blocking read hands back with its result, so its allocation is reclaimed once the previous frame has been written. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: stop a needle stream once its volume becomes unavailable Taking the store lock out of StreamingBody also dropped its per-chunk unavailable_error() check. With -hasSlowRead a writer can take the data-file lease between chunks, fail its fsync and its truncate, and mark the volume unavailable; the stream then kept serving the rest of the needle from its pinned handle. The volume's io_unavailable reason is now an Arc-shared leaf mutex that the read plan hands to the stream. Each chunk checks it under its data-file lease, where the writer marks it, and fails with the same "volume is unavailable: <reason>" error the old check returned. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2026-10-01 23:15:38 +08:00
3 people	8a9563e53d	volume: test that only the disk holding the replaced replica gets the free-slot credit (#11486) ... * volume server: split volume_copy into phases and type the delete-after-status gate volume_copy was one ~400-line handler, and the rule that an existing local replica is deleted only after the source's ReadVolumeFileStatus succeeded was held by statement order alone. The keep_remote_data=true that the pre-copy delete and the failed-copy rollback must share was kept in sync by a comment pointing from one to the other. The handler is now a ~60-line orchestrator over connect_to_copy_source, SourceVolumeStatus::fetch, delete_existing_replica, plan_copy_destination and a VolumeCopyJob whose run() drives preallocate_dat, transfer_files, finish_copied_files and mount_and_reply, with cleanup_failed_copy on error. delete_existing_replica takes a &SourceVolumeStatus, which only fetch can construct (private field in a child module), so the delete cannot be called before the status RPC. Both deletes go through delete_replica_keep_remote. Pure refactor: call order, status codes and messages, cancellation checks, throttling, progress reports and cleanup are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: find space for a VolumeCopy before deleting the replica it replaces VolumeCopy deleted an existing local replica as soon as the source answered ReadVolumeFileStatus and only then looked for a location with room for the copy. With no usable location (disk full, low-disk, wrong disk type) the call errored after the delete, leaving the node with neither the old replica nor the new one. Plan the destination first, as Go does: find_free_location_replacing credits the location holding the replaced volume with that volume's slot, so a disk at its volume limit that holds the replica still accepts the copy. Only then delete the replica and write the .note (still after the delete, as in Go). delete_existing_replica now takes the planned CopyDestination, so the delete cannot precede the plan. find_free_location_predicate keeps its behaviour. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: describe the replace-credit test against the current VolumeCopy flow Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2026-10-01 23:13:17 +08:00
4 people	eeec9ec09a	volume server: refuse to compact a volume tiered to remote storage (#11545) ... * volume server: refuse a tier move while compacting, and a commit once tiered A tier move to remote and a vacuum compaction of the same volume could interleave and leave the volume unreadable: - A compaction committing while the upload ran swapped .dat/.idx under the transfer, which reopens the .dat by path per part. The move then published an object holding the old (or a mixed) layout against the compacted .idx, and with keep_local_dat_file=false deleted the only compacted .dat. - A tier move finishing while the compaction copy ran (or between the copy and the commit) let the commit swap in the compacted .idx while the reload served the pre-compaction remote object through it. The tier move now refuses to start while the volume is compacting, and re-checks the compaction revision under the store write lock before it records the remote file; on a mismatch it deletes the uploaded object and fails with FailedPrecondition, leaving the volume local. Committing a compaction on a volume that has a remote file is refused and its .cpd/.cpx removed, since the reload would read the remote object through the compacted index. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: abort a tier move whose volume was replaced or removed The tier-up bookkeeping looked the volume up by id only and compared the compaction revision. A delete and re-create of the same id during the upload yields a fresh volume at the same revision, so the move recorded the old volume's object on the new one and, without keep_local_dat_file, removed the new .dat. An unmounted volume was skipped and the move reported success, leaving the uploaded object referenced by nothing. Capture the volume instance (its data-file access control Arc, as the scan and read plans do) with the revision, and require both under the store write lock. A replaced volume fails with FailedPrecondition, a missing one with NotFound; either way nothing is recorded and the object is deleted after the lock is released. Go fails in both cases because deleting or unmounting closes the descriptor its copy reads. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: refuse to compact a volume tiered to remote storage Committing a compaction of a tiered volume is refused, since the reload would read the remote object through the compacted index. The compaction itself still started: a tiered volume's data backend is the remote object (the local .dat is dropped or deleted on tier-up), so an explicit vacuum streamed the whole .dat out of remote storage into a .cpd that the commit then discarded. Refuse at the start of the compaction instead, before the .cpd is created, at the point where Go's copy opens the local .dat. The truncated-index test now uses a read-only local volume for its sorted index, since a tiered one no longer reaches the copy. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: match the ReadOnly(VolumeId) variant in write_volume_needles #11543 matched VolumeError::ReadOnly as a unit variant in Store::write_volume_needles, and #11544 changed it to ReadOnly(VolumeId) in the same merge window. Each passed CI on its own, but master no longer compiles the Rust volume server. Carry the volume id through. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2026-10-01 23:11:16 +08:00
Chris Lu	3fd5e4c329	volume-rust: match the tuple variant VolumeError::ReadOnly in write_volume_needles	2026-10-01 06:24:49 -07:00
3 people	9d3907e36c	volume: say "volume N is read only" like Go, so filer retries deletes (#11544) ... VolumeError::ReadOnly displayed "volume is read-only". Go's store and volume say "volume %d is read only", and the filer's deletion classifier requeues a failed delete only when the error contains "is read only". Against a Rust volume server a BatchDelete on a read-only volume (tier move, maintenance) was booked as a permanent failure and the chunk was never deleted. ReadOnly now carries the volume id and displays Go's text. The text reaches clients through BatchDelete results, the HTTP write and delete error bodies, and gRPC statuses; the gRPC code (FailedPrecondition) and the HTTP/BatchDelete status codes are unchanged. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-10-01 21:00:04 +08:00
3 people	bd953b0f84	volume server: group-commit fsync writes in the write queue (#11543) ... * volume: split the write path into reusable steps do_write_request ran its pre-append checks, the append, the sync rollback, the index publish and the post-write bookkeeping inline, so a batched write could only reuse it one needle at a time. Pull the steps out (check_writable, prepare_write, undo_unsynced_append, publish_write, finish_write) and the store's volume lookup plus disk-space check (writable_volume_mut). do_write_request composes them in the same order with the same early returns; no behaviour change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: group-commit fsync writes in the write queue The write queue holds one store lock for a batch of up to 128 needles but wrote them one at a time, so every fsync needle paid its own .dat sync and its own .idx sync: 2N syncs per batch. Add Volume::write_needles_grouped, after Go's processBatch. A volume's entries are split into runs of distinct needle ids (a repeated id starts a new run, so its dedup and cookie checks see the earlier write). A run with a durable entry appends everything with append_at_ns chained through a local, syncs the .dat once, and only then publishes the entries and syncs the .idx once. A failed .dat sync truncates the .dat back to the run start (marking the volume unavailable if that fails), leaves last_append_at_ns and last_modified untouched, and fails every entry of the run. Runs with no durable entry go through the unchanged per-needle path. Store::write_volume_needles is the queue's entry point; the handlers' non-queue path is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: fail closed when a failed append's rollback fails append_needle discarded the truncate-back result, so a partial write that could not be rolled back left unindexed bytes on the .dat while the volume stayed writable; the next append would bury them mid-file, past the load-time tail check. Route the rollback through undo_unsynced_append, which marks the volume unavailable when the truncate fails, so nothing more is appended over an unverified tail. * volume server: keep a grouped run's I/O error streak from later appends A synced run stages every append before any entry finishes, so the success reset in finish_write ran after the failed appends queued behind the last write to land and erased their media-error streak. Sent one at a time, those errors would have counted and quarantined the volume. Skip the reset when an append after the last landed write added to the streak. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: replay a grouped run's I/O error streak in queue order Skipping the run's success reset whenever an append after the last landed write failed kept the errors from before that write as well, so a run like [EIO, EIO, landed, EIO] reached the quarantine count that the same writes one at a time (one error) do not. Mark the streak where each entry is staged and record the run's success at the last landed write's mark: errors before it are cleared, the ones after it still count. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: replay a run's I/O error streak in one atomic step Reads record their outcomes on the tracker without the volume's write lock, so record_success_at's separate load and store could drop an error a read counted in between, or restore errors a read had just cleared. Keep the count and the clear counter in one atomic word and apply the replay with a single fetch_update. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-10-01 20:59:40 +08:00
Eliah RusinandClaude Opus 5.5	fea14c01a7	volume server: refuse a tier move while compacting, and a commit once tiered (#11539) ... * volume server: refuse a tier move while compacting, and a commit once tiered A tier move to remote and a vacuum compaction of the same volume could interleave and leave the volume unreadable: - A compaction committing while the upload ran swapped .dat/.idx under the transfer, which reopens the .dat by path per part. The move then published an object holding the old (or a mixed) layout against the compacted .idx, and with keep_local_dat_file=false deleted the only compacted .dat. - A tier move finishing while the compaction copy ran (or between the copy and the commit) let the commit swap in the compacted .idx while the reload served the pre-compaction remote object through it. The tier move now refuses to start while the volume is compacting, and re-checks the compaction revision under the store write lock before it records the remote file; on a mismatch it deletes the uploaded object and fails with FailedPrecondition, leaving the volume local. Committing a compaction on a volume that has a remote file is refused and its .cpd/.cpx removed, since the reload would read the remote object through the compacted index. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: abort a tier move whose volume was replaced or removed The tier-up bookkeeping looked the volume up by id only and compared the compaction revision. A delete and re-create of the same id during the upload yields a fresh volume at the same revision, so the move recorded the old volume's object on the new one and, without keep_local_dat_file, removed the new .dat. An unmounted volume was skipped and the move reported success, leaving the uploaded object referenced by nothing. Capture the volume instance (its data-file access control Arc, as the scan and read plans do) with the revision, and require both under the store write lock. A replaced volume fails with FailedPrecondition, a missing one with NotFound; either way nothing is recorded and the object is deleted after the lock is released. Go fails in both cases because deleting or unmounting closes the descriptor its copy reads. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-01 20:20:16 +08:00
Eliah RusinandClaude Opus 5.5	f2498e122a	volume: read EC shards fully, like Go's ReadAt (#11537) ... EcVolumeShard::read_at and the scrub plan's EcLocalShard::read_at were a single pread/seek_read. That may legally return fewer bytes than asked mid-file (FUSE/NFS/CIFS mounts, a signal, very large requests), and an Interrupted error was not retried. Callers treat a short count as end of file or corruption: verify_ec_shards compared a zero tail and reported a parity mismatch, local scrub reported a broken shard, VolumeEcShardRead ended the stream early, and decode/rebuild/local needle reads failed. Add storage::io::read_full_at, which loops until the buffer is full or a read returns 0 and retries Interrupted, so a short count means EOF. Route both shard read_at methods through it, replace the encoder's private read_at_most with it, and reuse it for the Windows read_exact_at loop. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-01 20:19:43 +08:00
Eliah RusinandClaude Opus 5.5	62481f1673	volume server: ec.decode reads the .ecx from the index dir it was copied to (#11536) ... * volume server: ec.decode reads the .ecx from the index dir it was copied to VolumeEcShardsCopy writes the .ecx/.ecj into the receiver's -dir.idx, so with a split data/index dir the decode target has no .ecx beside its shards. VolumeEcShardsToVolume sized the .dat from the right .ecx but built the .idx from the data dir, failing with NotFound after the .dat was already published. It now reads .ecx/.ecj from where the EC volume opened them and writes the .idx beside the .dat, where Go leaves it. The live-entry check and the .dat size also ignored deletions recorded only in the .ecj, which Go folds into the .ecx (RebuildEcxFile) first: a fully deleted volume was decoded instead of reported as having no live entries, and deleted tail needles were copied into the .dat. Both now treat journaled ids as deleted, without rewriting the sealed .ecx. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: ec.decode keeps the decoded volume writable and reads every .ecj The rebuilt .idx copied a journaled tail needle's .ecx row verbatim after the .dat was cut short before it, so the mount saw a row past EOF and marked the decoded volume read-only. Rows of deleted needles the .dat no longer holds are now dropped, and each journaled needle still in the .dat gets one tombstone instead of one per journal entry. VolumeEcShardsCopy appends journals collected from other holders into the idx dir, but the decode read only the .ecj beside the .ecx, which sits in the data dir when this server generated the shards. It now reads both, once, in bounded chunks via the loader EcVolume uses. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: test ec.decode drops a sealed .ecx tail tombstone Covers the other half of the rule added in the previous commit: a tail needle tombstoned in the .ecx itself (Go's RebuildEcxFile) is cut from the .dat, and its row must not reach the rebuilt .idx either. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-01 20:19:17 +08:00
Eliah RusinandClaude Opus 5.5	5ece8dd63c	volume server: drop the unused unmount result in VolumeCopy validation, and test-only EC helpers (#11531) ... mount_and_reply ignored the Result of store.unmount_volume when a copied replica failed record count validation, tripping unused_must_use. The Err branch is unreachable there: the volume was mounted under the same store write guard, mount_volume refuses an already loaded vid so it is a fresh Volume with is_compacting false, and a compaction claim needs &mut Volume, i.e. the store lock. Ignore the result explicitly with a one-line reason. Store::delete_expired_ec_volumes and Store::remove_ec_volume are called only from test modules (the heartbeat uses the split find_expired_ec_volumes / remove_expired_ec_volumes halves), so mark them #[cfg(test)]. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-01 20:13:45 +08:00
Chris LuandDevin	0978e7f833	vacuum: keep disk-full read-only volumes reclaimable (#11519) ... * storage/topology: keep disk-full read-only volumes vacuumable The vacuum sweep skipped every read-only replica, so a volume that went read-only because its disk filled could never reclaim its garbage — the exact situation compaction exists for. The volume server now reports disk_space_low in VacuumVolumeCheckResponse, and the sweep skips a read-only replica only when the flag is clear. An explicit volumeId vacuum is unaffected: it already bypassed the read-only rule. The field takes number 4: 2 and 3 are downstream-allocated for tombstone retention, keeping the wire merge clean. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * storage: measure vacuum free space against live bytes The pre-compaction space check required the current .dat + .idx size free, which includes the garbage being reclaimed — on a nearly full disk that estimate can never fit, so the volume stayed garbage-bound forever. Measure against the estimated compacted output instead: superblock plus live index entries plus live content bytes, with the existing ten percent buffer unchanged. Mirrors the same check in the Rust volume server. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vacuum: count per-needle framing in the compacted-size estimate The live-bytes estimate covered each live needle's content and index entry but not its .dat framing (header, checksum, timestamp, padding — ~32 bytes on version 3). For small-needle volumes that is more than the 10% headroom, so a disk with space between the estimate and the real output still ran out mid-compaction. Rust side mirrors the same formula. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * storage: report disk_space_low only when it is the sole read-only cause Review feedback (ihnokim, greptile, devin): a volume read-only for low disk space AND an operator mark or I/O quarantine was still eligible for the automatic sweep, rewriting a copy meant to stay protected. The flag now reports only the benign sole-cause case in both servers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: fail closed when the read-only lookup misses in the sweep A heartbeat can drop the volume from the DataNode cache between the location-list copy and VacuumVolumeCheck; a lookup error previously skipped the read-only check entirely. Review feedback (coderabbit). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-30 17:32:41 +08:00
Chris LuandDevin	43fd5b8d82	volume: reclaim staged EC shard generations left by the 2PC switch (#11501) ... * volume: remove staged EC generation files on teardown and shard delete The 2PC generation switch stages each run as <base>.ecNN.v<N> plus versioned .ecx/.ecj/.vif files. Nothing on the volume server removes them: isEcDataShardFile only recognises the exact .ecNN name, so the staged files are invisible to every bookkeeping pass, and even full_teardown's wipe-all path left them behind. Each re-encode therefore leaks a full shard set per shard-holding disk. RemoveEcGenerationFiles sweeps <base>.ec*.v<N> and <base>.vif.v<N>, optionally keeping generations at or above a threshold; teardown and the reconcile wipe remove every generation, and a per-shard delete removes that shard's staged generations too. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: delete staged EC generations older than N via VolumeEcShardsDelete After a 2PC generation switch commits, the superseded generation's <base>.*.v<N> files sit on disk with no cleanup path: teardown removes everything, and a per-shard delete only touches the named shards, so the executor had no RPC that reclaims just the staged leftovers. delete_generations_older_than removes staged generation files strictly below the threshold on every disk. Versioned files are never mounted, so nothing is unloaded first; the committed generation and the canonical files are preserved. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: mirror staged EC generation cleanup Parity with the Go volume server: remove_ec_generation_files sweeps <base>.ec*.v<N> and <base>.vif.v<N> staged by the 2PC switch, called by remove_ec_volume_files (which covers both teardown paths) and the new delete_generations_older_than request field; delete_ec_shards removes a shard's staged generations along with the canonical file. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: match staged generation filenames literally filepath.Glob interprets metacharacters in the collection part of the base name, so a collection like a[bc] could match another volume's staged files (or miss its own). Scan the directory and compare names literally instead, mirroring the Rust read_dir implementation. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: report generation-sweep errors and drop the store lock first - snapshot the location base names under the read lock and run the filesystem sweep after dropping it, so a slow disk cannot stall the store; - record per-entry read_dir errors in remove_ec_generation_files and propagate them from remove_ec_shard_generations instead of flatten() skipping them; - warn when a staged-shard generation fails to delete rather than reporting success with files left behind. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: fail shard delete when the staged-generation listing fails A transient ReadDir failure fell back to removing canonical shard names only: staged .v<N> files survived while the RPC still reported success, leaving the leak invisible to retrying callers. ENOENT still means the disk simply has no such directory; other listing errors now propagate. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: propagate staged-generation removal failures delete_ec_shards logged remove_ec_shard_generations errors and the RPC returned success while staged .v<N> files remained, diverging from the Go handler which surfaces the failure. The sweep keeps processing the remaining shards, retains the first error, and volume_ec_shards_delete maps it to Status::internal so callers can retry. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: notify state change even when the shard sweep errors delete_ec_shards already deletes and unmounts the shards before returning a staged-generation failure, so returning early skipped volume_state_notify and the master kept routing to them until the next heartbeat. Notify before propagating the error. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-28 21:55:25 +08:00
3 people	e57f8c4d87	volume server: read GET/HEAD needles off the store lock, and only once (#11487) ... The GET/HEAD handler read the needle synchronously on the tokio worker while holding store.read(): first a stream-info read that loaded the whole record just to parse its meta, then, for every needle that was not streamed (small, compressed, chunk manifest, image ops), a second full read. For a tiered volume each read is an S3 GET under the store lock, and a writer queued behind it parks every other store reader. The regular-volume read now runs in spawn_blocking. Under the store guard it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat handle or the remote backend, offset, size); the guard is dropped before any needle data I/O. No data-file lease is held across the read either, since a writer waits for one while holding the store write lock. The index size decides the read, as in Go's readNeedle: a HEAD, a ranged read or a needle above the stream threshold reads only its header and meta tail (ReadNeedleMeta) and hands off to StreamingBody or the range path; everything else is read in full once, with its checksum verified. A compressed or manifest needle found by the meta read is then read in full once. The range-from-source read also moves to spawn_blocking. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2026-09-27 20:13:15 +08:00
hsdfat	02353444ac	fix(volume-rust): reserve a disk before replacing a replica in VolumeCopy, and check record counts (#11483) ... * fix(volume-rust): reserve a disk before replacing a replica in VolumeCopy, and check record counts Port of the Go VolumeCopy hardening in #11238 and #11252. - Pick the destination disk before deleting the existing replica, counting the slot that replica holds as free. If no disk qualifies, the healthy replica is kept instead of being deleted. - Read the source's VolumeStatus before and after the copy. When both succeed and the counts did not change, the mounted replica's file and deleted counts must match; on mismatch it is unmounted and its files removed. A failed "before" read skips the check; a failed "after" read fails the copy. * fix(volume-rust): let a departing caller cancel VolumeCopy's post-copy status read Go reads the source's status after the copy with stream.Context(), so the call ends when the caller leaves. The Rust call had no such link: a source that stalled there held the copied, unmounted files after the caller was gone. Race it against the response channel, like the other blocking steps, so the usual error cleanup removes the partial copy.	2026-09-27 19:40:07 +08:00
Eliah RusinandClaude Opus 5.5	68944e83a3	volume: typed tier errors so a missing remote object answers NotFound (#11484) ... remote_storage/s3_tier.rs returned Result<_, String> from every transfer (upload_file, download_file, read_range[_blocking], delete_file[_blocking]) and from the tier runtime helpers. The tier move handlers could only wrap that in Status::internal, so a .dat whose remote object is gone was indistinguishable from an I/O failure to weed shell. Add TierError { NotFound, Io, RuntimeUnavailable, Aborted }. Each variant carries the existing message verbatim. NotFound follows the rules remote_storage/s3.rs already uses: raw 404 status on HEAD, NoSuchKey code on GET; a bare 404 on GET stays Io. A progress-callback Err becomes Aborted. VolumeError gains a transparent Tier variant and From<VolumeError> for Status maps Tier(NotFound) to NotFound; the tier move handlers go through status_with_context, so their message text is unchanged. Every other tier failure is still Internal. The remote needle read path keeps io::Error::other, so its error kind and vacuum's handling of it do not change. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-27 19:39:25 +08:00
Eliah RusinandClaude Opus 5.5	00310f6588	volume server: run the vacuum compaction copy without the store lock (#11482) ... * volume server: run the vacuum compaction copy without the store lock VacuumVolumeCompact held the store write lock for the whole live-needle copy, including every progress blocking_send on the 16-deep stream. On a large volume that is minutes with every read, write and heartbeat on the node parked behind it, long enough for the master to unregister the node. Split compaction the way Go's CompactByIndex runs it. A short locked step claims the volume's compacting flag, records the makeup_diff watermark (index size and compaction revision) and opens fresh .dat/.idx handles. The copy then replays .idx up to the watermark and copies from those handles with the store lock released; writes that land meanwhile are replayed by makeup_diff at commit, as before. The flag is an Arc<AtomicBool> released when the job is dropped, so every exit path clears it. Because the flag is now visible to other callers, the operations that would pull the files out from under the copy refuse while it is set: unmount (and VolumeConfigure, which unmounts and remounts), delete (checked before the volume is removed from the map, which a refused destroy used to leave unmounted), cleanup, and index relocation. A second compact and a commit stay no-ops, as in Go. The pre-copy fsync is dropped: the copy reads its own handles through the page cache and .cpd/.cpx are fsynced before commit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: keep a read-only in-memory index's size for the compaction copy The unlocked copy replays .idx up to index_file_size(). A read-only volume whose .sdx could not be built loads its index into memory without a writer, so that size stayed 0: the copy came out empty and the commit replaced the volume with it. CompactNeedleMap::load_from_idx now records the rows it loaded, which is also what Go's IndexFileSize reports for a read-only index. The copy's index replay now stops reading at the recorded size instead of walking rows appended since, which makeup_diff replays anyway. Adds tests for compacting a read-only volume on both the sorted index and the in-memory fallback, and for VolumeConfigure stopping when the unmount is refused during a copy. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: stop a vacuum copy as soon as its client is gone The progress callback only noticed a closed response stream when a report was due, every 128 MiB. With the copy now running outside the store lock, a copy nobody waits for keeps the volume marked compacting and so keeps refusing unmount, delete and cleanup until that next report. Check the stream on every callback. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-27 19:38:59 +08:00
Eliah RusinandClaude Opus 5.5	5c9c424a84	volume server: stream ReadAllNeedles without holding the store lock (#11481) ... * volume server: stream ReadAllNeedles without holding the store lock read_all_needles held store.read() while Volume::read_all_needles read every live needle of the volume into a Vec, and kept holding it through the whole blocking_send loop. Memory grew with the volume, and a slow client parked the scan in a send with the guard held; needle writes and the heartbeat take store.write() on a writer-preferring lock, so the node stopped serving until the client caught up. Take a DatScanPlan (fresh .dat open, end bound) under a short guard and walk it with the guard released, sending one needle at a time. Each record is checked against the live needle map under a brief read guard, as the scan reaches it, and only a live record is parsed, so a damaged stale copy does not fail the stream. Records appended while a pass ran are walked by a follow-up plan, so a needle overwritten during the scan is streamed once, as its new copy. A vacuum commit or re-create of the volume during the scan fails the stream, since the map's offsets no longer describe the pinned file; the plan carries the volume instance and compaction revision for that check. DatScanPlan::scan_records yields records unparsed; scan keeps its behaviour on top of it. Volume::read_all_needles has no caller left and is removed; its tests move to the RPC. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: check ReadAllNeedles liveness only once the send can proceed The per-record liveness check ran before blocking_send, so a scan parked on a full channel held a record it had already judged live. An overwrite landing during that park left the old copy in the stream, and the continuation over appended records then streamed the new copy as well. Reserve channel space first, then take the store read guard, check the record against the needle map and enqueue it through the permit before releasing the guard. The wait for space still happens without the lock; the record is parsed before the guard is taken, and its parse error only counts if the record turns out to be live. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-27 19:38:43 +08:00
Eliah RusinandClaude Opus 5.5	5218e68554	volume server: collect heartbeats under the store read lock, off the runtime (#11480) ... * volume server: collect heartbeats under the store read lock, off the runtime Every pulse tick, options change and volume-state notification took store.write() for a whole heartbeat pass, directly on the async heartbeat task. The pass fstats every volume's .dat twice and hashes its report, so on a server with many volumes it held the store exclusively for the whole scan: reads and writes stalled, and with the writer-preferring RwLock a pending pass parked every new reader too. The pass only needs to mutate the store for a few rare actions: removing expired EC volumes, deleting expired volumes past their removal delay, and setting no-write on IO-quarantined volumes. It now runs under store.read(), records those as (disk, volume id) actions, and applies them afterwards under a short store.write() that is only taken when there is something to do. Each action re-checks its target under the write lock, so a volume written to, replaced or removed in between is left alone. Expired EC volumes are still removed before the volume pass, as before, because the EC shard count feeds the disk-space-low max volume count. Every pass runs on the blocking pool via spawn_blocking. The heartbeat message is unchanged for the same store state. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: take has_no_ec_shards with the heartbeat's volume list The heartbeat pass took has_no_ec_shards from the EC phase's read lock, then built the volume list under a second one. An EC shard mounted in between went out as "no EC shards" beside a volume list taken after the mount, and the master clears a server's EC registrations on that flag. has_no_ec_shards is now computed under the same read lock as the volume list, with the EC phase's filter: not expired, not quarantined, at least one shard. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-27 19:38:23 +08:00
Chris LuandDevin	3c1e8ca7a8	volume: never finish serving a needle whose data fails its CRC (#11467) ... * storage: hold back last chunk until CRC verifies on whole-needle reads Above PagedReadLimit the needle is streamed: headers and body go out before the checksum is computed, so a corrupted needle was served as 200 with bad bytes and readers could not fall back to a replica. The final chunk is now written only after the checksum verifies; on a mismatch the response ends short of Content-Length and the client sees a failed transfer. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: verify needle checksum on streamed reads (parity) Mirror the Go fix: carry the needle checksum in NeedleStreamInfo and have StreamingBody accumulate the CRC and verify it before emitting the last frame; a mismatch ends the body with an error so the client sees the transfer fail rather than receiving corrupt bytes that look complete. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * server: abort the transfer when a committed stream fails A writeFn failure after any byte or WriteHeader call leaves the declared status and Content-Length already sent; http.Error's text then joins the body and can exactly fill the withheld tail of a corrupted needle read — the client sees a complete 200 instead of a failed transfer to retry. Track whether the response is committed (headers sent, or bytes buffered for the deferred flush) and panic with http.ErrAbortHandler instead of appending an error body; pre-commit failures keep the 500 path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * server: drop the response writer wrapper from the committed-response check Counting buffered writes is enough: with no bytes buffered the status and headers cannot have gone out, and the range branches commit via the explicit WriteHeader call before writeFn runs. The extra ResponseWriter wrapper added a new Write sink site that CodeQL flags. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-26 16:50:41 +08:00
Chris LuandDevin	0f3ba98e11	volume: make volume.scrub report a live needle whose stored id is damaged (#11468) ... * storage: scrub live needles' stored id against the index key scrubVolumeData only compared the needle's stored id for tombstones, so header damage on a live needle — where the data CRC cannot see it — passed every scrub mode while reads of that needle kept failing or serving the wrong key's data. Compare the id for every indexed needle. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: scrub live needles' stored id against the index key (parity) Mirror the Go scrub fix: compare the stored needle id with the index key for live needles too, not only for deleted ones. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: cover damaged live needle id in scrub test The tombstone test proved the index-key check fires for deleted entries; add the live-needle mirror of Go's TestScrubVolumeDataChecksLiveNeedleId so a regression in the live path is caught in Rust too. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-26 16:10:14 +08:00
Chris LuandDevin	8ad2f29e3e	shell: let volume.deleteEmpty drop volumes with no live needles (#11437) ... * shell: let volume.deleteEmpty drop volumes with no live needles The candidate check only accepted a .dat at superblock size, so a volume whose every needle was deleted still had to be vacuumed first — minutes of compaction to rewrite bytes that were all garbage anyway. FileCount counts every indexed entry and DeleteCount every entry made garbage by overwrite or delete, so FileCount <= DeleteCount means nothing live remains and the volume can be unlinked directly. The quietFor guard is unchanged. * volume server: add only_garbage VolumeDelete guard VolumeDelete(only_empty) refuses every volume that ever held data, so a volume whose needles are all deleted could only be removed after a vacuum rewrote it. The new only_garbage flag deletes only when the byte counters show nothing live: DeletedSize covering all of ContentSize, the same all-garbage state vacuum measures. Byte counters are used because the file/delete counts drift on index reload. * rust volume: mirror only_garbage VolumeDelete guard Same check as the Go server: a volume deletes under only_garbage when its deleted bytes cover all content bytes. The grpc handler rejects before the store drops the volume from its map, since destroy errors after removal would still unmount it. * volume delete: let either enabled check pass, keep onlyEmpty on the wire An upgraded shell sending only_garbage to a pre-upgrade server would be read as an unconditional delete (field ignored, only_empty false). The request now keeps only_empty set so old servers check emptiness and refuse, while new servers delete when either check passes. * volume.deleteEmpty: skip remote-backed and protected read-only volumes A remote-tiered replica shares its cloud object with the other replicas, so keepRemoteData=false on one delete removes data they still reference. Protected read-only volumes are quarantined or under maintenance, which is exactly when a replica should not be dropped. * volume delete: validate guarded copies across disks before deleting * volume delete: hold copy locks across guarded validate-and-delete CheckVolumeDeletable released each copy's locks before Destroy ran, so a write landing on a later copy between the two passes refused its destroy after earlier copies were already removed. Pin every copy's dataFileAccessLock (and its location's volumesLock) across validation and removal so a refused delete leaves all copies intact. * volume delete: send deleted-volume notices after releasing locks A blocking send on a full DeletedVolumesChan under volumesLock can stall the heartbeat loop that drains it while it waits on the same locks. Collect the notices under the lock span and send after release. * pb: restore generated-file cosmetics to match the repo's protoc version Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-26 15:53:39 +08:00
3 people	bb9942c646	volume server: sweep stale EC artifacts before VolumeEcShardsGenerate re-encodes (#11413) ... * volume server: sweep stale EC artifacts before VolumeEcShardsGenerate re-encodes The Rust VolumeEcShardsGenerate went straight into write_ec_files: no unload of an already-mounted EC volume and no stale-artifact sweep. Only .ec00..ecNN on the encoding disk were truncated, so a retry could mix two encode runs. A stale N.ec03 left on a sibling disk survived, reconcile later mounted it against the new .ecx, and the new .vif made the encode_ts_ns identity guard pass, so reads served old-run bytes at new-run offsets. Mirror Go's VolumeEcShardsGenerate (#9880 / #9953): UnloadEcVolume on every disk, then removeStaleEcArtifacts on every disk location before encoding. remove_ec_volume_files_full_teardown already has removeStaleEcArtifacts' semantics (.ec00..ec31, .ecx/.ecj/.ecsum[.vN] in both the data and idx dirs, .vif only on a shard-only disk; never the source .dat/.idx), so reuse it. Add Store::unload_ec_volume, which unlike remove_ec_volume does not stop at the first disk and closes the descriptors so the unlink frees the inodes. The store write lock covers only unload + sweep, not the encode. The failure arm now also drops the generation-0 .ecsum, as Go's defer does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * volume server: wake the heartbeat after VolumeEcShardsGenerate unloads shards The pre-encode unload drops mounted EC shards from memory, but unlike every other unmount path it did not wake the heartbeat, so the master kept routing reads to shards this server no longer serves until the next pulse. Notify once the store lock is released, and before the sweep error propagates: a failed sweep has unloaded the shards too. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * volume server: clean up encode artifacts when the .vif write fails too Go's shouldCleanup defer covers every error before the .vif commits, not just a failed encode. A serialize or write failure on the .vif left the fresh .ecNN/.ecx/.ecsum behind, which the next generate would have to rely on the new sweep to remove. Extract the cleanup and run it on the .vif error paths as well. * volume server: write the EC .vif atomically Go's SaveVolumeInfo writes a temp file, syncs it, and renames it over the target, so a failed write leaves the previous metadata intact and a read-only .vif fails the save. The direct fs::write truncated the file first, so a write or sync failure could leave an empty .vif even after cleanup_encode removed the generated shards. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2026-09-24 07:10:54 +08:00
3 people	f0afcf904d	volume: an EC volume needs an .ecx to mount, and a 0-byte stub never outranks a real index (#11415) ... * volume: an EC volume needs a non-empty .ecx to mount Two gaps against Go in how the Rust volume server treats the .ecx. EcVolume::new mounted with no index at all. The per-shard VolumeEcShardsMount path picks the disk by shard file alone, so a shard whose .ecx was on no local directory still registered and was advertised to the master; every VolumeEcShardRead then failed with "ecx file not open", and add_shard's 0-byte guard was neutralised because ecx_file_size stayed 0. Go's NewEcVolume returns an error wrapping os.ErrNotExist. EcVolume::new now fails with NotFound, and Store::mount_ec_shard looks up the .ecx owner across all disks first (findEcxIdxDirForVolume) so a shard on a sibling disk of its index still mounts instead of turning into a hard failure. A 0-byte .ecx stub, as left by a failed EC distribute copy, counted as a valid index. Go requires Size() > 0 wherever the file steers a decision: HasEcxFileOnDisk, findEcxIdxDirForVolume, indexEcxOwners (shared by reconcile and mirror), and VolumeEcShardsCopy removes a copied 0-byte .ecx and fails the copy. Mirror each through one is_usable_ecx_file helper. NewEcVolume itself still accepts a lone 0-byte .ecx as a legitimate empty index, but prefers a non-empty copy, local directory first, over a stub in the other directory; the resolution in EcVolume::new now follows the same order. Tests that mounted EC volumes without any .ecx get a real fixture. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * volume: mount_ec_shard tries every disk; reconcile ignores a 0-byte local .ecx mount_ec_shard returned the first disk's error, so an unusable shard copy (a 0-byte .ecNN left by an interrupted move) hid a good copy on the next disk. Like Go's MountEcShards, keep scanning: NotFound means "not this disk", any other failure is collected, and an all-disks-fail error names every disk tried. "No .ecx on any local disk" is now told apart from "shard not on this server". The orphan-shard reconcile took its locally-mirrored fast path whenever a local .ecx existed at all. A 0-byte stub there registered the shards against an empty index while the owner index skipped that same stub. Go gates the fast path on HasEcxFileOnDisk; do the same. ec_local_ecx_path loses its last production caller and becomes test-only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * volume: match Go's mount error text and skip the owner stat on the owning disk MountEcShards in Go skips the HasEcxFileOnDisk stat when the disk's own directories already hold the .ecx, dedups a shared -dir.idx across locations in findEcxIdxDirForVolume, and reports "load failures" with the same wording. Also drop two issue-number references from comments. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2026-09-24 07:08:56 +08:00
3 people	94a68fa9b9	volume: walk_index_file keeps row alignment across short reads (#11412) ... * volume: walk_index_file keeps row alignment across short reads walk_index_file issued one Read::read per batch and decoded whatever came back. Read::read may legally return a short count that is not a multiple of the 17-byte entry size (FUSE and network filesystems, a BufReader whose capacity is not a multiple of 17). The split entry at the end of the batch was dropped with no carry and the next read started mid-entry, so every later row was decoded from misaligned bytes and fed to the index as a garbage key/offset/size. This function backs every in-memory index load. Go's WalkIndexFile is immune because it reads through io.ReaderAt, which returns a full buffer or an error. Fill the batch buffer until it is full or the reader reports EOF, retrying ErrorKind::Interrupted, and only then decode whole entries. Reads stay batched at ROWS_TO_READ entries. EOF semantics are unchanged and match Go: on io.EOF Go decodes the whole entries in the final buffer, ignores a trailing partial entry and returns nil. A torn final entry is still skipped without an error here. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * volume: trim walk_index_file comments The batch-fill loop and the ShortReader test helper each carried a paragraph where a sentence suffices. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2026-09-24 07:08:26 +08:00
ssshr-66andChris Lu	b9ad62fc16	[Volume] Keep DAT and index state consistent after async batch Sync failure (#11425) ... * fix 11400 * persist failed-recovery quarantine and harden rollback - record the unavailable state in a .unavailable marker, fsync it, and re-arm it on load so a restart cannot serve an unverified pair - quarantine the volume so heartbeats stop advertising it - block MarkVolumeWritable while unavailable, rechecked under noWriteLock - fail every request of a failed batch, not only the succeeded ones - restore the needle map and truncate .dat on inline fsync rollback failure - add truncateIndex for the sorted-file needle map - mirror the fail-closed semantics in the Rust volume server * volume: erase rolled-back mappings instead of leaving tombstones A rolled-back batch or failed inline write used Delete() to undo a needle that did not exist beforehand, leaving a tombstoned map entry whose stale offset makes the next write to that needle fail reading a header that no longer exists. Add removeMapping/restoreMapping to the mappers so recovery erases entries that were absent before the batch and reinstates the exact prior offset/size for ones that were, including tombstones. The index row still goes through Delete so a replay forgets the needle. * volume: gate bulk readers on unavailable and fsync the marker's dir - fsync_dir(&self.dir) synced the volume dir's parent, not the dir holding .unavailable; pass the marker path so the create survives a host crash - export UnavailableError and check it in ReadAllNeedles, VolumeTailSender, VolumeIncrementalCopy, and IncrementalBackup so replica-sync paths cannot stream or append data from an unverified .dat/.idx pair; mirror on the Rust side via read_dat_slice, read_all_needles, dat_scan_plan, and the incremental-copy handler * volume: drop issue references from comments near touched code * volume: stop active scans when the volume becomes unavailable The stream entry-point checks ran once per RPC, so a volume quarantined by a failed recovery mid-scan kept serving data. Recheck availability per needle/chunk on the detached read paths: tail scan and heartbeat, read-all, incremental copy, incremental backup writes, and the Rust StreamingBody chunk reads. Rust incremental copy also rejects a quarantined volume before sync_to_disk touches the backend. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-09-24 06:57:44 +08:00
Chris Lu	d848b8ed00	rust volume: regression test for makeup_diff replay across a 32 GiB offset boundary (#11410) (#11427) ... * rust volume: test makeup_diff replay across a 32 GiB offset boundary Issue #11410 corrupted a replayed write's index offset in Go's makeupDiff by patching only four of the five offset bytes. The Rust makeup_diff already encodes the whole offset through idx_entry_to_bytes and Offset::from_actual_offset; this adds the mirror of TestConcurrentWriteCrossesOffsetBoundary so a regression would fail here the same way it does under -tags=5BytesOffset on the Go side. Sparse-truncate the .dat to 64 GiB, compact, write, commit: the index offset must equal the .cpd size and the needle must stay readable through a second vacuum. Gated on the 5bytes feature since a 64 GiB .dat exceeds the 32 GiB range of 4-byte offsets. * rust volume: skip the offset-boundary replay test on Windows Windows set_len allocates the full 64 GiB extension instead of a sparse range, so the test fails with StorageFull on CI runners. Gate it to unix, where set_len leaves the extension unallocated.	2026-09-23 18:08:20 +08:00
3 people	0f2ecb766f	volume server: reject non-ASCII input instead of panicking (#11406) ... * volume server: reject non-ASCII input instead of panicking Three parsers sliced attacker-supplied strings by byte offset, so a multi-byte character split inside itself and panicked the task: - parse_needle_id_cookie took the last 8 bytes as the cookie and the rest as the needle id. Reachable from VolumeServer.BatchDelete, whose file_ids come straight off the wire as protobuf strings; that handler already answers 400 per bad fid, so the guard turns a panicked RPC into the error it was already written to return. - TTL::read took the unit as the last byte and the count as everything before it, so "?ttl=5<multi-byte>" split mid-character. The HTTP upload path does TTL::read(..).ok() and drops an invalid TTL; AllocateVolume maps the Err to InvalidArgument. Both now reject non-ASCII up front. Hex and a digits-plus-unit TTL are ASCII by definition, so no accepted input changes -- covered by tests alongside the rejection cases. The six response-* header overrides were inserted with parse().unwrap(). They come from the query string, so "?response-cache-control=%0Aevil" decodes to a value HeaderValue rejects and the unwrap panicked the connection task, unauthenticated. They now skip the override, matching the if-let the chunked-response path in the same file already uses. ReplicaPlacement::from_string was reported as a fourth site but is not one: reaching chars[2] requires chars[0] and chars[1] to be ASCII digits, which forces the padded string to be three single-byte characters, so a multi-byte character always lands on a to_digit() None first. Kept as a regression test rather than a change. Each fix was confirmed against the unfixed code first: the parser tests panic with "byte index N is not a char boundary", and the integration tests panic at handlers.rs:1413 and ttl.rs:88. Not a vector, contrary to the report: the HTTP request line. The path is not percent-decoded before parsing, so "%C3%A9" stays ASCII and fails the length check. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * volume server: fall back to needle MIME when response-content-type is invalid Skipping an unparseable override left the response without any Content-Type because the override had already bypassed the normal MIME selection. Also correct a test comment that described a chars[2] panic which cannot be reached. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>	2026-09-20 23:35:05 -07:00
3 people	ca62d4297b	volume: load the .ecj deletion journal in chunks, and repair a torn tail (#11408) ... * volume: load the .ecj deletion journal in chunks, and repair a torn tail Two independent defects in the EC deletion journal's load path. 1. The loader issued one NEEDLE_ID_SIZE-byte positional read per entry. That is fine for a healthy journal -- kilobytes -- and pathological for a large one. A `.ecj` is semantically a SET of deleted needle ids but is written as an append-only log that nothing dedupes, and several paths append a peer's ENTIRE journal onto the local one (VolumeEcShardsCopy with copy_ecj_file, EC index recovery, and ec_decode's deliberate cross-holder merge), so a volume whose shards are repeatedly balanced between two servers grows the file without bound. Observed in production: 1.51 TB and 1.30 TB on the two holders of one 10+4 volume containing ~100 distinct ids. At that size the per-entry loop is ~188e9 syscalls, run synchronously while holding the deleted_needles write lock and before the HTTP port opens. The process sits at 100% of one core with a small RSS -- the set stays tiny because the ids repeat -- reading at a few MiB/s because 8-byte reads defeat readahead, logs nothing after "Adding storage location", and ignores SIGTERM. The master then unregisters every volume it holds and reads of them fail. 4.46 and 4.47 are both affected. Read in 1 MiB chunks and build into a local set, merging once at the end so the write lock is not held for the whole scan. Measured on a 256 MiB journal of 100 distinct ids: 33,554,500 syscalls -> 257, identical resulting set. 2. A torn tail silently corrupted later deletes. The journal handle is in append mode, so writes land at the physical end regardless of alignment. A trailing partial record therefore pushed every later append out of alignment: the loader skipped the partial bytes, but the next mount decoded them together with the leading bytes of the following entry, producing one garbage id and dropping the delete that came after the tear -- after acknowledging it. Truncate to a whole number of records at mount, before anything can append. The repair uses its own read+write (non-append) handle: on Windows, append(true) requests FILE_APPEND_DATA without FILE_WRITE_DATA (and .write(true) is subsumed by .append(true)), so SetEndOfFile through the journal handle fails with ERROR_ACCESS_DENIED. The same trap exists in journal_delete's recovery path, which calls set_len on the append handle to roll back a partial write whose sync failed. It is error-handled rather than fatal, so on Windows that rollback silently does not happen. Untouched here; worth a separate fix. Bounding the journal's growth needs compaction, which is deliberately not in this change: replacing the file under a store that can hold several EcVolume instances for one volume id requires coordinating with the other holders, and that belongs at the store layer. Sent separately. Tests: a journal spanning several read chunks loads every entry; a trailing partial record is ignored rather than panicking; a torn tail is truncated at mount and a delete taken afterwards survives a remount. * volume: roll back a failed .ecj append through a dedicated write handle The append handle lacks FILE_WRITE_DATA on Windows, so the set_len rollback after a failed sync silently did nothing and the journal could drift one record past deleted_needles. Same trap as the torn-tail repair in this file; fix it the same way. Also format the new tests. * volume: mirror chunked .ecj load and torn-tail repair in Go --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> Co-authored-by: Devin <devin@cognition.ai>	2026-09-20 23:26:41 -07:00
4 people	818f3bb71b	rust volume: share the I/O-error tracker between Volume and EcVolume (#11351) ... * rust volume: share the I/O-error tracker between Volume and EcVolume Volume and EcVolume each carried the same three fields - a mutex-held last error, a consecutive count and a sticky quarantine flag - and the same four methods over them, identical except for the path qualifier on is_storage_io_error. The tolerance the count is compared against was a fourth copy: heartbeat.rs held VOLUME_IO_ERROR_TOLERANCE for volumes, ec_volume.rs held IO_ERROR_TOLERANCE for EC, and the volume test helper open-coded the same 3, so the two paths could drift apart silently. Go keeps this in one place already: weed/storage/io_error.go holds IoErrorTracker, IoErrorTolerance and isStorageIoError, and Volume embeds the tracker. Go's EcVolume has to re-implement it only because those fields are unexported and EC lives in another package. storage::io_error::IoErrorTracker now owns that state, with record / state / should_quarantine / mark_quarantined / reset and the single IO_ERROR_TOLERANCE. is_storage_io_error moves into the same file, so it sits with the tracker that is now its only caller, the way io_error.go is laid out. Both volume kinds embed one tracker and keep their existing method names as delegates, so the ~16 internal call sites and the readers in heartbeat.rs, store.rs and grpc_server.rs change only where the two threshold comparisons become should_quarantine(). Volume::last_io_error and EcVolume::reset_io_error_state had no callers and are gone. Unchanged: what counts as a storage-media error - is_storage_io_error changed file, not body, and is still the single predicate both volume kinds share, where Go's EcVolume tests EIO directly and so misses the Windows codes. Also unchanged: the tolerance value, the metric increment on every counted error, and the sticky quarantine - a success clears the count and the last error but never the flag, which only reset lifts. In the heartbeat the state read moved inside the quarantine branch, so the common path no longer takes the tracker's mutex or clones the last-error string; should_quarantine's two relaxed loads run either way. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * rust volume: hoist absolute_display_path into server handlers.rs and ui.rs each held a byte-identical copy of the helper that turns a configured -dir into an absolute path for display. The status JSON and the status page are meant to show the same directory, so the two copies had to be edited together to stay that way. The helper now lives in server/mod.rs as pub(crate) and both callers use it. No behaviour change: same body, same call sites. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * rust volume: keep EcVolume::reset_io_error_state Moving both volume types onto the shared IoErrorTracker dropped EcVolume's public reset while Volume kept its own, so the two sides of the tracker drifted apart. mark_quarantined is sticky: a later successful read clears the error count through record(), but the quarantine flag only comes down through reset(). Without the delegate an EC volume that hit sustained media errors could not be returned to service in place once the storage was repaired. Go exposes the same method as EcVolume.ResetIoErrorState (weed/storage/erasure_coding/ec_volume.go:114). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * rust volume: name the shared tracker after Go's IoErrorTracker - check_read_write_error, get_io_error_state, mark_io_quarantined, reset_io_error_state match weed/storage/io_error.go one to one - io_error module is pub(crate) like the io module beside it - restore EcVolume::reset_io_error_state so both volume kinds expose the same recovery surface - trim comments that restate the code Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-20 16:57:23 -07:00
Eliah RusinandClaude Fable 5.1	7dbbdac030	rust volume: keep the EC shard-location map, its refresh time and stale mark under one lock (#11356) ... The per-EcVolume shard-location cache was three fields under three locks: an RwLock<HashMap> for the map, a Mutex<Option<Instant>> for the time it was last refreshed, and a Mutex<bool> for the stale mark. Nothing tied them together. merge_shard_locations published the merged map, released the write lock, and only then stamped the refresh time; both readers (scrub_ec_volume_distributed's snapshot and build_snapshot) took the two guards one after the other. A reader landing between the two writes paired a freshly merged map with the previous lookup's timestamp -- and that pair is exactly what needs_refresh judges, so a read went back to the master for a map that had just been refreshed. Go keeps the same state in one struct behind one ShardLocationsLock. replace_shard_locations documented itself as "a single observable step" while being two. Fold the three fields into one ShardLocationCache behind a single RwLock. merge_shard_locations upserts and stamps in one write section, shard_locations_snapshot returns the map and its time from one read section, and mark_shard_locations_stale / claim_shard_locations_refresh move the mark's read-and-consume onto the cache. The three zero-caller accessors -- set_shard_locations, replace_shard_locations, get_shard_locations -- are deleted, and the field is now private, so the invariant cannot be sidestepped from outside the module. The two test seeding sites go through merge_shard_locations, which already produces the state they were writing by hand. Unchanged: the freshness rule. needs_refresh keeps its thresholds and still judges the caller's snapshot -- the map that caller will actually read from, not whatever is cached by the time the claim runs -- so only the stale mark is read from under the new lock. The master lookup, the completeness guard in write_back_shard_locations and the per-shard upsert semantics are untouched. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-20 13:30:15 -07:00
4 people	6d676eda67	rust volume: typed errors for store compaction so gRPC can answer NotFound (#11355) ... * rust volume: typed errors for store compaction so gRPC can answer NotFound The vacuum entry points on `Store` returned `Result<_, String>`, so the gRPC layer had nothing to branch on and answered `Status::internal` for every failure. A vacuum loop that races a volume being moved or deleted saw the same code as a disk going bad, and `weed shell` could only tell the two apart by matching on the message text. `VolumeError` gains `VolumeNotFound(VolumeId)` — the existing `NotFound` is needle-level and carries no payload — and `InsufficientSpace`, and `compact_volume`, `commit_compact_volume`, `cleanup_compact_volume` and `delete_collection` return it. `impl From<VolumeError> for tonic::Status` in `server/mod.rs` maps not-found to `not_found`, read-only to `failed_precondition`, insufficient space to `resource_exhausted`, already-exists to `already_exists`, and everything else to `internal`; the four RPCs prefix their own context with `status_with_context`, so a message reads "commit compact volume 7: volume id 7 is not found". The store-side "during compact" / "during commit compact" / "during cleaning up" suffixes are gone, and the free-space message drops the volume id the prefix already supplies. `check_compact_volume` had no callers — `VacuumVolumeCheck` computes the garbage level from its own `find_volume` — and is deleted. `compact_volume` folded the size estimate into its first lookup, dropping the `unwrap()` re-lookup that only existed to dodge a borrow. `ascending_visit` on `CompactNeedleMap`, `RedbNeedleMap`, `SortedFileNeedleMap` and the `NeedleMap` dispatch is now generic over the visitor's error type, like `CompactMap::ascending_visit` already was. The three signatures that can fail on their own bound `E: From<String>` to carry those failures; the in-memory walk in `iter_entries` names `Infallible`, which says in the type what its comment used to say in prose. No Go shell command matches on the old error text: the strings exist only in weed/storage/store_vacuum.go. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * volume server: trim comments and answer the same codes from Go - vacuum_volume_check reports VolumeError::VolumeNotFound like the other vacuum RPCs instead of its own "not found volume id" wording - drop doc comments that restate what the code says - Go volume server wraps ErrVolumeNotFound/ErrInsufficientSpace from store_vacuum.go so VacuumVolumeCheck/Compact/Commit/Cleanup and DeleteCollection answer NotFound/ResourceExhausted, matching the Rust volume server; volumeDeleteStatusError generalized to volumeStatusError Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume server: prefix operation context on vacuum errors Lower-level errors forwarded by CompactVolume, CommitCompactVolume, CommitCleanupVolume and DeleteCollection carry no volume id or operation name. Wrap with %w so the status mapping still sees the sentinel chain, matching the context the Rust server's status_with_context adds. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume server: map NotEmpty to FailedPrecondition, share mapper in VolumeDelete Go's volumeStatusError maps ErrVolumeNotEmpty to FailedPrecondition; the Rust Status conversion was missing it and volume_delete kept a hand-rolled match. Route it through status_with_context like the vacuum handlers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-20 13:30:02 -07:00
Eliah RusinandClaude Fable 5.1	d002481037	rust volume: derive has_remote_file instead of mirroring it (#11353) ... Volume carried `pub has_remote_file: bool` next to `pub volume_info`, and the bool was only ever the answer to `!volume_info.files.is_empty()`: outside the two constructors, `refresh_remote_write_mode` was the single writer. Both fields being public made the pair a convention rather than an invariant. Every caller that touched `volume_info.files` — load_vif twice, the tier-up handler, the tier-down handler and its rollback — had to remember to call `refresh_remote_write_mode` afterwards, and a caller that forgot would leave the volume advertising a write mode its .vif contradicts, or serving a remote .dat through a writable needle map. The bool becomes `has_remote_file()`, computed from the list, so it cannot drift. `volume_info` becomes private with a `volume_info()` reader, and edits to the reference list go through `update_remote_files(|files| ...)`, which applies the closure and then refreshes the derived write mode and the needle map. With no caller left outside the module, `refresh_remote_write_mode` is private. Unchanged: the refresh logic itself, the order of operations in both tier handlers, and the tier-down rollback semantics. The rollback still snapshots the removed reference before the refresh runs, restores it on failure, and re-refreshes unconditionally on the error path — the second `update_remote_files` call runs with a no-op closure when there was nothing to restore, exactly as the old code re-ran the refresh whether or not it had re-inserted a reference. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-20 13:29:47 -07:00
Eliah RusinandClaude Fable 5.1	1df8c05bc3	rust volume: document every unsafe block and stop mutating the process env in tests (#11352) ... Three production `unsafe` blocks carried no `// SAFETY:` comment at all (`libc::fallocate`, `libc::sysinfo`, `libc::statvfs`), and nothing made that an error: `clippy::undocumented_unsafe_blocks` is a `restriction` lint, allow-by-default, and appeared nowhere in either crate. Turn it on in `seaweed-volume`'s `[lints.clippy]` and in the worker workspace's `[workspace.lints.clippy]`, then document what each block relies on. `memory_status.rs` and `disk_location.rs` get their blocks narrowed to the `zeroed()` and the libc call, so each comment sits next to the operation it justifies and the arithmetic is outside the block. Both turn the success test into an early return on failure; the casts, the multiplication order and the values returned on either path are unchanged. The bigger problem was in `config.rs`'s tests. `with_temp_env_var` and `with_cleared_security_env` called `std::env::set_var`/`remove_var`, claiming soundness because every caller holds `process_state_lock()`. That mutex only serialises the fourteen annotated tests in this module. The same lib test binary runs the `grpc_server.rs` tests, which bind a `TcpListener`, dial loopback and drive a multi-thread tokio runtime, and tonic/hyper/rustls/aws-sdk all read the environment lazily on those threads — which is exactly the race Rust 2024 made these calls unsafe for. `restore_env_var` had no SAFETY comment at all. `#[serial]` would not have helped: it serialises annotated tests, which the mutex already did. So the config layer no longer reads the environment implicitly. An `EnvLookup<'a> = &'a dyn Fn(&str) -> Option<OsString>` is threaded from the public entry points down to every reader — `HOME`, `USERPROFILE`, the twenty-four `WEED_*` keys and `SEAWEED_WRITE_QUEUE`. `parse_cli` and `parse_security_config` keep their signatures and pass `process_env`, a thin wrapper over `std::env::var_os`; `resolve_config` becomes `resolve_config_with_env` (private, one caller). Tests build one with `fake_env` instead, so no test touches the real environment and every `unsafe` in the module is gone. `process_state_lock()` stays, with a smaller job: `set_current_dir` is safe but still process-global, so the tests that move the working directory are still serialised against the ones that read it. Tests naming an explicit config file never reach that search and no longer take the lock. No production behaviour changes: the same keys are read in the same order with the same precedence, and `env_string` reproduces `std::env::var(key).ok()` — absent and non-UTF-8 both read as unset. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-20 13:29:07 -07:00
hsdfat	110b485bae	fix(volume): stop ScanVolumeFileFrom at a header it cannot advance past (#11398) ... fix(volume): stop scans at a header they cannot advance past A corrupt .dat header with a very negative size gives a record length (NeedleHeaderSize + NeedleBodyLength) of zero or less: v3 sizes -43..-36 and v2 sizes -35..-28 give exactly zero, and smaller sizes give a negative length. ScanVolumeFileFrom advanced by that length, so it re-read the same header forever or stepped back into the record before it. weed fix, weed export, weed compact, incremental weed backup and the tail sender behind volume.move and volume.merge could hang on such a volume, and weed compact could also finish with a .cpx that had dropped every needle after the header. Return an error wrapping needle.ErrorCorrupted instead. The check runs after the visitor has seen the record, so the rebuild scanner still stops quietly with io.EOF. Smaller negative sizes whose record length is positive are still stepped over, preserving the salvage behavior compaction relies on. Mirror the guard into the Rust volume scans: DatScanPlan::scan and read_all_needles fail on a non-positive record length, as does scan_dat_head, so a corrupt header cannot stall a tail pass or leave the repair scan walking stale offsets.	2026-09-19 22:26:56 -07:00
hsdfatandChris Lu	06dda12e4b	fix(volume): validate sizes in ReadNeedleBlob and WriteNeedleBlob (#11399) ... * fix(volume): reject negative sizes in ReadNeedleBlob and WriteNeedleBlob A ReadNeedleBlob RPC with a size of -44 or below (-36 on v2 volumes) panics in makeslice inside needle.ReadNeedleBlob. The volume gRPC server has no recovery interceptor, so one request kills the process. Smaller negative sizes return bytes that are not a record. WriteNeedleBlob accepted a negative size whenever the blob header carried the same value: it appended the blob to .dat and indexed the needle with that size, which reads as deleted. Reject size < 0 in both Volume methods. Size 0 still passes, since delete records carry it. The Rust volume server got the same storage guards in #11345. * fix(volume): reject needle blobs whose length does not match their size WriteNeedleBlob appends the blob as is. A blob that is not the length its size implies leaves .dat off the 8-byte grid, and every later ordinary write to the volume is indexed at a truncated offset and reads back as EOF. A blob off by 8 bytes keeps the grid but leaves bytes that a .dat scan reads as the next record. The in-tree callers already send exact lengths. The one case this newly refuses is a copy between volumes of different needle versions, and that case already writes a broken record: a v3 record lands on a v2 volume with 8 extra bytes, and a v2 record on a v3 volume either fails the timestamp check or lands 8 bytes short. This is separate from the negative-size guards, whose Rust counterpart is #11345. The Rust server does not check the length yet. * fix(volume): guard the blob buffer allocation in needle.ReadNeedleBlob Volume.ReadNeedleBlob rejected negative sizes, but needle.ReadNeedleBlob still sized its buffer from the size and is called directly by vacuum and other paths. Reject a deletion marker before make() there too, and use size.IsDeleted() in the volume-level checks. * fix(volume): mirror the blob length check in the rust volume server write_needle_blob_and_index checked the size against the blob header but appended the blob verbatim, so a blob that is not the length its size implies still leaves .dat off the record grid. Match the Go check. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-09-19 21:28:37 -07:00
ssshr-66andChris Lu	cd1e738422	[Volume] Scrub local deletion tombstones during FULL scrub (#11396) ... * fix 11388 * fix(volume): scrub validates local deletion tombstones TombstoneFileSize (-1) is an .idx-only sentinel; the physical record it points at carries a zero-sized body. Normalize deleted index sizes to 0 via onDiskSize before computing disk usage and calling ReadData, so corrupted or truncated tombstone records are detected instead of skipped. Offset-zero entries (remote logical deletes, no .dat record) remain skipped, and the physical needle id is checked against the index key. Mirror the behavior in the Rust volume server. * fix(volume): scrub preserves physical size of deleted non-tombstone entries Size.Raw()/raw() already encodes the index-to-disk mapping: tombstone (-1) -> 0, other negative sizes -> their absolute value (the offset then points at the original record, per the ReadDeleted path). Use it instead of mapping every deleted size to 0. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-09-19 18:55:27 -07:00
Chris Lu	ce1e0dc30a	s3api: don't delete chunks when CreateEntry outcome is ambiguous (#11376) ... * s3api: map ambiguous filer transport errors to retryable 503 Canceled, DeadlineExceeded and Unavailable can be returned after the filer applied the write, so the outcome is ambiguous. Reporting them as a 4xx tells the client not to retry; report ServiceUnavailable instead. * s3api: verify entry existence before deleting orphaned chunks A failed CreateEntry can still have landed on the filer when the error is a transport failure, and entryCreated=false would tombstone chunks a live entry references, leaving a dangling pointer that survives only because reads pass readDeleted=true until vacuum reclaims the needle. Before deleting, look the entry up: if it is stored with the same chunks, the write succeeded; if the lookup cannot be answered, keep the chunks for vacuum to reclaim; only a confirmed absence still cleans up. * s3api: regression tests for ambiguous CreateEntry outcomes Covers the three post-create-failure cases in putToFiler: the entry landed despite the error (treat as success, keep chunks), the entry is confirmed absent (delete orphans), and the outcome is unverifiable (keep chunks, return error). * volume: count reads served from deleted needles A readDeleted read succeeding on a tombstoned needle is the signal that metadata still points at deleted data. Count it under a readDeletedNeedle handler label in both the Go and Rust volume servers so the condition is visible before vacuum turns it into a 404. * s3api: never delete chunks on an ambiguous create error Review feedback on the first fix showed verification could still go wrong in both directions: a stale or lagged lookup could report not-found for a committed entry, a prefix object stores its chunks on a directory entry, and filer-side manifestization rewrites the top-level chunk ids the comparison relied on. Rework the rule so the outcome classes are asymmetric: - A transport-level error (anything filerErrorToS3Error maps to a retryable 503) is ambiguous and never deletes chunks; the lookup can only upgrade the write to success. - Any other error is a definitive filer refusal and still cleans up. confirmCreateLanded asks the write owner first, resolves the stored entry through chunk manifests, requires an exact match of the uploaded file ids, and on success runs the finalize callback the failed create skipped (under the object write lock, with the same rmObject undo the create path uses). Zero-chunk writes stay ambiguous since they cannot be told apart by chunks. * s3api: cover definitive refusals and stale entries in put tests The confirmed-failure case now uses a definitive refusal so it still exercises orphan cleanup, and a new case keeps chunks when the stored entry belongs to an older object rather than this PUT. * volume: count deleted-needle reads once per request Streamed Go reads ran the deleted check in readNeedle and again in readNeedleDataInto, and non-streamed Rust reads in stream_info and the full-read fallback, double-counting one request. Count at the single entry probe each implementation takes per GET: readNeedle in Go, read_needle_stream_info in Rust. * s3api: run recovered-write rollback under the object lock Two follow-ups from review: ResolveChunkManifest returns traversed manifest blobs in its manifestChunks output, so requiring it empty rejected every manifestized landing; and the rmObject undo ran after the object write lock was released, so a concurrent newer write could be deleted between finalize failure and rollback. Compare only the resolved data chunks and keep the undo inside the lock. * s3api: verify, finalize and roll back recovered creates in one lock A lookup done before the object write lock let a concurrent PUT replace the entry between the chunk comparison and the finalize/rollback section, so a failed afterCreate could rmObject a newer write. Run the owner lookup, manifest resolution, chunk comparison, afterCreate and the conditional undo inside a single withObjectWriteLock section.	2026-09-17 19:58:49 -07:00
Eliah RusinandClaude Fable 5.1	799c495226	rust volume: one positional read helper; never seek a dup'd handle on Windows (#11342) ... * rust volume: one positional read helper; never seek a dup'd handle on Windows Positional read-exact was hand-rolled four times: the complete cross-platform version in needle_map/sorted_file.rs, a Windows-only half in volume.rs whose unix half was inlined as a cfg(unix)/cfg(windows)/compile_error! triple at three call sites, a byte-identical Windows-only copy in ec_volume.rs, and read_full_at in ec_bitrot.rs. Three more sites -- EcVolumeShard::read_at, EcLocalShard::read_at and ec_encoder::read_at_most -- hand-rolled the short-read-permitted variant with a cfg(not(unix)) arm that try_clone()s the handle and seeks it. That last arm is wrong. A duplicated descriptor shares one kernel file offset with the original, so seek-then-read is two syscalls against state another thread can move in between: a concurrent reader or an append repositions the offset and the read returns bytes from somewhere else entirely. EcLocalShard::read_at documents that it must never seek, one line above the seek. Windows seek_read carries its own offset in a single call, so that window does not exist. All seven now go through storage::io::{read_exact_at, read_at}, whose module doc records why duplicating a handle is not a way to get a private file position -- opening the file again is, as Volume::dat_scan_plan already does. read_at_most keeps its own fill-until-EOF loop; only the per-iteration positional read changes. Behaviour on unix is unchanged: every unix arm was already FileExt::read_exact_at or FileExt::read_at. The one exception is ec_bitrot::verify_shard_blocks, which now retries on EINTR (std's read_exact_at does; the loop it replaces did not) and, on unix, reports the standard "failed to fill whole buffer" text instead of "short read on shard block". The Windows arm still says "unexpected EOF in seek_read"; both carry ErrorKind::UnexpectedEof, as before. NeedleStreamSource::read_exact_at and Volume::read_exact_at_backend keep their signatures; only their bodies shrink. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * rust volume: retry Interrupted in Windows read_exact_at Unix std's FileExt::read_exact_at ignores ErrorKind::Interrupted and retries, but the Windows seek_read loop propagated it, so the shared exact-read contract differed by platform. seek_read can surface ERROR_OPERATION_ABORTED, which std maps to Interrupted. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-17 11:47:38 -07:00
Eliah RusinandChris Lu	0eb638f503	fix(ec): BatchDelete cookie fail-closed via locate_data geometry (#11348) ... * fix(ec): BatchDelete cookie fail-closed via locate_data geometry * fix(ec): honor skip_cookie_check, require full cookie header * fix(ec): retry short cookie header reads, still fail closed on EOF * chore(ec): trim cookie validation comments --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-09-16 16:25:03 -07:00
Eliah Rusin	caf3d157e6	fix(ec): encode drops tombstoned needles, last-wins replay (#11347) ... * fix(ec): encode drops tombstoned needles, last-wins replay * fix(ec): drop zero-offset rows in encode, match readNeedleMap	2026-09-16 15:40:22 -07:00
Eliah Rusin	def25ca84d	fix(ec): validate ShardId at gRPC boundary, reject >=32 (#11346)	2026-09-16 08:33:22 -07:00
Eliah Rusin	701e397337	fix(volume): reject negative Size, recover poisoned store lock (#11345)	2026-09-16 08:12:11 -07:00
Eliah Rusin	4fc9ada2ec	ci: run seaweed-volume unit tests on Windows (#11349)	2026-09-16 08:06:21 -07:00
3 people	563c729e70	rust volume: stream the tail scan and release the store lock (#11275) ... * rust volume: add a .dat scan plan that runs without the store lock DatScanPlan captures a fresh .dat handle, the version, the start offset and an end bound while the caller holds a store guard, then visits one record at a time with positional reads that never touch the Volume, the way Go's ScanVolumeFileFrom feeds a scanner. The handle pins the inode the offset was resolved against: a vacuum commit renames .cpd over .dat and destroy unlinks it, and neither rewrites the pinned bytes. The end bound is read while no writer can hold store.write(), so the scan never meets a partial append. It is a fresh open, not try_clone, because on Windows read_exact_at uses seek_read, which moves a cursor a clone shares with the writer. A header whose size is negative, or does not fit before the end bound, ends the pass before the body length is computed or anything is allocated. In today's scan a negative size reaches needle_body_length and either overflows the buffer size or walks the scan from a wrong offset. A size near i32::MAX overflows padding_length's i32 arithmetic, which panics in debug builds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018VF7E9SHPihG1jC1grU9H3 * rust volume: stream the tail scan with the store lock released volume_tail_sender read every needle from the start offset to EOF into a Vec while holding store.read(). volume.merge tails from zero, so that was the whole volume in memory. And because needle writes and the heartbeat take store.write() on a lock that prefers writers, the whole node stopped serving until the scan finished: the failure #11235 fixed for EC scrub. Each pass now runs on a blocking thread. Under one store guard it resolves the start offset and captures a DatScanPlan, then drops the guard and sends each needle as it is read, as Go's VolumeFileScanner4Tailing does. This replaces the one-guard-across- search-and-scan rule from the previous commit with a stronger invariant: the offset, the handle and the end bound come from the same guard, and the handle pins the inode, so a vacuum commit mid-scan cannot point the offset into the compacted file. A scan error now ends the stream with Status::internal instead of a clean EOF, as Go's `streamFollow: %w` does. Once needles stream, a clean EOF after a partial pass would let volume.move treat a truncated tail as complete. A panic in the pass is reported the same way. A receiver that hangs up is also noticed between skipped needles, not only on a send. Unchanged: the append_at_ns filter, the header on every 2MB chunk, the caught-up heartbeat without a scan, and the draining countdown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018VF7E9SHPihG1jC1grU9H3 * rust volume: fail the tail pass on a short read below the snapshot end DatScanPlan::scan treated an UnexpectedEof on the header or body read as the end of the data and returned Ok. Every byte below the captured end existed when the plan was taken, so a short read there can only mean the inode was truncated under the plan: an unmount followed by a VolumeCopy of the same volume id reopens .dat with truncate(true). The pass then reported Scanned, the next pass found the volume gone, and the stream ended cleanly after a prefix of the planned records, which volume.move would take as a complete tail. Both short-read arms now fail the scan with an I/O error that names the offset and the snapshot end, so tail_pass reports Status::internal as it does for every other read failure. The break arms were carried over from scan_raw_needles_from, where the whole scan ran under the store guard and nothing could truncate the file. Found by the Devin and Greptile reviews on #11275. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * rust volume: sum the needle padding in i64 so a corrupt size cannot overflow padding_length added the header, checksum and timestamp widths to the needle size in i32. A size read from a corrupt header can sit near i32::MAX, and that sum then overflows: a panic with overflow checks, a wrapped padding without. DatScanPlan::scan bounds the size against the bytes left before computing the body length, but that only keeps such a size out of the arithmetic while under 2 GiB of the file remains, so on a large volume the scan could still reach the overflow and, in release, size a buffer from garbage. Sum in i64 in both version branches. The result is at most NEEDLE_PADDING_SIZE, so it still fits Size. The scan comment no longer claims the bound check prevents the overflow. Found by the CodeRabbit review on #11275. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * rust volume: propagate dat scan parse failures --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-09-15 14:40:06 -07:00
3 people	166af06a2b	rust: cargo fmt both crates, with a commented-out fmt --check CI step (#11329) ... * rust: migrate seaweed-volume and seaweed-worker to tonic 0.14 / prost 0.14 tonic 0.14 boxes the contents of tonic::Status, which is what made every RPC path trip clippy's result_large_err; the allow for that lint goes in the next commit. The prost codec moved out of tonic into tonic-prost and tonic-prost-build, so both build scripts now call tonic_prost_build::configure() and both crates depend on tonic-prost for the generated code. The `tls` feature was split into a per-backend feature; `tls-aws-lc` is the same backend both crates already install through rustls::crypto::aws_lc_rs. tonic 0.14 depends on axum 0.8 and tower 0.5, which would have left a second axum and a second tower in each tree next to the 0.7 / 0.4 the crates named themselves. Bumping them keeps one copy of each: axum 0.8 only changes the path-parameter syntax for the routes here (`/:vid` -> `/{vid}`, `/*path` -> `/{*path}`), tower 0.5 needs the `util` feature named explicitly for ServiceExt::oneshot (it used to arrive through tonic's feature unification), and tower-http 0.6 is the matching release. Lock files move only through cargo's own resolution for the new versions; no other dependency was refreshed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust: drop the result_large_err allow now that tonic::Status is boxed tonic 0.14 stores Status behind a Box, so Result<_, Status> is no longer a large-Err type and clippy has nothing to say about it. Both crates pass `cargo clippy --all-targets -- -D warnings` without the allow (seaweed-volume in both feature sets), so the policy entry and its comment go. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: drop the unused headers argument of try_expand_chunk_manifest The parameter was already named `_headers`; nothing in the body reads it. With it gone the function is under clippy's argument threshold and the expect goes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: pass EC peer reads an EcInterval instead of ten arguments fetch_one_interval, read_remote_ec_shard_interval, do_read_remote_ec_shard_interval and recover_one_remote_ec_shard_interval all took the same (vid, needle_id, shard_id, shard_offset, size, expected_encode_ts_ns) tuple, and the two that reconstruct also took the location map with the data/parity counts. Those are now EcInterval (Copy) and EcShardMap (a borrow of the map plus the counts). The fan-out inside recovery builds its per-shard request with `EcInterval { shard_id: sid, ..iv }`, which is the one place the old argument list was easy to get wrong. Bodies destructure at the top, so the code below the signatures is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: give the EC encoder an EcEncodeLayout and an EncodeRun encode_dat_file took the Reed-Solomon shape and three block sizes as five loose integers; they are now one Copy struct, EcEncodeLayout, which is what Go calls ECContext. The per-row and per-batch helpers took the same six sinks and the offsets; they become methods on EncodeRun, which owns the borrows for one run, so each call names only the offset and block size that vary. The byte-level work is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: describe a .dat rebuild with DatRebuild instead of nine arguments write_dat_file_from_shards, its _with_dirs twin and the private write_dat_file were three layers over one nine-argument signature. One public function now takes a DatRebuild, whose shard_dirs is None when every shard sits beside the .dat and Some(dirs) for the cross-disk reconciled layout. The field docs carry what the function doc used to say about the encode-time size and the block layout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: split copy_file_from_source's fifteen arguments into two structs CopyFileSpec is the per-file request (what to ask the source for, where it lands, whether its bytes count as progress); CopyProgress is the sender, throttler and report state that all three files of one VolumeCopy share, held by &mut across the calls. The three production call sites now read as the .dat/.idx/.vif literals they are, instead of positional trues and falses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: create volumes from a VolumeSpec Volume::new, DiskLocation::create_volume and Store::add_volume each took the same five-value tail of Go's NewVolume argument list: collection, replica placement, TTL, preallocation and needle version. That tail is now VolumeSpec, a Copy struct whose Default is what almost every test wanted anyway (empty collection, no replication, no TTL, no preallocation, current version), so most of the 104 call sites shrink to `&VolumeSpec::default()` or name the one field they set. The id, directories, index kind and disk type stay positional because they differ at every site. Two imports that only test modules use moved into those modules, and DiskLocation no longer imports ReplicaPlacement. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-worker: run cargo fmt Layout only; no token in the workspace changes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: run cargo fmt Layout only; no token in the crate changes. Every earlier Rust PR here formatted only the blocks it touched so as not to drown its diff in this one, and this commit is that debt paid in a single place. rustfmt needed two passes to settle one block in handlers.rs; the committed form is the fixed point, so `cargo fmt --check` is clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * ci: add a commented-out cargo fmt --check step to both Rust workflows Same shape as the commented clippy step from #11312: the check is written out so that making formatting a gate is a one-line uncomment, and whether to do that stays a maintainer call. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2026-09-15 09:29:22 -07:00