mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-12 06:48:45 +01:00
ca4ce2c5816d5b20c75d36d5b64b0104e3bd6fd3
180
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
ca4ce2c581 |
volume: a read plan waits for a vacuum commit instead of answering 404 (#12375)
* volume: a read plan waits for a vacuum commit instead of answering 404 A vacuum commit closes the needle map, swaps the compacted files in and reloads the map. It holds the data-file write lease for that and takes no store lock. needle_read_plan, which the HTTP GET path uses, read the needle map without the data-file read lease, so a GET planned in that window found no map and answered 404 for a live needle. needle_read_plan now takes the read lease first, as read_needle_with_option does, so the plan waits for the commit and then reads the reloaded map. Go's readNeedle does the same under dataFileAccessLock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: plan a read with the store guard released A plan that waits on a vacuum commit's data-file lease under the store read guard parks store writers (allocate, mount, EC ops) on that commit, and every store reader queued behind them. Look the volume up under the guard and plan after releasing it, as Go's readNeedle does, and look the volume up again if its handle was closed while the plan waited. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: retry a read plan while its handle keeps closing A plan that waited out an unmount looked the volume up only once more, so a replacement handle that also closed before its plan took the lease answered NotFound for a needle a later mount holds. Repeat the lookup while the planned handle is closed, at most three times, so a closed handle that stays in the map still ends the loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: start the swap-window check once the plan holds the volume The swap-window test gave the plan 300 ms to answer early, so a reader thread that started late passed the check without waiting on the commit. Share the writer test's wait for the plan's volume handle and start both timed checks after it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: run the swap-window read-plan test on a redb map too A commit reloads the index through load_index_redb for the redb kinds, a path the in-memory run never took during the swap window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: say a needle read plan is resolved under the data-file lease The plan no longer resolves under the store guard, and resolving it takes the data-file read lease. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: read the needle through the plan in the read-plan tests The tests checked only that the plan resolved and then read the needle through Store::read_volume_needle, so a plan with a wrong offset or size after the commit still passed. Read through the plan, as GET does, and compare the data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: test the read plan's lookup after an unmount closed the handle No test reached the closed-handle retry, so dropping it left every test green. Take the lookup as a closure so a test can hand the plan the handle an unmount closed and then the remounted volume, and check that the plan reads the needle from the remount. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandClaude Opus 5.5
|
5c99744e00 |
rust volume: count sorted-map files and deletes the way Go does (#12365)
* rust volume: count sorted-map files and deletes the way Go does The Rust sorted needle map still used the counter rule Go dropped when the index loaders were unified: every .idx row, tombstones included, counted as a file, and each delete added one file and two deletions. Go counts put rows as files and a delete as one deletion of the live row's bytes, so the Go and Rust copies of one volume reported different FileCount and DeleteCount for the same index. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rust volume: test canDelete volume counters across restarts Mirrors the Go store test: a canDelete volume reloaded from .sdx must count its deletes, and the counters it reports must match what the next reload computes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d84ae4f2ab |
volume: don't let an unreachable tier backend hold up volume server startup (#12312)
* s3 tier backend: bound connect and response-header waits The tier client used the SDK default transport: a 30s dial timeout and no limit on waiting for response headers. A blackholed endpoint cost about two minutes per request with the SDK's retries, and an endpoint that accepts connections but never answers held the request forever. Dial with a 10s timeout and give up on a response whose headers do not arrive within a minute. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: stop the disk scan from waiting on a stalled tier backend A volume server registers with the master only after every volume on its disks has loaded, and loading a tiered volume reads its superblock from the remote object. With the S3 endpoint unreachable, each read waited out the client's timeouts and retries on one of the loader workers, so start-up grew with the number of tiered volumes and the local volumes were not served until it finished. A tiered volume already loads without its superblock when the read fails. Give that read 10s, and once it stalls, let the backend's other volumes skip it for a minute instead of each waiting out the timeout. The volume reads from the backend as before once the endpoint answers again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: bound connect and read waits of the S3 tier client Mirror the Go tier client: a 10s connect timeout instead of the SDK's 3.1s default, and a minute to receive a response, so an endpoint that accepts connections but never answers cannot hold a read forever. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: stop volume loading from waiting on a stalled tier backend Mirror the Go loader: the superblock read of a tiered volume gets 10s, and once it stalls the backend's other volumes skip it for a minute. The volume loads with its .vif version as before when the read fails, and reads from the backend once the endpoint answers again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: keep the .vif replication when a remote superblock read fails Go applies the .vif replication to a tiered volume whether or not its superblock read succeeded, so a volume loaded while its backend is down still reports its replica placement. The Rust loader applied it only on success and reported the default placement instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: skip the .idx head repair while the tier backend is unreachable The repair reads needle headers from the remote .dat, so after the superblock read failed it would wait on the same backend again and hold up the load. It runs on the next load that reaches the backend. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: skip the .idx head repair while the tier backend is unreachable Mirrors the Go load: after the remote superblock read failed, the repair's remote header reads would wait on the same backend again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 tier backend: give up a ranged read whose body stops arriving The response-header timeout ends once headers arrive, so an endpoint that answers and then sends no body held ReadAt, and a superblock read's worker, forever. Cancel the request once the body makes no progress for s3ReadTimeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: test that a tier read whose body stalls gives up The SDK's stalled-stream protection already fails it, matching the Go read's no-progress cancel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): keep the S3 read timeout off upload_part The SDK's read timeout runs from the start of the request, so a 64 MiB part on a slow link would time out while its body is still being sent. Go's ResponseHeaderTimeout only starts after the body is written. upload_part now keeps the connect timeout and drops the read timeout; stalled-stream protection still fails an upload that stops moving. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): bound the upload_part response wait from when the part is sent Without a read timeout, an upload_part whose body was accepted but never answered waited forever. The part body now goes to the connection in 64 KiB chunks, and once the last chunk is taken the response must arrive within S3_READ_TIMEOUT, like Go's ResponseHeaderTimeout. A retry rebuilds the body and restarts that clock. The body keeps its in-memory contents, so signing and the checksum header are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): keep a retried part's response clock from an earlier attempt The send state was a single flag, and a retry body could be rebuilt and fully sent before the waiter saw the flag drop, so the waiter kept the earlier attempt's deadline. Each body now carries its attempt number, only the latest attempt marks the part sent, and the deadline holds only while no later attempt has started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
80aad76986 |
rust volume: remove the redb index when a failed VolumeCopy cleans up (#12250)
* rust volume: remove the redb index when a failed VolumeCopy cleans up A copy that mounts and then fails validation is unmounted, not deleted. With a redb needle map the mount has already built <idx base>.rdb, and cleanup_failed_copy removed .dat/.idx/.vif/.note but not the .rdb, so the index database outlived the copy. A later copy of the same volume to that server reuses it (RedbNeedleMap::try_reuse_rdb keeps an .rdb whose recorded idx size is not larger than the new .idx and replays only the tail), which is wrong once the source has been compacted in between. Move the partial-file removal into CopyDestination::remove_partial_files and include the .rdb, next to the .idx where Volume::file_name puts it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * rust volume: clear a stale index before a copy writes, and report cleanup failures A crashed copy, or one whose partial-file removal failed, leaves a stale .rdb/.sdx next to the .idx; the next copy's mount would reuse it via try_reuse_rdb's tail replay. remove_partial_files now also covers .sdx and returns the first non-NotFound error: callers that can still abort (the pre-copy clear) do so, while the post-failure cleanup warns so a surviving index is visible instead of silently poisoning the next copy. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: restore the pre-copy leftover clear A leftover index check-in accidentally gated remove_partial_files behind a constant false, so a stale .rdb/.sdx at the destination survived the copy that was meant to clear it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(volume): expect 409 when a read-only upload is refused x-seaweed-write-outcome: refused-before-append. The HTTP integration test still expected 500, so Rust unit tests failed on Linux and Windows. Deletes stay 500. * rust volume: keep a shared .vif, and clear a stale index beside the .dat remove_partial_files deleted <data>.vif even when a sealed .ecx still shares it, undoing the keep in Volume::destroy. It also cleared the index only under the index base, so a leftover .idx beside the .dat still made load trust that directory's .rdb. Skip .vif while a non-empty .ecx is present, and clear both bases when the index directory differs from the data directory. * rust volume: do not delete .dat files in the index directory The index directory can be another location's data directory. Clearing .dat and .note there removes that replica. Those files come off the data base only; .idx, .rdb, and .sdx are still cleared on both. * rust volume: serialize copies of a volume on one server A second VolumeCopy of the same volume could pass the existing-volume snapshot before the first copy mounts, then clear or overwrite the files the first copy still owns. Hold a per-volume claim for the whole copy, like ec_decodes_in_flight does for decodes, and fail a duplicate fast so the caller can retry. Mirrors the Go volumeCopiesInFlight guard. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: keep files when the mounted-copy delete is refused When a copy mounted and then failed, deleting through the store can be refused (e.g. the volume is compacting), leaving it registered. The cleanup then unlinked its files anyway, stranding a live replica. Keep the files while the volume is still in the store. Mirrors the Go copy cleanup. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: init volume_copies_in_flight in the http test state Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: refuse a mount while that volume is being copied volume_copy claims the vid, snapshots the store, then waits on the source. A VolumeMount in that window can load the replica, and the copy's pre-write cleanup then unlinks its files. Check the claim under the store write lock and return Unavailable. * rust volume: refuse configure while that volume is being copied volume_configure unmounts and remounts. A copy that already snapshotted the volume as absent can then unlink the files that remount loaded. Check the copy claim under the store write lock and return Unavailable. * rust volume: preserve files a copy displaces so a failed copy restores them * rust volume: keep an interrupted copy's .note with the files it marks displace_existing_files skipped the .note, so a crashed copy's leftovers were saved without the marker the load scan rejects them by, and remove_partial_files then deleted it. A retry that failed restored the partial set marker-free: after a restart the server could load an unfinished copy as a normal replica and serve wrong data. Displace the .note like every other file at the basename so the restore puts back the exact state the copy found. * rust volume: restore displaced files over a surviving partial restore_displaced_files dropped the backup whenever the basename was still occupied. When the partial cleanup could not remove every file the copy landed, an existing path just meant the removal failed — deleting the saved file destroyed the old replica and let the surviving partial pair with restored siblings of another generation. Clear the leftover first so the displaced file goes back, and keep the backup when it cannot: the operator can recover from copying/ rather than lose the replica outright. * rust volume: back up an EC-shared .vif the copy truncates in place A .vif a sealed .ecx shares was left out of the displaced set so the EC volume keeps it — but the incoming copy opens the same path with truncate, so a failed copy left the shared .vif overwritten by the rejected copy's settings with no way back. Go keeps the original via a hard link because its staged file swaps in by rename; truncating the shared inode makes a link useless here. Copy the content into copying/replaced_ like the other backups instead: the file stays in place for the EC volume during the copy, and the restore puts the original settings back on failure. * rust volume: serialize reload loads with the copy claim A SIGHUP reload scans every volume basename and sweeps .note-marked files as interrupted copies — including one a live VolumeCopy is still writing. Mirror Go's loadGuard: the copy-claim set is shared with each disk location, the scan skips a claimed vid outright and claims any other through its load, and reconcile_compact_states takes the same skip before rolling a leftover commit over the copy's files. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: hold the load claim through open_volumes The scan released each claim when its iteration ended, before the queued volume's index read ran in open_volumes. A VolumeCopy taking the vid in that gap could displace or remove files out from under Volume::new — the register the guard exists to prevent. Queued claims now live until the open pass returns; a second collection of an already-claimed vid merges into the entry instead of re-claiming. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: refuse allocation while that volume is being copied AllocateVolume registered a volume while a copy of it held the destination: create_volume loads the index before registration, and the copy then clears or rewrites files the live replica already opened, leaving it mounted on a stale index — the window VolumeMount and VolumeConfigure already refuse. Claim the volume id like the Go AllocateVolume does, so registration either wins the claim before the copy starts or sees it and refuses with Unavailable. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: refuse staged adoption while that volume is being copied The from_staged branch of volume_ec_shards_to_volume renames .copying files into place and mounts, all while a copy of the same vid could be rewriting the destination — the same registration window VolumeMount refuses. Hold the copy claim for the whole adoption, as Go's adoptStagedVolume does. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: drop a partial .note when its write aborts the copy A write_note failure after displace_existing_files restores the saved replica but leaves the empty or partial marker it just made; the next load scan then deletes the restored replica as an unfinished copy. Remove the marker before restoring, as the Go side's error defer does. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chris.lu@gmail.com> |
||
|
|
43a312e309 |
rust volume: keep a copied replica on deleted-count drift, as Go does (#12251)
* rust volume: keep a copied replica on deleted-count drift, as Go does check_copy_counts rejected a VolumeCopy whenever the target's deleted count differed from the source's. On a redb needle map that count is an estimate: metrics_from_idx rebuilds it at load through a bloom filter (0.1% false positives per entry, each counted as an overwrite), and the source holds the estimate from its own last load plus exact runtime updates. A source written since its load and a fresh load of the same files disagree by a few, so good copies were refused with "target deleted count [N+1] is not same as origin deleted count [N]". An index of unique keys 1..=128 already reports one deletion. Go fixed the same rejection in #11701 (issue #11678) by warning instead of failing. Do the same for the deleted count. The file count stays strict: it is exact on both sides (every non-tombstone .idx row is one file), and the .dat/.idx size checks still gate integrity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * test(volume): expect 409 when a read-only upload is refused x-seaweed-write-outcome: refused-before-append. The HTTP integration test still expected 500, so Rust unit tests failed on Linux and Windows. Deletes stay 500. * rust volume: keep the strict count check when only deletes drift Deletes append tombstone rows without moving the file count, so a copy whose source only deleted mid-flight still has an exactly comparable file count. Gate the strict check on file-count stability alone; the deleted-count drift is already a warning inside check_copy_counts. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
12dac68d66 | seaweed-volume: replace deprecated AtomicU64::fetch_update with try_update | ||
|
Chris Lu
|
94e9b8a49a |
fix(volume): return conflict and explicit proof for read-only upload refusals (#12236)
* fix(volume): report read-only HTTP refusals in Rust * fix(volume): mirror HTTP write refusal proof in Go * test(volume): cover HTTP read-only refusal parity * test(volume): cover local refusal after remote append * fix(volume): honor decoded replica controls before retry proof * fix(operation): reassign proven HTTP conflict refusals * fix(operation): stop same-target retries on HTTP conflict * fix(volume): replace replica marker when forwarding requests * fix(java): gate conflict reassignment on refusal proof * fix(volume): preserve authentication query encoding in replication * fix(operation): stop unproven conflicts in forever retries |
||
|
Chris LuandDevin
|
cba409cae7 |
backup: carry index-only tombstones through incremental copies (#12231)
* volume server: add idx_content to VolumeIncrementalCopyResponse A remote-tier delete is an index-only tombstone row with no .dat record, so it cannot reach an incremental copy through the .dat stream. Carrying the .idx tail verbatim lets the backup reproduce those rows at their exact positions. * backup: carry index-only tombstones through incremental copies A remote-tier delete leaves an offset-0 tombstone in .idx with no .dat record. The copy only streamed .dat bytes and regenerated .idx from them, so the tombstone never reached the backup: the deleted needle stayed live and every later incremental diverged and rebuilt again. backupCursor and ResumeByLastEntry now accept an index-only tombstone as a boundary, matched by key and size since there is no record to read a cookie from. VolumeIncrementalCopy appends the .idx tail after the .dat tail, and IncrementalBackup writes it at the index boundary, truncates, and loads the new rows — so tombstones land at the exact positions the replica has them. * seaweed-volume: mirror index-only tombstone handling in incremental copy resume_by_last_entry accepts an offset-0 tombstone boundary matched on key and size, skips its missing cookie check, and resumes .dat streaming at the next entry that has a record — index-only rows ride the new idx_content stream instead of reporting divergence. The server captures the .idx bound before the .dat one (a row is published only after its bytes) and streams the tail after the data. * backup: keep incremental copy bounds and streams consistent Review fixes: * Bound capture moved after the resume reads and reads .dat/.idx sizes together (Go FileStat, Rust file_stat under one lock), so every row resume saw is inside the index bound and each row's record is always inside the data bound. * A compaction commit changes CompactionRevision: the server now rejects the copy when the revision moved between resume, file opens, or the end of the stream, so a backup never mixes revisions. The .idx is opened up front to pin the inode. * An index tail that ends before the captured bound is an error, not a silent success. * IncrementalBackup rolls the .idx tail and .dat bytes back to the pre-copy boundary when the stream fails, so a partial row cannot wedge backupCursor and unapplied rows cannot pass for current ones. * The index tail is replayed through a small indexTailReceiver interface, covering the leveldb needle maps too, not just the in-memory map. * backup: bound the resume scan to the captured index size The resume and the stream bounds are now captured in one order: revision, then the .dat/.idx pair, then the scan. ResumeByLastEntry and resume_by_last_entry take the index bound and scan no further, so a row appended while the copy is being set up can never enter the index stream without its .dat bytes. The index-tail replay stays on the in-memory map only: the leveldb map's DoOffsetLoading writes to the .cpldb compaction database, not the live one, so it cannot serve a live tail replay; other maps keep regenerating the index from .dat as before. * seaweed-volume: guard incremental copy total against a located offset past .dat A corrupt index can place the resume offset beyond the captured .dat size; has_data already refuses the reader, but total still evaluates dat_size - s and underflows. * backup: bound the incremental .dat tail by index-covered records A record staged but not yet indexed can sit inside the captured .dat bound while its row stays outside the index bound — the fsync gap between staging and publishing is not covered by one stat. Streaming those bytes leaves them unindexed on the backup, and the next run appends the record again. Bound the .dat tail by the last record the streamed rows cover on both producers, and cover the bound walk with tests. * backup: cap the .dat stream at the captured bound and roll back replay failures sendFileContent could read past stopOffset on its last chunk, so a needle appended after FileStat arrived without its index row and the next copy appended at the wrong position. Each read is now limited to the remaining captured bytes. A failed truncateIndex or DoOffsetLoading after a successful stream left the index tail on disk unapplied, which a later backupCursor could mistake for applied rows; both now roll the tail back to the pre-copy boundary like the stream-error path. * backup: bound the incremental .dat tail by the maximum covered offset A key-sorted .idx ends on the highest-key row, which need not be the record physically last in .dat, so the last-row bound could cut off live records. Take the end of the highest-offset row inside the bound instead, and cover the case with a sorted-idx test in Go and Rust. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * backup: restore .dat and the needle map when index replay fails A failed DoOffsetLoading left the received .dat bytes in place and could leave partial mappings for rows the index rollback just removed, so the next copy appended the same records again or read stale entries. Truncate .dat back to the copy boundary and restore the touched mappings and metrics from a pre-replay snapshot. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * backup: cover no index rows with the superblock bound When no row inside the index bound names a record — an empty index or a bound of index-only tombstones — the covered end is the superblock, not the captured .dat size, so staged bytes are never streamed without a covering row. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * seaweed-volume: apply the covered bound even when the index streams no rows The bound was gated on idx_end > idx_start, so an empty captured index skipped it entirely and a staged .dat tail still shipped. Gate on the stream-index predicate instead, matching Go's streamIndex check. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * backup: propagate index coverage read errors * backup: rebuild the needle map from the truncated index on replay failure * seaweed-volume: propagate index coverage read errors --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
349576b63e |
backup: resume incremental copies by index position instead of appendAtNs (#12228)
* volume server: add last-entry cursor fields to VolumeIncrementalCopy * backup: resume incremental copies by index position instead of appendAtNs appendAtNs is stamped by whichever replica received the write, so the same needle carries a different timestamp on every replica. A backup that resumes from a different replica than last time can append the last needle again or skip needles written inside the clock gap. Send the local copy's index entry count and last entry identity (key, size, cookie) with VolumeIncrementalCopy. The replica verifies the same record sits at that position and streams from the next index entry, so the resume no longer depends on any replica's clock. When the boundary record does not match, or the local copy is ahead, the replica returns FailedPrecondition and the backup rebuilds the local copy from scratch. Old servers ignore the new fields and keep the timestamp behavior. * rust volume server: mirror positional resume in VolumeIncrementalCopy * volume server: add entries_hash prefix checksum to VolumeIncrementalCopy * backup: verify the whole index prefix, not just the boundary entry * rust volume server: verify the index prefix hash in resume_by_last_entry * backup: keep the source's compaction revision on rebuilt copies * backup: diverge on a zero-offset boundary entry A remote-tier tombstone lives in the .idx with no .dat record, so readNeedleAt(0) parses the superblock and its error bypassed the ErrIncrementalCopyDiverged mapping; the backup got a generic failure instead of FailedPrecondition and never rebuilt. * rust volume server: mirror zero-offset boundary divergence * rust volume server: gate the empty-dat early return on entry_count With a positional resume the Go handler still reaches ResumeByLastEntry, so an empty replica reports divergence; the Rust early return answered a clean empty stream and weed backup kept a stale copy. * backup: hash the index entry's size field, not trailing offset bytes In a 5BytesOffset build the offset occupies bytes 8-12 and the size starts at byte 13, so the hash folded in one byte of the offset. Matching records at different offsets then hashed differently and a valid backup was reported diverged and rebuilt. * backup: read the index in whole-entry chunks for the prefix hash When entries are 17 bytes a 1 MiB read buffer does not end on an entry boundary: the partial tail entry was skipped by the loop but still counted as consumed, so every later chunk was hashed at a misaligned position. * rust volume server: hash the index entry's size field after the offset Mirrors the Go hashIndexEntries change: with the 5bytes feature the offset occupies bytes 8-12, so starting the size slice at byte 12 folded one offset byte into the hash and matching records at different offsets compared diverged. * rust volume server: read the index in whole-entry chunks for the prefix hash Mirrors the Go change. With 17-byte entries a 1 MiB buffer does not end on an entry boundary, so slicing the partial tail entry panicked once a backup held 61681 entries. * rust volume server: default the new incremental copy request fields in tests |
||
|
Chris LuandDevin
|
2c510cc30e |
ec.rebuild: remount shards the rebuild replaced, keep regenerated copies (#12193)
* volume: remount ec shards the rebuild replaced under a mounted handle Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec.rebuild: keep copied shards the rebuild regenerated in place Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: reopen regenerated ec shards in place instead of remounting UnmountEcShards and MountEcShards send deltas through separate buffered channels, so a remount can reach the master as add-then-delete and drop the repaired shard from lookups until the next full EC heartbeat. ReopenEcShard swaps the mounted shard's descriptor under the volume lock instead: registration, disk type, and heartbeat state never flip, and readers holding the old shard degrade to a clean error the parity path already recovers from. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: reopen rebuilt ec shards in place, mirroring Go volume_ec_shards_rebuild recreates a shard file under a mounted handle whose descriptor still serves the detached inode (a mounted shard whose file was deleted under it). DiskLocation::reopen_ec_shard swaps the mounted descriptor for a fresh open, matching Go's ReopenEcShard — the registration and heartbeat view never flip. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: fail ec rebuild when a regenerated shard cannot be reopened * rust volume: fail ec shards rebuild when a rebuilt shard cannot be reopened * volume: repoint mounted ec shards after rebuild across disks A mounted shard whose file was deleted under the mount is "missing" to the rebuild: the regenerated file lands on the rebuild location while the handle keeps serving the detached inode on another disk. Reopening the mounted disk then failed the whole RPC, so the caller never saw the rebuilt ids and its deferred cleanup deleted the regenerated copies. Repoint every mounted handle of the rebuilt volume instead of only this run's outputs: reopen in place when the rebuilt file is on the mounted disk, otherwise drop the stale handle and remount the shard from whichever local disk holds the file via a delta-free load split out of MountEcShards — a same-server move, so the master's registration never flips. A shard that can be neither reopened nor remounted is left unmounted rather than serving replaced bytes, and the repoint step no longer fails the RPC, so RebuiltShardIds always reaches the caller. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: repoint mounted ec shards after rebuild across disks Mirror the Go repoint loop: iterate every mounted shard of the rebuilt volume, reopen in place when the rebuilt file is on the mounted disk, and otherwise drop the stale handle and remount the shard from whichever local disk holds it. The repoint no longer fails the RPC, so rebuilt_shard_ids still reaches the caller and its copy cleanup keeps the regenerated shards. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: still rebuild ecx and repoint mounts when no shard is missing * master: keep a node in an ec shard's lookup while a disk still holds it A rebuild that relocates a mounted shard between a node's disks reports the move as a delete on the source disk plus an add on the destination. Registering first is a no-op because shard lookups key on the node, then the unregister dropped the node outright — and steady-state heartbeats never restore it, so readers and later rebuilds lose the repaired copy. Skip the delete while another disk on the node still holds the shard. UnregisterDataNode now clears the node's own ec records before dropping its registrations, so the retained-shard check cannot skip the teardown, and each unregister gets a ShardsInfo copy since DeleteEcShard drains the shared one. * rust volume: fold the journal on a mounted index instead of rewriting it The rebuild handler regenerated .ecx through File::create even while the volume was mounted, truncating the index under readers that search the open descriptor with its old size — a live lookup could hit EOF mid-scan. Go folds the deletion journal in place on a mounted volume; do the same through rebuild_ecx_from_journal and reserve the full regen for the unmounted path, where nothing holds the file open to interrupt. * volume, rust volume: report rebuilt shards on late failures, fix rust corrupt discovery Two review findings the repoint rework left open: - A RebuildEcxFile failure still returns a plain error after shards were regenerated, so the caller discards the response holding the rebuilt ids and the deferred cleanup deletes copies the rebuild regenerated in place. The Go handler now packs the ids into the gRPC status details and the shell pulls them back out on error; the Rust handler carries them in grpc-status-details-bin the same way. - The Rust rebuild still counted a sidecar-corrupt shard as present, so it fed bad bytes into reconstruction and never regenerated the shard. Discovery now loads the generation-0 .ecsum the way Go does, verifies every present shard, and reclassifies mismatches (and zero-size residue) as rebuild targets written to a .rebuilding temp file, verified, and atomically renamed over the original only after every output passes the sidecar check. Honors unsafe_ignore_sidecar, added to VolumeEcShardsRebuildRequest to match the Go proto. * volume: repoint sibling journals sharing a folded .ecj after rebuild RebuildEcxFile unlinks the deletion journal and the mounted path only reopens its own handle — a sibling mount journaling into the same file keeps appending to the detached inode, and those deletes never reach a path-based reader again. Reopen every mounted volume sharing the index base before leaving the rebuild. * rust volume: repoint sibling journals sharing a folded .ecj after rebuild rebuild_ecx_from_journal replaces the journal and reopens only its own handle; mounts on sibling disks sharing that .ecj keep the detached handle, so their next appends are lost at restart. Adopt the folded journal on every mounted runtime sharing the path before releasing the store lock, clearing their in-memory set — every journaled id is now an .ecx tombstone. * ec: swap the deletion journal handle only after the new file opens A failed reopen left ecjFile nil, so every later delete returned "ec volume closed". Open first, then close and swap — mirroring the Rust adopt_folded_journal ordering. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec.rebuild: fail the rebuild when a sibling journal cannot be repointed A warning left the sibling mount journaling to a detached inode while the RPC reported success. Propagate the error like the Rust handler's adopt_folded_journal map_err. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec: keep deletes loud when the journal reopen cannot open the replacement Reverts the swap-on-success ordering: a detached handle kept on failure lets deletes append and sync to the removed inode, then vanish at restart. Closing first also frees the descriptor before the open. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: drop the folded journal handle before reopening it adopt_folded_journal and the post-fold recreate now clear ecj_file before opening the replacement, so a failed reopen leaves deletes erroring instead of appending to the detached inode. Mirrors Go's ReopenDeletionJournal. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec: drop the cached journal size when its reopen fails Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
ae95b2c777 |
ec: rebuild a needle's suspect shard ranges when it fails to parse (#12170)
* ec: let shard-interval recovery exclude a set of suspect shards gatherEcShardIntervals and recoverOneRemoteEcShardInterval skipped only the shard being rebuilt. A needle-level rebuild wants every suspect shard kept out of the gather so a bad copy cannot poison the reconstruction it is being compared against. * ec: rebuild a needle's suspect shard ranges when it fails to parse A bit-rotted data shard serves a clean-looking but wrong interval, so the missing/unreadable-shard fallback never engages and the needle CRC check in ReadEcShardNeedle returns the corruption to the caller. Each interval's source shard is a suspect: rebuild the suspect ranges from the surviving shards — all of them at once while parity covers the suspect set, or one at a time when the needle spans more shards than that — and let a successful needle parse arbitrate which copy was corrupt. Shards whose range changed are dropped from the location cache so later reads go straight to reconstruction. * ec: mirror the corrupt-shard needle rebuild in the Rust volume server The snapshot now keeps every interval's shard placement so a parse failure can name its suspects. rebuild_ec_shard_needle rebuilds the suspect ranges through recover_one_remote_ec_shard_interval with the suspect set excluded from the gather, and the winning candidate's differing shards are dropped via forget_shard_location. * ec: recover suspect ranges under the requested needle id A failed ReadBytes re-reads the id out of the possibly corrupt header, so the rebuild passed whatever it decoded to peer recovery requests; carry the requested id through instead, and reject a candidate whose parsed id does not match it. * ec: retry a failed needle rebuild with each gathered input dropped The gather behind a suspect-range rebuild can itself serve a second corrupt shard: Reed-Solomon then reconstructs wrong bytes and the needle parse rejects the only candidate. Gather every surviving shard once, then re-decode with each of the first decode's inputs dropped in turn; the parse arbitrates the honest subset. * ec: mirror the dropped-input rebuild and id check in the Rust volume server Same two guards as the Go side: a candidate that parses under a different needle id is rejected, and a failed candidate whose gather fed it a corrupt input is retried by dropping one gathered shard at a time. The gather is shared through a want-bounded gather_ec_shard_intervals so the retry pays no extra fan-out. * ec: answer deleted when a rebuild's holder reports the needle deleted The rebuild discarded the deleted flag its interval recovery collected, so a needle deleted on its other holders but live in the local index came back as a corruption error. A deleted report is authoritative, so it now short-circuits the hypothesis search and answers ErrorDeleted. * ec: hydrate rebuild candidates into a scratch needle A failed ReadBytes leaves partially parsed fields on the needle, and optional sections do not reset when absent, so a corrupt first parse could leak stale name/mime/ttl into a successful candidate. Parse each candidate into a fresh needle and adopt it only on a matching id. * ec: corrupt parity+1 shards in the exceeds-parity read test The loop skipped the suspect's own shard when it fell inside the parity range, leaving only ParityShards corrupt shards — a case recovery can legitimately fix. Count the additional corruptions so the test always exercises the boundary it names. * ec: mirror the deleted-report short-circuit in the Rust volume server A holder's deleted answer during a rebuild propagates out as EcMiss::Deleted instead of surfacing as InvalidData, matching the Go path's authoritative-delete rule. |
||
|
|
6ebc055a94 |
master, volume, filer: start on a read-only mounted directory and serve existing data read-only (#12081)
* master, volume, filer: start on a read-only mounted directory and serve existing data read-only A read-only mount was treated as writable (permission bits only), so every service failed on its first write: the master panicked writing raft conf.tmp, the filer exited opening the leveldb LOCK, and the volume server skipped every volume while still offering the directory as free capacity. - util.IsReadOnlyMount: access(W_OK) == EROFS. - volume: on a read-only -dir take the existing O_RDONLY/noWriteOrDelete load, index .idx in memory when no .sdx exists, offer no free slots (same branches as isDiskSpaceLow), skip the vid stamp there. - master: on a read-only -mdir run raft from a private copy and refuse new volume ids. - filer: leveldb2/leveldb3 set their existing ReadOnly option. * volume: skip read-only mounts when picking EC shard copy targets A shard copy landing on a read-only mount fails at write time; offer only writable locations like the other placement paths do. * volume: mirror the read-only mount handling in the Rust volume server The .dat open only fell back to read-only on EACCES, so a real read-only mount (EROFS) still failed the load. The index side already degrades once no_write_or_delete is set. Add is_read_only_mount via access(W_OK) == EROFS, mark DiskLocation.read_only_mount, and mirror the Go checks: no free slots, heartbeat max = used, no vid-stamp writes, no EC shard copy targets, and a read-only index dir forces the volume read-only. * volume: write vid stamps while only the index dir is read-only * filer: skip the write-bit check when a leveldb store opens read-only * filer: serve reads of buckets with no leveldb3 dir on a read-only store * volume: derive a stable dir uuid when a read-only mount cannot store one * master: keep the read-only raft copy across restarts so votes survive * volume: index a missing .idx in memory on a read-only mount * volume: release the .idx handle after an in-memory read-only load * volume: mount EC shards read-only when their index dir is on a read-only mount * volume: mark the index dir read-only before the tail repair opens .idx * volume: write vid stamps while only the index dir is read-only (Rust) * volume: derive a stable dir uuid and report no free slots on a read-only mount (Rust) * volume: index a missing .idx in memory on a read-only mount (Rust) * volume: refuse an EC copy whose shard owner is on a read-only disk * master: key the read-only raft copy by the absolute -mdir Two masters sharing the same relative -mdir text from different working directories would have collided on one temp copy and shared raft state. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: mix device and host into the read-only dir uuid Deriving the uuid from the path alone let independent servers at the same mount point report one uuid, which RegisterUuids rejects. Hash in the device id and hostname so restarts stay stable but disks differ. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: vic6996-web <320442181+vic6996-web@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
0d721fccee |
fix(volume): keep preallocated growth above -volume.minFreeSpace (#12067)
* fix(volume): keep preallocated growth above -volume.minFreeSpace When free space fell below -volume.minFreeSpace, every volume went read-only and the master kept growing new volumes. With preallocation on, each new volume reserved its full size, driving the disk further below the floor (15 Oct incident: 66 empty volumes reserved about 1980 GiB). Two causes, both in weed/storage/store.go: - MaybeAdjustVolumeMax sized a disk's volume slots from raw free space, so space inside the floor was counted as available. It now counts only free space above the floor. - addVolume trusted the once-a-minute low-disk flag. A preallocated volume is now refused when the disk is low now, or when its reservation would not fit above the floor. Volumes without preallocation, including merges that copy existing data, are not affected. addVolume runs under a store-level lock, so two concurrent creates cannot both pass the room check against the same free space. * volume: mirror headroom guard in the rust store find_free_location only trusted the once-a-probe is_disk_space_low flag, so a burst of preallocated creates could still run the disk under -volume.minFreeSpace between probes, and maybe_adjust_volume_max sized slots from raw free space. Adds MinFreeSpace::is_low/available_space to match Go and applies both checks the same way. store.write() already serializes add_volume, so no extra lock is needed. * volume: read rust headroom tests from a fake disk The floor arithmetic used a live free-space reading, so unrelated disk activity could flip a test between pass and fail. Store.disk_stats_of is now the same seam Go uses (diskStatusOf): the growth-loop and floor tests run against a controlled disk, mirroring the Go suite one for one. * volume: hold back create metadata from the headroom check An exact-fit preallocation left free space exactly at the floor, so the .vif and .idx writes that follow a create could dip under it and get the location flagged read-only at the next probe. Reserve room for .vif on the data filesystem, and for .idx growth only when dir_idx shares it. * volume: mirror the metadata allowance in the rust store Same rule as the Go side: charge .vif against the checked filesystem and .idx only when dir_idx shares it, and grow the exact-fit test's fake disk by the per-create cost. * volume: skip the reservation check where create cannot preallocate * volume: mirror the platform preallocation gate in the rust store * volume.copy: count the preallocation reservation in the destination space check The copy path picked a location with room for the copied .dat, then reserved the master's full volume size limit on it. A volume smaller than the limit could land on a disk whose free space above -volume.minFreeSpace fit the copy but not the reservation, driving the disk under the floor the same way the growth loop did. Fetch the preallocation setting before choosing the location and require room for the reservation itself, only where CreateVolumeFile actually reserves blocks. * seaweed-volume: store min-free-space-adjusted available space like Go Go's DiskLocation.AvailableSpace holds free space above the -volume.minFreeSpace floor; the Rust probe cached raw free instead, so the VolumeCopy location check could accept a disk already inside the floor. Store the same adjusted figure and cover it with a test. * seaweed-volume: reserve the copied .dat like Go's CreateVolumeFile VolumeCopy called set_len for the reservation, which only grows a sparse file — no blocks are held, so a disk with room for the copied bytes but not the full reservation only fails mid-copy. preallocate_file fallocates KEEP_SIZE on Linux like Go, reserving the blocks up front; off Linux it is a no-op, matching PREALLOCATE_SUPPORTED. * seaweed-volume: charge a copy's destination pick for the .dat reservation Go's VolumeCopy now sizes its free-location check against the master's volume size limit when the platform preallocates, not just the copied bytes. The Rust pick only measured the copied bytes, so a destination could accept a copy whose reservation then dips under the floor. The config fetch moves ahead of the pick (same order as Go) and feeds both the space check and the reservation itself. * volume.copy: charge the picked location's free space under volumeCreateLock The copy path picked a destination off the once-a-minute AvailableSpace probe and released the decision long before the .dat reservation landed. Two copies could pass the same cached headroom and together drive the disk under -volume.minFreeSpace. ReserveFreeLocation serializes the pick under volumeCreateLock, verifies the disk still has live room for the reservation, and charges it against the location's cached free space until the next probe refreshes it. * seaweed-volume: mirror the copy pick's live-check and headroom charge Go's VolumeCopy now picks through Store.ReserveFreeLocation: live disk stats under the store write lock, then the reservation charged against the location's cached free space. The Rust plan_copy_destination gains the same store.reserve_free_location so two copies cannot spend the same headroom before the first .dat reservation lands. * volume.copy: saturate the free-space charge against a live-space drop The two statfs reads bracketing a pick can race a large external write; a plain subtraction would underflow and leave the location looking roomy until the next probe. * volume.copy: release the headroom reservation when a copy does not land ReserveFreeLocation charges the picked location's cached free space, but a failed copy never returned it, so the headroom stayed booked until the next disk-space probe and blocked retries. ReleaseFreeLocation refunds the charge; the copy's cleanup defer now registers with the reservation so every failure path (existing-replica delete, .note write, stream send) releases it, including the post-transfer mount/validate failures covered by the outer defer. * seaweed-volume: mirror the copy headroom release on failed copies release_free_location refunds reserve_free_location's charge. The copy returns it when the plan's existing-replica delete fails, when the .note write fails, and in cleanup_failed_copy, so a failed copy stops holding the headroom until the next disk-space probe. * volume.copy: keep pending copy headroom out of probe readings ReleaseFreeLocation refunded the reservation into AvailableSpace even when a CheckDiskSpace probe had already rebased the cache on live stats — the probe's reading still shows the in-flight copy's bytes as free, so the refund double-counted headroom and two copies could spend the same space, driving the disk under -volume.minFreeSpace. Track the outstanding charge in DiskLocation.ReservedSpace instead: probes subtract it so they cannot re-spend it, a failed copy removes only its own charge, and a landed copy commits it — the charge and the new usage cancel out. hasRoomForNewVolume respects pending headroom too, so a preallocating create cannot overrun an in-flight copy. * seaweed-volume: mirror pending copy headroom across disk probes reserved_space tracks the in-flight copy's charge separately from the cached free space: check_disk_space subtracts it, release removes only the failed copy's charge, a landed copy commits it, and has_room_for_new_volume respects it — matching the Go side. * volume.copy: settle a copy's headroom as its bytes land on disk The pending charge held the copy's full reservation until commit, but written and fallocated bytes already show up in statfs, so the live-space check counted them twice. A slow copy could then block other copies and volume growth despite free space above -volume.minFreeSpace. Settle the charge instead: commit the bytes CreateVolumeFile actually reserved (a failed fallocate leaves nothing committed), then settle each file's write progress as it lands. The failure paths now release only the unsettled remainder; allocatedSize is exported for the .dat's st_blocks. * seaweed-volume: mirror copy headroom settling as bytes land reserved_space keeps only the promised bytes statfs cannot see yet: preallocate_dat settles whatever blocks the filesystem actually reserved (0 when fallocate cannot), each copied file's progress settles the rest as it lands, and a failed copy releases only the unsettled remainder — matching the Go side. * volume.copy: release a failed copy's headroom charge only once Both defers fired on the same in-closure error: the inner released the unsettled charge and the outer released it again, driving ReservedSpace negative (an atomic.Uint64 underflow) and inflating AvailableSpace — the location would then fail every later headroom check, or worse, pass them with fabricated space. Zero the charge once it is released; the Rust mirror already releases via swap(0) for the same reason. * volume.copy: write into the preallocated .dat instead of staging over it The .dat stream staged into .copying and renamed over the file CreateVolumeFile had just fallocated, so the copy's transient footprint was preallocate+dat — more than the reservation the pick just checked — and the reservation itself was discarded at rename. Writing in place reuses the fallocated extent, keeps the peak at the reserved maximum, and leaves the copied volume preallocated like a created one. * seaweed-volume: mirror the in-place write into the preallocated .dat The .dat copy opened the just-fallocated file with truncate, freeing its blocks at open and spending the headroom again as the stream re-grew it. Append mode preserves them, matching the Go side. --------- Co-authored-by: chrislusf <chrislusf@gmail.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
2e7e691bed |
volume: drop stale index entries that resolve to a different needle (#11948)
* volume: drop index entries that resolve to a different needle An index row whose offset lands on a needle with a different id and a different size cannot have been produced by any write path: every appender indexes a key at the offset and size of the record it wrote, and WriteNeedleBlob keeps its indexed size consistent with the blob header. Such rows are corruption (for example an offset replayed with stale high bytes) and until now made the volume reject every write for the key with "mismatching cookie" while reads resolved to the wrong needle. When the duplicate check finds the indexed record belongs to a different key and size, drop the stale entry and let the append re-index the key. Same-key conflicts still fail the cookie check, and entries staying consistent with a re-keyed blob keep their existing semantics. * volume: verify the indexed needle header before streaming reads The buffered read path re-parses the record and rejects a needle whose stored size disagrees with the index, but the streamed path serves data bytes without ever looking at the header, so a stale index row would stream another needle's payload. Probe the header once up front and report a miss when the record is neither the requested key nor the indexed size, the same staleness test the write path applies. * rust volume: drop index entries that resolve to a different needle Mirror of the Go volume change: when the duplicate check finds the indexed record belongs to a different key and size, delete the stale entry and let the append re-index the key. * volume: overwrite a stale index entry on commit, not up front Deleting the stale row during the duplicate check published a tombstone before the replacement append and, for durable writes, before its fsync. A failed append or sync then rolled back the data but left the key deleted. The publish step already overwrites any older offset, so leave the entry in place and let it be replaced once the record commits. * volume: report streaming header-probe failures to the io error tracker A failed header read returned before the tracked body-read loop, so repeated EIO on the probe never reached checkReadWriteError. Skip io.EOF the way the loop does: it is a truncated record, not a disk fault. * rust volume: overwrite a stale index entry on commit, not up front Mirror of the Go volume change: the staged delete published a tombstone before the replacement append and its .dat flush, so a failed append or sync left the key deleted. publish_write already overwrites any older offset once the record commits. * rust volume: report needle meta read failures to the io error tracker read_meta is the header probe ahead of a streamed body: an EIO there escaped check_read_write_error entirely, so a failing disk could not trip the tolerance. Report the outcome the way read_full already does. * volume: retry the wrapped offset before judging an index entry stale A 4-byte index value can index past the 32 GiB wrap: the low offset may hold another needle while the real record sits at +MaxPossibleVolumeSize, which is where reads already retry on a size mismatch. Skipping the cookie check on the low-offset record alone would accept a foreign-cookie write over a live key. Retry the wrapped header read first, the same condition the read path uses; a wrapped read that fails for reasons other than EOF stops the write like the low-offset read does. * volume: probe the streamed needle header under the access lock With HasSlowRead the access lock is dropped after the index lookup, so a compaction commit could swap the backend between the lookup and the header probe and fail a valid download on the old offset. Take the read lock for the probe and resolve the compaction revision first, the way the streaming loop re-resolves it on every page. * rust volume: retry the wrapped offset before judging an index entry stale Mirror of the Go volume change: probe offset + MAX_POSSIBLE_VOLUME_SIZE under the 4-byte-offset build before treating a mismatched record as stale, so a live wrapped entry still gets its cookie checked. * rust volume: do not clear the io error streak on a successful meta read read_meta runs before the payload read in read_needle_for_get, so recording success there reset the streak before every failing download and the volume could never reach quarantine. Report failures only; read_full still records success after a complete read. * volume: record a meta-only read's success against the io error streak A meta read no longer clears the streak because a payload read still follows on GETs, but requests the meta read ends outright — HEAD answers and cookie mismatches — never reached the body path, so their success left earlier failures counted. Note the success on those terminal paths; mirrored in the Rust volume server. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: report streamed reads to the io error streak Streamed GETs never touched the tracker: a failing chunk read went uncounted and a completed download never cleared the streak, unlike readNeedleDataInto which does both. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: reset the io error streak only at terminal meta reads A streamed GET's metadata read runs before its payload read, so clearing the streak there erased the count before every failing download and the volume could never reach quarantine. Record the meta-read success only where the request actually ends: must-meta-only reads, HEAD responses, and cookie mismatches — matching the Rust volume server's placement. * rust volume: record the streamed read's success at the final frame Hyper can complete a Content-Length response without polling the body again, so a success recorded only when the stream is next polled could be skipped, leaving an earlier io error streak in place after a successful download. * volume: record a meta-only read's success on conditional 304 replies * rust volume: record a meta-only read's success on conditional 304 replies * volume: reset the io error streak only after the checksum passes A streamed read that ends in a CRC mismatch still cleared the EIO streak, so corruption could mask progress toward quarantine. The Rust side records success only after the final frame's checksum verifies; move the reset below the CRC check to match. * volume: satisfy codespell in the meta-read comment * rust volume: report range reads to the io error streak * rust volume: cover the range read's io error reporting --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
867a537802 |
volume.configure: keep the superblock in sync with the .vif (#11930)
* volume.configure: keep the superblock in sync with the .vif ConfigureVolume only rewrote the .vif's replication field, leaving the .dat superblock stale. A mounted volume also kept reporting the old placement in heartbeats, and its next compaction would rewrite the stale placement into a fresh .dat. Parse the placement once, then update the live volume when mounted: the in-memory .vif copy and superblock, and the .dat header through the volume's own backend under the data-file locks. For unmounted volumes rewrite the header directly; remote-only volumes skip it since the .vif carries the placement. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume.configure: update the .dat superblock in the Rust volume server Mirror the Go ConfigureVolume change: after rewriting the .vif, rewrite the replica placement in the .dat header too, so a remounted volume's superblock, heartbeats, and later compactions agree with the .vif. A missing .dat (remote-only volume) is skipped since the .vif carries the placement. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Junker der ProvinzandDan Foster
|
e645449872 |
fix(storage): count preallocated space when auto-sizing max volumes (#11919)
The max volume count subtracted each volume's remaining room from the free disk space, except when the master had preallocation turned on, in which case it subtracted nothing. That is wrong for a disk that mixes preallocated and plain volumes, and on platforms where preallocation is not supported. Use the larger of the .dat file's size and its allocated blocks, in the Go and the Rust volume server, so a preallocated volume counts as full and a plain one keeps its remaining room reserved. Fixes #5778 Co-authored-by: Dan Foster <4570306+danfoster@users.noreply.github.com> |
||
|
|
9c427b265f |
volume tier: accept deletes on a volume as soon as it is tiered (#11894)
* volume tier: accept deletes on a volume as soon as it is tiered
volume.tier.upload marks the volume read-only without deletes before the
upload (markVolumeReplicasWritable(..., false, false), i.e. noWriteOrDelete),
and nothing clears that until the volume server restarts, when load() gives
every tiered volume noWriteCanDelete. Every delete in between is refused with
"volume N is read only": the needle stays live in the index, and its bytes in
the remote object, until someone notices.
After the remote file is loaded, the upload handler now gives the volume the
flags a restart would (MarkTieredDeletable: writes refused, deletes accepted),
with two exceptions that keep the current behaviour:
- a replicated volume: the shell copies the uploaded replica's .idx and .vif
to the other replicas after the upload, and a tombstone written before that
copy would reach one replica only;
- a volume in I/O quarantine: that flag is not the upload's to lift.
Tests: TestTierUploadAcceptsDeletesOnASingleCopyVolume fails without the
change ("volume 1 is read only") and checks writes stay refused;
TestTierUploadKeepsRefusingDeletesOnAReplicatedVolume; TestMarkTieredDeletable
(quarantine kept).
* volume tier: check the I/O quarantine under noWriteLock
MarkTieredDeletable read the quarantine state before taking noWriteLock.
CollectHeartbeat could quarantine the volume and set noWriteOrDelete in
between, and the upload then cleared that refusal, so a quarantined volume
accepted deletes until the next heartbeat. Take noWriteLock first: the
heartbeat marks the quarantine before it takes noWriteLock, so either the
check sees it or the heartbeat's refusal lands afterwards and stays.
Also leave alone a volume past IoErrorTolerance that the next heartbeat
will quarantine, as the heartbeat itself does.
* volume: read noWriteOrDelete under noWriteLock in DeleteVolumeNeedle
DeleteVolumeNeedle checked v.noWriteOrDelete without the lock that
MarkVolumeReadonly, CollectHeartbeat and now MarkTieredDeletable hold when
they write it, so a delete racing a tier upload was a data race. Copy the
flag under the read lock; the delete path is otherwise unchanged.
* volume tier: mirror the delete acceptance rules in the Rust volume server
The Rust tier upload already promoted a freshly tiered volume to
no-write-can-delete via refresh_remote_write_mode, and did so for every
volume -- including the replicated and I/O-quarantined ones the Go
MarkTieredDeletable deliberately refuses. A live tier-up now goes
through mark_tiered_deletable (replicated volumes and quarantines keep
their upload-time mark until the next load), while load() keeps
promoting every tiered volume unconditionally, as Go does.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep a tiered volume delete-only when marked writable
MarkVolumeWritable preserved noWriteCanDelete on a remote-tiered volume
only when it was already set. A tiered volume still holding the
upload-time mark (a replicated volume, or one whose tier upload ran
before MarkTieredDeletable existed) had it clear, so marking writable
published the volume as fully writable while its .dat lives remotely --
writes then failed against a backend that cannot append, after the
master was told the volume is writable again.
load() gives every remote-tiered volume noWriteCanDelete; deriving the
same flag here keeps mark-writable consistent with a restart: "writable"
on a remote volume resolves to can-delete.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep a tiered volume delete-only when marked writable (rust)
set_writable preserved the remote delete-only mode only when the flag was
already set, so a tiered volume holding the upload-time read-only mark
(replicated, or quarantined under mark_tiered_deletable) was published
fully writable with its .dat remote. Derive no_write_can_delete from
has_remote_file() instead, matching load() and the Go MarkVolumeWritable.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume tier: serialize the quarantine check with the flag swap (rust)
A heartbeat refusal landing between should_quarantine() and the store
could be clobbered until the next heartbeat re-asserted it — the same
race the Go side closed by checking under noWriteLock. A
compare-exchange gives the same ordering: the swap fails, and the retry
sees the quarantine mark the heartbeat sets first.
* volume tier: MarkTieredDeletable refuses a non-tiered volume
A stray call on a local single-copy volume would pin it
read-only-can-delete. The only caller promotes right after
LoadRemoteFile, but the guard makes the method safe to reuse; the Rust
mark_tiered_deletable gets the same check, though its caller already
only reaches it under has_remote.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Chris LuandDevin
|
379abbe278 |
volume: repair torn tails on canDelete volumes; reject damaged 1-3 byte records (#11876)
- Go/Rust: run the unindexed .dat tail repair on noWriteCanDelete volumes too — deletes still append tombstone records, so the same crash window applies and they previously degraded to fully read-only. - Go readTailNeedle: reject a positive size of 1-3 for v2+ as corrupted before ReadBytes slices the body's 4-byte data-size field and panics; a torn last record is now cut instead of crashing the load. - Tests: TestVolumeLoadRepairsTailOnCanDeleteVolume and TestVolumeLoadCutsDamagedSmallSizeTailOnCanDeleteVolume (Go), and the Rust mirrors volume_load_repairs_tail_on_can_delete_volume / volume_load_cuts_damaged_small_size_tail. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
ab738bfd33 |
rust volume: repair an unindexed .dat tail at load (#11878)
* rust volume: repair an unindexed .dat tail at load Mirror Go's repairUnindexedDatTail (#11870): a crash between the .dat\nappend and the .idx append leaves a .dat tail no index entry covers,\nwhich sent the volume read-only on the next load and left acknowledged\nneedles unreadable. Before the index loads, re-index the complete\nneedles in the tail (a size 0 record as the tombstone or empty write\nresolveEmptyTailRecords reads it as) and cut only an incomplete last\nrecord. A tail an interrupted append does not explain is left as it is\nand the integrity check quarantines the volume read-only as before. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: tests for the unindexed .dat tail repair Reload volumes through the real load path after dropping .idx entries\nand cutting or damaging the .dat, mirroring Go's\nvolume_checking_tail_repair_test.go: complete unindexed needles are\nre-indexed (deletes as tombstones, empty writes as empty needles), an\nincomplete last record is cut, and a tail an interrupted append does\nnot explain is left untouched with the volume read-only. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: re-index a torn size 0 record as a live empty needle A delete and a write with no data append the identical size 0 record, so a recovered one cannot be told apart from the prior index state alone: the write path accepts an empty overwrite of a live key and indexes it with size 0, while a delete of the same key publishes a tombstone. Inferring a tombstone from prior live state turned a lost index row after an empty overwrite into a deletion -- the object read back not-found although the write was answered. Every re-indexed record is now indexed at its on-disk size, the entry publishWriteRequest writes for an empty write. The other misread is bounded: a recovered delete reads its key back as an empty object rather than not-found, and the size 0 record carries no data to resurrect. Review feedback on the Rust mirror (#11878): the same bug exists here; fixing both sides identically. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: re-index a torn size 0 record as a live empty needle Mirrors the Go fix in this pull request. A delete and a write with no data append the identical size 0 record, so resolve_empty_tail_records could not tell them apart from the prior index state: it treated a zero-size record of a live id as a tombstone, although the write path indexes the same bytes as a live size 0 entry. A crash after an empty overwrite of a live key therefore recovered the object as deleted. Every re-indexed record is now indexed at its on-disk size, the entry publish_write writes for an empty write. A recovered delete reads its key back as an empty object rather than not-found; the size 0 record carries no data to resurrect. Review: greptile-apps[bot] and devin-ai-integration[bot] on #11878. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: repair a .dat tail left past an empty .idx The writable index loader creates the .idx before the first needle append, so a crash after that append leaves a volume whose data nothing indexes. CheckVolumeDataIntegrity skipped an empty index outright, so the volume loaded writable and the surviving first needle stayed invisible while later writes appended past it. A zero-size index now runs the same repair with tailEntryPos -1: the walk starts at the first record past the superblock with no anchor needle and no append timestamp to compare against, gets the same complete-record validation and incomplete-tail policy, and re-verifies through the normal tail check once it wrote entries. A tail the repair cannot explain fails the check, so the volume loads read-only instead of writable on an unaccounted-for .dat. Review feedback on the Rust mirror (#11878): the same bug exists here; fixing both sides identically. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: repair a .dat tail left past an empty .idx Mirrors the Go fix in this pull request. The writable index loader creates the .idx before the first needle append, so a crash after that append leaves a volume whose data nothing indexes. Both the repair and the integrity check skipped an empty index, so the volume loaded writable and the surviving first needle stayed invisible while later writes appended past it. An empty .idx now runs the same repair with no anchor: the walk starts at the first record past the superblock, and a tail it cannot explain fails the load like any other repair failure, since every later check skips an empty index. Review: devin-ai-integration[bot] on #11878. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: sync the .dat before publishing re-indexed .idx rows A power loss after the repaired .idx rows become durable but before the .dat tail records they name are flushed leaves the index pointing past the end of the data file; the next load then keeps the volume read-only until an operator steps in. Sync the .dat first so the rows never outlive the records they reference. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: sync the .dat before publishing re-indexed .idx rows Mirrors the Go fix: a power loss after the repaired .idx rows become durable but before the .dat tail records they name are flushed leaves the index pointing past the end of the data file. Sync the .dat first. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
1cfd9bbb4c |
volume: run the durable sync outside the reader-visible lock (#11863)
* volume: run the durable sync outside the reader-visible lock Reads of a volume waited on that volume's fsync because the write lock was held across Sync()/sync_all. A new per-volume lock (Go dataFileSyncLock, Rust data_file_sync_lock) now serializes every .dat mutation and lifecycle op across append+publish+sync+rollback, while the fsync itself runs with dataFileAccessLock/the io write guard dropped — so readers proceed during a sync, and a failed sync's rollback still cannot clobber another write or a compaction swap. Same change in Go (volume_write.go syncWrite/processBatch, SyncToDisk) and Rust (write_needles_grouped/write_synced_run). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: publish after sync; fix Close/CommitCompact lock order Address review on #11863: - Close took dataFileSyncLock before waiting on isCompactionInProgress while CommitCompact claims the flag then takes the lock — a deadlock. Close now claims the flag before taking the locks (flag -> sync -> access). - Go now publishes needle-map entries only after the .dat sync, matching Rust's ordering: appendWriteRequest/appendDeleteRequest stage the bytes, publishWriteRequest/publishDeleteRequest index them after Sync. A failed sync rolls back bytes no reader ever resolved — the streamed-read offset-invalidation window is gone. processBatch splits runs at repeated needle ids (Rust parity) so dedup/cookie checks stay sequential; the old snapshot/restore rollback machinery is removed. - Rust commit_compact (store-lock caller) takes data_file_sync_lock via try_lock -> WouldBlock before the write lease; commit_compact_held and the tier-down _held paths document/require the caller-held sync lock, and the two vol.instance().write_lock() grpc sites take it first. The swap can no longer land between a durable write's append and publish. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: restore write watermarks when an inline fsync fails The syncWrite rollback truncated the .dat but left lastAppendAtNs, lastWriteAppendAtNs, lastWriteNeedleKey, lastWriteDeleted and lastModifiedTsSeconds describing the rejected write; recovery and compaction could read a watermark for a record that no longer exists. Snapshot and restore them on failure, as processBatchRun already does. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
6936bbdc3e |
rust volume: score EC shard placement with the volume's own ratio (#11862)
The auto-select branches of ReceiveFile (is_ec_volume) and VolumeEcShardsCopy passed the build-default DATA_SHARDS_COUNT into find_ec_shard_target_location, so its free-slot maths scored every volume as 10+4 even when the volume's .vif declares a custom ratio (e.g. 6+2). A wider-ratio volume's shards could be steered onto a disk too full to hold them all, and a narrower one would be refused slots it actually has. Add Store::lookup_ec_data_shards(collection, vid), which answers from the mounted EC volume first and then from the .vif on whichever disk holds it (via read_ec_shard_config_across_dirs, so split -dir/-dir.idx and rebuild-staging layouts are covered), falling back to the default when no metadata exists yet. Mirrors the enterprise fix (seaweed-mono c22fa8a15). |
||
|
Chris Lu
|
4feb30deff |
Federated volume-id bound via heartbeat stamps (#11763) (#11819)
* pb: heartbeat fields for the federated volume-id bound (#11763) Heartbeat carries max_volume_id_seen and last_seen_topology_id up, and HeartbeatResponse echoes max_volume_id and topology_id down, so the allocation bound is replicated to every volume server. * master: echo the volume-id bound and fold reported bounds in (#11763) Every heartbeat response carries the leader current max volume id and topology id; every heartbeat reported max_volume_id_seen is folded into the topology bound via UpAdjustMaxVolumeId. * volume: persist a per-dir vid stamp of the highest id ever seen (#11763) The store keeps max_volume_id_seen + last_seen_topology_id in a .vid_stamp file per volume dir, fed by heartbeat echoes, AllocateVolume, and local volume/EC creation, seeded at boot from stamps and local holdings, and reported in every heartbeat. A rebuilt master recovers the bound from the first surviving veteran report. * volume-rust: mirror the vid stamp and heartbeat echo handling (#11763) Same mechanism as the Go volume server: .vid_stamp per dir, bound folded in from add_location scans and allocate/mount calls, echoed fields applied on heartbeat responses, reported on every heartbeat. * volume: serialize vid stamp writes and fsync the directory (#11763) * volume-rust: serialize vid stamp writes and fsync the directory (#11763) * master: echo the allocation bound to connected servers on change The bound was only sent at connect time or on digest mismatch, so a healthy stream never learned later increases. If the high-id server then went offline and the master lost state, survivors restored a stale bound and the id could be reissued. Resend the heartbeat response whenever the bound or topology id moved. * volume: retry failed vid stamp saves on later updates A transient write failure left the in-memory bound raised but the stamp stale, and repeat updates at the same bound never retried. Track a dirty flag so any later update re-attempts the save. * volume: cover stamp bootstrap for pre-stamp dirs in test * volume-rust: retry failed vid stamp saves, mirroring Go * volume-rust: stamp every dir with the combined bound on add_location A dir whose own stamp sat below the combined bound kept its stale value after fold; losing the higher-stamped disk then restarted the server below ids it had already learned. Rewrite all stamps after each location folds in. * volume: stamp mounted volume and ec shard ids (#11763) * volume-rust: stamp mounted volume ids, mirroring Go (#11763) * volume: retry dirty vid stamp save on unchanged topology id (#11763) * volume-rust: retry dirty vid stamp save on unchanged topology id (#11763) * master: record echoed bound from the sent heartbeat response (#11763) |
||
|
Chris Lu
|
51a4c092c8 |
rust volume: release the store write lock during needle I/O (#11823)
* volume: give Volume interior locks mirroring Go's Volume's mutable state moves behind interior locks matching Go's per-volume lock domains, so the store can hand out shared handles instead of requiring exclusive &mut access: - io: RwLock<VolumeIo> owns the .dat/remote handles, needle map, append cursor, compaction bookkeeping and write watermarks — Go's dataFileAccessLock-adjacent mutable set; helpers take the guard as &mut VolumeIo so nothing re-locks. - super_block and volume_info each get a RwLock (superBlockAccessLock, volumeInfoRWLock); dir_idx gets its own so path building never contends with I/O. - no_write packs Go's noWriteLock pair into AtomicU8; last_modified_ts_seconds and compaction_byte_per_second become atomics. The data-file write lease (dataFileAccessLock) now covers every mutating entry that previously relied on borrow exclusivity: writes, deletes, blob writes, set_writable's flag/index swap, remote-file updates, local-backend swaps, index relocation, replica-placement rewrite, put_needle_index, close and destroy. set_read_only_persist leases only its can_delete idx-attach branch because mark_io_unavailable reaches it under a held lease. commit_compact takes try_write_lock — callers hold the store lock, so blocking would stall every volume behind one slow reader; WouldBlock surfaces as COMMIT_WAITS_FOR_READS for the caller to retry. Lock order is fixed: lease -> io -> volume_info/super_block/dir_idx. * storage: mount volumes as shared Arc<Volume> handles DiskLocation's map owns Arc<Volume> so the store can hand a caller a handle that stays valid across store-guard releases, instead of borrowing &mut Volume under the store write lock. - volumes: HashMap<VolumeId, Arc<Volume>>; find_volume returns a cloned Arc. - find_volume_mut and volumes_mut are gone — every Volume method is &self now, so shared handles cover what exclusive borrows did. - writable_volume_mut -> writable_volume(&self) -> Result<Arc<Volume>> keeps the same admission checks (exists, disk-space-low). - write_volume_needle/write_volume_needles/delete_volume_needle are &self; lifecycle methods that mutate the map or mounted state stay &mut self. * server: run needle I/O with the store guard released The store write lock previously wrapped needle writes and deletes end-to-end: process_batch held it across every grouped fsync, and the HTTP/gRPC fallbacks, tail receiver and fetch-and-write paths did the same. One volume's slow I/O then stalled admission and metadata on every other volume. - write_queue resolves each group's handle via writable_volume under a brief store read, then runs write_needles_grouped after the guard drops. - HTTP write/DELETE fallbacks, the gRPC batch-delete EC recheck + delete, tail receiver and fetch_and_write local writes, blob writes and the quarantine/sync/set_writable/remote-file paths all run under store.read() now that every volume method they reach is &self. - Lifecycle calls that mutate the volume map or mounted state (mount/unmount/delete, EC shard ops, compaction commits, load_new_volumes, heartbeat volume actions) keep store.write(). * test: cross-volume admission proceeds during another volume's write Regression coverage for the store-lock fix: - write_queue's test parks a volume-1 write inside its .dat append via a new needle_write_hook seam (mirroring needle_read_hook), then proves the store still hands out a lifecycle writer and a volume-2 handle — while volume 1's data-file lease still excludes same-volume writers. - tests/store_lock_repro.rs exercises the public API: concurrent read-guarded writes on two volumes, and a stale Arc<Volume> handle failing closed after unmount instead of writing into torn-down files. * test: drop store write guards that shared handles made read-only Test setup sites held store.write() purely to reach find_volume_mut; with Arc<Volume> and &self methods the guard is read-only (clippy readonly_write_lock). The volume lease still serializes the writes inside. * volume: drop a windows-only read_exact_at that shadows the io helper cargo's windows build failed E0255: the free fn copied with the interior-lock conversion collided with the unconditional crate::storage::io::read_exact_at import, which already carries the platform branches internally. * server: release the store read guard before needle I/O The read guard still ran through write_needle_blob_and_index, the tail receiver write, the fetch-and-write local write, and the HTTP write and DELETE fallbacks, so a slow append blocked mount/unmount writers (and, on writer-preferring locks, readers on other volumes). Resolve the Arc<Volume> under a short guard and write through the handle after it drops — the same scoping process_batch already used. * volume: serialize the read-only mark and its .vif save under the lease set_read_only_persist only leased the idx-attach branch, so concurrent mark-readonly and mark-writable calls could interleave flag updates and .vif writes — including two save_vif calls truncating and writing the file at once. The lease now spans the whole flag change and save, with a _held variant for mark_io_unavailable, which reaches this from write paths already holding the lease (Go: markVolumeReadonly runs under dataFileAccessLock). * volume,server: span the tier-down transition under one write lease update_remote_files cleared the remote-only write restriction and released its lease before open_local_dat_backend attached the local file, so a write could slip into the gap — pass check_writable, then fail appending to a missing dat_file. The handler now holds the lease across the reference trim, .vif save, and backend swap (and the rollback re-insert), via _held variants; the store read guard is scoped to the handle lookup. The swap still lands after the .vif trim — the ordering that keeps a crash from pointing the volume at a deleted remote object. * server: run a batch's volume groups on separate scoped threads A single batch can hold writes for several volumes; running the groups sequentially made every later volume wait behind a stalled append on an earlier one — the same head-of-line blocking the store guard release fixed, one level up. Each group now gets a scoped thread, preserving order within a group while independent volumes progress in parallel. * storage: close a volume handle a mount slot replaces With shared handles, inserting a new Volume over an existing map entry left the old handle alive in callers — two Volume instances could then write the same files with separate in-memory indexes. set_volume and create_volume now close() the replaced handle so its writers fail closed, matching what unload_volume already does. * test: key the write hook off a reserved needle id The hook registry is process-global and keyed by needle id, and other parallel tests write NeedleId(1) — an unrelated write could enter the barriers and leave the test's real writer parked forever. * server: wait out readers for vacuum commit without the store guard commit_compact try-locked the write lease because callers held the store write lock, returning COMMIT_WAITS_FOR_READS while a download was in flight — and the master reads that as a failed vacuum, discarding completed compaction work. The commit RPC now resolves the handle under a brief store read and blocks on the volume's lease, matching Go's CommitCompact under dataFileAccessLock: a slow reader stalls only this volume's vacuum. The try-lock path stays for callers that still hold the store lock. * server: scope the remaining store read guards to volume lookup Same class as the write-path fix: the batch-delete cookie check and tombstone, the query read, volume_needle_status, and the local chunk read all kept the read guard through the needle operation, so slow I/O there could still stall store writers. Each now resolves the Arc<Volume> under a short guard; the EC rechecks keep their guards. * seaweed-volume: fail closed on detached volume handles A handle resolved under a brief store read can outlive its mount: an unmount+remount between lookup and lease leaves the Arc pointing at a closed Volume while a new instance owns the files. Mark the handle in close() and reject it under the write lease in commit_compact_held, delete_needle, check_writable, set_read_only_persist and update_remote_files_held; the tier-down handler also checks the flag before deleting a remote object its replacement still reads. * seaweed-volume: drop the store guard before mark-readonly waits on the lease mark_volume_readonly held a store read across set_read_only_persist, which waits out the volume's leased readers; a slow download would stall every lifecycle operation behind the guard. Resolve the handle under a short read, then mark off the guard; a detached handle now maps to not_found. * seaweed-volume: retry batch deletes that raced an EC mount The store guard now drops before delete_needle, so ec.encode can mount shards between the pre-delete recheck and the tombstone; the encode then removes the .dat and the needle resurrects from pre-tombstone shards. Re-check has_ec_volume after each successful delete and return the retriable 503 so the filer requeues onto the EC path. * seaweed-volume: one write-queue worker per volume A stalled batch kept later batches for other volumes queued behind it: the single worker awaited process_batch before receiving again. Go's volume_write.go keeps one batching worker per volume, so give each volume its own FIFO worker spawned lazily, bounded by a global concurrent grouped-I/O semaphore and a global queued-request bound; workers exit only when their channel closes, and submit refuses unknown volume ids before spawning. * seaweed-volume: move the queued-write permit inside the request submit held the global queued-slot permit across the send and the response wait, so a canceled submit freed the slot while the worker still held the needle; repeated cancels could overflow the bound. The request now owns the permit until the worker drops it. * seaweed-volume: retire write workers whose volume left the store worker_for kept a sender per volume forever, so servers that mount and retire volumes accumulated idle workers. A worker now polls while idle and retires once its volume leaves the store: close the channel so racing senders fail fast, drain accepted writes, then release the map slot — a replacement worker can never start while this one still holds queued writes, so same-volume FIFO survives retirement. |
||
|
Chris Lu
|
ab39e3a51b |
Rust volume: stamp ExpireAtSec into the live volume_info on .vif persist (#11813)
The worker fix reads the source's TTL destroy time through ReadVolumeFileStatus, which returns the in-memory volume_info. Go's SaveVolumeInfo stamps volumeInfo.ExpireAtSec before writing; Rust's save_vif wrote it only to the serialized file, so a Rust source kept reporting the stale creation-time value and the EC .vif would carry a wrong expiry. Stamp the live volume_info first, mirroring Go and save_volume_info. |
||
|
Chris LuandDevin
|
5b4a707c43 |
shell: keep volume.tier.compact replicas on the same remote object (#11783)
* shell: keep volume.tier.compact replicas on the same remote object The compact cycle ran on a single replica: it deleted the shared remote object up front and never touched the siblings, leaving their .vif/.idx pointing at a removed object. It also remounted the working replica into a writable local volume mid-operation. Freeze all replicas first, keep the shared object while siblings still reference it, replicate the new metadata to every replica, and let the last tiered replica retire the old object. * storage: surface remote object delete failures from volume destroy Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: keep tier.compact replicas frozen until every copy lands A delete between the source unfreeze and a sibling's index copy could land past the copied snapshot, leaving the replica serving deleted data. Unfreeze all replicas only after every resync completes; a remote volume's readonly mark is inert, so the freeze adds no extra cost. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * storage: abort volume destroy when remote delete fails destroyLocked removed local files even when the backend delete errored, leaving a registered replica with no .dat/.idx and a leaked remote object. Run the remote delete before local cleanup and bail out on failure so the volume stays fully usable and the delete can be retried. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: mirror destroy ordering for remote delete failures Propagate a failed remote-object delete instead of warning past it, and reinsert the volume into the disk location registry so a still-intact replica is not unregistered. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: recover partial tier.compact freezes and report held replicas A failed freeze left already-marked replicas readonly with nothing changed, and later stage failures returned without saying the replicas stay frozen or how to resume. Roll back the partial mark when freezing fails, note in every mid-flight error that replicas remain readonly and rerun is the recovery path, and report how many replicas a partial unfreeze released. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * shell: retire abandoned remote objects during tier.compact resync A resync overwrote each replica's .vif, orphaning its previous remote object; an interrupted run could leave replicas on different keys, and only the last tiered replica's object was deleted. Read every replica's remote file list up front, count holders per object, and delete-before- copy whichever replica is an object's last holder, so shared and stranded objects alike are retired instead of leaked. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: keep collection members registered when destroy fails --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
acb44a68a9 |
fix(volume): skip unnecessary needle data reads on delete (#11711)
* fix(volume): do not read a needle's data to delete it A delete read the needle's whole payload from disk only to discard it. The delete handler and BatchDelete need the needle's cookie and its chunk-manifest flag, which live in the header and trailer, but fetched them with a full needle read, the same call a GET uses. readNeedle now reads the header and trailer only when MustMetaOnly is set, whatever the needle's size, and the delete paths ask for that. The Rust volume server honours must_meta_only the same way for its delete paths. A compressed needle or a chunk manifest still gets the full read, as before, because their callers need the bytes. HEAD is unchanged. It used to set MustMetaOnly, which nothing read; it now asks for AttemptMetaOnly alone, so it keeps reading a needle up to PagedReadLimit whole. On a spinning disk that one read is cheaper than a header read followed by a trailer read. Version 1 needles are always read in full. They have no data size field and no metadata after the data, which ReadNeedleMeta relies on, so a metadata-only read of one failed with "invalid needle meta size". That already happened for version 1 needles above PagedReadLimit read with AttemptMetaOnly. A delete therefore no longer verifies the data CRC of the needle it removes. The cookie check is unchanged. * fix(volume): read transformed images fully on HEAD too A HEAD carrying resize or crop params reports the transformed length, which a metadata read cannot produce. Go and Rust both answered such HEADs from the source header, returning the stored size instead of the cropped or resized one. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
1df165d514 |
fix(volume): keep the TTL clock across a vacuum commit instead of rescanning (#11630)
* fix(volume): keep the TTL clock across a vacuum commit instead of rescanning CommitCompact reloads the swapped files while holding dataFileAccessLock, and for a vacuumed TTL volume that reload re-derived lastModifiedTsSeconds by reading every live needle's append timestamp from the .dat: two random reads per needle, with every read of the volume blocked behind them. The in-memory clock is already current at that point. Every write since the volume loaded moved it, and makeupDiff only replays writes that went through that path. Carry it across the reload instead. This also stops an over-budget scan from falling back to the new .dat's mtime and restarting an expiring volume's TTL at the commit. * volume: carry the append watermark as the TTL clock across a vacuum commit The running append watermark is the clock the reload's recovery scan recomputes, so the commit can keep it directly. Client-supplied needle modified times can run ahead of or behind the append time; keeping lastModifiedTsSeconds itself would let a forged or stale timestamp move expiry through a vacuum, where the scan it replaces used server-side append timestamps. * storage: test that a vacuum commit keeps the append clock A write's client supplied modified time can lie ahead of or behind its append time; the commit must land the TTL clock on the append watermark, the same value the recovery scan would have recomputed. * volume: carry the append watermark as the TTL clock across a vacuum commit Mirrors the Go volume server: the running append watermark is the clock the reload's recovery scan recomputes, so the commit keeps it instead of rescanning live needles under the write lock. * volume: commit carries the last-write append time, not the latest append lastAppendAtNs counts tombstone appends and is reseeded from the .dat tail at every load, so it can sit ahead of the last write -- a delete freshens the commit clock -- or behind it: a restarted vacuumed volume's tail needle is not its newest write, and the commit would move the TTL clock backward into premature expiry. Track lastWriteAppendAtNs instead, bumped only on needle appends and seeded by the recovery scan, so the commit lands the clock on the same live-write maximum the rescan would have recomputed. * volume: rescan at commit when the newest write was deleted lastWriteAppendAtNs can hold a write the index no longer holds, so carrying it extends the TTL clock past what recovery over the compacted index would compute. Remember the key behind the watermark so its tombstone or index rollback can send the reload back through recoverLastModifiedTs, landing on the newest surviving write. * volume: a tombstone retires the write rows beneath it in the last-write scan A needle deleted after the compaction copy leaves its write row followed by a tombstone in the committed index. The reverse scan skipped the tombstone row but then counted the dead write, reseeding the watermark and flag as if it were alive. Track keys whose latest row is a tombstone so their earlier write rows stop counting, and exercise the delete-inside-the-commit-window ordering in the tests. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Joel Town RoadandChris Lu
|
23893eb378 |
volume: return error instead of panicking when .dat open fails (#11619)
* volume: return error instead of panicking when .dat open fails When backend.OpenVolumeFile returns an error (e.g. disk below -minFreeSpace), dataFile is nil. Calling backend.NewDiskFile(nil) immediately after caused a nil-pointer panic in f.Stat()/f.Name(). Move the existing error check to run right after OpenVolumeFile, before NewDiskFile is called, so the error is returned cleanly. Fixes seaweedfs/seaweedfs#11615 * volume: share .dat load error handling Extract datFileLoadError helper so open and create paths share one check. * volume: trim dat-open-fail test comments * volume(rust): cover unopenable .dat load path --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
10b0f2b8ad |
volume server: refuse the rest of a grouped run after a durable index failure (#11576)
* volume server: refuse the rest of a grouped run after a durable index failure A durable write whose needle-map put fails stops the volume taking writes (#10825): sent on its own, the next write then fails read only before it appends. The grouped run from #11543 appends and syncs every entry before publishing any, then kept publishing the entries after the failed one and acked them once the shared .idx sync went through. When the failed put tore its .idx row, the rows appended after it land off alignment, so the next load parses them as garbage and the acked writes are gone. Once a durable entry fails to publish, refuse every later entry of the run with ReadOnly, as the per-needle path does. The entries before it stay acked; their rows go down with the run's one .idx sync. The refused records stay on the .dat unindexed, as the failed one does on its own. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: refuse a grouped entry staged as a cookie mismatch too After a durable entry in a grouped run fails to index, the entries after it are refused as they would be on their own. On its own an entry meets check_writable before its cookie check, so one staged as a cookie mismatch now gets the refusal too, instead of keeping its staging error. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: trim a torn .idx row back so the next stays aligned A failed write_index_entry can leave half a row in the .idx. With the writer appending at the tail, every row written after it lands off alignment and the next load parses them as garbage, so a write acked behind a torn row does not come back. Trim the file back to idx_file_offset on a failed append, in both needle maps, and cover it with a test that writes past a torn row and reloads. * volume server: refuse queued Go writes once a durable index update fails processBatch kept writing after a failed nm.Put, and the single-write path checked IsReadOnly only outside the volume lock. A durable write whose index update fails now marks the volume noWriteOrDelete, and each queued request is checked before it appends, so the ones after a failed durable entry are refused the way a lone write is. Deletes get the same noWriteOrDelete refusal a lone delete gets. * volume server: refuse appends while a torn .idx row cannot be trimmed When trimming back a half-written .idx row itself fails, the next append would land after the torn bytes and every later row would parse off alignment on load. Latch the map as torn and refuse appends until the trim succeeds, on both CompactNeedleMap and RedbNeedleMap; the same latch covers an orphan row that could not be trimmed after a failed redb commit. The .idx writer is now opened with write+append access so truncate_to (set_len) works on Windows, where an append-only handle cannot trim. * volume server: write .idx rows at idx_file_offset, not via append mode Rust's OpenOptions on Windows strips FILE_WRITE_DATA whenever append is set so the handle stays strictly append-only, which makes set_len fail - the torn-row trim could never succeed there. Open the .idx writer with plain write access and seek to idx_file_offset before each row, the same positioned-write model the Go server uses. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
07da302da0 |
volume server: ec.decode verifies, cleans up and compacts like Go, off the runtime (#11547)
* volume server: ec.decode reads the .ecx from the index dir it was copied to VolumeEcShardsCopy writes the .ecx/.ecj into the receiver's -dir.idx, so with a split data/index dir the decode target has no .ecx beside its shards. VolumeEcShardsToVolume sized the .dat from the right .ecx but built the .idx from the data dir, failing with NotFound after the .dat was already published. It now reads .ecx/.ecj from where the EC volume opened them and writes the .idx beside the .dat, where Go leaves it. The live-entry check and the .dat size also ignored deletions recorded only in the .ecj, which Go folds into the .ecx (RebuildEcxFile) first: a fully deleted volume was decoded instead of reported as having no live entries, and deleted tail needles were copied into the .dat. Both now treat journaled ids as deleted, without rewriting the sealed .ecx. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: ec.decode keeps the decoded volume writable and reads every .ecj The rebuilt .idx copied a journaled tail needle's .ecx row verbatim after the .dat was cut short before it, so the mount saw a row past EOF and marked the decoded volume read-only. Rows of deleted needles the .dat no longer holds are now dropped, and each journaled needle still in the .dat gets one tombstone instead of one per journal entry. VolumeEcShardsCopy appends journals collected from other holders into the idx dir, but the decode read only the .ecj beside the .ecx, which sits in the data dir when this server generated the shards. It now reads both, once, in bounded chunks via the loader EcVolume uses. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: test ec.decode drops a sealed .ecx tail tombstone Covers the other half of the rule added in the previous commit: a tail needle tombstoned in the .ecx itself (Go's RebuildEcxFile) is cut from the .dat, and its row must not reach the rebuilt .idx either. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: ec.decode runs its file I/O off the async runtime VolumeEcShardsToVolume released the store lock before decoding, but read the .ecx/.ecj, rebuilt the .dat and wrote the .idx inside the async handler, parking a runtime worker for the length of a volume-sized copy. The decode now runs in spawn_blocking on inputs snapshotted under the store lock. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: ec.decode checks the rebuilt .dat is complete Go stats the decoded .dat before writing the .idx (VerifyDecodedDatFile) and fails the decode when it is shorter than the extent the EC index references, since the caller deletes the shards once the call returns. The Rust handler returned success without that check. The rebuild already fails on a short shard read, so this guards the published file itself. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: ec.decode drops the decoded volume's bitrot sidecars Go removes <base>.ecsum and <base>.ecsum.v<N> beside the .dat and beside the .ecx once the .idx is written, so a stale checksum sidecar cannot pass for the protection of a later re-encode. The Rust handler left them in place. Removal is best effort, as in Go. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: ec.decode compacts the decoded volume Go ends VolumeEcShardsToVolume with an offline CompactVolumeFiles, so the decoded volume holds only live needles. The Rust decode left every needle deleted through the .ecj in the .dat, tombstoned in the .idx, until a later vacuum reclaimed it. Store::compact_volume_files loads the unmounted volume, checks free space the way the vacuum does (the estimate now lives in one helper), and runs the vacuum's compact-by-index and commit. As in Go a failed compaction is logged and the decode still succeeds, so the uncompacted .idx rules stay: the tests that pin them now make the compaction fail. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: ec.decode keeps deletes journaled while the .dat is written The decode read the .ecj journals once, before rebuilding the .dat, so a delete that reached the EC volume during the rebuild was left out of the new .idx and the needle came back live. Each journal's read length is now kept, and the bytes appended since are read just before the .idx is written, after waiting out any journal append in flight (appends hold the store write lock), so every delete acknowledged by then is in the .idx. A delete after that point is still lost, as in Go. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Guard overlapping ec decode requests; serialize journal catch-up volume_ec_shards_to_volume runs its decode in spawn_blocking, so a dropped request leaves the job running and a retry would race it on the temporary and final volume files. Claim the vid in a per-server in-flight set until the blocking job finishes, and return Unavailable to an overlapping request. The Go handler has the same exposure and gets the same guard. Journal appends hold the store write lock through their sync-or-truncate, so holding a read lock across the catch-up read guarantees every record it sees is committed: a rolled-back delete can no longer leave a tombstone in the decoded index. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Reconcile the swap when offline compaction commit fails A CommitCompact that fails after the .cpc marker may have renamed .dat but not .idx. cleanup_compact refuses while the marker exists, so the mismatched pair survived until a restart reconciled it — and the decode caller treats the failure as non-fatal. Run reconcileCompactState on commit failure so a decided swap rolls forward and orphan temps are removed before the volume can mount. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Release the decode claim on panic * volume: add ec_decodes_in_flight to the integration-test state literal * volume server: hold the decode tail's lock through compaction The catch_up read released before the rebuilt .idx was written and the volume compacted, so a delete synced to .ecj in that window was durably journaled yet absent from the published index — resurrecting the needle. Rust now holds the store read lock from catch_up through compact, and Go mirrors it by holding the volume's journal lock from the journal- consuming index write through CompactVolumeFiles. * volume server: serialize ec decode's tail per volume, not per store Review follow-ups on the decode path: - Rust: holding the store read lock from journal catch-up through the offline compaction stalled every writer on unrelated volumes for the whole rewrite. The new ec_decode_tail set marks the vid only while its .idx is published and .cpd/.cpx swapped; the two local .ecj append paths (VolumeEcBlobDelete, the distributed delete's local journal) wait on a Notify for that span — Go's per-volume ecjFileAccessLock semantics without the global stall. VolumeMount and the staged-adopt path are also held off while a decode claim is in flight so neither can race the swap. - Rust: the initial journal read ran unlocked, so bytes a rolled-back append later truncated could be folded in as phantom tombstones. The first pass stays unlocked (a slow journal must not stall the store) and a rescan under the quiescing read lock re-reads only committed content; catch_up now rebuilds the id set when a regular journal shrank. - Go: the decode resolved the compaction DiskLocation through FindEcVolume while holding the journal lock, inverting DestroyEcVolume's map->journal order into a deadlock. The lookup now happens first, and DestroyEcVolume/deleteEcVolumeById/DiskLocation.Close destroy outside the map lock. - Go: RebuildEcxFile unlinks .ecj while the volume's ecjFile handle stays open, so later deletes could commit to a detached inode. Both call sites now fold under the journal lock and ReopenDeletionJournal repoints the handle at the live path, working on the volume's resolved .ecx dir (EcIndexBaseFileName) rather than the configured index dir. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume server: fence EC remounts behind the destroy tombstone DestroyEcVolume, deleteEcVolumeById, and the collection-delete sweep now remove the EcVolume from ecVolumes before destroying it off-lock, so a concurrent remount could re-open shard files that the in-flight destroy then unlinks — registering a detached fd. Each destroy records a per-vid tombstone channel in a new ecVolumesDestroying map before dropping the map entry and closes it when Destroy returns. The tombstone intentionally survives as the vid's destroy generation: loadEcShardWithIdxDir compares it before and after opening the shard, so a destroy that both started and finished inside the open window is still detected. A mismatch drops the just-opened shard (releasing its fd and mount gauge) and retries after the destroy completes; a successful mount clears the stale tombstone. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume server: rescan the .ecj under the store lock only after a rollback The decode's second journal pass ran a full rescan under the store read lock on every decode, stalling unrelated writers for the length of the scan. Bump a process-wide epoch whenever a failed append truncates its uncommitted tail; an unchanged epoch between the unlocked read and the quiesced pass proves every id folded in was committed, so catch_up() suffices. catch_up() also treats a journal that was read but has since disappeared as shrunk to zero, so its earlier ids cannot linger. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume server: check the decode tail under the store write lock on delete A blob delete waited for the publishing tail before taking the store write lock, so a decode that claimed the tail while the delete was parked behind the decoder's read lock could still see the journal append land after the rebuilt .idx — an acknowledged delete the mount would miss. Test tail membership under the write lock instead, retrying after the wait; journal_delete_local reports WouldBlock for the same recheck on the distributed path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume server: claim the vid for mount and staged adoption, per volume VolumeMount and the staged .copying adoption held the ec_decodes_in_flight set lock through slow file renames and mounts, stalling every unrelated volume's decode, mount, and adoption. Take the per-volume claim instead — the same exclusion against a racing decode for this vid, released when the call returns. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume server: fail the decode when a compaction commit marker survives CompactVolumeFiles' caller logged a compaction error and went on to delete the EC shards. When the commit marker (.cpc) is still on disk the .dat/.idx swap was decided but could not be reconciled, so the mounted pair may be mismatched — report the failure instead so the shards are kept and the caller can retry. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume server: gate the parked-delete test on the held write lock The releaser thread and the spawned delete raced for the store write lock; on a slow runner the delete could acquire it first and commit before the tail was ever claimed, failing !delete.is_finished() on the Windows unit-test job. Spawn the delete only after the thread reports the lock held. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
0ff7794c54 |
volume: compact an oversized .ecj at mount, safely (Rust + Go) (#11555)
* volume: compact an oversized .ecj at mount, safely (Rust + Go) Restore the mount-time compaction dropped from #11408, Rust + Go parity. A journal already bloated by repeated shard copies is folded down to the id set it encodes. - Trigger after load when file_records > max(threshold, 4x distinct), with a 1 MiB floor so small journals are never rewritten. The set is written to .ecj.compact.tmp + fsync, the handle dropped, renamed, the directory fsynced and the append handle reopened. A failure before the rename keeps the original journal and handle; a failure after it fails the mount. - Go never compacts after a failed journal load; the set would be partial and the rewrite would drop the unread records. - A per-path registry (ecj_registry.rs / ecj_registry.go) counts EcVolume holders and out-of-band writers of each .ecj. Compaction runs only when this volume is the sole holder and no copy is writing; holders and writers wait while one runs. This covers shared -dir.idx journals and cross-disk reconcile, where another EcVolume may hold the same journal. - VolumeEcShardsCopy and EC index recovery register as writers around their .ecj append and partial-file cleanup. - Under the reservation, re-check that the file on disk is still the inode and size that was loaded. - Publish errors are classified where they happen; a failed rename plus a failed restore reports both errors. - Compaction runs after the .vif / bitrot checks, so a refused mount leaves the journal untouched. - The tmp is opened like other volume files, removed at mount if a crash left it, and listed in every EC index cleanup path. Failure paths are tested through the real mount via injectable fs steps (open_with / newEcVolumeWith), plus sibling holders, active copies, changed-after-load, stale tmp cleanup, refused mounts and the Go load-error guard. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: fail the mount when the compacted .ecj's directory cannot be synced The Rust mount synced the journal's directory after renaming the compacted file over it through the crate's best-effort fsync_dir, which returns Ok when the directory cannot be opened. A rename needs only write and search permission, so on a directory without read permission the replacement was published, never synced, and the mount went on taking deletes against it. Sync through a helper that propagates the open error, as Go's util.FsyncDir already does, so that case fails the mount like any other post-rename sync failure. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: test the no-compaction-after-failed-load rule through the Go mount The test for it handed compactEcjAfterLoad an artificial error on a volume that had loaded cleanly, so it would not notice NewEcVolume dropping the real load error on the way to compaction. Make the journal read one of the injectable ecjFsOps steps and fail it inside the real mount, after the first chunk, on a journal whose last entry is an id the first chunk does not hold. The mount must leave the file byte for byte as it was; a clean remount then compacts and keeps that id. The Rust mount fails outright on a load error, so it has no equivalent path. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: register ReceiveFile's .ecj writes with the journal registry ReceiveFile refuses a mounted EC volume only once, when the info message arrives, then creates the .ecj and streams chunks into it. A volume that mounted on that journal mid-stream could find a bloated prefix, pass the inode-and-size re-check and rename a compacted file over it; the rest of the stream then went to the unlinked inode and was lost. Register the path as a writer before the file is created, in both the Go and Rust handlers, and hold it until the file is closed and any partial copy removed, as the shard-copy and index-recovery appends already do. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: skip .ecj compaction when a writer ran since the journal was loaded Compaction checked only that no writer was active at the reservation, and that the file was still the loaded inode at the loaded size. A ReceiveFile truncates and refills the journal in place, so one that ran during the mount's load, or after it, and finished before the reservation could leave different ids at the same length; compaction then wrote the stale set over them. Give each path a write generation that every writer bumps as it starts. A holder records it, and whether a writer was active, when it registers, which is before it opens and loads the journal. It may compact only if no writer was active then and the generation has not moved. Same rule in Go and Rust; the journal read becomes an injectable step in Rust as it is in Go, so both test the in-place rewrite through the real mount. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: match the ReadOnly(VolumeId) variant in write_volume_needles #11543 matched VolumeError::ReadOnly as a unit variant in Store::write_volume_needles, and #11544 changed it to ReadOnly(VolumeId) in the same merge window. Each passed CI on its own, but master no longer compiles the Rust volume server. Carry the volume id through. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
fa77cde7da |
vacuum: check compaction space against live bytes, not volume size (#11524)
* vacuum: size the compaction space check by live bytes, not volume size ensureCompactVolumeSpace required the volume's current .dat and .idx size as free space before compacting. That is the size of the garbage, not of what compaction writes, so on a disk that filled up until its volumes went read-only every compaction was refused, including all-garbage volumes that would compact to a superblock and an empty index. The sweep then retried every volume each cycle and reclaimed nothing (issue #11516). Estimate the output from what the needle map already tracks: live content bytes plus a per-needle framing upper bound behind a superblock, and one index entry per live needle. The estimate never exceeds the current volume size and preallocate still wins when larger. Volumes whose deleted sizes are unknown (.sdx converted back to .idx) keep the whole volume as the estimate. The disk probe moves behind a package variable so the tests can stand in for a full disk; the tests build real volumes instead of re-implementing the formula. * vacuum: space check reserves the index on top of preallocate, checks a separate index disk Review follow-ups: preallocate only stands in for the new .dat, so the rebuilt index is added on top of it; with separate index directories the data disk is checked for the .cpd and the index disk for the .cpx; and the estimates carry 1/16 headroom because counters rebuilt from an index file pass through a Bloom filter with a 0.1% false positive rate. Neither estimate exceeds the current file. * vacuum: split the space check by filesystem, not by directory name Two directories can sit on one filesystem and share its free space, so the data and index estimates are checked separately only when the index directory is on another device; otherwise the sum must fit. Unknown is treated as shared. * vacuum: ask the index directory for its share even when it looks like the same filesystem A volume mounted under the data directory's drive letter on Windows has the same volume name, so the identity check calls it shared. Checking the index directory for the index estimate as well costs one statfs and catches a full index mount either way. * vacuum: identify a Windows volume by its GUID, not its path prefix A volume can be reached through a drive letter and through a folder it is mounted on, so filepath.VolumeName says nothing about the free-space pool. Resolve each directory to its mount point and compare the volume GUIDs; when that fails the two are treated as shared. * vacuum: keep the framing and disk_space_low coverage the rebase displaced * rust volume: split the compaction space check across data and index disks Mirror the Go check: estimate the new .dat and rebuilt .idx separately — live content plus per-needle framing capped at the current file, with preallocate standing in for the data file when larger — and check each directory against its own filesystem's free space. Two directories on one filesystem are asked for the sum. * vacuum: tighten comments on the compaction space check Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
35b090a4df |
volume: merge .ecj as a set union on EC shard copy + index recovery (Rust+Go) (#11554)
* volume: merge .ecj as a set union on EC shard copy + index recovery (Rust+Go) An EC volume's deletion journal is a set of needle ids, but shard copy and index recovery appended the peer's whole journal, doubling the file on every ec_balance round trip. Fold the peer's ids in as a union instead: only ids the local journal lacks are appended. - The journal is never replaced. A mounted EcVolume merges a peer's ids through its live handle under the lock deletes take (Go MergeJournal / Rust merge_journal), wherever its journal lives. - An unmounted journal gets only the missing ids appended while mounts are excluded; the delta is read outside the lock and re-read if the journal changed. - The source .ecj streams into memory as an id set: no staging files, chunked reads, memory proportional to distinct ids. - Go and Rust agree that a source journal exists when it sends a modified time or any bytes. A missing source stays a no-op. - Rust runs every merge in spawn_blocking and shares one receive/merge path between shard copy and index recovery. The decode path and the journal format are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: route .ecj merges to the runtime that holds the journal open Disks sharing one index directory all resolved as the journal's owner, so the last one won and a sibling's mounted runtime was skipped: the merge appended behind its open handle and the sibling kept serving the peer's deleted needles until remount. Callers now name the receiving disk by its data directory; the merge goes through that disk's runtime, else a sibling runtime whose journal is the target file. In Go the unmounted append now holds every disk's EC lock (in location order) while it rechecks for a mount, so a sibling mounting from this disk's index during the unlocked read is merged through instead. In Rust a mount that lands during the read is merged through directly and its added count returned, rather than discarded and reported as zero. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: sync merged .ecj records outside the disks' EC locks The unmounted merge held every disk's EC read lock across its fsync, so a slow sync on one disk held off mounts on all of them, along with the EC reads queued behind those mounts. Mounts only need to be excluded while the records are written: the write now happens under the locks and the fsync after they are released, since a later mount reads the written records from the page cache. A failed fsync rolls back only if nothing has mounted the journal or appended to it since the write. A merge through a mounted volume now keeps only that volume's disk locked across its fsync. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: roll back an unsynced .ecj merge through a volume mounted mid-sync If a volume mounted after the unmounted merge wrote its records but before the fsync failed, the rollback kept the records because the journal was now open, leaving ids in the volume's deleted set that may never reach disk; a retried merge then saw them and synced nothing. The rollback now goes through that volume the way its own failed journal fsync does: truncate back and drop the ids from the in-memory set, so a retry appends and syncs them again. It still keeps the records if the volume journaled since, as truncating would lose that delete. No fsync runs under the disk locks. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: decide .ecj merge rollback from the journal's actual length Two runtimes can hold one journal (cross-disk mounts). The rollback of an unsynced merge checked one runtime's cached ecjFileSize, which another runtime's appends leave stale, so it could truncate a delete that runtime had already synced. The rollback now holds every holder's journal lock and truncates only if the file's actual length is still the append's end, then updates each holder's size and deleted set. Otherwise later records follow the merged ones, so they stay and are rewritten in place and synced outside the locks, rather than left possibly not durable. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: keep unsynced .ecj merge ids out of mounted deleted sets When a merge's fsync failed, later records blocked the rollback, and the rewrite-and-sync failed as well, the merged ids stayed in every mounted volume's deleted set without being shown durable, so a retried merge saw them as present and synced nothing. They now leave those sets while the records stay in the file, matching DeleteNeedleFromEcx, which publishes an id only after its record syncs. The merge returns the error and a retry appends and syncs them again. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: publish merged .ecj ids to every holder of the journal Two runtimes can journal into the same file when disks share an index directory. The merge went through only the first holder, leaving a sibling's in-memory deleted set without the ids, so it could keep serving a needle the peer deleted until it remounted. Every holder of the journal now gets the merged ids, in Go and in the volume server. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Publish merged .ecj ids to the journal actually written mountedEcJournal prefers the receiving disk's own runtime for the vid, whose journal may live in its data directory while the copied records name a sibling's journal in the index directory. Publishing by the requested ecjPath then marked a holder of a different file deleted on records that file never persisted, resurrecting the needles on remount. Publish by the picked runtime's journal path instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
d8f926cf46 |
volume server: stream needle chunks without the store lock (#11488)
* volume server: read GET/HEAD needles off the store lock, and only once The GET/HEAD handler read the needle synchronously on the tokio worker while holding store.read(): first a stream-info read that loaded the whole record just to parse its meta, then, for every needle that was not streamed (small, compressed, chunk manifest, image ops), a second full read. For a tiered volume each read is an S3 GET under the store lock, and a writer queued behind it parks every other store reader. The regular-volume read now runs in spawn_blocking. Under the store guard it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat handle or the remote backend, offset, size); the guard is dropped before any needle data I/O. No data-file lease is held across the read either, since a writer waits for one while holding the store write lock. The index size decides the read, as in Go's readNeedle: a HEAD, a ranged read or a needle above the stream threshold reads only its header and meta tail (ReadNeedleMeta) and hands off to StreamingBody or the range path; everything else is read in full once, with its checksum verified. A compressed or manifest needle found by the meta read is then read in full once. The range-from-source read also moves to spawn_blocking. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: stream needle chunks without the store lock StreamingBody::poll_frame took store.read() and find_volume for every chunk to compare the volume's compaction revision, dup'd the source handle, and allocated a fresh chunk buffer. With -hasSlowRead=false the stream also holds a data-file read lease for its whole life, while a writer waits for that lease under store.write(): the next chunk's store.read() then waits for the writer and the writer for the stream. The per-chunk re-lookup was also wrong. The stream reads a handle opened at plan time, which pins the .dat inode the offset was resolved against; a vacuum commit renames a new file over .dat and leaves that inode untouched. The re-looked-up offset belongs to the new file but was read from the old inode, so a stream whose needle a vacuum moved ended in a checksum error. The pinned offset stays valid, so the check, and with it every store access, is dropped, along with the now unused re_lookup_needle_data_offset and the revision fields of the read plan. The source is shared as an Arc instead of dup'd per chunk, and the chunk buffer is a BytesMut that the blocking read hands back with its result, so its allocation is reclaimed once the previous frame has been written. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: stop a needle stream once its volume becomes unavailable Taking the store lock out of StreamingBody also dropped its per-chunk unavailable_error() check. With -hasSlowRead a writer can take the data-file lease between chunks, fail its fsync and its truncate, and mark the volume unavailable; the stream then kept serving the rest of the needle from its pinned handle. The volume's io_unavailable reason is now an Arc-shared leaf mutex that the read plan hands to the stream. Each chunk checks it under its data-file lease, where the writer marks it, and fails with the same "volume is unavailable: <reason>" error the old check returned. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
8a9563e53d |
volume: test that only the disk holding the replaced replica gets the free-slot credit (#11486)
* volume server: split volume_copy into phases and type the delete-after-status gate volume_copy was one ~400-line handler, and the rule that an existing local replica is deleted only after the source's ReadVolumeFileStatus succeeded was held by statement order alone. The keep_remote_data=true that the pre-copy delete and the failed-copy rollback must share was kept in sync by a comment pointing from one to the other. The handler is now a ~60-line orchestrator over connect_to_copy_source, SourceVolumeStatus::fetch, delete_existing_replica, plan_copy_destination and a VolumeCopyJob whose run() drives preallocate_dat, transfer_files, finish_copied_files and mount_and_reply, with cleanup_failed_copy on error. delete_existing_replica takes a &SourceVolumeStatus, which only fetch can construct (private field in a child module), so the delete cannot be called before the status RPC. Both deletes go through delete_replica_keep_remote. Pure refactor: call order, status codes and messages, cancellation checks, throttling, progress reports and cleanup are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: find space for a VolumeCopy before deleting the replica it replaces VolumeCopy deleted an existing local replica as soon as the source answered ReadVolumeFileStatus and only then looked for a location with room for the copy. With no usable location (disk full, low-disk, wrong disk type) the call errored after the delete, leaving the node with neither the old replica nor the new one. Plan the destination first, as Go does: find_free_location_replacing credits the location holding the replaced volume with that volume's slot, so a disk at its volume limit that holds the replica still accepts the copy. Only then delete the replica and write the .note (still after the delete, as in Go). delete_existing_replica now takes the planned CopyDestination, so the delete cannot precede the plan. find_free_location_predicate keeps its behaviour. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: describe the replace-credit test against the current VolumeCopy flow Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
eeec9ec09a |
volume server: refuse to compact a volume tiered to remote storage (#11545)
* volume server: refuse a tier move while compacting, and a commit once tiered A tier move to remote and a vacuum compaction of the same volume could interleave and leave the volume unreadable: - A compaction committing while the upload ran swapped .dat/.idx under the transfer, which reopens the .dat by path per part. The move then published an object holding the old (or a mixed) layout against the compacted .idx, and with keep_local_dat_file=false deleted the only compacted .dat. - A tier move finishing while the compaction copy ran (or between the copy and the commit) let the commit swap in the compacted .idx while the reload served the pre-compaction remote object through it. The tier move now refuses to start while the volume is compacting, and re-checks the compaction revision under the store write lock before it records the remote file; on a mismatch it deletes the uploaded object and fails with FailedPrecondition, leaving the volume local. Committing a compaction on a volume that has a remote file is refused and its .cpd/.cpx removed, since the reload would read the remote object through the compacted index. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: abort a tier move whose volume was replaced or removed The tier-up bookkeeping looked the volume up by id only and compared the compaction revision. A delete and re-create of the same id during the upload yields a fresh volume at the same revision, so the move recorded the old volume's object on the new one and, without keep_local_dat_file, removed the new .dat. An unmounted volume was skipped and the move reported success, leaving the uploaded object referenced by nothing. Capture the volume instance (its data-file access control Arc, as the scan and read plans do) with the revision, and require both under the store write lock. A replaced volume fails with FailedPrecondition, a missing one with NotFound; either way nothing is recorded and the object is deleted after the lock is released. Go fails in both cases because deleting or unmounting closes the descriptor its copy reads. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: refuse to compact a volume tiered to remote storage Committing a compaction of a tiered volume is refused, since the reload would read the remote object through the compacted index. The compaction itself still started: a tiered volume's data backend is the remote object (the local .dat is dropped or deleted on tier-up), so an explicit vacuum streamed the whole .dat out of remote storage into a .cpd that the commit then discarded. Refuse at the start of the compaction instead, before the .cpd is created, at the point where Go's copy opens the local .dat. The truncated-index test now uses a read-only local volume for its sorted index, since a tiered one no longer reaches the copy. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: match the ReadOnly(VolumeId) variant in write_volume_needles #11543 matched VolumeError::ReadOnly as a unit variant in Store::write_volume_needles, and #11544 changed it to ReadOnly(VolumeId) in the same merge window. Each passed CI on its own, but master no longer compiles the Rust volume server. Carry the volume id through. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
3fd5e4c329 | volume-rust: match the tuple variant VolumeError::ReadOnly in write_volume_needles | ||
|
|
9d3907e36c |
volume: say "volume N is read only" like Go, so filer retries deletes (#11544)
VolumeError::ReadOnly displayed "volume is read-only". Go's store and volume say "volume %d is read only", and the filer's deletion classifier requeues a failed delete only when the error contains "is read only". Against a Rust volume server a BatchDelete on a read-only volume (tier move, maintenance) was booked as a permanent failure and the chunk was never deleted. ReadOnly now carries the volume id and displays Go's text. The text reaches clients through BatchDelete results, the HTTP write and delete error bodies, and gRPC statuses; the gRPC code (FailedPrecondition) and the HTTP/BatchDelete status codes are unchanged. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
bd953b0f84 |
volume server: group-commit fsync writes in the write queue (#11543)
* volume: split the write path into reusable steps do_write_request ran its pre-append checks, the append, the sync rollback, the index publish and the post-write bookkeeping inline, so a batched write could only reuse it one needle at a time. Pull the steps out (check_writable, prepare_write, undo_unsynced_append, publish_write, finish_write) and the store's volume lookup plus disk-space check (writable_volume_mut). do_write_request composes them in the same order with the same early returns; no behaviour change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: group-commit fsync writes in the write queue The write queue holds one store lock for a batch of up to 128 needles but wrote them one at a time, so every fsync needle paid its own .dat sync and its own .idx sync: 2N syncs per batch. Add Volume::write_needles_grouped, after Go's processBatch. A volume's entries are split into runs of distinct needle ids (a repeated id starts a new run, so its dedup and cookie checks see the earlier write). A run with a durable entry appends everything with append_at_ns chained through a local, syncs the .dat once, and only then publishes the entries and syncs the .idx once. A failed .dat sync truncates the .dat back to the run start (marking the volume unavailable if that fails), leaves last_append_at_ns and last_modified untouched, and fails every entry of the run. Runs with no durable entry go through the unchanged per-needle path. Store::write_volume_needles is the queue's entry point; the handlers' non-queue path is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: fail closed when a failed append's rollback fails append_needle discarded the truncate-back result, so a partial write that could not be rolled back left unindexed bytes on the .dat while the volume stayed writable; the next append would bury them mid-file, past the load-time tail check. Route the rollback through undo_unsynced_append, which marks the volume unavailable when the truncate fails, so nothing more is appended over an unverified tail. * volume server: keep a grouped run's I/O error streak from later appends A synced run stages every append before any entry finishes, so the success reset in finish_write ran after the failed appends queued behind the last write to land and erased their media-error streak. Sent one at a time, those errors would have counted and quarantined the volume. Skip the reset when an append after the last landed write added to the streak. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: replay a grouped run's I/O error streak in queue order Skipping the run's success reset whenever an append after the last landed write failed kept the errors from before that write as well, so a run like [EIO, EIO, landed, EIO] reached the quarantine count that the same writes one at a time (one error) do not. Mark the streak where each entry is staged and record the run's success at the last landed write's mark: errors before it are cleared, the ones after it still count. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: replay a run's I/O error streak in one atomic step Reads record their outcomes on the tracker without the volume's write lock, so record_success_at's separate load and store could drop an error a read counted in between, or restore errors a read had just cleared. Keep the count and the clear counter in one atomic word and apply the replay with a single fetch_update. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Eliah RusinandClaude Opus 5.5
|
fea14c01a7 |
volume server: refuse a tier move while compacting, and a commit once tiered (#11539)
* volume server: refuse a tier move while compacting, and a commit once tiered A tier move to remote and a vacuum compaction of the same volume could interleave and leave the volume unreadable: - A compaction committing while the upload ran swapped .dat/.idx under the transfer, which reopens the .dat by path per part. The move then published an object holding the old (or a mixed) layout against the compacted .idx, and with keep_local_dat_file=false deleted the only compacted .dat. - A tier move finishing while the compaction copy ran (or between the copy and the commit) let the commit swap in the compacted .idx while the reload served the pre-compaction remote object through it. The tier move now refuses to start while the volume is compacting, and re-checks the compaction revision under the store write lock before it records the remote file; on a mismatch it deletes the uploaded object and fails with FailedPrecondition, leaving the volume local. Committing a compaction on a volume that has a remote file is refused and its .cpd/.cpx removed, since the reload would read the remote object through the compacted index. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: abort a tier move whose volume was replaced or removed The tier-up bookkeeping looked the volume up by id only and compared the compaction revision. A delete and re-create of the same id during the upload yields a fresh volume at the same revision, so the move recorded the old volume's object on the new one and, without keep_local_dat_file, removed the new .dat. An unmounted volume was skipped and the move reported success, leaving the uploaded object referenced by nothing. Capture the volume instance (its data-file access control Arc, as the scan and read plans do) with the revision, and require both under the store write lock. A replaced volume fails with FailedPrecondition, a missing one with NotFound; either way nothing is recorded and the object is deleted after the lock is released. Go fails in both cases because deleting or unmounting closes the descriptor its copy reads. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Eliah RusinandClaude Opus 5.5
|
f2498e122a |
volume: read EC shards fully, like Go's ReadAt (#11537)
EcVolumeShard::read_at and the scrub plan's EcLocalShard::read_at were a single pread/seek_read. That may legally return fewer bytes than asked mid-file (FUSE/NFS/CIFS mounts, a signal, very large requests), and an Interrupted error was not retried. Callers treat a short count as end of file or corruption: verify_ec_shards compared a zero tail and reported a parity mismatch, local scrub reported a broken shard, VolumeEcShardRead ended the stream early, and decode/rebuild/local needle reads failed. Add storage::io::read_full_at, which loops until the buffer is full or a read returns 0 and retries Interrupted, so a short count means EOF. Route both shard read_at methods through it, replace the encoder's private read_at_most with it, and reuse it for the Windows read_exact_at loop. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Eliah RusinandClaude Opus 5.5
|
62481f1673 |
volume server: ec.decode reads the .ecx from the index dir it was copied to (#11536)
* volume server: ec.decode reads the .ecx from the index dir it was copied to VolumeEcShardsCopy writes the .ecx/.ecj into the receiver's -dir.idx, so with a split data/index dir the decode target has no .ecx beside its shards. VolumeEcShardsToVolume sized the .dat from the right .ecx but built the .idx from the data dir, failing with NotFound after the .dat was already published. It now reads .ecx/.ecj from where the EC volume opened them and writes the .idx beside the .dat, where Go leaves it. The live-entry check and the .dat size also ignored deletions recorded only in the .ecj, which Go folds into the .ecx (RebuildEcxFile) first: a fully deleted volume was decoded instead of reported as having no live entries, and deleted tail needles were copied into the .dat. Both now treat journaled ids as deleted, without rewriting the sealed .ecx. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: ec.decode keeps the decoded volume writable and reads every .ecj The rebuilt .idx copied a journaled tail needle's .ecx row verbatim after the .dat was cut short before it, so the mount saw a row past EOF and marked the decoded volume read-only. Rows of deleted needles the .dat no longer holds are now dropped, and each journaled needle still in the .dat gets one tombstone instead of one per journal entry. VolumeEcShardsCopy appends journals collected from other holders into the idx dir, but the decode read only the .ecj beside the .ecx, which sits in the data dir when this server generated the shards. It now reads both, once, in bounded chunks via the loader EcVolume uses. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: test ec.decode drops a sealed .ecx tail tombstone Covers the other half of the rule added in the previous commit: a tail needle tombstoned in the .ecx itself (Go's RebuildEcxFile) is cut from the .dat, and its row must not reach the rebuilt .idx either. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Eliah RusinandClaude Opus 5.5
|
5ece8dd63c |
volume server: drop the unused unmount result in VolumeCopy validation, and test-only EC helpers (#11531)
mount_and_reply ignored the Result of store.unmount_volume when a copied replica failed record count validation, tripping unused_must_use. The Err branch is unreachable there: the volume was mounted under the same store write guard, mount_volume refuses an already loaded vid so it is a fresh Volume with is_compacting false, and a compaction claim needs &mut Volume, i.e. the store lock. Ignore the result explicitly with a one-line reason. Store::delete_expired_ec_volumes and Store::remove_ec_volume are called only from test modules (the heartbeat uses the split find_expired_ec_volumes / remove_expired_ec_volumes halves), so mark them #[cfg(test)]. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Chris LuandDevin
|
0978e7f833 |
vacuum: keep disk-full read-only volumes reclaimable (#11519)
* storage/topology: keep disk-full read-only volumes vacuumable The vacuum sweep skipped every read-only replica, so a volume that went read-only because its disk filled could never reclaim its garbage — the exact situation compaction exists for. The volume server now reports disk_space_low in VacuumVolumeCheckResponse, and the sweep skips a read-only replica only when the flag is clear. An explicit volumeId vacuum is unaffected: it already bypassed the read-only rule. The field takes number 4: 2 and 3 are downstream-allocated for tombstone retention, keeping the wire merge clean. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * storage: measure vacuum free space against live bytes The pre-compaction space check required the current .dat + .idx size free, which includes the garbage being reclaimed — on a nearly full disk that estimate can never fit, so the volume stayed garbage-bound forever. Measure against the estimated compacted output instead: superblock plus live index entries plus live content bytes, with the existing ten percent buffer unchanged. Mirrors the same check in the Rust volume server. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vacuum: count per-needle framing in the compacted-size estimate The live-bytes estimate covered each live needle's content and index entry but not its .dat framing (header, checksum, timestamp, padding — ~32 bytes on version 3). For small-needle volumes that is more than the 10% headroom, so a disk with space between the estimate and the real output still ran out mid-compaction. Rust side mirrors the same formula. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * storage: report disk_space_low only when it is the sole read-only cause Review feedback (ihnokim, greptile, devin): a volume read-only for low disk space AND an operator mark or I/O quarantine was still eligible for the automatic sweep, rewriting a copy meant to stay protected. The flag now reports only the benign sole-cause case in both servers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: fail closed when the read-only lookup misses in the sweep A heartbeat can drop the volume from the DataNode cache between the location-list copy and VacuumVolumeCheck; a lookup error previously skipped the read-only check entirely. Review feedback (coderabbit). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
43fd5b8d82 |
volume: reclaim staged EC shard generations left by the 2PC switch (#11501)
* volume: remove staged EC generation files on teardown and shard delete The 2PC generation switch stages each run as <base>.ecNN.v<N> plus versioned .ecx/.ecj/.vif files. Nothing on the volume server removes them: isEcDataShardFile only recognises the exact .ecNN name, so the staged files are invisible to every bookkeeping pass, and even full_teardown's wipe-all path left them behind. Each re-encode therefore leaks a full shard set per shard-holding disk. RemoveEcGenerationFiles sweeps <base>.ec*.v<N> and <base>.vif.v<N>, optionally keeping generations at or above a threshold; teardown and the reconcile wipe remove every generation, and a per-shard delete removes that shard's staged generations too. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: delete staged EC generations older than N via VolumeEcShardsDelete After a 2PC generation switch commits, the superseded generation's <base>.*.v<N> files sit on disk with no cleanup path: teardown removes everything, and a per-shard delete only touches the named shards, so the executor had no RPC that reclaims just the staged leftovers. delete_generations_older_than removes staged generation files strictly below the threshold on every disk. Versioned files are never mounted, so nothing is unloaded first; the committed generation and the canonical files are preserved. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: mirror staged EC generation cleanup Parity with the Go volume server: remove_ec_generation_files sweeps <base>.ec*.v<N> and <base>.vif.v<N> staged by the 2PC switch, called by remove_ec_volume_files (which covers both teardown paths) and the new delete_generations_older_than request field; delete_ec_shards removes a shard's staged generations along with the canonical file. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: match staged generation filenames literally filepath.Glob interprets metacharacters in the collection part of the base name, so a collection like a[bc] could match another volume's staged files (or miss its own). Scan the directory and compare names literally instead, mirroring the Rust read_dir implementation. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: report generation-sweep errors and drop the store lock first - snapshot the location base names under the read lock and run the filesystem sweep after dropping it, so a slow disk cannot stall the store; - record per-entry read_dir errors in remove_ec_generation_files and propagate them from remove_ec_shard_generations instead of flatten() skipping them; - warn when a staged-shard generation fails to delete rather than reporting success with files left behind. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: fail shard delete when the staged-generation listing fails A transient ReadDir failure fell back to removing canonical shard names only: staged .v<N> files survived while the RPC still reported success, leaving the leak invisible to retrying callers. ENOENT still means the disk simply has no such directory; other listing errors now propagate. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: propagate staged-generation removal failures delete_ec_shards logged remove_ec_shard_generations errors and the RPC returned success while staged .v<N> files remained, diverging from the Go handler which surfaces the failure. The sweep keeps processing the remaining shards, retains the first error, and volume_ec_shards_delete maps it to Status::internal so callers can retry. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: notify state change even when the shard sweep errors delete_ec_shards already deletes and unmounts the shards before returning a staged-generation failure, so returning early skipped volume_state_notify and the master kept routing to them until the next heartbeat. Notify before propagating the error. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
e57f8c4d87 |
volume server: read GET/HEAD needles off the store lock, and only once (#11487)
The GET/HEAD handler read the needle synchronously on the tokio worker while holding store.read(): first a stream-info read that loaded the whole record just to parse its meta, then, for every needle that was not streamed (small, compressed, chunk manifest, image ops), a second full read. For a tiered volume each read is an S3 GET under the store lock, and a writer queued behind it parks every other store reader. The regular-volume read now runs in spawn_blocking. Under the store guard it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat handle or the remote backend, offset, size); the guard is dropped before any needle data I/O. No data-file lease is held across the read either, since a writer waits for one while holding the store write lock. The index size decides the read, as in Go's readNeedle: a HEAD, a ranged read or a needle above the stream threshold reads only its header and meta tail (ReadNeedleMeta) and hands off to StreamingBody or the range path; everything else is read in full once, with its checksum verified. A compressed or manifest needle found by the meta read is then read in full once. The range-from-source read also moves to spawn_blocking. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
hsdfat
|
02353444ac |
fix(volume-rust): reserve a disk before replacing a replica in VolumeCopy, and check record counts (#11483)
* fix(volume-rust): reserve a disk before replacing a replica in VolumeCopy, and check record counts Port of the Go VolumeCopy hardening in #11238 and #11252. - Pick the destination disk before deleting the existing replica, counting the slot that replica holds as free. If no disk qualifies, the healthy replica is kept instead of being deleted. - Read the source's VolumeStatus before and after the copy. When both succeed and the counts did not change, the mounted replica's file and deleted counts must match; on mismatch it is unmounted and its files removed. A failed "before" read skips the check; a failed "after" read fails the copy. * fix(volume-rust): let a departing caller cancel VolumeCopy's post-copy status read Go reads the source's status after the copy with stream.Context(), so the call ends when the caller leaves. The Rust call had no such link: a source that stalled there held the copied, unmounted files after the caller was gone. Race it against the response channel, like the other blocking steps, so the usual error cleanup removes the partial copy. |
||
|
Eliah RusinandClaude Opus 5.5
|
68944e83a3 |
volume: typed tier errors so a missing remote object answers NotFound (#11484)
remote_storage/s3_tier.rs returned Result<_, String> from every
transfer (upload_file, download_file, read_range[_blocking],
delete_file[_blocking]) and from the tier runtime helpers. The tier
move handlers could only wrap that in Status::internal, so a .dat whose
remote object is gone was indistinguishable from an I/O failure to
weed shell.
Add TierError { NotFound, Io, RuntimeUnavailable, Aborted }. Each
variant carries the existing message verbatim. NotFound follows the
rules remote_storage/s3.rs already uses: raw 404 status on HEAD,
NoSuchKey code on GET; a bare 404 on GET stays Io. A progress-callback
Err becomes Aborted. VolumeError gains a transparent Tier variant and
From<VolumeError> for Status maps Tier(NotFound) to NotFound; the tier
move handlers go through status_with_context, so their message text is
unchanged. Every other tier failure is still Internal.
The remote needle read path keeps io::Error::other, so its error kind
and vacuum's handling of it do not change.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
Eliah RusinandClaude Opus 5.5
|
00310f6588 |
volume server: run the vacuum compaction copy without the store lock (#11482)
* volume server: run the vacuum compaction copy without the store lock VacuumVolumeCompact held the store write lock for the whole live-needle copy, including every progress blocking_send on the 16-deep stream. On a large volume that is minutes with every read, write and heartbeat on the node parked behind it, long enough for the master to unregister the node. Split compaction the way Go's CompactByIndex runs it. A short locked step claims the volume's compacting flag, records the makeup_diff watermark (index size and compaction revision) and opens fresh .dat/.idx handles. The copy then replays .idx up to the watermark and copies from those handles with the store lock released; writes that land meanwhile are replayed by makeup_diff at commit, as before. The flag is an Arc<AtomicBool> released when the job is dropped, so every exit path clears it. Because the flag is now visible to other callers, the operations that would pull the files out from under the copy refuse while it is set: unmount (and VolumeConfigure, which unmounts and remounts), delete (checked before the volume is removed from the map, which a refused destroy used to leave unmounted), cleanup, and index relocation. A second compact and a commit stay no-ops, as in Go. The pre-copy fsync is dropped: the copy reads its own handles through the page cache and .cpd/.cpx are fsynced before commit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume: keep a read-only in-memory index's size for the compaction copy The unlocked copy replays .idx up to index_file_size(). A read-only volume whose .sdx could not be built loads its index into memory without a writer, so that size stayed 0: the copy came out empty and the commit replaced the volume with it. CompactNeedleMap::load_from_idx now records the rows it loaded, which is also what Go's IndexFileSize reports for a read-only index. The copy's index replay now stops reading at the recorded size instead of walking rows appended since, which makeup_diff replays anyway. Adds tests for compacting a read-only volume on both the sorted index and the in-memory fallback, and for VolumeConfigure stopping when the unmount is refused during a copy. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: stop a vacuum copy as soon as its client is gone The progress callback only noticed a closed response stream when a report was due, every 128 MiB. With the copy now running outside the store lock, a copy nobody waits for keeps the volume marked compacting and so keeps refusing unmount, delete and cleanup until that next report. Check the stream on every callback. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Eliah RusinandClaude Opus 5.5
|
5c9c424a84 |
volume server: stream ReadAllNeedles without holding the store lock (#11481)
* volume server: stream ReadAllNeedles without holding the store lock read_all_needles held store.read() while Volume::read_all_needles read every live needle of the volume into a Vec, and kept holding it through the whole blocking_send loop. Memory grew with the volume, and a slow client parked the scan in a send with the guard held; needle writes and the heartbeat take store.write() on a writer-preferring lock, so the node stopped serving until the client caught up. Take a DatScanPlan (fresh .dat open, end bound) under a short guard and walk it with the guard released, sending one needle at a time. Each record is checked against the live needle map under a brief read guard, as the scan reaches it, and only a live record is parsed, so a damaged stale copy does not fail the stream. Records appended while a pass ran are walked by a follow-up plan, so a needle overwritten during the scan is streamed once, as its new copy. A vacuum commit or re-create of the volume during the scan fails the stream, since the map's offsets no longer describe the pinned file; the plan carries the volume instance and compaction revision for that check. DatScanPlan::scan_records yields records unparsed; scan keeps its behaviour on top of it. Volume::read_all_needles has no caller left and is removed; its tests move to the RPC. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: check ReadAllNeedles liveness only once the send can proceed The per-record liveness check ran before blocking_send, so a scan parked on a full channel held a record it had already judged live. An overwrite landing during that park left the old copy in the stream, and the continuation over appended records then streamed the new copy as well. Reserve channel space first, then take the store read guard, check the record against the needle map and enqueue it through the permit before releasing the guard. The wait for space still happens without the lock; the record is parsed before the guard is taken, and its parse error only counts if the record turns out to be live. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |