Commit Graph
85 Commits
Author SHA1 Message Date
8ba33fae71 rust volume: bulk-load needle maps on redb 4.6 and compact the old floor (#12695)
* seaweed-volume: expect a fresh redb file to start at 72 KiB

The assertion fails on redb 4.2.0, whose empty database is about 1 MiB.

* seaweed-volume: bulk-load redb rebuilds with the stable cursor

redb 4.6.0 has no experimental_cursor feature. Full rebuilds insert
sorted needle ids through insert_before, and a new .rdb starts at 72 KiB.

* seaweed-volume: test which redb files are still at the old floor

The gate is missing, so the test does not compile yet.

* seaweed-volume: decide when a reused redb is still at the old floor

Compact only a file above 72 KiB and at most 1 MiB plus 8 KiB, and only
when floor_compact has not been recorded.

* seaweed-volume: test one-shot compact of an old-floor redb

Reuse, shrink, and storage-error recovery are specified. The helpers
they call are not implemented, so the tests do not compile yet.

* seaweed-volume: compact a reused redb that is still at the old floor

One compact runs when the file is above 72 KiB and at most 1 MiB plus
8 KiB. A storage error drops the handle and opens the same path again
so the volume does not keep a PreviousIo latch.

* seaweed-volume: publish a redb replay before compacting the old floor

compact() is what would make a Durability::None tail durable, and a
storage error then drops that tail on reopen. An in-band file is
committed first. The floor_compact flag is read only inside the band,
so a meta error cannot rebuild a larger index.

* seaweed-volume: drop the redb experimental cursor CI step

redb 4.6 has no redb-experimental-cursor feature. The needle-map tests
that used it now run in the normal cargo test step.

* seaweed-volume: drop the experimental-api-5 notes on redb reads

redb's experimental-api-5 can no longer be enabled, so these reads always
use the inherent ReadOnlyTable::get.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: drop the always-false flag from the old-floor band check

Both callers pass false and read floor_compact on their own, so the band
check only needs the file length.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 17:00:55 -08:00
80aad76986 rust volume: remove the redb index when a failed VolumeCopy cleans up (#12250)
* rust volume: remove the redb index when a failed VolumeCopy cleans up

A copy that mounts and then fails validation is unmounted, not deleted. With
a redb needle map the mount has already built <idx base>.rdb, and
cleanup_failed_copy removed .dat/.idx/.vif/.note but not the .rdb, so the
index database outlived the copy. A later copy of the same volume to that
server reuses it (RedbNeedleMap::try_reuse_rdb keeps an .rdb whose recorded
idx size is not larger than the new .idx and replays only the tail), which is
wrong once the source has been compacted in between.

Move the partial-file removal into CopyDestination::remove_partial_files and
include the .rdb, next to the .idx where Volume::file_name puts it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* rust volume: clear a stale index before a copy writes, and report cleanup failures

A crashed copy, or one whose partial-file removal failed, leaves a stale
.rdb/.sdx next to the .idx; the next copy's mount would reuse it via
try_reuse_rdb's tail replay. remove_partial_files now also covers .sdx
and returns the first non-NotFound error: callers that can still abort
(the pre-copy clear) do so, while the post-failure cleanup warns so a
surviving index is visible instead of silently poisoning the next copy.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: restore the pre-copy leftover clear

A leftover index check-in accidentally gated remove_partial_files behind
a constant false, so a stale .rdb/.sdx at the destination survived the
copy that was meant to clear it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(volume): expect 409 when a read-only upload is refused

x-seaweed-write-outcome: refused-before-append. The HTTP integration
test still expected 500, so Rust unit tests failed on Linux and Windows.
Deletes stay 500.

* rust volume: keep a shared .vif, and clear a stale index beside the .dat

remove_partial_files deleted <data>.vif even when a sealed .ecx still
shares it, undoing the keep in Volume::destroy. It also cleared the
index only under the index base, so a leftover .idx beside the .dat
still made load trust that directory's .rdb.

Skip .vif while a non-empty .ecx is present, and clear both bases when
the index directory differs from the data directory.

* rust volume: do not delete .dat files in the index directory

The index directory can be another location's data directory. Clearing
.dat and .note there removes that replica. Those files come off the
data base only; .idx, .rdb, and .sdx are still cleared on both.

* rust volume: serialize copies of a volume on one server

A second VolumeCopy of the same volume could pass the existing-volume
snapshot before the first copy mounts, then clear or overwrite the
files the first copy still owns. Hold a per-volume claim for the whole
copy, like ec_decodes_in_flight does for decodes, and fail a duplicate
fast so the caller can retry. Mirrors the Go volumeCopiesInFlight
guard.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: keep files when the mounted-copy delete is refused

When a copy mounted and then failed, deleting through the store can be
refused (e.g. the volume is compacting), leaving it registered. The
cleanup then unlinked its files anyway, stranding a live replica. Keep
the files while the volume is still in the store. Mirrors the Go copy
cleanup.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: init volume_copies_in_flight in the http test state

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: refuse a mount while that volume is being copied

volume_copy claims the vid, snapshots the store, then waits on the
source. A VolumeMount in that window can load the replica, and the
copy's pre-write cleanup then unlinks its files. Check the claim under
the store write lock and return Unavailable.

* rust volume: refuse configure while that volume is being copied

volume_configure unmounts and remounts. A copy that already snapshotted
the volume as absent can then unlink the files that remount loaded.
Check the copy claim under the store write lock and return Unavailable.

* rust volume: preserve files a copy displaces so a failed copy restores them

* rust volume: keep an interrupted copy's .note with the files it marks

displace_existing_files skipped the .note, so a crashed copy's leftovers
were saved without the marker the load scan rejects them by, and
remove_partial_files then deleted it. A retry that failed restored the
partial set marker-free: after a restart the server could load an
unfinished copy as a normal replica and serve wrong data.

Displace the .note like every other file at the basename so the restore
puts back the exact state the copy found.

* rust volume: restore displaced files over a surviving partial

restore_displaced_files dropped the backup whenever the basename was
still occupied. When the partial cleanup could not remove every file the
copy landed, an existing path just meant the removal failed — deleting
the saved file destroyed the old replica and let the surviving partial
pair with restored siblings of another generation.

Clear the leftover first so the displaced file goes back, and keep the
backup when it cannot: the operator can recover from copying/ rather
than lose the replica outright.

* rust volume: back up an EC-shared .vif the copy truncates in place

A .vif a sealed .ecx shares was left out of the displaced set so the EC
volume keeps it — but the incoming copy opens the same path with
truncate, so a failed copy left the shared .vif overwritten by the
rejected copy's settings with no way back. Go keeps the original via a
hard link because its staged file swaps in by rename; truncating the
shared inode makes a link useless here.

Copy the content into copying/replaced_ like the other backups instead:
the file stays in place for the EC volume during the copy, and the
restore puts the original settings back on failure.

* rust volume: serialize reload loads with the copy claim

A SIGHUP reload scans every volume basename and sweeps .note-marked files
as interrupted copies — including one a live VolumeCopy is still writing.
Mirror Go's loadGuard: the copy-claim set is shared with each disk
location, the scan skips a claimed vid outright and claims any other
through its load, and reconcile_compact_states takes the same skip before
rolling a leftover commit over the copy's files.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: hold the load claim through open_volumes

The scan released each claim when its iteration ended, before the queued
volume's index read ran in open_volumes. A VolumeCopy taking the vid in
that gap could displace or remove files out from under Volume::new — the
register the guard exists to prevent. Queued claims now live until the
open pass returns; a second collection of an already-claimed vid merges
into the entry instead of re-claiming.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: refuse allocation while that volume is being copied

AllocateVolume registered a volume while a copy of it held the
destination: create_volume loads the index before registration, and the
copy then clears or rewrites files the live replica already opened,
leaving it mounted on a stale index — the window VolumeMount and
VolumeConfigure already refuse.

Claim the volume id like the Go AllocateVolume does, so registration
either wins the claim before the copy starts or sees it and refuses
with Unavailable.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: refuse staged adoption while that volume is being copied

The from_staged branch of volume_ec_shards_to_volume renames .copying
files into place and mounts, all while a copy of the same vid could be
rewriting the destination — the same registration window VolumeMount
refuses. Hold the copy claim for the whole adoption, as Go's
adoptStagedVolume does.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: drop a partial .note when its write aborts the copy

A write_note failure after displace_existing_files restores the saved replica but leaves the empty or partial marker it just made; the next load scan then deletes the restored replica as an unfinished copy. Remove the marker before restoring, as the Go side's error defer does.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
2026-10-30 11:21:38 -07:00
43a312e309 rust volume: keep a copied replica on deleted-count drift, as Go does (#12251)
* rust volume: keep a copied replica on deleted-count drift, as Go does

check_copy_counts rejected a VolumeCopy whenever the target's deleted count
differed from the source's. On a redb needle map that count is an estimate:
metrics_from_idx rebuilds it at load through a bloom filter (0.1% false
positives per entry, each counted as an overwrite), and the source holds the
estimate from its own last load plus exact runtime updates. A source written
since its load and a fresh load of the same files disagree by a few, so good
copies were refused with "target deleted count [N+1] is not same as origin
deleted count [N]". An index of unique keys 1..=128 already reports one
deletion.

Go fixed the same rejection in #11701 (issue #11678) by warning instead of
failing. Do the same for the deleted count. The file count stays strict: it
is exact on both sides (every non-tombstone .idx row is one file), and the
.dat/.idx size checks still gate integrity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* test(volume): expect 409 when a read-only upload is refused

x-seaweed-write-outcome: refused-before-append. The HTTP integration
test still expected 500, so Rust unit tests failed on Linux and Windows.
Deletes stay 500.

* rust volume: keep the strict count check when only deletes drift

Deletes append tombstone rows without moving the file count, so a copy whose source only deleted mid-flight still has an exactly comparable file count. Gate the strict check on file-count stability alone; the deleted-count drift is already a warning inside check_copy_counts.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-30 11:20:33 -07:00
3c6700c310 volume balance: stop planning moves onto disk types a server does not have (#11762)
* fix(topology): capacity queries no longer create empty disk-type entries

AvailableSpaceFor and CapacityFor called getOrCreateDisk, so asking a
node whether it had room for a disk type it lacks created a zeroed
DiskUsageCounts entry for that type. Volume growth walks every data
center, rack and data node, and ToDataNodeInfo publishes diskUsages, so
every node ended up advertising an empty disk (MaxVolumeCount 0,
DiskId 0) of every type ever grown: HDD-only servers showed an "ssd"
disk and SSD-only servers an "hdd" disk.

Read the counters through a non-creating getDisk; a missing entry
answers 0, the same numbers as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* fix(admin): don't let an empty disk-type entry shadow a node's real disk

UpdateTopology keys disks by node and disk id. The master published an
empty entry (no capacity, nothing on it, DiskId 0) for every disk type a
capacity check had asked about, and it collided with the node's real
disk 0; map iteration order decided which survived. An SSD-only node
intermittently showed an empty HDD disk, and an HDD node lost its disk
-- and with it its volumes from the volume index, hence "Could not find
volume N on source server" in balance detection.

IsShadowedEmptyDiskEntry matches an empty entry whose disk id another
DiskInfo of the node, one with capacity or data, already uses.
UpdateTopology and CountTopologyResources skip it. A lone empty entry
is still an empty disk of unknown capacity and is kept, as
GetDisksWithEffectiveCapacity expects.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* fix(balance): don't seed balance candidates from shadowed empty disk entries

detectForDiskType registered every node with a DiskInfos entry of the
type as a zero-volume candidate. An empty entry the master made up for
a type the node lacks made that node the least-loaded destination, and
its zero max switched the whole disk type to raw-count balancing. The
volume server then refused every move: "no space left hdd" / "no space
left ssd".

Skip entries IsShadowedEmptyDiskEntry matches when seeding candidates.
A server that holds volumes of the type but reports no max still enters
through the metrics and still triggers the raw-count fallback.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* fix(admin,balance): ignore made-up disk-type entries at any disk id

The made-up entry is published with DiskId 0, but it need not collide
with a real disk: when location 0 is unavailable (not reported) the
node's usable disk is location 1. IsShadowedEmptyDiskEntry required a
collision, so such an entry survived both UpdateTopology -- where the
unknown-capacity fallback offers it to planners -- and balance
detection, which planned moves onto it again.

Its disk id cannot tell it apart: a DiskInfo with no volumes, no EC
shards and no per-disk max is published with DiskId 0 whichever disk
it stands for. IsPhantomDiskTypeEntry (renamed, since nothing needs to
be shadowed now) matches an empty entry on any node that has a disk of
another type with capacity or data. A real disk looks the same only
while it offers no volume slot, when it can take no volume either. A
node whose entries are all empty keeps them as disks of unknown
capacity, as before.

Reported by review on #11762.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* test(balance): assert where the real disks land in the empty-entry test

Check through GetNodeDisks that each server's real disk sits at the
subtest's disk id before planning. The id-1 case already landed on
physical disk 1 (SplitByPhysicalDisk folds all-zero record ids onto the
entry's DiskId), so this only makes the setup explicit; on the first
revision of the fix it shows the real disk at 1 beside the made-up
entry at 0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* fix(topology,balance,ec): don't publish or trust usage-only disk-type entries

ToDataNodeInfo started from the raw disk-usage counters, so a node that
was asked for capacity of a disk type it lacks kept advertising an empty
DiskInfo for that type even though no such disk was ever reported. Skip
usage entries that have no disk child and hold all-zero counters.

The same stale entries still misled planners that read DiskInfos
directly: the balance plugin's move precondition treated a phantom entry
as proof a target has that disk type, and the EC balancer counted it as
a matching disk. Filter them with IsPhantomDiskTypeEntry.

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-16 23:41:26 -07:00
Eliah RusinandChris Lu 924a683397 volume server: answer chunk-manifest reads from the sum of chunk sizes (#11628)
* volume: answer chunk-manifest HEAD from the chunk-size sum

* volume: size chunk-manifest reads by the sum of chunk sizes

* volume: saturate the sum of chunk-manifest Range lengths

* volume: fetch remote chunk-manifest ranges with a bounded Range

* volume: fmt chunk-manifest handlers

* volume: cap whole-window chunk reads at the visible window

* volume: reject an over-cap chunk span and an illegal manifest MIME

* volume: take the manifest transform extension from the URL first

Go's tryHandleChunkedFile keeps the extension parsed from the URL —
including a fid suffix like /vid,fid.png — and only falls back to the
resolved filename's extension when the URL has none. The manifest path
now does the same, so a transform requested on a fid-suffixed URL runs
as it does in Go.

Also: compute the chunk-size sum once, drop an unused remote_read_for
parameter, share the crop/resize predicate with parse_read_request, and
settle a couple of clippy lints.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-10 10:05:00 +08:00
Eliah RusinandClaude Opus 5.5 0bcebf708c ecbalancer: cap total shards per rack in Plan (#11623)
* ecbalancer: cap total shards per rack in Plan

Plan caps data and parity per rack separately (ceil(data/racks) and
ceil(parity/racks)), so with 10+4 over 8 racks a rack can legally hold
2 data + 1 parity. When a rack is one disk, losing two such racks loses
6 of 14 shards and the volume can't be read.

The cross-rack phase now also caps each rack's TOTAL shards of a
volume, sized with Place's rackTotalCap: ceil(shards/racks) unless the
racks lack room, counting a rack's own shards of the volume as room
since Plan can move them.

- A rack above the cap sheds parity until it fits. Those shards may go
  to a data-bearing rack, and when no rack is under the parity cap
  they fall back to a rack under the total cap. #11438's non-overflow
  candidates keep moving only to data-free racks.
- No cross-rack move lands on a rack at the cap.
- A rack above the cap triggers balancing regardless of the imbalance
  threshold.
- The fallback applies only while the source rack is above the cap;
  otherwise the next Plan moves the shard back.
- Options.RackTotalCapRaised reports volumes whose cap had to be raised
  above the even share; the worker and the shell log it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ecbalancer: finish the rack cap in one Plan, count SameRackCount room

Review follow-ups.

- The data pass can run out of destinations before the parity pass frees
  slots elsewhere, which left a data-heavy rack above the cap after one
  Plan (a one-shot shell balance stops there). The cross-rack phase now
  repeats while a rack is above the cap and the last round moved
  something. A shard moves at most once per plan, since each move runs
  as its own task.
- planRackTotalCap bounds each node's room by what SameRackCount still
  allows, as Place does. Counting the raw free slots sized the cap too
  low, so RackTotalCapRaised missed volumes it should report.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 20:35:25 +08:00
c0a751072d volume server: apply Range to chunk manifests and forward raw headers when proxying (#11538)
* volume server: read GET/HEAD needles off the store lock, and only once

The GET/HEAD handler read the needle synchronously on the tokio worker
while holding store.read(): first a stream-info read that loaded the
whole record just to parse its meta, then, for every needle that was not
streamed (small, compressed, chunk manifest, image ops), a second full
read. For a tiered volume each read is an S3 GET under the store lock,
and a writer queued behind it parks every other store reader.

The regular-volume read now runs in spawn_blocking. Under the store guard
it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat
handle or the remote backend, offset, size); the guard is dropped before
any needle data I/O. No data-file lease is held across the read either,
since a writer waits for one while holding the store write lock. The
index size decides the read, as in Go's readNeedle: a HEAD, a ranged read
or a needle above the stream threshold reads only its header and meta
tail (ReadNeedleMeta) and hands off to StreamingBody or the range path;
everything else is read in full once, with its checksum verified. A
compressed or manifest needle found by the meta read is then read in
full once. The range-from-source read also moves to spawn_blocking.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: stream needle chunks without the store lock

StreamingBody::poll_frame took store.read() and find_volume for every
chunk to compare the volume's compaction revision, dup'd the source
handle, and allocated a fresh chunk buffer. With -hasSlowRead=false the
stream also holds a data-file read lease for its whole life, while a
writer waits for that lease under store.write(): the next chunk's
store.read() then waits for the writer and the writer for the stream.

The per-chunk re-lookup was also wrong. The stream reads a handle opened
at plan time, which pins the .dat inode the offset was resolved against;
a vacuum commit renames a new file over .dat and leaves that inode
untouched. The re-looked-up offset belongs to the new file but was read
from the old inode, so a stream whose needle a vacuum moved ended in a
checksum error. The pinned offset stays valid, so the check, and with
it every store access, is dropped, along with the now unused
re_lookup_needle_data_offset and the revision fields of the read plan.

The source is shared as an Arc instead of dup'd per chunk, and the chunk
buffer is a BytesMut that the blocking read hands back with its result,
so its allocation is reclaimed once the previous frame has been written.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: split get_or_head_handler_inner into phases

get_or_head_handler_inner was a ~650-line function. Its middle resolved
the needle and set five mutable flags (stream_info, can_stream,
can_handle_head_from_meta, can_handle_range_from_source, bypass_cm) that
three if-let reply paths then re-tested, each re-checking stream_info.

It is now a 126-line orchestrator over named phases: reject_read_jwt,
proxy_missing_volume, wait_for_download_slot, parse_read_request,
read_ec_needle / read_volume_needle, etag_and_last_modified,
not_modified_response, read_response_headers, and the reply phases
stream_response, head_from_meta_response, range_from_source_response,
buffered_payload and buffered_response. The read phases return a
ReadPlan whose ReadStrategy enum (Stream, HeadFromMeta, RangeFromSource,
Buffered) carries the NeedleStreamInfo only on the variants that use it,
so the reply is one match instead of three flag checks.

Pure refactor: every status code, header and header order, error text,
metric increment, lock and data-file lease scope, spawn_blocking
boundary and side-effect order is unchanged. Phases that can end the
request return ControlFlow<Response, T>. A Range header that is not
visible ASCII still falls through to the buffered path, as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: stop a needle stream once its volume becomes unavailable

Taking the store lock out of StreamingBody also dropped its per-chunk
unavailable_error() check. With -hasSlowRead a writer can take the
data-file lease between chunks, fail its fsync and its truncate, and mark
the volume unavailable; the stream then kept serving the rest of the
needle from its pinned handle.

The volume's io_unavailable reason is now an Arc-shared leaf mutex that
the read plan hands to the stream. Each chunk checks it under its
data-file lease, where the writer marks it, and fails with the same
"volume is unavailable: <reason>" error the old check returned.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: read a non-ASCII or empty Range header as Go does

A Range value with a byte >= 0x80 (obs-text, which hyper accepts) failed
HeaderValue::to_str at both range gates. For a needle served as stored the
handler had already chosen a meta-only read, so it fell through to the
buffered path with no payload and answered 200 with an empty body; a
compressed or EC needle answered 200 with the full body. Go's parseRange
fails on a byte it can neither trim nor parse and answers 416 "invalid
range", and trims Unicode whitespace such as NBSP into a normal 206. An
empty Range value was also a 200 with an empty body, where Go sends the
whole payload.

Read Range once with from_utf8_lossy, dropping an empty value, and hand
that one value to the read plan and to both range gates. A replaced byte
never parses, so it is a 416; str::trim trims the same Unicode whitespace
as strings.TrimSpace. A range read from the data file now always answers
itself instead of falling through with an empty needle. The buffered path
answers HEAD before it looks at Range, as Go's writeResponseContent does,
so an EC HEAD with a Range is a 200 with the full length.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: apply Range to chunk manifests and forward raw headers when proxying

A GET of a chunk manifest assembled the object and always answered 200 with
the whole body, ignoring Range. Go serves the expanded manifest through
writeResponseContent, which answers HEAD first and then hands Range to
ProcessRangeRequest: 206 for one range, multipart/byteranges for several,
416 for an unsatisfiable or unparsable one. try_expand_chunk_manifest now
returns the assembled body and headers, and the caller answers through
buffered_response, the same helper the buffered needle path uses.

A proxied read forwarded a request header only if HeaderValue::to_str
succeeded, so a Range with an obs-text byte was dropped and the target
answered 200 with the full body. Go copies every header value as is.
Forward the raw HeaderValue for every header.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: reset Content-Type on range errors as net/http does

Go's http.Error sets text/plain and nosniff unconditionally; keeping the
needle's MIME type on a 416 mislabels the error body. Match it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: count manifest downloads toward the limit

Expanded manifests hard-coded track_download off, so chunked-file reads
bypassed in-flight byte accounting while every other path honored the
download limiter.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-06 09:54:55 +08:00
825c3dff8b volume server: fetch only the chunks a manifest Range needs, like Go (#11546)
* volume server: read GET/HEAD needles off the store lock, and only once

The GET/HEAD handler read the needle synchronously on the tokio worker
while holding store.read(): first a stream-info read that loaded the
whole record just to parse its meta, then, for every needle that was not
streamed (small, compressed, chunk manifest, image ops), a second full
read. For a tiered volume each read is an S3 GET under the store lock,
and a writer queued behind it parks every other store reader.

The regular-volume read now runs in spawn_blocking. Under the store guard
it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat
handle or the remote backend, offset, size); the guard is dropped before
any needle data I/O. No data-file lease is held across the read either,
since a writer waits for one while holding the store write lock. The
index size decides the read, as in Go's readNeedle: a HEAD, a ranged read
or a needle above the stream threshold reads only its header and meta
tail (ReadNeedleMeta) and hands off to StreamingBody or the range path;
everything else is read in full once, with its checksum verified. A
compressed or manifest needle found by the meta read is then read in
full once. The range-from-source read also moves to spawn_blocking.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: stream needle chunks without the store lock

StreamingBody::poll_frame took store.read() and find_volume for every
chunk to compare the volume's compaction revision, dup'd the source
handle, and allocated a fresh chunk buffer. With -hasSlowRead=false the
stream also holds a data-file read lease for its whole life, while a
writer waits for that lease under store.write(): the next chunk's
store.read() then waits for the writer and the writer for the stream.

The per-chunk re-lookup was also wrong. The stream reads a handle opened
at plan time, which pins the .dat inode the offset was resolved against;
a vacuum commit renames a new file over .dat and leaves that inode
untouched. The re-looked-up offset belongs to the new file but was read
from the old inode, so a stream whose needle a vacuum moved ended in a
checksum error. The pinned offset stays valid, so the check, and with
it every store access, is dropped, along with the now unused
re_lookup_needle_data_offset and the revision fields of the read plan.

The source is shared as an Arc instead of dup'd per chunk, and the chunk
buffer is a BytesMut that the blocking read hands back with its result,
so its allocation is reclaimed once the previous frame has been written.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: split get_or_head_handler_inner into phases

get_or_head_handler_inner was a ~650-line function. Its middle resolved
the needle and set five mutable flags (stream_info, can_stream,
can_handle_head_from_meta, can_handle_range_from_source, bypass_cm) that
three if-let reply paths then re-tested, each re-checking stream_info.

It is now a 126-line orchestrator over named phases: reject_read_jwt,
proxy_missing_volume, wait_for_download_slot, parse_read_request,
read_ec_needle / read_volume_needle, etag_and_last_modified,
not_modified_response, read_response_headers, and the reply phases
stream_response, head_from_meta_response, range_from_source_response,
buffered_payload and buffered_response. The read phases return a
ReadPlan whose ReadStrategy enum (Stream, HeadFromMeta, RangeFromSource,
Buffered) carries the NeedleStreamInfo only on the variants that use it,
so the reply is one match instead of three flag checks.

Pure refactor: every status code, header and header order, error text,
metric increment, lock and data-file lease scope, spawn_blocking
boundary and side-effect order is unchanged. Phases that can end the
request return ControlFlow<Response, T>. A Range header that is not
visible ASCII still falls through to the buffered path, as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: stop a needle stream once its volume becomes unavailable

Taking the store lock out of StreamingBody also dropped its per-chunk
unavailable_error() check. With -hasSlowRead a writer can take the
data-file lease between chunks, fail its fsync and its truncate, and mark
the volume unavailable; the stream then kept serving the rest of the
needle from its pinned handle.

The volume's io_unavailable reason is now an Arc-shared leaf mutex that
the read plan hands to the stream. Each chunk checks it under its
data-file lease, where the writer marks it, and fails with the same
"volume is unavailable: <reason>" error the old check returned.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: read a non-ASCII or empty Range header as Go does

A Range value with a byte >= 0x80 (obs-text, which hyper accepts) failed
HeaderValue::to_str at both range gates. For a needle served as stored the
handler had already chosen a meta-only read, so it fell through to the
buffered path with no payload and answered 200 with an empty body; a
compressed or EC needle answered 200 with the full body. Go's parseRange
fails on a byte it can neither trim nor parse and answers 416 "invalid
range", and trims Unicode whitespace such as NBSP into a normal 206. An
empty Range value was also a 200 with an empty body, where Go sends the
whole payload.

Read Range once with from_utf8_lossy, dropping an empty value, and hand
that one value to the read plan and to both range gates. A replaced byte
never parses, so it is a 416; str::trim trims the same Unicode whitespace
as strings.TrimSpace. A range read from the data file now always answers
itself instead of falling through with an empty needle. The buffered path
answers HEAD before it looks at Range, as Go's writeResponseContent does,
so an EC HEAD with a Range is a 200 with the full length.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: apply Range to chunk manifests and forward raw headers when proxying

A GET of a chunk manifest assembled the object and always answered 200 with
the whole body, ignoring Range. Go serves the expanded manifest through
writeResponseContent, which answers HEAD first and then hands Range to
ProcessRangeRequest: 206 for one range, multipart/byteranges for several,
416 for an unsatisfiable or unparsable one. try_expand_chunk_manifest now
returns the assembled body and headers, and the caller answers through
buffered_response, the same helper the buffered needle path uses.

A proxied read forwarded a request header only if HeaderValue::to_str
succeeded, so a Range with an obs-text byte was dropped and the target
answered 200 with the full body. Go copies every header value as is.
Forward the raw HeaderValue for every header.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: fetch only the chunks a manifest Range needs, like Go

A ranged GET of a chunk manifest fetched every chunk, assembled the whole
object and then sliced it, so reading a few bytes of a large object cost a
read of all of it, and a 416 still fetched everything. Go serves a manifest
through ChunkedFileReader, which seeks to each range and reads only the
chunks under it.

For a GET with a Range, try_expand_chunk_manifest now parses the ranges
against the manifest size, fetches only the chunks whose declared window
overlaps one of them (none when the reply carries no body), and answers
through handle_range_request_with, the reader-based core that
handle_range_request now wraps, so 206/416/multipart stay one code path.
The reader replays assembly: chunks clamped as before, later chunks over
earlier ones, zeros in gaps. HEAD, no-Range GETs and GETs that crop or
resize an image still assemble the whole object.

A missing chunk outside the requested ranges no longer turns a ranged GET
into a 500, as in Go; a missing chunk inside them still does, before any
headers are sent.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: reword a comment codespell flags

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: keep only range-covered bytes of fetched manifest chunks

A ranged GET retained every overlapping chunk's full contents; 1,000
overlapping 8 MiB chunks could pin ~8 GiB for a one-byte response. Clip
each fetched chunk to the bytes the requested ranges can actually read,
preserving the later-chunks-overwrite and zero-fill-gap semantics.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: bucket ranged manifest parts by range

Serving a multipart range scanned every retained part. Bucket the kept
intersections by their range so one range only reads its own parts.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-05 21:17:19 +08:00
7a19961074 volume server: read a non-ASCII or empty Range header as Go does (#11534)
* volume server: read GET/HEAD needles off the store lock, and only once

The GET/HEAD handler read the needle synchronously on the tokio worker
while holding store.read(): first a stream-info read that loaded the
whole record just to parse its meta, then, for every needle that was not
streamed (small, compressed, chunk manifest, image ops), a second full
read. For a tiered volume each read is an S3 GET under the store lock,
and a writer queued behind it parks every other store reader.

The regular-volume read now runs in spawn_blocking. Under the store guard
it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat
handle or the remote backend, offset, size); the guard is dropped before
any needle data I/O. No data-file lease is held across the read either,
since a writer waits for one while holding the store write lock. The
index size decides the read, as in Go's readNeedle: a HEAD, a ranged read
or a needle above the stream threshold reads only its header and meta
tail (ReadNeedleMeta) and hands off to StreamingBody or the range path;
everything else is read in full once, with its checksum verified. A
compressed or manifest needle found by the meta read is then read in
full once. The range-from-source read also moves to spawn_blocking.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: stream needle chunks without the store lock

StreamingBody::poll_frame took store.read() and find_volume for every
chunk to compare the volume's compaction revision, dup'd the source
handle, and allocated a fresh chunk buffer. With -hasSlowRead=false the
stream also holds a data-file read lease for its whole life, while a
writer waits for that lease under store.write(): the next chunk's
store.read() then waits for the writer and the writer for the stream.

The per-chunk re-lookup was also wrong. The stream reads a handle opened
at plan time, which pins the .dat inode the offset was resolved against;
a vacuum commit renames a new file over .dat and leaves that inode
untouched. The re-looked-up offset belongs to the new file but was read
from the old inode, so a stream whose needle a vacuum moved ended in a
checksum error. The pinned offset stays valid, so the check, and with
it every store access, is dropped, along with the now unused
re_lookup_needle_data_offset and the revision fields of the read plan.

The source is shared as an Arc instead of dup'd per chunk, and the chunk
buffer is a BytesMut that the blocking read hands back with its result,
so its allocation is reclaimed once the previous frame has been written.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: split get_or_head_handler_inner into phases

get_or_head_handler_inner was a ~650-line function. Its middle resolved
the needle and set five mutable flags (stream_info, can_stream,
can_handle_head_from_meta, can_handle_range_from_source, bypass_cm) that
three if-let reply paths then re-tested, each re-checking stream_info.

It is now a 126-line orchestrator over named phases: reject_read_jwt,
proxy_missing_volume, wait_for_download_slot, parse_read_request,
read_ec_needle / read_volume_needle, etag_and_last_modified,
not_modified_response, read_response_headers, and the reply phases
stream_response, head_from_meta_response, range_from_source_response,
buffered_payload and buffered_response. The read phases return a
ReadPlan whose ReadStrategy enum (Stream, HeadFromMeta, RangeFromSource,
Buffered) carries the NeedleStreamInfo only on the variants that use it,
so the reply is one match instead of three flag checks.

Pure refactor: every status code, header and header order, error text,
metric increment, lock and data-file lease scope, spawn_blocking
boundary and side-effect order is unchanged. Phases that can end the
request return ControlFlow<Response, T>. A Range header that is not
visible ASCII still falls through to the buffered path, as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: stop a needle stream once its volume becomes unavailable

Taking the store lock out of StreamingBody also dropped its per-chunk
unavailable_error() check. With -hasSlowRead a writer can take the
data-file lease between chunks, fail its fsync and its truncate, and mark
the volume unavailable; the stream then kept serving the rest of the
needle from its pinned handle.

The volume's io_unavailable reason is now an Arc-shared leaf mutex that
the read plan hands to the stream. Each chunk checks it under its
data-file lease, where the writer marks it, and fails with the same
"volume is unavailable: <reason>" error the old check returned.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: read a non-ASCII or empty Range header as Go does

A Range value with a byte >= 0x80 (obs-text, which hyper accepts) failed
HeaderValue::to_str at both range gates. For a needle served as stored the
handler had already chosen a meta-only read, so it fell through to the
buffered path with no payload and answered 200 with an empty body; a
compressed or EC needle answered 200 with the full body. Go's parseRange
fails on a byte it can neither trim nor parse and answers 416 "invalid
range", and trims Unicode whitespace such as NBSP into a normal 206. An
empty Range value was also a 200 with an empty body, where Go sends the
whole payload.

Read Range once with from_utf8_lossy, dropping an empty value, and hand
that one value to the read plan and to both range gates. A replaced byte
never parses, so it is a 416; str::trim trims the same Unicode whitespace
as strings.TrimSpace. A range read from the data file now always answers
itself instead of falling through with an empty needle. The buffered path
answers HEAD before it looks at Range, as Go's writeResponseContent does,
so an EC HEAD with a Range is a 200 with the full length.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-05 19:47:54 +08:00
10b0f2b8ad volume server: refuse the rest of a grouped run after a durable index failure (#11576)
* volume server: refuse the rest of a grouped run after a durable index failure

A durable write whose needle-map put fails stops the volume taking writes
(#10825): sent on its own, the next write then fails read only before it
appends. The grouped run from #11543 appends and syncs every entry before
publishing any, then kept publishing the entries after the failed one and
acked them once the shared .idx sync went through. When the failed put
tore its .idx row, the rows appended after it land off alignment, so the
next load parses them as garbage and the acked writes are gone.

Once a durable entry fails to publish, refuse every later entry of the run
with ReadOnly, as the per-needle path does. The entries before it stay
acked; their rows go down with the run's one .idx sync. The refused
records stay on the .dat unindexed, as the failed one does on its own.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: refuse a grouped entry staged as a cookie mismatch too

After a durable entry in a grouped run fails to index, the entries
after it are refused as they would be on their own. On its own an entry
meets check_writable before its cookie check, so one staged as a cookie
mismatch now gets the refusal too, instead of keeping its staging error.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: trim a torn .idx row back so the next stays aligned

A failed write_index_entry can leave half a row in the .idx. With the
writer appending at the tail, every row written after it lands off
alignment and the next load parses them as garbage, so a write acked
behind a torn row does not come back. Trim the file back to
idx_file_offset on a failed append, in both needle maps, and cover it
with a test that writes past a torn row and reloads.

* volume server: refuse queued Go writes once a durable index update fails

processBatch kept writing after a failed nm.Put, and the single-write
path checked IsReadOnly only outside the volume lock. A durable write
whose index update fails now marks the volume noWriteOrDelete, and each
queued request is checked before it appends, so the ones after a failed
durable entry are refused the way a lone write is. Deletes get the same
noWriteOrDelete refusal a lone delete gets.

* volume server: refuse appends while a torn .idx row cannot be trimmed

When trimming back a half-written .idx row itself fails, the next append
would land after the torn bytes and every later row would parse off
alignment on load. Latch the map as torn and refuse appends until the
trim succeeds, on both CompactNeedleMap and RedbNeedleMap; the same
latch covers an orphan row that could not be trimmed after a failed
redb commit.

The .idx writer is now opened with write+append access so truncate_to
(set_len) works on Windows, where an append-only handle cannot trim.

* volume server: write .idx rows at idx_file_offset, not via append mode

Rust's OpenOptions on Windows strips FILE_WRITE_DATA whenever append is
set so the handle stays strictly append-only, which makes set_len fail -
the torn-row trim could never succeed there. Open the .idx writer with
plain write access and seek to idx_file_offset before each row, the same
positioned-write model the Go server uses.

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-03 21:04:30 +08:00
8c1ebbee32 volume server: split get_or_head_handler_inner into phases (#11489)
* volume server: read GET/HEAD needles off the store lock, and only once

The GET/HEAD handler read the needle synchronously on the tokio worker
while holding store.read(): first a stream-info read that loaded the
whole record just to parse its meta, then, for every needle that was not
streamed (small, compressed, chunk manifest, image ops), a second full
read. For a tiered volume each read is an S3 GET under the store lock,
and a writer queued behind it parks every other store reader.

The regular-volume read now runs in spawn_blocking. Under the store guard
it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat
handle or the remote backend, offset, size); the guard is dropped before
any needle data I/O. No data-file lease is held across the read either,
since a writer waits for one while holding the store write lock. The
index size decides the read, as in Go's readNeedle: a HEAD, a ranged read
or a needle above the stream threshold reads only its header and meta
tail (ReadNeedleMeta) and hands off to StreamingBody or the range path;
everything else is read in full once, with its checksum verified. A
compressed or manifest needle found by the meta read is then read in
full once. The range-from-source read also moves to spawn_blocking.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: stream needle chunks without the store lock

StreamingBody::poll_frame took store.read() and find_volume for every
chunk to compare the volume's compaction revision, dup'd the source
handle, and allocated a fresh chunk buffer. With -hasSlowRead=false the
stream also holds a data-file read lease for its whole life, while a
writer waits for that lease under store.write(): the next chunk's
store.read() then waits for the writer and the writer for the stream.

The per-chunk re-lookup was also wrong. The stream reads a handle opened
at plan time, which pins the .dat inode the offset was resolved against;
a vacuum commit renames a new file over .dat and leaves that inode
untouched. The re-looked-up offset belongs to the new file but was read
from the old inode, so a stream whose needle a vacuum moved ended in a
checksum error. The pinned offset stays valid, so the check, and with
it every store access, is dropped, along with the now unused
re_lookup_needle_data_offset and the revision fields of the read plan.

The source is shared as an Arc instead of dup'd per chunk, and the chunk
buffer is a BytesMut that the blocking read hands back with its result,
so its allocation is reclaimed once the previous frame has been written.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: split get_or_head_handler_inner into phases

get_or_head_handler_inner was a ~650-line function. Its middle resolved
the needle and set five mutable flags (stream_info, can_stream,
can_handle_head_from_meta, can_handle_range_from_source, bypass_cm) that
three if-let reply paths then re-tested, each re-checking stream_info.

It is now a 126-line orchestrator over named phases: reject_read_jwt,
proxy_missing_volume, wait_for_download_slot, parse_read_request,
read_ec_needle / read_volume_needle, etag_and_last_modified,
not_modified_response, read_response_headers, and the reply phases
stream_response, head_from_meta_response, range_from_source_response,
buffered_payload and buffered_response. The read phases return a
ReadPlan whose ReadStrategy enum (Stream, HeadFromMeta, RangeFromSource,
Buffered) carries the NeedleStreamInfo only on the variants that use it,
so the reply is one match instead of three flag checks.

Pure refactor: every status code, header and header order, error text,
metric increment, lock and data-file lease scope, spawn_blocking
boundary and side-effect order is unchanged. Phases that can end the
request return ControlFlow<Response, T>. A Range header that is not
visible ASCII still falls through to the buffered path, as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: stop a needle stream once its volume becomes unavailable

Taking the store lock out of StreamingBody also dropped its per-chunk
unavailable_error() check. With -hasSlowRead a writer can take the
data-file lease between chunks, fail its fsync and its truncate, and mark
the volume unavailable; the stream then kept serving the rest of the
needle from its pinned handle.

The volume's io_unavailable reason is now an Arc-shared leaf mutex that
the read plan hands to the stream. Each chunk checks it under its
data-file lease, where the writer marks it, and fails with the same
"volume is unavailable: <reason>" error the old check returned.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: mirror Go order in the buffered read path

- check HEAD before Range in buffered_response (writeResponseContent
  order); an EC-volume HEAD with a Range header answered 206, Go
  answers 200
- treat the proxied flag as an exact query pair like Go's parsed
  lookup, not a substring
- name the phases after their Go counterparts: check_download_limit
  and read_ec_shard_needle; reuse has_replication()
- drop comments that restate the code or cite Go line numbers

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-03 17:54:46 +08:00
07da302da0 volume server: ec.decode verifies, cleans up and compacts like Go, off the runtime (#11547)
* volume server: ec.decode reads the .ecx from the index dir it was copied to

VolumeEcShardsCopy writes the .ecx/.ecj into the receiver's -dir.idx, so
with a split data/index dir the decode target has no .ecx beside its
shards. VolumeEcShardsToVolume sized the .dat from the right .ecx but
built the .idx from the data dir, failing with NotFound after the .dat
was already published. It now reads .ecx/.ecj from where the EC volume
opened them and writes the .idx beside the .dat, where Go leaves it.

The live-entry check and the .dat size also ignored deletions recorded
only in the .ecj, which Go folds into the .ecx (RebuildEcxFile) first:
a fully deleted volume was decoded instead of reported as having no live
entries, and deleted tail needles were copied into the .dat. Both now
treat journaled ids as deleted, without rewriting the sealed .ecx.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode keeps the decoded volume writable and reads every .ecj

The rebuilt .idx copied a journaled tail needle's .ecx row verbatim after
the .dat was cut short before it, so the mount saw a row past EOF and
marked the decoded volume read-only. Rows of deleted needles the .dat no
longer holds are now dropped, and each journaled needle still in the .dat
gets one tombstone instead of one per journal entry.

VolumeEcShardsCopy appends journals collected from other holders into
the idx dir, but the decode read only the .ecj beside the .ecx, which
sits in the data dir when this server generated the shards. It now
reads both, once, in bounded chunks via the loader EcVolume uses.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: test ec.decode drops a sealed .ecx tail tombstone

Covers the other half of the rule added in the previous commit: a tail
needle tombstoned in the .ecx itself (Go's RebuildEcxFile) is cut from
the .dat, and its row must not reach the rebuilt .idx either.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode runs its file I/O off the async runtime

VolumeEcShardsToVolume released the store lock before decoding, but read
the .ecx/.ecj, rebuilt the .dat and wrote the .idx inside the async
handler, parking a runtime worker for the length of a volume-sized copy.
The decode now runs in spawn_blocking on inputs snapshotted under the
store lock.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode checks the rebuilt .dat is complete

Go stats the decoded .dat before writing the .idx (VerifyDecodedDatFile)
and fails the decode when it is shorter than the extent the EC index
references, since the caller deletes the shards once the call returns.
The Rust handler returned success without that check. The rebuild
already fails on a short shard read, so this guards the published file
itself.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode drops the decoded volume's bitrot sidecars

Go removes <base>.ecsum and <base>.ecsum.v<N> beside the .dat and beside
the .ecx once the .idx is written, so a stale checksum sidecar cannot
pass for the protection of a later re-encode. The Rust handler left them
in place. Removal is best effort, as in Go.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode compacts the decoded volume

Go ends VolumeEcShardsToVolume with an offline CompactVolumeFiles, so the
decoded volume holds only live needles. The Rust decode left every needle
deleted through the .ecj in the .dat, tombstoned in the .idx, until a
later vacuum reclaimed it.

Store::compact_volume_files loads the unmounted volume, checks free space
the way the vacuum does (the estimate now lives in one helper), and runs
the vacuum's compact-by-index and commit. As in Go a failed compaction is
logged and the decode still succeeds, so the uncompacted .idx rules stay:
the tests that pin them now make the compaction fail.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode keeps deletes journaled while the .dat is written

The decode read the .ecj journals once, before rebuilding the .dat, so a
delete that reached the EC volume during the rebuild was left out of the
new .idx and the needle came back live. Each journal's read length is now
kept, and the bytes appended since are read just before the .idx is
written, after waiting out any journal append in flight (appends hold
the store write lock), so every delete acknowledged by then is in the
.idx. A delete after that point is still lost, as in Go.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Guard overlapping ec decode requests; serialize journal catch-up

volume_ec_shards_to_volume runs its decode in spawn_blocking, so a
dropped request leaves the job running and a retry would race it on the
temporary and final volume files. Claim the vid in a per-server
in-flight set until the blocking job finishes, and return Unavailable
to an overlapping request. The Go handler has the same exposure and
gets the same guard.

Journal appends hold the store write lock through their
sync-or-truncate, so holding a read lock across the catch-up read
guarantees every record it sees is committed: a rolled-back delete can
no longer leave a tombstone in the decoded index.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Reconcile the swap when offline compaction commit fails

A CommitCompact that fails after the .cpc marker may have renamed .dat
but not .idx. cleanup_compact refuses while the marker exists, so the
mismatched pair survived until a restart reconciled it — and the decode
caller treats the failure as non-fatal. Run reconcileCompactState on
commit failure so a decided swap rolls forward and orphan temps are
removed before the volume can mount.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Release the decode claim on panic

* volume: add ec_decodes_in_flight to the integration-test state literal

* volume server: hold the decode tail's lock through compaction

The catch_up read released before the rebuilt .idx was written and the
volume compacted, so a delete synced to .ecj in that window was durably
journaled yet absent from the published index — resurrecting the needle.
Rust now holds the store read lock from catch_up through compact, and Go
mirrors it by holding the volume's journal lock from the journal-
consuming index write through CompactVolumeFiles.

* volume server: serialize ec decode's tail per volume, not per store

Review follow-ups on the decode path:

- Rust: holding the store read lock from journal catch-up through the
  offline compaction stalled every writer on unrelated volumes for the
  whole rewrite. The new ec_decode_tail set marks the vid only while its
  .idx is published and .cpd/.cpx swapped; the two local .ecj append paths
  (VolumeEcBlobDelete, the distributed delete's local journal) wait on a
  Notify for that span — Go's per-volume ecjFileAccessLock semantics
  without the global stall. VolumeMount and the staged-adopt path are also
  held off while a decode claim is in flight so neither can race the swap.

- Rust: the initial journal read ran unlocked, so bytes a rolled-back
  append later truncated could be folded in as phantom tombstones. The
  first pass stays unlocked (a slow journal must not stall the store) and
  a rescan under the quiescing read lock re-reads only committed content;
  catch_up now rebuilds the id set when a regular journal shrank.

- Go: the decode resolved the compaction DiskLocation through
  FindEcVolume while holding the journal lock, inverting DestroyEcVolume's
  map->journal order into a deadlock. The lookup now happens first, and
  DestroyEcVolume/deleteEcVolumeById/DiskLocation.Close destroy outside
  the map lock.

- Go: RebuildEcxFile unlinks .ecj while the volume's ecjFile handle stays
  open, so later deletes could commit to a detached inode. Both call sites
  now fold under the journal lock and ReopenDeletionJournal repoints the
  handle at the live path, working on the volume's resolved .ecx dir
  (EcIndexBaseFileName) rather than the configured index dir.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: fence EC remounts behind the destroy tombstone

DestroyEcVolume, deleteEcVolumeById, and the collection-delete sweep now
remove the EcVolume from ecVolumes before destroying it off-lock, so a
concurrent remount could re-open shard files that the in-flight destroy
then unlinks — registering a detached fd.

Each destroy records a per-vid tombstone channel in a new
ecVolumesDestroying map before dropping the map entry and closes it when
Destroy returns. The tombstone intentionally survives as the vid's
destroy generation: loadEcShardWithIdxDir compares it before and after
opening the shard, so a destroy that both started and finished inside the
open window is still detected. A mismatch drops the just-opened shard
(releasing its fd and mount gauge) and retries after the destroy
completes; a successful mount clears the stale tombstone.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: rescan the .ecj under the store lock only after a rollback

The decode's second journal pass ran a full rescan under the store read
lock on every decode, stalling unrelated writers for the length of the
scan. Bump a process-wide epoch whenever a failed append truncates its
uncommitted tail; an unchanged epoch between the unlocked read and the
quiesced pass proves every id folded in was committed, so catch_up()
suffices. catch_up() also treats a journal that was read but has since
disappeared as shrunk to zero, so its earlier ids cannot linger.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: check the decode tail under the store write lock on delete

A blob delete waited for the publishing tail before taking the store
write lock, so a decode that claimed the tail while the delete was
parked behind the decoder's read lock could still see the journal append
land after the rebuilt .idx — an acknowledged delete the mount would
miss. Test tail membership under the write lock instead, retrying after
the wait; journal_delete_local reports WouldBlock for the same recheck
on the distributed path.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: claim the vid for mount and staged adoption, per volume

VolumeMount and the staged .copying adoption held the
ec_decodes_in_flight set lock through slow file renames and mounts,
stalling every unrelated volume's decode, mount, and adoption. Take the
per-volume claim instead — the same exclusion against a racing decode
for this vid, released when the call returns.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: fail the decode when a compaction commit marker survives

CompactVolumeFiles' caller logged a compaction error and went on to
delete the EC shards. When the commit marker (.cpc) is still on disk the
.dat/.idx swap was decided but could not be reconciled, so the mounted
pair may be mismatched — report the failure instead so the shards are
kept and the caller can retry.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: gate the parked-delete test on the held write lock

The releaser thread and the spawned delete raced for the store write
lock; on a slow runner the delete could acquire it first and commit
before the tail was ever claimed, failing !delete.is_finished() on the
Windows unit-test job. Spawn the delete only after the thread reports
the lock held.

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-03 14:55:15 +08:00
0ff7794c54 volume: compact an oversized .ecj at mount, safely (Rust + Go) (#11555)
* volume: compact an oversized .ecj at mount, safely (Rust + Go)

Restore the mount-time compaction dropped from #11408, Rust + Go parity.
A journal already bloated by repeated shard copies is folded down to the
id set it encodes.

- Trigger after load when file_records > max(threshold, 4x distinct),
  with a 1 MiB floor so small journals are never rewritten. The set is
  written to .ecj.compact.tmp + fsync, the handle dropped, renamed,
  the directory fsynced and the append handle reopened. A failure before
  the rename keeps the original journal and handle; a failure after it
  fails the mount.
- Go never compacts after a failed journal load; the set would be
  partial and the rewrite would drop the unread records.
- A per-path registry (ecj_registry.rs / ecj_registry.go) counts EcVolume
  holders and out-of-band writers of each .ecj. Compaction runs only
  when this volume is the sole holder and no copy is writing; holders
  and writers wait while one runs. This covers shared -dir.idx journals
  and cross-disk reconcile, where another EcVolume may hold the same
  journal.
- VolumeEcShardsCopy and EC index recovery register as writers around
  their .ecj append and partial-file cleanup.
- Under the reservation, re-check that the file on disk is still the
  inode and size that was loaded.
- Publish errors are classified where they happen; a failed rename plus
  a failed restore reports both errors.
- Compaction runs after the .vif / bitrot checks, so a refused mount
  leaves the journal untouched.
- The tmp is opened like other volume files, removed at mount if a crash
  left it, and listed in every EC index cleanup path.

Failure paths are tested through the real mount via injectable fs steps
(open_with / newEcVolumeWith), plus sibling holders, active copies,
changed-after-load, stale tmp cleanup, refused mounts and the Go
load-error guard.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: fail the mount when the compacted .ecj's directory cannot be synced

The Rust mount synced the journal's directory after renaming the compacted
file over it through the crate's best-effort fsync_dir, which returns Ok
when the directory cannot be opened. A rename needs only write and search
permission, so on a directory without read permission the replacement was
published, never synced, and the mount went on taking deletes against it.

Sync through a helper that propagates the open error, as Go's
util.FsyncDir already does, so that case fails the mount like any other
post-rename sync failure.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: test the no-compaction-after-failed-load rule through the Go mount

The test for it handed compactEcjAfterLoad an artificial error on a volume
that had loaded cleanly, so it would not notice NewEcVolume dropping the
real load error on the way to compaction.

Make the journal read one of the injectable ecjFsOps steps and fail it
inside the real mount, after the first chunk, on a journal whose last
entry is an id the first chunk does not hold. The mount must leave the
file byte for byte as it was; a clean remount then compacts and keeps
that id. The Rust mount fails outright on a load error, so it has no
equivalent path.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: register ReceiveFile's .ecj writes with the journal registry

ReceiveFile refuses a mounted EC volume only once, when the info message
arrives, then creates the .ecj and streams chunks into it. A volume that
mounted on that journal mid-stream could find a bloated prefix, pass the
inode-and-size re-check and rename a compacted file over it; the rest of
the stream then went to the unlinked inode and was lost.

Register the path as a writer before the file is created, in both the Go
and Rust handlers, and hold it until the file is closed and any partial
copy removed, as the shard-copy and index-recovery appends already do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: skip .ecj compaction when a writer ran since the journal was loaded

Compaction checked only that no writer was active at the reservation, and
that the file was still the loaded inode at the loaded size. A ReceiveFile
truncates and refills the journal in place, so one that ran during the
mount's load, or after it, and finished before the reservation could leave
different ids at the same length; compaction then wrote the stale set over
them.

Give each path a write generation that every writer bumps as it starts. A
holder records it, and whether a writer was active, when it registers,
which is before it opens and loads the journal. It may compact only if no
writer was active then and the generation has not moved. Same rule in Go
and Rust; the journal read becomes an injectable step in Rust as it is in
Go, so both test the in-place rewrite through the real mount.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: match the ReadOnly(VolumeId) variant in write_volume_needles

#11543 matched VolumeError::ReadOnly as a unit variant in Store::write_volume_needles, and #11544 changed it to ReadOnly(VolumeId) in the same merge window. Each passed CI on its own, but master no longer compiles the Rust volume server. Carry the volume id through.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-03 09:08:19 +08:00
35b090a4df volume: merge .ecj as a set union on EC shard copy + index recovery (Rust+Go) (#11554)
* volume: merge .ecj as a set union on EC shard copy + index recovery (Rust+Go)

An EC volume's deletion journal is a set of needle ids, but shard copy
and index recovery appended the peer's whole journal, doubling the file
on every ec_balance round trip. Fold the peer's ids in as a union
instead: only ids the local journal lacks are appended.

- The journal is never replaced. A mounted EcVolume merges a peer's ids
  through its live handle under the lock deletes take (Go
  MergeJournal / Rust merge_journal), wherever its journal lives.
- An unmounted journal gets only the missing ids appended while mounts
  are excluded; the delta is read outside the lock and re-read if the
  journal changed.
- The source .ecj streams into memory as an id set: no staging files,
  chunked reads, memory proportional to distinct ids.
- Go and Rust agree that a source journal exists when it sends a
  modified time or any bytes. A missing source stays a no-op.
- Rust runs every merge in spawn_blocking and shares one receive/merge
  path between shard copy and index recovery.

The decode path and the journal format are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: route .ecj merges to the runtime that holds the journal open

Disks sharing one index directory all resolved as the journal's owner, so
the last one won and a sibling's mounted runtime was skipped: the merge
appended behind its open handle and the sibling kept serving the peer's
deleted needles until remount. Callers now name the receiving disk by its
data directory; the merge goes through that disk's runtime, else a
sibling runtime whose journal is the target file.

In Go the unmounted append now holds every disk's EC lock (in location
order) while it rechecks for a mount, so a sibling mounting from this
disk's index during the unlocked read is merged through instead.

In Rust a mount that lands during the read is merged through directly and
its added count returned, rather than discarded and reported as zero.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: sync merged .ecj records outside the disks' EC locks

The unmounted merge held every disk's EC read lock across its fsync, so a
slow sync on one disk held off mounts on all of them, along with the EC
reads queued behind those mounts. Mounts only need to be excluded while
the records are written: the write now happens under the locks and the
fsync after they are released, since a later mount reads the written
records from the page cache. A failed fsync rolls back only if nothing
has mounted the journal or appended to it since the write.

A merge through a mounted volume now keeps only that volume's disk locked
across its fsync.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: roll back an unsynced .ecj merge through a volume mounted mid-sync

If a volume mounted after the unmounted merge wrote its records but before
the fsync failed, the rollback kept the records because the journal was now
open, leaving ids in the volume's deleted set that may never reach disk; a
retried merge then saw them and synced nothing. The rollback now goes
through that volume the way its own failed journal fsync does: truncate
back and drop the ids from the in-memory set, so a retry appends and syncs
them again. It still keeps the records if the volume journaled since, as
truncating would lose that delete. No fsync runs under the disk locks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: decide .ecj merge rollback from the journal's actual length

Two runtimes can hold one journal (cross-disk mounts). The rollback of an
unsynced merge checked one runtime's cached ecjFileSize, which another
runtime's appends leave stale, so it could truncate a delete that runtime
had already synced. The rollback now holds every holder's journal lock and
truncates only if the file's actual length is still the append's end,
then updates each holder's size and deleted set. Otherwise later records
follow the merged ones, so they stay and are rewritten in place and
synced outside the locks, rather than left possibly not durable.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: keep unsynced .ecj merge ids out of mounted deleted sets

When a merge's fsync failed, later records blocked the rollback, and the
rewrite-and-sync failed as well, the merged ids stayed in every mounted
volume's deleted set without being shown durable, so a retried merge saw
them as present and synced nothing. They now leave those sets while the
records stay in the file, matching DeleteNeedleFromEcx, which publishes an
id only after its record syncs. The merge returns the error and a retry
appends and syncs them again.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: publish merged .ecj ids to every holder of the journal

Two runtimes can journal into the same file when disks share an index
directory. The merge went through only the first holder, leaving a
sibling's in-memory deleted set without the ids, so it could keep
serving a needle the peer deleted until it remounted. Every holder of
the journal now gets the merged ids, in Go and in the volume server.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Publish merged .ecj ids to the journal actually written

mountedEcJournal prefers the receiving disk's own runtime for the vid,
whose journal may live in its data directory while the copied records
name a sibling's journal in the index directory. Publishing by the
requested ecjPath then marked a holder of a different file deleted on
records that file never persisted, resurrecting the needles on remount.
Publish by the picked runtime's journal path instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-02 22:46:36 +08:00
d8f926cf46 volume server: stream needle chunks without the store lock (#11488)
* volume server: read GET/HEAD needles off the store lock, and only once

The GET/HEAD handler read the needle synchronously on the tokio worker
while holding store.read(): first a stream-info read that loaded the
whole record just to parse its meta, then, for every needle that was not
streamed (small, compressed, chunk manifest, image ops), a second full
read. For a tiered volume each read is an S3 GET under the store lock,
and a writer queued behind it parks every other store reader.

The regular-volume read now runs in spawn_blocking. Under the store guard
it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat
handle or the remote backend, offset, size); the guard is dropped before
any needle data I/O. No data-file lease is held across the read either,
since a writer waits for one while holding the store write lock. The
index size decides the read, as in Go's readNeedle: a HEAD, a ranged read
or a needle above the stream threshold reads only its header and meta
tail (ReadNeedleMeta) and hands off to StreamingBody or the range path;
everything else is read in full once, with its checksum verified. A
compressed or manifest needle found by the meta read is then read in
full once. The range-from-source read also moves to spawn_blocking.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: stream needle chunks without the store lock

StreamingBody::poll_frame took store.read() and find_volume for every
chunk to compare the volume's compaction revision, dup'd the source
handle, and allocated a fresh chunk buffer. With -hasSlowRead=false the
stream also holds a data-file read lease for its whole life, while a
writer waits for that lease under store.write(): the next chunk's
store.read() then waits for the writer and the writer for the stream.

The per-chunk re-lookup was also wrong. The stream reads a handle opened
at plan time, which pins the .dat inode the offset was resolved against;
a vacuum commit renames a new file over .dat and leaves that inode
untouched. The re-looked-up offset belongs to the new file but was read
from the old inode, so a stream whose needle a vacuum moved ended in a
checksum error. The pinned offset stays valid, so the check, and with
it every store access, is dropped, along with the now unused
re_lookup_needle_data_offset and the revision fields of the read plan.

The source is shared as an Arc instead of dup'd per chunk, and the chunk
buffer is a BytesMut that the blocking read hands back with its result,
so its allocation is reclaimed once the previous frame has been written.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: stop a needle stream once its volume becomes unavailable

Taking the store lock out of StreamingBody also dropped its per-chunk
unavailable_error() check. With -hasSlowRead a writer can take the
data-file lease between chunks, fail its fsync and its truncate, and mark
the volume unavailable; the stream then kept serving the rest of the
needle from its pinned handle.

The volume's io_unavailable reason is now an Arc-shared leaf mutex that
the read plan hands to the stream. Each chunk checks it under its
data-file lease, where the writer marks it, and fails with the same
"volume is unavailable: <reason>" error the old check returned.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-01 23:15:38 +08:00
8a9563e53d volume: test that only the disk holding the replaced replica gets the free-slot credit (#11486)
* volume server: split volume_copy into phases and type the delete-after-status gate

volume_copy was one ~400-line handler, and the rule that an existing
local replica is deleted only after the source's ReadVolumeFileStatus
succeeded was held by statement order alone. The keep_remote_data=true
that the pre-copy delete and the failed-copy rollback must share was
kept in sync by a comment pointing from one to the other.

The handler is now a ~60-line orchestrator over connect_to_copy_source,
SourceVolumeStatus::fetch, delete_existing_replica,
plan_copy_destination and a VolumeCopyJob whose run() drives
preallocate_dat, transfer_files, finish_copied_files and
mount_and_reply, with cleanup_failed_copy on error.
delete_existing_replica takes a &SourceVolumeStatus, which only fetch
can construct (private field in a child module), so the delete cannot
be called before the status RPC. Both deletes go through
delete_replica_keep_remote.

Pure refactor: call order, status codes and messages, cancellation
checks, throttling, progress reports and cleanup are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: find space for a VolumeCopy before deleting the replica it replaces

VolumeCopy deleted an existing local replica as soon as the source answered
ReadVolumeFileStatus and only then looked for a location with room for the
copy. With no usable location (disk full, low-disk, wrong disk type) the call
errored after the delete, leaving the node with neither the old replica nor
the new one.

Plan the destination first, as Go does: find_free_location_replacing credits
the location holding the replaced volume with that volume's slot, so a disk at
its volume limit that holds the replica still accepts the copy. Only then
delete the replica and write the .note (still after the delete, as in Go).
delete_existing_replica now takes the planned CopyDestination, so the delete
cannot precede the plan. find_free_location_predicate keeps its behaviour.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: describe the replace-credit test against the current VolumeCopy flow

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-01 23:13:17 +08:00
eeec9ec09a volume server: refuse to compact a volume tiered to remote storage (#11545)
* volume server: refuse a tier move while compacting, and a commit once tiered

A tier move to remote and a vacuum compaction of the same volume could
interleave and leave the volume unreadable:

- A compaction committing while the upload ran swapped .dat/.idx under
  the transfer, which reopens the .dat by path per part. The move then
  published an object holding the old (or a mixed) layout against the
  compacted .idx, and with keep_local_dat_file=false deleted the only
  compacted .dat.
- A tier move finishing while the compaction copy ran (or between the
  copy and the commit) let the commit swap in the compacted .idx while
  the reload served the pre-compaction remote object through it.

The tier move now refuses to start while the volume is compacting, and
re-checks the compaction revision under the store write lock before it
records the remote file; on a mismatch it deletes the uploaded object
and fails with FailedPrecondition, leaving the volume local. Committing
a compaction on a volume that has a remote file is refused and its
.cpd/.cpx removed, since the reload would read the remote object
through the compacted index.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: abort a tier move whose volume was replaced or removed

The tier-up bookkeeping looked the volume up by id only and compared the
compaction revision. A delete and re-create of the same id during the upload
yields a fresh volume at the same revision, so the move recorded the old
volume's object on the new one and, without keep_local_dat_file, removed the
new .dat. An unmounted volume was skipped and the move reported success,
leaving the uploaded object referenced by nothing.

Capture the volume instance (its data-file access control Arc, as the scan
and read plans do) with the revision, and require both under the store write
lock. A replaced volume fails with FailedPrecondition, a missing one with
NotFound; either way nothing is recorded and the object is deleted after the
lock is released. Go fails in both cases because deleting or unmounting closes
the descriptor its copy reads.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: refuse to compact a volume tiered to remote storage

Committing a compaction of a tiered volume is refused, since the reload
would read the remote object through the compacted index. The compaction
itself still started: a tiered volume's data backend is the remote object
(the local .dat is dropped or deleted on tier-up), so an explicit vacuum
streamed the whole .dat out of remote storage into a .cpd that the commit
then discarded.

Refuse at the start of the compaction instead, before the .cpd is created,
at the point where Go's copy opens the local .dat. The truncated-index
test now uses a read-only local volume for its sorted index, since a
tiered one no longer reaches the copy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: match the ReadOnly(VolumeId) variant in write_volume_needles

#11543 matched VolumeError::ReadOnly as a unit variant in Store::write_volume_needles, and #11544 changed it to ReadOnly(VolumeId) in the same merge window. Each passed CI on its own, but master no longer compiles the Rust volume server. Carry the volume id through.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-01 23:11:16 +08:00
Eliah RusinandClaude Opus 5.5 2a42d56437 ecbalancer: honour total-shards-per-rack cap in Place / PlaceDurabilityFirst (#11553)
* ecbalancer: honour total-shards-per-rack cap in Place / PlaceDurabilityFirst

Worker auto-EC encode places via Topology.Place, which capped each shard
type independently (ceil(data/racks), ceil(parity/racks)). On an 8-rack
topology that permits 3 total shards on one rack, so losing two racks
strands 6/14 and a 10+4 volume becomes unreadable.

- tryPlace caps the total shards (data + parity) per rack in both modes,
  whether or not ReplicaPlacement is set.
- rackTotalCap picks the smallest per-rack total the racks' real room
  (free slots, bounded by the per-disk cap and node free slots, counting
  shards already placed) can satisfy. On a uniform cluster it is
  ceil(shards/racks); a nearly full rack raises it just enough that the
  cap alone never fails an encode.
- PlaceDurabilityFirst gets a last rung that drops the rack cap
  ("rack-total-cap" in Relaxed), so it fails only when no disk has room.
  PlaceStrict keeps the cap as a hard limit.
- chooseShardDest tries the next rack when the chosen one has no node
  that fits, and room checks count the per-disk cap, so a rack whose
  disks are all at the cap is no longer picked and then failed on
  (pre-existing: 3-node rack + single-disk rack failed at shard 9).
- Docs no longer claim the cap guarantees surviving rack loss; the
  placement error names the caps in effect; the encode warning no longer
  says replica placement when other constraints were relaxed.

place_rack_cap_test.go covers 10+4 over 8 racks (max 2/rack, 3/rack on
master), a starved rack, nearly full racks, the preferred-tag tier, the
full-disk rack, and rackTotalCap directly.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ecbalancer: size the rack total cap from room left under SameRackCount

The rack total cap counted each rack's free disk room, but attempts that
enforce ReplicaPlacement also stop a node at SameRackCount shards. With
SameRackCount=1, four one-node racks and four three-node racks got cap 2,
which fits only 12 of 14 shards: strict placement failed and
durability-first relaxed replica placement although 1 per small rack and
up to 3 per large rack fits.

Attempts that enforce ReplicaPlacement now use a cap sized from each
node's remaining SameRackCount allowance; attempts that relax it keep the
disk-room cap.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 22:22:34 +08:00
Eliah RusinandClaude Opus 5.5 164c3db606 s3: return 403, not 500, when an over-quota bucket refuses a write (#11552)
* s3: return 403, not 500, when an over-quota bucket refuses a write

Filer AssignVolume flattened ErrReadOnly into the free-text
AssignVolumeResponse.Error string, so S3 PutObject / PutObjectPart via
UploadReaderInChunks could not match it with errors.Is and fell through
to 500 InternalError: retryable, and it hides the quota.

Add FilerError READ_ONLY and AssignVolumeResponse.error_code, set it
alongside the unchanged error text, and rebuild the sentinel with
filer_pb.AssignVolumeResponseError. weed_server.ErrReadOnly now aliases
filer_pb.ErrReadOnly so errors.Is matches on both sides, and
mapChunkedUploadErrorToS3Error maps it to ErrAccessDenied. There is no
"read only" substring matching, so a volume server's "volume N is read
only" stays retryable.

Carrying the verdict as a response code rather than a gRPC status keeps
clients from treating it as a transport failure: the S3 gateway does not
fail over across filers and the Java client does not retry it.

Wrap per-chunk copy errors with %w so CopyObject keeps the sentinel, and
map UploadPartCopy chunk errors through mapCopyErrorToS3Error instead of
always returning 500.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci: re-run integration tests (PyPI download timeout)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 22:21:03 +08:00
6f9becaa37 volume server: answer BatchDelete on EC needles as Go does (#11541)
* volume server: VolumeNeedleStatus reads remote EC shards and reports deleted needles like Go

For an EC volume the handler read only locally mounted shards, so a node
that did not hold the shard with the needle's bytes answered Internal
"ec shard N not available locally". Go's ReadEcShardNeedle fetches the
interval from a peer or reconstructs it. It also mapped every regular
volume read error, including a tombstone, to NotFound "needle not found",
which fs.verify treats as a missing needle; Go returns ErrorDeleted as a
plain error ("already deleted"), which fs.verify skips.

The EC branch now drops the store guard and uses the distributed EC read
the HTTP GET path uses. Errors map like Go: needle absent -> NotFound
"needle not found <decimal id>", tombstoned (regular or EC .ecx/.ecj) ->
Unknown "already deleted", anything else -> Unknown with the error text.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: tell EC deletions and vanished volumes apart in VolumeNeedleStatus

The distributed EC reader returned Ok(None) for an absent needle, a needle
a peer reported deleted, and a volume unmounted after the handler's own
existence check. VolumeNeedleStatus answered all three NotFound "needle not
found", which fs.verify -pruneEntries counts as lost data. A reported
deletion was also lost when an earlier interval failed.

The reader now says why it has no needle (EcMiss: NotFound, Deleted,
VolumeNotFound), classifying the local tombstone itself and letting a
reported deletion outrank other interval errors, as Go's ReadEcShardNeedle
does. VolumeNeedleStatus maps Deleted to Unknown "already deleted" and
VolumeNotFound to "volume not found", and drops its separate EC pre-check.
read_ec_shard_needle_distributed keeps its Ok(None) for every miss, so the
other callers are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: answer BatchDelete on EC needles as Go does

With skip_cookie_check, which every weed/ client sends, an EC needle that
was already deleted came back 404 "ec needle <fid> not found". Go's
DeleteEcShardNeedle gets ErrorDeleted from its read and BatchDelete
answers 304 with no error; the filer's deletion classifier only forgives
"already deleted" or an exact "not found", so it booked the repeat delete
as a permanent failure. The same mode also compared the fid cookie and
refused chunk manifests with 406, while Go never reads the needle before
those checks when skipping, so the filer's delete of a manifest chunk's
own fid failed permanently too.

The EC branch now reads with read_ec_shard_needle_or_miss and answers as
Go: skipping, a deletion is 304 and any other miss is 500 with Go's text;
checking, every miss is 404 with Go's text ("already deleted",
"locate in local ec volume: FindNeedleFromEcx: needle not found",
"ec shard <vid> not found"). The cookie and manifest checks run only when
the caller asked for the cookie check, which leaves the non-EC path as it
was.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: match the ReadOnly(VolumeId) variant in write_volume_needles

#11543 matched VolumeError::ReadOnly as a unit variant in Store::write_volume_needles, and #11544 changed it to ReadOnly(VolumeId) in the same merge window. Each passed CI on its own, but master no longer compiles the Rust volume server. Carry the volume id through.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-01 22:17:18 +08:00
9d3907e36c volume: say "volume N is read only" like Go, so filer retries deletes (#11544)
VolumeError::ReadOnly displayed "volume is read-only". Go's store and
volume say "volume %d is read only", and the filer's deletion classifier
requeues a failed delete only when the error contains "is read only".
Against a Rust volume server a BatchDelete on a read-only volume (tier
move, maintenance) was booked as a permanent failure and the chunk was
never deleted.

ReadOnly now carries the volume id and displays Go's text. The text
reaches clients through BatchDelete results, the HTTP write and delete
error bodies, and gRPC statuses; the gRPC code (FailedPrecondition) and
the HTTP/BatchDelete status codes are unchanged.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-01 21:00:04 +08:00
bd953b0f84 volume server: group-commit fsync writes in the write queue (#11543)
* volume: split the write path into reusable steps

do_write_request ran its pre-append checks, the append, the sync
rollback, the index publish and the post-write bookkeeping inline, so a
batched write could only reuse it one needle at a time. Pull the steps
out (check_writable, prepare_write, undo_unsynced_append, publish_write,
finish_write) and the store's volume lookup plus disk-space check
(writable_volume_mut). do_write_request composes them in the same order
with the same early returns; no behaviour change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: group-commit fsync writes in the write queue

The write queue holds one store lock for a batch of up to 128 needles
but wrote them one at a time, so every fsync needle paid its own .dat
sync and its own .idx sync: 2N syncs per batch.

Add Volume::write_needles_grouped, after Go's processBatch. A volume's
entries are split into runs of distinct needle ids (a repeated id starts
a new run, so its dedup and cookie checks see the earlier write). A run
with a durable entry appends everything with append_at_ns chained
through a local, syncs the .dat once, and only then publishes the
entries and syncs the .idx once. A failed .dat sync truncates the .dat
back to the run start (marking the volume unavailable if that fails),
leaves last_append_at_ns and last_modified untouched, and fails every
entry of the run. Runs with no durable entry go through the unchanged
per-needle path. Store::write_volume_needles is the queue's entry point;
the handlers' non-queue path is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: fail closed when a failed append's rollback fails

append_needle discarded the truncate-back result, so a partial write
that could not be rolled back left unindexed bytes on the .dat while
the volume stayed writable; the next append would bury them mid-file,
past the load-time tail check. Route the rollback through
undo_unsynced_append, which marks the volume unavailable when the
truncate fails, so nothing more is appended over an unverified tail.

* volume server: keep a grouped run's I/O error streak from later appends

A synced run stages every append before any entry finishes, so the
success reset in finish_write ran after the failed appends queued
behind the last write to land and erased their media-error streak.
Sent one at a time, those errors would have counted and quarantined
the volume. Skip the reset when an append after the last landed write
added to the streak.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: replay a grouped run's I/O error streak in queue order

Skipping the run's success reset whenever an append after the last
landed write failed kept the errors from before that write as well,
so a run like [EIO, EIO, landed, EIO] reached the quarantine count
that the same writes one at a time (one error) do not. Mark the
streak where each entry is staged and record the run's success at
the last landed write's mark: errors before it are cleared, the ones
after it still count.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: replay a run's I/O error streak in one atomic step

Reads record their outcomes on the tracker without the volume's write
lock, so record_success_at's separate load and store could drop an
error a read counted in between, or restore errors a read had just
cleared. Keep the count and the clear counter in one atomic word and
apply the replay with a single fetch_update.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-01 20:59:40 +08:00
Eliah RusinandClaude Opus 5.5 7944cb4ba2 volume server: keep repeated response headers when proxying a read, like Go (#11542)
In readMode=proxy, proxy_request copied the target's response headers
with HeaderMap::insert, so a header the target sent more than once
(several Set-Cookie, Vary, Link, ...) reached the client with only its
last value. Go's proxyReqToTargetServer adds every value with
w.Header().Add. Append instead of insert; the Server header is still
dropped and status and body handling are unchanged.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 20:58:50 +08:00
15d3c65e0c volume server: take the heartbeat's remaining store reads off the runtime (#11540)
* volume server: collect EC heartbeats and adjust volume max off the runtime

The volume pass moved to the blocking pool, but the heartbeat task still
called collect_ec_heartbeat and the following EC shard snapshot, and
Store::maybe_adjust_volume_max, directly on its tokio worker.
maybe_adjust_volume_max runs statvfs on every auto-sized disk and stats the .dat
of every writable volume under the store read lock. All of them block the
worker on the node-wide RwLock<Store> whenever a writer holds it or is
queued, and every task sharing that worker stalls with it.

Run the adjustment, on the pulse and after the master changes volume
options, and the EC tick's heartbeat plus shard snapshot through
off_runtime, like the volume pass. apply_master_volume_options now only
reports whether the options changed; the loop adjusts off the runtime.
What is collected and sent, and in what order, is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: keep EC shard deltas out of the volume heartbeat's snapshot

EC shard deltas are the diff between the store's shards and last_ec_shards,
taken when volume_state_notify fires. But the volume tick and the
options-changed heartbeat re-took last_ec_shards from the store too, and a
volume heartbeat carries no shard list: a mount or unmount that landed while
the notify was pending or the volume pass was collecting was absorbed into
the baseline and never sent. The master only learned of it at the next EC
tick, 17 pulses later. The EC tick likewise built its heartbeat and its
baseline under two separate store guards, so a mount between them was lost
the same way.

A volume heartbeat now only takes out of the baseline the expired EC shards
it reports deleted itself, so the next delta does not repeat them. The EC
tick, and the initial EC heartbeat, build the full list and the baseline
under one read guard, still on the blocking pool.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: take the heartbeat's remaining store reads off the runtime

The heartbeat's volume and EC passes and the volume max adjustment already
run on the blocking pool, but several reads of the node-wide RwLock<Store>
were still taken directly on the heartbeat's tokio worker: the digest
report reset before the first heartbeat, the duplicate-UUID directory
lookup and the volume options a master response carries, the EC shard list
a state notification is diffed against, and the deregistration heartbeat
sent on stop and shutdown. The lock is writer-preferring, so with a writer
holding or queued for it each of these parks the worker, and every task
sharing that worker stalls with it.

Run each through off_runtime, which now takes a closure so a pass can carry
what it needs from the master's response. The notify branch's volume
snapshot and EC read become one blocking pass, still under two guards in
the same order. What is collected and sent, and in what order, is
unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-01 20:58:01 +08:00
Eliah RusinandClaude Opus 5.5 fea14c01a7 volume server: refuse a tier move while compacting, and a commit once tiered (#11539)
* volume server: refuse a tier move while compacting, and a commit once tiered

A tier move to remote and a vacuum compaction of the same volume could
interleave and leave the volume unreadable:

- A compaction committing while the upload ran swapped .dat/.idx under
  the transfer, which reopens the .dat by path per part. The move then
  published an object holding the old (or a mixed) layout against the
  compacted .idx, and with keep_local_dat_file=false deleted the only
  compacted .dat.
- A tier move finishing while the compaction copy ran (or between the
  copy and the commit) let the commit swap in the compacted .idx while
  the reload served the pre-compaction remote object through it.

The tier move now refuses to start while the volume is compacting, and
re-checks the compaction revision under the store write lock before it
records the remote file; on a mismatch it deletes the uploaded object
and fails with FailedPrecondition, leaving the volume local. Committing
a compaction on a volume that has a remote file is refused and its
.cpd/.cpx removed, since the reload would read the remote object
through the compacted index.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: abort a tier move whose volume was replaced or removed

The tier-up bookkeeping looked the volume up by id only and compared the
compaction revision. A delete and re-create of the same id during the upload
yields a fresh volume at the same revision, so the move recorded the old
volume's object on the new one and, without keep_local_dat_file, removed the
new .dat. An unmounted volume was skipped and the move reported success,
leaving the uploaded object referenced by nothing.

Capture the volume instance (its data-file access control Arc, as the scan
and read plans do) with the revision, and require both under the store write
lock. A replaced volume fails with FailedPrecondition, a missing one with
NotFound; either way nothing is recorded and the object is deleted after the
lock is released. Go fails in both cases because deleting or unmounting closes
the descriptor its copy reads.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 20:20:16 +08:00
Eliah RusinandClaude Opus 5.5 f2498e122a volume: read EC shards fully, like Go's ReadAt (#11537)
EcVolumeShard::read_at and the scrub plan's EcLocalShard::read_at were a
single pread/seek_read. That may legally return fewer bytes than asked
mid-file (FUSE/NFS/CIFS mounts, a signal, very large requests), and an
Interrupted error was not retried. Callers treat a short count as end of
file or corruption: verify_ec_shards compared a zero tail and reported a
parity mismatch, local scrub reported a broken shard, VolumeEcShardRead
ended the stream early, and decode/rebuild/local needle reads failed.

Add storage::io::read_full_at, which loops until the buffer is full or a
read returns 0 and retries Interrupted, so a short count means EOF. Route
both shard read_at methods through it, replace the encoder's private
read_at_most with it, and reuse it for the Windows read_exact_at loop.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 20:19:43 +08:00
Eliah RusinandClaude Opus 5.5 62481f1673 volume server: ec.decode reads the .ecx from the index dir it was copied to (#11536)
* volume server: ec.decode reads the .ecx from the index dir it was copied to

VolumeEcShardsCopy writes the .ecx/.ecj into the receiver's -dir.idx, so
with a split data/index dir the decode target has no .ecx beside its
shards. VolumeEcShardsToVolume sized the .dat from the right .ecx but
built the .idx from the data dir, failing with NotFound after the .dat
was already published. It now reads .ecx/.ecj from where the EC volume
opened them and writes the .idx beside the .dat, where Go leaves it.

The live-entry check and the .dat size also ignored deletions recorded
only in the .ecj, which Go folds into the .ecx (RebuildEcxFile) first:
a fully deleted volume was decoded instead of reported as having no live
entries, and deleted tail needles were copied into the .dat. Both now
treat journaled ids as deleted, without rewriting the sealed .ecx.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode keeps the decoded volume writable and reads every .ecj

The rebuilt .idx copied a journaled tail needle's .ecx row verbatim after
the .dat was cut short before it, so the mount saw a row past EOF and
marked the decoded volume read-only. Rows of deleted needles the .dat no
longer holds are now dropped, and each journaled needle still in the .dat
gets one tombstone instead of one per journal entry.

VolumeEcShardsCopy appends journals collected from other holders into
the idx dir, but the decode read only the .ecj beside the .ecx, which
sits in the data dir when this server generated the shards. It now
reads both, once, in bounded chunks via the loader EcVolume uses.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: test ec.decode drops a sealed .ecx tail tombstone

Covers the other half of the rule added in the previous commit: a tail
needle tombstoned in the .ecx itself (Go's RebuildEcxFile) is cut from
the .dat, and its row must not reach the rebuilt .idx either.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 20:19:17 +08:00
Eliah RusinandClaude Opus 5.5 b8f074b7d3 volume server: VolumeNeedleStatus reads remote EC shards and reports deleted needles like Go (#11535)
* volume server: VolumeNeedleStatus reads remote EC shards and reports deleted needles like Go

For an EC volume the handler read only locally mounted shards, so a node
that did not hold the shard with the needle's bytes answered Internal
"ec shard N not available locally". Go's ReadEcShardNeedle fetches the
interval from a peer or reconstructs it. It also mapped every regular
volume read error, including a tombstone, to NotFound "needle not found",
which fs.verify treats as a missing needle; Go returns ErrorDeleted as a
plain error ("already deleted"), which fs.verify skips.

The EC branch now drops the store guard and uses the distributed EC read
the HTTP GET path uses. Errors map like Go: needle absent -> NotFound
"needle not found <decimal id>", tombstoned (regular or EC .ecx/.ecj) ->
Unknown "already deleted", anything else -> Unknown with the error text.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: tell EC deletions and vanished volumes apart in VolumeNeedleStatus

The distributed EC reader returned Ok(None) for an absent needle, a needle
a peer reported deleted, and a volume unmounted after the handler's own
existence check. VolumeNeedleStatus answered all three NotFound "needle not
found", which fs.verify -pruneEntries counts as lost data. A reported
deletion was also lost when an earlier interval failed.

The reader now says why it has no needle (EcMiss: NotFound, Deleted,
VolumeNotFound), classifying the local tombstone itself and letting a
reported deletion outrank other interval errors, as Go's ReadEcShardNeedle
does. VolumeNeedleStatus maps Deleted to Unknown "already deleted" and
VolumeNotFound to "volume not found", and drops its separate EC pre-check.
read_ec_shard_needle_distributed keeps its Ok(None) for every miss, so the
other callers are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 20:18:34 +08:00
Eliah RusinandClaude Opus 5.5 cc1ec48151 volume server: collect EC heartbeats and adjust volume max off the runtime (#11532)
The volume pass moved to the blocking pool, but the heartbeat task still
called collect_ec_heartbeat and the following EC shard snapshot, and
Store::maybe_adjust_volume_max, directly on its tokio worker.
maybe_adjust_volume_max runs statvfs on every auto-sized disk and stats the .dat
of every writable volume under the store read lock. All of them block the
worker on the node-wide RwLock<Store> whenever a writer holds it or is
queued, and every task sharing that worker stalls with it.

Run the adjustment, on the pulse and after the master changes volume
options, and the EC tick's heartbeat plus shard snapshot through
off_runtime, like the volume pass. apply_master_volume_options now only
reports whether the options changed; the loop adjusts off the runtime.
What is collected and sent, and in what order, is unchanged.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 20:14:15 +08:00
Eliah RusinandClaude Opus 5.5 5ece8dd63c volume server: drop the unused unmount result in VolumeCopy validation, and test-only EC helpers (#11531)
mount_and_reply ignored the Result of store.unmount_volume when a copied
replica failed record count validation, tripping unused_must_use. The Err
branch is unreachable there: the volume was mounted under the same store
write guard, mount_volume refuses an already loaded vid so it is a fresh
Volume with is_compacting false, and a compaction claim needs &mut Volume,
i.e. the store lock. Ignore the result explicitly with a one-line reason.

Store::delete_expired_ec_volumes and Store::remove_ec_volume are called
only from test modules (the heartbeat uses the split
find_expired_ec_volumes / remove_expired_ec_volumes halves), so mark them
#[cfg(test)].

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 20:13:45 +08:00
e57f8c4d87 volume server: read GET/HEAD needles off the store lock, and only once (#11487)
The GET/HEAD handler read the needle synchronously on the tokio worker
while holding store.read(): first a stream-info read that loaded the
whole record just to parse its meta, then, for every needle that was not
streamed (small, compressed, chunk manifest, image ops), a second full
read. For a tiered volume each read is an S3 GET under the store lock,
and a writer queued behind it parks every other store reader.

The regular-volume read now runs in spawn_blocking. Under the store guard
it only resolves a NeedleReadPlan (index lookup, a freshly opened .dat
handle or the remote backend, offset, size); the guard is dropped before
any needle data I/O. No data-file lease is held across the read either,
since a writer waits for one while holding the store write lock. The
index size decides the read, as in Go's readNeedle: a HEAD, a ranged read
or a needle above the stream threshold reads only its header and meta
tail (ReadNeedleMeta) and hands off to StreamingBody or the range path;
everything else is read in full once, with its checksum verified. A
compressed or manifest needle found by the meta read is then read in
full once. The range-from-source read also moves to spawn_blocking.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-09-27 20:13:15 +08:00
67691a1eea volume server: split volume_copy into phases and type the delete-after-status gate (#11485)
volume_copy was one ~400-line handler, and the rule that an existing
local replica is deleted only after the source's ReadVolumeFileStatus
succeeded was held by statement order alone. The keep_remote_data=true
that the pre-copy delete and the failed-copy rollback must share was
kept in sync by a comment pointing from one to the other.

The handler is now a ~60-line orchestrator over connect_to_copy_source,
SourceVolumeStatus::fetch, delete_existing_replica,
plan_copy_destination and a VolumeCopyJob whose run() drives
preallocate_dat, transfer_files, finish_copied_files and
mount_and_reply, with cleanup_failed_copy on error.
delete_existing_replica takes a &SourceVolumeStatus, which only fetch
can construct (private field in a child module), so the delete cannot
be called before the status RPC. Both deletes go through
delete_replica_keep_remote.

Pure refactor: call order, status codes and messages, cancellation
checks, throttling, progress reports and cleanup are unchanged.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-09-27 20:12:55 +08:00
Eliah RusinandClaude Opus 5.5 68944e83a3 volume: typed tier errors so a missing remote object answers NotFound (#11484)
remote_storage/s3_tier.rs returned Result<_, String> from every
transfer (upload_file, download_file, read_range[_blocking],
delete_file[_blocking]) and from the tier runtime helpers. The tier
move handlers could only wrap that in Status::internal, so a .dat whose
remote object is gone was indistinguishable from an I/O failure to
weed shell.

Add TierError { NotFound, Io, RuntimeUnavailable, Aborted }. Each
variant carries the existing message verbatim. NotFound follows the
rules remote_storage/s3.rs already uses: raw 404 status on HEAD,
NoSuchKey code on GET; a bare 404 on GET stays Io. A progress-callback
Err becomes Aborted. VolumeError gains a transparent Tier variant and
From<VolumeError> for Status maps Tier(NotFound) to NotFound; the tier
move handlers go through status_with_context, so their message text is
unchanged. Every other tier failure is still Internal.

The remote needle read path keeps io::Error::other, so its error kind
and vacuum's handling of it do not change.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 19:39:25 +08:00
Eliah RusinandClaude Opus 5.5 00310f6588 volume server: run the vacuum compaction copy without the store lock (#11482)
* volume server: run the vacuum compaction copy without the store lock

VacuumVolumeCompact held the store write lock for the whole live-needle
copy, including every progress blocking_send on the 16-deep stream. On a
large volume that is minutes with every read, write and heartbeat on the
node parked behind it, long enough for the master to unregister the node.

Split compaction the way Go's CompactByIndex runs it. A short locked step
claims the volume's compacting flag, records the makeup_diff watermark
(index size and compaction revision) and opens fresh .dat/.idx handles.
The copy then replays .idx up to the watermark and copies from those
handles with the store lock released; writes that land meanwhile are
replayed by makeup_diff at commit, as before. The flag is an Arc<AtomicBool>
released when the job is dropped, so every exit path clears it.

Because the flag is now visible to other callers, the operations that would
pull the files out from under the copy refuse while it is set: unmount
(and VolumeConfigure, which unmounts and remounts), delete (checked before
the volume is removed from the map, which a refused destroy used to leave
unmounted), cleanup, and index relocation. A second compact and a commit
stay no-ops, as in Go. The pre-copy fsync is dropped: the copy reads its
own handles through the page cache and .cpd/.cpx are fsynced before commit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: keep a read-only in-memory index's size for the compaction copy

The unlocked copy replays .idx up to index_file_size(). A read-only
volume whose .sdx could not be built loads its index into memory without
a writer, so that size stayed 0: the copy came out empty and the commit
replaced the volume with it. CompactNeedleMap::load_from_idx now records
the rows it loaded, which is also what Go's IndexFileSize reports for a
read-only index.

The copy's index replay now stops reading at the recorded size instead
of walking rows appended since, which makeup_diff replays anyway.

Adds tests for compacting a read-only volume on both the sorted index
and the in-memory fallback, and for VolumeConfigure stopping when the
unmount is refused during a copy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: stop a vacuum copy as soon as its client is gone

The progress callback only noticed a closed response stream when a report
was due, every 128 MiB. With the copy now running outside the store lock,
a copy nobody waits for keeps the volume marked compacting and so keeps
refusing unmount, delete and cleanup until that next report. Check the
stream on every callback.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 19:38:59 +08:00
Eliah RusinandClaude Opus 5.5 5c9c424a84 volume server: stream ReadAllNeedles without holding the store lock (#11481)
* volume server: stream ReadAllNeedles without holding the store lock

read_all_needles held store.read() while Volume::read_all_needles read
every live needle of the volume into a Vec, and kept holding it through
the whole blocking_send loop. Memory grew with the volume, and a slow
client parked the scan in a send with the guard held; needle writes and
the heartbeat take store.write() on a writer-preferring lock, so the node
stopped serving until the client caught up.

Take a DatScanPlan (fresh .dat open, end bound) under a short guard and
walk it with the guard released, sending one needle at a time. Each
record is checked against the live needle map under a brief read guard,
as the scan reaches it, and only a live record is parsed, so a damaged
stale copy does not fail the stream. Records appended while a pass ran
are walked by a follow-up plan, so a needle overwritten during the scan
is streamed once, as its new copy. A vacuum commit or re-create of the
volume during the scan fails the stream, since the map's offsets no
longer describe the pinned file; the plan carries the volume instance
and compaction revision for that check.

DatScanPlan::scan_records yields records unparsed; scan keeps its
behaviour on top of it. Volume::read_all_needles has no caller left and
is removed; its tests move to the RPC.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: check ReadAllNeedles liveness only once the send can proceed

The per-record liveness check ran before blocking_send, so a scan parked
on a full channel held a record it had already judged live. An overwrite
landing during that park left the old copy in the stream, and the
continuation over appended records then streamed the new copy as well.

Reserve channel space first, then take the store read guard, check the
record against the needle map and enqueue it through the permit before
releasing the guard. The wait for space still happens without the lock;
the record is parsed before the guard is taken, and its parse error only
counts if the record turns out to be live.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 19:38:43 +08:00
Eliah RusinandClaude Opus 5.5 5218e68554 volume server: collect heartbeats under the store read lock, off the runtime (#11480)
* volume server: collect heartbeats under the store read lock, off the runtime

Every pulse tick, options change and volume-state notification took
store.write() for a whole heartbeat pass, directly on the async heartbeat
task. The pass fstats every volume's .dat twice and hashes its report, so
on a server with many volumes it held the store exclusively for the whole
scan: reads and writes stalled, and with the writer-preferring RwLock a
pending pass parked every new reader too.

The pass only needs to mutate the store for a few rare actions: removing
expired EC volumes, deleting expired volumes past their removal delay, and
setting no-write on IO-quarantined volumes. It now runs under store.read(),
records those as (disk, volume id) actions, and applies them afterwards
under a short store.write() that is only taken when there is something to
do. Each action re-checks its target under the write lock, so a volume
written to, replaced or removed in between is left alone. Expired EC
volumes are still removed before the volume pass, as before, because the
EC shard count feeds the disk-space-low max volume count. Every pass runs
on the blocking pool via spawn_blocking.

The heartbeat message is unchanged for the same store state.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: take has_no_ec_shards with the heartbeat's volume list

The heartbeat pass took has_no_ec_shards from the EC phase's read lock,
then built the volume list under a second one. An EC shard mounted in
between went out as "no EC shards" beside a volume list taken after the
mount, and the master clears a server's EC registrations on that flag.

has_no_ec_shards is now computed under the same read lock as the volume
list, with the EC phase's filter: not expired, not quarantined, at least
one shard.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 19:38:23 +08:00
f7680cf812 volume: one long-lived tokio runtime for blocking tiered S3 reads (#11448)
* volume: one long-lived tokio runtime for blocking tiered S3 reads

block_on_tier_future, behind read_range_blocking and delete_file_blocking,
spawned an OS thread and built a fresh current-thread tokio runtime on every
call, then tore the runtime down. On an S3-tiered volume that is once per
needle read, per streamed 64 KiB chunk, per DatScanPlan record and per
destroy. The SDK client's pooled HTTPS connections are driven by tasks on the
runtime a request ran on, so each teardown dropped the pool and every call
re-dialed and re-handshook TLS. A panic inside the SDK was also flattened to
the fixed string "tier runtime thread panicked".

Now one process-wide runtime (OnceLock, multi_thread, 2 workers named
tier-io) drives all tier I/O; block_on_tier_future spawns onto it and parks
the caller on an mpsc channel for the JoinHandle result. Blocking the caller
is unavoidable (the storage layer is synchronous) and is what the old code
did through thread::spawn().join(). Handle::block_on is not used because the
wrappers are also reached from inside another runtime's worker, where it
panics with "Cannot start a runtime from within a runtime". JoinError panics
are downcast to &str/String and the payload is kept in the error.

Tests cover runtime reuse (Handle::id equal across calls, thread name
tier-io), calls from a std thread, from spawn_blocking, and directly from
current-thread and multi-thread runtime contexts, and the panic payload.
Against the old body 7 of 9 fail.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume: return a tier runtime build failure instead of panicking

Review follow-up. tier_runtime() expect'ed the runtime build, so an OS
refusing threads panicked inside Volume::destroy (after the volume left
the in-memory map, before its files were removed) and inside needle
reads, bypassing their error paths. Keep the runtime in a
Mutex<Option<Runtime>> behind tier_handle() -> Result<Handle, String>:
a failed build is returned to the caller through block_on_tier_future's
existing Result and is not cached, so a later call retries once the
pressure is gone. The lock is held only while building.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume: trim comments on the shared tier I/O runtime

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 12:00:50 +08:00
4299fdf578 volume server: VolumeEcShardsDelete full teardown unloads every disk and keeps the shard gauge honest (#11446)
* volume server: VolumeEcShardsDelete full teardown unloads every disk and keeps the shard gauge honest

Go's VolumeEcShardsDelete full teardown calls vs.store.UnloadEcVolume in
the blanket path (weed/server/volume_grpc_erasure_coding.go:488) and
location.UnloadEcVolume in the generation-fenced path (:511): each disk
that had the volume registered drops it, closes its shard descriptors
and gives back its ec_shards gauge before the artifacts are unlinked.

The Rust handler used Store::remove_ec_volume / DiskLocation::remove_ec_volume
instead, which only remove the map entry. Store::remove_ec_volume also
stops at the FIRST disk holding the vid, so on a split-disk volume
(shards on several disks) the blanket teardown left the sibling disks'
EcVolume registered with open fds while the unlink loop deleted their
files underneath it: the heartbeat kept advertising shards whose files
were gone, the inodes stayed pinned by the open descriptors, and the
VOLUME_GAUGE{collection,"ec_shards"} never came back down. The fenced
path leaked the gauge and the descriptors the same way on the one disk
it wiped.

Both paths now use the unload_ec_volume helpers from #11413 (every disk
for the blanket teardown, the strictly-older disk for the fenced one),
and the two Status::internal messages name the disk directory like Go's
"... on %s: %w".

Regression tests build a two-disk store with the same vid mounted on
each disk (the SplitDiskEcFixture, which gains a collection knob so the
gauge read is isolated from parallel tests mounting under "") and assert
that a blanket teardown leaves no EcVolume registered on any disk and
returns the gauge to its pre-mount value, and that a fenced teardown
decrements the gauge for the older disk's shard while preserving the
newer disk. Both fail against the previous handler.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume server: trim comments on the EC full-teardown unload path

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 12:00:23 +08:00
3e9fc9e75b volume: reject invalid -minFreeSpace and duration flags at startup like Go (#11447)
* volume: reject invalid -minFreeSpace and duration flags at startup like Go

Symptom: the Rust volume server swallowed invalid values for three
startup flags and ran with a limit the operator did not ask for.
-minFreeSpace=150 became 150 bytes (Go: fatal, percent must be 0..=100),
-minFreeSpace=10GiBx and -minFreeSpace=50B fell through to the 1%
default (Go: fatal), and -inflightUploadDataTimeout=30sec, =abc or =30
silently became the 60 s default (Go: flag.Duration usage error, exit 2).
Go's byte-size units (1,024MB, 42 mib, 100Ki, 1eb) were also not
understood and mapped to 1%.

Go reference: weed/util/minfreespace.go:73-106 (MustParseMinFreeSpace /
ParseMinFreeSpace), weed/util/bytes.go:212-243 (ParseBytes and
bytesSizeTable), weed/command/volume.go:226-227 (flag.Duration, i.e.
time.ParseDuration grammar).

Fix: the two inflight timeout Cli fields are now std::time::Duration
with a clap value_parser that ports time.ParseDuration (units ns/us/µs/
μs/ms/s/m/h, fractions, bare 0, int64-nanosecond overflow), so clap
prints a usage error naming the value and exits 2 at the same layer as
Go's flag package. parse_min_free_spaces returns Result and is built
from Go-parity parse_min_free_space / parse_bytes; resolve_config_with_env
aborts with Go's "The value specified in -minFreeSpace not a valid
value" message, the fail-fast pattern it already uses for --max, --index
and --readMode. No new dependency. Negative durations and NaN percent
are rejected explicitly (Go quirks with no sane Rust representation).

Tests cover the Go fixtures from minfreespace_test.go and bytes_test.go,
the duration grammar, and flag-level rejection through Cli::try_parse_from
and resolve_config_with_env; the flag-level tests were confirmed to fail
against the previous parser.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume: say that -minFreeSpace commas separate directories, never thousands

Review follow-up. parse_bytes keeps Go's ParseBytes thousands-comma
support, but the flag is split on commas into per-directory entries
before any entry is parsed (Go's MustParseMinFreeSpace does the same),
so -minFreeSpace=1,024MB is the two entries 1 and 024MB. The doc
comment and the single-entry tests advertised the comma form as if it
were reachable from the flag. Document the split and add a flag-level
test: two entries from parse_min_free_spaces, and resolve_config
aborting on the directory-count mismatch with one -dir.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume: trim comments on the strict flag parsing

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 12:00:06 +08:00
83d44be0f3 volume: detect S3 not-found by typed SDK errors, not the "service error" string (#11444)
* volume: detect S3 not-found by typed SDK errors, not the "service error" string

remote_storage/s3.rs decided ObjectNotFound by matching the Display
output of an aws_sdk_s3 SdkError against "NoSuchKey" / "404" /
"NotFound". In the locked SDK (aws-smithy-runtime-api 1.11.6,
src/client/result.rs:487-497) that Display is a fixed string per
variant, "service error" for every S3 error, so ObjectNotFound was
unreachable: every missing remote object surfaced as
Other("s3 get object: service error") with the real cause discarded.

Go (weed/remote_storage/s3/s3_storage_client.go) uses typed checks:
HEAD (373-374): awserr.RequestFailure with StatusCode() == 404;
GET (436-437): awserr.Error with Code() == s3.ErrCodeNoSuchKey.

read_file now matches SdkError::ServiceError whose GetObjectError
is_no_such_key(); a bare 404 on GET stays a generic error, as in Go.
stat_file matches HeadObjectError::is_not_found() or a raw HTTP 404
status, Go's actual condition. Non-service errors fall through to
Other unchanged. Every SdkError message in s3.rs and s3_tier.rs is
formatted with DisplayErrorContext so the S3 error code and message
survive instead of "service error".

Six network-free unit tests drive the client through a canned
HttpClient (404 NoSuchKey, bare 404 on GET and HEAD, 404 with a
foreign body on HEAD, 403 AccessDenied on GET and HEAD). They need
aws-smithy-runtime-api as a dev-dependency; it is already in the lock
at a single version, so no new crates.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume: HEAD not-found is the raw 404 status alone, as in Go

Review follow-up. The HEAD arm also accepted the SDK's NotFound error
code on any status, so a 400 carrying <Code>NotFound</Code> became a
missing object. Go's stat looks only at RequestFailure.StatusCode() ==
404 (weed/remote_storage/s3/s3_storage_client.go:373); do the same. The
raw status still covers the body-less 404 the SDK turns into NotFound
and a 404 whose body names a foreign code. Regression test for the
non-404 NotFound body, which failed against the previous arm.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume: trim comments on the typed S3 not-found checks

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: franchb <ru.balloo@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 11:59:09 +08:00
df4995b894 volume server: VolumeMarkReadonly answers NotFound when the volume vanished under the lock (#11443)
* volume server: VolumeMarkReadonly answers NotFound when the volume vanished under the lock

make_volume_readonly looked the volume up, notified the master (step 1),
then took the store write lock (step 2) and marked the volume only `if
let Some(..)`. When the volume left the store during step 1 -- a master
round trip, during which an unmount or a heartbeat expiry can land --
the missing else meant the RPC reported success for a volume the server
no longer has, and step 3 told the master again that it is read-only.

Go's Store.MarkVolumeReadonly (weed/storage/store.go) returns
"volume %d not found" when findVolume comes back nil, and
makeVolumeReadonly (weed/server/volume_grpc_admin.go) returns that error
before the step-3 notification. The Rust step 2 now does the same:
find_volume_mut(vid) -> Status::not_found("volume {vid} not found"), and
the `?` skips step 3, as it already did for a set_read_only_persist
failure. The scrub caller already matches NotFound to skip such a
volume instead of failing the whole report; it now actually gets it.
volume_mark_writable already returns NotFound under its write lock.

The regression test opens the step-1 window deterministically: step 1
awaits the current_master_url read lock, so the test holds its write
guard, lets make_volume_readonly park there after its own lookup
succeeded, unmounts the volume, then releases the guard. With no master
configured the notification is a no-op, so the write lock in step 2 is
the only place left that can notice the volume is gone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume server: trim comments on the vanished-volume mark-readonly path

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-25 22:05:53 +08:00
bb9942c646 volume server: sweep stale EC artifacts before VolumeEcShardsGenerate re-encodes (#11413)
* volume server: sweep stale EC artifacts before VolumeEcShardsGenerate re-encodes

The Rust VolumeEcShardsGenerate went straight into write_ec_files: no unload
of an already-mounted EC volume and no stale-artifact sweep. Only .ec00..ecNN
on the encoding disk were truncated, so a retry could mix two encode runs. A
stale N.ec03 left on a sibling disk survived, reconcile later mounted it
against the new .ecx, and the new .vif made the encode_ts_ns identity guard
pass, so reads served old-run bytes at new-run offsets.

Mirror Go's VolumeEcShardsGenerate (#9880 / #9953): UnloadEcVolume on every
disk, then removeStaleEcArtifacts on every disk location before encoding.
remove_ec_volume_files_full_teardown already has removeStaleEcArtifacts'
semantics (.ec00..ec31, .ecx/.ecj/.ecsum[.vN] in both the data and idx dirs,
.vif only on a shard-only disk; never the source .dat/.idx), so reuse it. Add
Store::unload_ec_volume, which unlike remove_ec_volume does not stop at the
first disk and closes the descriptors so the unlink frees the inodes. The
store write lock covers only unload + sweep, not the encode.

The failure arm now also drops the generation-0 .ecsum, as Go's defer does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume server: wake the heartbeat after VolumeEcShardsGenerate unloads shards

The pre-encode unload drops mounted EC shards from memory, but unlike every
other unmount path it did not wake the heartbeat, so the master kept routing
reads to shards this server no longer serves until the next pulse. Notify
once the store lock is released, and before the sweep error propagates: a
failed sweep has unloaded the shards too.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume server: clean up encode artifacts when the .vif write fails too

Go's shouldCleanup defer covers every error before the .vif commits,
not just a failed encode. A serialize or write failure on the .vif left
the fresh .ecNN/.ecx/.ecsum behind, which the next generate would have
to rely on the new sweep to remove. Extract the cleanup and run it on
the .vif error paths as well.

* volume server: write the EC .vif atomically

Go's SaveVolumeInfo writes a temp file, syncs it, and renames it over
the target, so a failed write leaves the previous metadata intact and a
read-only .vif fails the save. The direct fs::write truncated the file
first, so a write or sync failure could leave an empty .vif even after
cleanup_encode removed the generated shards.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-09-24 07:10:54 +08:00
f0afcf904d volume: an EC volume needs an .ecx to mount, and a 0-byte stub never outranks a real index (#11415)
* volume: an EC volume needs a non-empty .ecx to mount

Two gaps against Go in how the Rust volume server treats the .ecx.

EcVolume::new mounted with no index at all. The per-shard
VolumeEcShardsMount path picks the disk by shard file alone, so a shard
whose .ecx was on no local directory still registered and was
advertised to the master; every VolumeEcShardRead then failed with
"ecx file not open", and add_shard's 0-byte guard was neutralised
because ecx_file_size stayed 0. Go's NewEcVolume returns an error
wrapping os.ErrNotExist. EcVolume::new now fails with NotFound, and
Store::mount_ec_shard looks up the .ecx owner across all disks first
(findEcxIdxDirForVolume) so a shard on a sibling disk of its index
still mounts instead of turning into a hard failure.

A 0-byte .ecx stub, as left by a failed EC distribute copy, counted as
a valid index. Go requires Size() > 0 wherever the file steers a
decision: HasEcxFileOnDisk, findEcxIdxDirForVolume, indexEcxOwners
(shared by reconcile and mirror), and VolumeEcShardsCopy removes a
copied 0-byte .ecx and fails the copy. Mirror each through one
is_usable_ecx_file helper. NewEcVolume itself still accepts a lone
0-byte .ecx as a legitimate empty index, but prefers a non-empty copy,
local directory first, over a stub in the other directory; the
resolution in EcVolume::new now follows the same order.

Tests that mounted EC volumes without any .ecx get a real fixture.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume: mount_ec_shard tries every disk; reconcile ignores a 0-byte local .ecx

mount_ec_shard returned the first disk's error, so an unusable shard copy
(a 0-byte .ecNN left by an interrupted move) hid a good copy on the next
disk. Like Go's MountEcShards, keep scanning: NotFound means "not this
disk", any other failure is collected, and an all-disks-fail error names
every disk tried. "No .ecx on any local disk" is now told apart from
"shard not on this server".

The orphan-shard reconcile took its locally-mirrored fast path whenever a
local .ecx existed at all. A 0-byte stub there registered the shards against
an empty index while the owner index skipped that same stub. Go gates the
fast path on HasEcxFileOnDisk; do the same. ec_local_ecx_path loses its last
production caller and becomes test-only.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume: match Go's mount error text and skip the owner stat on the owning disk

MountEcShards in Go skips the HasEcxFileOnDisk stat when the disk's own
directories already hold the .ecx, dedups a shared -dir.idx across
locations in findEcxIdxDirForVolume, and reports "load failures" with
the same wording. Also drop two issue-number references from comments.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-09-24 07:08:56 +08:00
Eliah RusinandClaude Fable 5.1 c1ccbcda13 volume server: the write queue answers uploads with the needle's real ETag (#11414)
With SEAWEED_WRITE_QUEUE=1 every upload came back with ETag "00000000".
The upload handler built the needle with Needle::default(), so its
checksum was CRC(0), and handed a clone of it to the queue. The CRC was
only computed in the write path, on the worker's clone, and WriteResult
carries no checksum back, so n.etag() in the handler formatted the zero
checksum. The direct path writes through &mut n and was correct.

Compute the checksum in the handler while building the needle, the way
Go's CreateNeedleFromRequest does, over the same bytes the write path
hashes (the stored data, gzipped or not). The ETag and the has-name flag
are read before the write, so the needle is moved into the queue instead
of cloned, which also drops a full payload copy per queued upload.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-24 07:08:36 +08:00
94a68fa9b9 volume: walk_index_file keeps row alignment across short reads (#11412)
* volume: walk_index_file keeps row alignment across short reads

walk_index_file issued one Read::read per batch and decoded whatever came
back. Read::read may legally return a short count that is not a multiple
of the 17-byte entry size (FUSE and network filesystems, a BufReader whose
capacity is not a multiple of 17). The split entry at the end of the batch
was dropped with no carry and the next read started mid-entry, so every
later row was decoded from misaligned bytes and fed to the index as a
garbage key/offset/size. This function backs every in-memory index load.

Go's WalkIndexFile is immune because it reads through io.ReaderAt, which
returns a full buffer or an error. Fill the batch buffer until it is full
or the reader reports EOF, retrying ErrorKind::Interrupted, and only then
decode whole entries. Reads stay batched at ROWS_TO_READ entries.

EOF semantics are unchanged and match Go: on io.EOF Go decodes the whole
entries in the final buffer, ignores a trailing partial entry and returns
nil. A torn final entry is still skipped without an error here.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume: trim walk_index_file comments

The batch-fill loop and the ShortReader test helper each carried a
paragraph where a sentence suffices.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-09-24 07:08:26 +08:00
0f2ecb766f volume server: reject non-ASCII input instead of panicking (#11406)
* volume server: reject non-ASCII input instead of panicking

Three parsers sliced attacker-supplied strings by byte offset, so a
multi-byte character split inside itself and panicked the task:

  - parse_needle_id_cookie took the last 8 bytes as the cookie and the
    rest as the needle id. Reachable from VolumeServer.BatchDelete,
    whose file_ids come straight off the wire as protobuf strings;
    that handler already answers 400 per bad fid, so the guard turns a
    panicked RPC into the error it was already written to return.

  - TTL::read took the unit as the last byte and the count as
    everything before it, so "?ttl=5<multi-byte>" split mid-character.
    The HTTP upload path does TTL::read(..).ok() and drops an invalid
    TTL; AllocateVolume maps the Err to InvalidArgument.

Both now reject non-ASCII up front. Hex and a digits-plus-unit TTL are
ASCII by definition, so no accepted input changes -- covered by tests
alongside the rejection cases.

The six response-* header overrides were inserted with
parse().unwrap(). They come from the query string, so
"?response-cache-control=%0Aevil" decodes to a value HeaderValue
rejects and the unwrap panicked the connection task,
unauthenticated. They now skip the override, matching the if-let the
chunked-response path in the same file already uses.

ReplicaPlacement::from_string was reported as a fourth site but is not
one: reaching chars[2] requires chars[0] and chars[1] to be ASCII
digits, which forces the padded string to be three single-byte
characters, so a multi-byte character always lands on a to_digit()
None first. Kept as a regression test rather than a change.

Each fix was confirmed against the unfixed code first: the parser
tests panic with "byte index N is not a char boundary", and the
integration tests panic at handlers.rs:1413 and ttl.rs:88.

Not a vector, contrary to the report: the HTTP request line. The path
is not percent-decoded before parsing, so "%C3%A9" stays ASCII and
fails the length check.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* volume server: fall back to needle MIME when response-content-type is invalid

Skipping an unparseable override left the response without any
Content-Type because the override had already bypassed the normal MIME
selection. Also correct a test comment that described a chars[2] panic
which cannot be reached.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
2026-09-20 23:35:05 -07:00
ca62d4297b volume: load the .ecj deletion journal in chunks, and repair a torn tail (#11408)
* volume: load the .ecj deletion journal in chunks, and repair a torn tail

Two independent defects in the EC deletion journal's load path.

1. The loader issued one NEEDLE_ID_SIZE-byte positional read per entry.

   That is fine for a healthy journal -- kilobytes -- and pathological for a
   large one. A `.ecj` is semantically a SET of deleted needle ids but is
   written as an append-only log that nothing dedupes, and several paths append
   a peer's ENTIRE journal onto the local one (VolumeEcShardsCopy with
   copy_ecj_file, EC index recovery, and ec_decode's deliberate cross-holder
   merge), so a volume whose shards are repeatedly balanced between two servers
   grows the file without bound.

   Observed in production: 1.51 TB and 1.30 TB on the two holders of one 10+4
   volume containing ~100 distinct ids. At that size the per-entry loop is
   ~188e9 syscalls, run synchronously while holding the deleted_needles write
   lock and before the HTTP port opens. The process sits at 100% of one core
   with a small RSS -- the set stays tiny because the ids repeat -- reading at a
   few MiB/s because 8-byte reads defeat readahead, logs nothing after "Adding
   storage location", and ignores SIGTERM. The master then unregisters every
   volume it holds and reads of them fail. 4.46 and 4.47 are both affected.

   Read in 1 MiB chunks and build into a local set, merging once at the end so
   the write lock is not held for the whole scan. Measured on a 256 MiB journal
   of 100 distinct ids: 33,554,500 syscalls -> 257, identical resulting set.

2. A torn tail silently corrupted later deletes.

   The journal handle is in append mode, so writes land at the physical end
   regardless of alignment. A trailing partial record therefore pushed every
   later append out of alignment: the loader skipped the partial bytes, but the
   next mount decoded them together with the leading bytes of the following
   entry, producing one garbage id and dropping the delete that came after the
   tear -- after acknowledging it.

   Truncate to a whole number of records at mount, before anything can append.
   The repair uses its own read+write (non-append) handle: on Windows,
   append(true) requests FILE_APPEND_DATA without FILE_WRITE_DATA (and
   .write(true) is subsumed by .append(true)), so SetEndOfFile through the
   journal handle fails with ERROR_ACCESS_DENIED.

   The same trap exists in journal_delete's recovery path, which calls set_len
   on the append handle to roll back a partial write whose sync failed. It is
   error-handled rather than fatal, so on Windows that rollback silently does
   not happen. Untouched here; worth a separate fix.

Bounding the journal's growth needs compaction, which is deliberately not in
this change: replacing the file under a store that can hold several EcVolume
instances for one volume id requires coordinating with the other holders, and
that belongs at the store layer. Sent separately.

Tests: a journal spanning several read chunks loads every entry; a trailing
partial record is ignored rather than panicking; a torn tail is truncated at
mount and a delete taken afterwards survives a remount.

* volume: roll back a failed .ecj append through a dedicated write handle

The append handle lacks FILE_WRITE_DATA on Windows, so the set_len
rollback after a failed sync silently did nothing and the journal could
drift one record past deleted_needles. Same trap as the torn-tail repair
in this file; fix it the same way. Also format the new tests.

* volume: mirror chunked .ecj load and torn-tail repair in Go

---------

Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>
2026-09-20 23:26:41 -07:00
4bb40732bb volume server: ReceiveFile loses bytes and hides fsync failures (#11407)
* volume server: ReceiveFile loses bytes and hides fsync failures

Three defects in one handler, all on the path that receives a pushed
.dat/.idx/.vif or EC shard:

  - `f.write(&content)` never compared the return to content.len().
    A short write (ENOSPC, NFS) counted only the bytes that landed,
    so every later chunk was written at a shifted offset and the RPC
    answered error: "" with a byte count that looked right. Go's
    os.File.Write loops. Now write_all.

  - `let _ = f.sync_all();` discarded EIO and answered success with
    the full byte count. Go omits the check too, but
    ReceiveFileResponse carries an `error` field and the caller
    renames the staged file into place on success -- so a silent
    fsync failure publishes a file whose data never reached the
    platter. Flush and fsync failures are now reported.

  - Both the per-chunk write and the final fsync were blocking
    std::fs calls inside the async fn, on the runtime worker that is
    also driving the stream. Switched to tokio::fs + BufWriter, the
    shape `drain_copy_stream_to_file` in this same file already uses
    and documents. The partial-file cleanup on the error path moves
    to tokio::fs::remove_file for the same reason.

The handler had no test at all, which is how the short-write bug
survived. Added a round-trip over a real connection with ragged chunk
boundaries, asserting the bytes on disk and not only the reported
count -- a dropped or reordered chunk changes the file even when
bytes_written still adds up.

That test guards the rewrite; it does not reproduce the original
faults. ENOSPC and EIO need fault injection that this suite has no
harness for, so the short-write and fsync paths are argued from the
code, not demonstrated by a failing test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* volume server: remove the staged file on every ReceiveFile error reply

Flush and fsync failures returned early and left the partial .copying or
shard file behind, as did the pre-existing write-error path. Route all
response-level errors through one cleanup block, matching Go's
close-and-remove on a failed write.

* volume server: tighten ReceiveFile comments

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>
2026-09-20 23:26:37 -07:00
8ff2e0777e volume server: HTTP DELETE on a distributed EC volume (#11405)
* volume server: HTTP DELETE on a distributed EC volume

The delete handler validated the cookie with EcVolume::read_ec_shard_needle,
which reads only locally-mounted shards and errors "ec shard N not available
locally" for any interval held by a peer. Every Err was mapped to 500 and no
.ecj tombstone was appended, so on a standard 10+4 spread over 14 servers an
HTTP delete of an EC needle could not succeed. The GET path already goes
through read_ec_shard_needle_distributed.

Route the delete's read through the same distributed reader. It does a
local-first pass in its snapshot phase, so the all-shards-local case costs
what it did before, and no store guard is held across the await (the reader
takes its own; RwLockReadGuard is !Send).

Two smaller corrections fall out of the new return type:

  - the reader reports both "needle not in the index" and "volume vanished
    between the has_ec check and the snapshot" as Ok(None), which collapses
    the old Some(Ok(None)) and None arms into one 404;
  - an io::ErrorKind::NotFound now answers 404 rather than 500, matching the
    GET path. Telling a caller to retry a delete that can never succeed was
    half the bug.

The cookie check and its ordering before the journal append are unchanged.

Not addressed here: Rust journals the tombstone locally while Go routes it to
the primary shard holder. That is a separate behaviour change and belongs in
its own PR against the same issue-10 checkbox.

The regression test mounts 13 of 14 shards, leaving out the one holding the
needle's interval. The distributed reader seeds its Reed-Solomon buffers from
locally mounted siblings, so with >= 10 survivors it reconstructs with no peer
fan-out -- which makes the bug reproducible on a single node. Against the
unfixed handler the test fails with 500 vs 202.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* volume server: fail the delete when the EC volume unmounts mid-request

find_ec_volume_mut returning None used to fall through to a 202 with no
.ecj tombstone written, reporting success for a delete that did not
happen. Answer 404 like the other volume-vanished arms so the caller can
retry after a remount.

* volume server: forward EC needle deletes to a primary-shard holder

Mirror Go's doDeleteNeedleFromAtLeastOneRemoteEcShards: the tombstone is
journaled on one holder of the needle's primary data shard via
VolumeEcBlobDelete (or the local journal when this server holds the
shard), falling back to any other shard holder when the primary has
none. Journaling only on the node that received the DELETE scattered
tombstones across whichever server took the request.

* volume server: route BatchDelete EC deletes through the same forwarding

BatchDelete had the same local-journal divergence as HTTP DELETE, plus a
gap the old code admitted in a comment: the .ecx index cannot supply the
needle's cookie, so EC deletes ran with no cookie check at all. A
distributed read now fills the needle for every EC entry — matching Go's
DeleteEcShardNeedle, which reads and compares the fid cookie even when
skip_cookie_check is set — and the tombstone forwards via
delete_ec_shard_needle_distributed. A needle deleted between read and
journal reports 304 like Go's ErrorDeleted; a vanished volume reports
500 so the filer retries.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>
2026-09-20 23:26:31 -07:00
Eliah RusinandClaude Fable 5.1 ac876eef21 rust volume: build outgoing gRPC clients in one place and give every dial a connect timeout (#11354)
server/grpc_client.rs stopped at build_grpc_endpoint() -> Endpoint, so all 13
production call sites hand-wrote the same .connect() + X::with_interceptor()
+ two max_*_message_size() lines. Four of them -- VolumeCopy,
VolumeTailReceiver, VolumeEcShardsCopy and the HTTP chunk batch-delete fan-out
-- dialed with no timeout at all, so an unreachable peer whose TCP handshake
never completes (SYN dropped, blackholed route, host behind a silent firewall)
left the operation waiting on the kernel's own retry budget, minutes long.

Add GrpcDialOptions (unary / long / stream presets), connect_channel(), and
volume_server_client() / master_client() / filer_client() constructors that
attach the request-id interceptor and lift both message-size limits, then
route all 13 sites through them. build_grpc_endpoint is private again, so
connect_channel is the only way out of the module and no call site can dial
without picking up a bound. Each site's existing timeouts are preserved
exactly; the four bare dials gain a 5 s connect timeout and nothing else. No
per-request deadline was added to any streaming call: Endpoint::timeout is a
per-request bound on time-to-first-response-headers for every request the
channel carries, so a value picked for one short call would also be the header
deadline for the whole-volume transfer sharing the dial.

The new bound covers the TCP handshake only -- tonic hands connect_timeout to
HttpConnector::set_connect_timeout. A peer that completes the handshake and
then stalls in the TLS or HTTP/2 exchange is still unbounded at those four
sites, as are the RPCs themselves. That is why the three ping_* helpers keep
their outer tokio::time::timeout: replacing it with connect_timeout would have
narrowed a whole-connect bound they already had.

main.rs no longer re-declares GRPC_MAX_MESSAGE_SIZE and the three
keepalive/window constants; it imports them from grpc_client.rs so the
inbound server and the outgoing clients cannot drift apart.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 19:41:37 -07:00
818f3bb71b rust volume: share the I/O-error tracker between Volume and EcVolume (#11351)
* rust volume: share the I/O-error tracker between Volume and EcVolume

Volume and EcVolume each carried the same three fields - a mutex-held
last error, a consecutive count and a sticky quarantine flag - and the
same four methods over them, identical except for the path qualifier on
is_storage_io_error. The tolerance the count is compared against was a
fourth copy: heartbeat.rs held VOLUME_IO_ERROR_TOLERANCE for volumes,
ec_volume.rs held IO_ERROR_TOLERANCE for EC, and the volume test helper
open-coded the same 3, so the two paths could drift apart silently.

Go keeps this in one place already: weed/storage/io_error.go holds
IoErrorTracker, IoErrorTolerance and isStorageIoError, and Volume embeds
the tracker. Go's EcVolume has to re-implement it only because those
fields are unexported and EC lives in another package.

storage::io_error::IoErrorTracker now owns that state, with record /
state / should_quarantine / mark_quarantined / reset and the single
IO_ERROR_TOLERANCE. is_storage_io_error moves into the same file, so it
sits with the tracker that is now its only caller, the way io_error.go
is laid out. Both volume kinds embed one tracker and keep their existing
method names as delegates, so the ~16 internal call sites and the
readers in heartbeat.rs, store.rs and grpc_server.rs change only where
the two threshold comparisons become should_quarantine().

Volume::last_io_error and EcVolume::reset_io_error_state had no callers
and are gone.

Unchanged: what counts as a storage-media error - is_storage_io_error
changed file, not body, and is still the single predicate both volume
kinds share, where Go's EcVolume tests EIO directly and so misses the
Windows codes. Also unchanged: the tolerance value, the metric increment
on every counted error, and the sticky quarantine - a success clears the
count and the last error but never the flag, which only reset lifts. In
the heartbeat the state read moved inside the quarantine branch, so the
common path no longer takes the tracker's mutex or clones the last-error
string; should_quarantine's two relaxed loads run either way.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* rust volume: hoist absolute_display_path into server

handlers.rs and ui.rs each held a byte-identical copy of the helper that
turns a configured -dir into an absolute path for display. The status
JSON and the status page are meant to show the same directory, so the
two copies had to be edited together to stay that way.

The helper now lives in server/mod.rs as pub(crate) and both callers use
it. No behaviour change: same body, same call sites.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* rust volume: keep EcVolume::reset_io_error_state

Moving both volume types onto the shared IoErrorTracker dropped
EcVolume's public reset while Volume kept its own, so the two sides of
the tracker drifted apart.

mark_quarantined is sticky: a later successful read clears the error
count through record(), but the quarantine flag only comes down through
reset(). Without the delegate an EC volume that hit sustained media
errors could not be returned to service in place once the storage was
repaired. Go exposes the same method as EcVolume.ResetIoErrorState
(weed/storage/erasure_coding/ec_volume.go:114).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* rust volume: name the shared tracker after Go's IoErrorTracker

- check_read_write_error, get_io_error_state, mark_io_quarantined,
  reset_io_error_state match weed/storage/io_error.go one to one
- io_error module is pub(crate) like the io module beside it
- restore EcVolume::reset_io_error_state so both volume kinds expose the
  same recovery surface
- trim comments that restate the code

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-20 16:57:23 -07:00
7643f4f541 rust: a seaweed-common crate for the address and TLS helpers both crates carry (#11358)
* rust: a seaweed-common crate for the address and TLS helpers both crates carry

seaweed-volume and seaweed-worker are separate cargo trees with separate
lockfiles and no root manifest, so anything both of them need has had to be
written twice. Two of those copies are a correctness risk rather than a typing
cost, and this crate is where they stop being copies.

address.rs is the HTTP<->gRPC port rule: `host:port` means gRPC on port+10000,
`host:port.grpcPort` names it outright. The two copies had already drifted —
the worker's bracketed IPv6 literals, the volume server's did not — so the rule
lives here once, returning a typed AddressError whose Display text is the volume
server's original wording, with join_host_port public beside it. A test asserts
two of those messages in full rather than by substring, because the wording is
the contract its callers hand to a Status or an io::Error; the other three end
in a std ParseIntError message, which is std's to reword. The enum is
#[non_exhaustive] so a future variant is not a breaking change for either
consumer. The tests are both crates' cases together, plus the IPv6,
already-bracketed and normalisation cases neither copy covered on its own.

tls.rs is install_default_crypto_provider. Both binaries link aws-lc-rs and ring
transitively, so rustls cannot auto-select and tonic's client TLS panics on
first use; each binary has to pin one and it has to be the same one, which is
exactly the kind of choice that should not exist twice. It is safe to share
because `cargo tree -i rustls` resolves a single rustls in each tree (0.23.37 in
seaweed-volume, 0.23.43 in seaweed-worker) and cargo unifies all
semver-compatible `rustls = "0.23"` requirements into one crate per binary, so
this crate writes the same process-wide static its consumer reads. rustls is
already in both graphs — directly in the volume server, through tonic's
tls-aws-lc in seaweed-worker-core — so the dependency adds no crate to either.

rust-version is 1.91.1, the lower of the two consumers' floors, so depending on
this crate cannot raise either tree's MSRV; verified with
`cargo +1.91.1 check --all-targets`. The lockfile is committed even though this
is a library: CI builds it directly, so a committed lock is what makes those
runs reproducible and their caches stable.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* rust: take the address and TLS helpers from seaweed-common

Both public signatures are kept, so no caller outside the two wrapper files
changes. parse_grpc_address stays `Result<String, String>` and maps the typed
error through Display; server_to_grpc_address stays `Option<String>` and drops
it with .ok(). Their doc comments and the volume server's 13 call sites are
otherwise untouched.

Three behaviours change, each in the direction of the copy that was already
right:

- The volume server now brackets IPv6 literals. `::1:19333` used to come back as
  `::1:29333`, which build_grpc_endpoint rejects with "invalid gRPC endpoint
  http://::1:19333: invalid authority" — an IPv6 master or EC peer could not be
  dialled at all. Two tests in grpc_client.rs pin it, one on the string and one
  on the endpoint the string builds.
- The volume server now emits the *parsed* gRPC port of the dotted form instead
  of the original text it had just validated, so `host:8080.018080` and
  `host:8080.+18080` come back as `host:18080` rather than as authorities the
  URI parser rejects. Same port either way; only malformed spellings change.
- The worker's dotted form now validates the HTTP port it discards.
  `server_to_grpc_address("host:abc.18080")` used to answer Some("host:18080");
  it now answers None, which is what the volume server's copy has always done.

install_default_crypto_provider becomes a re-export in both trees, so
`crate::security::tls::install_default_crypto_provider` and
`weed_lance_worker::tls::install_default_crypto_provider` still resolve. The
lance crate's `rustls = "0.23"` was its only direct use of rustls and goes away
with the body; seaweed-common states the same requirement, so neither the
resolved version nor the enabled features move in either lockfile.

The PEM test fixtures stay where they are. The two tests that use them are not
duplicates: the volume server's exercises build_grpc_endpoint, and the lance one
exists precisely because aws-lc-rs and ring are both linked in that crate's
graph. Only the literals are shared, and exporting test fixtures from a library
to dedupe two constants costs more than it saves.

A path dependency outside both trees means every build context that copies one
crate directory has to copy the other. The repo has one: the Rust source-build
stage of docker/Dockerfile.go_build, which now copies seaweed-common beside
seaweed-volume. Every workflow whose `paths:` filter keys on a crate directory
gains `seaweed-common/**` — the two Rust test workflows, rust_binaries_dev,
container_dev and performance. The tag- and dispatch-triggered ones
(rust_binaries_release, container_release_unified, container_latest) have no
`paths:` filter and need nothing.

The two Rust test workflows also run `cargo test` in seaweed-common, from their
unit-test job, because a path dependency is not a workspace member and neither
tree's own `cargo test` reaches it. Each step builds into its job's cached
target directory, and both cache keys now hash seaweed-common/Cargo.lock as well
so a change there invalidates the cache it would otherwise silently reuse.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docker: keep go_build working for BRANCH revisions without seaweed-common

The rust_builder stage copies seaweed-common unconditionally now that seaweed-volume path-depends on it, but BRANCH can name any revision — including ones that predate the crate. Create the directory in the builder stage so the COPY always has a source; an empty dir beside an old seaweed-volume is harmless.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-20 13:30:58 -07:00
f1ed270942 rust volume: one S3 tier registry instead of two kept in sync by hand (#11357)
* rust volume: one S3 tier registry instead of two kept in sync by hand

`VolumeServerState.s3_tier_registry` and `global_s3_tier_registry()` held
the same S3 tier backends. `apply_storage_backends` — the only production
writer — registered every backend into both, and each half of the tiering
code then read a different one: the gRPC tier-move handlers resolved the
backend from the per-server field, while `Volume`'s remote mount and
destroy paths resolved it from the global registry, because a `Volume` has
no handle to the server state. Two registries that must agree, kept in
agreement by a duplicated `register_s3_backend` call and a comment in a
test constructor explaining the hand-sync.

Delete the field and let both tier-move handlers resolve from the global
registry, so `apply_storage_backends` registers once and no longer needs
the server state at all. Injecting a registry handle through `VolumeSpec`
instead was considered and rejected here: it would touch every `Volume`
constructor for no functional gain, and the process-wide registry is what
`Volume` already uses.

Behaviour is unchanged: the same names were registered in both registries,
so every lookup resolves exactly as before. The tier-down test now
registers its backend only in the global registry — before this change it
fails with `remote storage s3.tier_down_delete not found from supported:
[]`. The tier-up handler had no test at all, so it gets a cheap probe:
register a backend only in the global registry, ask for that destination,
and check the call gets past the lookup — the response is dropped straight
away, so the transfer sees a departed caller and never opens a connection.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* rust volume: await the tier-up probe terminal error instead of racing it

Dropping the response left it to chance whether the detached transfer saw the closed channel before its initial check; if it won that race it went on to attempt the multipart upload with no one waiting on the outcome. Hold the stream and read until the dead endpoint fails the upload — the terminal error proves the task ran and finished, so no background network work outlives the test.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-20 13:30:40 -07:00
Eliah RusinandClaude Fable 5.1 7dbbdac030 rust volume: keep the EC shard-location map, its refresh time and stale mark under one lock (#11356)
The per-EcVolume shard-location cache was three fields under three locks:
an RwLock<HashMap> for the map, a Mutex<Option<Instant>> for the time it
was last refreshed, and a Mutex<bool> for the stale mark. Nothing tied
them together. merge_shard_locations published the merged map, released
the write lock, and only then stamped the refresh time; both readers
(scrub_ec_volume_distributed's snapshot and build_snapshot) took the two
guards one after the other. A reader landing between the two writes
paired a freshly merged map with the previous lookup's timestamp -- and
that pair is exactly what needs_refresh judges, so a read went back to
the master for a map that had just been refreshed. Go keeps the same
state in one struct behind one ShardLocationsLock. replace_shard_locations
documented itself as "a single observable step" while being two.

Fold the three fields into one ShardLocationCache behind a single RwLock.
merge_shard_locations upserts and stamps in one write section,
shard_locations_snapshot returns the map and its time from one read
section, and mark_shard_locations_stale / claim_shard_locations_refresh
move the mark's read-and-consume onto the cache. The three zero-caller
accessors -- set_shard_locations, replace_shard_locations,
get_shard_locations -- are deleted, and the field is now private, so the
invariant cannot be sidestepped from outside the module. The two test
seeding sites go through merge_shard_locations, which already produces
the state they were writing by hand.

Unchanged: the freshness rule. needs_refresh keeps its thresholds and
still judges the caller's snapshot -- the map that caller will actually
read from, not whatever is cached by the time the claim runs -- so only
the stale mark is read from under the new lock. The master lookup, the
completeness guard in write_back_shard_locations and the per-shard upsert
semantics are untouched.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 13:30:15 -07:00
6d676eda67 rust volume: typed errors for store compaction so gRPC can answer NotFound (#11355)
* rust volume: typed errors for store compaction so gRPC can answer NotFound

The vacuum entry points on `Store` returned `Result<_, String>`, so the
gRPC layer had nothing to branch on and answered `Status::internal` for
every failure. A vacuum loop that races a volume being moved or deleted
saw the same code as a disk going bad, and `weed shell` could only tell
the two apart by matching on the message text.

`VolumeError` gains `VolumeNotFound(VolumeId)` — the existing `NotFound`
is needle-level and carries no payload — and `InsufficientSpace`, and
`compact_volume`, `commit_compact_volume`, `cleanup_compact_volume` and
`delete_collection` return it. `impl From<VolumeError> for tonic::Status`
in `server/mod.rs` maps not-found to `not_found`, read-only to
`failed_precondition`, insufficient space to `resource_exhausted`,
already-exists to `already_exists`, and everything else to `internal`;
the four RPCs prefix their own context with `status_with_context`, so a
message reads "commit compact volume 7: volume id 7 is not found". The
store-side "during compact" / "during commit compact" / "during cleaning
up" suffixes are gone, and the free-space message drops the volume id the
prefix already supplies.

`check_compact_volume` had no callers — `VacuumVolumeCheck` computes the
garbage level from its own `find_volume` — and is deleted. `compact_volume`
folded the size estimate into its first lookup, dropping the `unwrap()`
re-lookup that only existed to dodge a borrow.

`ascending_visit` on `CompactNeedleMap`, `RedbNeedleMap`,
`SortedFileNeedleMap` and the `NeedleMap` dispatch is now generic over the
visitor's error type, like `CompactMap::ascending_visit` already was. The
three signatures that can fail on their own bound `E: From<String>` to
carry those failures; the in-memory walk in `iter_entries` names
`Infallible`, which says in the type what its comment used to say in prose.

No Go shell command matches on the old error text: the strings exist only
in weed/storage/store_vacuum.go.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume server: trim comments and answer the same codes from Go

- vacuum_volume_check reports VolumeError::VolumeNotFound like the other
  vacuum RPCs instead of its own "not found volume id" wording
- drop doc comments that restate what the code says
- Go volume server wraps ErrVolumeNotFound/ErrInsufficientSpace from
  store_vacuum.go so VacuumVolumeCheck/Compact/Commit/Cleanup and
  DeleteCollection answer NotFound/ResourceExhausted, matching the Rust
  volume server; volumeDeleteStatusError generalized to volumeStatusError

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: prefix operation context on vacuum errors

Lower-level errors forwarded by CompactVolume, CommitCompactVolume,
CommitCleanupVolume and DeleteCollection carry no volume id or operation
name. Wrap with %w so the status mapping still sees the sentinel chain,
matching the context the Rust server's status_with_context adds.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: map NotEmpty to FailedPrecondition, share mapper in VolumeDelete

Go's volumeStatusError maps ErrVolumeNotEmpty to FailedPrecondition; the
Rust Status conversion was missing it and volume_delete kept a hand-rolled
match. Route it through status_with_context like the vacuum handlers.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-20 13:30:02 -07:00
Eliah RusinandClaude Fable 5.1 d002481037 rust volume: derive has_remote_file instead of mirroring it (#11353)
Volume carried `pub has_remote_file: bool` next to `pub volume_info`,
and the bool was only ever the answer to `!volume_info.files.is_empty()`:
outside the two constructors, `refresh_remote_write_mode` was the single
writer. Both fields being public made the pair a convention rather than
an invariant. Every caller that touched `volume_info.files` — load_vif
twice, the tier-up handler, the tier-down handler and its rollback — had
to remember to call `refresh_remote_write_mode` afterwards, and a caller
that forgot would leave the volume advertising a write mode its .vif
contradicts, or serving a remote .dat through a writable needle map.

The bool becomes `has_remote_file()`, computed from the list, so it
cannot drift. `volume_info` becomes private with a `volume_info()`
reader, and edits to the reference list go through
`update_remote_files(|files| ...)`, which applies the closure and then
refreshes the derived write mode and the needle map. With no caller left
outside the module, `refresh_remote_write_mode` is private.

Unchanged: the refresh logic itself, the order of operations in both
tier handlers, and the tier-down rollback semantics. The rollback still
snapshots the removed reference before the refresh runs, restores it on
failure, and re-refreshes unconditionally on the error path — the second
`update_remote_files` call runs with a no-op closure when there was
nothing to restore, exactly as the old code re-ran the refresh whether
or not it had re-inserted a reference.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 13:29:47 -07:00
Eliah RusinandClaude Fable 5.1 1df8c05bc3 rust volume: document every unsafe block and stop mutating the process env in tests (#11352)
Three production `unsafe` blocks carried no `// SAFETY:` comment at all
(`libc::fallocate`, `libc::sysinfo`, `libc::statvfs`), and nothing made
that an error: `clippy::undocumented_unsafe_blocks` is a `restriction`
lint, allow-by-default, and appeared nowhere in either crate. Turn it on
in `seaweed-volume`'s `[lints.clippy]` and in the worker workspace's
`[workspace.lints.clippy]`, then document what each block relies on.

`memory_status.rs` and `disk_location.rs` get their blocks narrowed to
the `zeroed()` and the libc call, so each comment sits next to the
operation it justifies and the arithmetic is outside the block. Both
turn the success test into an early return on failure; the casts, the
multiplication order and the values returned on either path are
unchanged.

The bigger problem was in `config.rs`'s tests. `with_temp_env_var` and
`with_cleared_security_env` called `std::env::set_var`/`remove_var`,
claiming soundness because every caller holds `process_state_lock()`.
That mutex only serialises the fourteen annotated tests in this module.
The same lib test binary runs the `grpc_server.rs` tests, which bind a
`TcpListener`, dial loopback and drive a multi-thread tokio runtime, and
tonic/hyper/rustls/aws-sdk all read the environment lazily on those
threads — which is exactly the race Rust 2024 made these calls unsafe
for. `restore_env_var` had no SAFETY comment at all. `#[serial]` would
not have helped: it serialises annotated tests, which the mutex already
did.

So the config layer no longer reads the environment implicitly. An
`EnvLookup<'a> = &'a dyn Fn(&str) -> Option<OsString>` is threaded from
the public entry points down to every reader — `HOME`, `USERPROFILE`,
the twenty-four `WEED_*` keys and `SEAWEED_WRITE_QUEUE`. `parse_cli` and
`parse_security_config` keep their signatures and pass `process_env`, a
thin wrapper over `std::env::var_os`; `resolve_config` becomes
`resolve_config_with_env` (private, one caller). Tests build one with
`fake_env` instead, so no test touches the real environment and every
`unsafe` in the module is gone.

`process_state_lock()` stays, with a smaller job: `set_current_dir` is
safe but still process-global, so the tests that move the working
directory are still serialised against the ones that read it. Tests
naming an explicit config file never reach that search and no longer
take the lock.

No production behaviour changes: the same keys are read in the same
order with the same precedence, and `env_string` reproduces
`std::env::var(key).ok()` — absent and non-UTF-8 both read as unset.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 13:29:07 -07:00
Eliah RusinandClaude Fable 5.1 799c495226 rust volume: one positional read helper; never seek a dup'd handle on Windows (#11342)
* rust volume: one positional read helper; never seek a dup'd handle on Windows

Positional read-exact was hand-rolled four times: the complete
cross-platform version in needle_map/sorted_file.rs, a Windows-only half
in volume.rs whose unix half was inlined as a
cfg(unix)/cfg(windows)/compile_error! triple at three call sites, a
byte-identical Windows-only copy in ec_volume.rs, and read_full_at in
ec_bitrot.rs. Three more sites -- EcVolumeShard::read_at,
EcLocalShard::read_at and ec_encoder::read_at_most -- hand-rolled the
short-read-permitted variant with a cfg(not(unix)) arm that
try_clone()s the handle and seeks it.

That last arm is wrong. A duplicated descriptor shares one kernel file
offset with the original, so seek-then-read is two syscalls against
state another thread can move in between: a concurrent reader or an
append repositions the offset and the read returns bytes from somewhere
else entirely. EcLocalShard::read_at documents that it must never seek,
one line above the seek. Windows seek_read carries its own offset in a
single call, so that window does not exist.

All seven now go through storage::io::{read_exact_at, read_at}, whose
module doc records why duplicating a handle is not a way to get a
private file position -- opening the file again is, as
Volume::dat_scan_plan already does. read_at_most keeps its own
fill-until-EOF loop; only the per-iteration positional read changes.

Behaviour on unix is unchanged: every unix arm was already
FileExt::read_exact_at or FileExt::read_at. The one exception is
ec_bitrot::verify_shard_blocks, which now retries on EINTR (std's
read_exact_at does; the loop it replaces did not) and, on unix, reports
the standard "failed to fill whole buffer" text instead of "short read
on shard block". The Windows arm still says "unexpected EOF in
seek_read"; both carry ErrorKind::UnexpectedEof, as before.

NeedleStreamSource::read_exact_at and Volume::read_exact_at_backend keep
their signatures; only their bodies shrink.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* rust volume: retry Interrupted in Windows read_exact_at

Unix std's FileExt::read_exact_at ignores ErrorKind::Interrupted and
retries, but the Windows seek_read loop propagated it, so the shared
exact-read contract differed by platform. seek_read can surface
ERROR_OPERATION_ABORTED, which std maps to Interrupted.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-17 11:47:38 -07:00
Eliah RusinandChris Lu 0eb638f503 fix(ec): BatchDelete cookie fail-closed via locate_data geometry (#11348)
* fix(ec): BatchDelete cookie fail-closed via locate_data geometry

* fix(ec): honor skip_cookie_check, require full cookie header

* fix(ec): retry short cookie header reads, still fail closed on EOF

* chore(ec): trim cookie validation comments

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-16 16:25:03 -07:00
Eliah Rusin caf3d157e6 fix(ec): encode drops tombstoned needles, last-wins replay (#11347)
* fix(ec): encode drops tombstoned needles, last-wins replay

* fix(ec): drop zero-offset rows in encode, match readNeedleMap
2026-09-16 15:40:22 -07:00
Eliah Rusin def25ca84d fix(ec): validate ShardId at gRPC boundary, reject >=32 (#11346) 2026-09-16 08:33:22 -07:00
Eliah Rusin 701e397337 fix(volume): reject negative Size, recover poisoned store lock (#11345) 2026-09-16 08:12:11 -07:00
Eliah Rusin 4fc9ada2ec ci: run seaweed-volume unit tests on Windows (#11349) 2026-09-16 08:06:21 -07:00
Eliah RusinandClaude Fable 5.1 a73ba3adbb rust volume: parse vid/fid paths once; the proxy redirect drops the extension like Go (#11341)
handlers.rs split needle URLs in three places and the three disagreed.
Go does it once, in parseURLPath (weed/server/common.go:218-249), and
dispatches on the slash count: /vid/fid/filename takes the extension off
the filename and leaves the fid whole, /vid/fid takes it off the fid, and
the comma form splits the last segment on its last comma and dot.

Two of the Rust copies got that wrong:

- extract_file_id returned the path unchanged when it found no comma, so
  a JWT fid claim, which Go compares against vid + "," + fid for every URL
  form (volume_server_handlers.go:361-364), could never match a slash-form
  request. With a JWT key configured, every read, write or delete of
  /3/01637037d6 was a 401.
- build_proxy_request_info's slash branch had no extension handling, so a
  redirect for /3/01637037d6.jpg sent the client to /3,01637037d6.jpg.
  Go's proxyReqToTargetServer formats "%s/%s,%s" from the already-stripped
  fid (volume_server_handlers_read.go:128-137) and so emits
  /3,01637037d6. The peer still serves either form, since the comma form
  strips the extension again, so this one is parity rather than breakage.

Replace all three with one parse_needle_path returning vid, fid, ext and
filename borrowed from the path. The fid keeps its _delta suffix, as in
Go: parse_needle_id_cookie applies it and the JWT check strips it. The
leading slash stays optional, so chunk manifest fids still parse.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 01:13:06 -07:00
563c729e70 rust volume: stream the tail scan and release the store lock (#11275)
* rust volume: add a .dat scan plan that runs without the store lock

DatScanPlan captures a fresh .dat handle, the version, the start offset
and an end bound while the caller holds a store guard, then visits one
record at a time with positional reads that never touch the Volume, the
way Go's ScanVolumeFileFrom feeds a scanner.

The handle pins the inode the offset was resolved against: a vacuum
commit renames .cpd over .dat and destroy unlinks it, and neither
rewrites the pinned bytes. The end bound is read while no writer can
hold store.write(), so the scan never meets a partial append. It is a
fresh open, not try_clone, because on Windows read_exact_at uses
seek_read, which moves a cursor a clone shares with the writer.

A header whose size is negative, or does not fit before the end bound,
ends the pass before the body length is computed or anything is
allocated. In today's scan a negative size reaches needle_body_length
and either overflows the buffer size or walks the scan from a wrong
offset. A size near i32::MAX overflows padding_length's i32 arithmetic,
which panics in debug builds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018VF7E9SHPihG1jC1grU9H3

* rust volume: stream the tail scan with the store lock released

volume_tail_sender read every needle from the start offset to EOF into a
Vec while holding store.read(). volume.merge tails from zero, so that was
the whole volume in memory. And because needle writes and the heartbeat
take store.write() on a lock that prefers writers, the whole node stopped
serving until the scan finished: the failure #11235 fixed for EC scrub.

Each pass now runs on a blocking thread. Under one store guard it
resolves the start offset and captures a DatScanPlan, then drops the
guard and sends each needle as it is read, as Go's
VolumeFileScanner4Tailing does. This replaces the one-guard-across-
search-and-scan rule from the previous commit with a stronger invariant:
the offset, the handle and the end bound come from the same guard, and
the handle pins the inode, so a vacuum commit mid-scan cannot point the
offset into the compacted file.

A scan error now ends the stream with Status::internal instead of a
clean EOF, as Go's `streamFollow: %w` does. Once needles stream, a clean
EOF after a partial pass would let volume.move treat a truncated tail as
complete. A panic in the pass is reported the same way. A receiver that
hangs up is also noticed between skipped needles, not only on a send.

Unchanged: the append_at_ns filter, the header on every 2MB chunk, the
caught-up heartbeat without a scan, and the draining countdown.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018VF7E9SHPihG1jC1grU9H3

* rust volume: fail the tail pass on a short read below the snapshot end

DatScanPlan::scan treated an UnexpectedEof on the header or body read as
the end of the data and returned Ok. Every byte below the captured end
existed when the plan was taken, so a short read there can only mean the
inode was truncated under the plan: an unmount followed by a VolumeCopy
of the same volume id reopens .dat with truncate(true). The pass then
reported Scanned, the next pass found the volume gone, and the stream
ended cleanly after a prefix of the planned records, which volume.move
would take as a complete tail.

Both short-read arms now fail the scan with an I/O error that names the
offset and the snapshot end, so tail_pass reports Status::internal as it
does for every other read failure. The break arms were carried over from
scan_raw_needles_from, where the whole scan ran under the store guard and
nothing could truncate the file.

Found by the Devin and Greptile reviews on #11275.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* rust volume: sum the needle padding in i64 so a corrupt size cannot overflow

padding_length added the header, checksum and timestamp widths to the
needle size in i32. A size read from a corrupt header can sit near
i32::MAX, and that sum then overflows: a panic with overflow checks, a
wrapped padding without. DatScanPlan::scan bounds the size against the
bytes left before computing the body length, but that only keeps such a
size out of the arithmetic while under 2 GiB of the file remains, so on a
large volume the scan could still reach the overflow and, in release,
size a buffer from garbage.

Sum in i64 in both version branches. The result is at most
NEEDLE_PADDING_SIZE, so it still fits Size. The scan comment no longer
claims the bound check prevents the overflow.

Found by the CodeRabbit review on #11275.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* rust volume: propagate dat scan parse failures

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-15 14:40:06 -07:00
166af06a2b rust: cargo fmt both crates, with a commented-out fmt --check CI step (#11329)
* rust: migrate seaweed-volume and seaweed-worker to tonic 0.14 / prost 0.14

tonic 0.14 boxes the contents of tonic::Status, which is what made every
RPC path trip clippy's result_large_err; the allow for that lint goes in
the next commit. The prost codec moved out of tonic into tonic-prost and
tonic-prost-build, so both build scripts now call
tonic_prost_build::configure() and both crates depend on tonic-prost for
the generated code. The `tls` feature was split into a per-backend
feature; `tls-aws-lc` is the same backend both crates already install
through rustls::crypto::aws_lc_rs.

tonic 0.14 depends on axum 0.8 and tower 0.5, which would have left a
second axum and a second tower in each tree next to the 0.7 / 0.4 the
crates named themselves. Bumping them keeps one copy of each: axum 0.8
only changes the path-parameter syntax for the routes here (`/:vid` ->
`/{vid}`, `/*path` -> `/{*path}`), tower 0.5 needs the `util` feature
named explicitly for ServiceExt::oneshot (it used to arrive through
tonic's feature unification), and tower-http 0.6 is the matching
release.

Lock files move only through cargo's own resolution for the new
versions; no other dependency was refreshed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust: drop the result_large_err allow now that tonic::Status is boxed

tonic 0.14 stores Status behind a Box, so Result<_, Status> is no longer
a large-Err type and clippy has nothing to say about it. Both crates
pass `cargo clippy --all-targets -- -D warnings` without the allow
(seaweed-volume in both feature sets), so the policy entry and its
comment go.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust-volume: drop the unused headers argument of try_expand_chunk_manifest

The parameter was already named `_headers`; nothing in the body reads it.
With it gone the function is under clippy's argument threshold and the
expect goes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust-volume: pass EC peer reads an EcInterval instead of ten arguments

fetch_one_interval, read_remote_ec_shard_interval,
do_read_remote_ec_shard_interval and recover_one_remote_ec_shard_interval
all took the same (vid, needle_id, shard_id, shard_offset, size,
expected_encode_ts_ns) tuple, and the two that reconstruct also took the
location map with the data/parity counts. Those are now EcInterval (Copy)
and EcShardMap (a borrow of the map plus the counts). The fan-out inside
recovery builds its per-shard request with `EcInterval { shard_id: sid,
..iv }`, which is the one place the old argument list was easy to get
wrong. Bodies destructure at the top, so the code below the signatures
is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust-volume: give the EC encoder an EcEncodeLayout and an EncodeRun

encode_dat_file took the Reed-Solomon shape and three block sizes as five
loose integers; they are now one Copy struct, EcEncodeLayout, which is
what Go calls ECContext. The per-row and per-batch helpers took the same
six sinks and the offsets; they become methods on EncodeRun, which owns
the borrows for one run, so each call names only the offset and block
size that vary. The byte-level work is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust-volume: describe a .dat rebuild with DatRebuild instead of nine arguments

write_dat_file_from_shards, its _with_dirs twin and the private
write_dat_file were three layers over one nine-argument signature. One
public function now takes a DatRebuild, whose shard_dirs is None when
every shard sits beside the .dat and Some(dirs) for the cross-disk
reconciled layout. The field docs carry what the function doc used to
say about the encode-time size and the block layout.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust-volume: split copy_file_from_source's fifteen arguments into two structs

CopyFileSpec is the per-file request (what to ask the source for, where
it lands, whether its bytes count as progress); CopyProgress is the
sender, throttler and report state that all three files of one
VolumeCopy share, held by &mut across the calls. The three production
call sites now read as the .dat/.idx/.vif literals they are, instead of
positional trues and falses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust-volume: create volumes from a VolumeSpec

Volume::new, DiskLocation::create_volume and Store::add_volume each
took the same five-value tail of Go's NewVolume argument list:
collection, replica placement, TTL, preallocation and needle version.
That tail is now VolumeSpec, a Copy struct whose Default is what almost
every test wanted anyway (empty collection, no replication, no TTL, no
preallocation, current version), so most of the 104 call sites shrink
to `&VolumeSpec::default()` or name the one field they set. The id,
directories, index kind and disk type stay positional because they
differ at every site.

Two imports that only test modules use moved into those modules, and
DiskLocation no longer imports ReplicaPlacement.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust-worker: run cargo fmt

Layout only; no token in the workspace changes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust-volume: run cargo fmt

Layout only; no token in the crate changes. Every earlier Rust PR here
formatted only the blocks it touched so as not to drown its diff in
this one, and this commit is that debt paid in a single place. rustfmt
needed two passes to settle one block in handlers.rs; the committed
form is the fixed point, so `cargo fmt --check` is clean.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* ci: add a commented-out cargo fmt --check step to both Rust workflows

Same shape as the commented clippy step from #11312: the check is
written out so that making formatting a gate is a one-line uncomment,
and whether to do that stays a maintainer call.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-09-15 09:29:22 -07:00
Eliah RusinandClaude Fable 5.1 517f60e875 rust-volume: fold the 8–15-argument functions into parameter structs (#11328)
* rust: migrate seaweed-volume and seaweed-worker to tonic 0.14 / prost 0.14

tonic 0.14 boxes the contents of tonic::Status, which is what made every
RPC path trip clippy's result_large_err; the allow for that lint goes in
the next commit. The prost codec moved out of tonic into tonic-prost and
tonic-prost-build, so both build scripts now call
tonic_prost_build::configure() and both crates depend on tonic-prost for
the generated code. The `tls` feature was split into a per-backend
feature; `tls-aws-lc` is the same backend both crates already install
through rustls::crypto::aws_lc_rs.

tonic 0.14 depends on axum 0.8 and tower 0.5, which would have left a
second axum and a second tower in each tree next to the 0.7 / 0.4 the
crates named themselves. Bumping them keeps one copy of each: axum 0.8
only changes the path-parameter syntax for the routes here (`/:vid` ->
`/{vid}`, `/*path` -> `/{*path}`), tower 0.5 needs the `util` feature
named explicitly for ServiceExt::oneshot (it used to arrive through
tonic's feature unification), and tower-http 0.6 is the matching
release.

Lock files move only through cargo's own resolution for the new
versions; no other dependency was refreshed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust: drop the result_large_err allow now that tonic::Status is boxed

tonic 0.14 stores Status behind a Box, so Result<_, Status> is no longer
a large-Err type and clippy has nothing to say about it. Both crates
pass `cargo clippy --all-targets -- -D warnings` without the allow
(seaweed-volume in both feature sets), so the policy entry and its
comment go.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust-volume: drop the unused headers argument of try_expand_chunk_manifest

The parameter was already named `_headers`; nothing in the body reads it.
With it gone the function is under clippy's argument threshold and the
expect goes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust-volume: pass EC peer reads an EcInterval instead of ten arguments

fetch_one_interval, read_remote_ec_shard_interval,
do_read_remote_ec_shard_interval and recover_one_remote_ec_shard_interval
all took the same (vid, needle_id, shard_id, shard_offset, size,
expected_encode_ts_ns) tuple, and the two that reconstruct also took the
location map with the data/parity counts. Those are now EcInterval (Copy)
and EcShardMap (a borrow of the map plus the counts). The fan-out inside
recovery builds its per-shard request with `EcInterval { shard_id: sid,
..iv }`, which is the one place the old argument list was easy to get
wrong. Bodies destructure at the top, so the code below the signatures
is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust-volume: give the EC encoder an EcEncodeLayout and an EncodeRun

encode_dat_file took the Reed-Solomon shape and three block sizes as five
loose integers; they are now one Copy struct, EcEncodeLayout, which is
what Go calls ECContext. The per-row and per-batch helpers took the same
six sinks and the offsets; they become methods on EncodeRun, which owns
the borrows for one run, so each call names only the offset and block
size that vary. The byte-level work is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust-volume: describe a .dat rebuild with DatRebuild instead of nine arguments

write_dat_file_from_shards, its _with_dirs twin and the private
write_dat_file were three layers over one nine-argument signature. One
public function now takes a DatRebuild, whose shard_dirs is None when
every shard sits beside the .dat and Some(dirs) for the cross-disk
reconciled layout. The field docs carry what the function doc used to
say about the encode-time size and the block layout.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust-volume: split copy_file_from_source's fifteen arguments into two structs

CopyFileSpec is the per-file request (what to ask the source for, where
it lands, whether its bytes count as progress); CopyProgress is the
sender, throttler and report state that all three files of one
VolumeCopy share, held by &mut across the calls. The three production
call sites now read as the .dat/.idx/.vif literals they are, instead of
positional trues and falses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust-volume: create volumes from a VolumeSpec

Volume::new, DiskLocation::create_volume and Store::add_volume each
took the same five-value tail of Go's NewVolume argument list:
collection, replica placement, TTL, preallocation and needle version.
That tail is now VolumeSpec, a Copy struct whose Default is what almost
every test wanted anyway (empty collection, no replication, no TTL, no
preallocation, current version), so most of the 104 call sites shrink
to `&VolumeSpec::default()` or name the one field they set. The id,
directories, index kind and disk type stay positional because they
differ at every site.

Two imports that only test modules use moved into those modules, and
DiskLocation no longer imports ReplicaPlacement.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-15 09:04:16 -07:00
Eliah Rusin 49a680dd64 rust: tonic 0.14 / prost 0.14, drop the result_large_err allow (#11327) 2026-09-15 04:40:46 -07:00
adaf3534fa rust: clippy-clean both crates and adopt the std APIs the 1.91 MSRV allows (#11312)
* rust: apply clippy --fix to both crates

The mechanical part of a clippy sweep: `cargo clippy --all-targets --fix`
on seaweed-volume and the seaweed-worker workspace, hand-reviewed. Both
manifests declare their MSRV (1.91.1 and 1.94.1), so every suggestion
clippy applied is within it: the collapsible_if sites become let chains
(1.88, edition 2024), `% n == 0` becomes is_multiple_of (1.87),
chunks_exact with a constant becomes as_chunks (1.88), repeat().take()
becomes repeat_n (1.82), and io::Error::new(Other, ..) becomes
io::Error::other (1.74). The rest is redundant clones, borrows, casts,
closures and field names.

Nothing here changes behaviour. The three let_and_return sites in
needle_map.rs and store_ec.rs deserve a note: the `let result = ..;
result` shape was a deliberate edition-2021 workaround to drop a redb
guard before the table it borrows. Edition 2024 drops tail-expression
temporaries before locals, which is why clippy now flags it, and the
two comments that described the workaround say so instead.

Manual edits on top of the tool output: the blocks clippy rewrote are
re-indented the way rustfmt lays them out (only those blocks — the
crate is not rustfmt-clean and a whole-crate fmt would bury this diff),
the blank lines let_and_return left behind are removed, and the CRC
legacy_value test compares against a literal worked out from the
original shift formula rather than restating rotate_right.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust: clear the clippy warnings --fix cannot apply, and say why the rest stay

Hand fixes for the lints clippy only reports. Behaviour is unchanged
throughout; each rewrite is the one clippy names.

- needless_range_loop (7): index loops over shard vectors become
  iterator loops. Where the old code indexed `v[..n]` the new loop
  iterates `v[..n]` so an undersized vector still panics the same way.
- field_reassign_with_default (6): struct literals with `..Default`.
- redundant_pattern_matching (3): `if let Err(_) = guard.check()` becomes
  `.is_err()`, which also releases the read guard at the end of the
  condition instead of at the end of the block.
- manual_strip (2), manual_checked_ops, format_in_format_args,
  redundant_locals, wrong_self_convention (to_vif takes self by value,
  so it is into_vif; CompactEntry is Copy, so to_needle_value takes self).
- type_complexity (2): `OrphanShardLoad` and `RawNeedleEntry` name two
  tuples that were spelled out inline.
- new_without_default: CompactNeedleMap gets a Default that calls new().
- suspicious_open_options: a test helper spells out `.truncate(false)`,
  which is what `.create(true).write(true)` already did.

What stays, and the attribute that says so:

- too_many_arguments (10): `#[expect]` on each function. Folding 8–15
  parameters into a struct is a design change, not a lint fix.
- await_holding_lock / readonly_write_lock: one test holds the store
  write guard across a sleep on purpose, as a barrier that parks the
  copy task at the mount block. `#[expect(.., reason = ..)]` records it.
- module_inception: needle/needle.rs mirrors the Go package layout.

Two lints become crate-wide policy in `[lints.clippy]`, with the reason
next to each: result_large_err, because every RPC path returns
tonic::Status (176 bytes) and boxing it would change every handler
signature; and needless_update, because `..Default::default()` on a
protobuf message literal is what lets a proto gain a field without
touching every constructor (all 11 sites are pb messages). The worker
workspace gets the same table and its members opt in with
`lints.workspace = true`; its generated plugin.rs also allows
large_enum_variant on prost's oneof enums.

Both crates are now clean under `cargo clippy --all-targets -- -D warnings`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust volume: use the std APIs the 1.91 MSRV already pays for

The crate declares rust-version 1.91.1, so a few things the code still
worked around are plain std now. All of them come from the 1.85–1.91
release notes; nothing here needs a newer toolchain than the manifest
already requires.

- std::sync::LazyLock (1.80) replaces the lazy_static! block in
  metrics.rs, and the lazy_static dependency goes. Every use site reads
  the same through Deref, so no caller changes.
- Duration::from_mins / from_hours (1.91) replace `from_secs(v * 60)`
  and `from_secs(v * 3600)` in the option parser and the shard-location
  refresh TTLs. One difference for the parser: an absurd count that
  overflows u64 seconds now panics in release builds too, where the
  multiplication used to wrap.
- Result::flatten (1.89) replaces `.and_then(|r| r)` on the replication
  join handle.
- OsStr::display (1.87) replaces `to_string_lossy()` where the name was
  only being formatted; the output is byte-identical.
- `#[allow]` becomes `#[expect]` (1.81) on the suppressions that are
  meant to be permanent, so a suppression that stops being needed
  becomes a warning rather than lingering. Doing that found four that
  already had: dead_code on ChunkManifest, base_name and last_io_error,
  and too_many_arguments on read_from_data_shards, which is down to
  seven parameters. Those attributes are deleted. The three allows that
  depend on cfg (a unix-only mutation, a linux-only field set, a
  profiling-only parameter) stay as allow, because expect would be
  unfulfilled on the other platforms.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* ci: add a commented-out clippy step to both Rust workflows

Both crates are warning-free under `cargo clippy --all-targets
-D warnings` now. Whether that becomes a gate is a policy call, so the
step is present but commented out; uncommenting it is the whole change.
The comment points at the `[lints.clippy]` table where crate-wide
exceptions are recorded, so the gate does not become a reason to
sprinkle allows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU

* rust volume: guard parse_duration against overflow panics

Duration::from_mins/from_hours panic when the count overflows u64
seconds. Use checked_mul so an oversized CLI value falls back to the
parser default instead of crashing volume startup.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-14 11:29:29 -07:00
5b2fe374fc [Volume] Scrub every disk's EC shards for a volume id, not just the first (#11258)
* storage: add Store::find_all_ec_volumes for split-disk EC lookups

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* ec: add merge_ec_runtimes to resolve a vid's per-disk shard set

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* ec: replace dead slots.get(14) assertion with a width-14 pin

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* ec: build the checksum scrub plan from every per-disk runtime

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* ec: build the local scrub plan from every per-disk runtime

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* ec: prove the local scrub plan reaches every runtime's slots

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* ec: make the scrub plan tests falsifiable

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* ec: report unverifiable protection when the sidecar predates the scrubbed encode

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* ec: commit sidecar provenance with the sidecar it describes

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* volume server: scrub every disk's EC shards for CHECKSUM and LOCAL

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* ec: run the FULL/READS parity check across split-disk shards

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* volume server: report fenced-out runtimes in FULL/READS scrubs

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* ec: tighten verify_ec_shards ordering and missing-shard coverage

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* volume server: visit each EC volume id once in node-wide scrubs

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* ec: cover split-disk scrub aggregation end to end

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* volume server: pin fenced-out disks and sibling-disk shards in EC scrubs

Three scrub behaviors shipped without a test at the RPC seam. Task 8
showed the seam exists, so close them here.

FULL/READS (mode 2|5) now marks a volume broken when the identity fence
excludes a runtime, where it previously reported clean. Pinned against a
control fixture whose two disks AGREE and scrub clean, so the test fails
on the clean->broken transition, not only on the message text. That needs
a structurally valid, tombstone-only .ecx (so the needle walk finds
nothing to complain about) and a seeded shard-location cache (so the
absent master does not short-circuit the scrub with an error of its own).

LOCAL (mode 3) and CHECKSUM (mode 4) now build their plans from every
per-disk runtime. Made observable by moving shard 0 -- the shard the
volume's single needle spans and the one the checksum sidecar is checked
against -- to the SIBLING disk, leaving shard 5 on the disk the singular
find_ec_volume lookup returns. Built from that disk alone, neither scrub
ever looks at shard 0.

The split-disk fixture grows a config struct rather than more positional
arguments; its defaults reproduce the existing layout byte for byte, so
the node-wide dedupe test is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* ec: report fenced-out disks on a malformed sidecar too

`errors.extend(self.skipped)` sat below the whole status match, so only
`(Some(p), On)` ever reached it. The Invalid arm already returns a
non-empty error vector of its own, so the Go-parity contract that
silences the Off arm (`case BitrotOff: return 0, nil, nil`) does not
reach it -- appending the fence lines there costs nothing that contract
protects. A volume with BOTH a malformed sidecar and a disk the identity
fence excluded reported only the sidecar, hiding the unscanned disk
behind an unrelated integrity error.

Off stays byte-identical, and so does the `(None, On)` arm that is
documented as treating a missing payload defensively as protection off.
Off is now the ONLY status that drops the report, and the comment at the
On-path copy says so: that is the one place the parity constraint costs
us coverage.

Also corrects a false claim in the FULL/READS test's doc comment. It
said a fenced-out disk "is a disk this scrub did NOT read", which is true
only of the merge-driven parity half. The per-needle walk still resolves
`store.find_ec_volume` (store_ec.rs:281) and binds
`expected_encode_ts_ns` to that runtime (:311) -- position 0, the
EXCLUDED one on that fixture -- so `read_local_intervals`' generation
filter (:1204) makes it read the excluded disk and treat the anchor's
shards as non-local, the inverse of what `skipped` reports. The fixture's
tombstone-only .ecx walks nothing, so the test cannot tell the two apart;
the comment now says that rather than implying coverage it does not have.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* ec: take CHECKSUM's bitrot protection from the disk that has the sidecar

`EcChecksumScrubPlan::for_volumes` read `(prot, status)` off the ANCHOR.
The anchor is the first shard-bearing runtime at the maximum `encode_ts_ns`,
chosen with no regard for which disk holds the `.ecsum`.

That sidecar is deliberately NOT mirrored across disks -- `ec_metadata_dirs()`
exists so one authoritative copy stays reachable rather than being duplicated
-- and at mount `EcVolume::new` resolves it via `load_active_bitrot_sidecar(&[])`
with no sibling directories at all; only the `VolumeEcShardsMount` RPC ever
passes `ec_metadata_dirs()`. So after EVERY volume-server restart, the
split-disk runtime that does not physically hold the sidecar mounts
`BitrotStatus::Off`. When the one copy lives on disk 1 and the anchor is disk 0,
`run()` hit `case BitrotOff` and returned `(0, [], [])`: the whole volume
scrubbed clean, silently. That is the steady state for roughly half of all
mirrored split-disk layouts, and it is the exact failure this branch exists to
remove.

Source protection from the first MERGED runtime that has any -- `On` if one
does, else `Invalid`, else the anchor's `Off`. Two facts make that safe, and
both are load-bearing:

  - Every runtime that mounted `On` already passed the `geometry_matches` gate
    in `load_bitrot_for_generation`, so its manifest agrees with the volume's
    layout. A sidecar that contradicted it would have failed the mount.
  - All merged runtimes share the same `encode_ts_ns` by construction of the
    identity fence, so a sidecar from any of them describes the same encode run.

The `unverifiable_sidecar` provenance rule four lines down read
`anchor.bitrot_source_dir`; it now reads the SAME runtime `prot` came from.
Otherwise the two would describe different sidecars and the rule would vouch
for a manifest nobody is scanning against. One consequence worth naming: that
source dir is now non-empty by construction (a runtime with protection found a
file), where the anchor's was often "" and short-circuited the rule -- so on a
fenced volume whose anchor had no sidecar, an unverifiable-protection note now
surfaces where previously nothing was reported at all.

`run()` is untouched, and the `BitrotStatus::Off` arm still returns
`(0, [], [])` exactly, for Go parity with `case BitrotOff: return 0, nil, nil`.
`parity_shards` still comes from the anchor while `prot` may come from a
sibling; the geometry gate above makes them agree, and slot-width agreement is
handled separately.

The test drives mode 4 through the real RPC against a split-disk volume whose
sidecar exists only on dir1, and asserts up front that the anchor mounted `Off`
and the sibling `On` -- otherwise it would prove nothing. Reverting this commit's
one-line source change makes it report `[]` instead of `[0, 5]`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* ec: pin the slot width, contain the shard-size fallback, and cover multi-disk FULL

Five findings from the whole-branch review, none of which changes what a
healthy volume reports.

Slot width was undefined and the two consumers disagreed (ec_volume.rs).
`merge_ec_runtimes` sizes `slots` to the WIDEST merged runtime, but the identity
fence keys on `encode_ts_ns` alone and never on geometry -- so two
same-generation runtimes whose `.vif`s disagree do merge. The mode 2|5 arm
truncates to the anchor's `data+parity` and silently drops the surplus slots,
while `EcChecksumScrubPlan::for_volumes` iterated the full width and emitted
"present but missing from sidecar manifest" for exactly those ids. Nothing in
the volume describes them -- the sidecar manifest and the Reed-Solomon matrix
are both the anchor's -- so that message was the width disagreement talking, not
a finding. The `slots` field doc now states the contract (the range is the
anchor's geometry; every consumer truncates to it) and CHECKSUM truncates.

The LOCAL `shard_size` fallback had grown a node-wide blast radius
(ec_volume.rs). `anchor.shard_file_size()` returns the anchor's FIRST held
shard, not a maximum. Before aggregation the plan read only that runtime's own
shards, so a truncated shard was contained to its disk; now that one value sizes
every merged sibling's shards, mis-offsetting `locate_data` and manufacturing
needle corruption across the node. Take the max over the merged slots, which is
how `verify_ec_shards` already answers the same question
(`if size > shard_size { shard_size = size }`). Only on the legacy
`dat_file_size == 0` path.

Multi-disk `all_local` had no end-to-end test (grpc_server.rs). The parity check
is gated on every shard being present, and the one all-local fixture keeps them
in a single directory, so every entry of `dirs` is the same string and a
permutation or off-by-one in the `slots` -> `dirs` mapping is invisible;
`test_verify_ec_shards_reads_shards_from_multiple_dirs` builds its `dirs` by
hand and never goes through `merge_ec_runtimes`. The new fixture is a real 10+4
encode split 0..=6 / 7..=13 across two store locations (the `.dat`/`.idx` stay
outside both, so `prune_incomplete_ec_with_sibling_dat` has nothing to act on),
driven through the real RPC: clean first, then a corrupted PARITY shard on the
SECOND disk -- which only the parity half can see, and only through a correct
mapping. Shifting that mapping by one, or computing `all_local` from the anchor
alone, both make it report `[]` instead of `[13]`.

Deleted `test_ec_volume_enumeration_is_deduped` (store_ec_reconcile.rs). It
built `raw` from `store.locations` and then applied its OWN inline
`filter(|v| seen.insert(*v))`, asserting on that -- a property of
`HashSet::insert`, never reaching the production dedupe. That path is covered by
`test_scrub_ec_volume_node_wide_dedupes_a_split_disk_volume`, which does fail
(2 != 1) when the dedupe is removed.

Corrected `test_verify_ec_shards_treats_a_none_dir_as_missing`'s docstring
(ec_encoder.rs). It claimed the unmounted shard "must not drag the shards that
ARE mounted down with it", but `dirs[5] = None` puts shard 5 in `broken_shards`
before the block loop, so every iteration takes the `read_failed` arm and the
parity comparison never runs: corrupting a mounted shard in that fixture changes
nothing about the result. The assertions are unchanged; the docstring now states
what they actually establish.

Also refreshed two comments that cited `shard_file_size() - 1` as the reason
`merge_ec_runtimes` prefers a shard-bearing anchor -- true before this commit,
stale after it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* ec: correct the Fix 1 rationale and truncate the shard-size scan

The safety argument attached to `EcChecksumScrubPlan::for_volumes`'s protection
selection was false as written, and it is the argument a reviewer reads first.
`geometry_matches` compares a sidecar against the MOUNTING runtime's own
data/parity/block size, not the anchor's, and returns true vacuously when
`ec_shard_config` is `None` -- so it establishes agreement only when all merged
runtimes share one geometry, which an `encode_ts_ns`-only fence does not
guarantee and which `test_checksum_scrub_truncates_slots_to_the_anchors_geometry`
constructs a counterexample to. The second clause was weaker than stated too: a
`.ecsum` records no encode identity at all, so merged runtimes agreeing on
`encode_ts_ns` does not transfer to the sidecar.

Replace it with the property that is true, checkable from the selection itself,
and stronger for what actually matters. `anchor` is an element of `merged`, so
the `.unwrap_or(anchor)` fallback is reached only when no merged runtime is `On`
and none is `Invalid` -- in which case the anchor is necessarily `Off`. The
status can therefore only move `Off -> On`, `Off -> Invalid` or
`Invalid -> On`; never `On -> Off`, never `Invalid -> Off`. This selection
cannot stop a volume that was being scanned from being scanned, and cannot turn
a reported integrity error into silence: every change it makes is toward more
verification. The comment now also states what it does NOT establish -- geometry
agreement is not guaranteed -- and names geometry fencing as the follow-up that
would close it.

Second, `EcLocalScrubPlan::for_volumes`'s `shard_size` max scanned the FULL slot
width, violating the `slots` contract documented in the same commit that
introduced the max: the volume's shard-id range is the anchor's geometry and
every consumer must truncate to it. Pre-fix that input could not exist, because
`anchor.shard_file_size()` read only the anchor's own anchor-sized vector -- so
the max opened a new, narrow path to the same node-wide mis-sizing it exists to
close (same-generation runtimes with disagreeing `.vif`s, the wider one holding
an out-of-geometry shard larger than the in-geometry ones, `dat_file_size == 0`).
`.take(anchor.data_shards + anchor.parity_shards)` mirrors the truncation
already applied to the CHECKSUM shard scan.

The sibling `shards:` vector is left untruncated on purpose: every access in
`EcLocalScrubPlan::run` is `shards.get(sid)` with `sid < data_shards`, so the
surplus entries are inert.

No behavior change for any healthy volume, and no test added -- the suite is
unchanged at 575 passing, 0 failing, 0 warnings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUf2cmVKHNhAZPTNv39rDE

* ec: aggregate split-disk runtimes in Go scrubs, mirroring Rust

Go volume scrubs previously used FindEcVolume (first runtime only), so a
volume whose EC shards are split across multiple disks was scrubbed against
just one disk's shards and the others were silently skipped. Node-wide
ScrubEcVolume also appended each disk's EcVolumeIds without deduplication,
scrubbing a split-disk volume once per disk.

Add MergedEcRuntimes/MergeEcRuntimes (Go counterpart to Rust's
merge_ec_runtimes): select the maximum EncodeTsNs as the anchor generation,
fence out runtimes whose encode generation or geometry (DataShards,
ParityShards, BlockSize) disagrees with the anchor, merge shard handles by
shard ID, and report excluded runtimes rather than dropping them. Wire it
into every scrub mode:

- INDEX: scrub the anchor's index, report skipped runtimes.
- LOCAL: aggregate local shards across all merged runtimes via a synthetic
  EcVolume built from the merged shard slots.
- FULL/READS: resolve the runtime matching the anchor's encode generation
  (not the first match) so the needle walk and parity phase inspect one
  encode run; report skipped runtimes.
- CHECKSUM: take bitrot protection from the first merged runtime that has a
  valid sidecar (On, else Invalid, else anchor's Off), preserve invalid
  sidecar errors from every other merged runtime, and report skipped
  runtimes.

Deduplicate EC volume IDs in node-wide ScrubEcVolume so each volume is
scrubbed exactly once.

Refactor ScrubEcVolume to share the per-needle walk via scrubEcVolumeWalk,
called by both the legacy first-runtime path and the new merged path.

Add Go regression tests covering split-disk deduplication, encode-generation
fencing, geometry fencing, sibling-disk LOCAL reach, and merge anchor
selection.

Rust: keep the previously-landed merge/fence/checksum changes intact; revert
incidental cargo-fmt drift from unrelated files so the diff stays focused.

* ec: fence merged CHECKSUM on sidecar encode generation and fix legacy shard size

Address two review findings on the Go merged-runtime scrub:

1. Sidecar provenance: a merged runtime can load a bitrot sidecar from a
   sibling metadata directory (ReloadBitrotSidecar), and the merge fence may
   then exclude the runtime owning that directory. Generation-0 sidecars do
   not identify the encode run, so geometry validation alone cannot prove the
   borrowed manifest describes the anchor shards. If the sidecar records a
   non-zero EncodeTsNs that disagrees with the anchor, refuse the scan
   instead of applying stale checksums to current shards and reporting false
   corruption.

2. Legacy shard size: for volumes without datFileSize in .vif,
   LocateEcShardNeedleInterval derives the shard size from Shards[0].ecdFileSize.
   The merged shard set is compacted in shard-ID order, so a truncated
   lowest-ID shard would shrink every interval and misread intact sibling
   shards. Synthesize a datFileSize from the maximum mounted shard size when
   the anchor lacks one, so the datFileSize>0 path uses the largest shard
   size across all merged runtimes.

* ec: fix copylocks, legacy shard boundary, and encode-aware Rust lookups

Address review findings from CodeRabbit and Devin:

Go (ec_volume_merge.go):
- Remove bitrotLock copy from the synthetic EcVolume: copying a sync.RWMutex
  is a go vet copylocks error. The synthetic volume uses its own zero-value
  mutex; bitrot/bitrotStatus are set directly before ChecksumScrub reads them
  via BitrotProtection(), so no concurrent access occurs.
- Fix legacy shard-size boundary: synthesize datFileSize from
  (maxShardSize - 1) * DataShards, not maxShardSize * DataShards, to match
  the legacy fallback in LocateEcShardNeedleInterval (ecdFileSize - 1). An
  exact large-block boundary is ambiguous; the unadjusted size would select
  an extra large row and misread intact sibling shards.

Rust (store_ec.rs):
- Add find_ec_volume_for_scrub helper that resolves by encode generation
  (not first-match find_ec_volume) and use it in scrub_snapshot_under_lock,
  write_back_shard_locations, and the post-refresh shard-location read.
  Previously the encode-aware lookup was only used for the initial runtime
  selection; the cache write-back and per-needle snapshot still used
  first-match, so a split-disk volume whose first runtime was from an older
  encode run would write to and read from the wrong runtime's shard-location
  cache and falsely abort with 'remounted as a different encode run'.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-12 14:19:18 -07:00
3ae9e332ec rust volume: honour is_last in the tail sender instead of rescanning the whole volume (#11273)
* rust volume: honour is_last in the tail sender instead of rescanning

volume_tail_sender discarded the is_last flag from
binary_search_by_append_at_ns:

    Ok((offset, _is_last)) => {
        if offset.is_zero() { Ok(sb_size) } ...

is_last means the caller is already caught up. Go answers that with a
heartbeat and does not scan at all (volume_grpc_tail.go, `if isLastOne`).
Dropping it is expensive rather than untidy, because the branches interact:
when the search reports caught-up it returns Offset::default(), which is
zero, so the start offset falls back to sb_size -- the beginning of the
data -- and scan_raw_needles_from materialises every needle from there to
EOF into a Vec. The timestamp filter discards all of it, the loop sleeps
2s, and it happens again.

A volume being moved is marked read-only before the copy, so it is ALWAYS
caught up during the tail phase. Measured on one volume.move of a 2.15 GB
volume, sampling the source's cgroup anon every 2s against the move's own
phase output:

  copying   16 -> 37 MB          CopyFile streams correctly, stays bounded
  tailing   904 -> 2166 -> 629 -> 2166 -> 342 -> 2173 -> 2179 MB
  deleting  46 MB

Six full-volume allocate/free cycles in 35s, peak 2179 MB against a volume
of 2147 MiB. The destination never exceeded 35 MB, so this is entirely
source-side. Under a per-process memory cap it OOM-kills the source
whenever the volume exceeds the cap.

The ordering here is the whole fix and is easy to get wrong: resolve the
start offset and is_last under a brief lock, return the heartbeat
immediately when caught up, and only then reach the scan. An earlier cut
set the flag correctly but placed the early return after the block that
performs the scan -- the heartbeat fired and the destination received
nothing, yet every iteration still read the whole volume and discarded it.
Production showed no improvement (1770 MB across five cycles), which is
what caught it. The binary search is over the .idx and costs nothing; the
scan is the expensive part and must not run speculatively.

Three tests, and the last two matter as much as the first: a fix that
always reported "caught up" would make tailing silently lose needles, a
worse bug than the one being fixed. One asserts is_last for a caller at or
beyond the newest append_at_ns; one asserts NOT is_last for a caller that
is behind, so real tail data is still scanned and shipped; one asserts NOT
is_last when the only newer record is a delete, and that scanning from the
returned offset ships exactly that tombstone.

Left deliberately unfixed, and worth separate changes: the scan still
collects into a Vec rather than streaming through a visitor as Go's
ScanVolumeFileFrom does, and it runs while holding store.read(), the same
lock-across-a-large-read shape as #11235. Both are latent once the rescan
is gone, since remaining scans are bounded by genuinely new data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFr2v4BUqrXdgj4LEUAwVF
Claude-Session: https://claude.ai/code/session_018VF7E9SHPihG1jC1grU9H3

* rust volume: resolve and scan the tail under one store guard

The tail sender took store.read() once for the binary search and again
for the scan. A vacuum commit takes the store write lock and swaps
.dat/.idx, so it could land between the two: the offset resolved against
the old files would then be applied to the new ones and start the scan
inside an unrelated record. The code before the is_last fix held a
single guard for both. Restore that, and scan only when the caller is not
caught up, so the caught-up heartbeat still skips the scan and is sent
outside the lock.

Also pin the compacted-volume boundary raised in review. Compaction
writes .idx in needle-id order in both Go and Rust, so the search can
report caught-up while an earlier row is newer; such a caller's since_ns
is the last row's timestamp, so those rows were in the files it copied.
A write made afterwards is appended as the final row, which the search
cannot step past. The new test asserts it still reaches the scan.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018VF7E9SHPihG1jC1grU9H3

* rust volume: make the compaction tail test a genuine overwrite

The compaction regression test's second id=1 write reused the first
write's data, so write_needle's dedup short-circuit (is_file_unchanged)
returned without appending or updating append_at_ns. Compaction then
kept key 1's original (older) timestamp, so the test passed without
exercising the overwrite it describes -- key 2 was the final row only
because key 1 was never actually newer.

Give the overwrite distinct data so it appends a new record, and assert
key1_ns > key2_ns up front so a future dedup regression fails the test
instead of silently hollowing it out. Trim the verbose comments on the
tail sender and the binary-search tests to their essentials.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-11 09:42:20 -07:00
4f9bbd51cb rust volume: stop glibc retaining freed EC buffers as unreturnable heap (#11255)
* rust volume: stop glibc retaining freed EC buffers as unreturnable heap

A Rust volume server doing EC work accumulates hundreds of MB of resident
anonymous memory that it never gives back, and under a hard cgroup
MemoryMax that ends in an OOM kill while most of the resident set is
free-but-unreturned.

It is not a leak. glibc serves allocations >= M_MMAP_THRESHOLD with mmap
and munmaps them on free, but the threshold is ADAPTIVE: freeing an
mmap'd block raises it toward that block's size, up to 32 MiB. EC
reconstruction and needle reassembly allocate large short-lived buffers,
so the first few train the threshold upward and every later buffer is
carved from the heap instead. Heap pages only return to the OS from the
top of the arena, so they stay resident for the life of the process --
reusable, but anonymous, and anonymous pages cannot be reclaimed under
pressure the way page cache can. The retained footprint is exactly the
headroom a burst of maintenance work needs.

Measured on a 17-node cluster (EC 10+4, --index=redb), one node, two
identical `ec.scrub -mode full` rounds over 10912 EC files each, same
unit restarted with and without a pinned threshold:

                       baseline  round 1  round 2  60s idle
  default (adaptive)      10 MB    84 MB    88 MB     88 MB
  pinned threshold        10 MB    13 MB    14 MB     14 MB

78 MB retained versus 4 MB for identical work. On heavier mixed scrub
workloads the same effect reached ~600 MB per volume server against a
3 GiB cap, and restarting the process was the only way to release it.

Calling mallopt(M_MMAP_THRESHOLD, ...) sets the threshold and disables
the dynamic adjustment. Pin it to glibc's own default rather than
inventing a value: the goal is to stop the adaptation, not to second-guess
the default. MALLOC_MMAP_THRESHOLD_ still wins if an operator sets it,
glibc-only, and a failed mallopt is logged rather than fatal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFr2v4BUqrXdgj4LEUAwVF

* Address PR review: validate env overrides, honour GLIBC_TUNABLES, fix non-glibc test compile

Three review-bot findings on seaweed-volume/src/malloc_tuning.rs:

1. (CodeRabbit) The test used cfg!(...), which keeps both branches in
   compilation. On non-glibc targets DEFAULT_MMAP_THRESHOLD is undefined,
   so the test failed to compile. Split into #[cfg]-gated tests so each
   branch only references items defined for that target.

2. (Greptile) MALLOC_MMAP_THRESHOLD_ was checked by presence only. An
   empty or non-numeric value makes glibc ignore the override while we still
   skipped mallopt, leaving the adaptive threshold enabled -- exactly the
   behaviour this module exists to prevent. Now we defer only when the value
   is non-empty and parses as an integer; otherwise we fall through to
   pinning.

3. (Codex) The modern GLIBC_TUNABLES=glibc.malloc.mmap_threshold=... tunable
   was missed, so mallopt could overwrite an operator's explicit tunable. Now
   we detect that tunable (with the same validation) and defer to it.

The override check moved into the glibc-gated inner function, so off glibc
pin_mmap_threshold() always reports NotApplicable regardless of any
allocator env vars that happen to be set. The startup log for DeferredToEnv
is reworded to cover both override sources. Added tests for the override
parsers and the off-glibc no-op.

* Address round-2 review: match glibc's actual override parsing

Three follow-up review-bot findings after the first round of fixes, all
rooted in our validation not matching how glibc actually parses the
overrides:

1. (Greptile, P1) parse::<i64>() accepted negative values like "-1" and
   returned DeferredToEnv, but glibc's threshold is unsigned and rejects
   negatives — so we skipped mallopt while glibc also ignored the override,
   leaving the adaptive threshold enabled. Now we reject negatives and
   zero.

2. (Devin, BUG) glibc parses thresholds as unsigned (strtoul for tunables,
   atoi for the legacy var). Values above i64::MAX are valid for glibc but
   were rejected by parse::<i64>(), so we pinned 128 KiB over the operator's
   explicit setting. Now we parse as u64, accepting the full unsigned range.

3. (CodeRabbit, Major) Two issues in usable_glibc_tunable_threshold:
   a. A malformed sibling entry (e.g. glibc.malloc.check=2=2:...) makes
      glibc reject the entire GLIBC_TUNABLES string, but our per-entry scan
      still returned true for the valid-looking mmap_threshold entry. Now
      we validate every entry (exactly one '=') before accepting any.
   b. Hex values (0x20000) are accepted by glibc's strtoul but were rejected
      by parse::<i64>(). Now parse_strtoul_threshold handles 0x-prefixed hex.
      MALLOC_MMAP_THRESHOLD_ stays decimal-only (atoi), matching glibc.

Added regression tests for negatives, zero, >i64::MAX, hex tunables, and
malformed mixed GLIBC_TUNABLES entries. Verified: clippy clean and tests
pass on macOS (non-glibc); glibc-gated code type-checks for
x86_64-unknown-linux-gnu.

* Address round-3 review: match glibc's actual override parsing

Three follow-up review-bot findings (Greptile P1, Devin BUG, CodeRabbit
Major) all on the same issue: the round-2 fix rejected negative and zero
override values, but glibc actually accepts them.

Verified against the glibc source (malloc/malloc.c, malloc/arena.c,
elf/dl-tunables.c, elf/dl-misc.c):

- do_set_mmap_threshold(size_t value) does NO clamping — it just sets
  mp_.mmap_threshold = value and mp_.no_dyn_threshold = 1.
- MALLOC_MMAP_THRESHOLD_: glibc calls atoi(value) then mallopt, which
  always sets the threshold and disables dynamic adjustment — even for
  empty, negative, or non-numeric values (atoi returns 0). So ANY
  presence of the variable means the operator's override is in effect.
  Reverted to presence-only check for the legacy variable. The round-1
  Greptile comment claiming glibc "cannot apply the override" for
  empty/malformed values was incorrect.
- GLIBC_TUNABLES: glibc parses values with _dl_strtoul (elf/dl-misc.c),
  which accepts decimal, 0x hex, 0 octal, an optional sign (negatives
  wrap to unsigned long), and requires the entire value consumed
  (tunable_parse_num checks endptr == strval + len). Replaced
  parse_strtoul_threshold with dl_strtoul_consumes_all that replicates
  _dl_strtoul's parsing and checks full consumption. Now accepts -1
  (wraps to SIZE_MAX), 0, 0x20000, 010 (octal), and values above
  i64::MAX.

The duplicate-= validation for GLIBC_TUNABLES (from round 1) is kept —
glibc's parse_tunables_string returns -1 if any entry's value contains
a duplicate =, rejecting the entire string.

Added dl_strtoul_consumes_all tests covering decimal, hex, octal,
negative, zero, empty, whitespace, trailing garbage, and sign-only
inputs. Updated usable_glibc_tunable_threshold tests to accept
negative, zero, and empty values. Verified: clippy clean and tests
pass on macOS (non-glibc); glibc-gated code type-checks and clippy
clean for x86_64-unknown-linux-gnu.

* Address round-4 review: add overflow detection, fix sign-only test assertions

Two Greptile P1 findings:

1. Overflowing tunables bypass threshold pinning: dl_strtoul_consumes_all
   consumed every digit and returned true for values like
   18446744073709551616 (u64::MAX + 1), but glibc's _dl_strtoul stops at
   the overflowing digit (sets endptr there, returns UINT64_MAX), so
   tunable_parse_num rejects the value (endptr != strval + len). Added
   overflow detection matching glibc's cutoff/cutlim logic — on overflow,
   the parser stops and returns false.

2. Sign-only parser assertions fail: the test asserted
   !dl_strtoul_consumes_all("-") and !dl_strtoul_consumes_all("+"), but
   _dl_strtoul skips the sign, finds no digit, sets endptr to the position
   after the sign (== end of string), and returns 0. tunable_parse_num
   sees endptr == strval + len → true. So glibc accepts sign-only strings
   as value 0. Fixed the test assertions to expect true.

Also fixed "0x" with no hex digits: _dl_strtoul parses "0" as octal, then
stops at "x" (not an octal digit), so endptr != end of string → rejected.
The base-detection now requires a hex digit after "0x" before switching
to hex; otherwise "0" is parsed as octal and "x" stops the parser.

Added overflow regression tests: 18446744073709551616 (u64::MAX + 1),
99999999999999999999 (20 nines), 0x10000000000000000 (2^64). Verified:
clippy clean and tests pass on macOS (non-glibc); glibc-gated code
type-checks and clippy clean for x86_64-unknown-linux-gnu.

* Address round-5 review: accept bare 0x prefix, remove unused helper

Two review-bot findings (Devin BUG + CodeRabbit Major) on the same issue:
the round-4 fix required a hex digit after "0x" before switching to hex
base, but glibc's _dl_strtoul unconditionally advances past "0x"/"0X"
when the first char is '0' and the next is 'x'/'X' — even if no hex digit
follows. In that case the digit loop breaks immediately, endptr reaches
the end, and the value is 0. tunable_parse_num accepts it.

Removed the is_digit_in_base lookahead from the base-detection condition
and the now-unused is_digit_in_base helper. Updated the test assertions
for "0x" and "0X" to expect true (accepted as value 0).

The Greptile P1 overflow comment is invalid: glibc's _dl_strtoul rejects
18446744073709551616 (u64::MAX + 1) — on overflow it sets endptr to the
overflowing digit (not end of string) and returns UINT64_MAX, so
tunable_parse_num sees endptr != strval + len and rejects. My
implementation correctly returns false for this value, matching glibc.

Verified: clippy clean and tests pass on macOS (non-glibc); glibc-gated
code type-checks and clippy clean for x86_64-unknown-linux-gnu.

* Address round-6 review: rewrite tunable parser to match glibc exactly

Two Greptile P1 comments (3975151906, 3975151911) both invalid, but
investigation revealed a real bug in the split(':')-based parser:

Bug: usable_glibc_tunable_threshold used split(':') which loses the
distinction between an entry terminated by ':' (glibc skips it) and one
terminated by '\0' with no '=' (glibc rejects the entire string). Examples:
  - "glibc.malloc.mmap_threshold=262144:glibc.cpu.x" (no '=' at end):
    glibc rejects entire string, old code accepted it.
  - "glibc.malloc.mmap_threshold=262144:" (trailing ':'):
    glibc rejects entire string, old code accepted it.

Fix: replaced split(':') with a character-by-character parser matching
glibc's parse_tunables_string exactly. The parser tracks position in the
original string and correctly handles all three terminators ('=', ':', '\0')
for both name and value scanning.

Comment 3975151906 (near-maximum values): Invalid. Verified against
_dl_strtoul: for 18446744073709551615 (u64::MAX), cutoff = u64::MAX/10,
cutlim = u64::MAX%10 = 5. After 19 digits result == cutoff. 20th digit 5:
overflow check (digval > cutlim) is 5 > 5 = false → no overflow. glibc
accepts u64::MAX. Added regression test asserting it's accepted.

Comment 3975151911 (later malformed entry): Invalid. Verified against
parse_tunables (elf/dl-tunables.c): when parse_tunables_string returns -1,
parse_tunables prints a warning and returns immediately without applying
ANY tunable — including ones already parsed into the array. Added
regression test for "threshold=262144:check=2=2" (threshold before
malformed sibling) asserting it's rejected.

Added regression tests: u64::MAX accepted, threshold-before-malformed
rejected, no-'=' at end rejected, trailing ':' rejected, leading ':'
accepted. Verified: clippy clean and tests pass on macOS; glibc-gated
code type-checks and clippy clean for x86_64-unknown-linux-gnu.

* Fix CI: correct hex trailing-garbage test assertion

The test asserted !dl_strtoul_consumes_all("0x20000abc"), but in hex
mode a-f are valid digits — "0x20000abc" is a valid hex number
(0x20000abc = 536874044), not trailing garbage. _dl_strtoul consumes
the entire string and tunable_parse_num accepts it. The assertion
failed on Linux CI where the glibc-gated test actually runs.

Replaced with "0x20000g" — 'g' is not a hex digit, so _dl_strtoul
stops at 'g' and tunable_parse_num rejects the value.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-10 09:29:31 -07:00
516e251f9e rust volume: move the crate to edition 2024 (#11244)
* rust volume: move the crate to edition 2024

Edition 2024 turns three things in this crate into hard errors, and changes
drop order in a further 34 places without changing compilation. The compiler
errors are fixed here; the silent changes were audited against
`RUSTFLAGS='-W rust-2024-compatibility' cargo check --all-targets` output
captured before the flip, since edition 2024 stops reporting them.

`std::env::set_var`/`remove_var` are unsafe as of 2024 because they race with
concurrent readers. All six call sites are safe by construction rather than by
assertion, and the SAFETY comments say why: the build script runs
single-threaded before anything else in the process, and every test reaching
the `config.rs` helpers holds `process_state_lock()` for the duration.

The two `ref` bindings in handlers.rs sit in patterns that already borrow
implicitly, so removing the modifier leaves both bindings at `&String`.

On the 34 drop-order sites: no lock guard's scope is extended anywhere, and
`volume.rs` has none. Most are moved-from `Option`/`Result` husks — `if let
Some(v) = map.remove(&k)`, `while let Some(m) = stream.next().await` — where
the value is moved into the binding and the temporary has nothing left to drop;
where closing order actually matters these paths already call `v.close()`,
`ec_vol.destroy()` or `drop(writer)` explicitly. Two sites get strictly better
ordering: the metrics read guard in `run_metrics_push_loop` shrinks to the end
of its initializer block (it never crossed an `.await` either way), and an EC
test now closes the volume's descriptors before the `TempDir` removes the
directory.

No `rust-version` is declared. Edition 2024 needs rustc 1.85, but that is not
the binding constraint — the dependency tree already requires 1.91.1 through
the `aws-sdk-s3`/`aws-smithy-*` family, so `cargo +1.85 check` fails on the
deps regardless. CI builds on `dtolnay/rust-toolchain@stable`.

`vendor/reed-solomon-erasure` is a separate package and keeps edition 2021.
Cargo.lock is unchanged despite edition 2024 implying resolver 3.

Verified: `cargo test` 551 passed / 0 failed, `cargo test
--no-default-features` 550 passed / 0 failed (the two feature sets produce an
identical migration site list), `cargo build --release` clean. No automated
test covers shutdown ordering, so the channel and runtime sites in `main.rs`,
`write_queue.rs` and `grpc_server.rs` were read individually.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC

* rust volume: address edition-2024 review feedback

Three fixes from review of the edition bump.

Serialize the two environment-reading tests. The SAFETY comments on the
`env::set_var`/`remove_var` helpers claim every test touching the environment
holds `process_state_lock()`, but `test_resolve_config_defaults_dir_to_platform_temp_dir`
and `test_resolve_config_index_accepts_redb_and_leveldb_aliases` called
`resolve_config` — which reads HOME/USERPROFILE, SEAWEED_WRITE_QUEUE and the
WEED_* set — without taking it. `set_var` is unsafe precisely because a
concurrent *reader* is UB, not only a concurrent writer, so the comment was
overclaiming. An audit of the module found exactly these two; every other
environment-touching test already held the lock. The race predates edition
2024, which only made the requirement explicit.

Declare `rust-version = "1.91.1"`. The edition needs 1.85, but that was never
the binding constraint: `cargo +1.90 check --all-targets` fails on the
`aws-sdk-s3`/`aws-smithy-*` family, and 1.91.1 checks clean. Declaring the
verified floor turns a wall of per-dependency errors into one clear message.
Cargo.lock is unchanged despite this making the resolver MSRV-aware.

Update the README, which advertised "Rust 1.75+ (2021 edition)". 1.75 was
already stale before this branch — the tree has needed 1.91 for a while.

Verified: `cargo test` 551 passed / 0 failed, `cargo test --no-default-features`
550 passed / 0 failed, `cargo build --release` clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC

* rust volume: state the exact MSRV patch release in the README

The README said "Rust 1.91+", which reads as 1.91.0 and is wrong by one patch
release: `cargo +1.91.0 check --all-targets` fails on the aws-sdk-s3 family,
`cargo +1.91.1` passes. Say 1.91.1+, matching `rust-version` in Cargo.toml, and
call out that the patch component is load-bearing so nobody installs 1.91.0 and
hits the same wall.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC

* rust worker: move the workspace to edition 2024

Moves the seaweed-worker workspace (core, lance, sort) from edition 2021 to
2024, the same migration seaweed-volume just got in this branch.

Edition 2024 turns exactly one thing in this workspace into a hard error. The
baseline came from RUSTFLAGS='-W rust-2024-compatibility' cargo check
--all-targets, run before the flip; unlike seaweed-volume's 34 silent +
8 hard sites, the worker reports only the one hard site and no
tail_expr_drop_order or if_let_rescope sites at all. The worker is a much
smaller crate and none of its expressions hold a guard or temporary whose drop
order the edition changes, so there is nothing to audit on the silent side.

Fixed (1 site):

std::env::set_var is unsafe as of 2024 because it races with concurrent
readers. The single call is in crates/core/build.rs, which sets PROTOC from
protoc_bin_vendored the way seaweed-volume's build script does. A build
script's main runs single-threaded before anything else in the process, so
no other thread can be reading the environment concurrently; the SAFETY comment
says so. There are no config.rs-style test helpers here -- the worker's tests
do not mutate the environment -- so unlike the volume crate there are no
process_state_lock() callers to audit.

No redundant ref bindings to clean up: a grep for ref across the three
crates finds none.

MSRV:

rust-version = "1.94.1", verified rather than inferred. Edition 2024 only
needs 1.85, but the dependency tree needs more: lance's aws feature pulls in
a newer cut of the same aws-sdk-*/aws-smithy-* family that sets
seaweed-volume's 1.91.1 floor, and that newer cut requires 1.94.1.
cargo +1.94.0 check --all-targets fails on that family; cargo +1.94.1
check --all-targets is clean. The worker's floor is therefore higher than
the volume's, and moves with lance and the AWS SDK rather than with the
edition. CI builds on dtolnay/rust-toolchain@stable, so nothing changes
there.

The edition is set once in [workspace.package] and inherited by each member
via edition.workspace = true; rust-version is added the same way. The
workspace keeps its explicit resolver = "2" -- edition 2024 would default to
resolver 3, but the pin is deliberate and Cargo.lock is unchanged by this
commit either way.

The README gains a "Requires Rust 1.94.1+ (2024 edition)" line in its Building
section, matching the one seaweed-volume's README now carries, and calling out
that the patch release is load-bearing (1.94.0 does not build) so nobody
installs 1.94.0 and hits the same wall.

Verification:

* cargo check --all-targets -- clean, zero warnings (default toolchain 1.97)
* cargo +1.94.1 check --all-targets -- clean
* cargo +1.94.0 check --all-targets -- fails on the AWS SDK, as claimed
* cargo test --all-targets -- 40 passed, 0 failed
  (core 13, sort 11, lance lib 3, lance bin 2, compaction 6, lifecycle 1,
  sort integration 4)
* Cargo.lock unchanged

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-09 10:54:06 -07:00
9b12d13934 volume server: release the store lock before scrubbing EC volumes (#11235)
* volume server: release the store lock before scrubbing EC volumes

`ec.scrub` makes a Rust volume server stop serving for the duration of the
scrub, and then kills its own gRPC connection:

    error: rpc error: code = Unavailable desc = keepalive ping failed to
    receive ACK within timeout

Measured on a 4.46 cluster (17 Rust volume servers on one host, ~520 volumes
and 53 EC volumes, --index=redb, EC 10+4). It reproduces against a SINGLE
node in 30-70s, in checksum, index and local modes, at -maxParallelization 1.

## Cause

The CHECKSUM arm of scrub_ec_volume reads every byte of every local shard
while holding the caller's store.read() guard:

    let store = self.state.store.read().unwrap();
    let ecv = store.find_ec_volume(vid)...?;
    let (blocks, broken, errs) = ecv.checksum_scrub();   // GBs of I/O, lock held

VolumeServerState::store is a std::sync::RwLock, which is write-preferring.
The periodic heartbeat's collect_heartbeat_with_snapshot takes store.write()
and blocks; once that writer is pending, every later store.read() queues
behind it. Every HTTP handler takes store.read(), so the node serves nothing,
stops heart-beating, and cannot answer the scrub RPC's own keepalive - the
scrub kills the connection it is running on.

The INDEX and LOCAL arms have the same shape, and the node-wide scrub_volume
loop is worse: it held ONE guard across every volume on the node.

## Evidence

offcputime, off-CPU stacks >1s in a 30s window during a scrub:

    futex_wait
      seaweed_volume::server::heartbeat::collect_heartbeat_with_snapshot
      - tokio-rt-worker
        27967020        <- 27.97s blocked, of a 30s window

A single HTTP /status request issued 12s into a scrub, with 180s of patience,
was accepted and queued for 120 seconds, then served once the scrub released.
Thread states throughout: 1 D + 48 S. One thread working, 48 idle - not
executor starvation and no thread pileup, which is what a single lock holder
looks like.

Memory was tested and ruled out as the cause: the same scrub was run at
MemoryMax 3G, 8G and unlimited. With no limit there is no reclaim at all,
page cache grows freely to 22 GB, and the node still goes unresponsive at
t+30s. anon stays flat at 48-86 MB in every run.

## Fix

checksum_scrub, scrub_index and scrub_local gain plan types -
EcChecksumScrubPlan, EcIndexScrubPlan and EcLocalScrubPlan - snapshotted from
the volume under a brief guard. The handler builds a plan, drops the guard,
and runs the scan in spawn_blocking, off the async workers, since it is
synchronous CPU + file I/O either way.

A plan captures DESCRIPTORS, not paths. Resolving a path again after the
guard is dropped would let a writer that legitimately unlinks the files - the
heartbeat's delete_expired_ec_volumes, which reaches EcVolume::destroy(), or
volume_ec_shards_delete - surface an intentional removal as "scrub read
error: No such file or directory" and put the volume in broken_volume_ids. A
descriptor outlives the name.

For the shards it duplicates the handle the mounted EcVolumeShard already
holds (try_clone_file), which is what Go does: ChecksumScrub reads through
shard.ReadAt (weed/storage/erasure_coding/ec_volume_scrub.go:71), never
through a path. That also inherits open_volume_file's O_NOATIME and drops a
dead branch - the old code built {base}.ec{id}.v{gen} for a non-zero
generation, a name nothing in this tree writes. dup shares the kernel offset,
so shard reads stay positional; the .ecx gets a fresh open instead, since
check_index_file seeks.

FULL/READS is unchanged here: it already released the guard across the index
walk, and still re-takes it per needle in store_ec::scrub_snapshot_under_lock
for that needle's local shard intervals - short holds, many of them.

scrub_volume now takes the read guard PER VOLUME instead of across the whole
loop, so the heartbeat can land between volumes. Its per-volume work still
runs under the guard; Volume needs an equivalent plan to fix that properly,
left as a follow-up and noted in the code.

## A failed scrub task must not take the whole RPC down

Moving the scans into spawn_blocking changed where a panic lands. It no
longer unwinds inside the handler's own future; it comes back as a JoinError
at the .await, and all four join points sat behind a `?`. So one bad volume
out of six hundred returned Err from the entire handler: the
broken_volume_ids, broken_shard_infos and details already gathered for the
other 599 were dropped, and emit_scrub_metrics - the only writer of
SCRUB_LAST_TIME_SECONDS, SCRUB_VOLUME_FAILURES and SCRUB_SHARD_FAILURES - was
never reached, so the staleness alert kept firing while real corruption went
unreported.

And there is a reachable panic behind it. EcLocalScrubPlan::run() sized its
reassembly buffer with

    Vec::with_capacity(get_actual_size(size, version) as usize)

which for any negative size that is not the -1 tombstone skipped above is a
capacity-overflow abort. Mode 3 (LOCAL) is the default of `weed shell
ec.scrub`, and a scrub is what you point at an index you already suspect, so
an arbitrary i32 in a .ecx size field is in-scope input. The buffer is
Rust-only - Go appends to a nil slice and has no capacity hint here. Guard on
`want <= 0` and fall through with an empty buffer: locate_data returns no
intervals for a non-positive size, read stays 0, and the existing
`read != want` error reports the row exactly as Go does.

Each join point now records the failure against its own volume and continues.
A panic is evidence about the volume and counts as broken; a non-panic
JoinError is not - spawn_blocking only reports one when the runtime is going
down, the volume was never scanned, and counting it would put a false
corruption into SCRUB_VOLUME_FAILURES. total_volumes moves before the join in
modes 1, 3 and 4 (2|5 already counted there) so a failed join cannot silently
shrink it. Mode 2|5's verify_ec_shards join is the one that must not
`continue`: the needle walk above has already produced findings for that
volume.

The tombstone guard stays is_tombstone() on purpose. ScrubLocal in
ec_volume_scrub.go:228 skips only IsTombstone(), while the distributed walk
in store_ec.go:516 skips all IsDeleted() - the asymmetry is Go's, and both
Rust walks mirror their own counterpart.

## Both servers: a node-wide scrub skips a volume that vanished mid-run

Releasing the lock makes the volume set legitimately mutable during a scrub,
so a node-wide run can reach a volume that has since been unmounted. That is
not a scrub failure. A node-wide run now logs and skips it; an explicitly
requested volume id still returns NotFound. The Go server is changed the same
way, so both implementations answer the same shell command identically.
mark_broken_volumes_readonly tolerates the same teardown one step later,
instead of throwing away the whole scrub report.

## Test

test_scrub_plans_are_self_contained_and_match_direct_call drops the EcVolume
and runs both plans on another thread, asserting the results match the direct
calls. A plan that borrowed from EcVolume could do neither, so the test stops
compiling if the snapshot regresses to a borrow.

test_scrub_plans_survive_files_removed_after_snapshot unlinks every shard and
the .ecx after the plans are built, then asserts the results still equal the
direct call. Against a path-resolving version it fails with all 14 shards
reported as "No such file or directory".

test_local_scrub_plan_reports_negative_size_ecx_row rewrites a .ecx row's
size to -1000 and runs the local plan on another thread, so the join is the
assertion - that thread is the spawn_blocking whose panic used to fail the
RPC. Without the capacity guard it fails with "capacity overflow"; with it,
the row is reported.

The Go tests cover both halves of the vanished-volume rule for volumes and EC
volumes.

517 lib tests pass, plus 34 across the other targets (`cargo test`).
`go test ./weed/server -run Scrub` passes.

## Known remaining, not fixed here

`ec.scrub -volumeId=N` is still fanned out to every node, and a node that
holds no shard of N returns NotFound, so the shell command errors even when
the nodes that do hold shards scrub cleanly. That is a shell-side fan-out
question rather than a volume-server one, and both servers keep the existing
behaviour for an explicitly requested id.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DvHoW85w6SNKNBPvrqLMmK

* scrub: discard checksum block count from total_files; capture .ecx fd for FULL walk

Two review fixes:

1. CHECKSUM arm: plan.run() returns blocks scanned, not a file count.
   Go discards it (_, shardInfos, serrs = v.ChecksumScrub()) so TotalFiles
   stays a needle/file count. The Rust arm was adding it to total_files,
   inflating the count. Discard it to match Go.

2. FULL/READS (scrub_ec_volume_distributed): the needle walk reopened the
   .ecx by PATH after the store guard was released, so a concurrent teardown
   that unlinks or replaces the .ecx (heartbeat delete_expired_ec_volumes,
   volume_ec_shards_delete) could surface an intentional removal as a scrub
   error or mix index generations within one scrub. Capture a second .ecx
   descriptor under the guard (the index plan handle is consumed by its own
   structural walk, and both seek) and read through it instead -- the same
   descriptor-outlives-name invariant the checksum plan shard handles use.

* scrub: bind FULL/READS walk to one encode generation

Address Devin review: after capturing the .ecx descriptor under the guard,
scrub_snapshot_under_lock still re-resolves the volume by id per needle, so
a teardown-and-remount of the same vid between two rows would apply the
captured .ecx offsets to a replacement volume's shards -- falsely reporting
corruption.

Capture the volume's encode_ts_ns (encode-run identity) in Phase A and pass
it to scrub_snapshot_under_lock. If the mounted volume's encode_ts_ns no
longer matches, abort the walk like a mid-scan unmount instead of mixing
generations within one scrub.

* scrub: run FULL/READS index scan in the blocking pool

Address CodeRabbit review (5147767192): index_plan.run() reads the whole
.ecx synchronously, so running it on the async executor worker could block
unrelated RPC work handled on the same executor. Move it into spawn_blocking,
matching the treatment the CHECKSUM/LOCAL arms already give their plans. A
join failure (panic/cancellation) is reported as a seed error so the
per-volume findings below are not silently dropped.

* scrub: move ecx walk to blocking pool, classify join errors, guard encode_ts_ns==0

Three CodeRabbit review fixes (5148034447):

1. Move the FULL/READS needle walk (walk_index_file over the captured ecx
   descriptor) into spawn_blocking. It reads the full .ecx synchronously and
   was still running on the async executor worker, the same blocker the
   index_plan.run() fix in the previous commit addressed.

2. Preserve JoinError classification in both spawn_blocking join points in
   scrub_ec_volume_distributed. A panic is evidence about the volume and
   counts as broken; a cancellation only happens at runtime shutdown, the
   volume was never scanned, and returning it as an error would put a false
   corruption into broken_volume_ids (the FULL/READS arm marks the volume
   broken on any non-empty errs). Panics return an error; cancellations
   return clean.

3. Do not treat encode_ts_ns == 0 as a verified generation match. The .vif
   assigns 0 when it carries no encode-run identity (legacy/pre-feature
   volumes), so 0 == 0 would accept a teardown-and-remount and apply the old
   .ecx offsets to the replacement volume's shards. Only enforce the
   generation check when the captured identity is non-zero; when it is zero,
   fall back to the pre-check behavior (no generation binding) rather than
   aborting a scrub that was already running without the guard.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-08 19:12:30 -07:00
005012edcf rust volume: quick-repair redb on durable checkpoints (#11203)
* rust volume: insert redb rebuilds in needle-id order

Unlink the .rdb before create (create does not truncate). Collapse
last-write-wins, then insert live keys sorted so 4.2.0 packs leaves.

* rust volume: rebuild redb from a BTreeMap and clear leftover keys

Peak rebuild memory is one ordered map instead of HashMap + Vec +
stable-sort scratch. Unlink stays best-effort: if it fails, retain
clears the leftover table before sorted insert. Compute idx metrics
before the write so a read error does not unlink a committed .rdb.

* rust volume: drop the extra redb read transaction on put/delete

put uses insert()'s previous value. delete gets then inserts the
tombstone in the same write transaction. Truncate the .idx row on
any failed redb write after the append.

* rust volume: unpack redb blobs through packed_to_needle_value

save_to_idx, ascending_visit, and collect_entries used the same
length-check copy as get. Route them through the helper so a
wrong-length value is absent everywhere, not a panic.

* rust volume: quick-repair redb on durable checkpoints

set_quick_repair(true) on the durable checkpoint transaction so an
OOM-killed volume server opens without a full-file repair scan.

* rust volume: reopen redb from .idx on non-poisoned commit error

redb 4.2.0 can make a Durability::None commit visible before returning
Err(CommitError::Storage(..)). In that state the database refuses further
write transactions, so truncating the .idx row (the old behavior) would
leave a redb-only put or tombstone that the stored idx_size makes the
reload skip.

Distinguish CommitError::TransactionPoisoned (txn rolled back, db still
usable -- truncate the orphan .idx row as before) from other commit errors
(change may be visible, db refuses writes -- keep the .idx row, close the
database, and reopen from .idx to repair redb's internal state).

db becomes Option<Database> so reopen_from_idx can drop the old file lock
before load_from_idx opens the same path. rdb_path, version, and
cache_bytes are stored so the reopen uses the same configuration.

* rust volume: truncate .idx row when redb is closed in put

put appends the .idx entry before acquiring the write transaction.
When db_or_err() fails (db is None after a failed reopen), the ?
returned without calling truncate_idx_to_offset, so a write reported
failed remained in the authoritative .idx and was replayed on restart.

Handle db_or_err() explicitly and truncate the orphan .idx row before
returning the error, matching the existing handling for begin_write,
open_table, and insert failures.

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-07 13:50:49 -07:00
Eliah Rusin ed5b342f0c rust volume: optional redb insert_before bulk load (#11205)
* rust volume: quick-repair redb on durable checkpoints

set_quick_repair(true) on the durable checkpoint transaction so an
OOM-killed volume server opens without a full-file repair scan.

* rust volume: optional redb insert_before bulk load

Behind redb-experimental-cursor (default off). Production binary
stays on sorted insert(). CI unit tests run both feature settings.

* rust volume: exercise insert_before across leaf splits

Replace the 5-key cfg clone with a 4000-key reverse-order rebuild
so CursorMut::insert_before hits page splits. CI runs the feature
only on storage::needle_map unit tests.
2026-09-07 13:15:47 -07:00
Eliah RusinandChris Lu b6690cfbc8 rust volume: drop the extra redb read transaction on put/delete (#11204)
* rust volume: insert redb rebuilds in needle-id order

Unlink the .rdb before create (create does not truncate). Collapse
last-write-wins, then insert live keys sorted so 4.2.0 packs leaves.

* rust volume: rebuild redb from a BTreeMap and clear leftover keys

Peak rebuild memory is one ordered map instead of HashMap + Vec +
stable-sort scratch. Unlink stays best-effort: if it fails, retain
clears the leftover table before sorted insert. Compute idx metrics
before the write so a read error does not unlink a committed .rdb.

* rust volume: drop the extra redb read transaction on put/delete

put uses insert()'s previous value. delete gets then inserts the
tombstone in the same write transaction. Truncate the .idx row on
any failed redb write after the append.

* rust volume: unpack redb blobs through packed_to_needle_value

save_to_idx, ascending_visit, and collect_entries used the same
length-check copy as get. Route them through the helper so a
wrong-length value is absent everywhere, not a panic.

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-09-07 09:31:00 -07:00
Eliah Rusin 94b10c006d rust volume: insert redb rebuilds in needle-id order (#11202)
* rust volume: insert redb rebuilds in needle-id order

Unlink the .rdb before create (create does not truncate). Collapse
last-write-wins, then insert live keys sorted so 4.2.0 packs leaves.

* rust volume: rebuild redb from a BTreeMap and clear leftover keys

Peak rebuild memory is one ordered map instead of HashMap + Vec +
stable-sort scratch. Unlink stays best-effort: if it fails, retain
clears the leftover table before sorted insert. Compute idx metrics
before the write so a read error does not unlink a committed .rdb.
2026-09-07 09:29:12 -07:00
Eliah RusinandClaude Opus 5 ade4bdf9e6 rust volume: stop a tier move whose caller has gone (#11192)
Both tier-move handlers run in a detached tokio::spawn and report
progress through a closure that returns (), with the send result
discarded. Nothing observes the caller leaving, so an abandoned move
uploads or downloads the whole .dat anyway and then commits the
transition.

Go aborts both. Its progress callback returns `stream.Send`'s error,
which surfaces out of the reader in s3_upload.go:99 and the writer in
s3_download.go:84 and fails the transfer, so the volume info is never
rewritten. The Rust port dropped that by typing the callback as
FnMut(i64, f32) with no result.

Give the callback Go's signature -- FnMut(i64, f32) -> Result<(), String>
-- and abort when the caller's channel is closed. Checked on every part
rather than only where progress is reported, since the report is
rate-limited to one a second and would miss a caller that left in
between. A merely full channel is a slow reader, not a departed one, so
only TrySendError::Closed counts as cancellation.

Two consequences of aborting mid-transfer that the old code never had to
handle:

- upload_file now aborts the multipart upload when the transfer fails.
  An abandoned multipart upload does not show up in an ordinary object
  listing but still accrues storage charges until a lifecycle rule reaps
  it, and cancellation makes that a routine path rather than a rare one.
- The tier-down handler removes the partial .dat. download_file
  pre-allocates the destination to the object's full size, so an aborted
  download leaves a file of the right length and the wrong content --
  and this handler refuses to run at all when a local .dat exists, so
  leaving one wedges every retry on "already on local disk" and a
  restart would load the sparse file as the volume's data.

There is deliberately no check between a finished transfer and the
bookkeeping that follows. Once the object is in S3, or the .dat is on
disk, that bookkeeping is what makes the state consistent; stopping
there would leave an object paid for and referenced by nothing, or a
complete local .dat the volume still calls remote. Go does not gate
there either -- its callback only runs during the transfer.


Claude-Session: https://claude.ai/code/session_0122W3eqt6gmLUMxmRoZdPAb

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-06 12:18:47 -07:00
4b41329e12 rust volume: stop a VolumeCopy whose caller has gone (#11188)
* rust volume: stop a VolumeCopy whose caller has gone

VolumeCopy runs its copy in a detached tokio::spawn and reports progress
with the send error discarded, so nothing observes the client leaving.
When the caller cancels the RPC -- which weed-admin's batch balance does
routinely, starting far more copies than it finishes -- the server
streamed the whole volume from the source, wrote it to disk, and mounted
it. The destination is then left holding a volume nobody took delivery
of: its index cache is never reclaimed, and under replication=000 one
volume id ends up on two servers, both writable, which concurrent writes
can diverge.

Three checks now reach the task:

- Every chunk in copy_file_from_source, via the sender's is_closed().
  This is the one that matters in practice. The first progress report is
  128MB in, so for a smaller volume -- the ordinary balance move -- no
  send ever happens and its result says nothing; only the closed channel
  does. The sender is passed for the .idx and .vif copies too, with
  reporting gated separately, so those phases notice as well.
- The throttle sleep, which for a throttled copy runs for seconds at a
  time, now races the sender's closed() instead of being slept through.
- Immediately before mount_volume, and once at the top of the task.

Cancellation surfaces as an ordinary Err(Status::cancelled), so it lands
in the existing error branch that already removes the partial .dat/.idx/
.vif and the .note. That branch also logs now: the error otherwise went
to a channel nobody was reading, leaving the operator with the
balancer's "delete that copy, then re-run the move" and no cause.

This also clears the stranded read-only sources reported on the issue.
They are downstream of the orphan mount, not a separate defect:
LiveMoveVolume's cleanup probes the target before undoing the freeze
(volume_move.go:95, "the server can finish the copy and mount the target
even when the client loses the stream"), and when it finds a mounted
copy it cannot attribute, or cannot delete, it deliberately keeps the
source readonly rather than risk two writable replicas -- the messages
at volume_move.go:110 and :123. With nothing mounted on the target the
probe reports clean and the freeze is undone.

On Go parity: the progress send result is honoured here too, matching
`return false` in volume_grpc_copy.go. But that report is Go's only
abort signal, and measured against a 120MiB volume -- above the
throttler's activation threshold, below the 128MiB report interval -- a
Go destination mounts an abandoned copy as well. The issue's premise
that Go aborts holds only above the report interval. The Rust side now
stops in both cases; the Go behaviour is worth its own issue.

Tests: the integration test runs against both implementations and is
green on Go, red on Rust before this change. Its 192MiB fixture is sized
for two separate constraints, documented at the fixture: IoBytePerSecond
is a no-op below ~100ms of wall clock (64MiB copies in ~110ms on a tmpfs
loopback cluster), and the payload must exceed the 128MiB report
interval for the Go leg to pass at all. The two Rust unit tests cover
what the integration test cannot reach: cancellation detected with no
progress report at all, and the cleanup of the partial files plus the
.note.

Fixes #11186

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0122W3eqt6gmLUMxmRoZdPAb

* rust volume: surface VolumeCopy cancellation as Status::cancelled

copy_file_from_source returned Result<_, String>, so the per-chunk
cancellation path -- the one that matters in practice for volumes
below the 128MB report interval -- was wrapped to Status::internal at
the call sites. The spawn logging branch then classified it as a
generic failure instead of the intended "abandoned by caller",
defeating the logging change in the same PR for the case that occurs
most often.

Return Result<_, Status> from copy_file_from_source: Status::cancelled
for caller-gone, Status::internal for the existing errors. Drop the
.map_err(|e| Status::internal(e)) at the three call sites.

The unit test now asserts the code is Cancelled, not just the message
text, so the classification is locked in.

* rust volume: close cancellation gaps in VolumeCopy

Address two review findings on the same PR:

1. Roll back a mount that races a departing caller. The pre-mount
is_closed() check cannot close the window between the check and
mount_volume: if the receiver drops in that gap, the volume mounts
and the final tx.send(Ok(...)) fails, but its error was discarded
(let _ =), so the task returned Ok(()) and the error branch never
ran. The destination then held an orphaned mounted replica — the
exact defect this PR prevents.

   Fix: track a mounted flag. The final send now checks its result;
   on failure it returns Status::cancelled, and the error branch
   calls store.delete_volume (which unmounts AND removes the files)
   when mounted is true, instead of only unlinking.

2. Observe cancellation while awaiting the source stream. The
per-chunk is_closed() check only runs after stream.message().await
returns. A stalled source (slow disk, partition, GC pause) never
delivers a chunk, so a caller that has already left cannot preempt
the read: the task, the source connection, and the partial files
(including the .note) all outlive the caller indefinitely.

   Fix: race stream.message() against progress_tx.closed() in a
   tokio::select!, so a departing caller preempts a stalled source.

Adds test_volume_copy_after_mount_cancellation_rolls_back_mount to
cover the after-mount rollback path. cargo test --release green
(497 + 5 + 1 + 28).

* rust volume: keep remote data on after-mount rollback, race RPC startup

Two review findings on the after-mount rollback added in ff51f6a:

1. The rollback called delete_volume(vid, false, false), i.e.
   keep_remote_data=false. A remote-tier copy .vif points at the same
   cloud object the source replica references, so destroying the
   abandoned destination with keep_remote_data=false deletes the
   source remote data via Volume::destroy backend.delete_file_blocking.
   Use keep_remote_data=true, matching the pre-spawn delete_volume at
   the top of volume_copy.

2. client.copy_file(copy_req).await (the initial RPC establishment)
   was not raced against progress_tx.closed(). If the source stalls
   before sending response headers, the per-message select! added in
   ff51f6a is never reached, and the task, source connection, and
   preallocated files outlive a departed caller. Race the RPC
   establishment against progress_tx.closed() the same way.

cargo test --release green (497 + 5 + 1 + 28).

* rust volume: race master-configuration wait against caller cancellation

try_get_master_configuration().await was the last un-raced await in
the VolumeCopy task before copy_file_from_source. A stalled master
(or slow leader election) would hold the task and its .note past a
departing caller, since the per-chunk cancellation checks are never
reached. Race it against tx.closed() the same way the source stream
reads already are.

cargo test --release green (497 + 5 + 1 + 28).

* rust volume: fix after-mount rollback test to actually reach that path

The previous version of
test_volume_copy_after_mount_cancellation_rolls_back_mount dropped the
response immediately after volume_copy returned, so tx.is_closed() was
already true at the spawn first check and the task returned before
mount_volume. The test was green for the wrong reason: the mounted
flag, the rollback, and the keep_remote_data=true line were all
uncovered.

Use the store write lock as a seam: take it immediately after
volume_copy returns so the task runs the copy to completion with the
caller still attached, passes the pre-mount is_closed() check, then
parks entering the mount block. Drop the response (caller gone) and
release the guard: the task mounts, fails the final tx.send, and must
roll back via delete_volume.

Verified by setting the rollback guard to if false: the test fails
with "destination still holds a mounted volume". With the rollback
enabled, probe eprintlns confirmed the full path: about to mount ->
mounted = true -> final send failed -> rolling back mount -> rollback
done.

cargo test --release green (497 + 5 + 1 + 28).

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-06 10:47:18 -07:00
b82cb05d71 rust volume: checkpoint the redb index durably every 1000 writes (#11182)
* rust volume: checkpoint the redb index durably every 1000 writes

Every put and delete on a redb-backed volume committed with
Durability::None and nothing ever committed durably, on the theory that
the .idx file is the source of truth. redb, however, keeps an entry in
its transaction tracker for every non-durable commit and cannot recycle
pages that were on disk at the last durable commit until a durable one
happens. With no durable commit for the life of the process, both grew
with every write, and .rdb files could bloat toward double size after a
restart (#11179, the hash-table rehash stacks in the memleak output).

The needle map now counts non-durable commits and reports when a
checkpoint is due; the volume takes it, data first: flush the .dat, then
the map fsyncs the .idx and commits redb durably, recording in the same
transaction how much of the .idx the table reflects. A checkpoint makes
the index durable, so the bytes it points at must be down before it, or
after a power loss the index would reference past the end of the .dat
and the volume would load read-only. A failed .dat flush skips the
checkpoint; it is retried on the next write.

Volume::close() now closes the needle map instead of only syncing it,
and the redb map's close() takes the same checkpoint. Before, a clean
shutdown left the table durable (redb flushes on drop) but the recorded
.idx size stale at its load-time value, so the next load replayed every
entry written since load on top of the counters.

On load, the redb map's counters now come from the whole .idx history,
the way Go's LevelDB map rebuilds them (newest entry first, with a bloom
filter of seen keys), instead of from the table's final state. Both the
reuse and the full-rebuild path use it, so overwritten and deleted bytes
keep counting as garbage across restarts, and the incremental replay of
the .idx tail only touches the table, which makes it idempotent whether
or not the table is ahead of the recorded .idx size.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK

* rust volume: skip redundant .idx fsync on checkpoint after flush_idx

On the fsync=true write path, flush_idx() already fsyncs the .idx before
maybe_checkpoint_index() runs, so the checkpoint's own sync() fsyncs the
same file a second time for nothing. Thread an idx_already_synced flag
from the volume through maybe_checkpoint_index into checkpoint(sync_idx):
when it is true the checkpoint skips its .idx fsync and only does the
durable redb commit. The delete path and close() still sync (they have
not flushed the .idx beforehand).

Co-Authored-By: Chris Lu <chris.lu@gmail.com>

* rust volume: saturate writes_since_checkpoint to prevent u32 overflow

If checkpoints keep failing (e.g. a persistent .dat flush failure whose
error is not EIO and so does not mark the volume read-only), the counter
increments on every write with no upper bound and wraps at ~4.3 billion.
Use saturating_add so it pins at u32::MAX instead, which keeps
checkpoint_due() true and retries on every subsequent write.

Co-Authored-By: Chris Lu <chris.lu@gmail.com>

* rust volume: only update max_file_key on live entries in idx metric rebuild

metrics_from_idx called maybe_set_max_file_key on every entry including
tombstones, but the live on_put path only calls it for puts and on_delete
never does. A tombstone always has a preceding put for the same key that
already set max_file_key, so the result is the same today; restricting it
to live entries makes the parity with the live path exact and self-evident.

Co-Authored-By: Chris Lu <chris.lu@gmail.com>

* rust volume: advance idx_file_offset only after redb commit succeeds

put() and delete() appended to the .idx file and advanced idx_file_offset
before committing to redb. If the redb commit failed, the offset included
the orphan row that redb doesn't reflect. A later checkpoint would record
that offset as "the table reflects up to here," and the reload would skip
the orphan row entirely — the entry becomes permanently unindexed.

Move the idx_file_offset increment to after the successful redb commit.
The .idx file still has the orphan row (append-only), but idx_file_offset
stays behind it, so the next checkpoint records the smaller offset and
the reload replays the orphan row back into redb.

Co-Authored-By: Chris Lu <chris.lu@gmail.com>

* rust volume: skip index checkpoint on close when .dat sync fails

Volume::close() discarded the .dat sync_all() result and always
checkpointed the redb index. If the .dat sync failed, the checkpoint
made the index durable with entries that may point past the unflushed
.dat tail, and a power loss would leave the volume read-only on reload
(the max_needle_end check fires).

Check the .dat sync result: on success, checkpoint as before; on
failure, call close_without_checkpoint() — sync the .idx and drop the
writer without a durable redb commit. META_IDX_SIZE stays at the last
successful checkpoint, so the reload replays the uncheckpointed tail
(redb still flushes on drop, but without recording idx_size).

Co-Authored-By: Chris Lu <chris.lu@gmail.com>

* rust volume: schedule checkpoints on every index mutation path

maybe_checkpoint_index was only called from do_write_request and
do_delete_request. put_needle_index and write_needle_blob_and_index
also call NeedleMap::put, which increments writes_since_checkpoint,
but neither triggered the checkpoint. Through those paths the counter
could grow past the interval without ever being satisfied, leaving
non-durable redb transaction state until close().

Add maybe_checkpoint_index(false) after the successful nm.put in both
methods. The .dat flush inside maybe_checkpoint_index covers the blob
write in write_needle_blob_and_index; put_needle_index pairs with a
prior write_needle_blob, so the flush covers that too.

Co-Authored-By: Chris Lu <chris.lu@gmail.com>

* rust volume: truncate orphan .idx row on failed redb commit

Commit 947ee28 moved the idx_file_offset increment after the redb commit
so a failed commit doesn't advance the watermark. But the .idx file is
append-only: the orphan row stays in the file, and the next successful
write appends after it. That write's idx_file_offset += entry_size
advances past the orphan, so a later checkpoint records an offset that
makes the reload skip the orphan row — hiding a persisted put or
restoring a deleted needle.

On a failed redb commit, truncate the .idx file back to idx_file_offset
before returning the error. This removes the orphan row, so the next
write appends at the correct position and idx_file_offset stays a
contiguous replay watermark. Add a truncate_to method to IdxFileWriter
(set_len for std::fs::File) and a truncate_idx_to_offset helper.

Co-Authored-By: Chris Lu <chris.lu@gmail.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: chrislusf <chris@chrislusf.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-05 21:46:34 -07:00
Eliah RusinandClaude Fable 5.1 3f9b05946b rust volume: bound the redb index cache per volume by --index tier (#11180)
The Rust volume server opens one redb database per volume and built
each with redb's defaults, which give every database a 1 GiB page
cache (0.9 GiB read cache + 0.1 GiB write buffer). With hundreds of
volumes behind one disk the process-wide ceiling was volumes x 1 GiB:
memory grew in proportion to the pages traffic touched, never shrank
when traffic stopped, and hosts running many instances were OOM-killed
under bulk ingest. redb, redbMedium and redbLarge were also treated
identically, so the "memory~performance" tiers did nothing.

Size the cache per tier instead: 4, 8 and 16 MiB per volume, mirroring
the Go server's 3/6/12 MiB LevelDB block cache + write buffer. Thread
the budget through RedbNeedleMap::new/load_from_idx so every open path
(create, reuse, full rebuild) uses Database::builder().set_cache_size.

Fixes #11179


Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 10:56:42 -07:00
Eliah RusinandClaude Fable 5.1 adbee9452a rust volume: bump redb 3.1.3 -> 4.2.0 (#11181)
No source changes: the API surface the needle map uses (Database
create/open/builder, set_cache_size, set_durability, tables, iterators)
is unchanged and the on-disk format is still v3, so existing .rdb files
open as-is. The 4.0.0 breaking changes (Drop on AccessGuardMut, removal
of the Legacy type) do not touch this crate.

Relevant to the redb-backed index (#11179):
- 4.1.0: optimizes cache usage and memory usage; ~1.5x faster writes.
- 4.2.0: Durability::None commits ~2x faster; pages freed by a durable
  transaction are reused by the very next one; a crash-recovery fix for
  a crash during repair of an earlier crash.


Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 10:55:23 -07:00
Eliah RusinandChris Lu cfa8afec92 filer: guard FoundationDB 100KB value limit and pack earlier (#11161)
* filer: guard the FoundationDB value size limit, not the transaction limit

An entry's whole chunk list is one FoundationDB value, and FDB caps a value at
100,000 bytes while a transaction may reach 10MB. UpdateEntry checked the
transaction limit, so every entry between the two limits passed the guard and
was rejected by FDB itself with error 2103 (Value length exceeds limit). The
failure surfaced inside the store rather than at the guard, so the S3 layer
dropped the connection and clients saw a network fault instead of an error.

Check the value limit in UpdateEntry and KvPut instead, after gzip and before
the transaction, with an error that names the limit it hit. The removed
transaction-size constant guarded nothing else: DeleteFolderChildren batches by
entry count.

Refs #11158

* filer: fold at 500 chunks in the foundationdb build

Manifest packing is what keeps a large file's entry small, but it only ran once
a flat chunk list reached 10000 chunks. A FoundationDB value stops at 100,000
bytes and an entry's whole chunk list is one value, which at ~100 bytes per
chunk record is about 1000 chunks -- so on FDB the write always failed before
packing could help: a 3.3 GiB PutObject at the default -maxMB=4 was already
past the limit.

FoundationDB support is its own build (`go build -tags foundationdb`, shipped
as its own image), so the batch is a build-time choice and needs no negotiation
at run time. The tagged build folds at 500, every other build keeps 10000 and
is untouched.

500 is not arbitrary: a single fold level leaves (chunks/batch) manifest
pointers plus up to (batch-1) unfolded chunks in the entry, so the reachable
chunk count is highest when the two terms are near equal. For a 100,000-byte
budget that optimum is 500, which holds an entry inside the limit up to
~250,000 chunks -- ~1 TB at -maxMB=4, against ~4 GB before. Larger files need
nested packing, which no batch size substitutes for.

One binary serves every role in that image, so the filer and each client that
folds -- S3, mount, WebDAV, weed shell, filer.copy -- agree on the batch by
construction. A binary built with the tag but pointed at another store folds
earlier than that store requires, costing one manifest blob per 500 chunks and
one read to resolve it.

Fixes #11158

* filer: fold with rollback inside MaybeManifestize, not beside it

A fold that fails midway has already uploaded manifest blobs for its earlier
batches, and returns only the data chunks -- dropping the manifests it had
separated out of the caller's list. Both were wrong in ways that mattered:

  - AppendToEntry assigned that shortened list straight to entry.Chunks and
    created the entry, so an append to an already-folded file whose fold
    failed lost every previously folded chunk. weed mount had the same shape.
  - cleanupChunks logged the error as "not good, but should be ok" and then
    returned it through a named result, failing the whole CreateEntry or
    UpdateEntry, while the blobs it had written stayed behind referenced by
    nothing.

The S3 path was alone in handling this, through a private helper beside
MaybeManifestize. A second entry point next to the one everything else calls
just means the wrong one gets used, so the behaviour moves inside
MaybeManifestize: on failure it returns inputChunks as it received them, and
hands the blobs it saved to a deleteChunks callback. The filer, S3 and
filer.copy pass their existing deleters -- filer.copy already cleans up this
way after a failed upload -- and mount, WebDAV and weed shell pass nil, which
reports the blobs rather than collecting them, as before. Each caller keeps its
own error policy: the filer HTTP PUT path and filer.copy still fail the request,
the rest still continue with the flat list, which is a correct entry.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-04 23:12:27 -07:00
Eliah RusinandChris Lu c006dc563e ec: remove .ecsum sidecars on destroy / shard delete; align Go and Rust cleanup (#10307)
* fix(rust-volume): remove .ecsum sidecars on EC destroy / shard delete

Rust EcVolume::destroy removed shards and .ecx/.ecj/.vif but left bitrot
checksum sidecars (.ecsum / .ecsum.v*). On clusters that run weed-volume
(not Go weed volume), collection.delete therefore orphans every sidecar
while correctly wiping shards — observed live on 4.39 (14/14 .ecsum
survived after collection.delete on a freshly encoded EC volume).

Go Destroy already calls RemoveBitrotSidecars; this brings Rust to parity:
- hoist remove_bitrot_sidecars into ec_bitrot (shared helper)
- call it from EcVolume::destroy for dir / dir_idx / ecx_actual_dir
- call it from Store::delete_ec_shards when a disk has no remaining shards
- unit test: test_destroy_removes_bitrot_sidecar

* rust volume: gate the shard-delete sidecar sweep on a local shard removal

Only sweep a disk's .ecsum when this delete actually removed a shard file
there, matching Go's found gate: a delete that never touched a disk must not
strip a sidecar it does not own — a shared -dir.idx sibling with surviving
shards, or an ec.rebuild index-prep copy that lands .ecx/.ecsum before any
shard. The shard-presence probe now treats unexpected stat errors as
"exists" so a transient failure cannot orphan-classify live shards, and
check_all_ec_shards_deleted reuses it.

* rust volume: destroy() sidecar sweep needs only the data and idx bases

ecx_actual_dir is always one of the two, so the third branch could never
run; this is now exactly Go Destroy()'s two-base sweep.

* rust volume: call the shared sidecar removal helper directly

* rust volume: unit-test remove_bitrot_sidecars

Mirrors Go's TestRemoveBitrotSidecars: legacy and versioned sidecars are
removed, a shard file and a longer-vid sidecar survive, absent is success.

* rust volume: keep the shared idx-base sidecar while a sibling disk has shards

One -dir.idx serves every location, so emptying one disk must not sweep
<idx>/<vol>.ecsum out from under a sibling that still holds shards. Nothing
reads the idx-base sidecar today, but .ecx shows index-dir files are real;
this keeps the defensive sweep safe if a writer ever lands one there.

* ec shard delete: keep the shared idx-base sidecar while a sibling disk has shards

One -dir.idx serves every disk, so emptying one disk must not sweep
<idx>/<vol>.ecsum out from under a sibling that still holds shards of the
volume — the same gate the Rust volume server applies. A status error counts
as in-use so a transient failure never strips it early.

* rust volume: drop a shard-only disk's stale .vif with the node's last shard

Go's removeEcSharedIndexFiles also clears the data-base .vif in the
all-shards-gone pass, gated on .idx absence so a disk still hosting the
source volume keeps its live .vif; the Rust delete path left it behind.
Unexpected stat errors count as .idx-present so a transient failure never
strips a live volume's .vif.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-07-10 22:06:36 -07:00