Commit Graph
124 Commits
Author SHA1 Message Date
Chris LuandClaude Opus 5.5 194e6ede9f seaweed-volume: reject FetchAndWriteNeedle reads that miss the requested size (#12717)
The Go volume server fails the RPC when the remote read returns a
different length than req.Size; the Rust server cached whatever came
back as a full-size chunk.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:35:38 -08:00
Chris LuandClaude Opus 5.5 0c6c6868de volume: refuse a tier upload whose .dat was written during the copy (#12562)
* volume: refuse a tier upload whose .dat grew during the copy

AttachRemoteDatFile only checked that the data backend was still the one
copied, so a needle write acknowledged after the copy read the .dat was
missing from the remote object, and the upload then switched reads to it
and removed the local .dat. Compare the .dat size with the copy under the
same data-file locks; on a mismatch the handler deletes the copy and keeps
the local file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a tier upload whose .dat grew during the copy

Mirror the Go AttachRemoteDatFile check. The tier-up attach took the
write lease only inside each step, so a needle write acknowledged after
the upload read the .dat was missing from the object the volume then
switched to, and the local .dat holding it was removed. Hold one write
lease across the instance, revision and size checks, the remote-file
update, the .vif save and the backend swap, as tier-down does; a size
mismatch aborts, deleting the object and keeping the local .dat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: compare the physical .dat size before a tier attach

DiskFile pads its size past an unaligned tail, which a read-only load
keeps, while the upload copies the physical bytes. Stat the file itself
so such a volume is not refused as written during the copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:19:06 -08:00
Chris LuandClaude Opus 5.5 0ef0260844 volume: fail a vacuum step that finds the volume already compacting (#12553)
* volume: fail a vacuum step that finds a compaction already running

Compact, CompactByIndex and CommitCompact returned success when another
compaction held the volume. A second, uncoordinated vacuum (an expired
admin job still running, a shell volume.vacuum next to the worker) then
went on to commit or clean up the first run's .cpd/.cpx. Return a
"volume N is compacting" error instead, and refuse cleanup while a copy
is in flight, as the Rust volume server already did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check the compaction files before a commit closes the volume

CommitCompact closed the needle map and data file and wrote the .cpc
marker before it found .cpd/.cpx missing. A duplicate commit, such as
the second of two overlapping vacuums, then left the volume registered
with no open files: it dropped out of the heartbeat and looked lost to
volume.list and volume.fix.replication until a restart. Check for the
files first so the refused commit leaves the volume as it was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: run makeupDiff before a commit closes the volume

CommitCompact closed the needle map and data file before makeupDiff, so
a makeupDiff failure (a compaction revision another commit moved, an I/O
error) returned with them closed: the volume stayed registered but
missing from the heartbeat until a restart. makeupDiff reads the live
files through its own handles while the commit holds the write lock, so
close them only once it has succeeded, as the Rust volume server does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: log a volume the heartbeat skips because its files are closed

A volume left without its needle map or data file vanished from the
heartbeat with no log line, so the master saw a replica disappear while
its files were still on disk. Warn on each skip, as the deleted-.dat
case next to it already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: claim the compaction flag while cleanup removes the temp files

Cleanup only read the flag, so a compaction could start between the
check and the removal and lose its .cpd/.cpx. Claim it for the cleanup
instead, in Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:48:58 -08:00
Chris LuandClaude Opus 5.5 9af736da62 volume: stop serving EC reads from quarantined shards (#12450)
* storage: skip quarantined EC shards on local reads

A needle read tried the local shard first even when its disk was
quarantined after repeated IO errors, so every read paid the failing
disk's EIO latency before falling back. Treat such a shard as not local
so the read goes to peers or reconstruction, and read it only as a last
resort when no healthy copy answers. Scrub still reads it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: refuse EC shard reads from a quarantined disk

VolumeEcShardRead served peers from a shard whose disk was quarantined,
and it ignored the .ecx lookup error: a failed index read returns a
tombstone size, so the peer was told a live needle was deleted. Refuse
quarantined shards and fail the request on an index read error; a
needle missing from the index still reads, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: report quarantined EC shards deleted right away

Quarantining an EC volume only dropped it from later full EC heartbeats,
so the master kept routing reads to the faulty disk for up to 17
heartbeat intervals. The heartbeat that quarantines the volume now
carries its shards as deleted, once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: do not announce EC shards mounted onto a quarantined volume

MountEcShards and the shard load notifier reused a quarantined EcVolume
and announced the new shard, pointing the master back at the faulty disk
until the next full EC heartbeat. Skip the announcement, and fail the
mount so a caller moving a shard here keeps its source copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: count IO errors on local EC shard reads

Go's readLocalEcShardInterval feeds every local shard read into the EC
volume's IO error streak; the Rust needle read and recovery gather did
not, so a failing disk under local reads never moved toward quarantine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: skip quarantined EC shards on local reads

Mirror the Go read path: a needle read and the recovery gather treat a
shard on a quarantined disk as not local, and read it only as a last
resort once peers and reconstruction fail. Scrub still reads it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse EC shard reads from a quarantined disk

Mirror Go's VolumeEcShardRead: answer Unavailable for a shard whose disk
is quarantined, so the peer tries the next holder or reconstructs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: report quarantined EC shards deleted right away

Mirror Go: the volume heartbeat that quarantines an EC volume carries its
shards as deleted, once, instead of leaving the master to drop them at
the next full EC heartbeat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: fail EC shard mounts onto a quarantined volume

Mirror Go's MountEcShards: the heartbeat already leaves such a shard
unannounced, so fail the mount and let a caller moving the shard here
keep its source copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: keep reading EC needles when the shard lookup fails

A failed or short master lookup aborted the read before any shard was
tried, so the quarantined-disk last resort never ran once the master had
dropped those shards, and local shards went unread whenever the master
was unreachable. Read on with the cached locations, as the Rust server
already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: log a failed last-resort read of a quarantined shard

When reconstruction and the quarantined read both fail, say why the local
read failed too. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: drop a shard whose mount was refused on a quarantined disk

The refused mount left the new shard loaded on the quarantined volume,
where it was neither served nor announced, and the in-memory claim drew
later copies of that shard back to the same disk. Unload it again,
keeping a shard that was already mounted there. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: finish a mount batch before reporting a failed shard

A shard that failed to mount, now including one refused on a quarantined
disk, returned before the sidecar reload, and in Rust before the
heartbeat notify, so shards mounted earlier in the same batch kept
checksum protection off and reached the master only at the next full EC
heartbeat. Stop at the failed shard but finish the rest first. Go and
Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: keep a peer's deleted answer when its stream fails after it

A shard read recorded the holder's deleted answer, and a stream error
after it failed the read: Rust discarded the flag, and Go let the next
source or the reconstruction result overwrite it. With the last-resort
read of a quarantined shard, that could serve a needle a peer had
reported deleted. Once the deleted answer is in, end the read on a later
stream error instead of failing it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: use the platform media error in the quarantine heartbeat test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: rebuild EC intervals from quarantined shards when healthy ones fall short

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: rebuild EC intervals from quarantined shards when healthy ones fall short

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: serve a peer's EC shard read from a healthy copy on another disk

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: serve a peer's EC shard read from a healthy copy on another disk

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a quarantined EC shard read instead of serving a sibling copy

A sibling disk's copy can have its own deletion journal, and an EC blob
delete is journaled on one local copy only, so the sibling could serve a
deleted needle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: refuse a quarantined EC shard read instead of serving a sibling copy

A sibling disk's copy can have its own deletion journal, and
VolumeEcBlobDelete journals a delete on one local copy only, so the
sibling could serve a deleted needle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: check the disk an EC shard mounted on for quarantine

mount_ec_shard now returns the disk it mounted on, and the mount RPC
checks only that disk, like Go's MountEcShards. Before, a quarantined
sibling holding the same shard failed the mount while the healthy
disk's copy stayed mounted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: test an EC mount beside a quarantined sibling

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: check the quarantined-sibling mount test by its result

A server that remounts a shard on the disk already claiming it refuses
the mount there instead. The test now only requires that the result
matches whether the healthy disk ended up holding the shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: check the quarantined-sibling mount test by its result

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: expect the quarantined-sibling mount result from the disk the store picks

The test accepted a refused mount whenever the healthy disk stayed empty,
so a handler that refused before mounting passed too. Ask the store which
disk it picks first, then require success exactly when that is the healthy
disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: expect the quarantined-sibling mount result from the disk the store picks

Mirrors the Go test. add_location's cross-disk reconcile mounted shard 1 on
the healthy disk too, so the test now unmounts it there first, leaving the
quarantined sibling as the only claimant, as in Go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:38:15 -08:00
e44c16802d weed/server: verify data shards against bitrot sidecar before ec decode (#12422) (#12427)
* weed/server: verify data shards against bitrot sidecar before ec decode (#12422)

* ec.decode: verify data shards against the mounted volume's bitrot protection

The decode looked up only the generation-0 .ecsum beside the data base and
each location's data directory, so it missed a sidecar kept with the index
and ignored the generation the volume actually serves.

Take the protection the volume's per-disk runtimes resolved instead (On, else
Invalid, else Off), and test a corrupt block past the first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: check for live entries before verifying the shards

A volume whose needles are all deleted gets its shards purged on the
no-live-entries status; a failing checksum must not keep it around.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: verify data shards against the bitrot protection before a decode

Mirrors the Go volume server: the decode takes the protection the volume's
per-disk runtimes resolved and refuses a data shard that fails its .ecsum
checksum, after the no-live-entries check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: a decode refuses a data shard that fails its .ecsum checksum

Runs against both volume server implementations: the refused decode keeps
every shard and the sidecar, and the needle stays readable through EC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: reload the bitrot sidecar before verifying the shards

A rebuild's backfill writes the sidecar without reloading the mounted
runtimes, which then still report no protection; resolve it again across
the EC metadata directories, as a shard mount does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: reload the bitrot sidecar before verifying the shards

Mirrors the Go volume server: the decode job resolves the sidecar again on
every runtime of the volume before choosing its protection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: a decode finds a sidecar written after the shards were mounted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read through the EC shards after the refused decode

The source volume stayed mounted, so the final read never touched the
shards it is meant to check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: refuse when a sidecar reload reports another layout

ReloadBitrotSidecar logged and dropped the layout-mismatch error, so the
runtime stayed Off and the decode skipped the data-shard check. It now
returns that error and the decode refuses on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a decode when a sidecar reload reports another layout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: a decode refuses a late sidecar that records another layout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: refuse when an unmount left no runtime to resolve the sidecar

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a decode when an unmount left no runtime to resolve the sidecar

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 10:11:09 -08:00
Chris LuandClaude Opus 5.5 fe1cf5263f volume: bound each FetchAndWriteNeedle replica write (#12415)
* seaweed-volume: bound each FetchAndWriteNeedle replica write

A replica that accepted the connection and then went silent held the
request until TCP gave up, because the shared outgoing client has no
request timeout. Each replica write now fails after 30 s, from connect
to the response status, matching the Go volume server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: bound each FetchAndWriteNeedle replica write

The default replica path uploads through the guarded peer client, which
bounds only the dial and has no response header timeout, and the filer
calls FetchAndWriteNeedle with no deadline. A replica that accepted the
connection and went silent held the request for minutes. Each replica
write, retries included, now has 30 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: require the silent-peer test to fail at the deadline

A connection error that came back early also passed, since the test
only checked an upper bound. It now also requires the paused clock to
have reached FETCH_AND_WRITE_REPLICA_TIMEOUT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: grow the FetchAndWriteNeedle replica bound with the chunk

The reqwest timeout includes sending the body, and a remote-cache chunk
is a thousandth of the object once it passes the 1000-chunk cap, so a
fixed 30 s would fail a large chunk on a link under ~35 MB/s. The bound
is now 30 s plus 1 s per MiB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: grow the FetchAndWriteNeedle replica bound with the chunk

The context deadline includes sending the body, so a fixed 30 s would
fail a large remote-cache chunk on a slow link. The bound is now 30 s
plus 1 s per MiB, matching the Rust volume server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 03:37:49 -08:00
Chris LuandClaude Opus 5.5 fcf62db8d0 seaweed-volume: stage optional EC sidecar copies before replacing (#12403)
* seaweed-volume: stage optional EC sidecar copies before replacing

VolumeEcShardsCopy opened the destination .vif/.ecsum with File::create
and then streamed into it. A source without the file left a 0-byte .vif,
wiped an existing .ecsum (turning bitrot protection off on that holder),
and a failed stream removed the destination's existing copy.

Stream optional copies into <file>.copying and rename into place only
once the source sent bytes, as the Go volume server's writeToFile does
for ignore_source_file_not_found.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* seaweed-volume: keep the drain doc comment on drain_copy_stream_to_file

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: stage the .vif copied during missing-index recovery

fetch_ec_index_from_one_peer drained the optional .vif straight into the
local file, so a peer without one left a 0-byte .vif (losing a local EC
ratio) and a failed stream left a partial one. Stage it and rename only once
the peer sent bytes, as Go's recovery does through writeToFile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: test optional copies against a stream that fails mid-transfer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: remove a staged optional copy when the RPC is cancelled

A cancelled CopyFile handler drops its future at the stream await, so the
cleanup after it never ran and the .copying file stayed behind. Go's
writeToFile runs to completion and removes it. A guard now owns the
staging cleanup and is disarmed only once the rename commits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: test optional copies dropped mid-stream

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: cancel the staged-copy test only once the staging file exists

tokio's File::create runs on the blocking pool, so a fixed timeout could
drop the .ecsum copy before .copying existed and pass without exercising
the cleanup guard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 02:04:49 -08:00
80aad76986 rust volume: remove the redb index when a failed VolumeCopy cleans up (#12250)
* rust volume: remove the redb index when a failed VolumeCopy cleans up

A copy that mounts and then fails validation is unmounted, not deleted. With
a redb needle map the mount has already built <idx base>.rdb, and
cleanup_failed_copy removed .dat/.idx/.vif/.note but not the .rdb, so the
index database outlived the copy. A later copy of the same volume to that
server reuses it (RedbNeedleMap::try_reuse_rdb keeps an .rdb whose recorded
idx size is not larger than the new .idx and replays only the tail), which is
wrong once the source has been compacted in between.

Move the partial-file removal into CopyDestination::remove_partial_files and
include the .rdb, next to the .idx where Volume::file_name puts it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* rust volume: clear a stale index before a copy writes, and report cleanup failures

A crashed copy, or one whose partial-file removal failed, leaves a stale
.rdb/.sdx next to the .idx; the next copy's mount would reuse it via
try_reuse_rdb's tail replay. remove_partial_files now also covers .sdx
and returns the first non-NotFound error: callers that can still abort
(the pre-copy clear) do so, while the post-failure cleanup warns so a
surviving index is visible instead of silently poisoning the next copy.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: restore the pre-copy leftover clear

A leftover index check-in accidentally gated remove_partial_files behind
a constant false, so a stale .rdb/.sdx at the destination survived the
copy that was meant to clear it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(volume): expect 409 when a read-only upload is refused

x-seaweed-write-outcome: refused-before-append. The HTTP integration
test still expected 500, so Rust unit tests failed on Linux and Windows.
Deletes stay 500.

* rust volume: keep a shared .vif, and clear a stale index beside the .dat

remove_partial_files deleted <data>.vif even when a sealed .ecx still
shares it, undoing the keep in Volume::destroy. It also cleared the
index only under the index base, so a leftover .idx beside the .dat
still made load trust that directory's .rdb.

Skip .vif while a non-empty .ecx is present, and clear both bases when
the index directory differs from the data directory.

* rust volume: do not delete .dat files in the index directory

The index directory can be another location's data directory. Clearing
.dat and .note there removes that replica. Those files come off the
data base only; .idx, .rdb, and .sdx are still cleared on both.

* rust volume: serialize copies of a volume on one server

A second VolumeCopy of the same volume could pass the existing-volume
snapshot before the first copy mounts, then clear or overwrite the
files the first copy still owns. Hold a per-volume claim for the whole
copy, like ec_decodes_in_flight does for decodes, and fail a duplicate
fast so the caller can retry. Mirrors the Go volumeCopiesInFlight
guard.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: keep files when the mounted-copy delete is refused

When a copy mounted and then failed, deleting through the store can be
refused (e.g. the volume is compacting), leaving it registered. The
cleanup then unlinked its files anyway, stranding a live replica. Keep
the files while the volume is still in the store. Mirrors the Go copy
cleanup.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: init volume_copies_in_flight in the http test state

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: refuse a mount while that volume is being copied

volume_copy claims the vid, snapshots the store, then waits on the
source. A VolumeMount in that window can load the replica, and the
copy's pre-write cleanup then unlinks its files. Check the claim under
the store write lock and return Unavailable.

* rust volume: refuse configure while that volume is being copied

volume_configure unmounts and remounts. A copy that already snapshotted
the volume as absent can then unlink the files that remount loaded.
Check the copy claim under the store write lock and return Unavailable.

* rust volume: preserve files a copy displaces so a failed copy restores them

* rust volume: keep an interrupted copy's .note with the files it marks

displace_existing_files skipped the .note, so a crashed copy's leftovers
were saved without the marker the load scan rejects them by, and
remove_partial_files then deleted it. A retry that failed restored the
partial set marker-free: after a restart the server could load an
unfinished copy as a normal replica and serve wrong data.

Displace the .note like every other file at the basename so the restore
puts back the exact state the copy found.

* rust volume: restore displaced files over a surviving partial

restore_displaced_files dropped the backup whenever the basename was
still occupied. When the partial cleanup could not remove every file the
copy landed, an existing path just meant the removal failed — deleting
the saved file destroyed the old replica and let the surviving partial
pair with restored siblings of another generation.

Clear the leftover first so the displaced file goes back, and keep the
backup when it cannot: the operator can recover from copying/ rather
than lose the replica outright.

* rust volume: back up an EC-shared .vif the copy truncates in place

A .vif a sealed .ecx shares was left out of the displaced set so the EC
volume keeps it — but the incoming copy opens the same path with
truncate, so a failed copy left the shared .vif overwritten by the
rejected copy's settings with no way back. Go keeps the original via a
hard link because its staged file swaps in by rename; truncating the
shared inode makes a link useless here.

Copy the content into copying/replaced_ like the other backups instead:
the file stays in place for the EC volume during the copy, and the
restore puts the original settings back on failure.

* rust volume: serialize reload loads with the copy claim

A SIGHUP reload scans every volume basename and sweeps .note-marked files
as interrupted copies — including one a live VolumeCopy is still writing.
Mirror Go's loadGuard: the copy-claim set is shared with each disk
location, the scan skips a claimed vid outright and claims any other
through its load, and reconcile_compact_states takes the same skip before
rolling a leftover commit over the copy's files.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: hold the load claim through open_volumes

The scan released each claim when its iteration ended, before the queued
volume's index read ran in open_volumes. A VolumeCopy taking the vid in
that gap could displace or remove files out from under Volume::new — the
register the guard exists to prevent. Queued claims now live until the
open pass returns; a second collection of an already-claimed vid merges
into the entry instead of re-claiming.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: refuse allocation while that volume is being copied

AllocateVolume registered a volume while a copy of it held the
destination: create_volume loads the index before registration, and the
copy then clears or rewrites files the live replica already opened,
leaving it mounted on a stale index — the window VolumeMount and
VolumeConfigure already refuse.

Claim the volume id like the Go AllocateVolume does, so registration
either wins the claim before the copy starts or sees it and refuses
with Unavailable.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: refuse staged adoption while that volume is being copied

The from_staged branch of volume_ec_shards_to_volume renames .copying
files into place and mounts, all while a copy of the same vid could be
rewriting the destination — the same registration window VolumeMount
refuses. Hold the copy claim for the whole adoption, as Go's
adoptStagedVolume does.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: drop a partial .note when its write aborts the copy

A write_note failure after displace_existing_files restores the saved replica but leaves the empty or partial marker it just made; the next load scan then deletes the restored replica as an unfinished copy. Remove the marker before restoring, as the Go side's error defer does.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
2026-10-30 11:21:38 -07:00
43a312e309 rust volume: keep a copied replica on deleted-count drift, as Go does (#12251)
* rust volume: keep a copied replica on deleted-count drift, as Go does

check_copy_counts rejected a VolumeCopy whenever the target's deleted count
differed from the source's. On a redb needle map that count is an estimate:
metrics_from_idx rebuilds it at load through a bloom filter (0.1% false
positives per entry, each counted as an overwrite), and the source holds the
estimate from its own last load plus exact runtime updates. A source written
since its load and a fresh load of the same files disagree by a few, so good
copies were refused with "target deleted count [N+1] is not same as origin
deleted count [N]". An index of unique keys 1..=128 already reports one
deletion.

Go fixed the same rejection in #11701 (issue #11678) by warning instead of
failing. Do the same for the deleted count. The file count stays strict: it
is exact on both sides (every non-tombstone .idx row is one file), and the
.dat/.idx size checks still gate integrity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* test(volume): expect 409 when a read-only upload is refused

x-seaweed-write-outcome: refused-before-append. The HTTP integration
test still expected 500, so Rust unit tests failed on Linux and Windows.
Deletes stay 500.

* rust volume: keep the strict count check when only deletes drift

Deletes append tombstone rows without moving the file count, so a copy whose source only deleted mid-flight still has an exactly comparable file count. Gate the strict check on file-count stability alone; the deleted-count drift is already a warning inside check_copy_counts.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-30 11:20:33 -07:00
Chris LuandDevin cba409cae7 backup: carry index-only tombstones through incremental copies (#12231)
* volume server: add idx_content to VolumeIncrementalCopyResponse

A remote-tier delete is an index-only tombstone row with no .dat record,
so it cannot reach an incremental copy through the .dat stream. Carrying
the .idx tail verbatim lets the backup reproduce those rows at their
exact positions.

* backup: carry index-only tombstones through incremental copies

A remote-tier delete leaves an offset-0 tombstone in .idx with no .dat
record. The copy only streamed .dat bytes and regenerated .idx from
them, so the tombstone never reached the backup: the deleted needle
stayed live and every later incremental diverged and rebuilt again.

backupCursor and ResumeByLastEntry now accept an index-only tombstone
as a boundary, matched by key and size since there is no record to read
a cookie from. VolumeIncrementalCopy appends the .idx tail after the
.dat tail, and IncrementalBackup writes it at the index boundary,
truncates, and loads the new rows — so tombstones land at the exact
positions the replica has them.

* seaweed-volume: mirror index-only tombstone handling in incremental copy

resume_by_last_entry accepts an offset-0 tombstone boundary matched on
key and size, skips its missing cookie check, and resumes .dat streaming
at the next entry that has a record — index-only rows ride the new
idx_content stream instead of reporting divergence. The server captures
the .idx bound before the .dat one (a row is published only after its
bytes) and streams the tail after the data.

* backup: keep incremental copy bounds and streams consistent

Review fixes:

* Bound capture moved after the resume reads and reads .dat/.idx sizes
  together (Go FileStat, Rust file_stat under one lock), so every row
  resume saw is inside the index bound and each row's record is always
  inside the data bound.
* A compaction commit changes CompactionRevision: the server now rejects
  the copy when the revision moved between resume, file opens, or the
  end of the stream, so a backup never mixes revisions. The .idx is
  opened up front to pin the inode.
* An index tail that ends before the captured bound is an error, not a
  silent success.
* IncrementalBackup rolls the .idx tail and .dat bytes back to the
  pre-copy boundary when the stream fails, so a partial row cannot wedge
  backupCursor and unapplied rows cannot pass for current ones.
* The index tail is replayed through a small indexTailReceiver interface,
  covering the leveldb needle maps too, not just the in-memory map.

* backup: bound the resume scan to the captured index size

The resume and the stream bounds are now captured in one order:
revision, then the .dat/.idx pair, then the scan. ResumeByLastEntry and
resume_by_last_entry take the index bound and scan no further, so a row
appended while the copy is being set up can never enter the index
stream without its .dat bytes.

The index-tail replay stays on the in-memory map only: the leveldb
map's DoOffsetLoading writes to the .cpldb compaction database, not the
live one, so it cannot serve a live tail replay; other maps keep
regenerating the index from .dat as before.

* seaweed-volume: guard incremental copy total against a located offset past .dat

A corrupt index can place the resume offset beyond the captured .dat
size; has_data already refuses the reader, but total still evaluates
dat_size - s and underflows.

* backup: bound the incremental .dat tail by index-covered records

A record staged but not yet indexed can sit inside the captured .dat
bound while its row stays outside the index bound — the fsync gap
between staging and publishing is not covered by one stat. Streaming
those bytes leaves them unindexed on the backup, and the next run
appends the record again. Bound the .dat tail by the last record the
streamed rows cover on both producers, and cover the bound walk with
tests.

* backup: cap the .dat stream at the captured bound and roll back replay failures

sendFileContent could read past stopOffset on its last chunk, so a
needle appended after FileStat arrived without its index row and the
next copy appended at the wrong position. Each read is now limited to
the remaining captured bytes.

A failed truncateIndex or DoOffsetLoading after a successful stream
left the index tail on disk unapplied, which a later backupCursor could
mistake for applied rows; both now roll the tail back to the pre-copy
boundary like the stream-error path.

* backup: bound the incremental .dat tail by the maximum covered offset

A key-sorted .idx ends on the highest-key row, which need not be the
record physically last in .dat, so the last-row bound could cut off
live records. Take the end of the highest-offset row inside the bound
instead, and cover the case with a sorted-idx test in Go and Rust.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* backup: restore .dat and the needle map when index replay fails

A failed DoOffsetLoading left the received .dat bytes in place and could
leave partial mappings for rows the index rollback just removed, so the
next copy appended the same records again or read stale entries. Truncate
.dat back to the copy boundary and restore the touched mappings and
metrics from a pre-replay snapshot.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* backup: cover no index rows with the superblock bound

When no row inside the index bound names a record — an empty index or a
bound of index-only tombstones — the covered end is the superblock, not
the captured .dat size, so staged bytes are never streamed without a
covering row.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* seaweed-volume: apply the covered bound even when the index streams no rows

The bound was gated on idx_end > idx_start, so an empty captured index
skipped it entirely and a staged .dat tail still shipped. Gate on the
stream-index predicate instead, matching Go's streamIndex check.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* backup: propagate index coverage read errors

* backup: rebuild the needle map from the truncated index on replay failure

* seaweed-volume: propagate index coverage read errors

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 22:37:31 -07:00
Chris Lu 349576b63e backup: resume incremental copies by index position instead of appendAtNs (#12228)
* volume server: add last-entry cursor fields to VolumeIncrementalCopy

* backup: resume incremental copies by index position instead of appendAtNs

appendAtNs is stamped by whichever replica received the write, so the
same needle carries a different timestamp on every replica. A backup
that resumes from a different replica than last time can append the
last needle again or skip needles written inside the clock gap.

Send the local copy's index entry count and last entry identity (key,
size, cookie) with VolumeIncrementalCopy. The replica verifies the same
record sits at that position and streams from the next index entry, so
the resume no longer depends on any replica's clock. When the boundary
record does not match, or the local copy is ahead, the replica returns
FailedPrecondition and the backup rebuilds the local copy from scratch.

Old servers ignore the new fields and keep the timestamp behavior.

* rust volume server: mirror positional resume in VolumeIncrementalCopy

* volume server: add entries_hash prefix checksum to VolumeIncrementalCopy

* backup: verify the whole index prefix, not just the boundary entry

* rust volume server: verify the index prefix hash in resume_by_last_entry

* backup: keep the source's compaction revision on rebuilt copies

* backup: diverge on a zero-offset boundary entry

A remote-tier tombstone lives in the .idx with no .dat record, so
readNeedleAt(0) parses the superblock and its error bypassed the
ErrIncrementalCopyDiverged mapping; the backup got a generic failure
instead of FailedPrecondition and never rebuilt.

* rust volume server: mirror zero-offset boundary divergence

* rust volume server: gate the empty-dat early return on entry_count

With a positional resume the Go handler still reaches ResumeByLastEntry,
so an empty replica reports divergence; the Rust early return answered a
clean empty stream and weed backup kept a stale copy.

* backup: hash the index entry's size field, not trailing offset bytes

In a 5BytesOffset build the offset occupies bytes 8-12 and the size
starts at byte 13, so the hash folded in one byte of the offset.
Matching records at different offsets then hashed differently and a
valid backup was reported diverged and rebuilt.

* backup: read the index in whole-entry chunks for the prefix hash

When entries are 17 bytes a 1 MiB read buffer does not end on an entry
boundary: the partial tail entry was skipped by the loop but still
counted as consumed, so every later chunk was hashed at a misaligned
position.

* rust volume server: hash the index entry's size field after the offset

Mirrors the Go hashIndexEntries change: with the 5bytes feature the
offset occupies bytes 8-12, so starting the size slice at byte 12
folded one offset byte into the hash and matching records at different
offsets compared diverged.

* rust volume server: read the index in whole-entry chunks for the prefix hash

Mirrors the Go change. With 17-byte entries a 1 MiB buffer does not
end on an entry boundary, so slicing the partial tail entry panicked
once a backup held 61681 entries.

* rust volume server: default the new incremental copy request fields in tests
2026-10-29 18:56:25 -07:00
Chris LuandDevin 2c510cc30e ec.rebuild: remount shards the rebuild replaced, keep regenerated copies (#12193)
* volume: remount ec shards the rebuild replaced under a mounted handle

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec.rebuild: keep copied shards the rebuild regenerated in place

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: reopen regenerated ec shards in place instead of remounting

UnmountEcShards and MountEcShards send deltas through separate buffered
channels, so a remount can reach the master as add-then-delete and drop
the repaired shard from lookups until the next full EC heartbeat.
ReopenEcShard swaps the mounted shard's descriptor under the volume lock
instead: registration, disk type, and heartbeat state never flip, and
readers holding the old shard degrade to a clean error the parity path
already recovers from.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: reopen rebuilt ec shards in place, mirroring Go

volume_ec_shards_rebuild recreates a shard file under a mounted handle
whose descriptor still serves the detached inode (a mounted shard whose
file was deleted under it). DiskLocation::reopen_ec_shard swaps the
mounted descriptor for a fresh open, matching Go's ReopenEcShard — the
registration and heartbeat view never flip.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: fail ec rebuild when a regenerated shard cannot be reopened

* rust volume: fail ec shards rebuild when a rebuilt shard cannot be reopened

* volume: repoint mounted ec shards after rebuild across disks

A mounted shard whose file was deleted under the mount is "missing" to
the rebuild: the regenerated file lands on the rebuild location while
the handle keeps serving the detached inode on another disk. Reopening
the mounted disk then failed the whole RPC, so the caller never saw the
rebuilt ids and its deferred cleanup deleted the regenerated copies.

Repoint every mounted handle of the rebuilt volume instead of only this
run's outputs: reopen in place when the rebuilt file is on the mounted
disk, otherwise drop the stale handle and remount the shard from
whichever local disk holds the file via a delta-free load split out of
MountEcShards — a same-server move, so the master's registration never
flips. A shard that can be neither reopened nor remounted is left
unmounted rather than serving replaced bytes, and the repoint step no
longer fails the RPC, so RebuiltShardIds always reaches the caller.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: repoint mounted ec shards after rebuild across disks

Mirror the Go repoint loop: iterate every mounted shard of the rebuilt
volume, reopen in place when the rebuilt file is on the mounted disk,
and otherwise drop the stale handle and remount the shard from
whichever local disk holds it. The repoint no longer fails the RPC, so
rebuilt_shard_ids still reaches the caller and its copy cleanup keeps
the regenerated shards.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: still rebuild ecx and repoint mounts when no shard is missing

* master: keep a node in an ec shard's lookup while a disk still holds it

A rebuild that relocates a mounted shard between a node's disks reports
the move as a delete on the source disk plus an add on the destination.
Registering first is a no-op because shard lookups key on the node, then
the unregister dropped the node outright — and steady-state heartbeats
never restore it, so readers and later rebuilds lose the repaired copy.
Skip the delete while another disk on the node still holds the shard.

UnregisterDataNode now clears the node's own ec records before dropping
its registrations, so the retained-shard check cannot skip the teardown,
and each unregister gets a ShardsInfo copy since DeleteEcShard drains
the shared one.

* rust volume: fold the journal on a mounted index instead of rewriting it

The rebuild handler regenerated .ecx through File::create even while the
volume was mounted, truncating the index under readers that search the
open descriptor with its old size — a live lookup could hit EOF mid-scan.
Go folds the deletion journal in place on a mounted volume; do the same
through rebuild_ecx_from_journal and reserve the full regen for the
unmounted path, where nothing holds the file open to interrupt.

* volume, rust volume: report rebuilt shards on late failures, fix rust corrupt discovery

Two review findings the repoint rework left open:

- A RebuildEcxFile failure still returns a plain error after shards were
  regenerated, so the caller discards the response holding the rebuilt
  ids and the deferred cleanup deletes copies the rebuild regenerated
  in place. The Go handler now packs the ids into the gRPC status
  details and the shell pulls them back out on error; the Rust handler
  carries them in grpc-status-details-bin the same way.
- The Rust rebuild still counted a sidecar-corrupt shard as present, so
  it fed bad bytes into reconstruction and never regenerated the shard.
  Discovery now loads the generation-0 .ecsum the way Go does, verifies
  every present shard, and reclassifies mismatches (and zero-size
  residue) as rebuild targets written to a .rebuilding temp file,
  verified, and atomically renamed over the original only after every
  output passes the sidecar check. Honors unsafe_ignore_sidecar, added
  to VolumeEcShardsRebuildRequest to match the Go proto.

* volume: repoint sibling journals sharing a folded .ecj after rebuild

RebuildEcxFile unlinks the deletion journal and the mounted path only
reopens its own handle — a sibling mount journaling into the same file
keeps appending to the detached inode, and those deletes never reach a
path-based reader again. Reopen every mounted volume sharing the index
base before leaving the rebuild.

* rust volume: repoint sibling journals sharing a folded .ecj after rebuild

rebuild_ecx_from_journal replaces the journal and reopens only its own
handle; mounts on sibling disks sharing that .ecj keep the detached
handle, so their next appends are lost at restart. Adopt the folded
journal on every mounted runtime sharing the path before releasing the
store lock, clearing their in-memory set — every journaled id is now an
.ecx tombstone.

* ec: swap the deletion journal handle only after the new file opens

A failed reopen left ecjFile nil, so every later delete returned
"ec volume closed". Open first, then close and swap — mirroring the
Rust adopt_folded_journal ordering.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec.rebuild: fail the rebuild when a sibling journal cannot be repointed

A warning left the sibling mount journaling to a detached inode while
the RPC reported success. Propagate the error like the Rust handler's
adopt_folded_journal map_err.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec: keep deletes loud when the journal reopen cannot open the replacement

Reverts the swap-on-success ordering: a detached handle kept on failure
lets deletes append and sync to the removed inode, then vanish at
restart. Closing first also frees the descriptor before the open.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: drop the folded journal handle before reopening it

adopt_folded_journal and the post-fold recreate now clear ecj_file
before opening the replacement, so a failed reopen leaves deletes
erroring instead of appending to the detached inode. Mirrors Go's
ReopenDeletionJournal.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec: drop the cached journal size when its reopen fails

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 00:58:30 -07:00
5c92a54a12 remote: reuse one guarded remote storage client per configuration (#12189)
* remote: reuse one guarded remote storage client per configuration

BuildGuardedRemoteStorageClient (the filer's lazy remote stat and list,
remote deletes and renames, the volume server's cold reads, the S3
gateway's remote reads) built a new client, with its own http.Transport,
on every call. Each transport kept its connection idle for
IdleConnTimeout (60s) and was never reused or closed, so a steady stream
of remote lookups held one open connection per request. An endpoint
that caps connections per source then refuses every new connection.

- remote_storage: GetRemoteStorage's per-name cache becomes a bounded
  LRU (WEED_REMOTE_CLIENT_CACHE_MAX, default 64) keyed by kind, name, a
  hash of the whole configuration and a credentials version, built
  outside the lock once per key (singleflight) from a copy of the
  configuration. A client leaving the cache has its idle connections
  closed. GetRemoteStorageWith exposes the cache to other builders.
- BuildGuardedRemoteStorageClient caches the guarded client per
  configuration. The endpoint is checked against the deny-list when a
  client is built; every new connection still dials through the guarded
  dialer. A gcs credentials file, and for an S3-compatible remote using
  the default credential chain the shared credentials file and AWS_*
  variables, are part of the key by content.
- The shared transport keeps 32 idle connections per host, caps
  connections per host (WEED_REMOTE_MAX_CONNS_PER_HOST, default 64) and
  has dial, TLS handshake and response header deadlines, so a hung
  remote frees its slots.
- The filer drops the clients of removed remotes (and all but the newest
  client per name) on every /etc/remote reload, also when /etc/remote
  does not exist; reloads are serialized.

* remote: drop clients whose build spans a reload or whose remote is gone

* server: hash the shared AWS config file into the guarded credential version

* server: bound a stalled body on the shared remote transport

* remote: waive path-injection on the configured credentials file

* remote: keep caching remotes the local filer does not name

* remote: drop the inline codeql waiver the analyzer ignores

* remote: version the plain gcs client's credentials file

* remote: hash the environment's inline gcs credentials

GOOGLE_APPLICATION_CREDENTIALS can hold the JSON itself, which the gcs
client accepts; it is not in the configuration hash, so a changed value
kept the old client.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote: version the well-known gcs credentials file

With neither the configuration nor GOOGLE_APPLICATION_CREDENTIALS set,
the SDK reads application_default_credentials.json; rotating it left the
cache key unchanged and the old client behind.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote: share the gcs credentials version with the guarded client

The guarded gcs client resolves the same environment and well-known
credentials as the plain one; keying it only on a configured path left
those rotations invisible to the cache.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote: version s3 keys resolved from the environment

A remote whose access and secret keys both arrive through its type's
environment fallback (aliyun, baidu, contabo, filebase, storj, tencent)
never entered the default-chain branch of guardedCredentialsVersion, so
an in-process environment change kept serving the stale cached client.
The version now hashes the resolved keys for every S3-compatible remote.

* volume: reuse one remote storage client per configuration (rust)

fetch_and_write_needle built a fresh aws_sdk_s3::Client per request,
so every cold read kept its connector's idle connections without ever
reusing them — the same leak the Go side fixes. Cache clients keyed by
the configuration's protobuf encoding in an LRU of
WEED_REMOTE_CLIENT_CACHE_MAX entries (default 64), mirroring
weed/remote_storage/client_cache.go. An evicted entry drops the client's
last Arc and with it the connector's idle connections. The per-request
endpoint validation is unchanged: Rust has no dial-time guard yet, so
the up-front DNS check stays on every call.

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 00:57:37 -07:00
Chris LuandDevin 7a47c63d54 volume: replicate FetchAndWriteNeedle writes over the configured TLS scheme (#12128)
* volume: replicate FetchAndWriteNeedle writes with the configured TLS scheme

The guarded upload path hardcoded an http:// URL and used a bare
*http.Client, so a peer fronted by https rejected the replica write and
read-through of a remote-evicted object failed whenever the volume had
replicas. Normalize the upload URL through the shared global client and
let the guarded transport borrow its https.client TLS configuration, the
same settings the regular replication path resolves.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: keep the global HTTPS downgrade guard on replica uploads

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: cover TLS replicate uploads and the redirect downgrade guard

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: pin the downgrade guard against a dialable plaintext target

The previous redirect pointed at 169.254.169.254, which the guarded dialer
already refuses, so the test passed even with CheckRedirect removed.
Redirect to a live plaintext server on a dialable peer address and assert
it never receives the resent body.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume-rs: keep the HTTPS downgrade redirect guard on outbound writes

Mirrors the Go peer upload client: the shared outgoing HTTP client now stops redirects that would downgrade an https request to plaintext, so replicate uploads to TLS peers cannot leak data or auth headers via a 307 to http.

* volume-rs: fail rejected redirects instead of returning the 30x response

Go's rejectHttpsDowngrade uses ErrUseLastResponse, which surfaces to the
caller as a non-2xx status it rejects; attempt.stop() returned the redirect
response as Ok, so a refused downgrade could masquerade as a completed write
to any caller that does not check the status.

* volume-rs: fail replica writes on a non-2xx status

fetch_and_write_needle discarded the replica response without checking its
status, so a peer rejection — or a redirect the client refused to follow —
was reported back as a successful copy while no replica was written. Go's
doUploadData fails on any non-success status.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-27 09:09:04 -07:00
0d721fccee fix(volume): keep preallocated growth above -volume.minFreeSpace (#12067)
* fix(volume): keep preallocated growth above -volume.minFreeSpace

When free space fell below -volume.minFreeSpace, every volume went
read-only and the master kept growing new volumes. With preallocation on,
each new volume reserved its full size, driving the disk further below the
floor (15 Oct incident: 66 empty volumes reserved about 1980 GiB).

Two causes, both in weed/storage/store.go:

- MaybeAdjustVolumeMax sized a disk's volume slots from raw free space, so
  space inside the floor was counted as available. It now counts only free
  space above the floor.
- addVolume trusted the once-a-minute low-disk flag. A preallocated volume
  is now refused when the disk is low now, or when its reservation would
  not fit above the floor. Volumes without preallocation, including merges
  that copy existing data, are not affected.

addVolume runs under a store-level lock, so two concurrent creates cannot
both pass the room check against the same free space.

* volume: mirror headroom guard in the rust store

find_free_location only trusted the once-a-probe is_disk_space_low flag,
so a burst of preallocated creates could still run the disk under
-volume.minFreeSpace between probes, and maybe_adjust_volume_max sized
slots from raw free space. Adds MinFreeSpace::is_low/available_space to
match Go and applies both checks the same way. store.write() already
serializes add_volume, so no extra lock is needed.

* volume: read rust headroom tests from a fake disk

The floor arithmetic used a live free-space reading, so unrelated disk
activity could flip a test between pass and fail. Store.disk_stats_of is
now the same seam Go uses (diskStatusOf): the growth-loop and floor tests
run against a controlled disk, mirroring the Go suite one for one.

* volume: hold back create metadata from the headroom check

An exact-fit preallocation left free space exactly at the floor, so the
.vif and .idx writes that follow a create could dip under it and get the
location flagged read-only at the next probe. Reserve room for .vif on
the data filesystem, and for .idx growth only when dir_idx shares it.

* volume: mirror the metadata allowance in the rust store

Same rule as the Go side: charge .vif against the checked filesystem and
.idx only when dir_idx shares it, and grow the exact-fit test's fake
disk by the per-create cost.

* volume: skip the reservation check where create cannot preallocate

* volume: mirror the platform preallocation gate in the rust store

* volume.copy: count the preallocation reservation in the destination space check

The copy path picked a location with room for the copied .dat, then reserved
the master's full volume size limit on it. A volume smaller than the limit
could land on a disk whose free space above -volume.minFreeSpace fit the
copy but not the reservation, driving the disk under the floor the same way
the growth loop did.

Fetch the preallocation setting before choosing the location and require
room for the reservation itself, only where CreateVolumeFile actually
reserves blocks.

* seaweed-volume: store min-free-space-adjusted available space like Go

Go's DiskLocation.AvailableSpace holds free space above the
-volume.minFreeSpace floor; the Rust probe cached raw free instead, so the
VolumeCopy location check could accept a disk already inside the floor.
Store the same adjusted figure and cover it with a test.

* seaweed-volume: reserve the copied .dat like Go's CreateVolumeFile

VolumeCopy called set_len for the reservation, which only grows a sparse
file — no blocks are held, so a disk with room for the copied bytes but
not the full reservation only fails mid-copy. preallocate_file fallocates
KEEP_SIZE on Linux like Go, reserving the blocks up front; off Linux it
is a no-op, matching PREALLOCATE_SUPPORTED.

* seaweed-volume: charge a copy's destination pick for the .dat reservation

Go's VolumeCopy now sizes its free-location check against the master's
volume size limit when the platform preallocates, not just the copied
bytes. The Rust pick only measured the copied bytes, so a destination
could accept a copy whose reservation then dips under the floor. The
config fetch moves ahead of the pick (same order as Go) and feeds both
the space check and the reservation itself.

* volume.copy: charge the picked location's free space under volumeCreateLock

The copy path picked a destination off the once-a-minute AvailableSpace
probe and released the decision long before the .dat reservation landed.
Two copies could pass the same cached headroom and together drive the disk
under -volume.minFreeSpace.

ReserveFreeLocation serializes the pick under volumeCreateLock, verifies
the disk still has live room for the reservation, and charges it against
the location's cached free space until the next probe refreshes it.

* seaweed-volume: mirror the copy pick's live-check and headroom charge

Go's VolumeCopy now picks through Store.ReserveFreeLocation: live disk
stats under the store write lock, then the reservation charged against the
location's cached free space. The Rust plan_copy_destination gains the
same store.reserve_free_location so two copies cannot spend the same
headroom before the first .dat reservation lands.

* volume.copy: saturate the free-space charge against a live-space drop

The two statfs reads bracketing a pick can race a large external write; a
plain subtraction would underflow and leave the location looking roomy
until the next probe.

* volume.copy: release the headroom reservation when a copy does not land

ReserveFreeLocation charges the picked location's cached free space, but
a failed copy never returned it, so the headroom stayed booked until the
next disk-space probe and blocked retries. ReleaseFreeLocation refunds
the charge; the copy's cleanup defer now registers with the reservation
so every failure path (existing-replica delete, .note write, stream send)
releases it, including the post-transfer mount/validate failures covered
by the outer defer.

* seaweed-volume: mirror the copy headroom release on failed copies

release_free_location refunds reserve_free_location's charge. The copy
returns it when the plan's existing-replica delete fails, when the .note
write fails, and in cleanup_failed_copy, so a failed copy stops holding
the headroom until the next disk-space probe.

* volume.copy: keep pending copy headroom out of probe readings

ReleaseFreeLocation refunded the reservation into AvailableSpace even when
a CheckDiskSpace probe had already rebased the cache on live stats — the
probe's reading still shows the in-flight copy's bytes as free, so the
refund double-counted headroom and two copies could spend the same space,
driving the disk under -volume.minFreeSpace.

Track the outstanding charge in DiskLocation.ReservedSpace instead: probes
subtract it so they cannot re-spend it, a failed copy removes only its own
charge, and a landed copy commits it — the charge and the new usage cancel
out. hasRoomForNewVolume respects pending headroom too, so a preallocating
create cannot overrun an in-flight copy.

* seaweed-volume: mirror pending copy headroom across disk probes

reserved_space tracks the in-flight copy's charge separately from the
cached free space: check_disk_space subtracts it, release removes only the
failed copy's charge, a landed copy commits it, and has_room_for_new_volume
respects it — matching the Go side.

* volume.copy: settle a copy's headroom as its bytes land on disk

The pending charge held the copy's full reservation until commit, but
written and fallocated bytes already show up in statfs, so the live-space
check counted them twice. A slow copy could then block other copies and
volume growth despite free space above -volume.minFreeSpace.

Settle the charge instead: commit the bytes CreateVolumeFile actually
reserved (a failed fallocate leaves nothing committed), then settle each
file's write progress as it lands. The failure paths now release only the
unsettled remainder; allocatedSize is exported for the .dat's st_blocks.

* seaweed-volume: mirror copy headroom settling as bytes land

reserved_space keeps only the promised bytes statfs cannot see yet:
preallocate_dat settles whatever blocks the filesystem actually reserved
(0 when fallocate cannot), each copied file's progress settles the rest as
it lands, and a failed copy releases only the unsettled remainder —
matching the Go side.

* volume.copy: release a failed copy's headroom charge only once

Both defers fired on the same in-closure error: the inner released the
unsettled charge and the outer released it again, driving ReservedSpace
negative (an atomic.Uint64 underflow) and inflating AvailableSpace —
the location would then fail every later headroom check, or worse,
pass them with fabricated space. Zero the charge once it is released;
the Rust mirror already releases via swap(0) for the same reason.

* volume.copy: write into the preallocated .dat instead of staging over it

The .dat stream staged into .copying and renamed over the file
CreateVolumeFile had just fallocated, so the copy's transient footprint
was preallocate+dat — more than the reservation the pick just checked —
and the reservation itself was discarded at rename. Writing in place
reuses the fallocated extent, keeps the peak at the reserved maximum,
and leaves the copied volume preallocated like a created one.

* seaweed-volume: mirror the in-place write into the preallocated .dat

The .dat copy opened the just-fallocated file with truncate, freeing its
blocks at open and spending the headroom again as the stream re-grew it.
Append mode preserves them, matching the Go side.

---------

Co-authored-by: chrislusf <chrislusf@gmail.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-26 08:03:25 -07:00
23d8c62844 volume: route lookups to the leader master only (#11888)
* volume: route lookups to the leader master only (#11887)

After a leader change, the volume server kept sending volume and EC
shard lookups to masters that were not the leader:

* currentMaster was set as soon as a heartbeat stream opened, before
  the peer said whether it was leader, and kept a follower through the
  redirect and its 3s sleep. A follower answers LookupVolume from its
  own vid cache, which right after an election can be empty, so reads
  went 404 rather than failing over.
* EC lookups used Store.MasterAddress, a second, unlocked copy that the
  heartbeat loop set before each dial and cleared on errors, so EC and
  plain reads could disagree on the master or have none.
* the receive goroutine called stream.Send itself when the volume
  options changed, racing the send loop, and exited silently on
  failure, after which leader redirects were no longer read.

Publish currentMaster only once a master is known to be leader: the
peer answered a new stream without naming a leader, which the leader
always does, or a follower named the leader. Compare redirects against
the connected peer. Replace Store.MasterAddress with MasterFn, backed
by getCurrentMaster. Have the receiver ask the send loop for the extra
heartbeat instead of sending it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: ask the masters for the leader after losing the heartbeat

When the heartbeat stream to the leader failed, currentMaster kept the
lost master until a new stream answered, so lookups on a cache miss
kept dialing it through the retry delay even with a new leader already
elected.

On a regular heartbeat error, ask the configured masters for the raft
leader and publish it, within the retry delay rather than on top of it.
Without an answer the last known leader stays: clearing it would only
trade a dead address for an empty one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: stop the heartbeat on a failed assertion

The leader test closed stopChan only once every assertion had passed,
so a failure left the stream open and the fake master's GracefulStop
waited for the whole test run's timeout.

Start the heartbeat through a helper whose cleanup stops it, safe to
call twice, and waits for it to exit. It runs before the cleanups
registered earlier, so the stream is closed before the store and the
fake master go away. Both heartbeat tests use it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: clear the leader when no master names one

When the heartbeat lost its master and no configured master named a
leader, refreshLeaderFromPeers kept the lost one. That master is dead
or no longer leader, and a follower answers LookupVolume from its own
cache instead of refusing, so lookups could return locations no leader
vouched for.

Clear currentMaster instead. Lookups then fail fast ("missing address")
until a leader is known again, rather than reading a follower's cache
or waiting on a dial to a dead master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep volumeSizeLimit 64-bit aligned on 32-bit builds

MasterFn (a func, 4 bytes on 386) replaced MasterAddress (a string, 8
bytes) ahead of volumeSizeLimit and moved it to a 4-byte-aligned
offset, where atomic.LoadUint64 panics with an unaligned 64-bit
operation. The atomic.Uint64 type carries the alignment itself.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: publish the answering leader before a duplicate-uuid retry

Read the leader field first on each heartbeat reply. A follower reply
carries nothing else, so it no longer runs through the option updates;
and a leader that rejects the registration over duplicate uuids still
answers lookups, so it stays the lookup target through the retry wait
instead of leaving the previous master in place.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: publish the master only once it is known leader (rust)

* volume: ask the masters for the leader after losing the heartbeat (rust)

* volume: read the leader field first on each heartbeat reply (rust)

* volume: drop the seed fallback from ec shard lookups (rust)

cached_lookup_ec_shard_locations fell back to the static first seed when
no leader was published, so a mid-election EC lookup could land on a
follower and read its stale shard map. Error out like Go's
masterAddress() returning "" instead, via a current_master() getter
matching getCurrentMaster.

* volume: send volume lookups to the confirmed leader, not the seed (rust)

Every HTTP lookup path — proxy/redirect targets, replicated writes and
deletes, batch chunk deletes, remote chunk reads — dialed the static
first seed, so after a failover they kept asking a follower whose vid
map can be stale. Go routes all of these through GetMaster; switch the
lookup calls and their leader-known gates to current_master().

* volume: mark volumes readonly on the leader, retrying the named one (rust)

notify_master_volume_readonly fell back to the seed when no leader was
published, which the peer answers with "Not current leader". It now
uses only the confirmed leader, skips the call in masterless mode, and
on NotLeader asks the seed masters who leads and retries there — the
same path Go's notifyMasterVolumeReadonly takes. The peer leader query
is shared with refresh_leader_from_peers as lookup_leader_from_peers.

* volume: ask the confirmed leader for copy prealloc options (rust)

VolumeCopy's preallocate query went to the static seed; Go issues it
through GetMaster. Route it through current_master() too — when no
leader is known the lookup fails fast and the copy proceeds without
preallocating, as before.

* volume: keep the announced master port forms, not just http (rust)

publish_current_master stripped to http form, so a leader configured
with an explicit gRPC port (host:9333.19999) was redialed on the
default offset. Store the announced host:port[.grpcPort] form like Go's
currentMaster; callers pick to_http_address / to_grpc_address /
parse_grpc_address per use, and Ping admission compares on the http
form.

* volume: fail replicated writes and deletes when no leader is known (rust)

Gating replication on a known leader meant a write could be
acknowledged on fewer copies than required, and a delete could leave
the data readable on the other replicas. Go fails the request when the
replica-location lookup cannot run; drop the gate so the empty-leader
lookup errors the same way.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 13:36:05 -07:00
Chris LuandDevin 1cfd9bbb4c volume: run the durable sync outside the reader-visible lock (#11863)
* volume: run the durable sync outside the reader-visible lock

Reads of a volume waited on that volume's fsync because the write lock
was held across Sync()/sync_all. A new per-volume lock (Go
dataFileSyncLock, Rust data_file_sync_lock) now serializes every .dat
mutation and lifecycle op across append+publish+sync+rollback, while the
fsync itself runs with dataFileAccessLock/the io write guard dropped —
so readers proceed during a sync, and a failed sync's rollback still
cannot clobber another write or a compaction swap.

Same change in Go (volume_write.go syncWrite/processBatch, SyncToDisk)
and Rust (write_needles_grouped/write_synced_run).

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: publish after sync; fix Close/CommitCompact lock order

Address review on #11863:

- Close took dataFileSyncLock before waiting on isCompactionInProgress
  while CommitCompact claims the flag then takes the lock — a deadlock.
  Close now claims the flag before taking the locks (flag -> sync -> access).

- Go now publishes needle-map entries only after the .dat sync, matching
  Rust's ordering: appendWriteRequest/appendDeleteRequest stage the bytes,
  publishWriteRequest/publishDeleteRequest index them after Sync. A failed
  sync rolls back bytes no reader ever resolved — the streamed-read
  offset-invalidation window is gone. processBatch splits runs at repeated
  needle ids (Rust parity) so dedup/cookie checks stay sequential; the old
  snapshot/restore rollback machinery is removed.

- Rust commit_compact (store-lock caller) takes data_file_sync_lock via
  try_lock -> WouldBlock before the write lease; commit_compact_held and
  the tier-down _held paths document/require the caller-held sync lock,
  and the two vol.instance().write_lock() grpc sites take it first. The
  swap can no longer land between a durable write's append and publish.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: restore write watermarks when an inline fsync fails

The syncWrite rollback truncated the .dat but left lastAppendAtNs,
lastWriteAppendAtNs, lastWriteNeedleKey, lastWriteDeleted and
lastModifiedTsSeconds describing the rejected write; recovery and
compaction could read a watermark for a record that no longer exists.
Snapshot and restore them on failure, as processBatchRun already does.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 12:58:02 -07:00
Chris Lu 6936bbdc3e rust volume: score EC shard placement with the volume's own ratio (#11862)
The auto-select branches of ReceiveFile (is_ec_volume) and
VolumeEcShardsCopy passed the build-default DATA_SHARDS_COUNT into
find_ec_shard_target_location, so its free-slot maths scored every
volume as 10+4 even when the volume's .vif declares a custom ratio
(e.g. 6+2). A wider-ratio volume's shards could be steered onto a disk
too full to hold them all, and a narrower one would be refused slots
it actually has.

Add Store::lookup_ec_data_shards(collection, vid), which answers from
the mounted EC volume first and then from the .vif on whichever disk
holds it (via read_ec_shard_config_across_dirs, so split -dir/-dir.idx
and rebuild-staging layouts are covered), falling back to the default
when no metadata exists yet.

Mirrors the enterprise fix (seaweed-mono c22fa8a15).
2026-10-20 07:40:43 -07:00
Chris Lu 51a4c092c8 rust volume: release the store write lock during needle I/O (#11823)
* volume: give Volume interior locks mirroring Go's

Volume's mutable state moves behind interior locks matching Go's
per-volume lock domains, so the store can hand out shared handles
instead of requiring exclusive &mut access:

- io: RwLock<VolumeIo> owns the .dat/remote handles, needle map,
  append cursor, compaction bookkeeping and write watermarks —
  Go's dataFileAccessLock-adjacent mutable set; helpers take the
  guard as &mut VolumeIo so nothing re-locks.
- super_block and volume_info each get a RwLock
  (superBlockAccessLock, volumeInfoRWLock); dir_idx gets its own
  so path building never contends with I/O.
- no_write packs Go's noWriteLock pair into AtomicU8;
  last_modified_ts_seconds and compaction_byte_per_second become
  atomics.

The data-file write lease (dataFileAccessLock) now covers every
mutating entry that previously relied on borrow exclusivity:
writes, deletes, blob writes, set_writable's flag/index swap,
remote-file updates, local-backend swaps, index relocation,
replica-placement rewrite, put_needle_index, close and destroy.
set_read_only_persist leases only its can_delete idx-attach branch
because mark_io_unavailable reaches it under a held lease.
commit_compact takes try_write_lock — callers hold the store lock,
so blocking would stall every volume behind one slow reader;
WouldBlock surfaces as COMMIT_WAITS_FOR_READS for the caller to
retry.

Lock order is fixed: lease -> io -> volume_info/super_block/dir_idx.

* storage: mount volumes as shared Arc<Volume> handles

DiskLocation's map owns Arc<Volume> so the store can hand a caller a
handle that stays valid across store-guard releases, instead of
borrowing &mut Volume under the store write lock.

- volumes: HashMap<VolumeId, Arc<Volume>>; find_volume returns a
  cloned Arc.
- find_volume_mut and volumes_mut are gone — every Volume method is
  &self now, so shared handles cover what exclusive borrows did.
- writable_volume_mut -> writable_volume(&self) -> Result<Arc<Volume>>
  keeps the same admission checks (exists, disk-space-low).
- write_volume_needle/write_volume_needles/delete_volume_needle are
  &self; lifecycle methods that mutate the map or mounted state stay
  &mut self.

* server: run needle I/O with the store guard released

The store write lock previously wrapped needle writes and deletes
end-to-end: process_batch held it across every grouped fsync, and the
HTTP/gRPC fallbacks, tail receiver and fetch-and-write paths did the
same. One volume's slow I/O then stalled admission and metadata on
every other volume.

- write_queue resolves each group's handle via writable_volume under
  a brief store read, then runs write_needles_grouped after the guard
  drops.
- HTTP write/DELETE fallbacks, the gRPC batch-delete EC recheck +
  delete, tail receiver and fetch_and_write local writes, blob writes
  and the quarantine/sync/set_writable/remote-file paths all run
  under store.read() now that every volume method they reach is
  &self.
- Lifecycle calls that mutate the volume map or mounted state
  (mount/unmount/delete, EC shard ops, compaction commits,
  load_new_volumes, heartbeat volume actions) keep store.write().

* test: cross-volume admission proceeds during another volume's write

Regression coverage for the store-lock fix:

- write_queue's test parks a volume-1 write inside its .dat append via
  a new needle_write_hook seam (mirroring needle_read_hook), then
  proves the store still hands out a lifecycle writer and a volume-2
  handle — while volume 1's data-file lease still excludes same-volume
  writers.
- tests/store_lock_repro.rs exercises the public API: concurrent
  read-guarded writes on two volumes, and a stale Arc<Volume> handle
  failing closed after unmount instead of writing into torn-down
  files.

* test: drop store write guards that shared handles made read-only

Test setup sites held store.write() purely to reach find_volume_mut;
with Arc<Volume> and &self methods the guard is read-only (clippy
readonly_write_lock). The volume lease still serializes the writes
inside.

* volume: drop a windows-only read_exact_at that shadows the io helper

cargo's windows build failed E0255: the free fn copied with the
interior-lock conversion collided with the unconditional
crate::storage::io::read_exact_at import, which already carries the
platform branches internally.

* server: release the store read guard before needle I/O

The read guard still ran through write_needle_blob_and_index, the tail
receiver write, the fetch-and-write local write, and the HTTP write and
DELETE fallbacks, so a slow append blocked mount/unmount writers (and,
on writer-preferring locks, readers on other volumes). Resolve the
Arc<Volume> under a short guard and write through the handle after it
drops — the same scoping process_batch already used.

* volume: serialize the read-only mark and its .vif save under the lease

set_read_only_persist only leased the idx-attach branch, so concurrent
mark-readonly and mark-writable calls could interleave flag updates and
.vif writes — including two save_vif calls truncating and writing the
file at once. The lease now spans the whole flag change and save, with
a _held variant for mark_io_unavailable, which reaches this from write
paths already holding the lease (Go: markVolumeReadonly runs under
dataFileAccessLock).

* volume,server: span the tier-down transition under one write lease

update_remote_files cleared the remote-only write restriction and
released its lease before open_local_dat_backend attached the local
file, so a write could slip into the gap — pass check_writable, then
fail appending to a missing dat_file. The handler now holds the lease
across the reference trim, .vif save, and backend swap (and the rollback
re-insert), via _held variants; the store read guard is scoped to the
handle lookup. The swap still lands after the .vif trim — the ordering
that keeps a crash from pointing the volume at a deleted remote object.

* server: run a batch's volume groups on separate scoped threads

A single batch can hold writes for several volumes; running the groups
sequentially made every later volume wait behind a stalled append on an
earlier one — the same head-of-line blocking the store guard release
fixed, one level up. Each group now gets a scoped thread, preserving
order within a group while independent volumes progress in parallel.

* storage: close a volume handle a mount slot replaces

With shared handles, inserting a new Volume over an existing map entry
left the old handle alive in callers — two Volume instances could then
write the same files with separate in-memory indexes. set_volume and
create_volume now close() the replaced handle so its writers fail
closed, matching what unload_volume already does.

* test: key the write hook off a reserved needle id

The hook registry is process-global and keyed by needle id, and other
parallel tests write NeedleId(1) — an unrelated write could enter the
barriers and leave the test's real writer parked forever.

* server: wait out readers for vacuum commit without the store guard

commit_compact try-locked the write lease because callers held the store
write lock, returning COMMIT_WAITS_FOR_READS while a download was in
flight — and the master reads that as a failed vacuum, discarding
completed compaction work. The commit RPC now resolves the handle under
a brief store read and blocks on the volume's lease, matching Go's
CommitCompact under dataFileAccessLock: a slow reader stalls only this
volume's vacuum. The try-lock path stays for callers that still hold the
store lock.

* server: scope the remaining store read guards to volume lookup

Same class as the write-path fix: the batch-delete cookie check and
tombstone, the query read, volume_needle_status, and the local chunk
read all kept the read guard through the needle operation, so slow I/O
there could still stall store writers. Each now resolves the Arc<Volume>
under a short guard; the EC rechecks keep their guards.

* seaweed-volume: fail closed on detached volume handles

A handle resolved under a brief store read can outlive its mount: an
unmount+remount between lookup and lease leaves the Arc pointing at a
closed Volume while a new instance owns the files. Mark the handle in
close() and reject it under the write lease in commit_compact_held,
delete_needle, check_writable, set_read_only_persist and
update_remote_files_held; the tier-down handler also checks the flag
before deleting a remote object its replacement still reads.

* seaweed-volume: drop the store guard before mark-readonly waits on the lease

mark_volume_readonly held a store read across set_read_only_persist, which waits out the volume's leased readers; a slow download would stall every lifecycle operation behind the guard. Resolve the handle under a short read, then mark off the guard; a detached handle now maps to not_found.

* seaweed-volume: retry batch deletes that raced an EC mount

The store guard now drops before delete_needle, so ec.encode can mount shards between the pre-delete recheck and the tombstone; the encode then removes the .dat and the needle resurrects from pre-tombstone shards. Re-check has_ec_volume after each successful delete and return the retriable 503 so the filer requeues onto the EC path.

* seaweed-volume: one write-queue worker per volume

A stalled batch kept later batches for other volumes queued behind it: the single worker awaited process_batch before receiving again. Go's volume_write.go keeps one batching worker per volume, so give each volume its own FIFO worker spawned lazily, bounded by a global concurrent grouped-I/O semaphore and a global queued-request bound; workers exit only when their channel closes, and submit refuses unknown volume ids before spawning.

* seaweed-volume: move the queued-write permit inside the request

submit held the global queued-slot permit across the send and the response wait, so a canceled submit freed the slot while the worker still held the needle; repeated cancels could overflow the bound. The request now owns the permit until the worker drops it.

* seaweed-volume: retire write workers whose volume left the store

worker_for kept a sender per volume forever, so servers that mount and retire volumes accumulated idle workers. A worker now polls while idle and retires once its volume leaves the store: close the channel so racing senders fail fast, drain accepted writes, then release the map slot — a replacement worker can never start while this one still holds queued writes, so same-volume FIFO survives retirement.
2026-10-19 03:00:26 -07:00
Chris Lu ab39e3a51b Rust volume: stamp ExpireAtSec into the live volume_info on .vif persist (#11813)
The worker fix reads the source's TTL destroy time through
ReadVolumeFileStatus, which returns the in-memory volume_info. Go's
SaveVolumeInfo stamps volumeInfo.ExpireAtSec before writing; Rust's
save_vif wrote it only to the serialized file, so a Rust source kept
reporting the stale creation-time value and the EC .vif would carry a
wrong expiry. Stamp the live volume_info first, mirroring Go and
save_volume_info.
2026-10-18 04:29:48 -07:00
Chris Lu 887b1aac76 ec: carry the source volume's TTL expiry into the worker-written .vif (#11777)
* ec: carry the source volume's TTL expiry into the worker-written .vif

ReadVolumeFileStatus now reports the volume's ExpireAtSec (last write +
TTL), and the admin worker stamps it into the .vif it distributes with
the shards. Previously every worker-encoded .vif had expireAtSec 0, so
EC volumes of TTL collections never expired.

* ec: report expiry from the Rust volume server and cover the .vif carry

The seaweed-volume drop-in server omitted expire_at_sec from its
ReadVolumeFileStatus response, so worker-encoded volumes sourced from it
lost the TTL. Fall back to the nested VolumeInfo expiry for servers that
predate the top-level field, and pin the nonzero/zero carry with a test.
2026-10-17 12:34:55 -07:00
Peter Dodd 42ff535511 vacuum: bound concurrent commits per volume server (#11658)
CommitCompact holds the volume's write lock for the whole commit, and
reads of that volume wait on it. Volumes compacted together tend to
finish together, so their commits land at once and their fsyncs
contend: each commit's lock hold grows with how many run alongside it,
even when the disk is far from its throughput limit.

Add -compactionMaxParallelCommits (volume.compactionMaxParallelCommits
for server/mini). A commit waits for a slot before CommitCompactVolume,
so a queued commit holds no volume lock and reads keep flowing. The
wait counts against the caller's deadline and returns the context
error if it expires; a slot granted to an already-expired context is
handed back, so a request the caller gave up on never starts a commit,
and maintenance mode is checked again once the slot is held.
The slot is released whether the commit succeeds or fails. Compaction
itself is not gated, so -maxParallelVacuumPerServer still sets reclaim
throughput. 0, the default, keeps today's behaviour.

The limit sits in the volume server so it covers every commit caller:
the master's vacuum sweep, the vacuum worker task and volume.tier.compact.

The Rust volume server mirrors the flag and behaviour: queued commits
wait on a semaphore keyed to the grpc-timeout deadline, hold no store
lock, recheck maintenance mode once a permit is held, and release it
on every exit path.
2026-10-16 23:46:36 -07:00
acb44a68a9 fix(volume): skip unnecessary needle data reads on delete (#11711)
* fix(volume): do not read a needle's data to delete it

A delete read the needle's whole payload from disk only to discard it.
The delete handler and BatchDelete need the needle's cookie and its
chunk-manifest flag, which live in the header and trailer, but fetched
them with a full needle read, the same call a GET uses.

readNeedle now reads the header and trailer only when MustMetaOnly is
set, whatever the needle's size, and the delete paths ask for that. The
Rust volume server honours must_meta_only the same way for its delete
paths. A compressed needle or a chunk manifest still gets the full read,
as before, because their callers need the bytes.

HEAD is unchanged. It used to set MustMetaOnly, which nothing read; it
now asks for AttemptMetaOnly alone, so it keeps reading a needle up to
PagedReadLimit whole. On a spinning disk that one read is cheaper than a
header read followed by a trailer read.

Version 1 needles are always read in full. They have no data size field
and no metadata after the data, which ReadNeedleMeta relies on, so a
metadata-only read of one failed with "invalid needle meta size". That
already happened for version 1 needles above PagedReadLimit read with
AttemptMetaOnly.

A delete therefore no longer verifies the data CRC of the needle it
removes. The cookie check is unchanged.

* fix(volume): read transformed images fully on HEAD too

A HEAD carrying resize or crop params reports the transformed length,
which a metadata read cannot produce. Go and Rust both answered such
HEADs from the source header, returning the stored size instead of the
cropped or resized one.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 18:08:19 +08:00
07da302da0 volume server: ec.decode verifies, cleans up and compacts like Go, off the runtime (#11547)
* volume server: ec.decode reads the .ecx from the index dir it was copied to

VolumeEcShardsCopy writes the .ecx/.ecj into the receiver's -dir.idx, so
with a split data/index dir the decode target has no .ecx beside its
shards. VolumeEcShardsToVolume sized the .dat from the right .ecx but
built the .idx from the data dir, failing with NotFound after the .dat
was already published. It now reads .ecx/.ecj from where the EC volume
opened them and writes the .idx beside the .dat, where Go leaves it.

The live-entry check and the .dat size also ignored deletions recorded
only in the .ecj, which Go folds into the .ecx (RebuildEcxFile) first:
a fully deleted volume was decoded instead of reported as having no live
entries, and deleted tail needles were copied into the .dat. Both now
treat journaled ids as deleted, without rewriting the sealed .ecx.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode keeps the decoded volume writable and reads every .ecj

The rebuilt .idx copied a journaled tail needle's .ecx row verbatim after
the .dat was cut short before it, so the mount saw a row past EOF and
marked the decoded volume read-only. Rows of deleted needles the .dat no
longer holds are now dropped, and each journaled needle still in the .dat
gets one tombstone instead of one per journal entry.

VolumeEcShardsCopy appends journals collected from other holders into
the idx dir, but the decode read only the .ecj beside the .ecx, which
sits in the data dir when this server generated the shards. It now
reads both, once, in bounded chunks via the loader EcVolume uses.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: test ec.decode drops a sealed .ecx tail tombstone

Covers the other half of the rule added in the previous commit: a tail
needle tombstoned in the .ecx itself (Go's RebuildEcxFile) is cut from
the .dat, and its row must not reach the rebuilt .idx either.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode runs its file I/O off the async runtime

VolumeEcShardsToVolume released the store lock before decoding, but read
the .ecx/.ecj, rebuilt the .dat and wrote the .idx inside the async
handler, parking a runtime worker for the length of a volume-sized copy.
The decode now runs in spawn_blocking on inputs snapshotted under the
store lock.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode checks the rebuilt .dat is complete

Go stats the decoded .dat before writing the .idx (VerifyDecodedDatFile)
and fails the decode when it is shorter than the extent the EC index
references, since the caller deletes the shards once the call returns.
The Rust handler returned success without that check. The rebuild
already fails on a short shard read, so this guards the published file
itself.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode drops the decoded volume's bitrot sidecars

Go removes <base>.ecsum and <base>.ecsum.v<N> beside the .dat and beside
the .ecx once the .idx is written, so a stale checksum sidecar cannot
pass for the protection of a later re-encode. The Rust handler left them
in place. Removal is best effort, as in Go.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode compacts the decoded volume

Go ends VolumeEcShardsToVolume with an offline CompactVolumeFiles, so the
decoded volume holds only live needles. The Rust decode left every needle
deleted through the .ecj in the .dat, tombstoned in the .idx, until a
later vacuum reclaimed it.

Store::compact_volume_files loads the unmounted volume, checks free space
the way the vacuum does (the estimate now lives in one helper), and runs
the vacuum's compact-by-index and commit. As in Go a failed compaction is
logged and the decode still succeeds, so the uncompacted .idx rules stay:
the tests that pin them now make the compaction fail.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode keeps deletes journaled while the .dat is written

The decode read the .ecj journals once, before rebuilding the .dat, so a
delete that reached the EC volume during the rebuild was left out of the
new .idx and the needle came back live. Each journal's read length is now
kept, and the bytes appended since are read just before the .idx is
written, after waiting out any journal append in flight (appends hold
the store write lock), so every delete acknowledged by then is in the
.idx. A delete after that point is still lost, as in Go.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Guard overlapping ec decode requests; serialize journal catch-up

volume_ec_shards_to_volume runs its decode in spawn_blocking, so a
dropped request leaves the job running and a retry would race it on the
temporary and final volume files. Claim the vid in a per-server
in-flight set until the blocking job finishes, and return Unavailable
to an overlapping request. The Go handler has the same exposure and
gets the same guard.

Journal appends hold the store write lock through their
sync-or-truncate, so holding a read lock across the catch-up read
guarantees every record it sees is committed: a rolled-back delete can
no longer leave a tombstone in the decoded index.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Reconcile the swap when offline compaction commit fails

A CommitCompact that fails after the .cpc marker may have renamed .dat
but not .idx. cleanup_compact refuses while the marker exists, so the
mismatched pair survived until a restart reconciled it — and the decode
caller treats the failure as non-fatal. Run reconcileCompactState on
commit failure so a decided swap rolls forward and orphan temps are
removed before the volume can mount.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Release the decode claim on panic

* volume: add ec_decodes_in_flight to the integration-test state literal

* volume server: hold the decode tail's lock through compaction

The catch_up read released before the rebuilt .idx was written and the
volume compacted, so a delete synced to .ecj in that window was durably
journaled yet absent from the published index — resurrecting the needle.
Rust now holds the store read lock from catch_up through compact, and Go
mirrors it by holding the volume's journal lock from the journal-
consuming index write through CompactVolumeFiles.

* volume server: serialize ec decode's tail per volume, not per store

Review follow-ups on the decode path:

- Rust: holding the store read lock from journal catch-up through the
  offline compaction stalled every writer on unrelated volumes for the
  whole rewrite. The new ec_decode_tail set marks the vid only while its
  .idx is published and .cpd/.cpx swapped; the two local .ecj append paths
  (VolumeEcBlobDelete, the distributed delete's local journal) wait on a
  Notify for that span — Go's per-volume ecjFileAccessLock semantics
  without the global stall. VolumeMount and the staged-adopt path are also
  held off while a decode claim is in flight so neither can race the swap.

- Rust: the initial journal read ran unlocked, so bytes a rolled-back
  append later truncated could be folded in as phantom tombstones. The
  first pass stays unlocked (a slow journal must not stall the store) and
  a rescan under the quiescing read lock re-reads only committed content;
  catch_up now rebuilds the id set when a regular journal shrank.

- Go: the decode resolved the compaction DiskLocation through
  FindEcVolume while holding the journal lock, inverting DestroyEcVolume's
  map->journal order into a deadlock. The lookup now happens first, and
  DestroyEcVolume/deleteEcVolumeById/DiskLocation.Close destroy outside
  the map lock.

- Go: RebuildEcxFile unlinks .ecj while the volume's ecjFile handle stays
  open, so later deletes could commit to a detached inode. Both call sites
  now fold under the journal lock and ReopenDeletionJournal repoints the
  handle at the live path, working on the volume's resolved .ecx dir
  (EcIndexBaseFileName) rather than the configured index dir.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: fence EC remounts behind the destroy tombstone

DestroyEcVolume, deleteEcVolumeById, and the collection-delete sweep now
remove the EcVolume from ecVolumes before destroying it off-lock, so a
concurrent remount could re-open shard files that the in-flight destroy
then unlinks — registering a detached fd.

Each destroy records a per-vid tombstone channel in a new
ecVolumesDestroying map before dropping the map entry and closes it when
Destroy returns. The tombstone intentionally survives as the vid's
destroy generation: loadEcShardWithIdxDir compares it before and after
opening the shard, so a destroy that both started and finished inside the
open window is still detected. A mismatch drops the just-opened shard
(releasing its fd and mount gauge) and retries after the destroy
completes; a successful mount clears the stale tombstone.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: rescan the .ecj under the store lock only after a rollback

The decode's second journal pass ran a full rescan under the store read
lock on every decode, stalling unrelated writers for the length of the
scan. Bump a process-wide epoch whenever a failed append truncates its
uncommitted tail; an unchanged epoch between the unlocked read and the
quiesced pass proves every id folded in was committed, so catch_up()
suffices. catch_up() also treats a journal that was read but has since
disappeared as shrunk to zero, so its earlier ids cannot linger.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: check the decode tail under the store write lock on delete

A blob delete waited for the publishing tail before taking the store
write lock, so a decode that claimed the tail while the delete was
parked behind the decoder's read lock could still see the journal append
land after the rebuilt .idx — an acknowledged delete the mount would
miss. Test tail membership under the write lock instead, retrying after
the wait; journal_delete_local reports WouldBlock for the same recheck
on the distributed path.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: claim the vid for mount and staged adoption, per volume

VolumeMount and the staged .copying adoption held the
ec_decodes_in_flight set lock through slow file renames and mounts,
stalling every unrelated volume's decode, mount, and adoption. Take the
per-volume claim instead — the same exclusion against a racing decode
for this vid, released when the call returns.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: fail the decode when a compaction commit marker survives

CompactVolumeFiles' caller logged a compaction error and went on to
delete the EC shards. When the commit marker (.cpc) is still on disk the
.dat/.idx swap was decided but could not be reconciled, so the mounted
pair may be mismatched — report the failure instead so the shards are
kept and the caller can retry.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: gate the parked-delete test on the held write lock

The releaser thread and the spawned delete raced for the store write
lock; on a slow runner the delete could acquire it first and commit
before the tail was ever claimed, failing !delete.is_finished() on the
Windows unit-test job. Spawn the delete only after the thread reports
the lock held.

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-03 14:55:15 +08:00
0ff7794c54 volume: compact an oversized .ecj at mount, safely (Rust + Go) (#11555)
* volume: compact an oversized .ecj at mount, safely (Rust + Go)

Restore the mount-time compaction dropped from #11408, Rust + Go parity.
A journal already bloated by repeated shard copies is folded down to the
id set it encodes.

- Trigger after load when file_records > max(threshold, 4x distinct),
  with a 1 MiB floor so small journals are never rewritten. The set is
  written to .ecj.compact.tmp + fsync, the handle dropped, renamed,
  the directory fsynced and the append handle reopened. A failure before
  the rename keeps the original journal and handle; a failure after it
  fails the mount.
- Go never compacts after a failed journal load; the set would be
  partial and the rewrite would drop the unread records.
- A per-path registry (ecj_registry.rs / ecj_registry.go) counts EcVolume
  holders and out-of-band writers of each .ecj. Compaction runs only
  when this volume is the sole holder and no copy is writing; holders
  and writers wait while one runs. This covers shared -dir.idx journals
  and cross-disk reconcile, where another EcVolume may hold the same
  journal.
- VolumeEcShardsCopy and EC index recovery register as writers around
  their .ecj append and partial-file cleanup.
- Under the reservation, re-check that the file on disk is still the
  inode and size that was loaded.
- Publish errors are classified where they happen; a failed rename plus
  a failed restore reports both errors.
- Compaction runs after the .vif / bitrot checks, so a refused mount
  leaves the journal untouched.
- The tmp is opened like other volume files, removed at mount if a crash
  left it, and listed in every EC index cleanup path.

Failure paths are tested through the real mount via injectable fs steps
(open_with / newEcVolumeWith), plus sibling holders, active copies,
changed-after-load, stale tmp cleanup, refused mounts and the Go
load-error guard.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: fail the mount when the compacted .ecj's directory cannot be synced

The Rust mount synced the journal's directory after renaming the compacted
file over it through the crate's best-effort fsync_dir, which returns Ok
when the directory cannot be opened. A rename needs only write and search
permission, so on a directory without read permission the replacement was
published, never synced, and the mount went on taking deletes against it.

Sync through a helper that propagates the open error, as Go's
util.FsyncDir already does, so that case fails the mount like any other
post-rename sync failure.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: test the no-compaction-after-failed-load rule through the Go mount

The test for it handed compactEcjAfterLoad an artificial error on a volume
that had loaded cleanly, so it would not notice NewEcVolume dropping the
real load error on the way to compaction.

Make the journal read one of the injectable ecjFsOps steps and fail it
inside the real mount, after the first chunk, on a journal whose last
entry is an id the first chunk does not hold. The mount must leave the
file byte for byte as it was; a clean remount then compacts and keeps
that id. The Rust mount fails outright on a load error, so it has no
equivalent path.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: register ReceiveFile's .ecj writes with the journal registry

ReceiveFile refuses a mounted EC volume only once, when the info message
arrives, then creates the .ecj and streams chunks into it. A volume that
mounted on that journal mid-stream could find a bloated prefix, pass the
inode-and-size re-check and rename a compacted file over it; the rest of
the stream then went to the unlinked inode and was lost.

Register the path as a writer before the file is created, in both the Go
and Rust handlers, and hold it until the file is closed and any partial
copy removed, as the shard-copy and index-recovery appends already do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: skip .ecj compaction when a writer ran since the journal was loaded

Compaction checked only that no writer was active at the reservation, and
that the file was still the loaded inode at the loaded size. A ReceiveFile
truncates and refills the journal in place, so one that ran during the
mount's load, or after it, and finished before the reservation could leave
different ids at the same length; compaction then wrote the stale set over
them.

Give each path a write generation that every writer bumps as it starts. A
holder records it, and whether a writer was active, when it registers,
which is before it opens and loads the journal. It may compact only if no
writer was active then and the generation has not moved. Same rule in Go
and Rust; the journal read becomes an injectable step in Rust as it is in
Go, so both test the in-place rewrite through the real mount.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: match the ReadOnly(VolumeId) variant in write_volume_needles

#11543 matched VolumeError::ReadOnly as a unit variant in Store::write_volume_needles, and #11544 changed it to ReadOnly(VolumeId) in the same merge window. Each passed CI on its own, but master no longer compiles the Rust volume server. Carry the volume id through.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-03 09:08:19 +08:00
35b090a4df volume: merge .ecj as a set union on EC shard copy + index recovery (Rust+Go) (#11554)
* volume: merge .ecj as a set union on EC shard copy + index recovery (Rust+Go)

An EC volume's deletion journal is a set of needle ids, but shard copy
and index recovery appended the peer's whole journal, doubling the file
on every ec_balance round trip. Fold the peer's ids in as a union
instead: only ids the local journal lacks are appended.

- The journal is never replaced. A mounted EcVolume merges a peer's ids
  through its live handle under the lock deletes take (Go
  MergeJournal / Rust merge_journal), wherever its journal lives.
- An unmounted journal gets only the missing ids appended while mounts
  are excluded; the delta is read outside the lock and re-read if the
  journal changed.
- The source .ecj streams into memory as an id set: no staging files,
  chunked reads, memory proportional to distinct ids.
- Go and Rust agree that a source journal exists when it sends a
  modified time or any bytes. A missing source stays a no-op.
- Rust runs every merge in spawn_blocking and shares one receive/merge
  path between shard copy and index recovery.

The decode path and the journal format are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: route .ecj merges to the runtime that holds the journal open

Disks sharing one index directory all resolved as the journal's owner, so
the last one won and a sibling's mounted runtime was skipped: the merge
appended behind its open handle and the sibling kept serving the peer's
deleted needles until remount. Callers now name the receiving disk by its
data directory; the merge goes through that disk's runtime, else a
sibling runtime whose journal is the target file.

In Go the unmounted append now holds every disk's EC lock (in location
order) while it rechecks for a mount, so a sibling mounting from this
disk's index during the unlocked read is merged through instead.

In Rust a mount that lands during the read is merged through directly and
its added count returned, rather than discarded and reported as zero.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: sync merged .ecj records outside the disks' EC locks

The unmounted merge held every disk's EC read lock across its fsync, so a
slow sync on one disk held off mounts on all of them, along with the EC
reads queued behind those mounts. Mounts only need to be excluded while
the records are written: the write now happens under the locks and the
fsync after they are released, since a later mount reads the written
records from the page cache. A failed fsync rolls back only if nothing
has mounted the journal or appended to it since the write.

A merge through a mounted volume now keeps only that volume's disk locked
across its fsync.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: roll back an unsynced .ecj merge through a volume mounted mid-sync

If a volume mounted after the unmounted merge wrote its records but before
the fsync failed, the rollback kept the records because the journal was now
open, leaving ids in the volume's deleted set that may never reach disk; a
retried merge then saw them and synced nothing. The rollback now goes
through that volume the way its own failed journal fsync does: truncate
back and drop the ids from the in-memory set, so a retry appends and syncs
them again. It still keeps the records if the volume journaled since, as
truncating would lose that delete. No fsync runs under the disk locks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: decide .ecj merge rollback from the journal's actual length

Two runtimes can hold one journal (cross-disk mounts). The rollback of an
unsynced merge checked one runtime's cached ecjFileSize, which another
runtime's appends leave stale, so it could truncate a delete that runtime
had already synced. The rollback now holds every holder's journal lock and
truncates only if the file's actual length is still the append's end,
then updates each holder's size and deleted set. Otherwise later records
follow the merged ones, so they stay and are rewritten in place and
synced outside the locks, rather than left possibly not durable.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: keep unsynced .ecj merge ids out of mounted deleted sets

When a merge's fsync failed, later records blocked the rollback, and the
rewrite-and-sync failed as well, the merged ids stayed in every mounted
volume's deleted set without being shown durable, so a retried merge saw
them as present and synced nothing. They now leave those sets while the
records stay in the file, matching DeleteNeedleFromEcx, which publishes an
id only after its record syncs. The merge returns the error and a retry
appends and syncs them again.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: publish merged .ecj ids to every holder of the journal

Two runtimes can journal into the same file when disks share an index
directory. The merge went through only the first holder, leaving a
sibling's in-memory deleted set without the ids, so it could keep
serving a needle the peer deleted until it remounted. Every holder of
the journal now gets the merged ids, in Go and in the volume server.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Publish merged .ecj ids to the journal actually written

mountedEcJournal prefers the receiving disk's own runtime for the vid,
whose journal may live in its data directory while the copied records
name a sibling's journal in the index directory. Publishing by the
requested ecjPath then marked a holder of a different file deleted on
records that file never persisted, resurrecting the needles on remount.
Publish by the picked runtime's journal path instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-02 22:46:36 +08:00
eeec9ec09a volume server: refuse to compact a volume tiered to remote storage (#11545)
* volume server: refuse a tier move while compacting, and a commit once tiered

A tier move to remote and a vacuum compaction of the same volume could
interleave and leave the volume unreadable:

- A compaction committing while the upload ran swapped .dat/.idx under
  the transfer, which reopens the .dat by path per part. The move then
  published an object holding the old (or a mixed) layout against the
  compacted .idx, and with keep_local_dat_file=false deleted the only
  compacted .dat.
- A tier move finishing while the compaction copy ran (or between the
  copy and the commit) let the commit swap in the compacted .idx while
  the reload served the pre-compaction remote object through it.

The tier move now refuses to start while the volume is compacting, and
re-checks the compaction revision under the store write lock before it
records the remote file; on a mismatch it deletes the uploaded object
and fails with FailedPrecondition, leaving the volume local. Committing
a compaction on a volume that has a remote file is refused and its
.cpd/.cpx removed, since the reload would read the remote object
through the compacted index.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: abort a tier move whose volume was replaced or removed

The tier-up bookkeeping looked the volume up by id only and compared the
compaction revision. A delete and re-create of the same id during the upload
yields a fresh volume at the same revision, so the move recorded the old
volume's object on the new one and, without keep_local_dat_file, removed the
new .dat. An unmounted volume was skipped and the move reported success,
leaving the uploaded object referenced by nothing.

Capture the volume instance (its data-file access control Arc, as the scan
and read plans do) with the revision, and require both under the store write
lock. A replaced volume fails with FailedPrecondition, a missing one with
NotFound; either way nothing is recorded and the object is deleted after the
lock is released. Go fails in both cases because deleting or unmounting closes
the descriptor its copy reads.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: refuse to compact a volume tiered to remote storage

Committing a compaction of a tiered volume is refused, since the reload
would read the remote object through the compacted index. The compaction
itself still started: a tiered volume's data backend is the remote object
(the local .dat is dropped or deleted on tier-up), so an explicit vacuum
streamed the whole .dat out of remote storage into a .cpd that the commit
then discarded.

Refuse at the start of the compaction instead, before the .cpd is created,
at the point where Go's copy opens the local .dat. The truncated-index
test now uses a read-only local volume for its sorted index, since a
tiered one no longer reaches the copy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: match the ReadOnly(VolumeId) variant in write_volume_needles

#11543 matched VolumeError::ReadOnly as a unit variant in Store::write_volume_needles, and #11544 changed it to ReadOnly(VolumeId) in the same merge window. Each passed CI on its own, but master no longer compiles the Rust volume server. Carry the volume id through.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-01 23:11:16 +08:00
6f9becaa37 volume server: answer BatchDelete on EC needles as Go does (#11541)
* volume server: VolumeNeedleStatus reads remote EC shards and reports deleted needles like Go

For an EC volume the handler read only locally mounted shards, so a node
that did not hold the shard with the needle's bytes answered Internal
"ec shard N not available locally". Go's ReadEcShardNeedle fetches the
interval from a peer or reconstructs it. It also mapped every regular
volume read error, including a tombstone, to NotFound "needle not found",
which fs.verify treats as a missing needle; Go returns ErrorDeleted as a
plain error ("already deleted"), which fs.verify skips.

The EC branch now drops the store guard and uses the distributed EC read
the HTTP GET path uses. Errors map like Go: needle absent -> NotFound
"needle not found <decimal id>", tombstoned (regular or EC .ecx/.ecj) ->
Unknown "already deleted", anything else -> Unknown with the error text.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: tell EC deletions and vanished volumes apart in VolumeNeedleStatus

The distributed EC reader returned Ok(None) for an absent needle, a needle
a peer reported deleted, and a volume unmounted after the handler's own
existence check. VolumeNeedleStatus answered all three NotFound "needle not
found", which fs.verify -pruneEntries counts as lost data. A reported
deletion was also lost when an earlier interval failed.

The reader now says why it has no needle (EcMiss: NotFound, Deleted,
VolumeNotFound), classifying the local tombstone itself and letting a
reported deletion outrank other interval errors, as Go's ReadEcShardNeedle
does. VolumeNeedleStatus maps Deleted to Unknown "already deleted" and
VolumeNotFound to "volume not found", and drops its separate EC pre-check.
read_ec_shard_needle_distributed keeps its Ok(None) for every miss, so the
other callers are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: answer BatchDelete on EC needles as Go does

With skip_cookie_check, which every weed/ client sends, an EC needle that
was already deleted came back 404 "ec needle <fid> not found". Go's
DeleteEcShardNeedle gets ErrorDeleted from its read and BatchDelete
answers 304 with no error; the filer's deletion classifier only forgives
"already deleted" or an exact "not found", so it booked the repeat delete
as a permanent failure. The same mode also compared the fid cookie and
refused chunk manifests with 406, while Go never reads the needle before
those checks when skipping, so the filer's delete of a manifest chunk's
own fid failed permanently too.

The EC branch now reads with read_ec_shard_needle_or_miss and answers as
Go: skipping, a deletion is 304 and any other miss is 500 with Go's text;
checking, every miss is 404 with Go's text ("already deleted",
"locate in local ec volume: FindNeedleFromEcx: needle not found",
"ec shard <vid> not found"). The cookie and manifest checks run only when
the caller asked for the cookie check, which leaves the non-EC path as it
was.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: match the ReadOnly(VolumeId) variant in write_volume_needles

#11543 matched VolumeError::ReadOnly as a unit variant in Store::write_volume_needles, and #11544 changed it to ReadOnly(VolumeId) in the same merge window. Each passed CI on its own, but master no longer compiles the Rust volume server. Carry the volume id through.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-01 22:17:18 +08:00
9d3907e36c volume: say "volume N is read only" like Go, so filer retries deletes (#11544)
VolumeError::ReadOnly displayed "volume is read-only". Go's store and
volume say "volume %d is read only", and the filer's deletion classifier
requeues a failed delete only when the error contains "is read only".
Against a Rust volume server a BatchDelete on a read-only volume (tier
move, maintenance) was booked as a permanent failure and the chunk was
never deleted.

ReadOnly now carries the volume id and displays Go's text. The text
reaches clients through BatchDelete results, the HTTP write and delete
error bodies, and gRPC statuses; the gRPC code (FailedPrecondition) and
the HTTP/BatchDelete status codes are unchanged.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-01 21:00:04 +08:00
Eliah RusinandClaude Opus 5.5 fea14c01a7 volume server: refuse a tier move while compacting, and a commit once tiered (#11539)
* volume server: refuse a tier move while compacting, and a commit once tiered

A tier move to remote and a vacuum compaction of the same volume could
interleave and leave the volume unreadable:

- A compaction committing while the upload ran swapped .dat/.idx under
  the transfer, which reopens the .dat by path per part. The move then
  published an object holding the old (or a mixed) layout against the
  compacted .idx, and with keep_local_dat_file=false deleted the only
  compacted .dat.
- A tier move finishing while the compaction copy ran (or between the
  copy and the commit) let the commit swap in the compacted .idx while
  the reload served the pre-compaction remote object through it.

The tier move now refuses to start while the volume is compacting, and
re-checks the compaction revision under the store write lock before it
records the remote file; on a mismatch it deletes the uploaded object
and fails with FailedPrecondition, leaving the volume local. Committing
a compaction on a volume that has a remote file is refused and its
.cpd/.cpx removed, since the reload would read the remote object
through the compacted index.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: abort a tier move whose volume was replaced or removed

The tier-up bookkeeping looked the volume up by id only and compared the
compaction revision. A delete and re-create of the same id during the upload
yields a fresh volume at the same revision, so the move recorded the old
volume's object on the new one and, without keep_local_dat_file, removed the
new .dat. An unmounted volume was skipped and the move reported success,
leaving the uploaded object referenced by nothing.

Capture the volume instance (its data-file access control Arc, as the scan
and read plans do) with the revision, and require both under the store write
lock. A replaced volume fails with FailedPrecondition, a missing one with
NotFound; either way nothing is recorded and the object is deleted after the
lock is released. Go fails in both cases because deleting or unmounting closes
the descriptor its copy reads.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 20:20:16 +08:00
Eliah RusinandClaude Opus 5.5 62481f1673 volume server: ec.decode reads the .ecx from the index dir it was copied to (#11536)
* volume server: ec.decode reads the .ecx from the index dir it was copied to

VolumeEcShardsCopy writes the .ecx/.ecj into the receiver's -dir.idx, so
with a split data/index dir the decode target has no .ecx beside its
shards. VolumeEcShardsToVolume sized the .dat from the right .ecx but
built the .idx from the data dir, failing with NotFound after the .dat
was already published. It now reads .ecx/.ecj from where the EC volume
opened them and writes the .idx beside the .dat, where Go leaves it.

The live-entry check and the .dat size also ignored deletions recorded
only in the .ecj, which Go folds into the .ecx (RebuildEcxFile) first:
a fully deleted volume was decoded instead of reported as having no live
entries, and deleted tail needles were copied into the .dat. Both now
treat journaled ids as deleted, without rewriting the sealed .ecx.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: ec.decode keeps the decoded volume writable and reads every .ecj

The rebuilt .idx copied a journaled tail needle's .ecx row verbatim after
the .dat was cut short before it, so the mount saw a row past EOF and
marked the decoded volume read-only. Rows of deleted needles the .dat no
longer holds are now dropped, and each journaled needle still in the .dat
gets one tombstone instead of one per journal entry.

VolumeEcShardsCopy appends journals collected from other holders into
the idx dir, but the decode read only the .ecj beside the .ecx, which
sits in the data dir when this server generated the shards. It now
reads both, once, in bounded chunks via the loader EcVolume uses.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: test ec.decode drops a sealed .ecx tail tombstone

Covers the other half of the rule added in the previous commit: a tail
needle tombstoned in the .ecx itself (Go's RebuildEcxFile) is cut from
the .dat, and its row must not reach the rebuilt .idx either.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 20:19:17 +08:00
Eliah RusinandClaude Opus 5.5 b8f074b7d3 volume server: VolumeNeedleStatus reads remote EC shards and reports deleted needles like Go (#11535)
* volume server: VolumeNeedleStatus reads remote EC shards and reports deleted needles like Go

For an EC volume the handler read only locally mounted shards, so a node
that did not hold the shard with the needle's bytes answered Internal
"ec shard N not available locally". Go's ReadEcShardNeedle fetches the
interval from a peer or reconstructs it. It also mapped every regular
volume read error, including a tombstone, to NotFound "needle not found",
which fs.verify treats as a missing needle; Go returns ErrorDeleted as a
plain error ("already deleted"), which fs.verify skips.

The EC branch now drops the store guard and uses the distributed EC read
the HTTP GET path uses. Errors map like Go: needle absent -> NotFound
"needle not found <decimal id>", tombstoned (regular or EC .ecx/.ecj) ->
Unknown "already deleted", anything else -> Unknown with the error text.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: tell EC deletions and vanished volumes apart in VolumeNeedleStatus

The distributed EC reader returned Ok(None) for an absent needle, a needle
a peer reported deleted, and a volume unmounted after the handler's own
existence check. VolumeNeedleStatus answered all three NotFound "needle not
found", which fs.verify -pruneEntries counts as lost data. A reported
deletion was also lost when an earlier interval failed.

The reader now says why it has no needle (EcMiss: NotFound, Deleted,
VolumeNotFound), classifying the local tombstone itself and letting a
reported deletion outrank other interval errors, as Go's ReadEcShardNeedle
does. VolumeNeedleStatus maps Deleted to Unknown "already deleted" and
VolumeNotFound to "volume not found", and drops its separate EC pre-check.
read_ec_shard_needle_distributed keeps its Ok(None) for every miss, so the
other callers are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 20:18:34 +08:00
Eliah RusinandClaude Opus 5.5 5ece8dd63c volume server: drop the unused unmount result in VolumeCopy validation, and test-only EC helpers (#11531)
mount_and_reply ignored the Result of store.unmount_volume when a copied
replica failed record count validation, tripping unused_must_use. The Err
branch is unreachable there: the volume was mounted under the same store
write guard, mount_volume refuses an already loaded vid so it is a fresh
Volume with is_compacting false, and a compaction claim needs &mut Volume,
i.e. the store lock. Ignore the result explicitly with a one-line reason.

Store::delete_expired_ec_volumes and Store::remove_ec_volume are called
only from test modules (the heartbeat uses the split
find_expired_ec_volumes / remove_expired_ec_volumes halves), so mark them
#[cfg(test)].

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 20:13:45 +08:00
Chris LuandDevin 0978e7f833 vacuum: keep disk-full read-only volumes reclaimable (#11519)
* storage/topology: keep disk-full read-only volumes vacuumable

The vacuum sweep skipped every read-only replica, so a volume that went
read-only because its disk filled could never reclaim its garbage — the
exact situation compaction exists for. The volume server now reports
disk_space_low in VacuumVolumeCheckResponse, and the sweep skips a
read-only replica only when the flag is clear. An explicit volumeId
vacuum is unaffected: it already bypassed the read-only rule.

The field takes number 4: 2 and 3 are downstream-allocated for tombstone
retention, keeping the wire merge clean.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* storage: measure vacuum free space against live bytes

The pre-compaction space check required the current .dat + .idx size
free, which includes the garbage being reclaimed — on a nearly full disk
that estimate can never fit, so the volume stayed garbage-bound forever.
Measure against the estimated compacted output instead: superblock plus
live index entries plus live content bytes, with the existing ten
percent buffer unchanged. Mirrors the same check in the Rust volume
server.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vacuum: count per-needle framing in the compacted-size estimate

The live-bytes estimate covered each live needle's content and index
entry but not its .dat framing (header, checksum, timestamp, padding —
~32 bytes on version 3). For small-needle volumes that is more than the
10% headroom, so a disk with space between the estimate and the real
output still ran out mid-compaction. Rust side mirrors the same formula.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* storage: report disk_space_low only when it is the sole read-only cause

Review feedback (ihnokim, greptile, devin): a volume read-only for low
disk space AND an operator mark or I/O quarantine was still eligible for
the automatic sweep, rewriting a copy meant to stay protected. The flag
now reports only the benign sole-cause case in both servers.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: fail closed when the read-only lookup misses in the sweep

A heartbeat can drop the volume from the DataNode cache between the
location-list copy and VacuumVolumeCheck; a lookup error previously
skipped the read-only check entirely. Review feedback (coderabbit).

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-30 17:32:41 +08:00
Chris LuandDevin 43fd5b8d82 volume: reclaim staged EC shard generations left by the 2PC switch (#11501)
* volume: remove staged EC generation files on teardown and shard delete

The 2PC generation switch stages each run as <base>.ecNN.v<N> plus
versioned .ecx/.ecj/.vif files. Nothing on the volume server removes
them: isEcDataShardFile only recognises the exact .ecNN name, so the
staged files are invisible to every bookkeeping pass, and even
full_teardown's wipe-all path left them behind. Each re-encode therefore
leaks a full shard set per shard-holding disk.

RemoveEcGenerationFiles sweeps <base>.ec*.v<N> and <base>.vif.v<N>,
optionally keeping generations at or above a threshold; teardown and the
reconcile wipe remove every generation, and a per-shard delete removes
that shard's staged generations too.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: delete staged EC generations older than N via VolumeEcShardsDelete

After a 2PC generation switch commits, the superseded generation's
<base>.*.v<N> files sit on disk with no cleanup path: teardown removes
everything, and a per-shard delete only touches the named shards, so the
executor had no RPC that reclaims just the staged leftovers.

delete_generations_older_than removes staged generation files strictly
below the threshold on every disk. Versioned files are never mounted, so
nothing is unloaded first; the committed generation and the canonical
files are preserved.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: mirror staged EC generation cleanup

Parity with the Go volume server: remove_ec_generation_files sweeps
<base>.ec*.v<N> and <base>.vif.v<N> staged by the 2PC switch, called by
remove_ec_volume_files (which covers both teardown paths) and the new
delete_generations_older_than request field; delete_ec_shards removes a
shard's staged generations along with the canonical file.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: match staged generation filenames literally

filepath.Glob interprets metacharacters in the collection part of the
base name, so a collection like a[bc] could match another volume's
staged files (or miss its own). Scan the directory and compare names
literally instead, mirroring the Rust read_dir implementation.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: report generation-sweep errors and drop the store lock first

- snapshot the location base names under the read lock and run the
  filesystem sweep after dropping it, so a slow disk cannot stall the
  store;
- record per-entry read_dir errors in remove_ec_generation_files and
  propagate them from remove_ec_shard_generations instead of flatten()
  skipping them;
- warn when a staged-shard generation fails to delete rather than
  reporting success with files left behind.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: fail shard delete when the staged-generation listing fails

A transient ReadDir failure fell back to removing canonical shard names
only: staged .v<N> files survived while the RPC still reported success,
leaving the leak invisible to retrying callers. ENOENT still means the
disk simply has no such directory; other listing errors now propagate.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: propagate staged-generation removal failures

delete_ec_shards logged remove_ec_shard_generations errors and the RPC
returned success while staged .v<N> files remained, diverging from the
Go handler which surfaces the failure. The sweep keeps processing the
remaining shards, retains the first error, and volume_ec_shards_delete
maps it to Status::internal so callers can retry.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: notify state change even when the shard sweep errors

delete_ec_shards already deletes and unmounts the shards before
returning a staged-generation failure, so returning early skipped
volume_state_notify and the master kept routing to them until the next
heartbeat. Notify before propagating the error.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-28 21:55:25 +08:00
67691a1eea volume server: split volume_copy into phases and type the delete-after-status gate (#11485)
volume_copy was one ~400-line handler, and the rule that an existing
local replica is deleted only after the source's ReadVolumeFileStatus
succeeded was held by statement order alone. The keep_remote_data=true
that the pre-copy delete and the failed-copy rollback must share was
kept in sync by a comment pointing from one to the other.

The handler is now a ~60-line orchestrator over connect_to_copy_source,
SourceVolumeStatus::fetch, delete_existing_replica,
plan_copy_destination and a VolumeCopyJob whose run() drives
preallocate_dat, transfer_files, finish_copied_files and
mount_and_reply, with cleanup_failed_copy on error.
delete_existing_replica takes a &SourceVolumeStatus, which only fetch
can construct (private field in a child module), so the delete cannot
be called before the status RPC. Both deletes go through
delete_replica_keep_remote.

Pure refactor: call order, status codes and messages, cancellation
checks, throttling, progress reports and cleanup are unchanged.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-09-27 20:12:55 +08:00
hsdfat 02353444ac fix(volume-rust): reserve a disk before replacing a replica in VolumeCopy, and check record counts (#11483)
* fix(volume-rust): reserve a disk before replacing a replica in VolumeCopy, and check record counts

Port of the Go VolumeCopy hardening in #11238 and #11252.

- Pick the destination disk before deleting the existing replica, counting
  the slot that replica holds as free. If no disk qualifies, the healthy
  replica is kept instead of being deleted.
- Read the source's VolumeStatus before and after the copy. When both
  succeed and the counts did not change, the mounted replica's file and
  deleted counts must match; on mismatch it is unmounted and its files
  removed. A failed "before" read skips the check; a failed "after" read
  fails the copy.

* fix(volume-rust): let a departing caller cancel VolumeCopy's post-copy status read

Go reads the source's status after the copy with stream.Context(), so the
call ends when the caller leaves. The Rust call had no such link: a
source that stalled there held the copied, unmounted files after the
caller was gone. Race it against the response channel, like the other
blocking steps, so the usual error cleanup removes the partial copy.
2026-09-27 19:40:07 +08:00
Eliah RusinandClaude Opus 5.5 68944e83a3 volume: typed tier errors so a missing remote object answers NotFound (#11484)
remote_storage/s3_tier.rs returned Result<_, String> from every
transfer (upload_file, download_file, read_range[_blocking],
delete_file[_blocking]) and from the tier runtime helpers. The tier
move handlers could only wrap that in Status::internal, so a .dat whose
remote object is gone was indistinguishable from an I/O failure to
weed shell.

Add TierError { NotFound, Io, RuntimeUnavailable, Aborted }. Each
variant carries the existing message verbatim. NotFound follows the
rules remote_storage/s3.rs already uses: raw 404 status on HEAD,
NoSuchKey code on GET; a bare 404 on GET stays Io. A progress-callback
Err becomes Aborted. VolumeError gains a transparent Tier variant and
From<VolumeError> for Status maps Tier(NotFound) to NotFound; the tier
move handlers go through status_with_context, so their message text is
unchanged. Every other tier failure is still Internal.

The remote needle read path keeps io::Error::other, so its error kind
and vacuum's handling of it do not change.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 19:39:25 +08:00
Eliah RusinandClaude Opus 5.5 00310f6588 volume server: run the vacuum compaction copy without the store lock (#11482)
* volume server: run the vacuum compaction copy without the store lock

VacuumVolumeCompact held the store write lock for the whole live-needle
copy, including every progress blocking_send on the 16-deep stream. On a
large volume that is minutes with every read, write and heartbeat on the
node parked behind it, long enough for the master to unregister the node.

Split compaction the way Go's CompactByIndex runs it. A short locked step
claims the volume's compacting flag, records the makeup_diff watermark
(index size and compaction revision) and opens fresh .dat/.idx handles.
The copy then replays .idx up to the watermark and copies from those
handles with the store lock released; writes that land meanwhile are
replayed by makeup_diff at commit, as before. The flag is an Arc<AtomicBool>
released when the job is dropped, so every exit path clears it.

Because the flag is now visible to other callers, the operations that would
pull the files out from under the copy refuse while it is set: unmount
(and VolumeConfigure, which unmounts and remounts), delete (checked before
the volume is removed from the map, which a refused destroy used to leave
unmounted), cleanup, and index relocation. A second compact and a commit
stay no-ops, as in Go. The pre-copy fsync is dropped: the copy reads its
own handles through the page cache and .cpd/.cpx are fsynced before commit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume: keep a read-only in-memory index's size for the compaction copy

The unlocked copy replays .idx up to index_file_size(). A read-only
volume whose .sdx could not be built loads its index into memory without
a writer, so that size stayed 0: the copy came out empty and the commit
replaced the volume with it. CompactNeedleMap::load_from_idx now records
the rows it loaded, which is also what Go's IndexFileSize reports for a
read-only index.

The copy's index replay now stops reading at the recorded size instead
of walking rows appended since, which makeup_diff replays anyway.

Adds tests for compacting a read-only volume on both the sorted index
and the in-memory fallback, and for VolumeConfigure stopping when the
unmount is refused during a copy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: stop a vacuum copy as soon as its client is gone

The progress callback only noticed a closed response stream when a report
was due, every 128 MiB. With the copy now running outside the store lock,
a copy nobody waits for keeps the volume marked compacting and so keeps
refusing unmount, delete and cleanup until that next report. Check the
stream on every callback.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 19:38:59 +08:00
Eliah RusinandClaude Opus 5.5 5c9c424a84 volume server: stream ReadAllNeedles without holding the store lock (#11481)
* volume server: stream ReadAllNeedles without holding the store lock

read_all_needles held store.read() while Volume::read_all_needles read
every live needle of the volume into a Vec, and kept holding it through
the whole blocking_send loop. Memory grew with the volume, and a slow
client parked the scan in a send with the guard held; needle writes and
the heartbeat take store.write() on a writer-preferring lock, so the node
stopped serving until the client caught up.

Take a DatScanPlan (fresh .dat open, end bound) under a short guard and
walk it with the guard released, sending one needle at a time. Each
record is checked against the live needle map under a brief read guard,
as the scan reaches it, and only a live record is parsed, so a damaged
stale copy does not fail the stream. Records appended while a pass ran
are walked by a follow-up plan, so a needle overwritten during the scan
is streamed once, as its new copy. A vacuum commit or re-create of the
volume during the scan fails the stream, since the map's offsets no
longer describe the pinned file; the plan carries the volume instance
and compaction revision for that check.

DatScanPlan::scan_records yields records unparsed; scan keeps its
behaviour on top of it. Volume::read_all_needles has no caller left and
is removed; its tests move to the RPC.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: check ReadAllNeedles liveness only once the send can proceed

The per-record liveness check ran before blocking_send, so a scan parked
on a full channel held a record it had already judged live. An overwrite
landing during that park left the old copy in the stream, and the
continuation over appended records then streamed the new copy as well.

Reserve channel space first, then take the store read guard, check the
record against the needle map and enqueue it through the permit before
releasing the guard. The wait for space still happens without the lock;
the record is parsed before the guard is taken, and its parse error only
counts if the record turns out to be live.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 19:38:43 +08:00
Chris LuandDevin 8ad2f29e3e shell: let volume.deleteEmpty drop volumes with no live needles (#11437)
* shell: let volume.deleteEmpty drop volumes with no live needles

The candidate check only accepted a .dat at superblock size, so a volume
whose every needle was deleted still had to be vacuumed first — minutes
of compaction to rewrite bytes that were all garbage anyway. FileCount
counts every indexed entry and DeleteCount every entry made garbage by
overwrite or delete, so FileCount <= DeleteCount means nothing live
remains and the volume can be unlinked directly. The quietFor guard is
unchanged.

* volume server: add only_garbage VolumeDelete guard

VolumeDelete(only_empty) refuses every volume that ever held data, so a
volume whose needles are all deleted could only be removed after a
vacuum rewrote it. The new only_garbage flag deletes only when the byte
counters show nothing live: DeletedSize covering all of ContentSize, the
same all-garbage state vacuum measures. Byte counters are used because
the file/delete counts drift on index reload.

* rust volume: mirror only_garbage VolumeDelete guard

Same check as the Go server: a volume deletes under only_garbage when
its deleted bytes cover all content bytes. The grpc handler rejects
before the store drops the volume from its map, since destroy errors
after removal would still unmount it.

* volume delete: let either enabled check pass, keep onlyEmpty on the wire

An upgraded shell sending only_garbage to a pre-upgrade server would be
read as an unconditional delete (field ignored, only_empty false). The
request now keeps only_empty set so old servers check emptiness and
refuse, while new servers delete when either check passes.

* volume.deleteEmpty: skip remote-backed and protected read-only volumes

A remote-tiered replica shares its cloud object with the other replicas,
so keepRemoteData=false on one delete removes data they still reference.
Protected read-only volumes are quarantined or under maintenance, which
is exactly when a replica should not be dropped.

* volume delete: validate guarded copies across disks before deleting

* volume delete: hold copy locks across guarded validate-and-delete

CheckVolumeDeletable released each copy's locks before Destroy ran, so a
write landing on a later copy between the two passes refused its destroy
after earlier copies were already removed. Pin every copy's
dataFileAccessLock (and its location's volumesLock) across validation and
removal so a refused delete leaves all copies intact.

* volume delete: send deleted-volume notices after releasing locks

A blocking send on a full DeletedVolumesChan under volumesLock can stall
the heartbeat loop that drains it while it waits on the same locks.
Collect the notices under the lock span and send after release.

* pb: restore generated-file cosmetics to match the repo's protoc version

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 15:53:39 +08:00
4299fdf578 volume server: VolumeEcShardsDelete full teardown unloads every disk and keeps the shard gauge honest (#11446)
* volume server: VolumeEcShardsDelete full teardown unloads every disk and keeps the shard gauge honest

Go's VolumeEcShardsDelete full teardown calls vs.store.UnloadEcVolume in
the blanket path (weed/server/volume_grpc_erasure_coding.go:488) and
location.UnloadEcVolume in the generation-fenced path (:511): each disk
that had the volume registered drops it, closes its shard descriptors
and gives back its ec_shards gauge before the artifacts are unlinked.

The Rust handler used Store::remove_ec_volume / DiskLocation::remove_ec_volume
instead, which only remove the map entry. Store::remove_ec_volume also
stops at the FIRST disk holding the vid, so on a split-disk volume
(shards on several disks) the blanket teardown left the sibling disks'
EcVolume registered with open fds while the unlink loop deleted their
files underneath it: the heartbeat kept advertising shards whose files
were gone, the inodes stayed pinned by the open descriptors, and the
VOLUME_GAUGE{collection,"ec_shards"} never came back down. The fenced
path leaked the gauge and the descriptors the same way on the one disk
it wiped.

Both paths now use the unload_ec_volume helpers from #11413 (every disk
for the blanket teardown, the strictly-older disk for the fenced one),
and the two Status::internal messages name the disk directory like Go's
"... on %s: %w".

Regression tests build a two-disk store with the same vid mounted on
each disk (the SplitDiskEcFixture, which gains a collection knob so the
gauge read is isolated from parallel tests mounting under "") and assert
that a blanket teardown leaves no EcVolume registered on any disk and
returns the gauge to its pre-mount value, and that a fenced teardown
decrements the gauge for the older disk's shard while preserving the
newer disk. Both fail against the previous handler.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume server: trim comments on the EC full-teardown unload path

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 12:00:23 +08:00
df4995b894 volume server: VolumeMarkReadonly answers NotFound when the volume vanished under the lock (#11443)
* volume server: VolumeMarkReadonly answers NotFound when the volume vanished under the lock

make_volume_readonly looked the volume up, notified the master (step 1),
then took the store write lock (step 2) and marked the volume only `if
let Some(..)`. When the volume left the store during step 1 -- a master
round trip, during which an unmount or a heartbeat expiry can land --
the missing else meant the RPC reported success for a volume the server
no longer has, and step 3 told the master again that it is read-only.

Go's Store.MarkVolumeReadonly (weed/storage/store.go) returns
"volume %d not found" when findVolume comes back nil, and
makeVolumeReadonly (weed/server/volume_grpc_admin.go) returns that error
before the step-3 notification. The Rust step 2 now does the same:
find_volume_mut(vid) -> Status::not_found("volume {vid} not found"), and
the `?` skips step 3, as it already did for a set_read_only_persist
failure. The scrub caller already matches NotFound to skip such a
volume instead of failing the whole report; it now actually gets it.
volume_mark_writable already returns NotFound under its write lock.

The regression test opens the step-1 window deterministically: step 1
awaits the current_master_url read lock, so the test holds its write
guard, lets make_volume_readonly park there after its own lookup
succeeded, unmounts the volume, then releases the guard. With no master
configured the notification is a no-op, so the write lock in step 2 is
the only place left that can notice the volume is gone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume server: trim comments on the vanished-volume mark-readonly path

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-25 22:05:53 +08:00
Chris LuandDevin 317e756b9a volume server: validate ext and collection in gRPC CopyFile/ReceiveFile (Rust) (#11451)
* volume server: validate ext and collection in gRPC CopyFile

Port the Go-side checks (checkVolumeFileExtension, checkVolumeCollection)
to the Rust volume server so a client-supplied collection or ext carrying
a separator or ".." cannot fold a path outside the volume directory.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume server: validate ext and collection in gRPC ReceiveFile

Same port on the write path: the file ReceiveFile creates is built from
client-supplied fields, so reject traversal there too. Reported through
the response error field, matching Go's SendAndClose.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-25 09:47:16 +08:00
bb9942c646 volume server: sweep stale EC artifacts before VolumeEcShardsGenerate re-encodes (#11413)
* volume server: sweep stale EC artifacts before VolumeEcShardsGenerate re-encodes

The Rust VolumeEcShardsGenerate went straight into write_ec_files: no unload
of an already-mounted EC volume and no stale-artifact sweep. Only .ec00..ecNN
on the encoding disk were truncated, so a retry could mix two encode runs. A
stale N.ec03 left on a sibling disk survived, reconcile later mounted it
against the new .ecx, and the new .vif made the encode_ts_ns identity guard
pass, so reads served old-run bytes at new-run offsets.

Mirror Go's VolumeEcShardsGenerate (#9880 / #9953): UnloadEcVolume on every
disk, then removeStaleEcArtifacts on every disk location before encoding.
remove_ec_volume_files_full_teardown already has removeStaleEcArtifacts'
semantics (.ec00..ec31, .ecx/.ecj/.ecsum[.vN] in both the data and idx dirs,
.vif only on a shard-only disk; never the source .dat/.idx), so reuse it. Add
Store::unload_ec_volume, which unlike remove_ec_volume does not stop at the
first disk and closes the descriptors so the unlink frees the inodes. The
store write lock covers only unload + sweep, not the encode.

The failure arm now also drops the generation-0 .ecsum, as Go's defer does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume server: wake the heartbeat after VolumeEcShardsGenerate unloads shards

The pre-encode unload drops mounted EC shards from memory, but unlike every
other unmount path it did not wake the heartbeat, so the master kept routing
reads to shards this server no longer serves until the next pulse. Notify
once the store lock is released, and before the sweep error propagates: a
failed sweep has unloaded the shards too.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume server: clean up encode artifacts when the .vif write fails too

Go's shouldCleanup defer covers every error before the .vif commits,
not just a failed encode. A serialize or write failure on the .vif left
the fresh .ecNN/.ecx/.ecsum behind, which the next generate would have
to rely on the new sweep to remove. Extract the cleanup and run it on
the .vif error paths as well.

* volume server: write the EC .vif atomically

Go's SaveVolumeInfo writes a temp file, syncs it, and renames it over
the target, so a failed write leaves the previous metadata intact and a
read-only .vif fails the save. The direct fs::write truncated the file
first, so a write or sync failure could leave an empty .vif even after
cleanup_encode removed the generated shards.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-09-24 07:10:54 +08:00
f0afcf904d volume: an EC volume needs an .ecx to mount, and a 0-byte stub never outranks a real index (#11415)
* volume: an EC volume needs a non-empty .ecx to mount

Two gaps against Go in how the Rust volume server treats the .ecx.

EcVolume::new mounted with no index at all. The per-shard
VolumeEcShardsMount path picks the disk by shard file alone, so a shard
whose .ecx was on no local directory still registered and was
advertised to the master; every VolumeEcShardRead then failed with
"ecx file not open", and add_shard's 0-byte guard was neutralised
because ecx_file_size stayed 0. Go's NewEcVolume returns an error
wrapping os.ErrNotExist. EcVolume::new now fails with NotFound, and
Store::mount_ec_shard looks up the .ecx owner across all disks first
(findEcxIdxDirForVolume) so a shard on a sibling disk of its index
still mounts instead of turning into a hard failure.

A 0-byte .ecx stub, as left by a failed EC distribute copy, counted as
a valid index. Go requires Size() > 0 wherever the file steers a
decision: HasEcxFileOnDisk, findEcxIdxDirForVolume, indexEcxOwners
(shared by reconcile and mirror), and VolumeEcShardsCopy removes a
copied 0-byte .ecx and fails the copy. Mirror each through one
is_usable_ecx_file helper. NewEcVolume itself still accepts a lone
0-byte .ecx as a legitimate empty index, but prefers a non-empty copy,
local directory first, over a stub in the other directory; the
resolution in EcVolume::new now follows the same order.

Tests that mounted EC volumes without any .ecx get a real fixture.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume: mount_ec_shard tries every disk; reconcile ignores a 0-byte local .ecx

mount_ec_shard returned the first disk's error, so an unusable shard copy
(a 0-byte .ecNN left by an interrupted move) hid a good copy on the next
disk. Like Go's MountEcShards, keep scanning: NotFound means "not this
disk", any other failure is collected, and an all-disks-fail error names
every disk tried. "No .ecx on any local disk" is now told apart from
"shard not on this server".

The orphan-shard reconcile took its locally-mirrored fast path whenever a
local .ecx existed at all. A 0-byte stub there registered the shards against
an empty index while the owner index skipped that same stub. Go gates the
fast path on HasEcxFileOnDisk; do the same. ec_local_ecx_path loses its last
production caller and becomes test-only.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* volume: match Go's mount error text and skip the owner stat on the owning disk

MountEcShards in Go skips the HasEcxFileOnDisk stat when the disk's own
directories already hold the .ecx, dedups a shared -dir.idx across
locations in findEcxIdxDirForVolume, and reports "load failures" with
the same wording. Also drop two issue-number references from comments.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-09-24 07:08:56 +08:00
ssshr-66andChris Lu b9ad62fc16 [Volume] Keep DAT and index state consistent after async batch Sync failure (#11425)
* fix 11400

* persist failed-recovery quarantine and harden rollback

- record the unavailable state in a .unavailable marker, fsync it, and
  re-arm it on load so a restart cannot serve an unverified pair
- quarantine the volume so heartbeats stop advertising it
- block MarkVolumeWritable while unavailable, rechecked under noWriteLock
- fail every request of a failed batch, not only the succeeded ones
- restore the needle map and truncate .dat on inline fsync rollback failure
- add truncateIndex for the sorted-file needle map
- mirror the fail-closed semantics in the Rust volume server

* volume: erase rolled-back mappings instead of leaving tombstones

A rolled-back batch or failed inline write used Delete() to undo a
needle that did not exist beforehand, leaving a tombstoned map entry
whose stale offset makes the next write to that needle fail reading a
header that no longer exists. Add removeMapping/restoreMapping to the
mappers so recovery erases entries that were absent before the batch
and reinstates the exact prior offset/size for ones that were,
including tombstones. The index row still goes through Delete so a
replay forgets the needle.

* volume: gate bulk readers on unavailable and fsync the marker's dir

- fsync_dir(&self.dir) synced the volume dir's parent, not the dir
  holding .unavailable; pass the marker path so the create survives
  a host crash
- export UnavailableError and check it in ReadAllNeedles,
  VolumeTailSender, VolumeIncrementalCopy, and IncrementalBackup so
  replica-sync paths cannot stream or append data from an unverified
  .dat/.idx pair; mirror on the Rust side via read_dat_slice,
  read_all_needles, dat_scan_plan, and the incremental-copy handler

* volume: drop issue references from comments near touched code

* volume: stop active scans when the volume becomes unavailable

The stream entry-point checks ran once per RPC, so a volume quarantined
by a failed recovery mid-scan kept serving data. Recheck availability
per needle/chunk on the detached read paths: tail scan and heartbeat,
read-all, incremental copy, incremental backup writes, and the Rust
StreamingBody chunk reads. Rust incremental copy also rejects a
quarantined volume before sync_to_disk touches the backend.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-24 06:57:44 +08:00
4bb40732bb volume server: ReceiveFile loses bytes and hides fsync failures (#11407)
* volume server: ReceiveFile loses bytes and hides fsync failures

Three defects in one handler, all on the path that receives a pushed
.dat/.idx/.vif or EC shard:

  - `f.write(&content)` never compared the return to content.len().
    A short write (ENOSPC, NFS) counted only the bytes that landed,
    so every later chunk was written at a shifted offset and the RPC
    answered error: "" with a byte count that looked right. Go's
    os.File.Write loops. Now write_all.

  - `let _ = f.sync_all();` discarded EIO and answered success with
    the full byte count. Go omits the check too, but
    ReceiveFileResponse carries an `error` field and the caller
    renames the staged file into place on success -- so a silent
    fsync failure publishes a file whose data never reached the
    platter. Flush and fsync failures are now reported.

  - Both the per-chunk write and the final fsync were blocking
    std::fs calls inside the async fn, on the runtime worker that is
    also driving the stream. Switched to tokio::fs + BufWriter, the
    shape `drain_copy_stream_to_file` in this same file already uses
    and documents. The partial-file cleanup on the error path moves
    to tokio::fs::remove_file for the same reason.

The handler had no test at all, which is how the short-write bug
survived. Added a round-trip over a real connection with ragged chunk
boundaries, asserting the bytes on disk and not only the reported
count -- a dropped or reordered chunk changes the file even when
bytes_written still adds up.

That test guards the rewrite; it does not reproduce the original
faults. ENOSPC and EIO need fault injection that this suite has no
harness for, so the short-write and fsync paths are argued from the
code, not demonstrated by a failing test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* volume server: remove the staged file on every ReceiveFile error reply

Flush and fsync failures returned early and left the partial .copying or
shard file behind, as did the pre-existing write-error path. Route all
response-level errors through one cleanup block, matching Go's
close-and-remove on a failed write.

* volume server: tighten ReceiveFile comments

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>
2026-09-20 23:26:37 -07:00
8ff2e0777e volume server: HTTP DELETE on a distributed EC volume (#11405)
* volume server: HTTP DELETE on a distributed EC volume

The delete handler validated the cookie with EcVolume::read_ec_shard_needle,
which reads only locally-mounted shards and errors "ec shard N not available
locally" for any interval held by a peer. Every Err was mapped to 500 and no
.ecj tombstone was appended, so on a standard 10+4 spread over 14 servers an
HTTP delete of an EC needle could not succeed. The GET path already goes
through read_ec_shard_needle_distributed.

Route the delete's read through the same distributed reader. It does a
local-first pass in its snapshot phase, so the all-shards-local case costs
what it did before, and no store guard is held across the await (the reader
takes its own; RwLockReadGuard is !Send).

Two smaller corrections fall out of the new return type:

  - the reader reports both "needle not in the index" and "volume vanished
    between the has_ec check and the snapshot" as Ok(None), which collapses
    the old Some(Ok(None)) and None arms into one 404;
  - an io::ErrorKind::NotFound now answers 404 rather than 500, matching the
    GET path. Telling a caller to retry a delete that can never succeed was
    half the bug.

The cookie check and its ordering before the journal append are unchanged.

Not addressed here: Rust journals the tombstone locally while Go routes it to
the primary shard holder. That is a separate behaviour change and belongs in
its own PR against the same issue-10 checkbox.

The regression test mounts 13 of 14 shards, leaving out the one holding the
needle's interval. The distributed reader seeds its Reed-Solomon buffers from
locally mounted siblings, so with >= 10 survivors it reconstructs with no peer
fan-out -- which makes the bug reproducible on a single node. Against the
unfixed handler the test fails with 500 vs 202.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* volume server: fail the delete when the EC volume unmounts mid-request

find_ec_volume_mut returning None used to fall through to a 202 with no
.ecj tombstone written, reporting success for a delete that did not
happen. Answer 404 like the other volume-vanished arms so the caller can
retry after a remount.

* volume server: forward EC needle deletes to a primary-shard holder

Mirror Go's doDeleteNeedleFromAtLeastOneRemoteEcShards: the tombstone is
journaled on one holder of the needle's primary data shard via
VolumeEcBlobDelete (or the local journal when this server holds the
shard), falling back to any other shard holder when the primary has
none. Journaling only on the node that received the DELETE scattered
tombstones across whichever server took the request.

* volume server: route BatchDelete EC deletes through the same forwarding

BatchDelete had the same local-journal divergence as HTTP DELETE, plus a
gap the old code admitted in a comment: the .ecx index cannot supply the
needle's cookie, so EC deletes ran with no cookie check at all. A
distributed read now fills the needle for every EC entry — matching Go's
DeleteEcShardNeedle, which reads and compares the fid cookie even when
skip_cookie_check is set — and the tombstone forwards via
delete_ec_shard_needle_distributed. A needle deleted between read and
journal reports 304 like Go's ErrorDeleted; a vanished volume reports
500 so the filer retries.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>
2026-09-20 23:26:31 -07:00
Eliah RusinandClaude Fable 5.1 ac876eef21 rust volume: build outgoing gRPC clients in one place and give every dial a connect timeout (#11354)
server/grpc_client.rs stopped at build_grpc_endpoint() -> Endpoint, so all 13
production call sites hand-wrote the same .connect() + X::with_interceptor()
+ two max_*_message_size() lines. Four of them -- VolumeCopy,
VolumeTailReceiver, VolumeEcShardsCopy and the HTTP chunk batch-delete fan-out
-- dialed with no timeout at all, so an unreachable peer whose TCP handshake
never completes (SYN dropped, blackholed route, host behind a silent firewall)
left the operation waiting on the kernel's own retry budget, minutes long.

Add GrpcDialOptions (unary / long / stream presets), connect_channel(), and
volume_server_client() / master_client() / filer_client() constructors that
attach the request-id interceptor and lift both message-size limits, then
route all 13 sites through them. build_grpc_endpoint is private again, so
connect_channel is the only way out of the module and no call site can dial
without picking up a bound. Each site's existing timeouts are preserved
exactly; the four bare dials gain a 5 s connect timeout and nothing else. No
per-request deadline was added to any streaming call: Endpoint::timeout is a
per-request bound on time-to-first-response-headers for every request the
channel carries, so a value picked for one short call would also be the header
deadline for the whole-volume transfer sharing the dial.

The new bound covers the TCP handshake only -- tonic hands connect_timeout to
HttpConnector::set_connect_timeout. A peer that completes the handshake and
then stalls in the TLS or HTTP/2 exchange is still unbounded at those four
sites, as are the RPCs themselves. That is why the three ping_* helpers keep
their outer tokio::time::timeout: replacing it with connect_timeout would have
narrowed a whole-connect bound they already had.

main.rs no longer re-declares GRPC_MAX_MESSAGE_SIZE and the three
keepalive/window constants; it imports them from grpc_client.rs so the
inbound server and the outgoing clients cannot drift apart.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 19:41:37 -07:00