mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 15:41:15 +02:00
master
161
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
005012edcf |
rust volume: quick-repair redb on durable checkpoints (#11203)
* rust volume: insert redb rebuilds in needle-id order Unlink the .rdb before create (create does not truncate). Collapse last-write-wins, then insert live keys sorted so 4.2.0 packs leaves. * rust volume: rebuild redb from a BTreeMap and clear leftover keys Peak rebuild memory is one ordered map instead of HashMap + Vec + stable-sort scratch. Unlink stays best-effort: if it fails, retain clears the leftover table before sorted insert. Compute idx metrics before the write so a read error does not unlink a committed .rdb. * rust volume: drop the extra redb read transaction on put/delete put uses insert()'s previous value. delete gets then inserts the tombstone in the same write transaction. Truncate the .idx row on any failed redb write after the append. * rust volume: unpack redb blobs through packed_to_needle_value save_to_idx, ascending_visit, and collect_entries used the same length-check copy as get. Route them through the helper so a wrong-length value is absent everywhere, not a panic. * rust volume: quick-repair redb on durable checkpoints set_quick_repair(true) on the durable checkpoint transaction so an OOM-killed volume server opens without a full-file repair scan. * rust volume: reopen redb from .idx on non-poisoned commit error redb 4.2.0 can make a Durability::None commit visible before returning Err(CommitError::Storage(..)). In that state the database refuses further write transactions, so truncating the .idx row (the old behavior) would leave a redb-only put or tombstone that the stored idx_size makes the reload skip. Distinguish CommitError::TransactionPoisoned (txn rolled back, db still usable -- truncate the orphan .idx row as before) from other commit errors (change may be visible, db refuses writes -- keep the .idx row, close the database, and reopen from .idx to repair redb's internal state). db becomes Option<Database> so reopen_from_idx can drop the old file lock before load_from_idx opens the same path. rdb_path, version, and cache_bytes are stored so the reopen uses the same configuration. * rust volume: truncate .idx row when redb is closed in put put appends the .idx entry before acquiring the write transaction. When db_or_err() fails (db is None after a failed reopen), the ? returned without calling truncate_idx_to_offset, so a write reported failed remained in the authoritative .idx and was replayed on restart. Handle db_or_err() explicitly and truncate the orphan .idx row before returning the error, matching the existing handling for begin_write, open_table, and insert failures. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Eliah Rusin
|
ed5b342f0c |
rust volume: optional redb insert_before bulk load (#11205)
* rust volume: quick-repair redb on durable checkpoints set_quick_repair(true) on the durable checkpoint transaction so an OOM-killed volume server opens without a full-file repair scan. * rust volume: optional redb insert_before bulk load Behind redb-experimental-cursor (default off). Production binary stays on sorted insert(). CI unit tests run both feature settings. * rust volume: exercise insert_before across leaf splits Replace the 5-key cfg clone with a 4000-key reverse-order rebuild so CursorMut::insert_before hits page splits. CI runs the feature only on storage::needle_map unit tests. |
||
|
Eliah RusinandChris Lu
|
b6690cfbc8 |
rust volume: drop the extra redb read transaction on put/delete (#11204)
* rust volume: insert redb rebuilds in needle-id order Unlink the .rdb before create (create does not truncate). Collapse last-write-wins, then insert live keys sorted so 4.2.0 packs leaves. * rust volume: rebuild redb from a BTreeMap and clear leftover keys Peak rebuild memory is one ordered map instead of HashMap + Vec + stable-sort scratch. Unlink stays best-effort: if it fails, retain clears the leftover table before sorted insert. Compute idx metrics before the write so a read error does not unlink a committed .rdb. * rust volume: drop the extra redb read transaction on put/delete put uses insert()'s previous value. delete gets then inserts the tombstone in the same write transaction. Truncate the .idx row on any failed redb write after the append. * rust volume: unpack redb blobs through packed_to_needle_value save_to_idx, ascending_visit, and collect_entries used the same length-check copy as get. Route them through the helper so a wrong-length value is absent everywhere, not a panic. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Eliah Rusin
|
94b10c006d |
rust volume: insert redb rebuilds in needle-id order (#11202)
* rust volume: insert redb rebuilds in needle-id order Unlink the .rdb before create (create does not truncate). Collapse last-write-wins, then insert live keys sorted so 4.2.0 packs leaves. * rust volume: rebuild redb from a BTreeMap and clear leftover keys Peak rebuild memory is one ordered map instead of HashMap + Vec + stable-sort scratch. Unlink stays best-effort: if it fails, retain clears the leftover table before sorted insert. Compute idx metrics before the write so a read error does not unlink a committed .rdb. |
||
|
Chris Lu
|
15e4da65f7 |
volume: avoid read-only replica write targets (#11195)
* master: carry replica read-only state in volume lookups * volume: refresh writable replica targets * volume: preserve read-only replicas for deletes * master: propagate read-only delete capability * volume: target delete-capable replicas * volume: honor configured HTTPS for replica deletes * volume: reject insecure delete authorization forwarding * master: broadcast delete capability changes * volume: align Rust replica routing * http: protect credentialed replica redirects * master: preserve digest compatibility for delete capability * volume: propagate read-only state in short heartbeats * volume: report changed short volume state * http: guard TLS client redirects * master: announce mounted volume read-only state * volume: replace changed identity deltas * master: replace incremental volume layouts in order * master: keep moved volume lookup available * volume: announce read-only mounts |
||
|
Eliah RusinandClaude Opus 5
|
ade4bdf9e6 |
rust volume: stop a tier move whose caller has gone (#11192)
Both tier-move handlers run in a detached tokio::spawn and report progress through a closure that returns (), with the send result discarded. Nothing observes the caller leaving, so an abandoned move uploads or downloads the whole .dat anyway and then commits the transition. Go aborts both. Its progress callback returns `stream.Send`'s error, which surfaces out of the reader in s3_upload.go:99 and the writer in s3_download.go:84 and fails the transfer, so the volume info is never rewritten. The Rust port dropped that by typing the callback as FnMut(i64, f32) with no result. Give the callback Go's signature -- FnMut(i64, f32) -> Result<(), String> -- and abort when the caller's channel is closed. Checked on every part rather than only where progress is reported, since the report is rate-limited to one a second and would miss a caller that left in between. A merely full channel is a slow reader, not a departed one, so only TrySendError::Closed counts as cancellation. Two consequences of aborting mid-transfer that the old code never had to handle: - upload_file now aborts the multipart upload when the transfer fails. An abandoned multipart upload does not show up in an ordinary object listing but still accrues storage charges until a lifecycle rule reaps it, and cancellation makes that a routine path rather than a rare one. - The tier-down handler removes the partial .dat. download_file pre-allocates the destination to the object's full size, so an aborted download leaves a file of the right length and the wrong content -- and this handler refuses to run at all when a local .dat exists, so leaving one wedges every retry on "already on local disk" and a restart would load the sparse file as the volume's data. There is deliberately no check between a finished transfer and the bookkeeping that follows. Once the object is in S3, or the .dat is on disk, that bookkeeping is what makes the state consistent; stopping there would leave an object paid for and referenced by nothing, or a complete local .dat the volume still calls remote. Go does not gate there either -- its callback only runs during the transfer. Claude-Session: https://claude.ai/code/session_0122W3eqt6gmLUMxmRoZdPAb Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
4b41329e12 |
rust volume: stop a VolumeCopy whose caller has gone (#11188)
* rust volume: stop a VolumeCopy whose caller has gone VolumeCopy runs its copy in a detached tokio::spawn and reports progress with the send error discarded, so nothing observes the client leaving. When the caller cancels the RPC -- which weed-admin's batch balance does routinely, starting far more copies than it finishes -- the server streamed the whole volume from the source, wrote it to disk, and mounted it. The destination is then left holding a volume nobody took delivery of: its index cache is never reclaimed, and under replication=000 one volume id ends up on two servers, both writable, which concurrent writes can diverge. Three checks now reach the task: - Every chunk in copy_file_from_source, via the sender's is_closed(). This is the one that matters in practice. The first progress report is 128MB in, so for a smaller volume -- the ordinary balance move -- no send ever happens and its result says nothing; only the closed channel does. The sender is passed for the .idx and .vif copies too, with reporting gated separately, so those phases notice as well. - The throttle sleep, which for a throttled copy runs for seconds at a time, now races the sender's closed() instead of being slept through. - Immediately before mount_volume, and once at the top of the task. Cancellation surfaces as an ordinary Err(Status::cancelled), so it lands in the existing error branch that already removes the partial .dat/.idx/ .vif and the .note. That branch also logs now: the error otherwise went to a channel nobody was reading, leaving the operator with the balancer's "delete that copy, then re-run the move" and no cause. This also clears the stranded read-only sources reported on the issue. They are downstream of the orphan mount, not a separate defect: LiveMoveVolume's cleanup probes the target before undoing the freeze (volume_move.go:95, "the server can finish the copy and mount the target even when the client loses the stream"), and when it finds a mounted copy it cannot attribute, or cannot delete, it deliberately keeps the source readonly rather than risk two writable replicas -- the messages at volume_move.go:110 and :123. With nothing mounted on the target the probe reports clean and the freeze is undone. On Go parity: the progress send result is honoured here too, matching `return false` in volume_grpc_copy.go. But that report is Go's only abort signal, and measured against a 120MiB volume -- above the throttler's activation threshold, below the 128MiB report interval -- a Go destination mounts an abandoned copy as well. The issue's premise that Go aborts holds only above the report interval. The Rust side now stops in both cases; the Go behaviour is worth its own issue. Tests: the integration test runs against both implementations and is green on Go, red on Rust before this change. Its 192MiB fixture is sized for two separate constraints, documented at the fixture: IoBytePerSecond is a no-op below ~100ms of wall clock (64MiB copies in ~110ms on a tmpfs loopback cluster), and the payload must exceed the 128MiB report interval for the Go leg to pass at all. The two Rust unit tests cover what the integration test cannot reach: cancellation detected with no progress report at all, and the cleanup of the partial files plus the .note. Fixes #11186 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122W3eqt6gmLUMxmRoZdPAb * rust volume: surface VolumeCopy cancellation as Status::cancelled copy_file_from_source returned Result<_, String>, so the per-chunk cancellation path -- the one that matters in practice for volumes below the 128MB report interval -- was wrapped to Status::internal at the call sites. The spawn logging branch then classified it as a generic failure instead of the intended "abandoned by caller", defeating the logging change in the same PR for the case that occurs most often. Return Result<_, Status> from copy_file_from_source: Status::cancelled for caller-gone, Status::internal for the existing errors. Drop the .map_err(|e| Status::internal(e)) at the three call sites. The unit test now asserts the code is Cancelled, not just the message text, so the classification is locked in. * rust volume: close cancellation gaps in VolumeCopy Address two review findings on the same PR: 1. Roll back a mount that races a departing caller. The pre-mount is_closed() check cannot close the window between the check and mount_volume: if the receiver drops in that gap, the volume mounts and the final tx.send(Ok(...)) fails, but its error was discarded (let _ =), so the task returned Ok(()) and the error branch never ran. The destination then held an orphaned mounted replica — the exact defect this PR prevents. Fix: track a mounted flag. The final send now checks its result; on failure it returns Status::cancelled, and the error branch calls store.delete_volume (which unmounts AND removes the files) when mounted is true, instead of only unlinking. 2. Observe cancellation while awaiting the source stream. The per-chunk is_closed() check only runs after stream.message().await returns. A stalled source (slow disk, partition, GC pause) never delivers a chunk, so a caller that has already left cannot preempt the read: the task, the source connection, and the partial files (including the .note) all outlive the caller indefinitely. Fix: race stream.message() against progress_tx.closed() in a tokio::select!, so a departing caller preempts a stalled source. Adds test_volume_copy_after_mount_cancellation_rolls_back_mount to cover the after-mount rollback path. cargo test --release green (497 + 5 + 1 + 28). * rust volume: keep remote data on after-mount rollback, race RPC startup Two review findings on the after-mount rollback added in |
||
|
|
b82cb05d71 |
rust volume: checkpoint the redb index durably every 1000 writes (#11182)
* rust volume: checkpoint the redb index durably every 1000 writes
Every put and delete on a redb-backed volume committed with
Durability::None and nothing ever committed durably, on the theory that
the .idx file is the source of truth. redb, however, keeps an entry in
its transaction tracker for every non-durable commit and cannot recycle
pages that were on disk at the last durable commit until a durable one
happens. With no durable commit for the life of the process, both grew
with every write, and .rdb files could bloat toward double size after a
restart (#11179, the hash-table rehash stacks in the memleak output).
The needle map now counts non-durable commits and reports when a
checkpoint is due; the volume takes it, data first: flush the .dat, then
the map fsyncs the .idx and commits redb durably, recording in the same
transaction how much of the .idx the table reflects. A checkpoint makes
the index durable, so the bytes it points at must be down before it, or
after a power loss the index would reference past the end of the .dat
and the volume would load read-only. A failed .dat flush skips the
checkpoint; it is retried on the next write.
Volume::close() now closes the needle map instead of only syncing it,
and the redb map's close() takes the same checkpoint. Before, a clean
shutdown left the table durable (redb flushes on drop) but the recorded
.idx size stale at its load-time value, so the next load replayed every
entry written since load on top of the counters.
On load, the redb map's counters now come from the whole .idx history,
the way Go's LevelDB map rebuilds them (newest entry first, with a bloom
filter of seen keys), instead of from the table's final state. Both the
reuse and the full-rebuild path use it, so overwritten and deleted bytes
keep counting as garbage across restarts, and the incremental replay of
the .idx tail only touches the table, which makes it idempotent whether
or not the table is ahead of the recorded .idx size.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK
* rust volume: skip redundant .idx fsync on checkpoint after flush_idx
On the fsync=true write path, flush_idx() already fsyncs the .idx before
maybe_checkpoint_index() runs, so the checkpoint's own sync() fsyncs the
same file a second time for nothing. Thread an idx_already_synced flag
from the volume through maybe_checkpoint_index into checkpoint(sync_idx):
when it is true the checkpoint skips its .idx fsync and only does the
durable redb commit. The delete path and close() still sync (they have
not flushed the .idx beforehand).
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: saturate writes_since_checkpoint to prevent u32 overflow
If checkpoints keep failing (e.g. a persistent .dat flush failure whose
error is not EIO and so does not mark the volume read-only), the counter
increments on every write with no upper bound and wraps at ~4.3 billion.
Use saturating_add so it pins at u32::MAX instead, which keeps
checkpoint_due() true and retries on every subsequent write.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: only update max_file_key on live entries in idx metric rebuild
metrics_from_idx called maybe_set_max_file_key on every entry including
tombstones, but the live on_put path only calls it for puts and on_delete
never does. A tombstone always has a preceding put for the same key that
already set max_file_key, so the result is the same today; restricting it
to live entries makes the parity with the live path exact and self-evident.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: advance idx_file_offset only after redb commit succeeds
put() and delete() appended to the .idx file and advanced idx_file_offset
before committing to redb. If the redb commit failed, the offset included
the orphan row that redb doesn't reflect. A later checkpoint would record
that offset as "the table reflects up to here," and the reload would skip
the orphan row entirely — the entry becomes permanently unindexed.
Move the idx_file_offset increment to after the successful redb commit.
The .idx file still has the orphan row (append-only), but idx_file_offset
stays behind it, so the next checkpoint records the smaller offset and
the reload replays the orphan row back into redb.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: skip index checkpoint on close when .dat sync fails
Volume::close() discarded the .dat sync_all() result and always
checkpointed the redb index. If the .dat sync failed, the checkpoint
made the index durable with entries that may point past the unflushed
.dat tail, and a power loss would leave the volume read-only on reload
(the max_needle_end check fires).
Check the .dat sync result: on success, checkpoint as before; on
failure, call close_without_checkpoint() — sync the .idx and drop the
writer without a durable redb commit. META_IDX_SIZE stays at the last
successful checkpoint, so the reload replays the uncheckpointed tail
(redb still flushes on drop, but without recording idx_size).
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: schedule checkpoints on every index mutation path
maybe_checkpoint_index was only called from do_write_request and
do_delete_request. put_needle_index and write_needle_blob_and_index
also call NeedleMap::put, which increments writes_since_checkpoint,
but neither triggered the checkpoint. Through those paths the counter
could grow past the interval without ever being satisfied, leaving
non-durable redb transaction state until close().
Add maybe_checkpoint_index(false) after the successful nm.put in both
methods. The .dat flush inside maybe_checkpoint_index covers the blob
write in write_needle_blob_and_index; put_needle_index pairs with a
prior write_needle_blob, so the flush covers that too.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: truncate orphan .idx row on failed redb commit
Commit
|
||
|
Eliah RusinandClaude Fable 5.1
|
3f9b05946b |
rust volume: bound the redb index cache per volume by --index tier (#11180)
The Rust volume server opens one redb database per volume and built each with redb's defaults, which give every database a 1 GiB page cache (0.9 GiB read cache + 0.1 GiB write buffer). With hundreds of volumes behind one disk the process-wide ceiling was volumes x 1 GiB: memory grew in proportion to the pages traffic touched, never shrank when traffic stopped, and hosts running many instances were OOM-killed under bulk ingest. redb, redbMedium and redbLarge were also treated identically, so the "memory~performance" tiers did nothing. Size the cache per tier instead: 4, 8 and 16 MiB per volume, mirroring the Go server's 3/6/12 MiB LevelDB block cache + write buffer. Thread the budget through RedbNeedleMap::new/load_from_idx so every open path (create, reuse, full rebuild) uses Database::builder().set_cache_size. Fixes #11179 Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Eliah RusinandClaude Fable 5.1
|
adbee9452a |
rust volume: bump redb 3.1.3 -> 4.2.0 (#11181)
No source changes: the API surface the needle map uses (Database create/open/builder, set_cache_size, set_durability, tables, iterators) is unchanged and the on-disk format is still v3, so existing .rdb files open as-is. The 4.0.0 breaking changes (Drop on AccessGuardMut, removal of the Legacy type) do not touch this crate. Relevant to the redb-backed index (#11179): - 4.1.0: optimizes cache usage and memory usage; ~1.5x faster writes. - 4.2.0: Durability::None commits ~2x faster; pages freed by a durable transaction are reused by the very next one; a crash-recovery fix for a crash during repair of an earlier crash. Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris Luanddevin-ai-integration[bot]
|
811b8b5734 |
make the remote-mount cache wait configurable per mount (#11168)
* add a per-mount cache_wait_ms to the remote storage mount mapping A read of an uncached remote-only object waits on a hardcoded size tier before it can fall back to the origin, so every ranged read of a large remote-only object pays that wait. Carry the wait in the mount mapping so it can be tuned, or set to zero, per mount. * resolve the cache wait of an uncached remote-only read from its mount The wait came only from the object size, so an operator could not trade cache hits for time to first byte. Both read paths now resolve the mount covering the object and let its cache_wait_ms replace the size tiers. * read straight from the remote when a mount waits zero for its cache A mount used as a streaming source pays the cache wait on every ranged read of an object too large to finish caching, and the caching itself is wasted work. A zero wait now skips the cache call, so both read paths go to the origin immediately. * let remote.mount set the cache wait of a mount remote.mount -cacheWait=0 turns a mount into a streaming source, and any other duration trades cache hits against time to first byte. * keep the size based wait for a version-specific read A read pinned to a version cannot fall back to the origin, since the mounted remote only holds the current key, so a mount that opts out of caching would leave it on the 503 retry loop forever. * let the operator allow a remote-only read to dial an internal endpoint The remote-mount read paths in the filer and the S3 gateway always refused an endpoint resolving to a loopback or private host, so a mount backed by an internal S3 could never be read from its origin, only through the local cache. Both now take the allowance the volume server already has, still off by default. * skip the background cache of a mount that waits zero for its cache GetObjectHandler kicks off caching for every remote-only read, so a mount serving as a streaming source kept downloading whole objects even though no read ever waited for them. * cover a zero cache wait end to end The read has to reach a real origin, so the harness also opts the filer and the S3 gateway into dialing the loopback remote it already allows for the volume server. * resolve the S3 cache wait once so the background cache follows it too The background cache that GetObjectHandler starts read the mount on its own, so it skipped a version-specific read that the foreground path still waits for. Both now ask the same resolver. * answer 404 when the origin of a zero-wait read is gone Metadata can outlive the object it points at, and with no cache to fill the read would sit on the 503 retry path forever. The remote backends already report a missing object as ErrRemoteObjectNotFound. * open the origin at write time for a multipart range Every part of a multipart Range is prepared before any is written, so opening eagerly would hold one origin connection per part and leak the ones already opened when a later part fails to open. * reject a cache wait shorter than a millisecond The mapping stores milliseconds, so -cacheWait=500us truncated to zero and silently turned caching off instead of waiting. * restore the doc comment of cacheRemoteObjectForStreamingWithShortTimeout Extracting the wait resolver left its comment on the new function. * stat the origin before committing a multipart range Opening at write time keeps no connection through the preparation, but it also moved a failure past the point where the multipart body picks the response status, so a gone origin truncated a 206 instead of answering 404. One stat up front puts the status back. * stat the origin once per request Every part of a multipart Range is prepared on its own, so the preflight ran once per range instead of once per read. * map Azure and GCS stream not-found to ErrRemoteObjectNotFound ReadFileAsStream on Azure and GCS returned provider-specific not-found errors instead of ErrRemoteObjectNotFound, so a zero-wait read of a deleted object was misclassified as a transient cache failure and retried indefinitely. Map BlobNotFound and ErrObjectNotExist the same way StatFile already does. * Update weed/remote_storage/gcs/gcs_storage_client.go Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
f79d83abf4 |
volume: expire TTL volumes whose only traffic is deletes (#11167)
* volume: count a TTL volume's age from its last write, not the .dat mtime A delete appends a tombstone needle and vacuum rewrites the .dat wholesale, so the file's mtime moves without any write ever landing. The loader read lastModifiedTsSeconds back from that mtime, so every restart of a volume taking delete traffic re-armed expired() for another full TTL: an overwrite-heavy collection kept growing until it hit the max-volume cap. Recover the clock from the newest .idx entry that is not a tombstone and read that needle's append timestamp, falling back to the mtime when no write is recoverable. Only TTL volumes pay for the scan. Fixes #11160 * volume: count the .vif destroy time from the last write too ExpireAtSec is what an EC volume is reclaimed on, and it was recomputed as now+TTL every time the .vif was written. A read-only mark, a tier upload or an EC encode therefore handed an already expiring volume another full TTL, the same way the .dat mtime did. Derive it from the volume's last write, falling back to now for a volume that has not taken one yet so a fresh volume is not born expired. * volume: mirror the last-write TTL clock in the Rust volume server Same recovery as the Go loader: scan the .idx backwards for the newest entry that is not a tombstone and take that needle's append timestamp, leaving the clock on the .dat mtime when no write is recoverable. * volume: mirror the last-write destroy time in the Rust volume server Both .vif writers and the EC encode computed ExpireAtSec as now+TTL, the same way Go did, so the destroy time moved every time the sidecar was rewritten. Route all three through the volume's last write. * volume: report the .dat mtime in the Rust heartbeat, like Go does The Rust server reported its TTL clock as ModifiedAtSecond while Go reports the .dat mtime. The shell's quiet-period gates (volume.tier.move, volume.delete_empty) read that field as "last touched", which a delete has to count towards even though the TTL clock deliberately ignores it -- and with the clock now recovered from the last write, the two drift further apart. * volume: take the newest write by timestamp on a vacuumed volume The reverse .idx scan trusted position, which holds only while the .dat is append ordered. Vacuum rewrites it in key order, and since an overwrite keeps its original key, the highest-key survivor is not necessarily the newest write -- the recovered clock could land up to a TTL early and take the volume with data still inside its TTL. A volume that has been vacuumed (CompactionRevision > 0) now takes the maximum append timestamp over a bounded window of write entries instead. An append-ordered volume still answers in one read. * volume: never guess a vacuumed volume's last write, and resolve wrapped offsets Two holes in the reverse scan, both from review: A vacuumed volume's writes are ordered by key, so any of them can hold the newest timestamp. Reading a capped window sampled the highest keys, which could still miss a recently overwritten low-key needle and expire data inside its TTL. The scan now covers every write a vacuumed volume indexes, and a volume too large to scan keeps the .dat mtime rather than report a partial maximum -- late is recoverable, early is not. A .dat past MaxPossibleVolumeSize wraps the offsets in its .idx, so reading a timestamp at the unwrapped offset picks up an unrelated needle. Resolve the entry against the needle header first and retry one volume size in, the way doCheckAndFixVolumeData already does. * volume: drop GitHub issue references from TTL comments |
||
|
Chris LuandDevin
|
24b8646ec3 |
volume: let evacuation proceed on a server in maintenance mode (#11145)
Maintenance mode exists to fence a volume server so it can be evacuated without taking new writes (#7977), but the gate added in #8115 also rejected the RPCs evacuation issues against the source: VolumeMarkReadonly (the first step of every move, and the failure reported in #11066), VolumeDelete (the last step), and VolumeEcShardsDelete (the last step for EC shards). volumeServer.evacuate, volume.move and ec.balance therefore all failed on exactly the server they were meant to drain. Those three RPCs only remove data or restrict the server further, the same class as DeleteCollection and the unmount RPCs that were never gated, so they are exempted from the maintenance check in both the Go and Rust volume servers. Everything that adds data or reopens the server for writes (AllocateVolume, WriteNeedleBlob, BatchDelete, VolumeCopy, ReceiveFile, EC generate/copy/rebuild, vacuum, tiering, VolumeMarkWritable) stays blocked. A side effect is that scrub can now fence broken volumes readonly on a server already in maintenance. Fixes #11066 Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
8112f2733a |
filer: batch exact lookup RPC, authoritative volume lookup, VolumeDelete status codes (#11122)
* storage: make DeleteVolume errors inspectable with errors.Is An absent volume wraps ErrVolumeNotFound and an only-empty refusal now wraps ErrVolumeNotEmpty with %w instead of %v, so callers no longer have to match on the message. Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm * volume server: return NotFound and FailedPrecondition from VolumeDelete An absent volume maps to codes.NotFound and a non-empty volume under only_empty to codes.FailedPrecondition, so a caller retiring a volume can treat NotFound as already done. The store message is kept in the status description because the EC empty-replica sweep still matches on it. Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm * wdclient: add LookupVolumeIdsAuthoritative Bypasses the vid map and asks the provider directly, for callers where a stale positive location is unsafe. Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm * filer: add LookupDirectoryEntries batch lookup RPC Up to 4096 exact-path lookups in one call, resolved concurrently with results in request order, plus one deduplicated location lookup for every volume the returned entries reference and per-fid read tokens when the filer signs reads. unavailable_volume_is_miss lets cache-style callers take an entry whose volume has no live location as a miss, resolved against the master rather than the filer's location cache. Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm * filer: test that an expired file entry is deleted on read Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm * filer: test that AssignVolume and CreateEntry resolve the same TTL rule Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm * master: refuse partial lookups while warming up LookupVolume returned Unavailable during warm-up only when every requested volume was missing. A batch mixing a reported volume with one whose server has not reconnected yet came back as a partial answer with a per-volume not-found, which a caller treating the master as authoritative reads as gone. Any not-found during warm-up is now Unavailable, which callers already retry. Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm * filer: build batch test requests instead of copying a proto message Copying a generated message copies its internal mutex, which go vet's copylocks check rejects. Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm * filer: match ErrNotFound with errors.Is and state the miss rule's contract A wrapped not-found from the store would otherwise be reported as an error rather than a miss. The comments now say why a nil location map is the only sign of an unanswered lookup: the provider returns nil when it got no answer and a populated map, with unserved volumes reported as errors, when the master did answer. Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm * volume server: map absent and non-empty VolumeDelete errors in the Rust server Matches the Go server: an absent volume is NotFound and an only_empty refusal is FailedPrecondition instead of Internal, with the messages the EC empty-replica sweep matches on. Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm * filer: test that a malformed entry keeps its error outside cache mode Same test file as the enterprise tree, so the next sync sees one version. Claude-Session: https://claude.ai/code/session_01T4MEV3ETqFFKN46Uu2ZrUm |
||
|
Chris Lu
|
31fb46f693 |
volume: rebuild a missing .idx from the .dat (#11115)
* volume: rebuild a missing .idx from the .dat Pointing -dir.idx at a directory that holds no index aborted the whole volume server: checkIdxFile found no .idx and load() called glog.Fatalf. Every row of the index is derivable from the .dat, so walk it in append order and write the index back, which reproduces byte for byte what the server's own writes had left in the old directory. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: keep the index co-located with the data in the Rust server Go's load() drops back to the data directory when an .idx already sits beside the .dat, so naming a --dir.idx does not strand a pre-existing index. Rust had no such adjustment: it opened the new directory with create, and the volume came up on an empty index with every needle invisible. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: rebuild a missing .idx from the .dat in the Rust server Mirrors the Go side. Rust did not abort on a missing index the way checkIdxFile did; it opened the new directory with create and mounted the volume on an empty index, so every needle read as missing while the .dat still held the data. Walk the .dat in append order and write the index back, byte for byte what the server's own writes had left behind. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: stop the idx rebuild at a zero-padded .dat tail An all-zero needle header is unwritten space, not a record. Go's .dat walk keeps reading past it and would index a truncated data file's tail as millions of needle 0 rows; the Rust walk already stops there. Stop the Go rebuild at the same place. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: create the -dir.idx directory when it does not exist Rust's DiskLocation creates the index directory as it takes it; Go only resolved the path, so naming a directory that does not exist yet left every volume unable to open or rebuild its index and took the server down. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: stop the idx rebuild at a torn .dat record A crash between writing a needle's header and its body leaves a record whose declared size runs past the end of .dat. Indexing it puts a row in the .idx that points at bytes that do not exist, which fails every read of that needle and trips the past-EOF check on the next load. Stop at the first record that does not fit, in both servers. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: stop the idx rebuild at a negative-size header A corrupt header whose size field is negative makes the .dat walk advance backwards: NeedleBodyLength adds the negative size, so the next offset is lower than the current one. The Go walk then reads at a negative offset and the rebuild fails, which puts the volume server right back to exiting at startup; the Rust walk seeks past EOF and truncates the index instead. A negative size is never a record, so stop there. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: skip a volume whose index cannot be rebuilt, do not exit glog.Fatalf calls os.Exit(255), so a rebuild that could not write -- a full or read-only index directory -- put the server right back to dying at startup for one bad volume. Return the error instead: loadExistingVolume logs it and skips that volume, which is what the remote-volume branch just above already does and what the Rust loader has always done. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: create the index directory from the rebuild too The rebuild is the first thing to write into a fresh -dir.idx, and it runs before the loaders that create the directory on their way to opening .idx. Create it in both rebuilds so the ordering does not matter. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * ci: let codespell past the sme variable in the mount tests weedfs_stream_mutate_error_test.go names its *streamMutateError local sme, which codespell reads as a misspelling of same/some. It is an identifier, so exempt it beside the other variable-name entries. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ |
||
|
|
7620e96171 |
expose whether a volume replica is backed by remote storage, and prefer local replicas (#11105)
* expose whether a volume replica is backed by remote storage
Volume locations returned by lookups do not indicate whether a replica
has been tiered to remote storage. Readers cannot distinguish a local
replica from a remote-backed one, so they may hit a remote-backed
replica first even when a local replica is available.
Add DataInRemote to the lookup location message, populate it from the
master's volume info, and carry it through the wdclient vid map so
clients can prefer local replicas when resolving chunk locations.
* wdclient: prefer local volume replicas over remote-tier replicas on lookup
LookupFileIdWithFallback (and the publicUrl variant in FilerClient)
didn't honor the DataInRemote flag when shuffling URLs, so the
DataInRemote patch only took effect in LookupVolumeServerUrl. Apply
the same ReorderToFront(localUrls) to sameDcUrls/otherDcUrls so
non-remote replicas stay at the front, matching the existing vidMap
convention.
* wdclient: propagate DataInRemote across tier transitions on existing replicas
When a volume is tiered to remote storage or a remote-backed replica is
restored locally, the cached DataInRemote on the same volume-server URL
stayed at its old value because two pieces of state never updated:
* master_grpc_server.go only split newVolumes and (already-tracked) volumes
into NewVids vs RemoteVids. ChangedVolumes went straight to NewVids, so
the broadcast announced the re-classified volume as a fresh arrival and
the client had no way to tell whether its existing cache was stale.
* vid_map.addLocationToMap early-returned when an entry already had the
same URL. A tier transition reports the same URL with DataInRemote
flipped, so the cached entry stayed at the old classification.
Wire both sides together: ChangedVolumes now go through the same IsRemote
split as newVolumes, and addLocationToMap replaces the existing entry in
place when the URL matches but DataInRemote has changed. The server
reference key only depends on URL/grpc port, so the refcount does not
move across the flip.
Adds vid_map_remote_transition_test.go covering the local->remote and
remote->local paths so the in-place update and the cache-key stability
are pinned by tests.
* wdclient: prefer local replicas across data-center boundaries
The previous local-first ordering hoisted local URLs to the front of each
data-center bucket separately, then concatenated same-DC before other-DC.
That meant a same-DC remote replica could still be tried before an
other-DC local replica even though the local one would answer cheaply.
Reorder once across the full candidate list: concatenate same-DC and
other-DC first, then ReorderToFront pulls every local replica to the very
front while preserving the DC preference inside each tier. Apply the same
ordering in all four lookup paths so the cached vidMap, the
LookupFileIdWithFallback provider path, FilerClient.GetLookupFileIdFunction
(PublicUrl-preferred variant), and the deprecated filer.LookupFn all agree:
- weed/wdclient/vid_map.go (LookupVolumeServerUrl)
- weed/wdclient/vidmap_client.go (LookupFileIdWithFallback)
- weed/wdclient/filer_client.go (LookupFileId)
- weed/filer/reader_at.go (LookupFn)
Strengthen the existing local-first tests: vidmap_client_localfirst_test
now asserts both endpoints are present (not just the local one is first),
and slice_test asserts an exact match instead of accepting two orderings.
Add TestLookupFileIdWithFallbackGlobalLocalFirst to pin the cross-DC
ordering invariant: any local replica (same or other DC) precedes every
remote-tier replica; within each tier DC1 precedes DC2.
Add docstrings to ToVolumeLocations, ReorderToFront, LookupVolumeServerUrl,
LookupFileId, GetVidLocations, GetLocations, LookupFileIdWithFallback, and
updateVidMap so the touched lookup paths are described in one place.
* topology: broadcast tier transitions on existing replicas
When a volume replica is tiered to remote storage or restored locally, the
wdclient's cached DataInRemote went stale: every connected client kept
preferring a remote-backed replica over a freshly restored local one, or
demoted a freshly tiered remote replica. The fix in commit
|
||
|
Chris Lu
|
1996c6aec6 |
volume: open volume files with O_NOATIME (#11055)
* volume server: open volume files with O_NOATIME Nothing reads the atime of .dat, .idx, .sdx, or EC files, but every needle read still dirtied the inode: even relatime writes atime on the first read after each write, so an actively written volume paid a metadata write per read/write cycle, and strictatime mounts paid one per read. Open the serving handles with O_NOATIME, falling back to a plain open when the file belongs to another owner (EPERM). Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD * seaweed-volume: mirror the O_NOATIME volume file opens Same change as the Go volume server: serving handles for .dat, .idx, .sdx, .ecx, .ecj, and shard files open with O_NOATIME on Linux, with a plain-open fallback on EPERM. Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD * route the tier-down and recreate .dat opens through the no-atime helper Review caught the Rust tier-down swap opening the local .dat directly. The Go swapToLocalDatBackend and the zero-length read-only .dat recreate in maybeWriteSuperBlock had the same gap: all three install long-lived serving handles. Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD |
||
|
Guang Jiong LouandChris Lu
|
f740210235 |
get volume topology info without volume details (#11036)
* get volume topology info without volume details Signed-off-by: lou <alex1988@outlook.com> * master: rename VolumeListRequest.without_volumes to topology_only The field shapes the reply rather than selecting volumes, and it leaves out the ec shards too, which the old name denied. Match the message's *_only style and say what a master that predates the field does with it. Claude-Session: https://claude.ai/code/session_01QHnaNRgxnjzZsiz7WTFML5 * master: refuse topology_only combined with a volume selector A topology_only request that also names a collection or volume ids contradicts itself, and answering either half in silence surprises the caller. Answer InvalidArgument from both VolumeList and its stream, before the stream sends its header. Claude-Session: https://claude.ai/code/session_01QHnaNRgxnjzZsiz7WTFML5 --------- Signed-off-by: lou <alex1988@outlook.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
9bafeb6139 |
ec: refuse to mount a 0-byte shard file when the index has entries (#11030)
* ec: refuse to mount a 0-byte shard file when the index has entries The startup scan already skips (and eventually deletes) zero-sized shard files as residue of a failed copy, but the mount RPC path opens the file directly with no size check, so an explicit VolumeEcShardsMount over a truncated file registers a size-0 claim. A registered empty shard serves nothing while advertising ownership: with placement pinned to the owning disk, it would keep attracting re-copies to a file that was never valid. The one legitimate 0-byte shard is the empty volume's: encoding a volume with no live needles produces a 0-byte .ecx and 0-byte shards, and that mount must keep working (TestMountEcShards_EmptyEcxMountsSuccessfully). So the gate compares against the index: AddEcVolumeShard (Go) and EcVolume::add_shard (Rust) refuse a 0-byte shard file only when the volume's .ecx has entries. Go's AddEcVolumeShard grows an error return for this; the loader cleans up the refused shard and, when it just created the EcVolume, unregisters that too. The mount loop already collects non-ENOENT failures per disk and keeps scanning, so a sibling disk holding a real copy still wins. Regression tests in both trees: an empty shard beside an index with entries is refused and leaves nothing registered; an empty shard of an empty volume still mounts. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 * ec: release the duplicate shard when a mount retry re-loads it Review follow-up: AddEcVolumeShard keeps the existing shard and reports added=false for a shard this disk already registered, but the loader discarded that result, so every retried LoadEcShard leaked the duplicate it had just opened — an fd and a mount-gauge increment per retry. Release both and return the existing volume. Regression test pins the gauge. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 * ec: close the test DiskLocation instead of only its EC volumes Review follow-up: DiskLocation.Close() also stops the background goroutine NewDiskLocation starts; closeEcVolumes left it running for the rest of the test process. Both uses are this PR's own tests. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 * rust: unregister the just-created EcVolume when its first mount is refused Review follow-up: when the first mount of a volume rejects its shard (e.g. the new 0-byte-beside-nonempty-index refusal), the Rust mount path had already inserted the EcVolume and propagated the error without removing it — a zero-shard registration advertising a mount that serves no data while pinning the .ecx/.ecj descriptors (and, since placement's mounted tier keys off it, steering shard placement at this disk). Remove it on the way out, exactly as the Go loader already does; a volume that already holds shards keeps them (the RPC's first-error-aborts contract). Regression test covers both. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 * rust: skip already mounted shards on a mount retry Review follow-up: EcVolume::add_shard replaces self.shards[id] for a shard the volume already holds, and the mount loop then bumps the ec_shards gauge although the mounted count did not grow — gauge drift on every mount retry, and a serving fd swapped for no reason. Skip shard ids the volume already reports, mirroring Go's AddEcVolumeShard added=false handling. Regression test pins the gauge across a duplicate mount (unique collection label: the gauge is process-global and tests run in parallel). Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 |
||
|
Chris Lu
|
74b520113e |
ec: pin auto-selected shard placement to the disk that already owns the shard (#11029)
* ec: pin auto-selected shard placement to the disk that already owns the shard A multi-disk server legitimately mounts one EC volume on several disks, so FindEcShardTargetLocation's per-volume tiers tie at "mounted" and the free-shard-count tie-break decides — pointing at whichever disk is emptier, not at the disk that already holds the shard being placed. A re-copy of a shard the server already has (a retried ec.balance / ec.rebuild move) then lands on a sibling disk, and both disks register the same (volume, shard id): the shard is reported to the master from two disk ids, and which claimant serves reads or survives a later unmount/delete becomes an accident of Locations order. Add a tier above "mounted": a disk that already claims one of the shard ids being placed wins, ahead of the space filters too — re-copying in place needs no new shard slot, and a genuinely full disk should fail the write rather than silently split the claim. Applied to the Go selector and the VolumeEcShardsCopy auto-select (ReceiveFile refuses mounted EC volumes, so no claim can exist there) and mirrored in the Rust volume server. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 * ec: refuse a copy batch whose shards are already owned by different disks Review follow-up: ownership-aware selection ranks a mixed-owner batch (shard 0 on disk A, shard 2 on disk B — the legitimate multi-disk spread) into one destination, so the copy would still duplicate the losing disk's claim. No production caller sends such a batch (balance moves one shard, rebuild and encode copy shards the target lacks), so fail closed: report every owning disk via Store.EcShardOwnerDisks and refuse the copy with an error naming them, telling the caller to split per shard or pass disk_id. Go and Rust, with unit tests for the owner-reporting contract. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 |
||
|
Chris Lu
|
88c873ecd4 |
ec: uniform shard block layout (#10932)
* ec: uniform shard block layout An EC volume is striped as 1GiB blocks until less than one row remains, then 1MiB blocks, and consecutive blocks land on different shards. With ec.encode's -fullPercent 95 against the 30GiB default limit, ~30% of every volume sits in that 1MiB tail, so a 4MB filer chunk there is five stripes on five servers. New encodes now use one block per shard, sized ceil(datSize/dataShards) rounded up to 1MiB and recorded in the .vif (EcShardConfig.block_size, also carried by the .ecsum manifest). A needle now maps to one shard unless it is larger than the block or straddles a boundary. The chosen size equals the legacy layout's padded shard length for every input, so shard sizes, capacity math, and the shard-size credibility checks are unchanged; only the byte placement moved. Reads, decode, and scrub resolve the block sizes from the volume's .vif; absence keeps the legacy interpretation, so existing EC volumes read exactly as before. Rebuild is layout-agnostic. weed fix -ecx recovers the layout from the .vif, else the .ecsum sidecar, and with neither de-stripes under both candidate layouts and keeps the one that indexes more valid needles. Same change in the Rust volume server, which now also streams the encode in 256KB sub-batches like Go instead of allocating whole blocks, and computes the large-row count as shardSize/largeBlock to match Go on exact multiples. On a 26MB fixture both encoders produce byte-identical shards, and a Go-written .vif parses in Rust with the block size intact. * ec: resolve the rust ecx rebuild through the recorded layout The Rust rebuild path regenerated a lost .ecx by scanning the logical .dat through a hand-rolled pure-1MiB striping, which was already wrong for legacy volumes with large-block rows and is wrong for any uniform volume with a block past 1MiB. Route the scan through locate_data with the .vif-recorded block size, the same mapping the read path uses. Also seed the new tests' random data instead of the deprecated global math/rand.Read. * ec: fail the Rust ecx rebuild on any shard read error A read error mid-scan published the entries collected so far as a successful .ecx, and read_at's byte count was ignored so a legal short read passed as complete — a truncated or failing shard could produce a silently incomplete recovery index. Exact-read semantics in read_from_data_shards, error propagation in the needle walk, and a truncated-shard regression test. * ec: fail the mount on an unreadable or malformed vif Both servers silently fell back to the legacy layout when an existing .vif could not be read or parsed. Every new encode records a positive uniform block size there, so the fallback mounted the same shards with legacy offset math and could return wrong data. Absent stays legal (legacy volumes predate the sidecar), and a zero-byte stub still reads as absent (Go's MaybeLoadVolumeInfo convention, now mirrored in Rust); a present-but-unreadable or malformed .vif fails the mount instead. * ec: bound the reconstruct fan-out of one needle's intervals A degraded interval fans out a read to every reachable shard location, each with a buffer the size of the interval. Reading a needle's intervals in parallel multiplied that by the interval concurrency: a needle spanning 8 blocks could hold 8 x MaxShardCount remote reads and buffers at once, where the sequential version peaked at MaxShardCount. Give each needle a single reconstruct budget its intervals share, held for the buffer's lifetime, so separate reads stay independent but one read cannot multiply its own fan-out. * ec: drop the duplicated shard-size formula calculateExpectedShardSize reimplemented the padding rule that UniformBlockSize already owns — TestUniformBlockSizeMatchesLegacyShardSize asserts the two agree for every input — so a change to the rule would have had to be made in both. Defer to the helper, keeping the historic answer for an empty .dat. * ec: resolve the shard block layout from whatever records it Four places still answered the layout question by inference when a record of it was available, or accepted an answer that was not one: - A mount with no .vif defaulted to the legacy layout; the bitrot sidecar records the same config at encode time, so take it when present, as weed fix -ecx already does. The vif itself is now parsed once per mount rather than twice. - The Rust ecx rebuild derived its row count from the padded shard extent, which under the legacy layout reads a shard that is an exact large-block multiple as one row too many. Pass the encode-time .dat size from the .vif and keep the extent as the fallback. - weed fix -ecx read the block size outside the EC-config guard (collapsing the unknown sentinel into a definitive legacy), only wrote the recovered layout back when the .vif was absent rather than unusable, and broke a scan tie by candidate order instead of the documented reach. - The uniform layout tripped writeDatFile's large-block ambiguity guard, which cannot apply when the large and small blocks are the same size. * ec: give the index-recovery tests a parseable vif The fixtures wrote the literal bytes "volinfo" as the source .vif and the recovery copies it verbatim, so the receiving server then mounted the volume from a .vif it could not parse. That used to pass by silently defaulting to the legacy layout; a mount now refuses a vif it cannot read, which is what the tests were exercising all along without meaning to. * ec: validate the layout a vif records, not just its syntax Review follow-ups on the mount-strictness change: - A .vif can parse and still record a block size no encoder could have produced (negative, or not a whole number of small blocks). Both servers took it and mapped every read through it. ValidateBlockSize / the Rust mirror now refuse the mount, the same way an unparseable vif does; 0 stays valid as the legacy two-tier layout. - The bitrot-sidecar fallback accepted parity_shards == 0 and summed the counts in their own width, so values near the ceiling wrapped past the MaxShardCount bound. Require both counts and sum in a wider type. - weed fix -ecx treated a config with only DataShards > 0 as usable, so a half-written .vif suppressed the recovery paths AND survived the rewrite. Require a complete, in-range config before trusting it. - Returning the vif-load error left the .ecx and .ecj descriptors open; repeated mount attempts on malformed metadata could exhaust them. * ec: refuse to act on a layout the metadata does not establish - The worker encode only logged a failed .vif write and skipped it in the distribution set, and treated the .ecsum write as best-effort. A worker whose disk filled after the much larger shards landed could still distribute, mount, verify shard inventory, and delete the source replicas — leaving holders with shards whose geometry nothing records. Both writes and both inclusions are encode success conditions now. - A generation-matching .ecsum that disagreed with the .vif geometry only disabled checksums in Go, and in Rust was not compared at all, so protection stayed On while reads used the other layout. Both files record the layout their generation was encoded with, so a disagreement now fails the mount. * ec: reject an invalid recorded block size in weed fix -ecx A .vif with valid shard counts but a negative or unaligned block size was marked usable: a positive invalid value pinned the scan to a geometry that de-stripes to garbage, and a negative one ran the dual scan but left the invalid .vif in place afterwards. Validate it with the same rule the mount applies, and when it fails leave the layout unknown so the scan recovers it and the file is rewritten. * ec: validate the sidecar layout weed fix -ecx recovers from The .ecsum fallback was taken on DataShards > 0 alone, so a CRC-valid sidecar carrying the wrong generation, an incomplete ratio, or an unaligned block size would pin the reconstruction to one incorrect uniform-layout candidate instead of letting the dual scan decide. Require generation 0, a complete in-range ratio, and a valid block size; anything less leaves the layout unknown, which is the answer that still recovers by scanning. * ec: let only a genuinely absent sidecar choose the legacy layout With no .vif the bitrot sidecar is the only record of a volume's layout, and the mount fallback read a failed load, an unusable config, or a sidecar stamped for another generation as "assume legacy". A uniform generation-0 volume could therefore mount with legacy or another generation's geometry and answer reads with the wrong bytes. Present-but-unusable now fails the mount; only actual absence keeps the legacy defaults. Shared as EcShardConfigFromSidecar so every caller reads the sidecar the same way. * ec: treat a recorded-but-impossible layout as corruption, not as legacy - A .vif whose ecShardConfig is PRESENT but records an impossible ratio was answered with the default 10+4 and the legacy block layout, in both languages. That reads a uniform volume's shards at the wrong offsets and returns the wrong bytes. Only an entirely absent config still means "this predates the record"; a present one that cannot be true fails the mount. - The shard-count bound summed two uint32 counts as int, which wraps on a 32-bit build: 0x7fffffff + 0x7fffffff lands at -2 and slips under MaxShardCount. ValidEcShardCounts sums in uint64, and every EC call site that checked a recorded ratio now goes through it. * ec: rebuild on the geometry the sidecar records, and flag it when it disagrees The rebuild RPC passes BackgroundECContext, so RebuildEcFiles resolves the layout itself — and it resolved a missing or invalid .vif to the default 10+4 with the legacy block size. Two consequences: a 12+4 volume was reconstructed through a 10+4 matrix, which produces wrong bytes and never regenerates shards 14-15; and the chosen geometry then contradicted a valid uniform sidecar, which loadRebuildSidecar reported as BitrotOff — silently skipping the input and regenerated-shard checksum checks precisely when the volume had already lost its metadata. The layout now resolves from the bitrot sidecar (found across the server's disks, not just beside the base name) before falling back to the defaults, and a present-but-impossible ratio fails instead of being replaced. A sidecar that contradicts the chosen geometry is BitrotInvalid, which the existing unsafeIgnoreSidecar override still lets an operator push past. * ec: let the Rust rebuild read metadata off a sibling disk read_ec_shard_config searches only the location the rebuild writes into, so a volume whose .vif or generation-0 .ecsum sits on another of the server's disks resolved to the default 10+4 with the legacy block layout — the Rust half of the geometry-guessing the Go rebuild just stopped doing. It then reconstructs a custom-ratio or uniform volume through the wrong Reed-Solomon matrix and de-striping geometry. The rebuild now looks for the .vif in its own location and then each sibling, falls back to the generation-0 sidecar wherever that lives, and only defaults when neither exists anywhere. The encode-time .dat size the ecx rebuild needs is resolved the same way. * ec: resolve a rebuild's vif from every directory that may hold it RebuildEcFiles probed only <data-base>.vif. The caller knows the selected location's index directory and the sibling locations, but passed neither for metadata: additionalDirs carried shard directories only, and were searched for shards and the checksum sidecar. A split -dir/-dir.idx layout, or a disk holding only shards, therefore resolved a pre-sidecar custom-ratio volume to 10+4 and reconstructed through the wrong matrix — never regenerating shards 14-15. The caller now hands over the index and sibling directories, and the resolver probes the vif across all of them, matching what the Rust resolver already does for both the vif and the sidecar. * ec: make every rebuild consumer agree on the layout it resolved - The post-rebuild bitrot backfill re-derived the geometry from this directory's .vif alone and dropped the block size entirely, so a rebuild that resolved its layout from a sibling, the sidecar, or a uniform vif wrote a manifest describing a DIFFERENT layout — one later mounts reject, or that covers only the default shard count. The layout is resolved once now, through an exported ResolveRebuildECContext, and the rebuild and the backfill share that answer. - The Rust rebuild collected only each location's data directory, so a sibling's INDEX directory — where a split -dir/-dir.idx layout keeps .ecx/.ecj/.vif — was never probed, and a custom-ratio volume still resolved to 10+4 with the legacy layout. Both directories of every location are carried now, deduped against the rebuild's own. - A shard delivery can bring the checksum manifest with it, but the receive path only writes the file: a server that already had the volume mounted kept its resolved protection state (off) until a remount. The mount RPC re-resolves it once the shards it describes have been added. * ec: cover the rebuild's directory search with tests Reviewers flagged the sibling index directory twice, and the fix that closed it had no test of its own: the assembly sat inline in the rebuild handler, reachable only through a gRPC call against a populated store. Lifting it into rebuildSearchDirs / select_rebuild_location makes the rule assertable — a sibling contributes BOTH its data and its index directory, a shared index directory is listed once, and the rebuild's own data directory never repeats. Writing the Rust cases surfaced that the two implementations do not agree on where the rebuild's own index directory belongs, and both are right: Go's resolver takes a single directory list, so that directory has to be inside it, while Rust's takes the rebuild's data and index directories as their own arguments and would search them twice. The tests now state which contract each side is holding to, so neither drifts into the other's shape. Pure refactor otherwise; no behaviour change. * ec: search the index directory for the layout sidecar The Rust resolver looked for the generation-0 .ecsum in the rebuild's data directory and the sibling list, but not in the rebuild's own index directory — while the .vif lookup directly above it did, and Go's findBitrotSidecar has always checked both bases. On a split -dir/-dir.idx location that directory is where the metadata lives, and callers leave it out of the sibling list precisely because it is passed here separately, so nothing searched it. With no .vif anywhere the sidecar is the only surviving record of the layout. Missing it resolved a 12+4 uniform volume to 10+4 with the legacy striping — the test added here fails with (10, 4, 0) against the old code — and the rebuild then reconstructs through the wrong matrix and writes .ecx offsets that no reader can follow. * ec: let the rebuild see its own index directory The Rust rebuild takes a single flat directory list — the shape Go's RebuildEcFiles uses — so it cannot be handed the rebuild location's index directory separately the way the layout resolvers are, and the handler was passing the sibling list, which deliberately omits exactly that directory. On a split -dir/-dir.idx location that is where .ecx and .vif live, so the shard and index lookups could not see them. Go has always carried that directory in additionalDirs; this lines the two call sites up. * ec: let a config-free vif fall through to the layout sidecar A .vif that carries no ecShardConfig answers nothing about the layout, so it is no more informative than an absent one — but both trees treated its mere existence as the end of the search. Go went straight to the 10+4 legacy defaults without consulting the sidecar at all; Rust returned whatever ec_shard_config_from could make of a single directory. A 12+4 uniform volume with a legacy config-free vif therefore resolved as 10+4 legacy, and every read landed at the wrong shard offset. The sidecar lookup was also single-directory on both sides, while a split -dir/-dir.idx layout keeps .vif and .ecsum with the INDEX. Go's findBitrotSidecar has always taken both bases; the callers here passed only the data base, and the Rust bitrot resolver derived its path from the data base alone. Rust's layout resolver now takes a candidate directory list — data, index, then any siblings — and searches all of it, which also removes the early return that made the vif's presence decisive. load_vif_info_across_dirs reported `dir` even when load_vif_info had found the vif in `dir_idx`. Nothing reads that field today, so this changes no behaviour; it stops the next caller that resolves the rest of the volume's metadata against the answer from being sent to a disk holding none of it. Absence stays legal throughout: a volume with neither record is genuinely legacy. Present-but-unusable still fails the mount, now in the config-free-vif branch too. * ec: activate a delivered sidecar on every per-disk runtime A vid mounts as one EcVolume per disk, each with its own resolved protection state, but the post-delivery reload used the first-match lookup and so touched exactly one of them. The siblings kept reporting no protection until a remount — and since shard distribution deduplicates the metadata files onto the first target disk for a node, the runtime that got the .ecsum is not necessarily the one the lookup returns. Iterate every runtime instead, via a new FindAllEcVolumes and its Rust mut equivalent. Combined with each runtime now resolving its sidecar against its index directory as well as its data directory, a server sharing one -dir.idx across its disks activates all of them from the single delivered copy. The Rust volume server had no post-mount reload at all; it gets one here, matching Go. * ec: resolve the delivered sidecar across every EC metadata directory Reloading every per-disk runtime, added last round, did not by itself make the delivered manifest reachable. Startup mirroring copies .ecx/.ecj/.vif to every shard-bearing disk so each mounts self-contained, but deliberately not .ecsum, and a repair delivers exactly one copy. Each runtime was resolving against its own two directories, so every sibling of the disk that received the file kept reporting no protection however often it reloaded. Resolve one authoritative copy across every EC metadata directory instead of duplicating the file. Mirroring .ecsum would have to keep pace with a file that is rewritten as shards are repaired, and would not help the reported case at all: the delivery happens at runtime, and mirroring only runs at startup. The regression test pins both halves — a reload restricted to the volume's own directories still finds nothing, and the same reload given the server's metadata directories turns protection on. * ec: ask every directory before writing a TOFU baseline After a rebuild the opportunistic backfill asks whether this volume already has a checksum manifest, and answered from the data base alone. A split -dir/-dir.idx layout keeps the sidecar with the index, and a multi-disk server may keep it on a sibling, so an existing manifest read as absent. The consequence is worse than a missed read. On a false "no" the backfill writes a fresh sidecar at the data base from whatever the shards say right now — and the data base is the first candidate every resolver checks, so that TOFU baseline shadows the real manifest rather than sitting beside it. A shard that was silently corrupt gets blessed, and the record that would have caught it stops being consulted. FindBitrotSidecar exports the search the package already used internally, so the question is asked of the data base, the index base and the sibling disks — the same candidates the rebuild resolves its layout from. * ec: refuse a shard block size no encoder could have produced weed fix -ecx derived one from the raw shard extent, so a truncated or partially copied shard wrote a .vif that NewEcVolume then permanently refuses — the volume the tool was run to rescue could never mount again. An extent that is not a whole number of small blocks cannot have come from a uniform encode, so it is no longer offered as a candidate, and nothing unvalidated reaches the .vif. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: derive the .vif's dat size and block size from one measurement VolumeEcShardsGenerate stat'ed the .dat before the encode while WriteEcFiles stat'ed it again to size the blocks. A write landing between the two produced a .vif whose own two fields describe different files. WriteEcFiles now leaves both on the context, and fills a placeholder context in place so the caller can read them back. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: keep the source volume until every holder serves its shard layout The uniform layout rides in a .vif field older volume servers never knew: they discard it, mount the shards as legacy and return wrong bytes with nothing erroring, and the shard files are the same length either way so no other check notices. The upgrade order lived only in the release note. VolumeEcShardsInfo now reports the block size the holder actually serves, in both the Go and Rust servers, and the pre-delete verification refuses to drop the source unless every reachable holder echoes the one the shards were encoded with — while a rollback still exists. A server that predates the field answers 0, which is the negative answer. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: drop the rebuild's dead block-size parameters generateMissingEcFiles never reads largeBlockSize/smallBlockSize — Reed-Solomon reconstruction is layout-agnostic — so passing the legacy constants only advertised a layout the rebuild does not use. Also move UniformBlockSize's doc off ValidateBlockSize. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: warn about EC defaults only when the mount used them The "vif file not found, using defaults" warning fired even after the bitrot sidecar supplied a non-default layout, sending anyone triaging wrong bytes after the legacy layout the volume never mounted on. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: stat the distributed bitrot sidecar once The strict check re-stat'ed the file immediately before the stat that already gates inclusion, and a failed sidecar write now fails the encode outright, so the first could only fire on a deletion between the two lines. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: say what the reconstruct budget actually bounds A shard's buffer stays in bufs until its interval reconstructs, which is after the read that filled it released its permit, so the semaphore bounds round trips in flight and not retained bytes. Peak memory is the intervals reconstructing at once times the shards each reaches times the interval size. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * test: let the fake volume server report its delivered EC layout The pre-delete verification now asks each holder which shard block layout it serves, and a fake that always answered "unset" looked exactly like a volume server too old to know the field. Distribution ships the .vif to every holder alongside its shards, so read the layout back out of it as a real holder does. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 |
||
|
Guang Jiong LouandChris Lu
|
93666c90e9 |
filter by volume ids (#10983)
* filter by volume ids * master: carry the volume ids VolumeList asks about in one repeated field One id and a list of them ask the same question, so field 2 holds the list rather than standing beside a second field that supersedes it. Claude-Session: https://claude.ai/code/session_011qAmAdhrYvnzGkw7A9N4mP --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
3967ca23be | rust: cover the READS scrub reconstruction path (#11027) | ||
|
Chris LuandLisandro Pin
|
fcc2ea61d3 |
ec: scrub a volume through its parity data (#11006)
* Introduce a new `READS` scrub mode. `READS` performs a full volume scrub but, unlike `FULL`, it will attempt to reconstruct data for missing/damaged shard intervals from other shards in the cluster when necessary. The goal of this check is to ensure that EC volume contents _are readable by Seaweed_ even on a degraded storage state, by exercising parity data which is not read in `FULL` mode. This is useful not only to validate data is user-readable, but also to detect potential parity shard issues which may be difficult to pinpoint otherwise - particularly for older volumes lacking sidecar data, and hence unaffected by `CHECKSUM` scrubs. For regular volumes, this operation is equivalent to `FULL`. Example: ``` > ec.shard.unmount --volumeId=1 --shardId=0,3,11 --delete --apply Live shard topology for volume ID 1 (14 shards): 0@10.200.18.89:9001 1@10.200.18.89:9002 2@10.200.18.89:9003 3@10.200.18.89:9004 4@10.200.18.89:9005 5@10.200.18.89:9006 6@10.200.18.89:9007 7@10.200.18.89:9008 8@10.200.18.89:9009 9@10.200.18.89:9013 10@10.200.18.89:9010 11@10.200.18.89:9011 12@10.200.18.89:9012 13@10.200.18.89:9020 Will unmount + delete 3 shard(s): 0@10.200.18.89:9001 3@10.200.18.89:9004 11@10.200.18.89:9011 Unmounting shard 0@10.200.18.89:9001 for volume ID 1... Deleting shard 0@10.200.18.89:9001 for volume ID 1... Unmounting shard 3@10.200.18.89:9004 for volume ID 1... Deleting shard 3@10.200.18.89:9004 for volume ID 1... Unmounting shard 11@10.200.18.89:9011 for volume ID 1... Deleting shard 11@10.200.18.89:9011 for volume ID 1... All done! > ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=full using FULL mode Scrubbing 10.200.18.89:9002 (1/1)... Scrubbed 6 EC files and 1 volumes on 1 nodes Got scrub failures on 1 EC volumes and 1 EC shards :( Affected volumes: 10.200.18.89:9002:1 Affected shards: 10.200.18.89:9002:1:0 > ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=reads using READS mode Scrubbing 10.200.18.89:9002 (1/1)... Scrubbed 6 EC files and 1 volumes on 1 nodes ``` * ec: report the shards a READS scrub had to rebuild A READS scrub that recovers an interval was recording nothing, so a volume missing three shards came back clean and nobody repaired it. The unreadable shard is now recorded before the rebuild is attempted: READS reports the same broken shards as FULL and differs only in whether the needles themselves failed, which is the signal worth having - shards are gone, data is still there. forceDeletedNeedlesCheck now applies to READS as well, in the shell and in the RPC guard: it runs the same needle walk as FULL. Regenerated the proto instead of hand-editing it, so the pancis typo (which protoc-gen-go-grpc emits into eight other files here) and the header whitespace stay as generated. Mirrors into the Rust volume server, which also now honors force_deleted_needles_check rather than hardcoding it off. Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq * ec: answer a deleted needle from a READS rebuild as deleted #11020 gave the Rust recovery a deleted flag alongside its bytes, and it answers a deleted needle with no bytes at all. The READS scrub appended that empty answer, which does not compile against the new signature and, once it did, would leave the needle short and report the size mismatch as damage. Zero-fill the interval instead, the way the direct read beside it already does: the assembled needle then reaches read_bytes as the delete-state mismatch the walk already tolerates. Go takes the same branch off the flag its recovery returns, rather than discarding it. Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq --------- Co-authored-by: Lisandro Pin <lisandro.pin@proton.ch> |
||
|
Chris Lu
|
cd5013f116 |
Re-check an EC shard map a failed read has disproved (#11023)
* Re-check an EC shard map a failed read has disproved A read that fails against a cached location drops that shard from the map, which leaves it one short of complete -- and a map one short is trusted for seven more minutes. So a moment's trouble between volume servers cost minutes in which every read of that shard skipped the direct fetch and paid for a Reed-Solomon recovery instead, at DataShards times the memory and the peer load. Mark the map when a read disproves it, and re-check a marked map on the same eleven-second footing as one that never had enough shards to begin with. The mark clears on refresh, so it buys one prompt re-check rather than a master lookup per read. The tiers move into a helper; they were three overlapping conditions in one expression, and the reading of them was not obvious. Rust keeps the entry rather than dropping it -- a dead peer fails fast on the next attempt, and it was the freshness window, not the entry, hiding a shard that had moved. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Invalidate the location of an EC shard whose own read failed Recovery fans out to the other shards, so the one whose direct read just failed is the only location nothing ever invalidates: a shard that moved to another server was reconstructed on every read until the map's own window expired, up to thirty-seven minutes for a map still complete. Mark the map there too. The entry stays -- a moved shard's old holder fails fast, and the next refresh is seconds away. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Consume the stale mark before the lookup, not after A read that fails while the master is answering has disproved the very map that answer is about to install, and clearing the mark on the refresh's return swallowed it. Clear it where it is acted on instead. A lookup that then fails loses the mark, which costs nothing: the refresh time is only advanced on success, so the next read looks up regardless. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Judge the shard map and consume its mark in one critical section Reading the mark and clearing it were two separate acquisitions, so a mark raised between them was cleared by a refresh that had not seen it. In Go that gap was a few instructions; in Rust the mark was read when the read first snapshotted the volume and cleared at the decision point, with the local interval reads in between. Take both under one hold. Rust needs a mutex rather than an atomic to do it, and no longer carries the mark through the snapshot. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Put the stale mark back when the lookup does not answer for it Consuming the mark up front assumed the lookup would supersede it. A lookup that fails, or comes back with fewer than DataShards holders, supersedes nothing: the map is unchanged, its refresh time unadvanced, and with the mark gone the map a read had disproved is trusted for its full window again on the strength of a lookup that never landed. Put the mark back on both branches. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN |
||
|
Chris Lu
|
95248f7492 |
Bound the memory an EC shard recovery holds (#11020)
* Reconstruct an EC shard from the shards already on this server recoverOneRemoteEcShardInterval only ever fanned out to the cached shard locations, so a server holding shards of the volume still fetched them over gRPC from itself -- and when the peers were unreachable it could not reconstruct at all, even holding the whole volume on local disk. Seed the Reed-Solomon buffers from the locally mounted shards first; each one is a peer round trip, and an interval-sized buffer, the fan-out no longer needs. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Fetch only the EC shards reconstruction still needs The recovery fan-out read every surviving shard, so a 10+4 volume pulled 13 interval-sized buffers to feed Reed-Solomon 10 -- a third more memory held, and a third more load asked of peers that were, by definition, already having trouble. Fetch what is missing, and widen only when some of those reads fail. A shard reporting the needle deleted ends the walk: the rest would only answer the same. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Bound the bytes EC recovery holds in flight Recovery is the one read path that multiplies the served bytes: it holds an interval-sized buffer per shard until Reed-Solomon runs, and a peer that is slow to fail keeps them all alive for the whole gRPC timeout. Nothing bounded how many of those fan-outs ran at once, so a transient problem between volume servers turned every read into a DataShards-fold allocation and the server died of it -- 64 concurrent 4MB intervals pin 3.6GB, and that is a small burst. Charge each recovery against a process-wide budget, so a burst queues on the semaphore instead of on the heap. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Answer a deleted EC needle as deleted, not as a failed recovery A holder reporting the needle deleted is authoritative: deletes are never invented and never undone. Recovery already collected that flag, then dropped it on the branch where too few shards came back -- so a read of a deleted needle that had to recover surfaced as "cannot recover shard", and the volume server answered 500 where it owed a 404. Carry the flag out of the shortfall, and let it decide ahead of the error it came with. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Check the encode run of a locally seeded EC shard in Rust The Rust recovery seeded Reed-Solomon straight from the mounted shards, without the encode-run check the remote reads and Go's readLocalEcShardInterval both apply. A volume remounted from a newer encode between the read's snapshot and its recovery would have fed mixed-generation bytes into the reconstruction. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Say what the recovery budget actually guarantees Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Seed Rust EC recovery from shards on every local disk find_ec_volume returns the first disk's EcVolume, so a reconciled volume whose shards are split across data dirs had the siblings ignored and could report "cannot recover" while holding enough shards locally. Resolve each shard together with the disk that owns it, the way Go's recovery already does, and check that owner's encode run. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN |
||
|
Chris Lu
|
eed3c27d15 |
volume: cut the memory a server holding millions of volumes still uses (#10999)
* volume: stop the .vif guard depending on which entry the scan handed over A volume has both an .idx and a .vif, and loadExistingVolume skipped a .vif next to an .ecx as EC shard metadata. That was only ever correct because os.ReadDir sorted .idx ahead of .vif: an interrupted encode, where the .idx is still there, has to reach validateEcVolume to be reclaimed. Ask for the .idx instead of trusting the order. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * volume: walk volume directories in batches instead of listing them whole os.ReadDir builds, and sorts, a slice of every entry before the caller sees the first one. A disk holding millions of volumes has a .dat, .idx and .vif per volume, so each startup scan costs hundreds of MB of peak heap that the runtime is slow to hand back -- and there are several of them before the first volume loads. Walk in batches instead, and keep only the entries each scan acts on: loadAllEcShards now sorts and stats the shard and index files alone rather than every file on the disk. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * volume: skip the sibling-.dat scan when no EC volume is loaded pruneIncompleteEcWithSiblingDat only ever prunes EC volumes that are loaded, but it first walks every disk and keys a map by every .dat on the server. On a store with no EC volumes at all that is millions of map entries built to answer no question. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * volume: stop keeping a departure message for every volume The report state held a VolumeShortInformationMessage per volume copy so a departure could be named, but almost no volume ever departs. Hold a handle to the identity instead -- volumes share very few distinct ones -- and build the message on the way out. Measured over a populated report state: 195 -> 83 bytes per volume. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * rust volume: stop keeping a whole volume message per volume held The send loop kept a VolumeInformationMessage for every volume just to notice mounts and unmounts, and rebuilt the map from scratch on every beat. Keep the identity a delta names, which is what the Go report state keeps for the same reason. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * rust volume: keep only the EC files the shard scan acts on load_all_ec_shards named every file on the disk twice -- once in the dedup set and once in the sorted vector -- before deciding it only wanted .ec?? and .ecx. Filter while reading instead. Mirrors the same change in loadAllEcShards. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * volume: share the strings every .vif repeats A tiered volume's .vif names its replication and its backend, and every decode allocates a fresh copy, so a server holding millions of them holds millions of copies of the same handful of names. Route them through the interning table the volume info decode already uses. The remote key names one volume and is left alone. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy |
||
|
Chris Lu
|
12fd60f92e |
rust volume: stop racing the clock in torn_sdx_is_regenerated (#10966)
The test truncates a good .sdx and asserts the result still looks fresher than its .idx, on the reasoning that truncation bumps the mtime. That holds only at the filesystem's timestamp granularity: where both writes land in the same tick the precondition fails and the run reports a failure that says nothing about the code under test — as it did on CI. Backdate the .idx the way the sibling stale_sdx_is_regenerated already does. |
||
|
Chris Lu
|
b77d954f55 |
rust volume: fail closed on sorted-index failures and reconcile tier-up (#10956)
* rust volume: fail closed on sorted-index failures and reconcile tier-up Follow-ups to the .sdx sorted needle map (#10951): - get() folded open/read failures into None, so an EIO, a torn .sdx, or a failed pooled reopen answered reads with NotFound and let do_delete_request acknowledge the delete as Ok(0) without writing a tombstone. It now returns io::Result and every caller propagates; redb's get() had the same shape and is fixed with it. is_file_unchanged cannot propagate, so it reports unknown and logs rather than treating an unreadable index as proof of a change. - A delete whose .idx append landed but whose .sdx mark failed left the map still resolving the old live entry, so deleted content stayed readable until a reload. The map now records the tombstone before touching .sdx and only clears it once the mark lands; lookups consult that first and report the needle deleted, which is what the next reload concludes anyway. - Mode reconciliation ran one way. Entering remote mode made use_sorted_index() true, which returned early, so a volume tiered while the server runs kept its in-memory map and pinned .idx descriptor until restart — the RAM and fd win never applied. It now reconciles in both directions. - Tier-down dropped the remote reference before the fallible refresh, so a failure left volume_info local, the remote backend attached, the .vif still remote, and a retry reporting "already on local disk". The transition is snapshotted and rolled back. - The read-only fallback set no_write_or_delete but left no_write_can_delete, so metrics and mode checks called the volume delete-capable while every delete was refused. * rust volume: count a sorted-map delete against the durable .idx append The deletion counters sat after the in-place .sdx mark, so a mark that failed left them at their pre-delete values while the tombstone was already durable in .idx — and with retries now idempotent, nothing applied them later either. Heartbeats, status responses, and the garbage calculation would report the volume as free of that garbage until a reload. Move them to the append that makes the delete durable, which is also what a reload of .idx would count. Covered by a test that injects a mark failure through a cfg(test) seam: no portable filesystem trick reproduces it, since a read-only .sdx fails the borrow long before the mark. * rust volume: hide a pending tombstone from the sorted-map scans too The overlay that keeps a needle deleted after a failed .sdx mark was only consulted by get(). visit_live_entries still read the stale valid record straight off .sdx, so ascending_visit, iter_entries and save_to_idx all reported the needle live — and compaction takes iter_entries for the complete live set, so it would copy the deleted content forward and save_to_idx would write it back into the rebuilt .idx as live. Snapshot the overlay once per scan and skip its keys, which is the same conclusion the next reload reaches from the .idx tombstone. * rust volume: quarantine a durable write whose index lookup fails The prior-mapping lookup that decides whether to index a fresh append runs after the record is already down and flushed, so a failing lookup leaves exactly the state a failing put leaves: a durable .dat record nothing indexes. The put path marks the volume read only for it; this one returned the error and kept taking writes, and the next append would bury the orphan mid-file where the .dat tail check on reload cannot see it. Give it the same treatment. |
||
|
Chris Lu
|
b58d52ac16 |
rust volume: search .sdx for read-only volumes instead of holding the index (#10951)
* rust volume: search .sdx for read-only volumes instead of holding the index The Go volume server loads every read-only volume through SortedFileNeedleMap: the index lives on disk as a sorted .sdx, a lookup is a binary search, and since #10950 no descriptor is held between lookups. The Rust server had no counterpart. Read-only volumes built a full in-memory CompactNeedleMap, and cloud-tiered ones — noWriteCanDelete, so not the read-only branch — went through the writable path and pinned an .idx append handle on top of it. At the hundreds of thousands of tiered volumes a real server carries, that is an index in RAM and a descriptor each, for volumes nobody reads. Port the sorted map and the bounded handle pool. A tiered volume now costs zero descriptors and zero index bytes when idle; the pool keeps the hot handles open so a busy volume does not pay an open() per needle. Handles are Arc<File>, so an eviction cannot close one a reader still holds. The generated .sdx is byte-identical to Go's — same sort, same last-write-wins, same dropped tombstones — so a volume moved between a Go and a Rust server reads whichever copy is already on disk. A test pins the bytes against a Go-generated fixture. * rust volume: fail compaction on an unreadable .sdx, and rebuild the map on tier-down Two ways the sorted map could lose data. iter_entries swallowed read errors and returned however many entries it managed to collect. Compaction takes that vector for the complete live set, so a truncated .sdx or a mid-scan I/O fault would commit a volume missing every needle past the failure. Return a Result instead and abort. redb's collect_entries dropped errors the same way on the same path, so it goes with it. Tier-down clears the remote mode and publishes the volume as writable, but the map it booted with is the read-only sorted one. Its put always fails, so the first write would append to the local .dat and then fail to index it, leaving bytes nothing references — and a non-fsync write repeats it. Fold the reopen_idx_for_write swap into refresh_remote_write_mode so the map always matches the mode it just published; a rebuild that fails pins the volume read-only rather than letting it take writes it cannot record. Go reaches neither: its tier-down leaves noWriteCanDelete set, so the volume stays read-only until a reload or an explicit mark-writable, which already goes through reopenIdxForWrite. * rust volume: keep read-only volumes mountable on a read-only index dir, and batch the .sdx scan Building .sdx writes to the index directory, and load_index_sorted_file also created a missing .idx there. A volume whose index sits on a read-only mount took both paths and failed to load, where before it mounted read-only off an in-memory index and served reads. Create the .idx only where deletes are allowed, and fall back to the in-memory map when the sorted one cannot be built, so a directory nobody can write costs memory rather than availability. The end-to-end scan behind iter_entries, ascending_visit and save_to_idx read one entry per syscall. Read 1024 at a time instead, the batch size idx::walk_index_file uses. Positional reads, not a cursor: the handle is shared with any other borrower. Also gate the Go byte-parity fixture on the 5bytes feature it describes, which is otherwise dead code in a 4-byte-offset build. * rust volume: roll back a failed writable mark, and rebuild a torn .sdx set_writable clears the read-only flags before it can know the rest will succeed, but only the map rebuild rolled them back. An .idx writer that fails to attach left the volume advertising writable over a needle map with no writer, so puts landed in memory and were gone after a restart — the exact failure the function exists to prevent. The read-only-mount fallback made it reachable: that path loads an in-memory map with no writer attached. All three steps now run behind one rollback point. A .sdx whose length is not a whole number of entries was accepted as long as it looked fresh, and truncation is what makes it look fresh. The entry count then floored, hiding the last needle from lookups and from compaction, which would commit the shorter set. Treat a torn file like a stale one and rebuild it from .idx. Go writes .sdx in place rather than through a temporary, so a crash mid-generation is a real way to produce one. Appends now start at the last whole .idx entry too, so a torn tail there is overwritten by the next tombstone instead of misaligning every row after it. * rust volume: trim a torn .idx before writing to it, keep delete-only volumes online, count sorted-map deletes Three from review. Flooring the sorted map's append offset only protected its own positional writes. Every writable path appends at EOF instead, so a partial row left by a short write pushed the next row off alignment and the following load parsed the rest of the file as garbage. Drop the partial row before attaching any writable index writer — it is unrecoverable anyway, and every loader already skips it. Go refuses to load such a volume at all; trimming keeps it mountable with the rows before the tear intact. The unwritable-index-dir fallback stopped one step short for volumes that allow deletes, which is every tiered one: the in-memory loader opens .idx read-write there and fails on the same directory that just refused the .sdx, so the volume stayed offline. Give up the deletes instead — without a writer no tombstone could be recorded anyway — and a remount on a writable directory restores them. Sorted-map deletes left the counters untouched, so a tiered volume reported itself garbage-free until it restarted. They now land where a reload would put them: the tombstone is another .idx row, and both it and the row it supersedes count as deletions under the rule the load-time metric applies. Go skips this too, and should not. |
||
|
|
ef4c9d9178 |
filter volume by local or remote storage name (#10946)
* filter volume by local or remote storage name Signed-off-by: lou <alex1988@outlook.com> * fix SelectsEverything Signed-off-by: lou <alex1988@outlook.com> * keep the proto sync out of this change The branch copied weed/pb/*.proto over their seaweed-volume and Java counterparts and regenerated every .pb.go with a different protoc and protoc-gen-go-grpc. DiskStatus.error arriving that way broke the Rust build, and the rest is toolchain churn in files this change has nothing to say about. --------- Signed-off-by: lou <alex1988@outlook.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
51eb5333d3 |
ec: read a needle's intervals in parallel (#10911)
* ec: read a needle's intervals in parallel A needle spanning more than one EC block gets one interval per block, and consecutive blocks live on different shards. We read those intervals in sequence, so a 4MB chunk landing in a volume's 1MB small-block region cost five round trips to five different servers. Read them concurrently into disjoint slices of a single buffer, at most 8 in flight. Same change in the Rust volume server's phase C. * ec test: seed the random payload instead of the deprecated rand.Read |
||
|
Chris Lu
|
0f85d005ad |
server: 416 only when no requested range overlaps, with Content-Range, and the Rust mirror (#10889)
* filer, volume server: return 416 when no requested range overlaps the content * seaweed-volume: return 416 when no requested range overlaps the content * server: check the range test error, use the request context, fix the no-overlap comment boundary |
||
|
Chris Lu
|
c0a9b110dd |
volume: stop reporting read-only volumes that are no longer here (#10867)
* volume: clear per-collection metrics when a collection leaves a server The read-only and disk size gauges are only ever set for collections the heartbeat still finds here, and nothing zeroes the rest. volume.balance marks a volume read-only to move it, so the last heartbeat that saw it counts it read-only - and if it was the collection's last volume on that server, that count stands until the process restarts. The dashboard then shows read-only volumes that volume.list -readonly cannot find anywhere. Remember what each heartbeat set, and drop what is gone on the next one. * volume: stop the read-only volume count from wrapping at 256 The per-collection counters were uint8, so a server holding 256 read-only volumes of one collection reported zero of them. * volume: read the read-only flags once when counting them The heartbeat asked IsReadOnly for the verdict and then read noWriteOrDelete and noWriteCanDelete straight off the volume, unlocked, so the reasons could disagree with the verdict they were explaining. Take them together, under one lock. The location is now nil-checked rather than skipped by short-circuit evaluation, so a volume that has not joined a disk location yet stays safe. * volume: let only a surviving volume keep its collection reported A volume being deleted for expiry still made an entry in the read-only counts, which is what the cleanup reads as "this collection is still here". The collection's last volume could go and its series would stand for one more heartbeat. Count the survivors only. * volume: size a collection from the volumes it still has The size totals are rebuilt from scratch every heartbeat, so subtracting a volume that is about to be deleted took the surviving volumes' sizes down with it: a collection keeping a small volume and losing a larger one reported the difference, or lost its entry and kept the previous heartbeat's number. * volume: cover the deleted bytes total in the surviving volume test Deleted bytes are totalled the same way as sizes and were going unchecked, so the test now leaves deleted needles on both volumes and pins that gauge too. |
||
|
Chris Lu
|
3bd218e030 |
volume: cut idle memory at high volume counts (#10861)
* volume: start a volume's batch write worker on first use Mounting a volume started a goroutine parked on a 128-slot channel, plus the 128-entry batch slice it had already allocated. That is around 6.7KB per volume the server pays whether or not the volume ever takes a write: 7231 bytes per mounted volume, of which 4101 is goroutine stack. Only a write that asks for fsync ever reaches the worker, and a remote-tiered or read-only volume never can. Create the channel and its goroutine on the first such request instead, and let a write arriving after Destroy fall back to the inline path rather than queue onto a worker that has gone. Measured over 20000 mounted volumes: 7231 -> 1269 bytes each. * volume: update the heartbeat report state in place Every heartbeat built a second map of what it was about to tell the master, holding a freshly allocated short information message per volume, then swapped it in over the old one -- and computed departures through a third map of the live volume ids. A server holding 2M volumes rebuilt all three every VolumePulsePeriod for a report that usually says nothing. Number the heartbeats instead and mark the entry already held with the pass that found the copy, so a quiet volume costs a map lookup and no allocation. Departures are the entries a pass did not mark; the live-id map is now built only when there are some, sized to them. Measured over 10000 mounted volumes: 436 -> 196 bytes allocated per volume per heartbeat. * volume: fill one volume information message per heartbeat, not per volume The heartbeat built a message for every volume held so it could hash it, then dropped all but the few it had something to say about. At 2M volumes that is 2M messages allocated every VolumePulsePeriod to send almost none of them. Fill a message the caller supplies instead, and replace it only when the heartbeat keeps it, so a server with nothing to report fills the same one all the way through. Measured over 10000 mounted volumes: 196 -> 4 bytes allocated per volume per heartbeat, and a heartbeat runs a third faster. * volume: drop the per-volume trace from the heartbeat's status read glog.V(4).Infof evaluates its arguments whether or not the verbosity is on, so every volume boxed its id into a fresh interface slice on every heartbeat: 759 of the 773 allocations a 1000-volume heartbeat made, for a line that at this scale would print millions of unreadable rows. Measured over 1000 mounted volumes: 4776 -> 1792 bytes and 759 -> 14 allocations per heartbeat, which no longer grows with the volume count. * seaweed-volume: mirror the in-place heartbeat report state Same change as the Go volume server: number the heartbeats and mark the entry already held with the pass that found the copy, instead of building a second map of hashes and swapping it in. The volume snapshot must leave the reporting state as it found it, so it keeps asking through changed() while a real heartbeat marks through record(). * volume: refuse writes to a closed volume instead of dereferencing nil Close and Destroy leave the needle map and data backend nil, but a caller that already holds the volume can still reach the write path, where both are used unguarded: a write racing a volume deletion took the server down. syncDelete has always checked; syncWrite and the batch worker had not. Reachable before this series and now also from the inline fallback a durable write takes when the worker has gone. * seaweed-volume: guard the report state with one mutex, as Go does The full-list flag and the generation that answers it have to move together. Split across separate atomics they cannot: a request landing between begin's two reads returns full == false with the generation it just raised, and one landing between commit's read and its clear is marked answered by a heartbeat that carried no list. Either way the resend is dropped. Neither is reachable today -- every caller reaches this through the store's RwLock, the flag setters under a read lock and the heartbeat build under a write lock, so they cannot interleave. The type should not depend on that being true two files away, and Go holds a single mutex over exactly these fields. * test: build the servers under test to match the harness's offset size The mixed Go/Rust suites run both servers against one dataset, so both have to agree on the offset width. They did not: the harness built Go with no tags, 4-byte offsets, while the Rust crate defaults to its 5bytes feature, and the Rust server then refused the .vif the Go server had just written -- "bytes_offset mismatch: found 4, expected 5". Build each side to match the offset size the test binary itself was compiled with, so a plain `go test` and one with -tags 5BytesOffset both get a matched pair. |
||
|
Chris Lu
|
e0c4732e5e |
rust: stop writing when a durable write's index flush fails (#10825)
* rust: stop writing when a durable write's index flush fails A durable write flushes the .dat, publishes the needle map row, then flushes the .idx. If that last flush failed we returned the error and carried on: the row stayed live, the volume stayed writable, and the handler answered 500 without replicating. The primary then served a needle its replicas never saw, for a write the client was told had failed - and if the unflushed row was lost on restart, the durable .dat tail took the volume read only anyway. Taking the row back out is not an option: it means undoing published state on a disk that is already failing, and a truncate afterwards would leave an .idx row pointing past the end. So the volume stops taking writes instead, the same as when the truncate after a failed .dat flush cannot be done. Nothing more gets appended past a record whose index is in doubt, and the master routes writes elsewhere once the volume heartbeats read only. The divergence against the replicas is still there, but it is bounded and it is visible. A failed nm.put after the .dat is down leaves the same durable but unindexed record, so it takes the same route. * rust: drop the import the rollback removal left behind NeedleValue came in with rollback_unflushed_write, which went away when the durable path moved to flushing before it publishes. Nothing has used the type since. * rust: mark the test-only heartbeat helper as such collect_heartbeat has only ever been called from the tests - the send loop uses collect_heartbeat_with_snapshot, which it wraps - so a lib build rightly called it dead code. * rust: flush the index on a durable write that dedups A durable write matching content already in the volume flushed the .dat and returned before reaching the index flush. So a fsync=true write that deduped against an earlier non-durable one was acked with the row that indexes it still in the page cache - the same false promise the index flush exists to rule out, and the same read-only volume on restart if the row is lost. The dedup path now flushes both files, and the quarantine on a failed index flush moved into flush_idx so it applies wherever the flush is reached rather than only at the one call site that had it inline. |
||
|
Chris Lu
|
baead6901c |
ci: build protoc into the crate instead of installing it per job (#10830)
Every workflow that builds the Rust volume server first installed protoc from a package manager - twelve steps across apt, brew and choco. That is 37s per job on a good day, and this week archive.ubuntu.com stalled long enough for four jobs to burn their whole timeout without reaching a build. protoc-bin-vendored ships the compiler as a build-dependency, so it now arrives through the cargo registry the workflows already cache and there is nothing left to install. cargo build works on a machine with no protoc at all, which is worth as much locally as it is in CI. It also pins the version. The apt protoc on ubuntu-22.04 is 3.12, old enough to reject proto3 optional, which is why build.rs passes --experimental_allow_proto3_optional; the vendored one is 31.1. The flag stays, since it costs nothing and keeps a build against an older PROTOC working, and an explicit PROTOC still overrides the vendored binary for packagers who supply their own. |
||
|
Chris Lu
|
887910b377 |
rust: honor fsync on the volume server write path (#10816)
The Rust volume server ignored the fsync parameter completely: nothing parsed it, and write_volume_needle -> write_needle -> append_needle never flushed. So a ?fsync=true upload was acked out of the page cache, and since ReplicatedWrite forwards the parameter, a Go primary handing a durable write to a Rust replica got the same empty promise. The upload handler now reads fsync the way Go's r.FormValue does, off the decoded query fields, and threads it down to the volume. A durable write appends, flushes the .dat, publishes the needle map entry, then flushes the .idx, and only then is it acked. Nothing points at bytes that are not down yet, so a failed flush only has to take its own append back off the end - the index never moved and the volume's counters never saw the rejected write. If that truncate cannot be done the volume stops taking writes, rather than letting a later append bury the rejected record mid-file where the tail integrity check cannot see it. The .idx flush is what keeps the ack honest: load() rebuilds the map from .idx, so an acked write whose row was lost comes back as a .dat tail the integrity check cannot account for, and the volume loads read only. A dedup hit flushes too: there is nothing to append, but the write it matched may have been non-durable, and the caller is asking for the content to be on disk. Batched writes carry the flag per request rather than one flush per batch, so the write queue's module doc no longer claims otherwise. |
||
|
Chris Lu
|
6fda8c67f3 |
Guard the gcs credential path in FetchAndWriteNeedle like the other backends (#10796)
* volume: accept only static-key gcs credentials on the fetch request An inline credentials document of a federated type points the SDK at a url, file or executable of the caller's choosing for the token exchange, so the request-supplied value is no longer just a key. * volume: guard the gcs token endpoint like the other remote endpoints Inline credentials pick where the token request goes, so route the gcs client through the same deny-list and rebinding-safe dialer used for S3 and azure. * rust volume: pin that gcs has no credential-driven dial path * volume: only check gcs credentials on a gcs remote conf Only the gcs backend reads that field, so another backend carrying a stale value should not fail the request. * gcs: load credentials with the type the caller expects The untyped loader is deprecated because it reads whatever the document claims to be; callers handling credentials they do not control now name the types they accept. |
||
|
Chris Lu
|
d713ab49f9 |
volume: validate replica targets and restrict gcs credentials in FetchAndWriteNeedle (#10755)
* volume: validate replica upload targets in FetchAndWriteNeedle The replica leg forwarded the fetched needle to a caller-supplied address without checking it, so a malformed target could redirect the upload to an unintended host or path. Require each replica target to be a bare host:port whose host is not loopback / link-local / unspecified, reusing the address deny-list; cluster peers legitimately sit on private networks, so RFC 1918 / CGNAT stay allowed and -volume.allowUntrustedRemoteEndpoints still opts out. Validate every target up front so a bad one fails the request before the local write, and upload through a client that re-checks the resolved address at connect time so a replica hostname cannot rebind to a blocked address after validation. Mirrored in Rust (validation moved ahead of the local write; the Rust S3 path's connect-time re-check is still a follow-up there). * volume: only accept inline gcs credentials in FetchAndWriteNeedle The gcs credentials value on this request could name a local filesystem path, which the SDK reads from disk. Accept only inline JSON here; the server-side GOOGLE_APPLICATION_CREDENTIALS env var still supplies a path. The Rust volume server has no gcs backend, so there is nothing to mirror. |
||
|
Chris Lu
|
9125b9c835 |
volume: extend the remote-endpoint guard to the azure backend (#10754)
* remote_storage/azure: allow a per-request HTTP client Thread an optional *http.Client through NewAzBlobClient and add azure.MakeWithHTTPClient, mirroring the S3 backend. When set, the client overrides the azblob transport so a caller can pin the dial path. The existing makers pass nil, so behavior is unchanged. * volume: extend the remote-endpoint guard to the azure backend The endpoint validation and rebinding-safe dialer in FetchAndWriteNeedle covered the S3-SDK backends. The azure backend also dials a caller-supplied AzureEndpoint, so route both families through a single guardedRemoteClient helper that returns the endpoint each backend dials and a constructor bound to the guarded HTTP client. azure is guarded only when AzureEndpoint is set; an empty endpoint derives the public host from the account. -volume.allowUntrustedRemoteEndpoints still opts out. * rust volume: assert the azure endpoint has no remote-client path The Rust volume server has no azure backend, so make_remote_storage_client rejects the type before any client is built. Add a regression test pinning that invariant. |
||
|
Chris Lu
|
ae2cc8225e |
rust volume: mirror the VolumeConsolidateIndex RPC from Go (#10752)
The Go volume server has VolumeConsolidateIndex, which moves a volume's .idx out of the data directory into the configured -dir.idx directory (where an EC decode/reconstruct can leave it co-located) and reloads the volume in place. The Rust port's proto omitted the RPC entirely, so its generated VolumeServer trait was one method short of Go's. Add the proto message and rpc, the gated grpc handler, and Store::consolidate_volume_index / Volume::relocate_index_to, mirroring Go's Store.ConsolidateVolumeIndex and Volume.RelocateIndexTo -- including the cross-device copy fallback and the reopen-against-the-old-dir path when the move fails. Integration tests cover the real move (index relocated, volume still serves reads and the move is idempotent), the no-op paths (index already in place, no separate idx dir) and the not-found error, plus the grpc handler end to end. |
||
|
Chris Lu
|
c0f33d599b |
rust volume: mirror Go volume server logic to gate the admin RPCs (#10748)
rust volume: gate the remaining admin RPCs behind check_grpc_admin_auth
The Go volume server gates 29 destructive VolumeServer RPCs on the
-whiteList admin check; the Rust port only gated 14. Add the gate to the
other 15 -- batch_delete, read_all_needles, fetch_and_write_needle, the
EC-shard generate/rebuild/copy/unmount/to-volume RPCs, both tier-move RPCs,
volume_copy, volume_tail_receiver, set_state, scrub_ec_volume and
volume_needle_status -- so a configured whitelist restricts them the same
way it already does on the Go side.
check_grpc_admin_auth also required peer info before checking whether any
control was configured, unlike Go's `if vs.guard == nil { return nil }`.
Short-circuit when no whitelist and no signing key are set, so in-process
callers keep working with security inactive and only the gate ordering
changes for configured servers.
tests/admin_auth_coverage.rs mirrors the Go coverage test: every handler
must either gate or be listed as intentionally open with a reason, so the
two implementations can't silently drift apart again.
|
||
|
Chris Lu
|
76d3fd0e9d |
grpc: optional client_cert/client_key for outgoing mTLS connections (#10747)
* grpc: optional client_cert/client_key for outgoing mTLS connections * scaffold: list client_cert/client_key in each grpc section |
||
|
Chris Lu
|
c6e1387f59 |
shell: multi-target fs.mergeVolumes and volume.mark -readonlyCanDelete (#10706)
* shell: fs.mergeVolumes distributes one volume across multiple -toVolumeId targets * volume: volume.mark -readonlyCanDelete rejects writes but keeps accepting deletes * seaweed-volume: mirror readonlyCanDelete volume state |
||
|
Chris Lu
|
00c5572e8c |
volume: decode IPv6 transition addresses in the remote-endpoint guard (#10683)
* volume: decode IPv6 transition addresses in the remote-endpoint guard checkBlockedIP normalized only ::ffff: mapped IPv4, so NAT64 (64:ff9b::/96), 6to4 (2002::/16), Teredo (2001:0000::/32), and IPv4-compatible (::/96) addresses that embed an internal IPv4 (loopback, 169.254.169.254, RFC 1918) passed the endpoint guard even though the plain IPv4 forms are refused. Extract the embedded IPv4 from those forms and re-check it against the deny list, which covers both the up-front validation and the dial-time guard. Mirrored in the Rust volume server. * volume: require the full NAT64 well-known prefix before decoding Only 64:ff9b::/96 carries the embedded IPv4 in the low 32 bits, so also require bytes 4-11 to be zero before treating an address as NAT64; other 64:ff9b: prefixes place the IPv4 elsewhere and are left untouched. Add public-target coverage for 6to4, Teredo, and IPv4-compatible so every decoder is exercised on both a blocked and an allowed destination. Mirrored in the Rust volume server. |
||
|
Chris Lu
|
f09e8345c6 |
storage: stop keeping the remote storage key on the master (#10672)
A master decides nothing from it. Every caller that read it was asking whether a volume is remote, which the backend name answers, and the value itself is reported on demand by the server holding the volume, through the volume info in ReadVolumeFileStatus. It is also the one string here that cannot be shared: unique per volume, so unlike the collection and backend names it carries its own characters for every volume a master tracks. VolumeInfo goes from 136 bytes to 120. 800k volumes registered from a heartbeat that has been over the wire go from 214 to 163 B/volume when tiered. The volume server's own status page keeps showing the key, now read from the volume it holds rather than relayed through a master, which is also where the other volume server implementation reads it. The heartbeat digest drops it on the same grounds: a change to something the master does not hold cannot make its copy stale. Both implementations and their shared vectors move together, and the field-coverage test now names what is deliberately not retained rather than being loosened. |
||
|
Chris Lu
|
567052bfb6 |
s3: take bucket sizes from the master's summary (#10664)
* pb: ask the master what each collection holds Callers tracking usage were sent every volume in the cluster to add up themselves, which is the master's largest single allocation. * topology: summarise what each collection holds One pass over the topology, allocating per collection rather than per volume. Regular volumes count once each for logical totals and once per replica for physical, taken from the lookup index, which is already keyed by volume and so needs no set of seen ids. Ec shards are node-local so their sizes sum, while the file and delete counts describe the volume and resolve once every holder has been seen. Replicas of one volume disagree while a write is landing or a heartbeat is late. Walking a full listing took whichever replica the map iteration reached first, so the answer moved between runs; this takes the largest, which is stable and never reports usage below what some replica already holds. * s3: take bucket sizes from the master's summary The bucket size metrics pulled the whole volume list once a minute and added it up, which cost the master 184.6MB of allocation and 17.8MB on the wire for six numbers per collection. VolumeList over 550k volumes 184.6 MB allocated, 17.8 MB on the wire CollectionStatistics 176 bytes allocated, 47 bytes on the wire The aggregation moves to the master with it, so the cases the removed tests covered are now asserted against it directly. * topology: count the replica holding the most live data Quotas are enforced on size less deletions, and the replica with the biggest raw size can be the one that has deleted the most. Counting it reported a bucket smaller than it is and would leave one writable over its quota, which is the opposite of what picking the largest was meant to guarantee. * topology: cap a volume's deletions at what it holds Live usage is read as a collection's size less its deletions, so a volume reporting more deleted bytes than it has cancels live bytes belonging to other volumes in the same bucket and reports it smaller than it is. Replica selection already floored that volume's own live size at zero; the totals have to agree with it. |
||
|
Chris Lu
|
a2ffc7aadf |
heartbeat: keep the master current through collection churn (#10657)
* heartbeat: name departed volumes in delta heartbeats * master: release the lookup index with a deleted collection * master: keep a fresh grow safe from the report that raced it * volume: name the volumes a deleted collection took with it Deleting a collection left the master to work out what went by omission from the next full volume list, which it no longer gets: heartbeats carry the whole list only when the master asks for it. The volumes a bucket's churn creates and destroys between two of those requests are never named in either direction, so the master keeps counting their slots as occupied and a cluster that creates and drops collections quickly runs its free-slot accounting dry -- assigns fail with no free volumes left while the disk holds a handful of volumes. The destroy path already knows exactly which volumes it removed, so send them down the same channel every other deletion uses. * rust: name the volumes a deleted collection took with it Mirrors the Go volume server. The notify path derives its deltas by diffing snapshots, so a collection delete that does not wake it is invisible until the master next asks for the whole list. |
||
|
Chris Lu
|
ce7d388639 |
heartbeat: send only the volumes that changed (#10640)
* pb: let a heartbeat carry only the volumes that changed A partial list cannot travel in volumes: a master that did not understand it would read the absences as deletions. So changes get their own field, used only once the master has said it compares digests and can tell when it has fallen behind. * master: apply the volumes a heartbeat reports as changed Only the named volumes are touched. A full report says the server holds exactly these; a changed report says nothing about the ones it leaves out, so absence must not read as removal. Also advertises that the master compares digests, which is what lets a server stop sending its whole list. Advertising it once per connection means a server reconnecting to a master that does not is back to full lists straight away. * volume: send only the volumes that changed once the master accepts them The whole list goes on every heartbeat until the master says it compares digests, and again whenever it asks, so a master that cannot tell when it has fallen behind never has to. has_no_volumes stays derived from a full list alone. Deriving it from what a heartbeat happens to carry would make a quiet one read as a server that had lost every volume, and the master would drop them all. The digest still covers every volume held rather than the ones sent, which is what lets the master confirm that applying the changes left it current. Reporting state is per-connection: a server that reconnects, or reaches a different master, starts again from the full list. * volume: let the zero reporting state stand for having told no master anything A Store built as a literal, which tests do, left the reporting state nil and panicked on the first heartbeat. As a value its zero form already means nothing has been reported to anyone, which is exactly the state that sends the whole list. * rust: send only the volumes that changed once the master accepts them Mirrors the Go volume server, with one hazard the Go side does not have: mount and unmount deltas here are derived by diffing successive heartbeats, so a heartbeat that carries a partial list would report every volume it left out as unmounted. Collecting now returns the full set alongside the message, and every site that diffs uses that rather than what went on the wire. * volume: do not let a full-list request be lost to the heartbeat it raced The request arrived while a heartbeat was already being built as a delta, and committing that heartbeat cleared it, so the master waited for another digest mismatch before asking again. Count the requests and clear only the one the heartbeat answered. * rust: stop marking volumes reported by a heartbeat that is thrown away The state-notify path collected a heartbeat only to diff its volume list, then sent a delta message of its own and dropped the one it had collected. Once collecting recorded what the master had been told, every mount or unmount silently marked the changed volumes as sent, and the master learned of them only after a digest mismatch. Snapshotting no longer records anything, and no longer expires ec volumes whose deletion that path was already discarding. * master: announce only the volumes a change actually brought Every changed volume was broadcast as a new location. Volumes grow constantly and growth moves no location, so on a busy cluster that told every connected client about volumes it could already reach, filling bounded broadcast queues and pushing out the topology updates that matter. * master: ask for the full list when only one can repair the master Delta heartbeats stop the full report, and with it the only thing that re-registers a volume the lookup index lost. The volume server cannot see that divergence and its digest cannot show it, so the master now checks its own two indexes agree and asks for the list when they do not. A node reporting one volume id twice is kept on full lists for the same reason rather than merely skipped: its digest can never be verified, so nothing else would tell the master what it had stopped holding. * master: keep the volume options on every heartbeat response A volume server takes them from whatever response arrives, and preallocate is a bare bool with no way to tell off from unmentioned. A response sent to ask for the volume list therefore turned preallocation off until the server reconnected. Responses sent mid-stream now start from the configured options rather than being built field by field. * master: announce a volume the lookup index had lost Repairing the index makes the volume servable again, but clients were told it went when the node dropped out and nothing told them otherwise: the disk map still held it, so it did not count as an arrival. Reaching the lookup index is what makes a volume servable, so recovering an entry there is an arrival as far as clients are concerned, on both the full report and the changed-volume path. |