498 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandDevin	2c510cc30e	ec.rebuild: remount shards the rebuild replaced, keep regenerated copies (#12193) ... * volume: remount ec shards the rebuild replaced under a mounted handle Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec.rebuild: keep copied shards the rebuild regenerated in place Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: reopen regenerated ec shards in place instead of remounting UnmountEcShards and MountEcShards send deltas through separate buffered channels, so a remount can reach the master as add-then-delete and drop the repaired shard from lookups until the next full EC heartbeat. ReopenEcShard swaps the mounted shard's descriptor under the volume lock instead: registration, disk type, and heartbeat state never flip, and readers holding the old shard degrade to a clean error the parity path already recovers from. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: reopen rebuilt ec shards in place, mirroring Go volume_ec_shards_rebuild recreates a shard file under a mounted handle whose descriptor still serves the detached inode (a mounted shard whose file was deleted under it). DiskLocation::reopen_ec_shard swaps the mounted descriptor for a fresh open, matching Go's ReopenEcShard — the registration and heartbeat view never flip. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: fail ec rebuild when a regenerated shard cannot be reopened * rust volume: fail ec shards rebuild when a rebuilt shard cannot be reopened * volume: repoint mounted ec shards after rebuild across disks A mounted shard whose file was deleted under the mount is "missing" to the rebuild: the regenerated file lands on the rebuild location while the handle keeps serving the detached inode on another disk. Reopening the mounted disk then failed the whole RPC, so the caller never saw the rebuilt ids and its deferred cleanup deleted the regenerated copies. Repoint every mounted handle of the rebuilt volume instead of only this run's outputs: reopen in place when the rebuilt file is on the mounted disk, otherwise drop the stale handle and remount the shard from whichever local disk holds the file via a delta-free load split out of MountEcShards — a same-server move, so the master's registration never flips. A shard that can be neither reopened nor remounted is left unmounted rather than serving replaced bytes, and the repoint step no longer fails the RPC, so RebuiltShardIds always reaches the caller. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: repoint mounted ec shards after rebuild across disks Mirror the Go repoint loop: iterate every mounted shard of the rebuilt volume, reopen in place when the rebuilt file is on the mounted disk, and otherwise drop the stale handle and remount the shard from whichever local disk holds it. The repoint no longer fails the RPC, so rebuilt_shard_ids still reaches the caller and its copy cleanup keeps the regenerated shards. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: still rebuild ecx and repoint mounts when no shard is missing * master: keep a node in an ec shard's lookup while a disk still holds it A rebuild that relocates a mounted shard between a node's disks reports the move as a delete on the source disk plus an add on the destination. Registering first is a no-op because shard lookups key on the node, then the unregister dropped the node outright — and steady-state heartbeats never restore it, so readers and later rebuilds lose the repaired copy. Skip the delete while another disk on the node still holds the shard. UnregisterDataNode now clears the node's own ec records before dropping its registrations, so the retained-shard check cannot skip the teardown, and each unregister gets a ShardsInfo copy since DeleteEcShard drains the shared one. * rust volume: fold the journal on a mounted index instead of rewriting it The rebuild handler regenerated .ecx through File::create even while the volume was mounted, truncating the index under readers that search the open descriptor with its old size — a live lookup could hit EOF mid-scan. Go folds the deletion journal in place on a mounted volume; do the same through rebuild_ecx_from_journal and reserve the full regen for the unmounted path, where nothing holds the file open to interrupt. * volume, rust volume: report rebuilt shards on late failures, fix rust corrupt discovery Two review findings the repoint rework left open: - A RebuildEcxFile failure still returns a plain error after shards were regenerated, so the caller discards the response holding the rebuilt ids and the deferred cleanup deletes copies the rebuild regenerated in place. The Go handler now packs the ids into the gRPC status details and the shell pulls them back out on error; the Rust handler carries them in grpc-status-details-bin the same way. - The Rust rebuild still counted a sidecar-corrupt shard as present, so it fed bad bytes into reconstruction and never regenerated the shard. Discovery now loads the generation-0 .ecsum the way Go does, verifies every present shard, and reclassifies mismatches (and zero-size residue) as rebuild targets written to a .rebuilding temp file, verified, and atomically renamed over the original only after every output passes the sidecar check. Honors unsafe_ignore_sidecar, added to VolumeEcShardsRebuildRequest to match the Go proto. * volume: repoint sibling journals sharing a folded .ecj after rebuild RebuildEcxFile unlinks the deletion journal and the mounted path only reopens its own handle — a sibling mount journaling into the same file keeps appending to the detached inode, and those deletes never reach a path-based reader again. Reopen every mounted volume sharing the index base before leaving the rebuild. * rust volume: repoint sibling journals sharing a folded .ecj after rebuild rebuild_ecx_from_journal replaces the journal and reopens only its own handle; mounts on sibling disks sharing that .ecj keep the detached handle, so their next appends are lost at restart. Adopt the folded journal on every mounted runtime sharing the path before releasing the store lock, clearing their in-memory set — every journaled id is now an .ecx tombstone. * ec: swap the deletion journal handle only after the new file opens A failed reopen left ecjFile nil, so every later delete returned "ec volume closed". Open first, then close and swap — mirroring the Rust adopt_folded_journal ordering. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec.rebuild: fail the rebuild when a sibling journal cannot be repointed A warning left the sibling mount journaling to a detached inode while the RPC reported success. Propagate the error like the Rust handler's adopt_folded_journal map_err. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec: keep deletes loud when the journal reopen cannot open the replacement Reverts the swap-on-success ordering: a detached handle kept on failure lets deletes append and sync to the removed inode, then vanish at restart. Closing first also frees the descriptor before the open. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: drop the folded journal handle before reopening it adopt_folded_journal and the post-fold recreate now clear ecj_file before opening the replacement, so a failed reopen leaves deletes erroring instead of appending to the detached inode. Mirrors Go's ReopenDeletionJournal. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ec: drop the cached journal size when its reopen fails Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 00:58:30 -07:00
Chris LuandDevin	eec2452e30	master: replicate the recovery-wait growth unfreeze through raft (#12165) ... * master: replicate the recovery-wait growth unfreeze through raft state_loss_recovery_wait lifted the volume-growth freeze only in memory, while the freeze itself was replicated. A restart replayed the frozen flag and growth stayed refused until the wait expired again, on every restart, unless an operator ran volume.growth.resume. The lift now applies GrowthFrozen=false like ResumeVolumeGrowth, from a goroutine since IsGrowthFrozen is a read path that cannot block on raft. The apply runs under reportedTopologyIdsLock so a freeze landing after the lift keeps its command last in the log; the stale unfreeze skips itself when the flag is frozen again. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: cover raft replication of the recovery-wait unfreeze Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: retry and re-arm a growth unfreeze raft failed to record applyGrowthFreeze swallowed the raft error, so a rejected lift during a leader change left the flag clear while the log still held the freeze; nothing retried and a restart re-froze growth anyway. The apply now returns the error, replicateGrowthUnfreeze retries it, and on persistent failure re-arms the flag so the next IsGrowthFrozen attempts the lift again instead of diverging from the log. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: release the topology-id lock before the freeze raft write CheckReportedTopologyId and SetTopologyInitPending held reportedTopologyIdsLock across the blocking raft apply, so a slow apply stalled every later report — including the heartbeat that should trigger the next freeze. The lock now covers only the mismatch decision; the replicate step runs after it is released. * master: bind the recovery unfreeze to the freeze instant it lifts An unfreeze submitted while a newer freeze sat committed but not yet applied — e.g. after this node won an election — could land behind that freeze in the log and erase it. The command now carries the freeze instant cluster-wide (GrowthFrozenAt) and the lift is bound to the instant its submitter observed (GrowthUnfreezeFor): the state machine no-ops a lift while a strictly newer freeze exists, which also lets the unfreeze run without holding the topology-id lock at all. Concurrent freeze writes keep their own mutex so log order cannot regress the instant. * master: a superseded unfreeze retry leaves the resumed flag alone replicateGrowthUnfreeze broke out of its retry loop when a newer\nfreeze superseded the lift, but the break fell through to the\nre-arm: an operator resume in between was silently undone. Return\ninstead — the newer freeze generation owns the flag. * master: replayed freeze entries without an instant stay ageless Freeze commands written before the instant field existed carry zero.\nSubstituting the replay time made each restart stamp the freeze as\nnew, so a committed bound unfreeze always looked older and was\nrejected — the freeze came back on every restart. Keep the zero\ninstant: the recovery window reads as elapsed and bound lifts apply. * master: pack the growth freeze flag and generation into one atomic Separate flag and timestamp atomics left three holes: a resume's\nload-and-clear could lift a generation stamped in between, a bound\nlift compared stamps numerically although masters' clocks need not\nbe monotonic, and an ageless generation had no identity a bound lift\ncould match without colliding with the unconditional encoding.\n\nOne value now carries the state — 0 unfrozen, -1 an ageless freeze,\nelse the freeze instant — so every transition is a single atomic\nswap or compare-and-swap bound to an exact generation. * master: a repeat mismatch report keeps the armed freeze generation Re-stamping while already frozen never reaches the raft log, so the local generation diverges from the replicated one and every bound unfreeze becomes a no-op on replay — the freeze returns after restart or failover. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: snapshots read the freeze flag and generation in one load Save, Snapshot, and PersistMaxVolumeId read IsGrowthFrozen and the generation separately, so a resume landing between them persisted frozen=true with stamp 0 — restoring an ageless freeze a committed bound lift cannot clear. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 08:44:45 -07:00
Chris LuandDevin	3fd8d0cf09	master: let location-constrained assigns wait for pending growth (#12167) ... * topology: add AvailableSpaceWithinLocation for location-scoped free slots AvailableSpaceFor answers for the whole topology; a location-constrained assign needs to know whether the requested data center, rack, or data node still has room before deciding growth can satisfy it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: let location-constrained assigns wait for pending growth An assign carrying a data center or rack constraint failed on the first unsatisfied pick, even when the same request had just queued volume growth for that location — so the first write into a fresh collection errored out instead of picking up the volume growth landed a moment later. Keep the fail-fast path only when growth cannot help: growth disabled, no growth pending, or no free slots left inside the constraint. Otherwise the request takes the same wait/shed path as an unconstrained assign. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: sum leaf free slots in AvailableSpaceWithinLocation Rolled-up rack and data center counters keep a maintenance-mode server's capacity, and one overcommitted node can cancel a sibling's slots. Sum the matching data nodes directly so the location check agrees with what growth can actually use. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: let constrained growth initiators reach the shed path When a constrained assign's own growth concluded unfulfilled, the fast-fail check saw only that no grow was pending and returned the plain pick error — a code the assign client does not retry, so the first write died instead of re-triggering growth. Exclude the request's own initiated grow from the no-pending-growth condition so it reaches the existing re-pick and ResourceExhausted shed like an unconstrained assign. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: judge targeted grow requests by the location they name A grow request carrying a data center, rack, or data node constraint was discarded whenever the layout's writables elsewhere were healthy, so a location with free slots but no writable could never receive the volume its assign was waiting for. Add ShouldGrowVolumesWithinLocation and use it for constrained requests; the layout-wide check still governs untargeted ones. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: only positive leaf slots count in AvailableSpaceWithinLocation An over-committed node reports negative free slots; summed plainly it\ncancelled a sibling's room and a constrained assign could fail fast\nwhile the location still had space to grow into. * master: fail constrained assigns fast when growth cannot place A constrained assign with free slots in its location still waited out\ngrowth that could never satisfy the replica placement — a single-rack\ndata center asked for 010 sheds the same doomed grow on every client\nretry. Run the grow-time node search as a dry run and return its\nplacement error instead of a retryable shed. * master: judge constrained growth by layout, not free slots A slot-level feasibility probe races with the grow it judges: a\npending grow consuming the last slot in one rack reports the\nplacement impossible while it mints the request's own volume, and\nthe hard error skips the re-pick that would find it. GrowthBlocker\nnow counts racks and data nodes inside the named location — shape a\ngrow cannot consume — so only genuinely unplaceable layouts fail\nfast. * topology: GrowthBlocker mirrors the grow selector's feasibility The location filter bound every replica tier, so a 010 request naming one rack needed two racks inside it while the selector constrains only the first placement and counts the rest by room. And counting nodes without free slots let a full location queue a grow it could never serve, so the assign shed ResourceExhausted at timeout instead of failing fast. GrowthBlocker now applies the option's location to the primary placement chain only and leaf-sums free slots the way the selector does, so it reports a blocker exactly when the grow can never place. * master: repick once before surfacing a growth blocker An in-flight grow can spend the last free slot minting the request's own volume: the blocker's slot counts predate that registration by a heartbeat, so returning the placement error on the first failed pick could reject a write whose volume lands a moment later. A pending grow now earns one repick grace before the blocker error is returned. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 08:10:45 -07:00
Chris LuandDevin	18691e7b5b	server: keep the first write on a fresh weed server from stalling (#12149) ... * master: end the post-mint allocation hold on the embedded reporter's clean inventory DeferAllocation armed a warmup-length hold whenever the leader minted a fresh topology id, and on weed server the hold ran the whole window: the embedded volume server's join kept refreshing the quiet evidence clock, so the two-pulse early release still cost the first write 10-15s on every fresh start. weed mini already releases on its single pinned reporter's first inventory, but pinning refuses outside volume servers, which weed server must keep admitting. Add a non-exclusive variant: the topology names one expected reporter whose first clean full inventory lifts the hold, while other reporters stay admitted and still feed prior-state evidence and the topology-id freeze. A single-master weed server points it at its embedded volume server — the one reporter guaranteed to arrive quickly; a veteran's heartbeat carries prior state and keeps the window through the regular checks, so a wiped -dir with surviving external volume servers still trips the freeze on first report. Restarted clusters are untouched: a recovered topology id means no mint and no hold. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: announce on the startup peer scan for embedded single-master setups A fresh filer whose startup peer scan came back empty re-polled for the full confirm window before registering on the master, so the lock ring had no member and routed traffic such as S3 PUTs spun on "no lock server found" for ~10s on every fresh weed server / weed mini start. For setups that start with exactly one filer the scan is already final: a single-master weed server and weed mini announce as soon as it completes. A late external filer still joins and replays through the normal path — the confirm window only ever caught peers arriving inside it, and the residual risk mirrors the accepted volume-side trade-off: an embedded filer announcing a few seconds earlier can own the lock ring while a surviving external filer's backlog is still on its way. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: retry a failed heartbeat promptly until first registration A heartbeat stream refused before the leader is known — the common race when a volume server starts while an election is still running — always fell back to sleeping out the full pulse, so a fresh volume's first registration could wait ~5s behind a leader elected a second later. The peer refresh already identifies the leader during the wait, so cap the retry pause at a second once a leader is named; while no leader is known the same short retry applies until the first stream is confirmed, since an unregistered server has nothing else to do. Once the server was registered the pulse backoff is unchanged, so a lost leader still paces reconnects the way it did before. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: retry a failed heartbeat promptly until first registration (rust) Mirror the Go heartbeat retry: cap the post-refresh wait at a second when a leader is named, and keep the short retry until the first stream is confirmed so a fresh volume registers promptly when it races an election. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: leave admitted reporters a confirm pulse when the embedded report ends the hold The single-master weed server releases its post-mint allocation hold on the embedded volume server's clean report, but outside volume servers stay admitted and one may still carry prior cluster state — an inventory from the embedded server alone cannot prove none is due. Clearing the hold outright let a rebuilt master re-issue a volume id a reconnecting veteran still owned. Collapse the released hold to one heartbeat interval instead: late reporters land their evidence or a topology-id freeze inside the pulse, and a re-mint re-arms that pulse rather than the full warmup. The pinned single-reporter release (weed mini) still drops the hold outright, since no other reporter can be admitted. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: keep the empty-membership confirm on embedded single-master weed server A single-master weed server still admits external filers, so an empty startup peer scan is not final: after a master restart the existing filers simply have not reconnected yet. Announcing on the scan let the embedded filer take lock-ring ownership and serve writes that a peer's replayed backlog could then overwrite. Only weed mini keeps the skip, where the embedded filer is the only member its dev cluster can have. The confirm window only delays the master listing for a fresh store; the filer serves while it runs. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: tolerate an expired heartbeat retry deadline (rust) refresh_leader_from_peers can consume the whole pulse budget, leaving retry_at in the past; subtracting a later Instant panics and kills the heartbeat task instead of retrying. Match the Go side, where time.Until of a past deadline just sleeps zero. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: retry the seed list without a second pulse sleep (rust) The error path already slept its retry delay; an extra full pulse after losing the round delayed the next attempt past what Go's heartbeat() does, which paces retries only inside the per-master attempt. * master: keep a configured allocation hold shorter than the confirm pulse Once the embedded report lands, the collapsed hold replaced the configured deferral with a full pulse even when the operator set a shorter one, including zero (disabled). Cap at the pulse instead. * filer: end the empty-membership confirm early on a fresh-minted embedded master On a fresh single-master weed server the announce gate still held the filer for the full confirm window, so the master's lock ring stayed empty and the first write retried 'no lock server found' for ~7s on top of the collapsed allocation pulse. A fresh TopologyId mint proves the master's persisted state was empty, making the empty peer scan final; skip the rest of the confirm only then, and keep it whenever the master recovers a prior id. * filer: keep a short reconnect window on a fresh-minted embedded master A fresh TopologyId mint proves the master's persisted state was empty, not that no external filer survived its loss — one can still be mid-reconnect on its keepalive cycle when the confirm starts. Ending the empty-scan confirm on the first poll let the fresh filer register and serve routed reads before that peer was found and backfilled, so existing files could read as missing. The fresh-mint path now shortens the confirm to a few keepalive cycles instead of skipping it: a reconnecting filer still lands in time, while a genuinely fresh cluster announces after ~3s rather than ~10s. * volume: skip the heartbeat loop when no seed masters are configured An empty seed list made the outer loop spin with no await between passes and no path to observe the stop flag. A masterless volume server has nothing to register with, so the heartbeat exits after a warning. * volume: skip the heartbeat loop when no master addresses are given (rust) Mirrors the Go heartbeat: an empty seed list spun the outer loop with no await and no shutdown checks. Exit after a warning so library callers get the same behavior as the binary path that validates the list up front. * filer: re-check the fresh-mint hint on each confirm pass The topology id is minted after raft election, which can land after the empty-scan confirm has already started — a one-time check at entry missed the mint and held the full window. Evaluate the hint on every poll and shorten the confirm from the first observed mint, still capped by the normal confirm deadline. * master: mark a fresh topology mint only when the volume bound is zero Recovery restores the max volume id unconditionally but the topology id only when present, so an upgraded or partially rebuilt master can mint a new id while still holding prior volume state. Marking fresh there would let the embedded filer shortcut its peer confirm on a cluster that is not empty; only mark the mint fresh when no bound survived. * filer: drop the fresh-mint shortcut once a peer has been observed --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 08:08:26 -07:00
3 people	80f3cc0613	topology: keep oversized volumes out of the writable list across re-registration (#12120) ... * topology: keep the oversized verdict after the location list is gone The per-replica oversized marks live in the volume's location list, which is deleted when its last location leaves and recreated flagless on the next registration. Between that recreation and the next heartbeat that re-measures the volume, ensureCorrectWritables saw no mark and re-admitted an oversized volume to the writable list; two other paths (quarantine release, mark-writable) never consulted the marks at all. VolumeLayout now remembers the oversized verdict per vid, written only by heartbeat sizes and forgotten only when a report measures the volume under the limit. The writable admission gate itself refuses a marked volume, and the vacuum commit path drops a verdict its live size check has just disproved. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: mark oversized at registration when the report carries a size RegisterVolumeLayout ran EnsureCorrectWritables while the recreated location list still held no per-replica marks. When the registering report knows the volume's size, refresh the oversized state before the writable check instead of waiting for the next heartbeat pass. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: test that oversized volumes stay out of writables across re-registration Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: remember the oversized verdict per replica A shared per-volume verdict was cleared by any single replica's under-limit report, so a still-oversized replica whose location entry had died could re-register from a size-less report and put the volume back on the writable list. Keep one verdict per reporting replica: an under-limit report clears only its own, and a replica that never returns cannot pin its volume out of writables forever because the verdict ages out. The vacuum commit now feeds the node's fresh answer through the same per-replica bookkeeping instead of erasing the whole verdict. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: count a volume at the size limit as full on vacuum commit The layout's oversized test is >= the limit, but the vacuum commit and status checks used >, so a replica sitting exactly at the limit passed as not-full and the commit path could re-admit it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: cover per-replica oversized verdicts in tests One replica shrinking must not clear a still-oversized replica's remembered verdict; cover that plus a size-less re-registration before the second replica is re-measured. The vacuum-commit case now shrinks the stored record first, since the commit re-derives the mark from the node's own answer. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: judge the vacuum commit by the size it just measured SetVolumeAvailable rebuilt the oversized verdict from the cached VolumeInfo, which still carries the pre-compaction size until the next heartbeat. A commit that shrank a volume under the limit re-marked the replica oversized and never restored writes. The commit path already asks every replica for its fresh size; pass it through per replica and record it on the mark, the verdict, and the size tracking seed. * topology: hold an expired oversized verdict while its replica is back The TTL forgives a replica that never returns, but a remount announces the volume without a size and registration skips the oversized check. Deleting the verdict on expiry then re-admitted a still-oversized volume until the next size-bearing heartbeat. Keep counting an expired verdict while its replica sits in the live location list, and keep sweeping verdicts whose replicas stayed gone. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chrislusf@gmail.com>	2026-10-26 23:22:15 -07:00
4 people	6ebc055a94	master, volume, filer: start on a read-only mounted directory and serve existing data read-only (#12081) ... * master, volume, filer: start on a read-only mounted directory and serve existing data read-only A read-only mount was treated as writable (permission bits only), so every service failed on its first write: the master panicked writing raft conf.tmp, the filer exited opening the leveldb LOCK, and the volume server skipped every volume while still offering the directory as free capacity. - util.IsReadOnlyMount: access(W_OK) == EROFS. - volume: on a read-only -dir take the existing O_RDONLY/noWriteOrDelete load, index .idx in memory when no .sdx exists, offer no free slots (same branches as isDiskSpaceLow), skip the vid stamp there. - master: on a read-only -mdir run raft from a private copy and refuse new volume ids. - filer: leveldb2/leveldb3 set their existing ReadOnly option. * volume: skip read-only mounts when picking EC shard copy targets A shard copy landing on a read-only mount fails at write time; offer only writable locations like the other placement paths do. * volume: mirror the read-only mount handling in the Rust volume server The .dat open only fell back to read-only on EACCES, so a real read-only mount (EROFS) still failed the load. The index side already degrades once no_write_or_delete is set. Add is_read_only_mount via access(W_OK) == EROFS, mark DiskLocation.read_only_mount, and mirror the Go checks: no free slots, heartbeat max = used, no vid-stamp writes, no EC shard copy targets, and a read-only index dir forces the volume read-only. * volume: write vid stamps while only the index dir is read-only * filer: skip the write-bit check when a leveldb store opens read-only * filer: serve reads of buckets with no leveldb3 dir on a read-only store * volume: derive a stable dir uuid when a read-only mount cannot store one * master: keep the read-only raft copy across restarts so votes survive * volume: index a missing .idx in memory on a read-only mount * volume: release the .idx handle after an in-memory read-only load * volume: mount EC shards read-only when their index dir is on a read-only mount * volume: mark the index dir read-only before the tail repair opens .idx * volume: write vid stamps while only the index dir is read-only (Rust) * volume: derive a stable dir uuid and report no free slots on a read-only mount (Rust) * volume: index a missing .idx in memory on a read-only mount (Rust) * volume: refuse an EC copy whose shard owner is on a read-only disk * master: key the read-only raft copy by the absolute -mdir Two masters sharing the same relative -mdir text from different working directories would have collided on one temp copy and shared raft state. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: mix device and host into the read-only dir uuid Deriving the uuid from the path alone let independent servers at the same mount point report one uuid, which RegisterUuids rejects. Hash in the device id and hostname so restarts stay stable but disks differ. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: vic6996-web <320442181+vic6996-web@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-26 09:28:49 -07:00
Chris Lu	53834117a6	mini: lift the post-mint allocation hold on first inventory report (#12061) ... * mini: lift the post-mint allocation hold on first inventory report DeferAllocation armed a fixed warmup window whenever the leader minted a fresh topology id, so the first write on a brand-new weed mini blocked for the whole window even though no prior cluster state could exist. The window still has to cover a rebuilt master's veteran servers, which report on their first post-reconnect heartbeat. Mini has exactly one possible reporter — the embedded volume server, whose first heartbeat carries the full inventory — so hold allocation only until that report is applied. The bound raise and topology-id freeze checks run while the heartbeat is processed, so a wiped master state still raises the bound or trips the freeze before the hold lifts. Standalone weed master and weed server keep the fixed window: volume servers outside the process can still be on their way. * master: apply queued topology-id reports before opening the allocation gate SetTopologyInitPending(false) cleared the pending flag before replaying topology ids stashed while the local id was still being minted. A NextVolumeId landing in the drain window saw no gate and no freeze yet. The post-mint allocation hold used to cover the gap by delaying the winner past the freeze; once the hold can release on first inventory, the gate must stay closed until the queued checks have run. * mini: gate the first-report release on single-master mode releaseAllocOnFirstReport assumes the embedded volume server is the only possible reporter of prior cluster state. A mini started with -master.peers joins other masters, whose own volume servers can still be on their way when the first report lands, so keep the regular quiet-pulse hold there. * master: check reported topology id on empty full reports A HasNoVolumes heartbeat carries the same LastSeenTopologyId a populated list does, but the mismatch freeze only ran for non-empty reports. An EC-only veteran could therefore release the post-mint allocation hold through NoteInventoryApplied while its stale topology id never reached CheckReportedTopologyId. * master: drain the topology-id stash inside the init-gate lock CheckReportedTopologyId only held reportedTopologyIdsLock while stashing, so a report in flight during SetTopologyInitPending(false) could land its freeze after topologyInitPending cleared, letting a NextVolumeId slip through the gap. The whole check now runs under the lock, and the gate is released inside the same critical section as the drain. * mini, master: pin and enforce the single volume-server reporter isSingleMasterMode only checks master peers; SendHeartbeat still accepted any volume server, so a late foreign report could carry prior cluster state after the first-report release dropped the hold. Mini now pins the embedded volume server's address and the master refuses registration from any other reporter while ReleaseAllocOnFirstReport is set. * master: admit the volume-server reporter before applying heartbeat state * test: cover SendHeartbeat admission ordering A refused reporter must not freeze growth or raise the id bound, and the pinned reporter's stale topology id still must. * mini: re-derive the single-reporter release on every run The package-level master options survive in-process reruns, so a multi-master mini following a single-master one must not inherit the first-report release or a stale reporter pin. * master: admit addressless first heartbeats under the reporter pin An unregistered heartbeat with no Ip skipped the single-reporter admission check but still reached the seen-bound raise and the topology-id freeze before the delta-heartbeat skip. Admission now runs for every unregistered heartbeat; an empty address can never equal the pin.	2026-10-25 09:25:39 -07:00
Chris LuandDevin	da1d7444b8	master: shorten the first-write hold after a fresh topology id (#12059) ... * master: release the post-mint allocation hold after a quiet pulse DeferAllocation armed a fixed warmup window whenever the leader minted a fresh topology id, so the first write on a brand-new cluster blocked for the whole window even though no prior state could exist. The window still has to cover a rebuilt master's veteran servers, which report on their first post-reconnect heartbeat. Track the last heartbeat that could carry prior cluster state — a server joining, held volumes or shards, a nonzero id bound, or a foreign topology id — and lift the hold one pulse after it goes quiet, capped by the original window. * master: make the post-mint allocation hold configurable master.volume_growth.allocation_deferral overrides the warmup window a leader holds volume id allocation for after minting a fresh topology id. Unset keeps the existing window; "0s" disables the hold for setups that know the cluster is fresh (test harnesses, single-node labs); any other duration replaces the cap while the quiet-pulse early release still applies. * master: re-check the allocation deadline while it is held NextVolumeId slept for the whole computed delay, so a heartbeat landing mid-sleep that pushed the quiet deadline out was ignored — the pending allocation could still run ahead of a straggler's report. Wait in short slices, re-evaluating the deadline and the growth freeze each round. * master: count only ids above the issued bound as prior-state evidence Heartbeat echoes of volumes and shards this master allocated during the hold refreshed the quiet window, so each growth allocation re-extended the deferral until the hard cap. Evidence now requires an id beyond the bound this leader has issued, a higher reported bound, or a foreign topology id; a joining server (dn == nil) still counts once. * master: reject an unparseable allocation_deferral instead of disabling the hold Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: do not count a skipped heartbeat as a joining server Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: assert the allocation hold extends past the original quiet deadline Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: release the post-mint hold after two quiet pulses One pulse proved short for a veteran that reconnects on a staggered restart: with only five seconds of quiet, allocation could resume and reuse ids before a slower server landed its first heartbeat. Two pulses still halves the fresh-cluster first-write stall versus the full warmup cap, and known-fresh setups keep the "0s" override. * master: pick once more when assign times out as growth concludes The pick loop's final attempt can predate the growth it triggered concluding: growth registers its volumes in the layout before clearing the pending flag, so a timeout landing on that boundary used to shed a hard 406 over volumes that had just arrived. Pick once more after the loop in both the http and grpc assign paths before deciding how to shed. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-25 06:17:42 -07:00
Chris Lu	85470416cb	volume: overlap the local needle write with the replica fan-out (#10074) ... * volume: overlap the local needle write with the replica fan-out ReplicatedWrite wrote the needle to local disk and then fanned it out to the replicas serially, so a replicated write cost local + replicas. The needle is already fully buffered, so the two can run at once. Snapshot the fields the fan-out reads first (the local write stamps the volume TTL and updates needle flags, so the replica goroutine must not touch the shared needle), then run the local write and the replica fan-out concurrently and ack only after both. Where local disk and replica network are separate resources the write latency drops to max(local, replicas) instead of their sum; co-located on one disk the two writes contend, so this helps distributed placements, not single-host tests. * volume: return the local isUnchanged from the write goroutine * volume: trim overlap-write comments * volume-rs: overlap the replica fan-out with the local needle write * volume: test the overlapped local write and replica fan-out * volume: decode replica pairs only when replicas exist A single-copy or incoming replicate write never uses the decoded pair map, so a malformed Pairs blob only burned an error counter and a log line. Gate the decode on replicaCount. * volume: release the stalled replica on test failure paths An early t.Fatal left the httptest handler blocked on releaseReplica, so the deferred Close hung instead of reporting the failure. Release idempotently ahead of the server close. * test: mismatched If-None-Match serves the object over If-Modified-Since The precedence case asserted the old If-Modified-Since-first ordering and expected 304. The volume servers now evaluate If-None-Match first, so a non-matching tag must return 200 with the body even when If-Modified-Since would allow 304 (RFC 9110 13.1.3).	2026-10-24 15:16:16 -07:00
Chris LuandDevin	514c0c1e99	topology: stop a stale duplicate-vid report from re-quarantining the survivor (#12032) ... * topology: drop a duplicate-vid report whose own copy is already gone flagDuplicateVolumeId runs on a goroutine after registration, so by the time it executes the reporter's copy may have been unregistered (a finished move drops the losing copy). The surviving copies are then legitimate, yet the stale report still quarantined them and stored a phantom (node, vid) key for the departed reporter. Skip the marks when the reporter no longer serves the volume under its own layout key. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(topology): cover a stale duplicate report against a lone survivor The existing tests only flag the conflict before the losing copy is unregistered. Run the delayed duplicate check after the reporter's copy is already gone and assert nothing is quarantined and the survivor stays writable, so removing the stale-report guard fails the test outright instead of needing a lucky schedule. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(topology): cover the enforceQuarantine-driven release retry A failed writable push re-arms the record, and the next volume report drains that retry through enforceQuarantine itself - a different entry point than the pulse-driven pass the other tests exercise. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 07:05:46 -07:00
Chris LuandDevin	3e5aa8c3b3	topology: mark replica quarantined before recording the pair (#11983) ... * topology: serialize quarantine records with their layout marks A quarantine publishes a (node, vid) record and marks the replica read-only at the layout level; release and heartbeat flag updates must observe that pair as one state. Store-then-mark let a concurrent release delete the record before the mark landed, stranding the surviving replica read-only; mark-then-store let a heartbeat clear the mark before the record was visible, returning a conflicting replica to the writable list. quarantineLock now covers every record<->mark pair: flagDuplicateVolumeId's quarantine, releaseResolvedQuarantine's undo, the isQuarantined-gated marks on registration, and the quarantine-folded read-only flag writes on heartbeat paths. Remote server pushes stay outside the lock; a push that lands after the record was released is undone, and a failed release push re-adds the record for a later retry. The release test now waits for in-flight async flags to settle instead of asserting on a single read. * topology: fold quarantine into RegisterVolume's flag rebuild RegisterVolume rewrites every replica's location read-only flag from what its node last reported, and reported flags deliberately exclude quarantine. Any registration of a shared vid -- a reconnected replica, a new copy, the self-heal and delta-heartbeat re-register paths -- therefore cleared the quarantine mark on every replica in the location, while the gate afterwards only re-marked the registering node. Two call sites had no gate at all. RegisterVolume now takes an isQuarantined predicate and evaluates it under accessLock for every replica it touches, so the mark is never dropped in the first place: a release unmarks after the rebuild, and a quarantine mark re-applies on the next rebuild, with no window between. * topology: keep quarantine when a vacuum commit re-adds a replica SetVolumeAvailable re-adds the replica into the location list; a re-added slot starts with a cleared read-only flag, so a quarantined replica came back flag-free and the next ensureCorrectWritables could return a conflicting volume to the writable list. Pass the quarantine state in and set the flag back under the same accessLock, with the caller holding quarantineLock so release cannot interleave between the check and the mark. * topology: merge quarantine records instead of overwriting them enforceQuarantine wrote back the state copy it loaded before its remote push, so a concurrent writer's enforced=true was lost: the server stayed read-only while the record said unenforced, and a later release skipped the writable push it owed. The write-back now reloads the record under quarantineLock and merges -- enforced wins from either side, reported is taken from the newer pre-enforcement observation only while no push has landed yet. All three compensating writable pushes had the same hole in reverse: a failed un-quarantine left the server read-only with no record to retry from, and release's restore blindly overwrote any record a fresh flag pass had created in between. They now go through restoreQuarantineRecord, which adds the record only when absent and otherwise sets enforced on the current one, preserving its reported flag. * topology: hold quarantineLock over registration's quarantine check-and-mark RegisterVolume reads the quarantine record and marks the replica read-only under accessLock, but the pair ran without quarantineLock. A release could push writable, watch the record get re-read, and then delete it: the replica stayed read-only with no record left to clear, and unchanged writable heartbeats skip SetReplicaReadOnlyFlag so the mark never repaired. Holding quarantineLock around the check-and-mark in all three production callers serializes it against the release clear-and-delete. * topology: serialize remote quarantine pushes behind a record recheck releaseResolvedQuarantine deletes the record under quarantineLock and sends the writable request after unlock. A conflict flagged while that request is in flight creates a fresh record and marks the replica read-only again, and the older writable request can still land last, leaving the server writable under an active quarantine. The reverse flap strands a stale read-only mark the same way. Route every QuarantineVolumeFn call through pushQuarantine, which holds quarantinePushLock across the remote call and re-checks the record under quarantineLock right before sending: a read-only push requires the pair to exist, a writable push requires it to be gone. A skipped push means the decision already moved on, so the final server state always follows the newest record. * topology: serialize quarantine pushes per replica, not globally A single quarantinePushLock held across the remote mark call let one slow volume server (up to the 10s client timeout) delay pushes for unrelated replicas. Keying the lock by (node, vid) keeps pushes to one replica ordered in record-decision order while pushes to others run in parallel. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: restore the layout read-only mark with a released record A failed writable push re-armed the quarantine record but left the replica marked writable in the layout, so PickForWrite could route to a replica the server still holds read-only. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: wait out async duplicate checks in the release-retry test TestFailedReleasePushRestoresQuarantineMark asserts layout marks around a release pass, but the two RegisterVolumeLayout calls each start a background flagDuplicateVolumeId that can still be mid-flight. A late mark could land after the final release and fail the writable assertion even though the code behaved correctly. Track the async passes in a WaitGroup and drain them before the explicit flag call so the rest of the test is deterministic. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: keep quarantine when a replica's layout key is unreadable * topology: keep a failed writable push's restore ordered under the push lock * topology: retry the quarantine release on every duplicate check * ci: isolate Java and S3 test servers on persistent runners * ci: drop the duplicated -ip flags in the lifecycle test server The bind flags were already on the command; the isolation pass added a second pair. * topology: retry a failed quarantine writable push on the next report * topology: retry pending quarantine releases off the leader pulse A failed writable push re-arms the quarantine record with a retry that only enforceQuarantine fires, and that runs only for volumes a heartbeat carries. Once the server has reported the replica read-only it drops out of the delta reports, and restoreQuarantineRecord keeps the stored flag matching the server's so the digests agree and no full list is requested - the pending release then never retries and the idle volume stays read-only forever. Run the same release pass from the leader's pulse loop so a pending retry no longer waits on a report. * topology: exercise the pulse-driven quarantine release retry --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 07:00:45 -07:00
Chris LuandDevin	5e01caa522	master: cap parallel vacuums per rack via master.vacuum.max_parallel_per_rack (#11935) ... * master: cap parallel vacuums per rack via -maxParallelVacuumPerRack Hosts that run one volume server per disk under a single rack label could not bound host-level vacuum I/O: maxParallelVacuumPerServer limits each process, so every disk on the machine compacted at once. A second quota map keyed on rack id debits alongside the per-server one, so at most N volume copies compact in parallel within a rack. Defaults to 0 (no limit); the flag is wired on master, server, mini and master.follower. * master: resolve a data node's rack key once per vacuum sweep A volume server that unregisters mid-sweep loses its parent node, so the quota credit path calling dn.GetRack() panicked and crashed the master. Rack names are also local to their data center, so same-named racks in different centers shared one quota bucket. Each node's rack key is now captured when the limiters are built and carries the data center id, so checks, debits and credits never touch the live parent pointer. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: reserve vacuum quota per needed copy count, not per copy Each copy of a volume checked the shared server/rack quota independently, so several same-rack replicas all passed against one remaining slot and the debits drove the bucket negative. Needs are now counted per server and per rack first; a volume whose own copies exceed a cap reserves the whole bucket once it is free instead of pending forever. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master.follower: register -maxParallelVacuumPerRack as a flag It was only a fixed default, so passing the flag on the command line failed flag parsing. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: key the rack vacuum quota by rack node identity Joining data center and rack names with "/" still let two different racks share one bucket when the names themselves contained slashes. Key the limiter by the rack node instead, which is unique per rack regardless of naming. * master: keep the tiered-replica vacuum test through the rebase * master: document 0 as no limit for -maxParallelVacuumPerServer The clamped per-server need makes a non-positive cap mean "no limit", matching the new rack flag; say so in the flag help. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: key the rack-key cache by data node identity Node ids are only unique within a rack, so two servers that share an id in different racks would have reused the first server's cached rack and escaped the per-rack vacuum cap. * master: read the per-rack vacuum cap from master.toml -maxParallelVacuumPerRack becomes the master.max_parallel_vacuum_per_rack config key, so the new master limit is set the same way as other master options instead of a flag wired into every command that embeds a master. * master: let master.max_parallel_vacuum_per_server override the flag The released -maxParallelVacuumPerServer flag keeps working, but a value in master.toml now wins when set, matching how master.mdir already overrides -mdir. * master: group the vacuum caps under master.vacuum in master.toml Rename master.max_parallel_vacuum_per_rack to master.vacuum.max_parallel_per_rack and master.max_parallel_vacuum_per_server to master.vacuum.max_parallel_per_server, so the keys sit in one subsystem table like master.volume_growth and master.maintenance. * master: deprecate -maxParallelVacuumPerServer for master.toml The flag still works - and is overridden by master.vacuum.max_parallel_per_server when set - but the help text now points operators at the config key, matching the codebase style of help-text-only deprecation. * topology: read node parent atomically UnlinkChildNode clears a data node's parent under the rack's lock while readers like the vacuum sweep's rack capture reach it with no shared lock, so a torn interface read could crash the master. Store the parent in an atomic.Pointer and nil-guard the detached walk in DataNode.GetTopology. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-23 22:41:36 -07:00
Chris LuandDevin	168789d5f3	master: cap writable volumes per collection via -maxWritableVolumes (#11923) ... * master: cap writable volumes per collection via -maxWritableVolumes Some deployments serialize disk writes (e.g. an Unraid array where concurrent writes to multiple disks collapse throughput). With more than one writable volume in a collection, the master's weighted pick spreads assignments over all of them, so several disks get written at once. -maxWritableVolumes=N narrows the pick candidates to the N fullest writable volumes, so a collection's writes concentrate on one (or few) volumes at a time. 0 keeps the current spread-everywhere behavior. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: give master.follower a maxWritableVolumes default * master: cap writable volumes inside the requested location * master: pick the capped volumes without sorting the writable list * master: assert the capped candidate set instead of random pick outcomes * master: describe -maxWritableVolumes as a per-layout cap --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-23 17:04:36 -07:00
3 people	ac319fe318	volume: never compact a volume tiered to remote storage (#11895) ... * volume: never compact a volume tiered to remote storage A tiered volume's data backend is the remote object, but compaction reads the local .dat path. When tiering removed the local .dat, every vacuum of a tiered volume with garbage fails with "no such file or directory", and the master's sweep and the worker's vacuum task keep selecting it. When the local .dat was kept, the compaction succeeds against the stale copy and the commit reloads the volume to read the remote object (old layout) through the compacted index, so live needles stop reading. Mirror the Rust volume server (#11545): - CompactByIndex and CompactByVolumeData refuse a volume with a remote file before creating .cpd/.cpx ("volume N is tiered to remote storage, cannot compact"). - CommitCompact refuses under the data-file locks when the volume became tiered during the copy, and removes .cpd/.cpx. - Store.CheckCompactVolume reports no garbage for a tiered volume, so the master's vacuum and volume.vacuum do not select it. - The worker's vacuum detection skips a volume with a remote copy, as balance and EC detection already do. Tests (each fails on master): TestCompactRefusesTieredVolume, TestCommitCompactRefusesVolumeTieredDuringCopy (on master the commit succeeds and a surviving needle no longer reads), TestPickPrimaryReplicaSkipsTieredVolumes. * volume: skip tiered replicas in vacuum selection, keep their real ratio Reporting a zero garbage ratio for a tiered volume hid it from volume.tier.compact, whose preflight reads that ratio while the volume is still remote, and did not exclude it from a vacuum with a zero threshold. Report the real ratio again, as the Rust volume server does, and have the master's vacuum check skip a replica whose volume info names remote storage, whatever the threshold or request. When CommitCompact refuses a volume tiered during the copy, also close the temporary needle map and remove .cpldb: nothing else would, since no later vacuum selects the volume. The tiered-compaction test no longer depends on random needle sizes reaching a 0.3 garbage ratio. * volume tier: do not attach a copy of a .dat a compaction replaced The upload copies the .dat, then records the remote file in the .vif and switches the backend. A compaction commit in between swaps in a compacted .dat and index; the upload then served the old-layout remote object through the new index, so live needles stopped reading. CommitCompact's tiered check cannot see this: the volume is not tiered until the switch. AttachRemoteDatFile records the remote file and switches the backend under the data-file locks, and refuses when the backend is no longer the file that was copied. The upload then removes the unused remote copy. * volume tier: keep a committed .vif entry when only the dir fsync fails AttachRemoteDatFile rolled back the in-memory remote entry on any SaveVolumeInfo error. A NotCrashDurableError comes after the rename, so the .vif on disk already names the copy; with the entry gone, the upload handler deleted that object, and a reload would serve from a missing remote file. Keep the entry and switch to the copy as the .vif says, and still return the error so the caller keeps the local .dat. * vacuum worker: filter tiered replicas instead of skipping the volume A partially tiered volume is the leftover of an interrupted volume.tier.upload; skipping the whole volume left garbage on its local replicas unreclaimed. Per-replica filtering matches the master's batchVacuumVolumeCheck and is safe: a tiered replica refuses both compaction and commit. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2026-10-21 20:16:02 -07:00
Chris Lu	4d06742664	Detect and quarantine duplicate volume ids (#11763) (#11828) ... * master: detect and quarantine duplicate volume ids across layouts (#11763) * shell: volume.check.duplicate_ids audit command (#11763) * master: persist duplicate-vid quarantine across heartbeats (#11763) * master: quarantine all copies of a duplicated volume id Near-simultaneous registrations let each side quarantine only the other, so arrival order decided which copy stayed writable. A confirmed cross-layout collision means either copy could be the impostor, so flag every colliding copy read-only regardless of registration order. * topology: return a copy from VolumeLayout.Lookup Callers iterating the returned slice raced with registrations and removals replacing the location list in place. * master: release duplicate-vid quarantine when the conflict resolves A cross-node volume.move briefly shows the same id on two nodes under different disk layouts; permanent quarantine would pin a valid target read-only. Keep the first-reported state per quarantined copy and lift the marks once the conflicting copy unregisters, so only a persistent collision stays read-only. * shell: include ec shards in volume.check.duplicate_ids Ec-only volumes share the global id space. Compare shard collections against regular ones while allowing shards and regular copies of the same encoded volume. * master: keep quarantine release current and cover all removal paths Track the latest server-reported read-only flag while quarantined so a release restores the server s true state, and run the release check on collection deletion and data-node removal too, not only explicit volume unregistration. * shell: do not flag ec-only volumes as duplicates An ec-only volume has no regular copy, so its shard collections had nothing to compare against and always read as foreign. * master: quarantine every conflicting layout at the layout level (#11763) - collect all conflicting layouts, not just the first, so a vid spanning three layouts does not leave one writable - mark quarantine on the location list only: folding it into the stored report made the volume digest permanently disagree with the server and forced a full volume list every heartbeat - isAllWritable now honors the location read-only mark so a quarantined copy stays out of writables through resyncs and re-registration - re-check the conflict before keeping marks, and do not clear a node's quarantine while it still holds the vid in another layout - log the full layout keys on both sides of the conflict * master: push duplicate-vid quarantine to the volume server (#11763) A quarantined copy stayed writable on the volume server itself for a client holding an existing file id, since the master-only mark never reached the server's local write path. Quarantine now also calls VolumeMarkReadonly (can_delete so expiry still drains), is re-pushed if the server comes back reporting it writable, and VolumeMarkWritable runs when the conflict resolves and the server had reported writable. * master: retry unenforced quarantine marks and restore server writability on release (#11763) Dial the advertised grpc port, return mark errors, and keep a quarantine entry until the writable push succeeds so a released copy does not stay read-only on the server. * master: treat rp-only layout differences as an in-flight replication change (#11763) volume.configure.replication rolls through nodes one at a time, so a volume legitimately reports the old and new replica placement while it settles. Layouts differing only in rp within the same collection are a supported transition, not a reused id. * shell: group ec shards by collection in the duplicate id audit (#11763) EC shards of one volume legitimately span disk types when DiskTypePrefer spills, so grouping by disk type reports healthy volumes as duplicates.	2026-10-20 07:37:33 -07:00
Chris Lu	2874bd05fb	Freeze volume growth on topology id mismatch (#11763) (#11820) ... * pb: ResumeVolumeGrowth rpc for operator-driven growth resume (#11763) * master: freeze volume growth on topology id mismatch (#11763) * shell: volume.growth.resume command (#11763) * master: replicate the growth freeze and gate it on veteran heartbeats (#11763) * master: block allocation while the topology id is being minted DeferAllocation only armed after the topology id persisted, so growth requests arriving during init ran ahead of the warmup window. Gate NextVolumeId on an init-pending flag spanning the whole mint. * master: stash reported topology ids until the local id is minted (#11763) * drain stashed topology id reports when the id arrives outside the mint path * persist the growth-freeze flag in the mirror file and raft snapshots * log hashicorp raft marshal and apply errors in applyGrowthFreeze	2026-10-20 07:34:31 -07:00
Chris Lu	0b3ed4502f	Design + master mirror of max volume id outside raft (#11763) (#11818) ... * design: durable volume-id allocation boundary (#11763) * master: mirror max volume id to a file outside raft (#11763) The allocation bound lived only in raft state, so a master that lost it rebuilt the bound from whatever volume servers were connected and could reissue ids held by offline servers. NextVolumeId now persists the bound to <mdir>/max_volume_id before the raft apply, and both raft implementations persist on apply and recovery so follower mirrors stay current. The file also carries the topology id, which survives raft state cleanup the same way the snapshot recovery did. * master: serialize and fully sync the max volume id mirror (#11763) * master: rewrite the mirror when the topology id changes at the same bound A same-bound raft apply carrying a new topology id was skipped by the lastPersistedMaxVolumeId check, leaving the mirrored topology id empty until the next allocation. Track the mirrored topology id too and rewrite when either value differs. * drop the design doc from the PR stack	2026-10-19 02:42:42 -07:00
3 people	3c6700c310	volume balance: stop planning moves onto disk types a server does not have (#11762) ... * fix(topology): capacity queries no longer create empty disk-type entries AvailableSpaceFor and CapacityFor called getOrCreateDisk, so asking a node whether it had room for a disk type it lacks created a zeroed DiskUsageCounts entry for that type. Volume growth walks every data center, rack and data node, and ToDataNodeInfo publishes diskUsages, so every node ended up advertising an empty disk (MaxVolumeCount 0, DiskId 0) of every type ever grown: HDD-only servers showed an "ssd" disk and SSD-only servers an "hdd" disk. Read the counters through a non-creating getDisk; a missing entry answers 0, the same numbers as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * fix(admin): don't let an empty disk-type entry shadow a node's real disk UpdateTopology keys disks by node and disk id. The master published an empty entry (no capacity, nothing on it, DiskId 0) for every disk type a capacity check had asked about, and it collided with the node's real disk 0; map iteration order decided which survived. An SSD-only node intermittently showed an empty HDD disk, and an HDD node lost its disk -- and with it its volumes from the volume index, hence "Could not find volume N on source server" in balance detection. IsShadowedEmptyDiskEntry matches an empty entry whose disk id another DiskInfo of the node, one with capacity or data, already uses. UpdateTopology and CountTopologyResources skip it. A lone empty entry is still an empty disk of unknown capacity and is kept, as GetDisksWithEffectiveCapacity expects. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * fix(balance): don't seed balance candidates from shadowed empty disk entries detectForDiskType registered every node with a DiskInfos entry of the type as a zero-volume candidate. An empty entry the master made up for a type the node lacks made that node the least-loaded destination, and its zero max switched the whole disk type to raw-count balancing. The volume server then refused every move: "no space left hdd" / "no space left ssd". Skip entries IsShadowedEmptyDiskEntry matches when seeding candidates. A server that holds volumes of the type but reports no max still enters through the metrics and still triggers the raw-count fallback. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * fix(admin,balance): ignore made-up disk-type entries at any disk id The made-up entry is published with DiskId 0, but it need not collide with a real disk: when location 0 is unavailable (not reported) the node's usable disk is location 1. IsShadowedEmptyDiskEntry required a collision, so such an entry survived both UpdateTopology -- where the unknown-capacity fallback offers it to planners -- and balance detection, which planned moves onto it again. Its disk id cannot tell it apart: a DiskInfo with no volumes, no EC shards and no per-disk max is published with DiskId 0 whichever disk it stands for. IsPhantomDiskTypeEntry (renamed, since nothing needs to be shadowed now) matches an empty entry on any node that has a disk of another type with capacity or data. A real disk looks the same only while it offers no volume slot, when it can take no volume either. A node whose entries are all empty keeps them as disks of unknown capacity, as before. Reported by review on #11762. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * test(balance): assert where the real disks land in the empty-entry test Check through GetNodeDisks that each server's real disk sits at the subtest's disk id before planning. The id-1 case already landed on physical disk 1 (SplitByPhysicalDisk folds all-zero record ids onto the entry's DiskId), so this only makes the setup explicit; on the first revision of the fix it shows the real disk at 1 beside the made-up entry at 0. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq * fix(topology,balance,ec): don't publish or trust usage-only disk-type entries ToDataNodeInfo started from the raw disk-usage counters, so a node that was asked for capacity of a disk type it lacks kept advertising an empty DiskInfo for that type even though no such disk was ever reported. Skip usage entries that have no disk child and hold all-zero counters. The same stale entries still misled planners that read DiskInfos directly: the balance plugin's move precondition treated a phantom entry as proof a target has that disk type, and the EC balancer counted it as a matching disk. Filter them with IsPhantomDiskTypeEntry. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-10-16 23:41:26 -07:00
3 people	492e700d1f	Volume grow avg fill threshold (#11725) ... * master: add conditional grow by average fill ratio to VolumeGrow gRPC Callers that pre-grow volumes on fill level currently have to list all volumes of a collection and sum sizes themselves, which is expensive on clusters with millions of volumes. The master already tracks each writable volume's effective size in memory, so it can answer the question directly. Add grow_avg_fill_threshold to VolumeGrowRequest: when set (e.g. 0.6), growth only happens once the layout's writable volumes have reached that average fill ratio. A layout with no writable volumes skips the check, since that is the case growth exists to fix. VolumeGrowResponse now reports the observed avg_fill and writable_count whether or not growth was triggered. * master: unit tests for WritableAvgFill and FindVolumeLayout Covers the average-fill computation (mixed fills, no writables, a writable volume without a size-tracking entry yet) and that FindVolumeLayout resolves existing layouts without creating missing collections. * address review: report avg_fill and writable_count for every VolumeGrow The stats were only read when grow_avg_fill_threshold was set, so a request without a threshold got AvgFill=0 and WritableCount=0 even when writable volumes existed. Read the layout stats unconditionally and use the threshold only to decide whether growth happens. * master: scope the grow fill check to the requested target A VolumeGrow aimed at a data center, rack or node must not skip growth because fill elsewhere keeps the layout average below the threshold. Measure only writables serving the target; a target with none still grows. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-13 05:30:03 +08:00
4 people	41260a55f6	master: record writableVolumeCount on HTTP /dir/assign like gRPC Assign (#11716) ... * master: extract the periodic per-layout grow decision Move the must-grow and crowded-step decision of the periodic growth scan in ProcessGrowRequest into periodicGrowCount, a pure function of the layout's lastGrowCount, writable and crowded counts, so it can be tested without running the scan loop or growing real volumes. No behavior change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: record writableVolumeCount on HTTP /dir/assign The periodic growth scan (#5781) keeps a layout supplied ahead of demand only when it knows how many writable volumes clients want: lastGrowCount, which gRPC Assign records with SetLastGrowCount. HTTP /dir/assign parsed writableVolumeCount but only used it to size the emergency grow when no writable volume was left, so the scan always saw lastGrowCount 0 for HTTP-only writers. Since #5821 removed the per-assign threshold grow, such a layout only grew by the rack-aware step of 2, and only once every writable volume was crowded. When a batch of volumes filled in lockstep, those 2 took every write and could fill before the next scan, leaving no writable volumes until an assign-triggered emergency grow landed, with 503s for every other assign meanwhile. Record writableVolumeCount on HTTP assigns as gRPC Assign does. Clients that do not send it are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: reject unknown rack/dataNode before recording grow targets An assign naming a rack or data node outside the topology can never be served, but lastGrowCount was recorded first. The periodic growth scan drops the requested placement, so it kept topping the layout up with volumes the requester could not use. Both assign paths now validate the placement like they already validate the data center. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: scope placement existence checks to the requested path A rack in another data center or a data node under another rack passed validation yet could never serve the request, and the grow count was still recorded. The rack and data-node lookups now take the requested parents, matching the filters the placement engine applies. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-12 18:10:41 +08:00
Chris Lu	0ca484c354	vacuum: bound master vacuum RPCs with phase deadlines (#11579) ... * vacuum: bound the commit RPC with a phase deadline VacuumVolumeCommit ran on context.Background(), so a volume server that keeps the call pending would hold the topology-wide vacuum guard forever and every later sweep would be skipped. Give the call a deadline scaled like the existing phase waits (one minute per GB of the volume size limit) so a stalled commit ends as an error instead of blocking the sweep; the timeout is a var so tests can shrink it. * vacuum: bound the replica status probe with a phase deadline The VolumeStatus call on replicas that were not compacted also ran on context.Background(), so a stalled replica could pin the sweep the same way a stalled commit can. Give it the same per-phase deadline. * vacuum: bound the cleanup RPC with a phase deadline VacuumVolumeCleanup also ran on context.Background(); a stalled server would keep the sweep worker and the shared vacuum guard pending forever. Give it the same per-phase deadline. * vacuum: let the check and compact phase waits cancel their RPCs The coordinator wait timers fired while the check and compact calls still ran on context.Background(), so the sweep gave up but the RPC goroutine stayed until the server answered, and a compact stream kept writing on the server. Share one deadline context between the wait and the calls so an expired wait actually cancels them. * vacuum: test that a stalled volume server releases the vacuum guard A fake volume server keeps one vacuum-phase RPC pending until the client context is cancelled. Before the phase deadlines, Vacuum never returned and vacuumLockCounter stayed held; now each phase cancels on its deadline and the guard is free for the next request. * volume: stop compaction at the next needle when the client cancels The progress callback only noticed a gone client when a 128 MiB report failed to send, so an aborted VacuumVolumeCompact kept copying for up to a whole interval while the master had already moved on to cleanup. Check the stream context on every needle, the same early return the Rust volume server does with tx.is_closed(). * vacuum: assert the stalled phase RPC is cancelled, not just bypassed The check and compact coordinator waits already returned on timeout before the deadlines existed, so a regression that put the calls back on context.Background() would pass unnoticed. Wait for the fake server to report that the phase RPC context ended. * vacuum: give the stalled-RPC test room to reach the handler The 50ms phase budget starts before goroutine scheduling and the gRPC dial, so a busy test host could expire it before the fake server saw the call. Raise the override to 250ms; the test still finishes in about a second. * vacuum: describe the phase deadline as scaled, not per-GB The formula keeps the exact expression the check and compact waits already used (floor plus one at 1 GiB granularity); it is a backstop, not a per-GB SLO.	2026-10-03 12:32:42 +08:00
ihnokim	11e8c4c288	master: follow heartbeat read-only changes in the layout's replica flag (#11527) ... A replica's read-only flag in the volume layout only moved on registration and on volume.mark. A change that arrived in the regular heartbeat updated the node's record, which the writable list follows, but not the layout flag, which the vacuum sweep reads. So the sweep kept trying volumes on a disk that had gone read-only while the server ran, and after a restart it skipped volumes that had since become writable again until the next restart (issue #11516). Apply the reported state to the flag for every changed volume. Only the flag: the writable list stays with EnsureCorrectWritables and its capacity guards.	2026-10-01 01:09:09 +08:00
Chris LuandDevin	0978e7f833	vacuum: keep disk-full read-only volumes reclaimable (#11519) ... * storage/topology: keep disk-full read-only volumes vacuumable The vacuum sweep skipped every read-only replica, so a volume that went read-only because its disk filled could never reclaim its garbage — the exact situation compaction exists for. The volume server now reports disk_space_low in VacuumVolumeCheckResponse, and the sweep skips a read-only replica only when the flag is clear. An explicit volumeId vacuum is unaffected: it already bypassed the read-only rule. The field takes number 4: 2 and 3 are downstream-allocated for tombstone retention, keeping the wire merge clean. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * storage: measure vacuum free space against live bytes The pre-compaction space check required the current .dat + .idx size free, which includes the garbage being reclaimed — on a nearly full disk that estimate can never fit, so the volume stayed garbage-bound forever. Measure against the estimated compacted output instead: superblock plus live index entries plus live content bytes, with the existing ten percent buffer unchanged. Mirrors the same check in the Rust volume server. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vacuum: count per-needle framing in the compacted-size estimate The live-bytes estimate covered each live needle's content and index entry but not its .dat framing (header, checksum, timestamp, padding — ~32 bytes on version 3). For small-needle volumes that is more than the 10% headroom, so a disk with space between the estimate and the real output still ran out mid-compaction. Rust side mirrors the same formula. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * storage: report disk_space_low only when it is the sole read-only cause Review feedback (ihnokim, greptile, devin): a volume read-only for low disk space AND an operator mark or I/O quarantine was still eligible for the automatic sweep, rewriting a copy meant to stay protected. The flag now reports only the benign sole-cause case in both servers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: fail closed when the read-only lookup misses in the sweep A heartbeat can drop the volume from the DataNode cache between the location-list copy and VacuumVolumeCheck; a lookup error previously skipped the read-only check entirely. Review feedback (coderabbit). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-30 17:32:41 +08:00
yi111andChris Lu	150a69fe11	master: make volume capacity reservation timeout configurable (#11426) (#11497) ... * master: make volume capacity reservation timeout configurable (#11426) * master: expire reservations on reads, fix int timeout units - AvailableSpaceForReservation now expires reservations too: a node that is full of reservations is filtered out before TryReserveCapacity can clean them, which stranded expired capacity indefinitely. - Drop TryReserveCapacityWithTimeout: a per-call timeout lets one caller expire another's live reservations, and the Node interface stays stable for implementations outside this tree. - parseReservationTimeout no longer routes integer values through GetDuration, which read them as nanoseconds; bare numbers are seconds. The 5m fallback is now the shared DefaultReservationTimeout. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2026-09-28 18:40:11 +08:00
Chris LuandDevin	a261f90e18	vacuum: let the sweep release volumes that stay empty and quiet (#11477) ... * vacuum: let the sweep release volumes that stay empty and quiet Vacuuming reclaims bytes but not slots: a fully emptied volume stays registered to its collection forever, and since growth is gated only on slot count a store at 99% free disk can still refuse writes to other collections (#11429). volume.deleteEmpty exists but is manual-only. With -vacuumDeleteEmptyAfterSeconds (or master.vacuumDeleteEmptyAfterSeconds under weed server/mini; default 0, off) the automatic sweep now deletes replica copies that have stayed empty and quiet for that long, the same rule volume.deleteEmpty applies on demand: remote-backed copies are skipped, and every delete carries the volume server's onlyEmpty / onlyGarbage guards so a copy written since the last report is refused rather than removed. Copies that still hold data or were written recently stay; only a volume whose every copy is deleted leaves the sweep's work map, sparing a compaction of bytes that are all deleted. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vacuum: harden empty-volume sweep against partial and racing deletes Review follow-up on #11477: - delete a volume only when every replica copy is a verifiable empty-and-quiet candidate; deleting the empty copy of a volume whose sibling holds live files would silently cut its replica count (greptile P1). - drain the volume out of the writable list before deleting, the same drain the compact pass uses, so PickForWrite stops assigning it and pending writes settle (devin). - bound the VolumeDelete RPC so one stalled server cannot hold the vacuum lock indefinitely (greptile P1, reusing allocateVolumeTimeout). The vid2location panic scenario raised in review does not exist: VolumeLocationList methods are nil-receiver safe and a missing vid just fails enoughCopies, so a partially deleted volume skips compaction instead of crashing the sweep. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vacuum: unregister deleted empty replicas and prune the sweep list A successful VolumeDelete only updates the volume server; the master still tracked the replica and kept it in the sweep's location list for the compaction pass (coderabbit on #11477). Unregister the replica right after its delete succeeds and drop it from the sweep copy, so a partially deleted volume only compacts copies that still exist. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vacuum: pin deleting volumes out of the writable list across heartbeats Review follow-up on #11477 (greptile): DrainAndRemoveFromWritable only removed the volume once; a heartbeat landing between the drain and the replica deletes re-evaluated writability and re-added it, so a client write could reach a replica whose siblings were already gone and leave the volume under-replicated when the last copy refused its onlyEmpty delete. MarkDeleting records the vid in deletingVolumes — checked inside setVolumeWritable so heartbeat, capacity-recovery, and admin re-add paths all hold it out — and UnmarkDeleting releases it once the sweep finishes the copy pass. A partially deleted volume's surviving replicas then return to writable through the normal heartbeat path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vacuum: restore writability when a sweep delete survives Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-27 07:05:22 +08:00
Chris Lu	56d2f05ccd	topology: wake the vacuum dispatcher when a worker frees quota (#11436) ... * topology: wake the vacuum dispatcher when a worker frees quota The dispatch loop slept a fixed 10s whenever every pending volume was waiting for a per-server quota slot, so a sweep took volumes x 10s regardless of how fast the compactions were. Workers now signal on a buffered channel after crediting quota; the dispatcher waits on it with the 10s sleep kept only as a timeout. * master: add -vacuumIntervalSeconds to tune the automatic sweep interval The 14-minute base interval was a literal inside the refresh loop while every neighbouring vacuum knob was already a flag. Defaults to 840s, unchanged. * topology: keep the 14 minute floor on the vacuum interval A zero-valued MasterOption or a negative -vacuumIntervalSeconds left the sweep sleeping only its jitter, so treat non-positive intervals as the previous default.	2026-09-25 02:14:59 +08:00
hsdfatandChris Lu	a93a1ab2eb	fix(volume): return an error instead of 201 when a write lands on no volume (#11397) ... * fix(volume): return an error instead of 201 when a write lands on no volume ReplicatedWrite only writes locally when this server holds the volume. For a volume id no server holds, the master lookup returns no locations, so the write went nowhere and the upload still got 201 Created. The same happened for a type=replicate write to a server without the volume, so the primary, or the S3 chunk fan-out, counted a replica that was never written. A server without the volume still forwards the write to the replicas the master lists. When there is nothing to forward to, fail with "volume N not found on host:port". PostHandler returns that as 500, the status the Rust volume server already returns here, and uploaders re-assign on 5xx. Fixes #6609 * volume: reuse Store.HasVolume, drop issue ref from test comment --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-09-19 19:03:21 -07:00
Chris Lu	5d5ea18287	topology: fix fatal concurrent map read/write on VolumeLayout.crowded (#11216) ... SetVolumeCrowded mutated the crowded map under accessLock.RLock(), while GetWritableVolumeCount reads the same map under RLock() on the Assign hot path. Two concurrent RLock holders with one writing and one reading the map triggers a fatal "concurrent map read and map write" that kills the master process (unrecoverable, bypasses recover). Take the write lock in SetVolumeCrowded instead. This event path is a low-frequency single consumer driven by the crowded-volume event loop, and every other mutation of crowded already holds Lock(); setVolumeCrowded takes no nested locks, so there is no deadlock path. The hot readers (GetWritableVolumeCount, CloneWritableVolumes) keep using RLock. Adds a -race regression test that fails (race detected) on the old RLock and passes with the write lock. Fixes #11211	2026-09-07 13:47:22 -07:00
Chris Lu	15e4da65f7	volume: avoid read-only replica write targets (#11195) ... * master: carry replica read-only state in volume lookups * volume: refresh writable replica targets * volume: preserve read-only replicas for deletes * master: propagate read-only delete capability * volume: target delete-capable replicas * volume: honor configured HTTPS for replica deletes * volume: reject insecure delete authorization forwarding * master: broadcast delete capability changes * volume: align Rust replica routing * http: protect credentialed replica redirects * master: preserve digest compatibility for delete capability * volume: propagate read-only state in short heartbeats * volume: report changed short volume state * http: guard TLS client redirects * master: announce mounted volume read-only state * volume: replace changed identity deltas * master: replace incremental volume layouts in order * master: keep moved volume lookup available * volume: announce read-only mounts	2026-09-07 09:23:56 -07:00
yanglongwei	eb717199d0	master: delete replica_placement_mismatch labels when volumes leave topology (#11062) ... * master: delete replica_placement_mismatch labels when volumes leave topology Fixes #10804. Setting the gauge to 0 left stale Prometheus time series that grew unbounded with volume churn; remove the label set on unregister instead. * master: delete replica_placement_mismatch only after last placement leaves Unconditional DeleteLabelValues on UnRegisterVolumeLayout dropped the series while other data nodes still held the volume, hiding under-replication until the next collect cycle. Delete only when Lookup is empty, and cover the two-copy case in a regression test.	2026-09-05 12:52:27 -07:00
Chris LuandDevin	cc281dabc9	master: keep new volumes and writes off servers in maintenance mode (#11147) ... * master: keep new volumes and writes off servers in maintenance mode The master recorded a volume server's maintenance flag from the heartbeat but never consulted it. A server in maintenance (#7977) is being drained, yet the master kept creating volumes on it whenever it had free slots and kept handing out its volumes for writes. Nothing on the volume server blocks plain HTTP uploads either, so "read-only mode" was only a name. Volume growth: a data node in maintenance mode reports zero free slots through AvailableSpaceFor, which takes it out of every candidate list, feasibility count and capacity reservation. Its slots still roll up into its rack and data center, so the random offset drawn from those totals for an other-rack or other-DC replica could land in space the walk then skips and fail with "No free volume slot found!" while siblings had room; the walk now folds the offset into the space that is actually eligible. This also covers the pre-existing case of an over-committed sibling. Assignment: a replica on a server in maintenance mode is treated like a read-only replica in isAllWritable, so its volume leaves the writable list and returns when the flag clears. Topology.SetDataNodeMaintenanceMode re-evaluates the node's volumes on every change, since heartbeats are digest-based and a full volume list may not follow for a long time. Reads and lookups are untouched. The flag moves to an atomic so the assign and growth paths can read it without the node lock. Heartbeat: the Go volume server sent its state only when it changed, so a master elected while a server sat in maintenance never learned about it. The state now rides along on every heartbeat, as the Rust server already does; the master's compare is an atomic swap, and only a change does work. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: hold maintenance mode through vacuum commit and mark-writable SetVolumeAvailable and SetVolumeWritable put a volume back on the writable list on the replica count alone. A vacuum that started before the server entered maintenance, or a vacuum worker's mark-writable arriving after it, handed the volume back to assignment with a replica on the draining server. Heartbeats carry only changed volumes, so nothing re-evaluated it until the volume itself changed. Apply isAllWritable on both paths, the same test EnsureCorrectWritables uses. Also pin that re-evaluating a volume a concurrent disconnect already removed from its layout is a no-op. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: record a server's read-only notification on its node before judging the volume A volume server notifies the master the moment it flips a volume between read-only and writable, ahead of the heartbeat that repeats the flag. The layout only set its per-location flag, so isAllWritable, which reads the node's heartbeat copy, still saw the old value: a mark-writable was withheld until the next heartbeat, and a re-evaluation landing between a mark-readonly and its heartbeat put the volume back on the writable list. Record the flag on the node's volume first. AddOrUpdateVolume keeps the digest and the active volume count in step, so the heartbeat that follows finds nothing to change. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: a read-only mark does not confirm a provisional volume DataNode.SetVolumeReadOnly went through Disk.AddOrUpdateVolume, which treats its input as a server report and so ended the grace period that keeps a just-grown volume safe from a full report collected before the grow. A volume marked read-only before its first report could then be removed by that stale report. Give Disk a SetVolumeReadOnly that flips the flag and keeps the digest and active volume count in step without touching volumeAddedAt. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-03 23:50:12 -07:00
3 people	7620e96171	expose whether a volume replica is backed by remote storage, and prefer local replicas (#11105) ... * expose whether a volume replica is backed by remote storage Volume locations returned by lookups do not indicate whether a replica has been tiered to remote storage. Readers cannot distinguish a local replica from a remote-backed one, so they may hit a remote-backed replica first even when a local replica is available. Add DataInRemote to the lookup location message, populate it from the master's volume info, and carry it through the wdclient vid map so clients can prefer local replicas when resolving chunk locations. * wdclient: prefer local volume replicas over remote-tier replicas on lookup LookupFileIdWithFallback (and the publicUrl variant in FilerClient) didn't honor the DataInRemote flag when shuffling URLs, so the DataInRemote patch only took effect in LookupVolumeServerUrl. Apply the same ReorderToFront(localUrls) to sameDcUrls/otherDcUrls so non-remote replicas stay at the front, matching the existing vidMap convention. * wdclient: propagate DataInRemote across tier transitions on existing replicas When a volume is tiered to remote storage or a remote-backed replica is restored locally, the cached DataInRemote on the same volume-server URL stayed at its old value because two pieces of state never updated: * master_grpc_server.go only split newVolumes and (already-tracked) volumes into NewVids vs RemoteVids. ChangedVolumes went straight to NewVids, so the broadcast announced the re-classified volume as a fresh arrival and the client had no way to tell whether its existing cache was stale. * vid_map.addLocationToMap early-returned when an entry already had the same URL. A tier transition reports the same URL with DataInRemote flipped, so the cached entry stayed at the old classification. Wire both sides together: ChangedVolumes now go through the same IsRemote split as newVolumes, and addLocationToMap replaces the existing entry in place when the URL matches but DataInRemote has changed. The server reference key only depends on URL/grpc port, so the refcount does not move across the flip. Adds vid_map_remote_transition_test.go covering the local->remote and remote->local paths so the in-place update and the cache-key stability are pinned by tests. * wdclient: prefer local replicas across data-center boundaries The previous local-first ordering hoisted local URLs to the front of each data-center bucket separately, then concatenated same-DC before other-DC. That meant a same-DC remote replica could still be tried before an other-DC local replica even though the local one would answer cheaply. Reorder once across the full candidate list: concatenate same-DC and other-DC first, then ReorderToFront pulls every local replica to the very front while preserving the DC preference inside each tier. Apply the same ordering in all four lookup paths so the cached vidMap, the LookupFileIdWithFallback provider path, FilerClient.GetLookupFileIdFunction (PublicUrl-preferred variant), and the deprecated filer.LookupFn all agree: - weed/wdclient/vid_map.go (LookupVolumeServerUrl) - weed/wdclient/vidmap_client.go (LookupFileIdWithFallback) - weed/wdclient/filer_client.go (LookupFileId) - weed/filer/reader_at.go (LookupFn) Strengthen the existing local-first tests: vidmap_client_localfirst_test now asserts both endpoints are present (not just the local one is first), and slice_test asserts an exact match instead of accepting two orderings. Add TestLookupFileIdWithFallbackGlobalLocalFirst to pin the cross-DC ordering invariant: any local replica (same or other DC) precedes every remote-tier replica; within each tier DC1 precedes DC2. Add docstrings to ToVolumeLocations, ReorderToFront, LookupVolumeServerUrl, LookupFileId, GetVidLocations, GetLocations, LookupFileIdWithFallback, and updateVidMap so the touched lookup paths are described in one place. * topology: broadcast tier transitions on existing replicas When a volume replica is tiered to remote storage or restored locally, the wdclient's cached DataInRemote went stale: every connected client kept preferring a remote-backed replica over a freshly restored local one, or demoted a freshly tiered remote replica. The fix in commit 116982595 routed ChangedVolumes to NewVids/RemoteVids on the master, but ApplyVolumeChanges returned only fresh arrivals and previously servable replicas. An existing replica whose IsRemote() classification flipped was neither, so it never reached the broadcast loop and the wdclient never learned. Make Disk.doAddOrUpdateVolume return a third signal -- tierTransition -- true exactly when an existing replica's IsRemote() flips. ApplyVolumeChanges treats that as an arrival so the existing SendHeartbeat routing loop now sees it. Add a master-side end-to-end test covering local->remote, remote->local, no-op re-reports, and a mixed heartbeat that only announces the tier transition. Also add docstrings to LookupFileId, wdclientLocationsToPb, and LookupVolume where the prior change touched their bodies. * topology: broadcast tier transitions received through full reconciliation The previous commit added tier-transition routing on the ChangedVolumes delta path, but that is not the only way a re-tiered replica reaches the master. After a digest mismatch the volume server resends a full Volumes list, and SyncDataNodeRegistration applies the new IsRemote() classification silently -- the changedVolumes return value was being thrown away. The master therefore never broadcast NewVids/RemoteVids, and a wdclient connected during the recovery kept the stale DataInRemote until it lost contact with the master. Surface the changed set through UpdateVolumes.changedVolumes (now covering both ReadOnly flips and tier flips) and SyncDataNodeRegistration, then route it through NewVids/RemoteVids in SendHeartbeat the same way the delta path already does. Add an end-to-end test for the full reconciliation path. * master: keep an EC volume's locations in the volume lookup The nodes that answer for an EC volume hold shards, not a volume record, so asking them for one fails. Dropping the location on that failure emptied the result and turned every EC read through the master's HTTP lookup and fid redirect into a 404. Treat an absent volume record as a local read and keep the node in the answer. The per-node conversion moves into topologyLocation so the EC case is covered by a test. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW * wdclient: replace a tier-flipped location without writing under a reader GetLocations hands back the entry's own slice and the caller walks it after the read lock is dropped, which is why every other mutation here builds a new slice. Writing the flipped replica into the array in place raced LookupVolumeServerUrl, reported by -race. Copy the slice, swap the one element, and publish it. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW * master: keep a remote volume on NewVids for older clients Moving remote-tier volumes out of NewVids and into RemoteVids alone is a wire break in the wrong direction. A master upgraded ahead of its filers and mounts -- the usual order -- announces a tiered volume only on a field the older client ignores, so the volume drops out of that client's vid map entirely and reads for it fail. Announce every volume on NewVids and repeat the remote-tier subset on RemoteVids, so a new client still learns the tier and an old one keeps the location. The routing moves into announceVolume, which the heartbeat paths and their tests now share instead of each restating it. On the client, RemoteVids no longer needs a second write per volume: the tier is settled before anything is added. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW * topology: split the volume snapshot by tier without copying the records ToVolumeLocations runs on every KeepConnected, so a filer or mount connecting made the master allocate a full VolumeInfo per volume per node just to read four bytes of id off each one. AppendVolumeIds exists to avoid exactly that. Extend it to fill the remote-tier list alongside the full one, and use it again in the snapshot. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW * wdclient: keep the data-center preference ahead of the local-first ordering Hoisting every local replica to the very front puts an other-DC local read ahead of a same-DC remote one. When the remote tier sits in the same region as the replicas -- the common arrangement -- that trades an in-region GET for a WAN round trip and costs more than the remote read it avoids. Reorder inside each data-center bucket instead, so local still wins among equals and the data-center preference still wins overall. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW * operation: pick the read replica from one list The local-preferring lookup built a list of local URLs and then branched on whether it was empty, duplicating the random pick. Fall back by filling the same list with every replica instead. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW --------- Co-authored-by: Bruce Zou <gift_secondst@msn.com> Co-authored-by: bruce-zzz <bruce.zou@hhy-data.com>	2026-09-02 16:12:46 -07:00
Chris Lu	5f787a25c3	master: survive a volume layout deleted twice (#11098) ... * master: survive a layout deleted twice Two volume servers dropping the last replica of volumes that share a layout both find it empty and both delete it. The loser's lookup misses, and the single-value type assertion on the result crashed the master before the caller could look at the found flag. Claude-Session: https://claude.ai/code/session_01WmX6Rchx298NQksHDXg7sk * master: remove a layout and read it back in one step DeleteVolumeLayout looked the layout up and then deleted it, so two deleters could each release the lookup ownership of the same layout, or one could find nothing to release at all. Have the map hand back what it removed. Claude-Session: https://claude.ai/code/session_01WmX6Rchx298NQksHDXg7sk	2026-09-02 11:50:48 -07:00
Chris Lu	8f2daad338	topology: mirror the writable volume list in a set (#11076) ... Membership was a linear scan over a slice, and ensureCorrectWritables runs it for every volume on every heartbeat, so the master's steady-state cost per volume server is quadratic in that server's volume count. BenchmarkSyncDataNodeRegistration, median of 3: 1000 volumes 565.7us -> 535.0us -5.4% 100000 volumes 1.665s -> 55.3ms -96.7% Allocations are unchanged at both sizes, so the difference is the scan. Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1	2026-09-01 14:21:26 -07:00
Chris Lu	9d5525e747	master: keep periodic volume growth to the data centers a layout lives in (#11060) ... * master: keep the periodic growth scan to data centers hosting the layout The rack-aware scan planned growth for every data center in the topology, so a collection pinned to one DC (fs.configure -dataCenter) sprouted volumes in all the others within one scan cycle. Plan only for data centers already hosting the layout's volumes; an empty DC gets its volumes from the DC-constrained assign that first asks for them. The lastGrowCount divisor likewise counts only the racks the scan can plan for. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 * master: pin periodic must-grow growth to a single-DC layout's data center The must-grow and crowded paths of the periodic loop grow with no DataCenter, so even with the scan fixed a pinned collection's volumes could still land in any DC once lastGrowCount demands more writables. Stamp the grow request with the layout's data center when its volumes all live in one; layouts spanning DCs keep unconstrained growth. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 * master: never pin growth of a cross-DC-replicated layout A layout whose replication spans data centers cannot legitimately live in one DC; observing a single hosting DC there means the other DCs are down. Do not encode that outage as a placement constraint. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 * master: bound the hosting-DC walk by the answer it needs listVolumeDataCenters walked every location of the layout under accessLock — ~190ms for a million volumes, twice per layout per cycle, stalling assigns behind the read lock. Stop once enough distinct DCs answer the caller's question: two for the single-DC check, the topology's DC count for the scan. A spanning million-volume layout now finishes in microseconds; only a layout truly confined to fewer DCs still pays a full walk, the same cost class as the under-replication count this loop already takes each cycle. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5	2026-09-01 00:40:50 -07:00
Chris Lu	0c59c0fb05	master: scope the startup capacity shed to a truly empty topology (#11058) ... * master: scope the startup capacity shed to a truly empty topology The retryable "no volume server capacity registered yet" shed checked capacity for the requested disk type, so a cluster serving only other media -- where that capacity will never register -- shed every assign until the client's deadline instead of failing fast. An unsteered write to such a cluster hung for its full HTTP deadline and surfaced "context deadline exceeded" in place of "No writable volumes". Shed only while no disk type has any registered capacity, and name the unserved medium in the fast failure. Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R * master: name the unserved medium for every fail-fast caller The diagnostic sat in the growth-initiator block, so a follower joining an in-flight growth and a growth-disabled master failed the same way with only the generic pick error. Wrap at the fail-fast break instead, which every caller reaches, and cover all three paths in the test. Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R	2026-08-31 21:29:27 -07:00
Guang Jiong LouandChris Lu	f740210235	get volume topology info without volume details (#11036) ... * get volume topology info without volume details Signed-off-by: lou <alex1988@outlook.com> * master: rename VolumeListRequest.without_volumes to topology_only The field shapes the reply rather than selecting volumes, and it leaves out the ec shards too, which the old name denied. Match the message's *_only style and say what a master that predates the field does with it. Claude-Session: https://claude.ai/code/session_01QHnaNRgxnjzZsiz7WTFML5 * master: refuse topology_only combined with a volume selector A topology_only request that also names a collection or volume ids contradicts itself, and answering either half in silence surprises the caller. Answer InvalidArgument from both VolumeList and its stream, before the stream sends its header. Claude-Session: https://claude.ai/code/session_01QHnaNRgxnjzZsiz7WTFML5 --------- Signed-off-by: lou <alex1988@outlook.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-31 09:45:22 -07:00
Chris Lu	d3b8030a69	master: shed assigns retryably until volume servers register capacity (#11032) ... An assign arriving before any volume server has heartbeated saw zero available space and failed outright with a plain error no client retries, so the first write to a fresh bucket answered 500 while the cluster was still starting. Distinguish a topology with no registered capacity from a genuinely full one: fail fast only when registered capacity is exhausted, and shed ResourceExhausted otherwise so the client's retry budget rides out the startup window. Claude-Session: https://claude.ai/code/session_018G9kWFgy8BaBAEkYV3YL9n	2026-08-30 11:08:53 -07:00
Guang Jiong LouandChris Lu	93666c90e9	filter by volume ids (#10983) ... * filter by volume ids * master: carry the volume ids VolumeList asks about in one repeated field One id and a list of them ask the same question, so field 2 holds the list rather than standing beside a second field that supersedes it. Claude-Session: https://claude.ai/code/session_011qAmAdhrYvnzGkw7A9N4mP --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-28 20:43:29 -07:00
Chris Lu	a02c0024e5	master: cap the reported capacity at what the disks hold (#10960) ... * master: cap the reported capacity at what the disks hold Statistics reported max volume count times the volume size limit, which is how many volumes the cluster is allowed to place, not how much space it has. A cluster given far more slots than its disks can fill reported a capacity it could never reach -- 65536 slots at 30GB read as 1.9PB on a 460GB disk -- and the number never moved, since writing data changes neither the slot count nor the size limit. The volume servers already report each filesystem's total and free bytes in their heartbeats, so bound the answer by what they say is left. * mount: keep the last known sizes when filer statistics fails A failed Statistics call returned before df's answer was filled in, so a mount whose filer or master was briefly unreachable reported an empty filesystem rather than the sizes it already had. * master: drop the disk ceiling when a volume server does not report A cluster part way through an upgrade has volume servers that predate the disk bytes in the heartbeat. Summing only the ones that answered left the quiet server's free space out of the total, and the server holding the room is exactly the one that could make the cluster read as full. Answer with the disks only when every one of them reported.	2026-08-26 00:12:56 -07:00
3 people	ef4c9d9178	filter volume by local or remote storage name (#10946) ... * filter volume by local or remote storage name Signed-off-by: lou <alex1988@outlook.com> * fix SelectsEverything Signed-off-by: lou <alex1988@outlook.com> * keep the proto sync out of this change The branch copied weed/pb/*.proto over their seaweed-volume and Java counterparts and regenerated every .pb.go with a different protoc and protoc-gen-go-grpc. DiskStatus.error arriving that way broke the Rust build, and the rest is toolchain churn in files this change has nothing to say about. --------- Signed-off-by: lou <alex1988@outlook.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-25 13:05:33 -07:00
Chris Lu	b77431c142	master: stop hintless small-file assigns from marking volumes full (#10944) ... * master: estimate a hintless assign's size from the volume's average file size An assign that carries no dataSize hint charged a flat 1MB per file id against the volume's effective size. A small-file workload overpays by orders of magnitude: bulk-writing 4KB files marks volumes holding a few hundred MB of real data as crowded and then full, so the master grows unnecessary volumes and, once every volume is spuriously full, fails all assigns. Estimate from the volume's own average file size instead, and keep the 1MB fallback only for volumes with no history. * master: decay pending assign sizes for volumes gone quiet The decay that corrects pending assign estimates runs only when a heartbeat reports the volume, and a heartbeat only reports a volume whose content changed. A volume held out of the writable list takes no writes, so once inflated estimates mark every volume full, nothing is ever reported again, nothing decays, and the cluster refuses all writes until a restart. Run the decay from the master's periodic loop for volumes no heartbeat has reported within two pulses, feeding the last reported size back through the same path an unchanged heartbeat would take. * master: trim the comments on the assign size estimate * master: keep the periodic decay out of the replica-dedup window UpdateVolumeSize ignores a report arriving within two seconds of the last one, so replicas of the same volume do not each halve the pending estimate. The periodic decay went through the same path and stamped that window, so a real heartbeat landing right behind it was dropped along with its reported size and compact revision. Only a volume whose content changed is reported at all, so nothing would send that size again and the master kept a stale one. Let the dedup window belong to volume server reports alone. * master: let the decay read the size record under the lock it mutates The periodic decay picked its volumes under a read lock and replayed them under a write one, carrying the size it had read across the gap. A heartbeat landing in between was rolled back: the replay wrote the older size and compact revision over the fresh ones, and a compaction report lost that way is never resent, since only a volume whose content changed is reported. The decay has no size of its own to contribute, so it now reads the record under the same lock it mutates. * master: let a heartbeat that beat the decay stand for the cycle The decay chooses its volumes under a read lock and applies them under a write one. A heartbeat landing in that gap already did the halving the cycle owed, so applying the decay on top of it halved twice and forgot pending bytes the volume has not written yet - the double-halving the replica-dedup window exists to prevent. Both callers now give way to a report already handled for this cycle; only a real report still advances lastUpdateTime, so a quiet volume keeps decaying every pulse. * master: keep genuinely full volumes out of the decay pass A volume the disk really did fill keeps its fullSince set for good, so it was selected every pulse for a decay that cannot help it: UpdateVolumeSize refuses to recover a volume whose reported size is at the limit, and replaying a size that cannot move leaves the record as it found it. Full and quiet is the ordinary resting state of a cluster, so this was most of the pass, taking the layout write lock away from the heartbeats to do nothing. On a million tracked volumes with a hundredth of them phantom-full it costs ten thousand write locks a pulse instead of a million. * master: put the stale-replay test back on the path it guards Giving the decay the dedup window left this test short-circuiting there, so it no longer reached the locked read it was written for and passed with that read removed. Age the record past the window, which is the only case where reading it under the lock is what saves the report.	2026-08-25 10:16:48 -07:00
Chris Lu	9c8d3b6a81	ec: refund the cleared leftover shards' slots in the encode source health check (#10903) ... * erasure_coding: one home for the shard-count to volume-slots conversion * ec: refund the cleared leftover shards' slots in the encode source health check	2026-08-23 21:48:20 -07:00
Chris Lu	35d53a20f6	master: let the leader admit a master that starts with no raft state (#10865) ... * master: answer with the leader raft already knows Topo.Leader() backs off for up to 20 seconds waiting for an election. Callers that a health probe or a client is blocked on cannot afford that: /cluster/status, /cluster/healthz and /readyz all sit past the probe timeout of both the helm chart and the operator, so a master that is still joining looks dead rather than joining, and the kubelet restarts it. informNewLeader and SendHeartbeat hold the client on a master that cannot serve it, exactly when it should move on to find the one that can. Answer these from MaybeLeader instead, which reports what raft knows right now. MaybeLeader takes over the "am I the leader myself" fallback that Leader() used to apply on top of it, so one non-blocking call is still correct; Leader() keeps the backoff for callers that must wait. * master: let the leader admit a master that starts with no raft state Neither raft implementation lets a server outside the configuration campaign: goraft's promotable() requires a non-empty log, and hashicorp rejects vote requests from a candidate that is not in its configuration. A master that comes up with fresh state therefore cannot elect itself in — the leader has to pull it in. Nothing did. The peer list is static, rendered from the replica count, so scaling it up leaves the sitting leader running the old list with no idea the new masters exist. Under goraft they wait forever. Under hashicorp they are worse off: each bootstraps a cluster of its own from the new list, and two of them form a quorum next to the live leader, with their own TopologyId. That is the split brain SetTopologyId kills a master over. Admit the peer where it registers instead. Only the leader gets past the IsLeader check in KeepConnected, and a joining master's client lands there, so that is the moment it joins. The broadcast OnPeerUpdate rides on is not enough on its own: it only reaches masters already connected, which is why a leader that came up first missed both newcomers. RaftAddServer grew a goraft branch on the way, so cluster.raft.add stops silently doing nothing on the default raft, and RaftRemoveServer with it. Bootstrapping is now one call for both implementations, made only after the peers confirm nobody has a leader, and retried until this master is in rather than checked once and dropped. * master: do not evict a peer that is still in -peers The hashicorp leader drops a master from the raft configuration as soon as it stops answering pings. A master that is merely restarting answers nothing, so an ordinary bounce shrinks the quorum behind the operator's back — and then races its own return: the master comes back, registers, gets re-admitted, and the eviction lands after it. A randomized start/stop walk lands on it. Two of three masters running, the leader evicts the one that just went down, the restart re-adds it, the removal commits late and takes the leader's own leadership with it. What is left is a two-server configuration whose other half is down, and a running master that nobody will ask for a vote — no quorum, no way back until the third master returns. -peers is what declares membership. updatePeers already reconciles the configuration against it on every leadership change, and an operator who really means to drop a master can say so with cluster.raft.remove, so keep the eviction for masters that are no longer listed at all. * test: bounce masters at random and hold the election to it Twelve rounds of stopping or starting a random master, on both raft implementations, checking the two things an election must never get wrong: two masters claiming leadership at once, and a quorum that comes back without agreeing on one. The cluster's identity has to survive the whole walk, since a master that re-mints a TopologyId is the split brain SetTopologyId kills its peers over. The seed is random and logged, so a failure names the walk that reproduces it. Below a quorum the walk moves straight on. A master that has lost its quorum cannot commit anything, and goraft only checks whether it still has one on an election-timeout ticker, after its peers have been quiet for a full timeout — measured taking over 30 seconds to step down. That direction belongs to TestTwoMastersDownAndRestart, which was giving it ten seconds and would have started failing on a slower machine; it now waits on that behaviour explicitly rather than sleeping twice and hoping. WaitForTopologyId returns the id it waited for. Reading it separately raced the leader applying the raft entry that carries it, which shows up as an empty id right after an election rather than as a wrong one.	2026-08-21 15:22:22 -07:00
Chris Lu	3bd218e030	volume: cut idle memory at high volume counts (#10861) ... * volume: start a volume's batch write worker on first use Mounting a volume started a goroutine parked on a 128-slot channel, plus the 128-entry batch slice it had already allocated. That is around 6.7KB per volume the server pays whether or not the volume ever takes a write: 7231 bytes per mounted volume, of which 4101 is goroutine stack. Only a write that asks for fsync ever reaches the worker, and a remote-tiered or read-only volume never can. Create the channel and its goroutine on the first such request instead, and let a write arriving after Destroy fall back to the inline path rather than queue onto a worker that has gone. Measured over 20000 mounted volumes: 7231 -> 1269 bytes each. * volume: update the heartbeat report state in place Every heartbeat built a second map of what it was about to tell the master, holding a freshly allocated short information message per volume, then swapped it in over the old one -- and computed departures through a third map of the live volume ids. A server holding 2M volumes rebuilt all three every VolumePulsePeriod for a report that usually says nothing. Number the heartbeats instead and mark the entry already held with the pass that found the copy, so a quiet volume costs a map lookup and no allocation. Departures are the entries a pass did not mark; the live-id map is now built only when there are some, sized to them. Measured over 10000 mounted volumes: 436 -> 196 bytes allocated per volume per heartbeat. * volume: fill one volume information message per heartbeat, not per volume The heartbeat built a message for every volume held so it could hash it, then dropped all but the few it had something to say about. At 2M volumes that is 2M messages allocated every VolumePulsePeriod to send almost none of them. Fill a message the caller supplies instead, and replace it only when the heartbeat keeps it, so a server with nothing to report fills the same one all the way through. Measured over 10000 mounted volumes: 196 -> 4 bytes allocated per volume per heartbeat, and a heartbeat runs a third faster. * volume: drop the per-volume trace from the heartbeat's status read glog.V(4).Infof evaluates its arguments whether or not the verbosity is on, so every volume boxed its id into a fresh interface slice on every heartbeat: 759 of the 773 allocations a 1000-volume heartbeat made, for a line that at this scale would print millions of unreadable rows. Measured over 1000 mounted volumes: 4776 -> 1792 bytes and 759 -> 14 allocations per heartbeat, which no longer grows with the volume count. * seaweed-volume: mirror the in-place heartbeat report state Same change as the Go volume server: number the heartbeats and mark the entry already held with the pass that found the copy, instead of building a second map of hashes and swapping it in. The volume snapshot must leave the reporting state as it found it, so it keeps asking through changed() while a real heartbeat marks through record(). * volume: refuse writes to a closed volume instead of dereferencing nil Close and Destroy leave the needle map and data backend nil, but a caller that already holds the volume can still reach the write path, where both are used unguarded: a write racing a volume deletion took the server down. syncDelete has always checked; syncWrite and the batch worker had not. Reachable before this series and now also from the inline fallback a durable write takes when the worker has gone. * seaweed-volume: guard the report state with one mutex, as Go does The full-list flag and the generation that answers it have to move together. Split across separate atomics they cannot: a request landing between begin's two reads returns full == false with the generation it just raised, and one landing between commit's read and its clear is marked answered by a heartbeat that carried no list. Either way the resend is dropped. Neither is reachable today -- every caller reaches this through the store's RwLock, the flag setters under a read lock and the heartbeat build under a write lock, so they cannot interleave. The type should not depend on that being true two files away, and Go holds a single mutex over exactly these fields. * test: build the servers under test to match the harness's offset size The mixed Go/Rust suites run both servers against one dataset, so both have to agree on the offset width. They did not: the harness built Go with no tags, 4-byte offsets, while the Rust crate defaults to its 5bytes feature, and the Rust server then refused the .vif the Go server had just written -- "bytes_offset mismatch: found 4, expected 5". Build each side to match the offset size the test binary itself was compiled with, so a plain `go test` and one with -tags 5BytesOffset both get a matched pair.	2026-08-21 13:04:56 -07:00
3 people	f7c4636d22	topology: refresh oversized mark on every heartbeat (#10829) ... * topology: refresh oversized mark on every heartbeat The oversized flag on a volume location was only set when the volume was registered (RegisterVolume). A volume that later grew past the size limit kept its stale "not oversized" mark, so the heartbeat path (ensureCorrectWritables) kept re-adding it to the writable list while RecordAssign removed it on every assign - a writable/unwritable flip loop that let writes continue past the limit and made vacuum race in-flight writes. Refresh the mark from each heartbeat's reported size in both heartbeat paths (ApplyVolumeChanges and SyncDataNodeRegistration), mirroring what RegisterVolume already did at registration time. A volume that grew past the limit now stays unwritable, and one that shrank back clears the mark and can recover. * topology: order heartbeat writable correction after decay and honor cooldown Review feedback (Greptile, CodeRabbit) on the oversized-mark refresh: 1. Greptile: clearing the oversized mark before EnsureCorrectWritables let the delay-unaware helper re-add a just-compacted volume to writables, bypassing capacityRecoveryDelay. ensureCorrectWritables now checks fullSince and skips the re-add while the cooldown is pending, so a volume removed for capacity only recovers through UpdateVolumeSize's heartbeat recovery path. 2. CodeRabbit: in the full-heartbeat path the mark was refreshed after the writable correction, so a newly oversized volume stayed writable for an extra heartbeat cycle. The standalone changedVolumes loop is merged into the volumeInfos loop and EnsureCorrectWritables now runs after UpdateOversizedState + UpdateVolumeSize in both heartbeat paths, using the freshly refreshed mark. 3. TestHandlingVolumeServerHeartbeat used a size (254320) that is past the test's volumeSizeLimit (32768); it only passed because the stale mark hid the oversized state. Sized down to 30000 to keep testing the add/remove flow, and added TestEnsureCorrectWritablesHonorsRecoveryCooldown covering the cooldown window and the recovery after it. * topology: do not restore a still-crowded volume after the cooldown Greptile review: after capacityRecoveryDelay elapses, ensureCorrectWritables could restore a volume whose effective size is still past the crowded threshold. UpdateVolumeSize refuses the recovery (effectiveSize > crowded threshold -> setVolumeCrowded + return false), but the cooldown check in ensureCorrectWritables only looked at fullSince, so once the delay passed it re-added the volume even though capacity tracking still considers it crowded. Check the crowded mark before re-adding: a volume UpdateVolumeSize just marked crowded must not be restored here, otherwise assignments resume while the volume is still flagged for growth. Adds TestEnsureCorrectWritablesDoesNotRestoreCrowdedVolume: effectiveSize decays to 10500 (past the 9000 crowded threshold) after a report of 8000, and ensureCorrectWritables keeps the volume unwritable past the cooldown. * ci: trigger re-run of flaky FUSE jobs * topology: gate the writable restore on the limit, not on crowded A crowded volume is above the growth threshold, not full, and is normally writable. Refusing to restore one locks it out for good: nothing writes to a volume that is not writable, so its size can never fall back under the threshold. Gate on the same size the assign path uses to remove it. * topology: let only the heartbeat refresh set the oversized mark Registration also set it, from whatever VolumeInfo it was handed. The incremental path builds that from a short heartbeat message, which carries no size, so every arrival announcement cleared the mark and handed the volume back to the writable list until the next full report. * topology: use the re-resolved layout after a dropped one is replaced A layout dropped with its collection makes RegisterVolume refuse, and the full heartbeat then re-registered against a fresh layout but kept applying the size, oversized and writable updates to the dropped one. --------- Co-authored-by: hzsunchao <hzsunchao@corp.netease.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-20 09:56:53 -07:00
3 people	9575032b4c	volume: forward fsync=true to replicas in ReplicatedWrite (#10805) ... * volume: forward fsync=true to replicas in ReplicatedWrite When a write request carries fsync=true, only the primary volume server flushed to disk: the replica fan-out URL in ReplicatedWrite only carried type/ttl/ts/cm, so replicas always wrote without fsync even when the client explicitly requested a durable write. Forward the fsync request parameter to the replica volume servers so a durable write means every replica has flushed to disk, not just the primary. Replicas without fsync are untouched (zero behavior change). * storage: flush a durable write inline while stopping The fsync flag on the write path really selects the async batch worker, and it was switched off once the store is stopping. So a fsync=true write landing during the pre-stop drain got acked without ever being flushed - and now that ReplicatedWrite forwards fsync, that covers replicas too. Flush it inline instead of queueing it. The drain keeps accepting writes, which is the whole point of preStopSeconds, and the ack still means the .dat is on disk. If the fsync fails, the append comes back off the .dat and the needle map goes back to what it pointed at before, so nothing resolves to an offset past the truncated end. * storage: make the store's stopping flag atomic SetStopping runs on the signal handler goroutine while the write and vacuum paths read the flag, so every read of it was racy. Nothing about the shutdown ordering changes; only the flag itself is now safe to read. * topology: check the errors the replication test was dropping The mock replica ignored its response write and the mock master ignored whatever Serve returned, so a broken mock would have shown up as a confusing timeout rather than a failure. Also drops the explicit listener close: grpc.Server.Stop already closes the listener it was given. --------- Co-authored-by: hzsunchao <hzsunchao@corp.netease.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-18 17:17:50 -07:00
Chris Lu	980471c818	storage: count a volume's needles in uint32 (#10718) ... FileCount and DeleteCount were int, so each cost a word on every replica the master holds. A volume caps at 30GB on a 4-byte-offset build and 8TB on a 5-byte one, and neither holds 4.29 billion needles. That takes VolumeInfo from 120 bytes to 112, which is its own size class rather than rounding up into the 128 one, so a replica costs 135.7 bytes in the map instead of 151.7 -- about 25MB across the 1.6M replicas in a cluster the size of the one this came from. Counts are narrowed where they are read rather than assigned across, so a report claiming more than a volume can hold pins at the ceiling instead of wrapping to a small number.	2026-08-11 16:36:36 -07:00
Chris Lu	753cb8cda8	master: stop copying the cluster to name it (#10700) ... * topology: name a node's volumes without copying them ToVolumeLocations reads a volume id off every volume in the cluster, and got there through GetVolumes, which copies a whole storage.VolumeInfo per volume to be read for four bytes of it. Every client that connects asks for this. At 800k volumes the walk goes from 94.6MB to 16.0MB, which is the ids themselves. * master: log why a client send failed, not what was sent The message names every volume on a newly connected node, so a client going away had the master format a protobuf that size into text -- through the one log level that is always on. The error is the part worth having.	2026-08-10 11:02:09 -07:00
Chris Lu	5e9b7833ee	topology: keep the volume map's values out of its slots (#10680) ... Go stores a map value inline once it fits in 128 bytes, and indirectly above that. Shrinking storage.VolumeInfo to 120 bytes therefore moved 1.6M of them into the map's own slots, which are allocated to capacity rather than to occupancy, and the map grew by 149MB even though each volume got smaller. Hold pointers, so the size of a volume record stops deciding how it is stored. Updates are written through the pointer already there, so a heartbeat that changes a volume allocates nothing, as it did when the struct was above the threshold and Go was doing this itself. At 800k volume ids across 3 servers the resident topology goes from 705.3MB to 546.6MB, with heartbeat cost unchanged at 111.6MB.	2026-08-10 11:00:05 -07:00