Commit Graph
124 Commits
Author SHA1 Message Date
Chris LuandClaude Opus 5.5 e0beb59ef0 topology: treat a nil ttl as the empty ttl when looking for duplicate volume ids (#12601)
Volume layouts are keyed by ttl.String(), so a nil ttl and the empty ttl
share one layout. ttlEqual told them apart, so a replica reported with a nil
ttl into a layout created with the empty ttl matched its own layout as a
duplicate, and every copy was quarantined.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:09:07 -08:00
Chris LuandClaude Opus 5.5 5c910ad546 topology: list collections without nesting the collection map read lock (#12495)
* topology: list collections without nesting the collection map read lock

Items() already takes the read lock and copies the values. Holding it
around the call made ListCollections take the read lock twice, so a
DeleteCollection waiting for the write lock in between blocked the
second read lock and wedged the master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: test ListCollections against a concurrent collection delete

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 15:05:41 -08:00
d127be610f admin: re-send the master vacuum disable every poll so it survives leader changes (#12430)
* admin: re-send the master vacuum disable every poll so it survives leader changes

The admin disabled the master's scheduled vacuum only when a vacuum plugin
worker connected or disconnected. The master keeps that flag in memory and it
is not replicated through raft, so after a leader change or a leader restart
the new leader ran its own vacuum next to the plugin worker. When both picked
the same volume the master's commit failed ("compact swap aborted: missing
.cpd/.cpx") and it dropped that replica, leaving the volume under-replicated.

Send the desired state on every poll instead, and log on the master only
when the flag actually changes.

* admin: shorten the vacuum sync comments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Kevin <kevin@leetify.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:07:50 -08:00
Chris LuandDevin eec2452e30 master: replicate the recovery-wait growth unfreeze through raft (#12165)
* master: replicate the recovery-wait growth unfreeze through raft

state_loss_recovery_wait lifted the volume-growth freeze only in
memory, while the freeze itself was replicated. A restart replayed the
frozen flag and growth stayed refused until the wait expired again, on
every restart, unless an operator ran volume.growth.resume.

The lift now applies GrowthFrozen=false like ResumeVolumeGrowth, from a
goroutine since IsGrowthFrozen is a read path that cannot block on
raft. The apply runs under reportedTopologyIdsLock so a freeze landing
after the lift keeps its command last in the log; the stale unfreeze
skips itself when the flag is frozen again.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: cover raft replication of the recovery-wait unfreeze

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: retry and re-arm a growth unfreeze raft failed to record

applyGrowthFreeze swallowed the raft error, so a rejected lift during a
leader change left the flag clear while the log still held the freeze;
nothing retried and a restart re-froze growth anyway. The apply now
returns the error, replicateGrowthUnfreeze retries it, and on
persistent failure re-arms the flag so the next IsGrowthFrozen
attempts the lift again instead of diverging from the log.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: release the topology-id lock before the freeze raft write

CheckReportedTopologyId and SetTopologyInitPending held
reportedTopologyIdsLock across the blocking raft apply, so a slow
apply stalled every later report — including the heartbeat that
should trigger the next freeze. The lock now covers only the
mismatch decision; the replicate step runs after it is released.

* master: bind the recovery unfreeze to the freeze instant it lifts

An unfreeze submitted while a newer freeze sat committed but not yet
applied — e.g. after this node won an election — could land behind
that freeze in the log and erase it. The command now carries the
freeze instant cluster-wide (GrowthFrozenAt) and the lift is bound
to the instant its submitter observed (GrowthUnfreezeFor): the state
machine no-ops a lift while a strictly newer freeze exists, which
also lets the unfreeze run without holding the topology-id lock at
all. Concurrent freeze writes keep their own mutex so log order
cannot regress the instant.

* master: a superseded unfreeze retry leaves the resumed flag alone

replicateGrowthUnfreeze broke out of its retry loop when a newer\nfreeze superseded the lift, but the break fell through to the\nre-arm: an operator resume in between was silently undone. Return\ninstead — the newer freeze generation owns the flag.

* master: replayed freeze entries without an instant stay ageless

Freeze commands written before the instant field existed carry zero.\nSubstituting the replay time made each restart stamp the freeze as\nnew, so a committed bound unfreeze always looked older and was\nrejected — the freeze came back on every restart. Keep the zero\ninstant: the recovery window reads as elapsed and bound lifts apply.

* master: pack the growth freeze flag and generation into one atomic

Separate flag and timestamp atomics left three holes: a resume's\nload-and-clear could lift a generation stamped in between, a bound\nlift compared stamps numerically although masters' clocks need not\nbe monotonic, and an ageless generation had no identity a bound lift\ncould match without colliding with the unconditional encoding.\n\nOne value now carries the state — 0 unfrozen, -1 an ageless freeze,\nelse the freeze instant — so every transition is a single atomic\nswap or compare-and-swap bound to an exact generation.

* master: a repeat mismatch report keeps the armed freeze generation

Re-stamping while already frozen never reaches the raft log, so the local generation diverges from the replicated one and every bound unfreeze becomes a no-op on replay — the freeze returns after restart or failover.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: snapshots read the freeze flag and generation in one load

Save, Snapshot, and PersistMaxVolumeId read IsGrowthFrozen and the generation separately, so a resume landing between them persisted frozen=true with stamp 0 — restoring an ageless freeze a committed bound lift cannot clear.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 08:44:45 -07:00
Chris LuandDevin 3fd8d0cf09 master: let location-constrained assigns wait for pending growth (#12167)
* topology: add AvailableSpaceWithinLocation for location-scoped free slots

AvailableSpaceFor answers for the whole topology; a location-constrained
assign needs to know whether the requested data center, rack, or data node
still has room before deciding growth can satisfy it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: let location-constrained assigns wait for pending growth

An assign carrying a data center or rack constraint failed on the first
unsatisfied pick, even when the same request had just queued volume growth
for that location — so the first write into a fresh collection errored out
instead of picking up the volume growth landed a moment later.

Keep the fail-fast path only when growth cannot help: growth disabled, no
growth pending, or no free slots left inside the constraint. Otherwise the
request takes the same wait/shed path as an unconstrained assign.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: sum leaf free slots in AvailableSpaceWithinLocation

Rolled-up rack and data center counters keep a maintenance-mode server's
capacity, and one overcommitted node can cancel a sibling's slots. Sum the
matching data nodes directly so the location check agrees with what growth
can actually use.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: let constrained growth initiators reach the shed path

When a constrained assign's own growth concluded unfulfilled, the fast-fail
check saw only that no grow was pending and returned the plain pick error —
a code the assign client does not retry, so the first write died instead of
re-triggering growth. Exclude the request's own initiated grow from the
no-pending-growth condition so it reaches the existing re-pick and
ResourceExhausted shed like an unconstrained assign.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: judge targeted grow requests by the location they name

A grow request carrying a data center, rack, or data node constraint was
discarded whenever the layout's writables elsewhere were healthy, so a
location with free slots but no writable could never receive the volume its
assign was waiting for. Add ShouldGrowVolumesWithinLocation and use it for
constrained requests; the layout-wide check still governs untargeted ones.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: only positive leaf slots count in AvailableSpaceWithinLocation

An over-committed node reports negative free slots; summed plainly it\ncancelled a sibling's room and a constrained assign could fail fast\nwhile the location still had space to grow into.

* master: fail constrained assigns fast when growth cannot place

A constrained assign with free slots in its location still waited out\ngrowth that could never satisfy the replica placement — a single-rack\ndata center asked for 010 sheds the same doomed grow on every client\nretry. Run the grow-time node search as a dry run and return its\nplacement error instead of a retryable shed.

* master: judge constrained growth by layout, not free slots

A slot-level feasibility probe races with the grow it judges: a\npending grow consuming the last slot in one rack reports the\nplacement impossible while it mints the request's own volume, and\nthe hard error skips the re-pick that would find it. GrowthBlocker\nnow counts racks and data nodes inside the named location — shape a\ngrow cannot consume — so only genuinely unplaceable layouts fail\nfast.

* topology: GrowthBlocker mirrors the grow selector's feasibility

The location filter bound every replica tier, so a 010 request naming one rack needed two racks inside it while the selector constrains only the first placement and counts the rest by room. And counting nodes without free slots let a full location queue a grow it could never serve, so the assign shed ResourceExhausted at timeout instead of failing fast. GrowthBlocker now applies the option's location to the primary placement chain only and leaf-sums free slots the way the selector does, so it reports a blocker exactly when the grow can never place.

* master: repick once before surfacing a growth blocker

An in-flight grow can spend the last free slot minting the request's own volume: the blocker's slot counts predate that registration by a heartbeat, so returning the placement error on the first failed pick could reject a write whose volume lands a moment later. A pending grow now earns one repick grace before the blocker error is returned.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 08:10:45 -07:00
Chris LuandDevin 18691e7b5b server: keep the first write on a fresh weed server from stalling (#12149)
* master: end the post-mint allocation hold on the embedded reporter's clean inventory

DeferAllocation armed a warmup-length hold whenever the leader minted a
fresh topology id, and on weed server the hold ran the whole window: the
embedded volume server's join kept refreshing the quiet evidence clock, so
the two-pulse early release still cost the first write 10-15s on every
fresh start. weed mini already releases on its single pinned reporter's
first inventory, but pinning refuses outside volume servers, which weed
server must keep admitting.

Add a non-exclusive variant: the topology names one expected reporter
whose first clean full inventory lifts the hold, while other reporters
stay admitted and still feed prior-state evidence and the topology-id
freeze. A single-master weed server points it at its embedded volume
server — the one reporter guaranteed to arrive quickly; a veteran's
heartbeat carries prior state and keeps the window through the regular
checks, so a wiped -dir with surviving external volume servers still
trips the freeze on first report. Restarted clusters are untouched: a
recovered topology id means no mint and no hold.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: announce on the startup peer scan for embedded single-master setups

A fresh filer whose startup peer scan came back empty re-polled for the
full confirm window before registering on the master, so the lock ring
had no member and routed traffic such as S3 PUTs spun on "no lock server
found" for ~10s on every fresh weed server / weed mini start.

For setups that start with exactly one filer the scan is already final:
a single-master weed server and weed mini announce as soon as it
completes. A late external filer still joins and replays through the
normal path — the confirm window only ever caught peers arriving inside
it, and the residual risk mirrors the accepted volume-side trade-off: an
embedded filer announcing a few seconds earlier can own the lock ring
while a surviving external filer's backlog is still on its way.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: retry a failed heartbeat promptly until first registration

A heartbeat stream refused before the leader is known — the common race
when a volume server starts while an election is still running — always
fell back to sleeping out the full pulse, so a fresh volume's first
registration could wait ~5s behind a leader elected a second later.

The peer refresh already identifies the leader during the wait, so cap
the retry pause at a second once a leader is named; while no leader is
known the same short retry applies until the first stream is confirmed,
since an unregistered server has nothing else to do. Once the server was
registered the pulse backoff is unchanged, so a lost leader still paces
reconnects the way it did before.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: retry a failed heartbeat promptly until first registration (rust)

Mirror the Go heartbeat retry: cap the post-refresh wait at a second when
a leader is named, and keep the short retry until the first stream is
confirmed so a fresh volume registers promptly when it races an election.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: leave admitted reporters a confirm pulse when the embedded report ends the hold

The single-master weed server releases its post-mint allocation hold on
the embedded volume server's clean report, but outside volume servers
stay admitted and one may still carry prior cluster state — an inventory
from the embedded server alone cannot prove none is due. Clearing the
hold outright let a rebuilt master re-issue a volume id a reconnecting
veteran still owned.

Collapse the released hold to one heartbeat interval instead: late
reporters land their evidence or a topology-id freeze inside the pulse,
and a re-mint re-arms that pulse rather than the full warmup. The pinned
single-reporter release (weed mini) still drops the hold outright, since
no other reporter can be admitted.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: keep the empty-membership confirm on embedded single-master weed server

A single-master weed server still admits external filers, so an empty
startup peer scan is not final: after a master restart the existing
filers simply have not reconnected yet. Announcing on the scan let the
embedded filer take lock-ring ownership and serve writes that a peer's
replayed backlog could then overwrite.

Only weed mini keeps the skip, where the embedded filer is the only
member its dev cluster can have. The confirm window only delays the
master listing for a fresh store; the filer serves while it runs.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: tolerate an expired heartbeat retry deadline (rust)

refresh_leader_from_peers can consume the whole pulse budget, leaving
retry_at in the past; subtracting a later Instant panics and kills the
heartbeat task instead of retrying. Match the Go side, where
time.Until of a past deadline just sleeps zero.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: retry the seed list without a second pulse sleep (rust)

The error path already slept its retry delay; an extra full pulse after
losing the round delayed the next attempt past what Go's heartbeat()
does, which paces retries only inside the per-master attempt.

* master: keep a configured allocation hold shorter than the confirm pulse

Once the embedded report lands, the collapsed hold replaced the
configured deferral with a full pulse even when the operator set a
shorter one, including zero (disabled). Cap at the pulse instead.

* filer: end the empty-membership confirm early on a fresh-minted embedded master

On a fresh single-master weed server the announce gate still held the
filer for the full confirm window, so the master's lock ring stayed
empty and the first write retried 'no lock server found' for ~7s on
top of the collapsed allocation pulse. A fresh TopologyId mint proves
the master's persisted state was empty, making the empty peer scan
final; skip the rest of the confirm only then, and keep it whenever
the master recovers a prior id.

* filer: keep a short reconnect window on a fresh-minted embedded master

A fresh TopologyId mint proves the master's persisted state was empty,
not that no external filer survived its loss — one can still be
mid-reconnect on its keepalive cycle when the confirm starts. Ending
the empty-scan confirm on the first poll let the fresh filer register
and serve routed reads before that peer was found and backfilled, so
existing files could read as missing. The fresh-mint path now shortens
the confirm to a few keepalive cycles instead of skipping it: a
reconnecting filer still lands in time, while a genuinely fresh cluster
announces after ~3s rather than ~10s.

* volume: skip the heartbeat loop when no seed masters are configured

An empty seed list made the outer loop spin with no await between
passes and no path to observe the stop flag. A masterless volume server
has nothing to register with, so the heartbeat exits after a warning.

* volume: skip the heartbeat loop when no master addresses are given (rust)

Mirrors the Go heartbeat: an empty seed list spun the outer loop with
no await and no shutdown checks. Exit after a warning so library
callers get the same behavior as the binary path that validates the
list up front.

* filer: re-check the fresh-mint hint on each confirm pass

The topology id is minted after raft election, which can land after
the empty-scan confirm has already started — a one-time check at entry
missed the mint and held the full window. Evaluate the hint on every
poll and shorten the confirm from the first observed mint, still
capped by the normal confirm deadline.

* master: mark a fresh topology mint only when the volume bound is zero

Recovery restores the max volume id unconditionally but the topology id
only when present, so an upgraded or partially rebuilt master can mint a
new id while still holding prior volume state. Marking fresh there would
let the embedded filer shortcut its peer confirm on a cluster that is
not empty; only mark the mint fresh when no bound survived.

* filer: drop the fresh-mint shortcut once a peer has been observed

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 08:08:26 -07:00
80f3cc0613 topology: keep oversized volumes out of the writable list across re-registration (#12120)
* topology: keep the oversized verdict after the location list is gone

The per-replica oversized marks live in the volume's location list, which
is deleted when its last location leaves and recreated flagless on the
next registration. Between that recreation and the next heartbeat that
re-measures the volume, ensureCorrectWritables saw no mark and
re-admitted an oversized volume to the writable list; two other paths
(quarantine release, mark-writable) never consulted the marks at all.

VolumeLayout now remembers the oversized verdict per vid, written only
by heartbeat sizes and forgotten only when a report measures the volume
under the limit. The writable admission gate itself refuses a marked
volume, and the vacuum commit path drops a verdict its live size check
has just disproved.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: mark oversized at registration when the report carries a size

RegisterVolumeLayout ran EnsureCorrectWritables while the recreated
location list still held no per-replica marks. When the registering
report knows the volume's size, refresh the oversized state before the
writable check instead of waiting for the next heartbeat pass.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: test that oversized volumes stay out of writables across re-registration

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: remember the oversized verdict per replica

A shared per-volume verdict was cleared by any single replica's
under-limit report, so a still-oversized replica whose location entry
had died could re-register from a size-less report and put the volume
back on the writable list. Keep one verdict per reporting replica: an
under-limit report clears only its own, and a replica that never
returns cannot pin its volume out of writables forever because the
verdict ages out. The vacuum commit now feeds the node's fresh answer
through the same per-replica bookkeeping instead of erasing the whole
verdict.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: count a volume at the size limit as full on vacuum commit

The layout's oversized test is >= the limit, but the vacuum commit and
status checks used >, so a replica sitting exactly at the limit passed
as not-full and the commit path could re-admit it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: cover per-replica oversized verdicts in tests

One replica shrinking must not clear a still-oversized replica's
remembered verdict; cover that plus a size-less re-registration before
the second replica is re-measured. The vacuum-commit case now shrinks
the stored record first, since the commit re-derives the mark from the
node's own answer.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: judge the vacuum commit by the size it just measured

SetVolumeAvailable rebuilt the oversized verdict from the cached
VolumeInfo, which still carries the pre-compaction size until the next
heartbeat. A commit that shrank a volume under the limit re-marked the
replica oversized and never restored writes. The commit path already
asks every replica for its fresh size; pass it through per replica and
record it on the mark, the verdict, and the size tracking seed.

* topology: hold an expired oversized verdict while its replica is back

The TTL forgives a replica that never returns, but a remount announces
the volume without a size and registration skips the oversized check.
Deleting the verdict on expiry then re-admitted a still-oversized
volume until the next size-bearing heartbeat. Keep counting an expired
verdict while its replica sits in the live location list, and keep
sweeping verdicts whose replicas stayed gone.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chrislusf@gmail.com>
2026-10-26 23:22:15 -07:00
6ebc055a94 master, volume, filer: start on a read-only mounted directory and serve existing data read-only (#12081)
* master, volume, filer: start on a read-only mounted directory and serve existing data read-only

A read-only mount was treated as writable (permission bits only), so every
service failed on its first write: the master panicked writing raft conf.tmp,
the filer exited opening the leveldb LOCK, and the volume server skipped every
volume while still offering the directory as free capacity.

- util.IsReadOnlyMount: access(W_OK) == EROFS.
- volume: on a read-only -dir take the existing O_RDONLY/noWriteOrDelete load,
  index .idx in memory when no .sdx exists, offer no free slots (same branches
  as isDiskSpaceLow), skip the vid stamp there.
- master: on a read-only -mdir run raft from a private copy and refuse new
  volume ids.
- filer: leveldb2/leveldb3 set their existing ReadOnly option.

* volume: skip read-only mounts when picking EC shard copy targets

A shard copy landing on a read-only mount fails at write time; offer
only writable locations like the other placement paths do.

* volume: mirror the read-only mount handling in the Rust volume server

The .dat open only fell back to read-only on EACCES, so a real read-only
mount (EROFS) still failed the load. The index side already degrades
once no_write_or_delete is set. Add is_read_only_mount via access(W_OK)
== EROFS, mark DiskLocation.read_only_mount, and mirror the Go checks:
no free slots, heartbeat max = used, no vid-stamp writes, no EC shard
copy targets, and a read-only index dir forces the volume read-only.

* volume: write vid stamps while only the index dir is read-only

* filer: skip the write-bit check when a leveldb store opens read-only

* filer: serve reads of buckets with no leveldb3 dir on a read-only store

* volume: derive a stable dir uuid when a read-only mount cannot store one

* master: keep the read-only raft copy across restarts so votes survive

* volume: index a missing .idx in memory on a read-only mount

* volume: release the .idx handle after an in-memory read-only load

* volume: mount EC shards read-only when their index dir is on a read-only mount

* volume: mark the index dir read-only before the tail repair opens .idx

* volume: write vid stamps while only the index dir is read-only (Rust)

* volume: derive a stable dir uuid and report no free slots on a read-only mount (Rust)

* volume: index a missing .idx in memory on a read-only mount (Rust)

* volume: refuse an EC copy whose shard owner is on a read-only disk

* master: key the read-only raft copy by the absolute -mdir

Two masters sharing the same relative -mdir text from different working
directories would have collided on one temp copy and shared raft state.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: mix device and host into the read-only dir uuid

Deriving the uuid from the path alone let independent servers at the
same mount point report one uuid, which RegisterUuids rejects. Hash in
the device id and hostname so restarts stay stable but disks differ.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: vic6996-web <320442181+vic6996-web@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 09:28:49 -07:00
Chris Lu 53834117a6 mini: lift the post-mint allocation hold on first inventory report (#12061)
* mini: lift the post-mint allocation hold on first inventory report

DeferAllocation armed a fixed warmup window whenever the leader minted a
fresh topology id, so the first write on a brand-new weed mini blocked
for the whole window even though no prior cluster state could exist.
The window still has to cover a rebuilt master's veteran servers, which
report on their first post-reconnect heartbeat.

Mini has exactly one possible reporter — the embedded volume server,
whose first heartbeat carries the full inventory — so hold allocation
only until that report is applied. The bound raise and topology-id
freeze checks run while the heartbeat is processed, so a wiped master
state still raises the bound or trips the freeze before the hold lifts.

Standalone weed master and weed server keep the fixed window: volume
servers outside the process can still be on their way.

* master: apply queued topology-id reports before opening the allocation gate

SetTopologyInitPending(false) cleared the pending flag before replaying
topology ids stashed while the local id was still being minted. A
NextVolumeId landing in the drain window saw no gate and no freeze yet.
The post-mint allocation hold used to cover the gap by delaying the
winner past the freeze; once the hold can release on first inventory,
the gate must stay closed until the queued checks have run.

* mini: gate the first-report release on single-master mode

releaseAllocOnFirstReport assumes the embedded volume server is the only
possible reporter of prior cluster state. A mini started with
-master.peers joins other masters, whose own volume servers can still be
on their way when the first report lands, so keep the regular
quiet-pulse hold there.

* master: check reported topology id on empty full reports

A HasNoVolumes heartbeat carries the same LastSeenTopologyId a populated
list does, but the mismatch freeze only ran for non-empty reports. An
EC-only veteran could therefore release the post-mint allocation hold
through NoteInventoryApplied while its stale topology id never reached
CheckReportedTopologyId.

* master: drain the topology-id stash inside the init-gate lock

CheckReportedTopologyId only held reportedTopologyIdsLock while stashing,
so a report in flight during SetTopologyInitPending(false) could land its
freeze after topologyInitPending cleared, letting a NextVolumeId slip
through the gap. The whole check now runs under the lock, and the gate is
released inside the same critical section as the drain.

* mini, master: pin and enforce the single volume-server reporter

isSingleMasterMode only checks master peers; SendHeartbeat still accepted
any volume server, so a late foreign report could carry prior cluster
state after the first-report release dropped the hold. Mini now pins the
embedded volume server's address and the master refuses registration from
any other reporter while ReleaseAllocOnFirstReport is set.

* master: admit the volume-server reporter before applying heartbeat state

* test: cover SendHeartbeat admission ordering

A refused reporter must not freeze growth or raise the id bound, and the
pinned reporter's stale topology id still must.

* mini: re-derive the single-reporter release on every run

The package-level master options survive in-process reruns, so a
multi-master mini following a single-master one must not inherit the
first-report release or a stale reporter pin.

* master: admit addressless first heartbeats under the reporter pin

An unregistered heartbeat with no Ip skipped the single-reporter
admission check but still reached the seen-bound raise and the
topology-id freeze before the delta-heartbeat skip. Admission now runs
for every unregistered heartbeat; an empty address can never equal the
pin.
2026-10-25 09:25:39 -07:00
Chris LuandDevin da1d7444b8 master: shorten the first-write hold after a fresh topology id (#12059)
* master: release the post-mint allocation hold after a quiet pulse

DeferAllocation armed a fixed warmup window whenever the leader minted a
fresh topology id, so the first write on a brand-new cluster blocked for
the whole window even though no prior state could exist. The window
still has to cover a rebuilt master's veteran servers, which report on
their first post-reconnect heartbeat.

Track the last heartbeat that could carry prior cluster state — a server
joining, held volumes or shards, a nonzero id bound, or a foreign
topology id — and lift the hold one pulse after it goes quiet, capped by
the original window.

* master: make the post-mint allocation hold configurable

master.volume_growth.allocation_deferral overrides the warmup window a
leader holds volume id allocation for after minting a fresh topology
id. Unset keeps the existing window; "0s" disables the hold for setups
that know the cluster is fresh (test harnesses, single-node labs); any
other duration replaces the cap while the quiet-pulse early release
still applies.

* master: re-check the allocation deadline while it is held

NextVolumeId slept for the whole computed delay, so a heartbeat landing
mid-sleep that pushed the quiet deadline out was ignored — the pending
allocation could still run ahead of a straggler's report. Wait in short
slices, re-evaluating the deadline and the growth freeze each round.

* master: count only ids above the issued bound as prior-state evidence

Heartbeat echoes of volumes and shards this master allocated during the
hold refreshed the quiet window, so each growth allocation re-extended
the deferral until the hard cap. Evidence now requires an id beyond the
bound this leader has issued, a higher reported bound, or a foreign
topology id; a joining server (dn == nil) still counts once.

* master: reject an unparseable allocation_deferral instead of disabling the hold

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: do not count a skipped heartbeat as a joining server

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: assert the allocation hold extends past the original quiet deadline

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: release the post-mint hold after two quiet pulses

One pulse proved short for a veteran that reconnects on a staggered
restart: with only five seconds of quiet, allocation could resume and
reuse ids before a slower server landed its first heartbeat. Two pulses
still halves the fresh-cluster first-write stall versus the full warmup
cap, and known-fresh setups keep the "0s" override.

* master: pick once more when assign times out as growth concludes

The pick loop's final attempt can predate the growth it triggered
concluding: growth registers its volumes in the layout before clearing
the pending flag, so a timeout landing on that boundary used to shed a
hard 406 over volumes that had just arrived. Pick once more after the
loop in both the http and grpc assign paths before deciding how to
shed.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-25 06:17:42 -07:00
Chris LuandDevin 514c0c1e99 topology: stop a stale duplicate-vid report from re-quarantining the survivor (#12032)
* topology: drop a duplicate-vid report whose own copy is already gone

flagDuplicateVolumeId runs on a goroutine after registration, so by the
time it executes the reporter's copy may have been unregistered (a
finished move drops the losing copy). The surviving copies are then
legitimate, yet the stale report still quarantined them and stored a
phantom (node, vid) key for the departed reporter. Skip the marks when
the reporter no longer serves the volume under its own layout key.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(topology): cover a stale duplicate report against a lone survivor

The existing tests only flag the conflict before the losing copy is
unregistered. Run the delayed duplicate check after the reporter's copy
is already gone and assert nothing is quarantined and the survivor stays
writable, so removing the stale-report guard fails the test outright
instead of needing a lucky schedule.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(topology): cover the enforceQuarantine-driven release retry

A failed writable push re-arms the record, and the next volume report
drains that retry through enforceQuarantine itself - a different entry
point than the pulse-driven pass the other tests exercise.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:05:46 -07:00
Chris LuandDevin 3e5aa8c3b3 topology: mark replica quarantined before recording the pair (#11983)
* topology: serialize quarantine records with their layout marks

A quarantine publishes a (node, vid) record and marks the replica read-only at the layout level; release and heartbeat flag updates must observe that pair as one state. Store-then-mark let a concurrent release delete the record before the mark landed, stranding the surviving replica read-only; mark-then-store let a heartbeat clear the mark before the record was visible, returning a conflicting replica to the writable list.

quarantineLock now covers every record<->mark pair: flagDuplicateVolumeId's quarantine, releaseResolvedQuarantine's undo, the isQuarantined-gated marks on registration, and the quarantine-folded read-only flag writes on heartbeat paths. Remote server pushes stay outside the lock; a push that lands after the record was released is undone, and a failed release push re-adds the record for a later retry.

The release test now waits for in-flight async flags to settle instead of asserting on a single read.

* topology: fold quarantine into RegisterVolume's flag rebuild

RegisterVolume rewrites every replica's location read-only flag from what
its node last reported, and reported flags deliberately exclude quarantine.
Any registration of a shared vid -- a reconnected replica, a new copy, the
self-heal and delta-heartbeat re-register paths -- therefore cleared the
quarantine mark on every replica in the location, while the gate afterwards
only re-marked the registering node. Two call sites had no gate at all.

RegisterVolume now takes an isQuarantined predicate and evaluates it under
accessLock for every replica it touches, so the mark is never dropped in
the first place: a release unmarks after the rebuild, and a quarantine
mark re-applies on the next rebuild, with no window between.

* topology: keep quarantine when a vacuum commit re-adds a replica

SetVolumeAvailable re-adds the replica into the location list; a re-added
slot starts with a cleared read-only flag, so a quarantined replica came
back flag-free and the next ensureCorrectWritables could return a
conflicting volume to the writable list. Pass the quarantine state in and
set the flag back under the same accessLock, with the caller holding
quarantineLock so release cannot interleave between the check and the mark.

* topology: merge quarantine records instead of overwriting them

enforceQuarantine wrote back the state copy it loaded before its remote
push, so a concurrent writer's enforced=true was lost: the server stayed
read-only while the record said unenforced, and a later release skipped
the writable push it owed. The write-back now reloads the record under
quarantineLock and merges -- enforced wins from either side, reported is
taken from the newer pre-enforcement observation only while no push has
landed yet.

All three compensating writable pushes had the same hole in reverse: a
failed un-quarantine left the server read-only with no record to retry
from, and release's restore blindly overwrote any record a fresh flag
pass had created in between. They now go through restoreQuarantineRecord,
which adds the record only when absent and otherwise sets enforced on
the current one, preserving its reported flag.

* topology: hold quarantineLock over registration's quarantine check-and-mark

RegisterVolume reads the quarantine record and marks the replica
read-only under accessLock, but the pair ran without quarantineLock.
A release could push writable, watch the record get re-read, and then
delete it: the replica stayed read-only with no record left to clear,
and unchanged writable heartbeats skip SetReplicaReadOnlyFlag so the
mark never repaired. Holding quarantineLock around the check-and-mark
in all three production callers serializes it against the release
clear-and-delete.

* topology: serialize remote quarantine pushes behind a record recheck

releaseResolvedQuarantine deletes the record under quarantineLock and
sends the writable request after unlock. A conflict flagged while that
request is in flight creates a fresh record and marks the replica
read-only again, and the older writable request can still land last,
leaving the server writable under an active quarantine. The reverse
flap strands a stale read-only mark the same way.

Route every QuarantineVolumeFn call through pushQuarantine, which
holds quarantinePushLock across the remote call and re-checks the
record under quarantineLock right before sending: a read-only push
requires the pair to exist, a writable push requires it to be gone.
A skipped push means the decision already moved on, so the final
server state always follows the newest record.

* topology: serialize quarantine pushes per replica, not globally

A single quarantinePushLock held across the remote mark call let one slow
volume server (up to the 10s client timeout) delay pushes for unrelated
replicas. Keying the lock by (node, vid) keeps pushes to one replica
ordered in record-decision order while pushes to others run in parallel.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: restore the layout read-only mark with a released record

A failed writable push re-armed the quarantine record but left the
replica marked writable in the layout, so PickForWrite could route to a
replica the server still holds read-only.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: wait out async duplicate checks in the release-retry test

TestFailedReleasePushRestoresQuarantineMark asserts layout marks around a
release pass, but the two RegisterVolumeLayout calls each start a
background flagDuplicateVolumeId that can still be mid-flight. A late
mark could land after the final release and fail the writable assertion
even though the code behaved correctly. Track the async passes in a
WaitGroup and drain them before the explicit flag call so the rest of
the test is deterministic.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: keep quarantine when a replica's layout key is unreadable

* topology: keep a failed writable push's restore ordered under the push lock

* topology: retry the quarantine release on every duplicate check

* ci: isolate Java and S3 test servers on persistent runners

* ci: drop the duplicated -ip flags in the lifecycle test server

The bind flags were already on the command; the isolation pass added a
second pair.

* topology: retry a failed quarantine writable push on the next report

* topology: retry pending quarantine releases off the leader pulse

A failed writable push re-arms the quarantine record with a retry that only
enforceQuarantine fires, and that runs only for volumes a heartbeat carries.
Once the server has reported the replica read-only it drops out of the delta
reports, and restoreQuarantineRecord keeps the stored flag matching the
server's so the digests agree and no full list is requested - the pending
release then never retries and the idle volume stays read-only forever. Run
the same release pass from the leader's pulse loop so a pending retry no
longer waits on a report.

* topology: exercise the pulse-driven quarantine release retry

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:00:45 -07:00
Chris LuandDevin 168789d5f3 master: cap writable volumes per collection via -maxWritableVolumes (#11923)
* master: cap writable volumes per collection via -maxWritableVolumes

Some deployments serialize disk writes (e.g. an Unraid array where
concurrent writes to multiple disks collapse throughput). With more
than one writable volume in a collection, the master's weighted pick
spreads assignments over all of them, so several disks get written at
once.

-maxWritableVolumes=N narrows the pick candidates to the N fullest
writable volumes, so a collection's writes concentrate on one (or few)
volumes at a time. 0 keeps the current spread-everywhere behavior.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: give master.follower a maxWritableVolumes default

* master: cap writable volumes inside the requested location

* master: pick the capped volumes without sorting the writable list

* master: assert the capped candidate set instead of random pick outcomes

* master: describe -maxWritableVolumes as a per-layout cap

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 17:04:36 -07:00
Chris Lu 4d06742664 Detect and quarantine duplicate volume ids (#11763) (#11828)
* master: detect and quarantine duplicate volume ids across layouts (#11763)

* shell: volume.check.duplicate_ids audit command (#11763)

* master: persist duplicate-vid quarantine across heartbeats (#11763)

* master: quarantine all copies of a duplicated volume id

Near-simultaneous registrations let each side quarantine only the other,
so arrival order decided which copy stayed writable. A confirmed
cross-layout collision means either copy could be the impostor, so flag
every colliding copy read-only regardless of registration order.

* topology: return a copy from VolumeLayout.Lookup

Callers iterating the returned slice raced with registrations and
removals replacing the location list in place.

* master: release duplicate-vid quarantine when the conflict resolves

A cross-node volume.move briefly shows the same id on two nodes under
different disk layouts; permanent quarantine would pin a valid target
read-only. Keep the first-reported state per quarantined copy and lift
the marks once the conflicting copy unregisters, so only a persistent
collision stays read-only.

* shell: include ec shards in volume.check.duplicate_ids

Ec-only volumes share the global id space. Compare shard collections
against regular ones while allowing shards and regular copies of the
same encoded volume.

* master: keep quarantine release current and cover all removal paths

Track the latest server-reported read-only flag while quarantined so a
release restores the server s true state, and run the release check on
collection deletion and data-node removal too, not only explicit volume
unregistration.

* shell: do not flag ec-only volumes as duplicates

An ec-only volume has no regular copy, so its shard collections had
nothing to compare against and always read as foreign.

* master: quarantine every conflicting layout at the layout level (#11763)

- collect all conflicting layouts, not just the first, so a vid spanning
  three layouts does not leave one writable
- mark quarantine on the location list only: folding it into the stored
  report made the volume digest permanently disagree with the server and
  forced a full volume list every heartbeat
- isAllWritable now honors the location read-only mark so a quarantined
  copy stays out of writables through resyncs and re-registration
- re-check the conflict before keeping marks, and do not clear a node's
  quarantine while it still holds the vid in another layout
- log the full layout keys on both sides of the conflict

* master: push duplicate-vid quarantine to the volume server (#11763)

A quarantined copy stayed writable on the volume server itself for a
client holding an existing file id, since the master-only mark never
reached the server's local write path. Quarantine now also calls
VolumeMarkReadonly (can_delete so expiry still drains), is re-pushed if
the server comes back reporting it writable, and VolumeMarkWritable
runs when the conflict resolves and the server had reported writable.

* master: retry unenforced quarantine marks and restore server writability on release (#11763)

Dial the advertised grpc port, return mark errors, and keep a
quarantine entry until the writable push succeeds so a released
copy does not stay read-only on the server.

* master: treat rp-only layout differences as an in-flight replication change (#11763)

volume.configure.replication rolls through nodes one at a time, so a
volume legitimately reports the old and new replica placement while it
settles. Layouts differing only in rp within the same collection are a
supported transition, not a reused id.

* shell: group ec shards by collection in the duplicate id audit (#11763)

EC shards of one volume legitimately span disk types when
DiskTypePrefer spills, so grouping by disk type reports healthy
volumes as duplicates.
2026-10-20 07:37:33 -07:00
Chris Lu 2874bd05fb Freeze volume growth on topology id mismatch (#11763) (#11820)
* pb: ResumeVolumeGrowth rpc for operator-driven growth resume (#11763)

* master: freeze volume growth on topology id mismatch (#11763)

* shell: volume.growth.resume command (#11763)

* master: replicate the growth freeze and gate it on veteran heartbeats (#11763)

* master: block allocation while the topology id is being minted

DeferAllocation only armed after the topology id persisted, so growth
requests arriving during init ran ahead of the warmup window. Gate
NextVolumeId on an init-pending flag spanning the whole mint.

* master: stash reported topology ids until the local id is minted (#11763)

* drain stashed topology id reports when the id arrives outside the mint path

* persist the growth-freeze flag in the mirror file and raft snapshots

* log hashicorp raft marshal and apply errors in applyGrowthFreeze
2026-10-20 07:34:31 -07:00
Chris Lu 0b3ed4502f Design + master mirror of max volume id outside raft (#11763) (#11818)
* design: durable volume-id allocation boundary (#11763)

* master: mirror max volume id to a file outside raft (#11763)

The allocation bound lived only in raft state, so a master that lost it
rebuilt the bound from whatever volume servers were connected and could
reissue ids held by offline servers.

NextVolumeId now persists the bound to <mdir>/max_volume_id before the
raft apply, and both raft implementations persist on apply and recovery
so follower mirrors stay current. The file also carries the topology id,
which survives raft state cleanup the same way the snapshot recovery did.

* master: serialize and fully sync the max volume id mirror (#11763)

* master: rewrite the mirror when the topology id changes at the same bound

A same-bound raft apply carrying a new topology id was skipped by the
lastPersistedMaxVolumeId check, leaving the mirrored topology id empty
until the next allocation. Track the mirrored topology id too and
rewrite when either value differs.

* drop the design doc from the PR stack
2026-10-19 02:42:42 -07:00
492e700d1f Volume grow avg fill threshold (#11725)
* master: add conditional grow by average fill ratio to VolumeGrow gRPC

Callers that pre-grow volumes on fill level currently have to list all
volumes of a collection and sum sizes themselves, which is expensive on
clusters with millions of volumes. The master already tracks each
writable volume's effective size in memory, so it can answer the
question directly.

Add grow_avg_fill_threshold to VolumeGrowRequest: when set (e.g. 0.6),
growth only happens once the layout's writable volumes have reached
that average fill ratio. A layout with no writable volumes skips the
check, since that is the case growth exists to fix. VolumeGrowResponse
now reports the observed avg_fill and writable_count whether or not
growth was triggered.

* master: unit tests for WritableAvgFill and FindVolumeLayout

Covers the average-fill computation (mixed fills, no writables, a
writable volume without a size-tracking entry yet) and that
FindVolumeLayout resolves existing layouts without creating missing
collections.

* address review: report avg_fill and writable_count for every VolumeGrow

The stats were only read when grow_avg_fill_threshold was set, so a
request without a threshold got AvgFill=0 and WritableCount=0 even when
writable volumes existed. Read the layout stats unconditionally and use
the threshold only to decide whether growth happens.

* master: scope the grow fill check to the requested target

A VolumeGrow aimed at a data center, rack or node must not skip growth
because fill elsewhere keeps the layout average below the threshold.
Measure only writables serving the target; a target with none still
grows.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 05:30:03 +08:00
41260a55f6 master: record writableVolumeCount on HTTP /dir/assign like gRPC Assign (#11716)
* master: extract the periodic per-layout grow decision

Move the must-grow and crowded-step decision of the periodic growth scan
in ProcessGrowRequest into periodicGrowCount, a pure function of the
layout's lastGrowCount, writable and crowded counts, so it can be tested
without running the scan loop or growing real volumes.

No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: record writableVolumeCount on HTTP /dir/assign

The periodic growth scan (#5781) keeps a layout supplied ahead of
demand only when it knows how many writable volumes clients want:
lastGrowCount, which gRPC Assign records with SetLastGrowCount. HTTP
/dir/assign parsed writableVolumeCount but only used it to size the
emergency grow when no writable volume was left, so the scan always saw
lastGrowCount 0 for HTTP-only writers.

Since #5821 removed the per-assign threshold grow, such a layout only
grew by the rack-aware step of 2, and only once every writable volume
was crowded. When a batch of volumes filled in lockstep, those 2 took
every write and could fill before the next scan, leaving no writable
volumes until an assign-triggered emergency grow landed, with 503s for
every other assign meanwhile.

Record writableVolumeCount on HTTP assigns as gRPC Assign does. Clients
that do not send it are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: reject unknown rack/dataNode before recording grow targets

An assign naming a rack or data node outside the topology can never be
served, but lastGrowCount was recorded first. The periodic growth scan
drops the requested placement, so it kept topping the layout up with
volumes the requester could not use. Both assign paths now validate the
placement like they already validate the data center.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: scope placement existence checks to the requested path

A rack in another data center or a data node under another rack passed
validation yet could never serve the request, and the grow count was
still recorded. The rack and data-node lookups now take the requested
parents, matching the filters the placement engine applies.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 18:10:41 +08:00
ihnokim 11e8c4c288 master: follow heartbeat read-only changes in the layout's replica flag (#11527)
A replica's read-only flag in the volume layout only moved on registration
and on volume.mark. A change that arrived in the regular heartbeat updated
the node's record, which the writable list follows, but not the layout
flag, which the vacuum sweep reads. So the sweep kept trying volumes on a
disk that had gone read-only while the server ran, and after a restart it
skipped volumes that had since become writable again until the next
restart (issue #11516).

Apply the reported state to the flag for every changed volume. Only the
flag: the writable list stays with EnsureCorrectWritables and its
capacity guards.
2026-10-01 01:09:09 +08:00
Chris Lu 15e4da65f7 volume: avoid read-only replica write targets (#11195)
* master: carry replica read-only state in volume lookups

* volume: refresh writable replica targets

* volume: preserve read-only replicas for deletes

* master: propagate read-only delete capability

* volume: target delete-capable replicas

* volume: honor configured HTTPS for replica deletes

* volume: reject insecure delete authorization forwarding

* master: broadcast delete capability changes

* volume: align Rust replica routing

* http: protect credentialed replica redirects

* master: preserve digest compatibility for delete capability

* volume: propagate read-only state in short heartbeats

* volume: report changed short volume state

* http: guard TLS client redirects

* master: announce mounted volume read-only state

* volume: replace changed identity deltas

* master: replace incremental volume layouts in order

* master: keep moved volume lookup available

* volume: announce read-only mounts
2026-09-07 09:23:56 -07:00
yanglongwei eb717199d0 master: delete replica_placement_mismatch labels when volumes leave topology (#11062)
* master: delete replica_placement_mismatch labels when volumes leave topology

Fixes #10804. Setting the gauge to 0 left stale Prometheus time series
that grew unbounded with volume churn; remove the label set on unregister
instead.

* master: delete replica_placement_mismatch only after last placement leaves

Unconditional DeleteLabelValues on UnRegisterVolumeLayout dropped the series
while other data nodes still held the volume, hiding under-replication until
the next collect cycle. Delete only when Lookup is empty, and cover the
two-copy case in a regression test.
2026-09-05 12:52:27 -07:00
7620e96171 expose whether a volume replica is backed by remote storage, and prefer local replicas (#11105)
* expose whether a volume replica is backed by remote storage

Volume locations returned by lookups do not indicate whether a replica
has been tiered to remote storage. Readers cannot distinguish a local
replica from a remote-backed one, so they may hit a remote-backed
replica first even when a local replica is available.

Add DataInRemote to the lookup location message, populate it from the
master's volume info, and carry it through the wdclient vid map so
clients can prefer local replicas when resolving chunk locations.

* wdclient: prefer local volume replicas over remote-tier replicas on lookup

LookupFileIdWithFallback (and the publicUrl variant in FilerClient)
didn't honor the DataInRemote flag when shuffling URLs, so the
DataInRemote patch only took effect in LookupVolumeServerUrl. Apply
the same ReorderToFront(localUrls) to sameDcUrls/otherDcUrls so
non-remote replicas stay at the front, matching the existing vidMap
convention.

* wdclient: propagate DataInRemote across tier transitions on existing replicas

When a volume is tiered to remote storage or a remote-backed replica is
restored locally, the cached DataInRemote on the same volume-server URL
stayed at its old value because two pieces of state never updated:

* master_grpc_server.go only split newVolumes and (already-tracked) volumes
  into NewVids vs RemoteVids. ChangedVolumes went straight to NewVids, so
  the broadcast announced the re-classified volume as a fresh arrival and
  the client had no way to tell whether its existing cache was stale.

* vid_map.addLocationToMap early-returned when an entry already had the
  same URL. A tier transition reports the same URL with DataInRemote
  flipped, so the cached entry stayed at the old classification.

Wire both sides together: ChangedVolumes now go through the same IsRemote
split as newVolumes, and addLocationToMap replaces the existing entry in
place when the URL matches but DataInRemote has changed. The server
reference key only depends on URL/grpc port, so the refcount does not
move across the flip.

Adds vid_map_remote_transition_test.go covering the local->remote and
remote->local paths so the in-place update and the cache-key stability
are pinned by tests.

* wdclient: prefer local replicas across data-center boundaries

The previous local-first ordering hoisted local URLs to the front of each
data-center bucket separately, then concatenated same-DC before other-DC.
That meant a same-DC remote replica could still be tried before an
other-DC local replica even though the local one would answer cheaply.

Reorder once across the full candidate list: concatenate same-DC and
other-DC first, then ReorderToFront pulls every local replica to the very
front while preserving the DC preference inside each tier. Apply the same
ordering in all four lookup paths so the cached vidMap, the
LookupFileIdWithFallback provider path, FilerClient.GetLookupFileIdFunction
(PublicUrl-preferred variant), and the deprecated filer.LookupFn all agree:
- weed/wdclient/vid_map.go (LookupVolumeServerUrl)
- weed/wdclient/vidmap_client.go (LookupFileIdWithFallback)
- weed/wdclient/filer_client.go (LookupFileId)
- weed/filer/reader_at.go (LookupFn)

Strengthen the existing local-first tests: vidmap_client_localfirst_test
now asserts both endpoints are present (not just the local one is first),
and slice_test asserts an exact match instead of accepting two orderings.

Add TestLookupFileIdWithFallbackGlobalLocalFirst to pin the cross-DC
ordering invariant: any local replica (same or other DC) precedes every
remote-tier replica; within each tier DC1 precedes DC2.

Add docstrings to ToVolumeLocations, ReorderToFront, LookupVolumeServerUrl,
LookupFileId, GetVidLocations, GetLocations, LookupFileIdWithFallback, and
updateVidMap so the touched lookup paths are described in one place.

* topology: broadcast tier transitions on existing replicas

When a volume replica is tiered to remote storage or restored locally, the
wdclient's cached DataInRemote went stale: every connected client kept
preferring a remote-backed replica over a freshly restored local one, or
demoted a freshly tiered remote replica. The fix in commit 116982595 routed
ChangedVolumes to NewVids/RemoteVids on the master, but ApplyVolumeChanges
returned only fresh arrivals and previously servable replicas. An existing
replica whose IsRemote() classification flipped was neither, so it never
reached the broadcast loop and the wdclient never learned.

Make Disk.doAddOrUpdateVolume return a third signal -- tierTransition --
true exactly when an existing replica's IsRemote() flips. ApplyVolumeChanges
treats that as an arrival so the existing SendHeartbeat routing loop now
sees it. Add a master-side end-to-end test covering local->remote,
remote->local, no-op re-reports, and a mixed heartbeat that only announces
the tier transition.

Also add docstrings to LookupFileId, wdclientLocationsToPb, and
LookupVolume where the prior change touched their bodies.

* topology: broadcast tier transitions received through full reconciliation

The previous commit added tier-transition routing on the ChangedVolumes
delta path, but that is not the only way a re-tiered replica reaches the
master. After a digest mismatch the volume server resends a full Volumes
list, and SyncDataNodeRegistration applies the new IsRemote() classification
silently -- the changedVolumes return value was being thrown away. The
master therefore never broadcast NewVids/RemoteVids, and a wdclient connected
during the recovery kept the stale DataInRemote until it lost contact with
the master.

Surface the changed set through UpdateVolumes.changedVolumes (now covering
both ReadOnly flips and tier flips) and SyncDataNodeRegistration, then route
it through NewVids/RemoteVids in SendHeartbeat the same way the delta path
already does. Add an end-to-end test for the full reconciliation path.

* master: keep an EC volume's locations in the volume lookup

The nodes that answer for an EC volume hold shards, not a volume record,
so asking them for one fails. Dropping the location on that failure
emptied the result and turned every EC read through the master's HTTP
lookup and fid redirect into a 404.

Treat an absent volume record as a local read and keep the node in the
answer. The per-node conversion moves into topologyLocation so the EC
case is covered by a test.

Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW

* wdclient: replace a tier-flipped location without writing under a reader

GetLocations hands back the entry's own slice and the caller walks it
after the read lock is dropped, which is why every other mutation here
builds a new slice. Writing the flipped replica into the array in place
raced LookupVolumeServerUrl, reported by -race.

Copy the slice, swap the one element, and publish it.

Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW

* master: keep a remote volume on NewVids for older clients

Moving remote-tier volumes out of NewVids and into RemoteVids alone is a
wire break in the wrong direction. A master upgraded ahead of its filers
and mounts -- the usual order -- announces a tiered volume only on a
field the older client ignores, so the volume drops out of that client's
vid map entirely and reads for it fail.

Announce every volume on NewVids and repeat the remote-tier subset on
RemoteVids, so a new client still learns the tier and an old one keeps
the location. The routing moves into announceVolume, which the heartbeat
paths and their tests now share instead of each restating it.

On the client, RemoteVids no longer needs a second write per volume: the
tier is settled before anything is added.

Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW

* topology: split the volume snapshot by tier without copying the records

ToVolumeLocations runs on every KeepConnected, so a filer or mount
connecting made the master allocate a full VolumeInfo per volume per node
just to read four bytes of id off each one. AppendVolumeIds exists to
avoid exactly that.

Extend it to fill the remote-tier list alongside the full one, and use it
again in the snapshot.

Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW

* wdclient: keep the data-center preference ahead of the local-first ordering

Hoisting every local replica to the very front puts an other-DC local
read ahead of a same-DC remote one. When the remote tier sits in the same
region as the replicas -- the common arrangement -- that trades an
in-region GET for a WAN round trip and costs more than the remote read it
avoids.

Reorder inside each data-center bucket instead, so local still wins among
equals and the data-center preference still wins overall.

Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW

* operation: pick the read replica from one list

The local-preferring lookup built a list of local URLs and then branched
on whether it was empty, duplicating the random pick. Fall back by
filling the same list with every replica instead.

Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW

---------

Co-authored-by: Bruce Zou <gift_secondst@msn.com>
Co-authored-by: bruce-zzz <bruce.zou@hhy-data.com>
2026-09-02 16:12:46 -07:00
Chris Lu 5f787a25c3 master: survive a volume layout deleted twice (#11098)
* master: survive a layout deleted twice

Two volume servers dropping the last replica of volumes that share a layout
both find it empty and both delete it. The loser's lookup misses, and the
single-value type assertion on the result crashed the master before the
caller could look at the found flag.

Claude-Session: https://claude.ai/code/session_01WmX6Rchx298NQksHDXg7sk

* master: remove a layout and read it back in one step

DeleteVolumeLayout looked the layout up and then deleted it, so two deleters
could each release the lookup ownership of the same layout, or one could find
nothing to release at all. Have the map hand back what it removed.

Claude-Session: https://claude.ai/code/session_01WmX6Rchx298NQksHDXg7sk
2026-09-02 11:50:48 -07:00
Chris Lu a02c0024e5 master: cap the reported capacity at what the disks hold (#10960)
* master: cap the reported capacity at what the disks hold

Statistics reported max volume count times the volume size limit, which is
how many volumes the cluster is allowed to place, not how much space it has.
A cluster given far more slots than its disks can fill reported a capacity it
could never reach -- 65536 slots at 30GB read as 1.9PB on a 460GB disk -- and
the number never moved, since writing data changes neither the slot count nor
the size limit.

The volume servers already report each filesystem's total and free bytes in
their heartbeats, so bound the answer by what they say is left.

* mount: keep the last known sizes when filer statistics fails

A failed Statistics call returned before df's answer was filled in, so a
mount whose filer or master was briefly unreachable reported an empty
filesystem rather than the sizes it already had.

* master: drop the disk ceiling when a volume server does not report

A cluster part way through an upgrade has volume servers that predate the disk
bytes in the heartbeat. Summing only the ones that answered left the quiet
server's free space out of the total, and the server holding the room is
exactly the one that could make the cluster read as full.

Answer with the disks only when every one of them reported.
2026-08-26 00:12:56 -07:00
Chris Lu b77431c142 master: stop hintless small-file assigns from marking volumes full (#10944)
* master: estimate a hintless assign's size from the volume's average file size

An assign that carries no dataSize hint charged a flat 1MB per file id
against the volume's effective size. A small-file workload overpays by
orders of magnitude: bulk-writing 4KB files marks volumes holding a few
hundred MB of real data as crowded and then full, so the master grows
unnecessary volumes and, once every volume is spuriously full, fails all
assigns. Estimate from the volume's own average file size instead, and
keep the 1MB fallback only for volumes with no history.

* master: decay pending assign sizes for volumes gone quiet

The decay that corrects pending assign estimates runs only when a
heartbeat reports the volume, and a heartbeat only reports a volume
whose content changed. A volume held out of the writable list takes no
writes, so once inflated estimates mark every volume full, nothing is
ever reported again, nothing decays, and the cluster refuses all writes
until a restart. Run the decay from the master's periodic loop for
volumes no heartbeat has reported within two pulses, feeding the last
reported size back through the same path an unchanged heartbeat would
take.

* master: trim the comments on the assign size estimate

* master: keep the periodic decay out of the replica-dedup window

UpdateVolumeSize ignores a report arriving within two seconds of the last
one, so replicas of the same volume do not each halve the pending
estimate. The periodic decay went through the same path and stamped that
window, so a real heartbeat landing right behind it was dropped along
with its reported size and compact revision. Only a volume whose content
changed is reported at all, so nothing would send that size again and
the master kept a stale one. Let the dedup window belong to volume
server reports alone.

* master: let the decay read the size record under the lock it mutates

The periodic decay picked its volumes under a read lock and replayed
them under a write one, carrying the size it had read across the gap. A
heartbeat landing in between was rolled back: the replay wrote the older
size and compact revision over the fresh ones, and a compaction report
lost that way is never resent, since only a volume whose content changed
is reported. The decay has no size of its own to contribute, so it now
reads the record under the same lock it mutates.

* master: let a heartbeat that beat the decay stand for the cycle

The decay chooses its volumes under a read lock and applies them under a
write one. A heartbeat landing in that gap already did the halving the
cycle owed, so applying the decay on top of it halved twice and forgot
pending bytes the volume has not written yet - the double-halving the
replica-dedup window exists to prevent. Both callers now give way to a
report already handled for this cycle; only a real report still advances
lastUpdateTime, so a quiet volume keeps decaying every pulse.

* master: keep genuinely full volumes out of the decay pass

A volume the disk really did fill keeps its fullSince set for good, so it
was selected every pulse for a decay that cannot help it: UpdateVolumeSize
refuses to recover a volume whose reported size is at the limit, and
replaying a size that cannot move leaves the record as it found it. Full
and quiet is the ordinary resting state of a cluster, so this was most of
the pass, taking the layout write lock away from the heartbeats to do
nothing. On a million tracked volumes with a hundredth of them phantom-full
it costs ten thousand write locks a pulse instead of a million.

* master: put the stale-replay test back on the path it guards

Giving the decay the dedup window left this test short-circuiting there,
so it no longer reached the locked read it was written for and passed
with that read removed. Age the record past the window, which is the only
case where reading it under the lock is what saves the report.
2026-08-25 10:16:48 -07:00
Chris Lu 35d53a20f6 master: let the leader admit a master that starts with no raft state (#10865)
* master: answer with the leader raft already knows

Topo.Leader() backs off for up to 20 seconds waiting for an election.
Callers that a health probe or a client is blocked on cannot afford that:
/cluster/status, /cluster/healthz and /readyz all sit past the probe
timeout of both the helm chart and the operator, so a master that is
still joining looks dead rather than joining, and the kubelet restarts
it. informNewLeader and SendHeartbeat hold the client on a master that
cannot serve it, exactly when it should move on to find the one that can.

Answer these from MaybeLeader instead, which reports what raft knows
right now. MaybeLeader takes over the "am I the leader myself" fallback
that Leader() used to apply on top of it, so one non-blocking call is
still correct; Leader() keeps the backoff for callers that must wait.

* master: let the leader admit a master that starts with no raft state

Neither raft implementation lets a server outside the configuration
campaign: goraft's promotable() requires a non-empty log, and hashicorp
rejects vote requests from a candidate that is not in its configuration.
A master that comes up with fresh state therefore cannot elect itself in
— the leader has to pull it in. Nothing did.

The peer list is static, rendered from the replica count, so scaling it
up leaves the sitting leader running the old list with no idea the new
masters exist. Under goraft they wait forever. Under hashicorp they are
worse off: each bootstraps a cluster of its own from the new list, and
two of them form a quorum next to the live leader, with their own
TopologyId. That is the split brain SetTopologyId kills a master over.

Admit the peer where it registers instead. Only the leader gets past the
IsLeader check in KeepConnected, and a joining master's client lands
there, so that is the moment it joins. The broadcast OnPeerUpdate rides
on is not enough on its own: it only reaches masters already connected,
which is why a leader that came up first missed both newcomers.

RaftAddServer grew a goraft branch on the way, so cluster.raft.add stops
silently doing nothing on the default raft, and RaftRemoveServer with it.
Bootstrapping is now one call for both implementations, made only after
the peers confirm nobody has a leader, and retried until this master is
in rather than checked once and dropped.

* master: do not evict a peer that is still in -peers

The hashicorp leader drops a master from the raft configuration as soon
as it stops answering pings. A master that is merely restarting answers
nothing, so an ordinary bounce shrinks the quorum behind the operator's
back — and then races its own return: the master comes back, registers,
gets re-admitted, and the eviction lands after it.

A randomized start/stop walk lands on it. Two of three masters running,
the leader evicts the one that just went down, the restart re-adds it,
the removal commits late and takes the leader's own leadership with it.
What is left is a two-server configuration whose other half is down, and
a running master that nobody will ask for a vote — no quorum, no way
back until the third master returns.

-peers is what declares membership. updatePeers already reconciles the
configuration against it on every leadership change, and an operator who
really means to drop a master can say so with cluster.raft.remove, so
keep the eviction for masters that are no longer listed at all.

* test: bounce masters at random and hold the election to it

Twelve rounds of stopping or starting a random master, on both raft
implementations, checking the two things an election must never get
wrong: two masters claiming leadership at once, and a quorum that comes
back without agreeing on one. The cluster's identity has to survive the
whole walk, since a master that re-mints a TopologyId is the split brain
SetTopologyId kills its peers over. The seed is random and logged, so a
failure names the walk that reproduces it.

Below a quorum the walk moves straight on. A master that has lost its
quorum cannot commit anything, and goraft only checks whether it still
has one on an election-timeout ticker, after its peers have been quiet
for a full timeout — measured taking over 30 seconds to step down. That
direction belongs to TestTwoMastersDownAndRestart, which was giving it
ten seconds and would have started failing on a slower machine; it now
waits on that behaviour explicitly rather than sleeping twice and hoping.

WaitForTopologyId returns the id it waited for. Reading it separately
raced the leader applying the raft entry that carries it, which shows up
as an empty id right after an election rather than as a wrong one.
2026-08-21 15:22:22 -07:00
f7c4636d22 topology: refresh oversized mark on every heartbeat (#10829)
* topology: refresh oversized mark on every heartbeat

The oversized flag on a volume location was only set when the volume was
registered (RegisterVolume). A volume that later grew past the size limit
kept its stale "not oversized" mark, so the heartbeat path
(ensureCorrectWritables) kept re-adding it to the writable list while
RecordAssign removed it on every assign - a writable/unwritable flip loop
that let writes continue past the limit and made vacuum race in-flight
writes.

Refresh the mark from each heartbeat's reported size in both heartbeat
paths (ApplyVolumeChanges and SyncDataNodeRegistration), mirroring what
RegisterVolume already did at registration time. A volume that grew past
the limit now stays unwritable, and one that shrank back clears the mark
and can recover.

* topology: order heartbeat writable correction after decay and honor cooldown

Review feedback (Greptile, CodeRabbit) on the oversized-mark refresh:

1. Greptile: clearing the oversized mark before EnsureCorrectWritables let
   the delay-unaware helper re-add a just-compacted volume to writables,
   bypassing capacityRecoveryDelay. ensureCorrectWritables now checks
   fullSince and skips the re-add while the cooldown is pending, so a
   volume removed for capacity only recovers through UpdateVolumeSize's
   heartbeat recovery path.

2. CodeRabbit: in the full-heartbeat path the mark was refreshed after
   the writable correction, so a newly oversized volume stayed writable
   for an extra heartbeat cycle. The standalone changedVolumes loop is
   merged into the volumeInfos loop and EnsureCorrectWritables now runs
   after UpdateOversizedState + UpdateVolumeSize in both heartbeat paths,
   using the freshly refreshed mark.

3. TestHandlingVolumeServerHeartbeat used a size (254320) that is past
   the test's volumeSizeLimit (32768); it only passed because the stale
   mark hid the oversized state. Sized down to 30000 to keep testing the
   add/remove flow, and added TestEnsureCorrectWritablesHonorsRecoveryCooldown
   covering the cooldown window and the recovery after it.

* topology: do not restore a still-crowded volume after the cooldown

Greptile review: after capacityRecoveryDelay elapses, ensureCorrectWritables
could restore a volume whose effective size is still past the crowded
threshold. UpdateVolumeSize refuses the recovery (effectiveSize > crowded
threshold -> setVolumeCrowded + return false), but the cooldown check in
ensureCorrectWritables only looked at fullSince, so once the delay passed
it re-added the volume even though capacity tracking still considers it
crowded.

Check the crowded mark before re-adding: a volume UpdateVolumeSize just
marked crowded must not be restored here, otherwise assignments resume
while the volume is still flagged for growth.

Adds TestEnsureCorrectWritablesDoesNotRestoreCrowdedVolume: effectiveSize
decays to 10500 (past the 9000 crowded threshold) after a report of 8000,
and ensureCorrectWritables keeps the volume unwritable past the cooldown.

* ci: trigger re-run of flaky FUSE jobs

* topology: gate the writable restore on the limit, not on crowded

A crowded volume is above the growth threshold, not full, and is normally
writable. Refusing to restore one locks it out for good: nothing writes to
a volume that is not writable, so its size can never fall back under the
threshold. Gate on the same size the assign path uses to remove it.

* topology: let only the heartbeat refresh set the oversized mark

Registration also set it, from whatever VolumeInfo it was handed. The
incremental path builds that from a short heartbeat message, which carries
no size, so every arrival announcement cleared the mark and handed the
volume back to the writable list until the next full report.

* topology: use the re-resolved layout after a dropped one is replaced

A layout dropped with its collection makes RegisterVolume refuse, and the
full heartbeat then re-registered against a fresh layout but kept applying
the size, oversized and writable updates to the dropped one.

---------

Co-authored-by: hzsunchao <hzsunchao@corp.netease.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-08-20 09:56:53 -07:00
Chris Lu 3911e4c548 master: keep a racing registration out of a dying collection (#10677) 2026-08-09 22:20:34 -07:00
Chris Lu a2ffc7aadf heartbeat: keep the master current through collection churn (#10657)
* heartbeat: name departed volumes in delta heartbeats

* master: release the lookup index with a deleted collection

* master: keep a fresh grow safe from the report that raced it

* volume: name the volumes a deleted collection took with it

Deleting a collection left the master to work out what went by omission from
the next full volume list, which it no longer gets: heartbeats carry the whole
list only when the master asks for it. The volumes a bucket's churn creates and
destroys between two of those requests are never named in either direction, so
the master keeps counting their slots as occupied and a cluster that creates
and drops collections quickly runs its free-slot accounting dry -- assigns fail
with no free volumes left while the disk holds a handful of volumes.

The destroy path already knows exactly which volumes it removed, so send them
down the same channel every other deletion uses.

* rust: name the volumes a deleted collection took with it

Mirrors the Go volume server. The notify path derives its deltas by diffing
snapshots, so a collection delete that does not wake it is invisible until the
master next asks for the whole list.
2026-08-08 20:23:10 -07:00
Chris Lu ce7d388639 heartbeat: send only the volumes that changed (#10640)
* pb: let a heartbeat carry only the volumes that changed

A partial list cannot travel in volumes: a master that did not understand it
would read the absences as deletions. So changes get their own field, used only
once the master has said it compares digests and can tell when it has fallen
behind.

* master: apply the volumes a heartbeat reports as changed

Only the named volumes are touched. A full report says the server holds exactly
these; a changed report says nothing about the ones it leaves out, so absence
must not read as removal.

Also advertises that the master compares digests, which is what lets a server
stop sending its whole list. Advertising it once per connection means a server
reconnecting to a master that does not is back to full lists straight away.

* volume: send only the volumes that changed once the master accepts them

The whole list goes on every heartbeat until the master says it compares
digests, and again whenever it asks, so a master that cannot tell when it has
fallen behind never has to.

has_no_volumes stays derived from a full list alone. Deriving it from what a
heartbeat happens to carry would make a quiet one read as a server that had
lost every volume, and the master would drop them all.

The digest still covers every volume held rather than the ones sent, which is
what lets the master confirm that applying the changes left it current.

Reporting state is per-connection: a server that reconnects, or reaches a
different master, starts again from the full list.

* volume: let the zero reporting state stand for having told no master anything

A Store built as a literal, which tests do, left the reporting state nil and
panicked on the first heartbeat. As a value its zero form already means nothing
has been reported to anyone, which is exactly the state that sends the whole
list.

* rust: send only the volumes that changed once the master accepts them

Mirrors the Go volume server, with one hazard the Go side does not have: mount
and unmount deltas here are derived by diffing successive heartbeats, so a
heartbeat that carries a partial list would report every volume it left out as
unmounted. Collecting now returns the full set alongside the message, and every
site that diffs uses that rather than what went on the wire.

* volume: do not let a full-list request be lost to the heartbeat it raced

The request arrived while a heartbeat was already being built as a delta, and
committing that heartbeat cleared it, so the master waited for another digest
mismatch before asking again. Count the requests and clear only the one the
heartbeat answered.

* rust: stop marking volumes reported by a heartbeat that is thrown away

The state-notify path collected a heartbeat only to diff its volume list, then
sent a delta message of its own and dropped the one it had collected. Once
collecting recorded what the master had been told, every mount or unmount
silently marked the changed volumes as sent, and the master learned of them
only after a digest mismatch.

Snapshotting no longer records anything, and no longer expires ec volumes
whose deletion that path was already discarding.

* master: announce only the volumes a change actually brought

Every changed volume was broadcast as a new location. Volumes grow constantly
and growth moves no location, so on a busy cluster that told every connected
client about volumes it could already reach, filling bounded broadcast queues
and pushing out the topology updates that matter.

* master: ask for the full list when only one can repair the master

Delta heartbeats stop the full report, and with it the only thing that
re-registers a volume the lookup index lost. The volume server cannot see that
divergence and its digest cannot show it, so the master now checks its own two
indexes agree and asks for the list when they do not.

A node reporting one volume id twice is kept on full lists for the same reason
rather than merely skipped: its digest can never be verified, so nothing else
would tell the master what it had stopped holding.

* master: keep the volume options on every heartbeat response

A volume server takes them from whatever response arrives, and preallocate is a
bare bool with no way to tell off from unmentioned. A response sent to ask for
the volume list therefore turned preallocation off until the server reconnected.

Responses sent mid-stream now start from the configured options rather than
being built field by field.

* master: announce a volume the lookup index had lost

Repairing the index makes the volume servable again, but clients were told it
went when the node dropped out and nothing told them otherwise: the disk map
still held it, so it did not count as an arrival.

Reaching the lookup index is what makes a volume servable, so recovering an
entry there is an arrival as far as clients are concerned, on both the full
report and the changed-volume path.
2026-08-07 23:36:28 -07:00
Chris Lu 3fce1a938d perf(weed/topology): preallocate the heartbeat volume conversion slice (#10607)
* test(weed/topology): benchmark the per-heartbeat volume sync

A volume server re-sends its entire volume list every VolumePulsePeriod, so
SyncDataNodeRegistration is the master's steady-state per-server cost. Give it
a benchmark so allocation regressions show up.

* perf(weed/topology): preallocate the heartbeat volume conversion slice

The slice grows to one entry per volume on the data node, so at 100k volumes
the doubling copies allocate 60MB of garbage per heartbeat. The final length is
known up front.

BenchmarkSyncDataNodeRegistration/100000Volumes  199670102 B/op -> 137725872 B/op
2026-08-07 00:21:16 -07:00
Chris Lu fee3fcb55a mount: report data sizes to df with -df.logical (#10459)
df on a mount shows the space the cluster gives up to the data: every
replica of a regular volume, every shard of an ec one. That is the honest
answer for capacity planning, but it is not the question a user asks when
they want to know how much of their data is stored.

Add -df.logical. The master reports the logical sizes alongside the raw
ones: one replica per regular volume, the data shards of each ec volume
counted once. Free space is divided by the copies the requested
replication makes, so used plus available stays the amount of data the
mount can still write, and it comes off the cluster-wide usage rather
than one collection's, since capacity is cluster-wide too.

Statistics through a filer resolves an unset replication to the filer's
default rather than the master's, matching where the writes it is sizing
for actually land.

The flag governs the quota check too, so a mount has one notion of how
much it is using. A filer that predates the new fields sends zeros, and
the mount keeps reporting the raw sizes.
2026-07-27 14:28:29 -07:00
Chris Lu 152f1a2096 master: count EC volumes in statistics used size (#10457)
Statistics aggregates the volume layouts of a collection, but EC volumes
are tracked outside collectionMap, so they were reported as nothing. A
mount over a cluster whose volumes have mostly been encoded showed a df
used size of a few GiB against terabytes of EC data.

Walk the data nodes and add the EC volumes of the requested collection.
Every shard copy counts, parity included, the way a regular volume's used
size counts every replica, so used size stays the space the cluster
actually occupies.

File count comes from the volume-wide .ecx and .ecj counts, taking the
largest a holder reports rather than summing them: both files travel with
the shards on a move, so several nodes can report the same tombstones.
2026-07-27 14:24:51 -07:00
Chris Lu 19dc085e33 master: statistics used size covers all collections and layouts (#10319)
StatFs on a mount reported cluster-wide total capacity but used size from
a single volume layout keyed by collection, replication, ttl, and disk
type. A mount without -collection therefore showed only the default
collection's usage, hiding data in named collections, and even a
collection-scoped mount missed volumes with a different replication,
ttl, or disk type.

Aggregate used size and file count across all layouts of the requested
collection, and across every collection when the collection is empty,
matching how Topology.Lookup treats an empty collection. Looking up
stats no longer creates a phantom collection as a side effect.
2026-07-12 12:56:10 -07:00
Chris Lu 0f1e50f9ec fix(master): re-register volumes missing from the lookup index
A disconnect/reconnect race could drop a volume from vid2location while it stayed in the data node's disk map, so it showed in volume.list and the admin UI but LookupVolume returned "volume id not found" and never self-healed (the full heartbeat only registered volumes new to the disk map). The full heartbeat now re-registers any reported volume missing from the lookup index, reusing the already-resolved VolumeLayout.
2026-05-24 15:11:09 -07:00
Chris Lu 10cc06333b cluster: restrict Ping RPC to known peers of the requested type (#9445)
Ping previously dialled whatever host:port the caller asked for. Gate
each server's Ping handler on cluster membership: masters check the
topology, registered cluster nodes, and configured master peers; volume
servers only accept their seed/current masters; filers accept tracked
peer filers, the master-learned volume server set, and configured
masters.

Use address-indexed peer lookups to keep Ping target validation O(1):
- topology maintains a pb.ServerAddress -> *DataNode index alongside
  the dc/rack/node tree, kept in sync from doLinkChildNode and
  UnlinkChildNode plus the ip/port-rewrite branch in
  GetOrCreateDataNode. GetTopology now returns nil on a detached
  subtree instead of panicking, so the linkage hooks can no-op safely.
- vid_map tracks a refcount per volume-server address so
  hasVolumeServer answers without scanning every vid location. The
  add path skips empty-address entries the same way the delete path
  already does, so a zero-value Location cannot leak a permanent
  serverRefCount[""] bucket.
- masters reuse a cached master-address set from MasterClient instead
  of walking the configured peer slice on every request.
- volume servers compare against a pre-built seed-master set and
  protect currentMaster reads/writes with an RWMutex, fixing the
  data race with the heartbeat goroutine. The seed slice is copied
  on construction so external mutation cannot desync it from the
  frozen lookup set.
- cluster.check drops the direct volume-to-volume sweep; volume
  servers no longer carry a peer-volume list, and the note next to
  the dropped probe is reworded to make clear that direct
  volume-to-volume reachability is intentionally not validated by
  this command.

Update the volume-server integration tests that drove Ping through the
new admission gate: success-path coverage now targets the master peer
(the only type a volume server tracks), and the unknown/unreachable
path asserts the InvalidArgument the gate now returns instead of the
old downstream dial error.

Mirror the same admission gate in the Rust volume server crate: a
seed-master HashSet built once at startup plus a tokio RwLock over the
heartbeat-tracked current master, both consulted in is_known_ping_target
on every Ping, with InvalidArgument returned for any target that isn't
a recognised master.
2026-05-12 13:00:52 -07:00
Chris Lu ecc0390795 fix(master): eagerly remove volume from writable when assign hits limit (#9108)
* fix(master): eagerly remove volume from writable when RecordAssign hits limit

Previously, a volume was only removed from the writable list by the
heartbeat-driven CollectDeadNodeAndFullVolumes pass, which runs every
pulse (5s) after a 5s heartbeat. Under sustained concurrent writes,
fio-style workloads observed in the field grew volumes 8-20x past the
configured 100MB limit (median 530MB, peak 1.98GB) during that
5-15s detection window.

RecordAssign already tracks effective size (reported + pending) on each
/dir/assign. It now also removes the volume from writable the moment
effectiveSize reaches volumeSizeLimit, and mirrors the activeVolumeCount
decrement that Topology.SetVolumeCapacityFull would have done on the
next heartbeat. The heartbeat path remains unchanged and idempotent
(vl.SetVolumeCapacityFull returns false if already removed, so no
double-decrement).

Recovery still works: if a heartbeat later reports size < limit and
the volume is not oversized, EnsureCorrectWritables adds it back.

- weed/topology/volume_layout.go: RecordAssign returns reachedCapacity
  bool; adds AdjustActiveVolumeCountForFull helper.
- weed/topology/topology.go: PickForWrite invokes the decrement on
  eager full transitions.
- TestPickForWrite: pass a 1024-byte hint instead of 0 so the default
  1MB pendingDelta does not immediately bust the test's 32KB limit.
- New TestRecordAssignReachingCapacityRemovesFromWritable covers the
  eager removal, active count accounting, and no-double-accounting.

* fix(master): recover eagerly-removed volume once decay clears pending

After RecordAssign eagerly removes a volume from writables because
effectiveSize reached the limit, decay can later bring effectiveSize
back under the limit (e.g., when a burst of assigns didn't all result
in uploads). Without recovery the volume would stay non-writable until
vacuum or a ReadOnly flip.

UpdateVolumeSize now re-adds the volume to writables once all of the
following hold:

  * RecordAssign is what removed it (tracked via fullSince timestamp)
  * at least capacityRecoveryDelay has elapsed since the removal (30s)
    — this prevents bouncing during a steady stream of assigns near
    the limit
  * effectiveSize has decayed below the crowded threshold (90% of limit)
  * reportedSize is under the limit (actual disk is not over)
  * standard EnsureCorrectWritables preconditions: enough copies, all
    copies writable, not oversized

The caller (SyncDataNodeRegistration) re-increments activeVolumeCount
symmetrically with the decrement done on eager removal.

* review: release VolumeLayout lock before UpAdjustDiskUsageDelta

adjustActiveVolumeCount held vl.accessLock across the tree-climbing
UpAdjustDiskUsageDelta walk. That walk takes per-level DiskUsages
locks and could be re-entered from other call paths that hold a
node-level lock and then acquire vl.accessLock. Copy the node list
under the VolumeLayout lock and release it before the tree walk to
eliminate the lock-ordering hazard.
2026-04-16 12:50:30 -07:00
Chris Lu dfecd664f9 fix(master): do not re-enter warmup when a fresh cluster grows its first volume (#9092)
* fix(master): do not re-enter warmup when a fresh cluster grows its first volume

Follow-up investigation on #8777. After fixing the writable-chunk cap
deadlock, a second issue surfaced on the same fio reproducer: the
weed mount would log thousands of

  upload data X: filerGrpcAddress assign volume: assign volume failure
    count:1 path:"/test2/X": assign volume: rpc error: code = Canceled
    desc = grpc: the client connection is closing

cascading from the filer's handler. The mount's own cached gRPC
connection to the filer was never invalidated — the "client connection
is closing" text is the FILER's cached connection to the MASTER going
away, and the message is forwarded verbatim in the filer's
AssignVolumeResponse.Error.

Root cause: Topology.IsWarmingUp checks the *live* GetMaxVolumeId. On
a fresh cluster the master is initialized with SetLastLeaderChangeTime
(master_server.go:239 "Seed the warmup timestamp so IsWarmingUp() is
active even if the leader change event hasn't fired yet"), but the
MaxVolumeId==0 guard is supposed to short-circuit IsWarmingUp for
bootstraps so there is no wait. That guard breaks the moment the
first volume is grown inside the warmup window: MaxVolumeId flips
from 0 to 1, the lastLeaderChangeTime is still within 3*pulse (15 s
default), and IsWarmingUp retroactively returns true for the next
several seconds. Every AssignVolume in that window returns
codes.Unavailable, which trips the filer's shouldInvalidateConnection
guard and tears down its cached master connection, which in turn
surfaces as "client connection is closing" to every concurrent
in-flight call from the mount's file-close flush storm. fio reports
EIO on close and the user sees a thousand scary error lines in the
mount log for an otherwise correct run.

Fix: snapshot `hadVolumesAtLeaderChange` inside SetLastLeaderChangeTime
and read that snapshot in IsWarmingUp instead of the live MaxVolumeId.
A fresh cluster snapshots "no volumes at leader change" → IsWarmingUp
stays false through the entire warmup window regardless of how fast
the first grow lands. A real leader transition on a populated cluster
still snapshots "has volumes" → IsWarmingUp behaves exactly as before
until the 3*pulse window closes. The lock ordering in
SetLastLeaderChangeTime reads MaxVolumeId before taking the
lastLeaderChangeTimeLock so the two calls cannot interleave weirdly;
IsWarmingUp reads both fields under a single RLock acquisition.

Verified with the same containerized reproducer used for the deadlock
fix: 4 jobs × 250 nrfiles × 40 MiB × 4k randwrite direct.
  - baseline (master): fio rc=1 (EIO on close), 2809 mount error
    lines matching "filerGrpcAddress assign volume ... client
    connection is closing", 788 filer lines matching "warming up",
    331 "Removing cached gRPC connection to ...19333 due to error:
    master is warming up".
  - patched: fio rc=0 in 1.9 s at 79.2 MiB/s, 0 mount errors,
    0 filer "warming up" lines, 0 cached-master-conn invalidations.
go test ./weed/topology/... passes.

* fix(topology): make NodeImpl.maxVolumeId atomic

CodeRabbit flagged on #9092 that my new IsWarmingUp snapshot
(hadVolumesAtLeaderChange) reads through GetMaxVolumeId(), which until
now returned an unprotected int field on NodeImpl. UpAdjustMaxVolumeId
is called from the volume server heartbeat path in parallel with
GetMaxVolumeId reads on the assign path, and neither side had any
synchronization — a long-standing data race the race detector would
flag if the warmup test suite stressed both sides.

Switch maxVolumeId to atomic.Uint32 (needle.VolumeId is uint32) and
implement UpAdjustMaxVolumeId as a CAS loop so the check-then-set
stays linearizable: two heartbeats racing to promote the field will
land the higher value deterministically and propagate to the parent
exactly once. GetMaxVolumeId is a single atomic load. Callers of both
helpers are unchanged; the struct field comment documents why the
atomic is necessary.

go test -race ./weed/topology/... passes.

* refactor(topology): use WarmupDuration helper in IsWarmingUp

Gemini review on #9092 flagged that IsWarmingUp re-derives the
warmup duration from pulse and WarmupPulseMultiplier instead of
using the existing WarmupDuration() helper, which RemainingWarmupDuration
already uses. Fold the duration calculation through the helper and
short-circuit on lastChange.IsZero() before the time.Since call.
No behavior change.
2026-04-15 14:06:23 -07:00
Chris Lu b37bbf541a feat(master): drain pending size before marking volume readonly (#9036)
* feat(master): drain pending size before marking volume readonly

When vacuum, volume move, or EC encoding marks a volume readonly,
in-flight assigned bytes may still be pending. This adds a drain step:
immediately remove from writable list (stop new assigns), then wait
for pending to decay below 4MB or 30s timeout.

- Add volumeSizeTracking struct consolidating effectiveSize,
  reportedSize, and compactRevision into a single map
- Add GetPendingSize, waitForPendingDrain, DrainAndRemoveFromWritable,
  DrainAndSetVolumeReadOnly to VolumeLayout
- UpdateVolumeSize detects compaction via compactRevision change and
  resets effectiveSize instead of decaying
- Wire drain into vacuum (topology_vacuum.go) and volume mark readonly
  (master_grpc_server_volume.go)

* fix: use 2MB pending size drain threshold

* fix: check crowded state on initial UpdateVolumeSize registration

* fix: respect context cancellation in drain, relax test timing

- DrainAndSetVolumeReadOnly now accepts context.Context and returns
  early on cancellation (for gRPC handler timeout/cancel)
- waitForPendingDrain uses select on ctx.Done instead of time.Sleep
- Increase concurrent heartbeat test timeout from 10s to 15s for CI

* fix: use time-based dedup so decay runs even when reported size is unchanged

The value-based dedup (same reportedSize + compactRevision = skip) prevented
decay from running when pending bytes existed but no writes had landed on
disk yet. The reported size stayed the same across heartbeats, so the excess
never decayed.

Fix: dedup replicas within the same heartbeat cycle using a 2-second time
window instead of comparing values. This allows decay to run once per
heartbeat cycle even when the reported size is unchanged.

Also confirmed finding 1 (draining re-add race) is a false positive:
- Vacuum: ensureCorrectWritables only runs for ReadOnly-changed volumes
- Move/EC: readonlyVolumes flag prevents re-adding during drain

* fix: make VolumeMarkReadonly non-blocking to fix EC integration test timeout

The DrainAndSetVolumeReadOnly call in VolumeMarkReadonly gRPC blocked up
to 30s waiting for pending bytes to decay. In integration tests (and
real clusters during EC encoding), this caused timeouts because multiple
volumes are marked readonly sequentially and heartbeats may not arrive
fast enough to decay pending within the drain window.

Fix: VolumeMarkReadonly now calls SetVolumeReadOnly immediately (stops
new assigns) and only logs a warning if pending bytes remain. The drain
wait is kept only for vacuum (DrainAndRemoveFromWritable) which runs
inside the master's own goroutine pool.

Remove DrainAndSetVolumeReadOnly as it's no longer used.

* fix: relax test timing, rename test, add post-condition assert

* test: add vacuum integration tests with CI workflow

Full-cluster integration test for vacuum, modeled on the EC integration
tests. Starts a real master + 2 volume servers, uploads data, deletes
entries to create garbage, runs volume.vacuum via shell command, and
verifies garbage cleanup and data integrity.

Test flow:
1. Start cluster (master + 2 volume servers)
2. Upload 10 files to create volume with data
3. Delete 5 files to create ~50% garbage
4. Verify garbage ratio > 10%
5. Run volume.vacuum command
6. Verify garbage cleaned up
7. Verify remaining 5 files are still accessible

CI workflow runs on push/PR to master with 15-minute timeout.
Log collection on failure via artifact upload.

* fix: use 500KB files and delete 75% to exceed vacuum garbage threshold

* fix: add shell lock before vacuum command, fix compilation error

* fix: strengthen vacuum integration test assertions

- waitForServer: use net.DialTimeout instead of grpc.NewClient for
  real TCP readiness check
- verify_garbage_before_vacuum: t.Fatal instead of warning when no
  garbage detected
- verify_cleanup_after_vacuum: t.Fatal if no server reported the
  volume or cleanup wasn't verified
- verify_remaining_data: read actual file contents via HTTP and
  compare byte-for-byte against original uploaded payloads

* fix: use http.Client with timeout and close body before retry
2026-04-11 18:29:11 -07:00
Chris Lu 10b0bdce02 feat: pass expected_data_size from clients for size-aware assignment (#9032)
* feat: pass expected_data_size from clients for size-aware assignment

Add expected_data_size field to AssignRequest (master proto) and
AssignVolumeRequest (filer proto) so clients can hint how large the
data will be. The master uses this instead of the 1MB default when
tracking pending volume sizes for weighted assignment.

- Add expected_data_size to master.proto AssignRequest
- Add expected_data_size to filer.proto AssignVolumeRequest
- Wire through filer AssignVolume handler
- Wire through HTTP submit handler (uses actual upload size)
- Add ExpectedDataSize to VolumeAssignRequest in operation package
- Topology.PickForWrite accepts optional expectedDataSize parameter

* fix: guard integer conversions in expected_data_size path

- common.go: clamp OriginalDataSize to non-negative before uint64 cast
- topology.go: cap expectedDataSize at math.MaxInt64 before int64 cast

* fix: parse dataSize hint in HTTP /dir/assign and test non-zero expectedDataSize

- HTTP /dir/assign now parses optional "dataSize" query parameter
  and passes it to PickForWrite instead of hardcoded 0
- Add test assertion for PickForWrite with non-zero expectedDataSize
2026-04-11 11:30:47 -07:00
Chris LuandCopilot Autofix powered by AI e2c79af6ec feat(master): size-aware volume assignment with weighted selection (#9031)
* feat(master): size-aware volume assignment with weighted selection

PickForWrite now selects volumes proportional to remaining capacity
instead of uniform random, so emptier volumes receive more writes.

- Add vid2size map to VolumeLayout tracking effective volume sizes
- Weighted pick via random sampling (k=3) for O(1) cost
- RecordAssign tracks estimated pending bytes between heartbeats
- Exponential decay on heartbeat: halve excess each cycle
- Proactive crowded detection using effective size
- Zero extra heap allocations on the unconstrained hot path

Benchmark (20 writable volumes, unconstrained):
  Before: 36 ns/op, 32 B/op, 2 allocs/op
  After:  85 ns/op, 32 B/op, 2 allocs/op

* fix: address review feedback on size-aware assignment

- RecordAssign: use write lock (Lock) instead of read lock (RLock)
  since it mutates vid2size map and crowded set
- RegisterVolume: clear crowded flag when heartbeat decay drops
  effective size below the threshold
- pickWeightedByRemaining: fix misleading Fisher-Yates comment,
  simplify to plain random sampling (duplicates are harmless)
- ShouldGrowVolumesByDcAndRack: read vid2size under RLock

* fix: decay once per heartbeat cycle, not per replica

RegisterVolume is called once per replica of a volume. For replicated
volumes, the pending size decay was running multiple times per heartbeat
cycle, reducing the excess by 75% instead of 50% (for 2 replicas).

Fix: track vid2reportedSize and only run decay when the heartbeat-
reported size actually changes. A second replica reporting the same
size in the same cycle is a no-op.

Also fix CodeQL alert: cap count*EstimatedNeedleSizeBytes to avoid
uint64→int64 overflow in RecordAssign call.

* Potential fix for pull request finding 'CodeQL / Incorrect conversion between integer types'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fix: fail fast in test setup on JSON errors

- setupWithLimit now takes testing.TB and calls t.Fatalf on unmarshal
  errors or type assertion failures instead of printing and continuing
- benchSetup removed; benchmarks reuse setupWithLimit directly

* fix: run size decay on every heartbeat, not just new volumes

RegisterVolume is only called for newly discovered volumes, not on
every heartbeat. The pending size decay was never running in production.

- Extract decay logic into UpdateVolumeSize(), called from
  SyncDataNodeRegistration for every reported volume on every heartbeat
- RegisterVolume only initializes vid2size for brand-new volumes
- Constrained PickForWrite: scan from random offset, collect up to
  pickSampleSize matches in a stack array (no append allocation)
- Tests now exercise UpdateVolumeSize directly instead of RegisterVolume
  to match the production heartbeat path

* fix: compute pending bytes in uint64 to satisfy CodeQL

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2026-04-11 09:19:05 -07:00
8cde3d4486 Add data file compaction to iceberg maintenance (Phase 2) (#8503)
* Add iceberg_maintenance plugin worker handler (Phase 1)

Implement automated Iceberg table maintenance as a new plugin worker job
type. The handler scans S3 table buckets for tables needing maintenance
and executes operations in the correct Iceberg order: expire snapshots,
remove orphan files, and rewrite manifests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add data file compaction to iceberg maintenance handler (Phase 2)

Implement bin-packing compaction for small Parquet data files:
- Enumerate data files from manifests, group by partition
- Merge small files using parquet-go (read rows, write merged output)
- Create new manifest with ADDED/DELETED/EXISTING entries
- Commit new snapshot with compaction metadata

Add 'compact' operation to maintenance order (runs before expire_snapshots),
configurable via target_file_size_bytes and min_input_files thresholds.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix memory exhaustion in mergeParquetFiles by processing files sequentially

Previously all source Parquet files were loaded into memory simultaneously,
risking OOM when a compaction bin contained many small files. Now each file
is loaded, its rows are streamed into the output writer, and its data is
released before the next file is loaded — keeping peak memory proportional
to one input file plus the output buffer.

* Validate bucket/namespace/table names against path traversal

Reject names containing '..', '/', or '\' in Execute to prevent
directory traversal via crafted job parameters.

* Add filer address failover in iceberg maintenance handler

Try each filer address from cluster context in order instead of only
using the first one. This improves resilience when the primary filer
is temporarily unreachable.

* Add separate MinManifestsToRewrite config for manifest rewrite threshold

The rewrite_manifests operation was reusing MinInputFiles (meant for
compaction bin file counts) as its manifest count threshold. Add a
dedicated MinManifestsToRewrite field with its own config UI section
and default value (5) so the two thresholds can be tuned independently.

* Fix risky mtime fallback in orphan removal that could delete new files

When entry.Attributes is nil, mtime defaulted to Unix epoch (1970),
which would always be older than the safety threshold, causing the
file to be treated as eligible for deletion. Skip entries with nil
Attributes instead, matching the safer logic in operations.go.

* Fix undefined function references in iceberg_maintenance_handler.go

Use the exported function names (ShouldSkipDetectionByInterval,
BuildDetectorActivity, BuildExecutorActivity) matching their
definitions in vacuum_handler.go.

* Remove duplicated iceberg maintenance handler in favor of iceberg/ subpackage

The IcebergMaintenanceHandler and its compaction code in the parent
pluginworker package duplicated the logic already present in the
iceberg/ subpackage (which self-registers via init()). The old code
lacked stale-plan guards, proper path normalization, CAS-based xattr
updates, and error-returning parseOperations.

Since the registry pattern (default "all") makes the old handler
unreachable, remove it entirely. All functionality is provided by
iceberg.Handler with the reviewed improvements.

* Fix MinManifestsToRewrite clamping to match UI minimum of 2

The clamp reset values below 2 to the default of 5, contradicting the
UI's advertised MinValue of 2. Clamp to 2 instead.

* Sort entries by size descending in splitOversizedBin for better packing

Entries were processed in insertion order which is non-deterministic
from map iteration. Sorting largest-first before the splitting loop
improves bin packing efficiency by filling bins more evenly.

* Add context cancellation check to drainReader loop

The row-streaming loop in drainReader did not check ctx between
iterations, making long compaction merges uncancellable. Check
ctx.Done() at the top of each iteration.

* Fix splitOversizedBin to always respect targetSize limit

The minFiles check in the split condition allowed bins to grow past
targetSize when they had fewer than minFiles entries, defeating the
OOM protection. Now bins always split at targetSize, and a trailing
runt with fewer than minFiles entries is merged into the previous bin.

* Add integration tests for iceberg table maintenance plugin worker

Tests start a real weed mini cluster, create S3 buckets and Iceberg
table metadata via filer gRPC, then exercise the iceberg.Handler
operations (ExpireSnapshots, RemoveOrphans, RewriteManifests) against
the live filer. A full maintenance cycle test runs all operations in
sequence and verifies metadata consistency.

Also adds exported method wrappers (testing_api.go) so the integration
test package can call the unexported handler methods.

* Fix splitOversizedBin dropping files and add source path to drainReader errors

The runt-merge step could leave leading bins with fewer than minFiles
entries (e.g. [80,80,10,10] with targetSize=100, minFiles=2 would drop
the first 80-byte file). Replace the filter-based approach with an
iterative merge that folds any sub-minFiles bin into its smallest
neighbor, preserving all eligible files.

Also add the source file path to drainReader error messages so callers
can identify which Parquet file caused a read/write failure.

* Harden integration test error handling

- s3put: fail immediately on HTTP 4xx/5xx instead of logging and
  continuing
- lookupEntry: distinguish NotFound (return nil) from unexpected RPC
  errors (fail the test)
- writeOrphan and orphan creation in FullMaintenanceCycle: check
  CreateEntryResponse.Error in addition to the RPC error

* go fmt

---------

Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 11:27:42 -07:00
Chris Lu baae672b6f feat: auto-disable master vacuum when plugin worker is active (#8624)
* feat: auto-disable master vacuum when plugin vacuum worker is active

When a vacuum-capable plugin worker connects to the admin server, the
admin server calls DisableVacuum on the master to prevent the automatic
scheduled vacuum from conflicting with the plugin worker's vacuum. When
the worker disconnects, EnableVacuum is called to restore the default
behavior. A safety net in the topology refresh loop re-enables vacuum
if the admin server disconnects without cleanup.

* rename isAdminServerConnected to isAdminServerConnectedFunc

* add 5s timeout to DisableVacuum/EnableVacuum gRPC calls

Prevents the monitor goroutine from blocking indefinitely if the
master is unresponsive.

* track plugin ownership of vacuum disable to avoid overriding operator

- Add vacuumDisabledByPlugin flag to Topology, set when DisableVacuum
  is called while admin server is connected (i.e., by plugin monitor)
- Safety net only re-enables vacuum when it was disabled by plugin,
  not when an operator intentionally disabled it via shell command
- EnableVacuum clears the plugin flag

* extract syncVacuumState for testability, add fake toggler tests

Extract the single sync step into syncVacuumState() with a
vacuumToggler interface. Add TestSyncVacuumState with a fake
toggler that verifies disable/enable calls on state transitions.

* use atomic.Bool for isDisableVacuum and vacuumDisabledByPlugin

Both fields are written by gRPC handlers and read by the vacuum
goroutine, causing a data race. Use atomic.Bool with Store/Load
for thread-safe access.

* use explicit by_plugin field instead of connection heuristic

Add by_plugin bool to DisableVacuumRequest proto so the caller
declares intent explicitly. The admin server monitor sets it to
true; shell commands leave it false. This prevents an operator's
intentional disable from being auto-reversed by the safety net.

* use setter for admin server callback instead of function parameter

Move isAdminServerConnected from StartRefreshWritableVolumes
parameter to Topology.SetAdminServerConnectedFunc() setter.
Keeps the function signature stable and decouples the topology
layer from the admin server concept.

* suppress repeated log messages on persistent sync failures

Add retrying parameter to syncVacuumState so the initial
state transition is logged at V(0) but subsequent retries
of the same transition are silent until the call succeeds.

* clear plugin ownership flag on manual DisableVacuum

Prevents stale plugin flag from causing incorrect auto-enable
when an operator manually disables vacuum after a plugin had
previously disabled it.

* add by_plugin to EnableVacuumRequest for symmetric ownership tracking

Plugin-driven EnableVacuum now only re-enables if the plugin was
the one that disabled it. If an operator manually disabled vacuum
after the plugin, the plugin's EnableVacuum is a no-op. This
prevents the plugin monitor from overriding operator intent on
worker disconnect.

* use cancellable context for monitorVacuumWorker goroutine

Replace context.Background() with a cancellable context stored
as bgCancel on AdminServer. Shutdown() calls bgCancel() so
monitorVacuumWorker exits cleanly via ctx.Done().

* track operator and plugin vacuum disables independently

Replace single isDisableVacuum flag with two independent flags:
vacuumDisabledByOperator and vacuumDisabledByPlugin. Each caller
only flips its own flag. The effective disabled state is the OR
of both. This prevents a plugin connect/disconnect cycle from
overriding an operator's manual disable, and vice versa.

* fix safety net to clear plugin flag, not operator flag

The safety net should call EnableVacuumByPlugin() to clear only
the plugin disable flag when the admin server disconnects. The
previous call to EnableVacuum() incorrectly cleared the operator
flag instead.
2026-03-13 22:49:12 -07:00
Chris Lu 2ec0a67ee3 master: return 503/Unavailable during topology warmup after leader change (#8529)
* master: return 503/Unavailable during topology warmup after leader change

After a master restart or leader change, the topology is empty until
volume servers reconnect and send heartbeats. During this warmup window
(3 heartbeat intervals = 15 seconds), volume lookups that fail now
return 503 Service Unavailable (HTTP) or gRPC Unavailable instead of
404 Not Found, signaling clients to retry with other masters.

* master: skip warmup 503 on fresh start and single-master setups

- Check MaxVolumeId > 0 to distinguish restart from fresh start
  (MaxVolumeId is Raft-persisted, so 0 means no prior data)
- Check peer count > 1 so single-master deployments aren't affected
  (no point suggesting "retry with other masters" if there are none)

* master: address review feedback and block assigns during warmup

- Protect LastLeaderChangeTime with dedicated mutex (fix data race)
- Extract warmup multiplier as WarmupPulseMultiplier constant
- Derive Retry-After header from pulse config instead of hardcoding
- Only trigger warmup 503 for "not found" errors, not parse errors
- Return nil response (not partial) on gRPC Unavailable
- Add doc comments to IsWarmingUp, getter/setter, WarmupDuration
- Block volume assign requests (HTTP and gRPC) during warmup,
  since the topology is incomplete and assignments would be unreliable
- Skip warmup behavior for single-master setups (no peers to retry)

* master: apply warmup to all setups, skip only on fresh start

Single-master restarts still have an empty topology until heartbeats
arrive, so warmup protection should apply there too. The only case
to skip is a fresh cluster start (MaxVolumeId == 0), which already
has no volumes to look up.

- Remove GetMasterCount() > 1 guard from all warmup checks
- Remove now-unused GetMasterCount helper
- Update error messages to "topology is still loading" (not
  "retry with other masters" which doesn't apply to single-master)

* master: add client-side retry on Unavailable for lookup and assign

The server-side 503/Unavailable during warmup needs client cooperation.
Previously, LookupVolumeIds and Assign would immediately propagate the
error without retry.

Now both paths retry with exponential backoff (1s -> 1.5s -> ... up to
6s) when receiving Unavailable, respecting context cancellation. This
covers the warmup window where the master's topology is still loading
after a restart or leader change.

* master: seed warmup timestamp in legacy raft path at setup

The legacy raft path only set lastLeaderChangeTime inside the event
listener callback, which could fire after IsLeader() was already
observed as true in SetRaftServer. Seed the timestamp at setup time
(matching the hashicorp path) so IsWarmingUp() is active immediately.

* master: fix assign retry loop to cover full warmup window

The retry loop used waitTime <= maxWaitTime as a stop condition,
causing it to give up after ~13s while warmup lasts 15s. Now cap
each individual sleep at maxWaitTime but keep retrying until the
context is cancelled.

* master: preserve gRPC status in lookup retry and fix retry window

Return the raw gRPC error instead of wrapping with fmt.Errorf so
status.FromError() can extract the status code. Use proper gRPC
status check (codes.Unavailable) instead of string matching. Also
cap individual sleep at maxWaitTime while retrying until ctx is done.

* master: use gRPC status code instead of string matching in assign retry

Use status.FromError/codes.Unavailable instead of brittle
strings.Contains for detecting retriable gRPC errors in the
assign retry loop.

* master: use remaining warmup duration for Retry-After header

Set Retry-After to the remaining warmup time instead of the full
warmup duration, so clients don't wait longer than necessary.

* master: reset ret.Replicas before populating from assign response

Clear Replicas slice before appending to prevent duplicate entries
when the assign response is retried or when alternative requests
are attempted.

* master: add unit tests for warmup retry behavior

Test that Assign() and LookupVolumeIds() retry on codes.Unavailable
and stop promptly when the context is cancelled.

* master: record leader change time before initialization work

Move SetLastLeaderChangeTime() to fire immediately when the leader
change event is received, before DoBarrier(), EnsureTopologyId(),
and updatePeers(), so the warmup clock starts at the true moment
of leadership transition.

* master: use topology warmup duration in volume growth wait loop

Replace hardcoded constants.VolumePulsePeriod * 2 with
topo.IsWarmingUp() and topo.WarmupDuration() so the growth wait
stays in sync with the configured warmup window. Remove unused
constants import.

* master: resolve master before creating RPC timeout context

Move GetMaster() call before context.WithTimeout() so master
resolution blocking doesn't consume the gRPC call timeout.

* master: use NotFound flag instead of string matching for volume lookup

Add a NotFound field to LookupResult and set it in findVolumeLocation
when a volume is genuinely missing. Update HTTP and gRPC warmup
checks to use this flag instead of strings.Contains on the error
message.

* master: bound assign retry loop to 30s for deadline-free contexts

Without a context deadline, the Unavailable retry loop could spin
forever. Add a maxRetryDuration of 30s so the loop gives up even
when no context deadline is set.

* master: strengthen assign retry cancellation test

Verify the retry loop actually retried (callCount > 1) and that
the returned error is context.DeadlineExceeded, not just any error.

* master: extract shared retry-with-backoff utility

Add util.RetryWithBackoff for context-aware, bounded retry with
exponential backoff. Refactor both Assign() and LookupVolumeIds()
to use it instead of duplicating the retry/sleep/backoff logic.

* master: cap waitTime in RetryWithBackoff to prevent unbounded growth

Cap the backoff waitTime at maxWaitTime so it doesn't grow
indefinitely in long-running retry scenarios.

* master: only return Unavailable during warmup when all lookups failed

For batched LookupVolume requests, return partial results when some
volumes are found. Only return codes.Unavailable when no volumes
were successfully resolved, so clients benefit from partial results
instead of retrying unnecessarily.

* master: set retriable error message in 503 response body

When returning 503 during warmup, replace the "not found" error
in the JSON body with "service warming up, please retry" so
clients don't treat it as a permanent error.

* master: guard empty master address in LookupVolumeIds

If GetMaster() returns empty (no master found or ctx cancelled),
return an appropriate error instead of dialing an empty address.
Returns ctx.Err() if context is done, otherwise codes.Unavailable
to trigger retry.

* master: add comprehensive tests for RetryWithBackoff

Test success after retries, non-retryable error handling, context
cancellation, and maxDuration cap with context.Background().

* master: enforce hard maxDuration bound in RetryWithBackoff

Use a deadline instead of elapsed-time check so the last sleep is
capped to remaining time. This prevents the total retry duration
from overshooting maxDuration by up to one full backoff interval.

* master: respect fresh-start bypass in RemainingWarmupDuration

Check IsWarmingUp() first (which returns false when MaxVolumeId==0)
so RemainingWarmupDuration returns 0 on fresh clusters.

* master: round up Retry-After seconds to avoid underestimating

Use math.Ceil so fractional remaining seconds (e.g. 1.9s) round
up to the next integer (2) instead of flooring down (1).

* master: tighten batch lookup warmup to all-NotFound only

Only return codes.Unavailable when every requested volume ID was
a transient not-found. Mixed cases with non-NotFound errors now
return the response with per-volume error details preserved.

* master: reduce retry log noise and fix timer leak

Lower per-attempt retry log from V(0) to V(1) to reduce noise
during warmup. Replace time.After with time.NewTimer to avoid
lingering timers when context is cancelled.

* master: add per-attempt timeout for assign RPC

Use a 10s per-attempt timeout so a single slow RPC can't consume
the entire 30s retry budget when ctx has no deadline.

* master: share single 30s retry deadline across assign request entries

The Assign() function iterates over primary and fallback requests,
previously giving each its own 30s RetryWithBackoff budget. With a
primary + fallback, the total could reach 60s. Compute one deadline
up front and pass the remaining budget to each RetryWithBackoff call
so the entire Assign() call stays within a single 30s cap.

* master: strengthen context-cancel test with DeadlineExceeded and retry assertions

Assert errors.Is(err, context.DeadlineExceeded) to verify the error
is specifically from the context deadline, and check callCount > 1
to prove retries actually occurred before cancellation. Mirrors the
pattern used in TestAssignStopsOnContextCancel.

* master: bound GetMaster with per-attempt timeout in LookupVolumeIds

GetMaster() calls WaitUntilConnected() which can block indefinitely
if no master is available. Previously it used the outer ctx, so a
slow master resolution could consume the entire RetryWithBackoff
budget in a single attempt. Move the per-attempt timeoutCtx creation
before the GetMaster call so both master resolution and the gRPC
LookupVolume RPC share one grpcTimeout-bounded attempt.

* master: use deadline-aware context for assign retry budget

The shared 30s deadline only limited RetryWithBackoff's internal
wall-clock tracking, but per-attempt contexts were still derived
from the original ctx and could run for up to 10s even when the
budget was nearly exhausted. Create a deadlineCtx from the computed
deadline and derive both RetryWithBackoff and per-attempt timeouts
from it so all operations honor the shared 30s cap.

* master: skip warmup gate for empty lookup requests

When VolumeOrFileIds is empty, notFoundCount == len(req.VolumeOrFileIds)
is 0 == 0 which is true, causing empty lookup batches during warmup to
return codes.Unavailable and be retried endlessly. Add a
len(req.VolumeOrFileIds) > 0 guard so empty requests pass through.

* master: validate request fields before warmup gate in Assign

Move Replication and Ttl parsing before the IsWarmingUp() check so
invalid inputs get a proper validation error instead of being masked
by codes.Unavailable during warmup. Pure syntactic validation does
not depend on topology state and should run first.

* master: check deadline and context before starting retry attempt

RetryWithBackoff only checked the deadline and context after an
attempt completed or during the sleep select. If the deadline
expired or context was canceled during sleep, the next iteration
would still call operation() before detecting it. Add pre-operation
checks so no new attempt starts after the budget is exhausted.

* master: always return ctx.Err() on context cancellation in RetryWithBackoff

When ctx.Err() is non-nil, the pre-operation check was returning
lastErr instead of ctx.Err(). This broke callers checking
errors.Is(err, context.DeadlineExceeded) and contradicted the
documented contract. Always return ctx.Err() so the cancellation
reason is properly surfaced.

* master: handle warmup errors in StreamAssign without killing the stream

StreamAssign was returning codes.Unavailable errors from Assign
directly, which terminates the gRPC stream and breaks pooled
connections. Instead, return transient errors as in-band error
responses so the stream survives warmup periods.

Also reset assignClient in doAssign on Send/Recv failures so a
broken stream doesn't leave the proxy permanently dead.

* master: wait for warmup before slot search in findAndGrow

findEmptySlotsForOneVolume was called before the warmup wait loop,
selecting slots from an incomplete topology. Move the warmup wait
before slot search so volume placement uses the fully warmed-up
topology with all servers registered.

* master: add Retry-After header to /dir/assign warmup response

The /dir/lookup handler already sets Retry-After during warmup but
/dir/assign did not, leaving HTTP clients without guidance on when
to retry. Add the same header using RemainingWarmupDuration().

* master: only seed warmup timestamp on leader at startup

SetLastLeaderChangeTime was called unconditionally for both leader
and follower nodes. Followers don't need warmup state, and the
leader change event listener handles real elections. Move the seed
into the IsLeader() block so only the startup leader gets warmup
initialized.

* master: preserve codes.Unavailable for StreamAssign warmup errors in doAssign

StreamAssign returns transient warmup errors as in-band
AssignResponse.Error messages. doAssign was converting these to plain
fmt.Errorf, losing the codes.Unavailable classification needed for
the caller's retry logic. Detect warmup error messages and wrap them
as status.Error(codes.Unavailable) so RetryWithBackoff can retry.
2026-03-08 16:05:45 -07:00
Chris Lu b08bb8237c Fix master leader election startup issue (#8340)
* Fix master leader election startup issue

Fixes #error-log-leader-not-selected-yet

* Fix master leader election startup issue

This change improves server address comparison using the 'Equals' method and handles recursion in topology leader lookup, resolving the 'leader not selected yet' error during master startup.

* Merge user improvements: use MaybeLeader for non-blocking checks

* not useful test

* Address code review: optimize Equals, fix deadlock in IsLeader, safe access in Leader
2026-02-13 15:39:39 -08:00
Chris Lu 753e1db096 Prevent split-brain: Persistent ClusterID and Join Validation (#8022)
* Prevent split-brain: Persistent ClusterID and Join Validation

- Persist ClusterId in Raft store to survive restarts.
- Validate ClusterId on Raft command application (piggybacked on MaxVolumeId).
- Prevent masters with conflicting ClusterIds from joining/operating together.
- Update Telemetry to report the persistent ClusterId.

* Refine ClusterID validation based on feedback

- Improved error message in cluster_commands.go.
- Added ClusterId mismatch check in RaftServer.Recovery.

* Handle Raft errors and support Hashicorp Raft for ClusterId

- Check for errors when persisting ClusterId in legacy Raft.
- Implement ClusterId generation and persistence for Hashicorp Raft leader changes.
- Ensure consistent error logging.

* Refactor ClusterId validation

- Centralize ClusterId mismatch check in Topology.SetClusterId.
- Simplify MaxVolumeIdCommand.Apply and RaftServer.Recovery to rely on SetClusterId.

* Fix goroutine leak and add timeout

- Handle channel closure in Hashicorp Raft leader listener.
- Add timeout to Raft Apply call to prevent blocking.

* Fix deadlock in legacy Raft listener

- Wrap ClusterId generation/persistence in a goroutine to avoid blocking the Raft event loop (deadlock).

* Rename ClusterId to SystemId

- Renamed ClusterId to SystemId across the codebase (protobuf, topology, server, telemetry).
- Regenerated telemetry.pb.go with new field.

* Rename SystemId to TopologyId

- Rename to SystemId was intermediate step.
- Final name is TopologyId for the persistent cluster identifier.
- Updated protobuf, topology, raft server, master server, and telemetry.

* Optimize Hashicorp Raft listener

- Integrated TopologyId generation into existing monitorLeaderLoop.
- Removed extra goroutine in master_server.go.

* Fix optimistic TopologyId update

- Removed premature local state update of TopologyId in master_server.go and raft_hashicorp.go.
- State is now solely updated via the Raft state machine Apply/Restore methods after consensus.

* Add explicit log for recovered TopologyId

- Added glog.V(0) info log in RaftServer.Recovery to print the recovered TopologyId on startup.

* Add Raft barrier to prevent TopologyId race condition

- Implement ensureTopologyId helper method
- Send no-op MaxVolumeIdCommand to sync Raft log before checking TopologyId
- Ensures persisted TopologyId is recovered before generating new one
- Prevents race where generation happens during log replay

* Serialize TopologyId generation with mutex

- Add topologyIdGenLock mutex to MasterServer struct
- Wrap ensureTopologyId method with lock to prevent concurrent generation
- Fixes race where event listener and manual leadership check both generate IDs
- Second caller waits for first to complete and sees the generated ID

* Add TopologyId recovery logging to Apply method

- Change log level from V(1) to V(0) for visibility
- Log 'Recovered TopologyId' when applying from Raft log
- Ensures recovery is visible whether from snapshot or log replay
- Matches Recovery() method logging for consistency

* Fix Raft barrier timing issue

- Add 100ms delay after barrier command to ensure log application completes
- Add debug logging to track barrier execution and TopologyId state
- Return early if barrier command fails
- Prevents TopologyId generation before old logs are fully applied

* ensure leader

* address comments

* address comments

* redundant

* clean up

* double check

* refactoring

* comment
2026-01-18 14:02:34 -08:00
Chris Luandgemini-code-assist[bot] 7acebf11ea Master: volume assignment concurrency (#7159)
* volume assginment concurrency

* accurate tests

* ensure uniqness

* reserve atomically

* address comments

* atomic

* ReserveOneVolumeForReservation

* duplicated

* Update weed/topology/node.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update weed/topology/node.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* atomic counter

* dedup

* select the appropriate functions based on the useReservations flag

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-08-23 21:02:30 -07:00
Lisandro Pin cea34dc21a Fix implementation of master_pb.CollectionList RPC call (#6715) 2025-04-16 14:28:58 -07:00
Konstantin Lebedev e2e97db917 [master] avoid timeout when assigning for main request with filter by DC or rack (#6291)
* avoid timeout when assigning for main request with filter by DC or rack

https://github.com/seaweedfs/seaweedfs/issues/6290

* use constant NoWritableVolumes
2024-11-26 08:33:31 -08:00
Konstantin Lebedev 8836fa19b6 use ShouldGrowVolumesByDcAndRack (#6280) 2024-11-25 09:30:37 -08:00