7 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandDevin	3fd8d0cf09	master: let location-constrained assigns wait for pending growth (#12167) ... * topology: add AvailableSpaceWithinLocation for location-scoped free slots AvailableSpaceFor answers for the whole topology; a location-constrained assign needs to know whether the requested data center, rack, or data node still has room before deciding growth can satisfy it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: let location-constrained assigns wait for pending growth An assign carrying a data center or rack constraint failed on the first unsatisfied pick, even when the same request had just queued volume growth for that location — so the first write into a fresh collection errored out instead of picking up the volume growth landed a moment later. Keep the fail-fast path only when growth cannot help: growth disabled, no growth pending, or no free slots left inside the constraint. Otherwise the request takes the same wait/shed path as an unconstrained assign. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: sum leaf free slots in AvailableSpaceWithinLocation Rolled-up rack and data center counters keep a maintenance-mode server's capacity, and one overcommitted node can cancel a sibling's slots. Sum the matching data nodes directly so the location check agrees with what growth can actually use. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: let constrained growth initiators reach the shed path When a constrained assign's own growth concluded unfulfilled, the fast-fail check saw only that no grow was pending and returned the plain pick error — a code the assign client does not retry, so the first write died instead of re-triggering growth. Exclude the request's own initiated grow from the no-pending-growth condition so it reaches the existing re-pick and ResourceExhausted shed like an unconstrained assign. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: judge targeted grow requests by the location they name A grow request carrying a data center, rack, or data node constraint was discarded whenever the layout's writables elsewhere were healthy, so a location with free slots but no writable could never receive the volume its assign was waiting for. Add ShouldGrowVolumesWithinLocation and use it for constrained requests; the layout-wide check still governs untargeted ones. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: only positive leaf slots count in AvailableSpaceWithinLocation An over-committed node reports negative free slots; summed plainly it\ncancelled a sibling's room and a constrained assign could fail fast\nwhile the location still had space to grow into. * master: fail constrained assigns fast when growth cannot place A constrained assign with free slots in its location still waited out\ngrowth that could never satisfy the replica placement — a single-rack\ndata center asked for 010 sheds the same doomed grow on every client\nretry. Run the grow-time node search as a dry run and return its\nplacement error instead of a retryable shed. * master: judge constrained growth by layout, not free slots A slot-level feasibility probe races with the grow it judges: a\npending grow consuming the last slot in one rack reports the\nplacement impossible while it mints the request's own volume, and\nthe hard error skips the re-pick that would find it. GrowthBlocker\nnow counts racks and data nodes inside the named location — shape a\ngrow cannot consume — so only genuinely unplaceable layouts fail\nfast. * topology: GrowthBlocker mirrors the grow selector's feasibility The location filter bound every replica tier, so a 010 request naming one rack needed two racks inside it while the selector constrains only the first placement and counts the rest by room. And counting nodes without free slots let a full location queue a grow it could never serve, so the assign shed ResourceExhausted at timeout instead of failing fast. GrowthBlocker now applies the option's location to the primary placement chain only and leaf-sums free slots the way the selector does, so it reports a blocker exactly when the grow can never place. * master: repick once before surfacing a growth blocker An in-flight grow can spend the last free slot minting the request's own volume: the blocker's slot counts predate that registration by a heartbeat, so returning the placement error on the first failed pick could reject a write whose volume lands a moment later. A pending grow now earns one repick grace before the blocker error is returned. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 08:10:45 -07:00
Chris LuandDevin	3e5aa8c3b3	topology: mark replica quarantined before recording the pair (#11983) ... * topology: serialize quarantine records with their layout marks A quarantine publishes a (node, vid) record and marks the replica read-only at the layout level; release and heartbeat flag updates must observe that pair as one state. Store-then-mark let a concurrent release delete the record before the mark landed, stranding the surviving replica read-only; mark-then-store let a heartbeat clear the mark before the record was visible, returning a conflicting replica to the writable list. quarantineLock now covers every record<->mark pair: flagDuplicateVolumeId's quarantine, releaseResolvedQuarantine's undo, the isQuarantined-gated marks on registration, and the quarantine-folded read-only flag writes on heartbeat paths. Remote server pushes stay outside the lock; a push that lands after the record was released is undone, and a failed release push re-adds the record for a later retry. The release test now waits for in-flight async flags to settle instead of asserting on a single read. * topology: fold quarantine into RegisterVolume's flag rebuild RegisterVolume rewrites every replica's location read-only flag from what its node last reported, and reported flags deliberately exclude quarantine. Any registration of a shared vid -- a reconnected replica, a new copy, the self-heal and delta-heartbeat re-register paths -- therefore cleared the quarantine mark on every replica in the location, while the gate afterwards only re-marked the registering node. Two call sites had no gate at all. RegisterVolume now takes an isQuarantined predicate and evaluates it under accessLock for every replica it touches, so the mark is never dropped in the first place: a release unmarks after the rebuild, and a quarantine mark re-applies on the next rebuild, with no window between. * topology: keep quarantine when a vacuum commit re-adds a replica SetVolumeAvailable re-adds the replica into the location list; a re-added slot starts with a cleared read-only flag, so a quarantined replica came back flag-free and the next ensureCorrectWritables could return a conflicting volume to the writable list. Pass the quarantine state in and set the flag back under the same accessLock, with the caller holding quarantineLock so release cannot interleave between the check and the mark. * topology: merge quarantine records instead of overwriting them enforceQuarantine wrote back the state copy it loaded before its remote push, so a concurrent writer's enforced=true was lost: the server stayed read-only while the record said unenforced, and a later release skipped the writable push it owed. The write-back now reloads the record under quarantineLock and merges -- enforced wins from either side, reported is taken from the newer pre-enforcement observation only while no push has landed yet. All three compensating writable pushes had the same hole in reverse: a failed un-quarantine left the server read-only with no record to retry from, and release's restore blindly overwrote any record a fresh flag pass had created in between. They now go through restoreQuarantineRecord, which adds the record only when absent and otherwise sets enforced on the current one, preserving its reported flag. * topology: hold quarantineLock over registration's quarantine check-and-mark RegisterVolume reads the quarantine record and marks the replica read-only under accessLock, but the pair ran without quarantineLock. A release could push writable, watch the record get re-read, and then delete it: the replica stayed read-only with no record left to clear, and unchanged writable heartbeats skip SetReplicaReadOnlyFlag so the mark never repaired. Holding quarantineLock around the check-and-mark in all three production callers serializes it against the release clear-and-delete. * topology: serialize remote quarantine pushes behind a record recheck releaseResolvedQuarantine deletes the record under quarantineLock and sends the writable request after unlock. A conflict flagged while that request is in flight creates a fresh record and marks the replica read-only again, and the older writable request can still land last, leaving the server writable under an active quarantine. The reverse flap strands a stale read-only mark the same way. Route every QuarantineVolumeFn call through pushQuarantine, which holds quarantinePushLock across the remote call and re-checks the record under quarantineLock right before sending: a read-only push requires the pair to exist, a writable push requires it to be gone. A skipped push means the decision already moved on, so the final server state always follows the newest record. * topology: serialize quarantine pushes per replica, not globally A single quarantinePushLock held across the remote mark call let one slow volume server (up to the 10s client timeout) delay pushes for unrelated replicas. Keying the lock by (node, vid) keeps pushes to one replica ordered in record-decision order while pushes to others run in parallel. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: restore the layout read-only mark with a released record A failed writable push re-armed the quarantine record but left the replica marked writable in the layout, so PickForWrite could route to a replica the server still holds read-only. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: wait out async duplicate checks in the release-retry test TestFailedReleasePushRestoresQuarantineMark asserts layout marks around a release pass, but the two RegisterVolumeLayout calls each start a background flagDuplicateVolumeId that can still be mid-flight. A late mark could land after the final release and fail the writable assertion even though the code behaved correctly. Track the async passes in a WaitGroup and drain them before the explicit flag call so the rest of the test is deterministic. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: keep quarantine when a replica's layout key is unreadable * topology: keep a failed writable push's restore ordered under the push lock * topology: retry the quarantine release on every duplicate check * ci: isolate Java and S3 test servers on persistent runners * ci: drop the duplicated -ip flags in the lifecycle test server The bind flags were already on the command; the isolation pass added a second pair. * topology: retry a failed quarantine writable push on the next report * topology: retry pending quarantine releases off the leader pulse A failed writable push re-arms the quarantine record with a retry that only enforceQuarantine fires, and that runs only for volumes a heartbeat carries. Once the server has reported the replica read-only it drops out of the delta reports, and restoreQuarantineRecord keeps the stored flag matching the server's so the digests agree and no full list is requested - the pending release then never retries and the idle volume stays read-only forever. Run the same release pass from the leader's pulse loop so a pending retry no longer waits on a report. * topology: exercise the pulse-driven quarantine release retry --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 07:00:45 -07:00
Chris Lu	8f2daad338	topology: mirror the writable volume list in a set (#11076) ... Membership was a linear scan over a slice, and ensureCorrectWritables runs it for every volume on every heartbeat, so the master's steady-state cost per volume server is quadratic in that server's volume count. BenchmarkSyncDataNodeRegistration, median of 3: 1000 volumes 565.7us -> 535.0us -5.4% 100000 volumes 1.665s -> 55.3ms -96.7% Allocations are unchanged at both sizes, so the difference is the scan. Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1	2026-09-01 14:21:26 -07:00
Chris Lu	b77431c142	master: stop hintless small-file assigns from marking volumes full (#10944) ... * master: estimate a hintless assign's size from the volume's average file size An assign that carries no dataSize hint charged a flat 1MB per file id against the volume's effective size. A small-file workload overpays by orders of magnitude: bulk-writing 4KB files marks volumes holding a few hundred MB of real data as crowded and then full, so the master grows unnecessary volumes and, once every volume is spuriously full, fails all assigns. Estimate from the volume's own average file size instead, and keep the 1MB fallback only for volumes with no history. * master: decay pending assign sizes for volumes gone quiet The decay that corrects pending assign estimates runs only when a heartbeat reports the volume, and a heartbeat only reports a volume whose content changed. A volume held out of the writable list takes no writes, so once inflated estimates mark every volume full, nothing is ever reported again, nothing decays, and the cluster refuses all writes until a restart. Run the decay from the master's periodic loop for volumes no heartbeat has reported within two pulses, feeding the last reported size back through the same path an unchanged heartbeat would take. * master: trim the comments on the assign size estimate * master: keep the periodic decay out of the replica-dedup window UpdateVolumeSize ignores a report arriving within two seconds of the last one, so replicas of the same volume do not each halve the pending estimate. The periodic decay went through the same path and stamped that window, so a real heartbeat landing right behind it was dropped along with its reported size and compact revision. Only a volume whose content changed is reported at all, so nothing would send that size again and the master kept a stale one. Let the dedup window belong to volume server reports alone. * master: let the decay read the size record under the lock it mutates The periodic decay picked its volumes under a read lock and replayed them under a write one, carrying the size it had read across the gap. A heartbeat landing in between was rolled back: the replay wrote the older size and compact revision over the fresh ones, and a compaction report lost that way is never resent, since only a volume whose content changed is reported. The decay has no size of its own to contribute, so it now reads the record under the same lock it mutates. * master: let a heartbeat that beat the decay stand for the cycle The decay chooses its volumes under a read lock and applies them under a write one. A heartbeat landing in that gap already did the halving the cycle owed, so applying the decay on top of it halved twice and forgot pending bytes the volume has not written yet - the double-halving the replica-dedup window exists to prevent. Both callers now give way to a report already handled for this cycle; only a real report still advances lastUpdateTime, so a quiet volume keeps decaying every pulse. * master: keep genuinely full volumes out of the decay pass A volume the disk really did fill keeps its fullSince set for good, so it was selected every pulse for a decay that cannot help it: UpdateVolumeSize refuses to recover a volume whose reported size is at the limit, and replaying a size that cannot move leaves the record as it found it. Full and quiet is the ordinary resting state of a cluster, so this was most of the pass, taking the layout write lock away from the heartbeats to do nothing. On a million tracked volumes with a hundredth of them phantom-full it costs ten thousand write locks a pulse instead of a million. * master: put the stale-replay test back on the path it guards Giving the decay the dedup window left this test short-circuiting there, so it no longer reached the locked read it was written for and passed with that read removed. Age the record past the window, which is the only case where reading it under the lock is what saves the report.	2026-08-25 10:16:48 -07:00
Chris Lu	ecc0390795	fix(master): eagerly remove volume from writable when assign hits limit (#9108) ... * fix(master): eagerly remove volume from writable when RecordAssign hits limit Previously, a volume was only removed from the writable list by the heartbeat-driven CollectDeadNodeAndFullVolumes pass, which runs every pulse (5s) after a 5s heartbeat. Under sustained concurrent writes, fio-style workloads observed in the field grew volumes 8-20x past the configured 100MB limit (median 530MB, peak 1.98GB) during that 5-15s detection window. RecordAssign already tracks effective size (reported + pending) on each /dir/assign. It now also removes the volume from writable the moment effectiveSize reaches volumeSizeLimit, and mirrors the activeVolumeCount decrement that Topology.SetVolumeCapacityFull would have done on the next heartbeat. The heartbeat path remains unchanged and idempotent (vl.SetVolumeCapacityFull returns false if already removed, so no double-decrement). Recovery still works: if a heartbeat later reports size < limit and the volume is not oversized, EnsureCorrectWritables adds it back. - weed/topology/volume_layout.go: RecordAssign returns reachedCapacity bool; adds AdjustActiveVolumeCountForFull helper. - weed/topology/topology.go: PickForWrite invokes the decrement on eager full transitions. - TestPickForWrite: pass a 1024-byte hint instead of 0 so the default 1MB pendingDelta does not immediately bust the test's 32KB limit. - New TestRecordAssignReachingCapacityRemovesFromWritable covers the eager removal, active count accounting, and no-double-accounting. * fix(master): recover eagerly-removed volume once decay clears pending After RecordAssign eagerly removes a volume from writables because effectiveSize reached the limit, decay can later bring effectiveSize back under the limit (e.g., when a burst of assigns didn't all result in uploads). Without recovery the volume would stay non-writable until vacuum or a ReadOnly flip. UpdateVolumeSize now re-adds the volume to writables once all of the following hold: * RecordAssign is what removed it (tracked via fullSince timestamp) * at least capacityRecoveryDelay has elapsed since the removal (30s) — this prevents bouncing during a steady stream of assigns near the limit * effectiveSize has decayed below the crowded threshold (90% of limit) * reportedSize is under the limit (actual disk is not over) * standard EnsureCorrectWritables preconditions: enough copies, all copies writable, not oversized The caller (SyncDataNodeRegistration) re-increments activeVolumeCount symmetrically with the decrement done on eager removal. * review: release VolumeLayout lock before UpAdjustDiskUsageDelta adjustActiveVolumeCount held vl.accessLock across the tree-climbing UpAdjustDiskUsageDelta walk. That walk takes per-level DiskUsages locks and could be re-entered from other call paths that hold a node-level lock and then acquire vl.accessLock. Copy the node list under the VolumeLayout lock and release it before the tree walk to eliminate the lock-ordering hazard.	2026-04-16 12:50:30 -07:00
Chris Lu	b37bbf541a	feat(master): drain pending size before marking volume readonly (#9036) ... * feat(master): drain pending size before marking volume readonly When vacuum, volume move, or EC encoding marks a volume readonly, in-flight assigned bytes may still be pending. This adds a drain step: immediately remove from writable list (stop new assigns), then wait for pending to decay below 4MB or 30s timeout. - Add volumeSizeTracking struct consolidating effectiveSize, reportedSize, and compactRevision into a single map - Add GetPendingSize, waitForPendingDrain, DrainAndRemoveFromWritable, DrainAndSetVolumeReadOnly to VolumeLayout - UpdateVolumeSize detects compaction via compactRevision change and resets effectiveSize instead of decaying - Wire drain into vacuum (topology_vacuum.go) and volume mark readonly (master_grpc_server_volume.go) * fix: use 2MB pending size drain threshold * fix: check crowded state on initial UpdateVolumeSize registration * fix: respect context cancellation in drain, relax test timing - DrainAndSetVolumeReadOnly now accepts context.Context and returns early on cancellation (for gRPC handler timeout/cancel) - waitForPendingDrain uses select on ctx.Done instead of time.Sleep - Increase concurrent heartbeat test timeout from 10s to 15s for CI * fix: use time-based dedup so decay runs even when reported size is unchanged The value-based dedup (same reportedSize + compactRevision = skip) prevented decay from running when pending bytes existed but no writes had landed on disk yet. The reported size stayed the same across heartbeats, so the excess never decayed. Fix: dedup replicas within the same heartbeat cycle using a 2-second time window instead of comparing values. This allows decay to run once per heartbeat cycle even when the reported size is unchanged. Also confirmed finding 1 (draining re-add race) is a false positive: - Vacuum: ensureCorrectWritables only runs for ReadOnly-changed volumes - Move/EC: readonlyVolumes flag prevents re-adding during drain * fix: make VolumeMarkReadonly non-blocking to fix EC integration test timeout The DrainAndSetVolumeReadOnly call in VolumeMarkReadonly gRPC blocked up to 30s waiting for pending bytes to decay. In integration tests (and real clusters during EC encoding), this caused timeouts because multiple volumes are marked readonly sequentially and heartbeats may not arrive fast enough to decay pending within the drain window. Fix: VolumeMarkReadonly now calls SetVolumeReadOnly immediately (stops new assigns) and only logs a warning if pending bytes remain. The drain wait is kept only for vacuum (DrainAndRemoveFromWritable) which runs inside the master's own goroutine pool. Remove DrainAndSetVolumeReadOnly as it's no longer used. * fix: relax test timing, rename test, add post-condition assert * test: add vacuum integration tests with CI workflow Full-cluster integration test for vacuum, modeled on the EC integration tests. Starts a real master + 2 volume servers, uploads data, deletes entries to create garbage, runs volume.vacuum via shell command, and verifies garbage cleanup and data integrity. Test flow: 1. Start cluster (master + 2 volume servers) 2. Upload 10 files to create volume with data 3. Delete 5 files to create ~50% garbage 4. Verify garbage ratio > 10% 5. Run volume.vacuum command 6. Verify garbage cleaned up 7. Verify remaining 5 files are still accessible CI workflow runs on push/PR to master with 15-minute timeout. Log collection on failure via artifact upload. * fix: use 500KB files and delete 75% to exceed vacuum garbage threshold * fix: add shell lock before vacuum command, fix compilation error * fix: strengthen vacuum integration test assertions - waitForServer: use net.DialTimeout instead of grpc.NewClient for real TCP readiness check - verify_garbage_before_vacuum: t.Fatal instead of warning when no garbage detected - verify_cleanup_after_vacuum: t.Fatal if no server reported the volume or cleanup wasn't verified - verify_remaining_data: read actual file contents via HTTP and compare byte-for-byte against original uploaded payloads * fix: use http.Client with timeout and close body before retry	2026-04-11 18:29:11 -07:00
Chris LuandCopilot Autofix powered by AI	e2c79af6ec	feat(master): size-aware volume assignment with weighted selection (#9031) ... * feat(master): size-aware volume assignment with weighted selection PickForWrite now selects volumes proportional to remaining capacity instead of uniform random, so emptier volumes receive more writes. - Add vid2size map to VolumeLayout tracking effective volume sizes - Weighted pick via random sampling (k=3) for O(1) cost - RecordAssign tracks estimated pending bytes between heartbeats - Exponential decay on heartbeat: halve excess each cycle - Proactive crowded detection using effective size - Zero extra heap allocations on the unconstrained hot path Benchmark (20 writable volumes, unconstrained): Before: 36 ns/op, 32 B/op, 2 allocs/op After: 85 ns/op, 32 B/op, 2 allocs/op * fix: address review feedback on size-aware assignment - RecordAssign: use write lock (Lock) instead of read lock (RLock) since it mutates vid2size map and crowded set - RegisterVolume: clear crowded flag when heartbeat decay drops effective size below the threshold - pickWeightedByRemaining: fix misleading Fisher-Yates comment, simplify to plain random sampling (duplicates are harmless) - ShouldGrowVolumesByDcAndRack: read vid2size under RLock * fix: decay once per heartbeat cycle, not per replica RegisterVolume is called once per replica of a volume. For replicated volumes, the pending size decay was running multiple times per heartbeat cycle, reducing the excess by 75% instead of 50% (for 2 replicas). Fix: track vid2reportedSize and only run decay when the heartbeat- reported size actually changes. A second replica reporting the same size in the same cycle is a no-op. Also fix CodeQL alert: cap count*EstimatedNeedleSizeBytes to avoid uint64→int64 overflow in RecordAssign call. * Potential fix for pull request finding 'CodeQL / Incorrect conversion between integer types' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * fix: fail fast in test setup on JSON errors - setupWithLimit now takes testing.TB and calls t.Fatalf on unmarshal errors or type assertion failures instead of printing and continuing - benchSetup removed; benchmarks reuse setupWithLimit directly * fix: run size decay on every heartbeat, not just new volumes RegisterVolume is only called for newly discovered volumes, not on every heartbeat. The pending size decay was never running in production. - Extract decay logic into UpdateVolumeSize(), called from SyncDataNodeRegistration for every reported volume on every heartbeat - RegisterVolume only initializes vid2size for brand-new volumes - Constrained PickForWrite: scan from random offset, collect up to pickSampleSize matches in a stack array (no append allocation) - Tests now exercise UpdateVolumeSize directly instead of RegisterVolume to match the production heartbeat path * fix: compute pending bytes in uint64 to satisfy CodeQL --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>	2026-04-11 09:19:05 -07:00