3 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	ecc0390795	fix(master): eagerly remove volume from writable when assign hits limit (#9108) ... * fix(master): eagerly remove volume from writable when RecordAssign hits limit Previously, a volume was only removed from the writable list by the heartbeat-driven CollectDeadNodeAndFullVolumes pass, which runs every pulse (5s) after a 5s heartbeat. Under sustained concurrent writes, fio-style workloads observed in the field grew volumes 8-20x past the configured 100MB limit (median 530MB, peak 1.98GB) during that 5-15s detection window. RecordAssign already tracks effective size (reported + pending) on each /dir/assign. It now also removes the volume from writable the moment effectiveSize reaches volumeSizeLimit, and mirrors the activeVolumeCount decrement that Topology.SetVolumeCapacityFull would have done on the next heartbeat. The heartbeat path remains unchanged and idempotent (vl.SetVolumeCapacityFull returns false if already removed, so no double-decrement). Recovery still works: if a heartbeat later reports size < limit and the volume is not oversized, EnsureCorrectWritables adds it back. - weed/topology/volume_layout.go: RecordAssign returns reachedCapacity bool; adds AdjustActiveVolumeCountForFull helper. - weed/topology/topology.go: PickForWrite invokes the decrement on eager full transitions. - TestPickForWrite: pass a 1024-byte hint instead of 0 so the default 1MB pendingDelta does not immediately bust the test's 32KB limit. - New TestRecordAssignReachingCapacityRemovesFromWritable covers the eager removal, active count accounting, and no-double-accounting. * fix(master): recover eagerly-removed volume once decay clears pending After RecordAssign eagerly removes a volume from writables because effectiveSize reached the limit, decay can later bring effectiveSize back under the limit (e.g., when a burst of assigns didn't all result in uploads). Without recovery the volume would stay non-writable until vacuum or a ReadOnly flip. UpdateVolumeSize now re-adds the volume to writables once all of the following hold: * RecordAssign is what removed it (tracked via fullSince timestamp) * at least capacityRecoveryDelay has elapsed since the removal (30s) — this prevents bouncing during a steady stream of assigns near the limit * effectiveSize has decayed below the crowded threshold (90% of limit) * reportedSize is under the limit (actual disk is not over) * standard EnsureCorrectWritables preconditions: enough copies, all copies writable, not oversized The caller (SyncDataNodeRegistration) re-increments activeVolumeCount symmetrically with the decrement done on eager removal. * review: release VolumeLayout lock before UpAdjustDiskUsageDelta adjustActiveVolumeCount held vl.accessLock across the tree-climbing UpAdjustDiskUsageDelta walk. That walk takes per-level DiskUsages locks and could be re-entered from other call paths that hold a node-level lock and then acquire vl.accessLock. Copy the node list under the VolumeLayout lock and release it before the tree walk to eliminate the lock-ordering hazard.	2026-04-16 12:50:30 -07:00
Chris Lu	b37bbf541a	feat(master): drain pending size before marking volume readonly (#9036) ... * feat(master): drain pending size before marking volume readonly When vacuum, volume move, or EC encoding marks a volume readonly, in-flight assigned bytes may still be pending. This adds a drain step: immediately remove from writable list (stop new assigns), then wait for pending to decay below 4MB or 30s timeout. - Add volumeSizeTracking struct consolidating effectiveSize, reportedSize, and compactRevision into a single map - Add GetPendingSize, waitForPendingDrain, DrainAndRemoveFromWritable, DrainAndSetVolumeReadOnly to VolumeLayout - UpdateVolumeSize detects compaction via compactRevision change and resets effectiveSize instead of decaying - Wire drain into vacuum (topology_vacuum.go) and volume mark readonly (master_grpc_server_volume.go) * fix: use 2MB pending size drain threshold * fix: check crowded state on initial UpdateVolumeSize registration * fix: respect context cancellation in drain, relax test timing - DrainAndSetVolumeReadOnly now accepts context.Context and returns early on cancellation (for gRPC handler timeout/cancel) - waitForPendingDrain uses select on ctx.Done instead of time.Sleep - Increase concurrent heartbeat test timeout from 10s to 15s for CI * fix: use time-based dedup so decay runs even when reported size is unchanged The value-based dedup (same reportedSize + compactRevision = skip) prevented decay from running when pending bytes existed but no writes had landed on disk yet. The reported size stayed the same across heartbeats, so the excess never decayed. Fix: dedup replicas within the same heartbeat cycle using a 2-second time window instead of comparing values. This allows decay to run once per heartbeat cycle even when the reported size is unchanged. Also confirmed finding 1 (draining re-add race) is a false positive: - Vacuum: ensureCorrectWritables only runs for ReadOnly-changed volumes - Move/EC: readonlyVolumes flag prevents re-adding during drain * fix: make VolumeMarkReadonly non-blocking to fix EC integration test timeout The DrainAndSetVolumeReadOnly call in VolumeMarkReadonly gRPC blocked up to 30s waiting for pending bytes to decay. In integration tests (and real clusters during EC encoding), this caused timeouts because multiple volumes are marked readonly sequentially and heartbeats may not arrive fast enough to decay pending within the drain window. Fix: VolumeMarkReadonly now calls SetVolumeReadOnly immediately (stops new assigns) and only logs a warning if pending bytes remain. The drain wait is kept only for vacuum (DrainAndRemoveFromWritable) which runs inside the master's own goroutine pool. Remove DrainAndSetVolumeReadOnly as it's no longer used. * fix: relax test timing, rename test, add post-condition assert * test: add vacuum integration tests with CI workflow Full-cluster integration test for vacuum, modeled on the EC integration tests. Starts a real master + 2 volume servers, uploads data, deletes entries to create garbage, runs volume.vacuum via shell command, and verifies garbage cleanup and data integrity. Test flow: 1. Start cluster (master + 2 volume servers) 2. Upload 10 files to create volume with data 3. Delete 5 files to create ~50% garbage 4. Verify garbage ratio > 10% 5. Run volume.vacuum command 6. Verify garbage cleaned up 7. Verify remaining 5 files are still accessible CI workflow runs on push/PR to master with 15-minute timeout. Log collection on failure via artifact upload. * fix: use 500KB files and delete 75% to exceed vacuum garbage threshold * fix: add shell lock before vacuum command, fix compilation error * fix: strengthen vacuum integration test assertions - waitForServer: use net.DialTimeout instead of grpc.NewClient for real TCP readiness check - verify_garbage_before_vacuum: t.Fatal instead of warning when no garbage detected - verify_cleanup_after_vacuum: t.Fatal if no server reported the volume or cleanup wasn't verified - verify_remaining_data: read actual file contents via HTTP and compare byte-for-byte against original uploaded payloads * fix: use http.Client with timeout and close body before retry	2026-04-11 18:29:11 -07:00
Chris LuandCopilot Autofix powered by AI	e2c79af6ec	feat(master): size-aware volume assignment with weighted selection (#9031) ... * feat(master): size-aware volume assignment with weighted selection PickForWrite now selects volumes proportional to remaining capacity instead of uniform random, so emptier volumes receive more writes. - Add vid2size map to VolumeLayout tracking effective volume sizes - Weighted pick via random sampling (k=3) for O(1) cost - RecordAssign tracks estimated pending bytes between heartbeats - Exponential decay on heartbeat: halve excess each cycle - Proactive crowded detection using effective size - Zero extra heap allocations on the unconstrained hot path Benchmark (20 writable volumes, unconstrained): Before: 36 ns/op, 32 B/op, 2 allocs/op After: 85 ns/op, 32 B/op, 2 allocs/op * fix: address review feedback on size-aware assignment - RecordAssign: use write lock (Lock) instead of read lock (RLock) since it mutates vid2size map and crowded set - RegisterVolume: clear crowded flag when heartbeat decay drops effective size below the threshold - pickWeightedByRemaining: fix misleading Fisher-Yates comment, simplify to plain random sampling (duplicates are harmless) - ShouldGrowVolumesByDcAndRack: read vid2size under RLock * fix: decay once per heartbeat cycle, not per replica RegisterVolume is called once per replica of a volume. For replicated volumes, the pending size decay was running multiple times per heartbeat cycle, reducing the excess by 75% instead of 50% (for 2 replicas). Fix: track vid2reportedSize and only run decay when the heartbeat- reported size actually changes. A second replica reporting the same size in the same cycle is a no-op. Also fix CodeQL alert: cap count*EstimatedNeedleSizeBytes to avoid uint64→int64 overflow in RecordAssign call. * Potential fix for pull request finding 'CodeQL / Incorrect conversion between integer types' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * fix: fail fast in test setup on JSON errors - setupWithLimit now takes testing.TB and calls t.Fatalf on unmarshal errors or type assertion failures instead of printing and continuing - benchSetup removed; benchmarks reuse setupWithLimit directly * fix: run size decay on every heartbeat, not just new volumes RegisterVolume is only called for newly discovered volumes, not on every heartbeat. The pending size decay was never running in production. - Extract decay logic into UpdateVolumeSize(), called from SyncDataNodeRegistration for every reported volume on every heartbeat - RegisterVolume only initializes vid2size for brand-new volumes - Constrained PickForWrite: scan from random offset, collect up to pickSampleSize matches in a stack array (no append allocation) - Tests now exercise UpdateVolumeSize directly instead of RegisterVolume to match the production heartbeat path * fix: compute pending bytes in uint64 to satisfy CodeQL --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>	2026-04-11 09:19:05 -07:00