mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-19 10:54:08 +01:00
c7d8b106d50f72da1ab10df8aee3cd2fa080caa4
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
5681be241e |
topology: never report negative free volume slots (#12698)
* topology: never report negative free volume slots Free volume slots are computed as max + remote - volume - ecSlots. When phantom or over-registered volumes push the count past the configured max, the result is negative and rolls up into the cluster total, causing volume grow and assign to fail with a negative 'volumes left' count even while healthy servers have capacity. Clamp every free-volume count at zero so an over-committed server contributes no slots instead of subtracting from the cluster. * topology: drop the issue reference from the free-slot test comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * topology: test free slots next to an over-committed server A server with more volumes than slots must not hide the free slots of the other servers in its rack, data center or topology. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * topology: sum free slots per data node in racks, data centers and topology Clamping the rolled-up counters only clamped their sum, so one server with more volumes than slots still hid the free slots of every other server in its rack, data center and topology. Interior nodes now add up their children's free slots, so each data node counts as zero at worst. A data node in maintenance mode no longer adds slots to its parents either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * topology: test that disk info keeps an over-committed deficit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * topology: keep the signed free volume count in disk info EC encode retries and the balance planners add refunds and planned moves to the published FreeVolumeCount. Clamping it at zero dropped an over-committed disk's deficit, so a refund could make a full disk look free. Volume growth already reads the clamped AvailableSpaceFor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * topology: test placement next to an over-committed server Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * topology: accept either data center order in the placement spread check Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3fd8d0cf09 |
master: let location-constrained assigns wait for pending growth (#12167)
* topology: add AvailableSpaceWithinLocation for location-scoped free slots AvailableSpaceFor answers for the whole topology; a location-constrained assign needs to know whether the requested data center, rack, or data node still has room before deciding growth can satisfy it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: let location-constrained assigns wait for pending growth An assign carrying a data center or rack constraint failed on the first unsatisfied pick, even when the same request had just queued volume growth for that location — so the first write into a fresh collection errored out instead of picking up the volume growth landed a moment later. Keep the fail-fast path only when growth cannot help: growth disabled, no growth pending, or no free slots left inside the constraint. Otherwise the request takes the same wait/shed path as an unconstrained assign. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: sum leaf free slots in AvailableSpaceWithinLocation Rolled-up rack and data center counters keep a maintenance-mode server's capacity, and one overcommitted node can cancel a sibling's slots. Sum the matching data nodes directly so the location check agrees with what growth can actually use. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: let constrained growth initiators reach the shed path When a constrained assign's own growth concluded unfulfilled, the fast-fail check saw only that no grow was pending and returned the plain pick error — a code the assign client does not retry, so the first write died instead of re-triggering growth. Exclude the request's own initiated grow from the no-pending-growth condition so it reaches the existing re-pick and ResourceExhausted shed like an unconstrained assign. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: judge targeted grow requests by the location they name A grow request carrying a data center, rack, or data node constraint was discarded whenever the layout's writables elsewhere were healthy, so a location with free slots but no writable could never receive the volume its assign was waiting for. Add ShouldGrowVolumesWithinLocation and use it for constrained requests; the layout-wide check still governs untargeted ones. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: only positive leaf slots count in AvailableSpaceWithinLocation An over-committed node reports negative free slots; summed plainly it\ncancelled a sibling's room and a constrained assign could fail fast\nwhile the location still had space to grow into. * master: fail constrained assigns fast when growth cannot place A constrained assign with free slots in its location still waited out\ngrowth that could never satisfy the replica placement — a single-rack\ndata center asked for 010 sheds the same doomed grow on every client\nretry. Run the grow-time node search as a dry run and return its\nplacement error instead of a retryable shed. * master: judge constrained growth by layout, not free slots A slot-level feasibility probe races with the grow it judges: a\npending grow consuming the last slot in one rack reports the\nplacement impossible while it mints the request's own volume, and\nthe hard error skips the re-pick that would find it. GrowthBlocker\nnow counts racks and data nodes inside the named location — shape a\ngrow cannot consume — so only genuinely unplaceable layouts fail\nfast. * topology: GrowthBlocker mirrors the grow selector's feasibility The location filter bound every replica tier, so a 010 request naming one rack needed two racks inside it while the selector constrains only the first placement and counts the rest by room. And counting nodes without free slots let a full location queue a grow it could never serve, so the assign shed ResourceExhausted at timeout instead of failing fast. GrowthBlocker now applies the option's location to the primary placement chain only and leaf-sums free slots the way the selector does, so it reports a blocker exactly when the grow can never place. * master: repick once before surfacing a growth blocker An in-flight grow can spend the last free slot minting the request's own volume: the blocker's slot counts predate that registration by a heartbeat, so returning the placement error on the first failed pick could reject a write whose volume lands a moment later. A pending grow now earns one repick grace before the blocker error is returned. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
80f3cc0613 |
topology: keep oversized volumes out of the writable list across re-registration (#12120)
* topology: keep the oversized verdict after the location list is gone The per-replica oversized marks live in the volume's location list, which is deleted when its last location leaves and recreated flagless on the next registration. Between that recreation and the next heartbeat that re-measures the volume, ensureCorrectWritables saw no mark and re-admitted an oversized volume to the writable list; two other paths (quarantine release, mark-writable) never consulted the marks at all. VolumeLayout now remembers the oversized verdict per vid, written only by heartbeat sizes and forgotten only when a report measures the volume under the limit. The writable admission gate itself refuses a marked volume, and the vacuum commit path drops a verdict its live size check has just disproved. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: mark oversized at registration when the report carries a size RegisterVolumeLayout ran EnsureCorrectWritables while the recreated location list still held no per-replica marks. When the registering report knows the volume's size, refresh the oversized state before the writable check instead of waiting for the next heartbeat pass. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: test that oversized volumes stay out of writables across re-registration Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: remember the oversized verdict per replica A shared per-volume verdict was cleared by any single replica's under-limit report, so a still-oversized replica whose location entry had died could re-register from a size-less report and put the volume back on the writable list. Keep one verdict per reporting replica: an under-limit report clears only its own, and a replica that never returns cannot pin its volume out of writables forever because the verdict ages out. The vacuum commit now feeds the node's fresh answer through the same per-replica bookkeeping instead of erasing the whole verdict. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: count a volume at the size limit as full on vacuum commit The layout's oversized test is >= the limit, but the vacuum commit and status checks used >, so a replica sitting exactly at the limit passed as not-full and the commit path could re-admit it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: cover per-replica oversized verdicts in tests One replica shrinking must not clear a still-oversized replica's remembered verdict; cover that plus a size-less re-registration before the second replica is re-measured. The vacuum-commit case now shrinks the stored record first, since the commit re-derives the mark from the node's own answer. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: judge the vacuum commit by the size it just measured SetVolumeAvailable rebuilt the oversized verdict from the cached VolumeInfo, which still carries the pre-compaction size until the next heartbeat. A commit that shrank a volume under the limit re-marked the replica oversized and never restored writes. The commit path already asks every replica for its fresh size; pass it through per replica and record it on the mark, the verdict, and the size tracking seed. * topology: hold an expired oversized verdict while its replica is back The TTL forgives a replica that never returns, but a remount announces the volume without a size and registration skips the oversized check. Deleting the verdict on expiry then re-admitted a still-oversized volume until the next size-bearing heartbeat. Keep counting an expired verdict while its replica sits in the live location list, and keep sweeping verdicts whose replicas stayed gone. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chrislusf@gmail.com> |
||
|
|
3e5aa8c3b3 |
topology: mark replica quarantined before recording the pair (#11983)
* topology: serialize quarantine records with their layout marks A quarantine publishes a (node, vid) record and marks the replica read-only at the layout level; release and heartbeat flag updates must observe that pair as one state. Store-then-mark let a concurrent release delete the record before the mark landed, stranding the surviving replica read-only; mark-then-store let a heartbeat clear the mark before the record was visible, returning a conflicting replica to the writable list. quarantineLock now covers every record<->mark pair: flagDuplicateVolumeId's quarantine, releaseResolvedQuarantine's undo, the isQuarantined-gated marks on registration, and the quarantine-folded read-only flag writes on heartbeat paths. Remote server pushes stay outside the lock; a push that lands after the record was released is undone, and a failed release push re-adds the record for a later retry. The release test now waits for in-flight async flags to settle instead of asserting on a single read. * topology: fold quarantine into RegisterVolume's flag rebuild RegisterVolume rewrites every replica's location read-only flag from what its node last reported, and reported flags deliberately exclude quarantine. Any registration of a shared vid -- a reconnected replica, a new copy, the self-heal and delta-heartbeat re-register paths -- therefore cleared the quarantine mark on every replica in the location, while the gate afterwards only re-marked the registering node. Two call sites had no gate at all. RegisterVolume now takes an isQuarantined predicate and evaluates it under accessLock for every replica it touches, so the mark is never dropped in the first place: a release unmarks after the rebuild, and a quarantine mark re-applies on the next rebuild, with no window between. * topology: keep quarantine when a vacuum commit re-adds a replica SetVolumeAvailable re-adds the replica into the location list; a re-added slot starts with a cleared read-only flag, so a quarantined replica came back flag-free and the next ensureCorrectWritables could return a conflicting volume to the writable list. Pass the quarantine state in and set the flag back under the same accessLock, with the caller holding quarantineLock so release cannot interleave between the check and the mark. * topology: merge quarantine records instead of overwriting them enforceQuarantine wrote back the state copy it loaded before its remote push, so a concurrent writer's enforced=true was lost: the server stayed read-only while the record said unenforced, and a later release skipped the writable push it owed. The write-back now reloads the record under quarantineLock and merges -- enforced wins from either side, reported is taken from the newer pre-enforcement observation only while no push has landed yet. All three compensating writable pushes had the same hole in reverse: a failed un-quarantine left the server read-only with no record to retry from, and release's restore blindly overwrote any record a fresh flag pass had created in between. They now go through restoreQuarantineRecord, which adds the record only when absent and otherwise sets enforced on the current one, preserving its reported flag. * topology: hold quarantineLock over registration's quarantine check-and-mark RegisterVolume reads the quarantine record and marks the replica read-only under accessLock, but the pair ran without quarantineLock. A release could push writable, watch the record get re-read, and then delete it: the replica stayed read-only with no record left to clear, and unchanged writable heartbeats skip SetReplicaReadOnlyFlag so the mark never repaired. Holding quarantineLock around the check-and-mark in all three production callers serializes it against the release clear-and-delete. * topology: serialize remote quarantine pushes behind a record recheck releaseResolvedQuarantine deletes the record under quarantineLock and sends the writable request after unlock. A conflict flagged while that request is in flight creates a fresh record and marks the replica read-only again, and the older writable request can still land last, leaving the server writable under an active quarantine. The reverse flap strands a stale read-only mark the same way. Route every QuarantineVolumeFn call through pushQuarantine, which holds quarantinePushLock across the remote call and re-checks the record under quarantineLock right before sending: a read-only push requires the pair to exist, a writable push requires it to be gone. A skipped push means the decision already moved on, so the final server state always follows the newest record. * topology: serialize quarantine pushes per replica, not globally A single quarantinePushLock held across the remote mark call let one slow volume server (up to the 10s client timeout) delay pushes for unrelated replicas. Keying the lock by (node, vid) keeps pushes to one replica ordered in record-decision order while pushes to others run in parallel. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: restore the layout read-only mark with a released record A failed writable push re-armed the quarantine record but left the replica marked writable in the layout, so PickForWrite could route to a replica the server still holds read-only. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: wait out async duplicate checks in the release-retry test TestFailedReleasePushRestoresQuarantineMark asserts layout marks around a release pass, but the two RegisterVolumeLayout calls each start a background flagDuplicateVolumeId that can still be mid-flight. A late mark could land after the final release and fail the writable assertion even though the code behaved correctly. Track the async passes in a WaitGroup and drain them before the explicit flag call so the rest of the test is deterministic. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: keep quarantine when a replica's layout key is unreadable * topology: keep a failed writable push's restore ordered under the push lock * topology: retry the quarantine release on every duplicate check * ci: isolate Java and S3 test servers on persistent runners * ci: drop the duplicated -ip flags in the lifecycle test server The bind flags were already on the command; the isolation pass added a second pair. * topology: retry a failed quarantine writable push on the next report * topology: retry pending quarantine releases off the leader pulse A failed writable push re-arms the quarantine record with a retry that only enforceQuarantine fires, and that runs only for volumes a heartbeat carries. Once the server has reported the replica read-only it drops out of the delta reports, and restoreQuarantineRecord keeps the stored flag matching the server's so the digests agree and no full list is requested - the pending release then never retries and the idle volume stays read-only forever. Run the same release pass from the leader's pulse loop so a pending retry no longer waits on a report. * topology: exercise the pulse-driven quarantine release retry --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
cc281dabc9 |
master: keep new volumes and writes off servers in maintenance mode (#11147)
* master: keep new volumes and writes off servers in maintenance mode The master recorded a volume server's maintenance flag from the heartbeat but never consulted it. A server in maintenance (#7977) is being drained, yet the master kept creating volumes on it whenever it had free slots and kept handing out its volumes for writes. Nothing on the volume server blocks plain HTTP uploads either, so "read-only mode" was only a name. Volume growth: a data node in maintenance mode reports zero free slots through AvailableSpaceFor, which takes it out of every candidate list, feasibility count and capacity reservation. Its slots still roll up into its rack and data center, so the random offset drawn from those totals for an other-rack or other-DC replica could land in space the walk then skips and fail with "No free volume slot found!" while siblings had room; the walk now folds the offset into the space that is actually eligible. This also covers the pre-existing case of an over-committed sibling. Assignment: a replica on a server in maintenance mode is treated like a read-only replica in isAllWritable, so its volume leaves the writable list and returns when the flag clears. Topology.SetDataNodeMaintenanceMode re-evaluates the node's volumes on every change, since heartbeats are digest-based and a full volume list may not follow for a long time. Reads and lookups are untouched. The flag moves to an atomic so the assign and growth paths can read it without the node lock. Heartbeat: the Go volume server sent its state only when it changed, so a master elected while a server sat in maintenance never learned about it. The state now rides along on every heartbeat, as the Rust server already does; the master's compare is an atomic swap, and only a change does work. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: hold maintenance mode through vacuum commit and mark-writable SetVolumeAvailable and SetVolumeWritable put a volume back on the writable list on the replica count alone. A vacuum that started before the server entered maintenance, or a vacuum worker's mark-writable arriving after it, handed the volume back to assignment with a replica on the draining server. Heartbeats carry only changed volumes, so nothing re-evaluated it until the volume itself changed. Apply isAllWritable on both paths, the same test EnsureCorrectWritables uses. Also pin that re-evaluating a volume a concurrent disconnect already removed from its layout is a no-op. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: record a server's read-only notification on its node before judging the volume A volume server notifies the master the moment it flips a volume between read-only and writable, ahead of the heartbeat that repeats the flag. The layout only set its per-location flag, so isAllWritable, which reads the node's heartbeat copy, still saw the old value: a mark-writable was withheld until the next heartbeat, and a re-evaluation landing between a mark-readonly and its heartbeat put the volume back on the writable list. Record the flag on the node's volume first. AddOrUpdateVolume keeps the digest and the active volume count in step, so the heartbeat that follows finds nothing to change. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: a read-only mark does not confirm a provisional volume DataNode.SetVolumeReadOnly went through Disk.AddOrUpdateVolume, which treats its input as a server report and so ended the grace period that keeps a just-grown volume safe from a full report collected before the grow. A volume marked read-only before its first report could then be removed by that stale report. Give Disk a SetVolumeReadOnly that flips the flag and keeps the digest and active volume count in step without touching volumeAddedAt. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |