Commit Graph
23 Commits
Author SHA1 Message Date
Chris LuandDevin 3fd8d0cf09 master: let location-constrained assigns wait for pending growth (#12167)
* topology: add AvailableSpaceWithinLocation for location-scoped free slots

AvailableSpaceFor answers for the whole topology; a location-constrained
assign needs to know whether the requested data center, rack, or data node
still has room before deciding growth can satisfy it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: let location-constrained assigns wait for pending growth

An assign carrying a data center or rack constraint failed on the first
unsatisfied pick, even when the same request had just queued volume growth
for that location — so the first write into a fresh collection errored out
instead of picking up the volume growth landed a moment later.

Keep the fail-fast path only when growth cannot help: growth disabled, no
growth pending, or no free slots left inside the constraint. Otherwise the
request takes the same wait/shed path as an unconstrained assign.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: sum leaf free slots in AvailableSpaceWithinLocation

Rolled-up rack and data center counters keep a maintenance-mode server's
capacity, and one overcommitted node can cancel a sibling's slots. Sum the
matching data nodes directly so the location check agrees with what growth
can actually use.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: let constrained growth initiators reach the shed path

When a constrained assign's own growth concluded unfulfilled, the fast-fail
check saw only that no grow was pending and returned the plain pick error —
a code the assign client does not retry, so the first write died instead of
re-triggering growth. Exclude the request's own initiated grow from the
no-pending-growth condition so it reaches the existing re-pick and
ResourceExhausted shed like an unconstrained assign.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: judge targeted grow requests by the location they name

A grow request carrying a data center, rack, or data node constraint was
discarded whenever the layout's writables elsewhere were healthy, so a
location with free slots but no writable could never receive the volume its
assign was waiting for. Add ShouldGrowVolumesWithinLocation and use it for
constrained requests; the layout-wide check still governs untargeted ones.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: only positive leaf slots count in AvailableSpaceWithinLocation

An over-committed node reports negative free slots; summed plainly it\ncancelled a sibling's room and a constrained assign could fail fast\nwhile the location still had space to grow into.

* master: fail constrained assigns fast when growth cannot place

A constrained assign with free slots in its location still waited out\ngrowth that could never satisfy the replica placement — a single-rack\ndata center asked for 010 sheds the same doomed grow on every client\nretry. Run the grow-time node search as a dry run and return its\nplacement error instead of a retryable shed.

* master: judge constrained growth by layout, not free slots

A slot-level feasibility probe races with the grow it judges: a\npending grow consuming the last slot in one rack reports the\nplacement impossible while it mints the request's own volume, and\nthe hard error skips the re-pick that would find it. GrowthBlocker\nnow counts racks and data nodes inside the named location — shape a\ngrow cannot consume — so only genuinely unplaceable layouts fail\nfast.

* topology: GrowthBlocker mirrors the grow selector's feasibility

The location filter bound every replica tier, so a 010 request naming one rack needed two racks inside it while the selector constrains only the first placement and counts the rest by room. And counting nodes without free slots let a full location queue a grow it could never serve, so the assign shed ResourceExhausted at timeout instead of failing fast. GrowthBlocker now applies the option's location to the primary placement chain only and leaf-sums free slots the way the selector does, so it reports a blocker exactly when the grow can never place.

* master: repick once before surfacing a growth blocker

An in-flight grow can spend the last free slot minting the request's own volume: the blocker's slot counts predate that registration by a heartbeat, so returning the placement error on the first failed pick could reject a write whose volume lands a moment later. A pending grow now earns one repick grace before the blocker error is returned.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 08:10:45 -07:00
Chris LuandDevin 3e5aa8c3b3 topology: mark replica quarantined before recording the pair (#11983)
* topology: serialize quarantine records with their layout marks

A quarantine publishes a (node, vid) record and marks the replica read-only at the layout level; release and heartbeat flag updates must observe that pair as one state. Store-then-mark let a concurrent release delete the record before the mark landed, stranding the surviving replica read-only; mark-then-store let a heartbeat clear the mark before the record was visible, returning a conflicting replica to the writable list.

quarantineLock now covers every record<->mark pair: flagDuplicateVolumeId's quarantine, releaseResolvedQuarantine's undo, the isQuarantined-gated marks on registration, and the quarantine-folded read-only flag writes on heartbeat paths. Remote server pushes stay outside the lock; a push that lands after the record was released is undone, and a failed release push re-adds the record for a later retry.

The release test now waits for in-flight async flags to settle instead of asserting on a single read.

* topology: fold quarantine into RegisterVolume's flag rebuild

RegisterVolume rewrites every replica's location read-only flag from what
its node last reported, and reported flags deliberately exclude quarantine.
Any registration of a shared vid -- a reconnected replica, a new copy, the
self-heal and delta-heartbeat re-register paths -- therefore cleared the
quarantine mark on every replica in the location, while the gate afterwards
only re-marked the registering node. Two call sites had no gate at all.

RegisterVolume now takes an isQuarantined predicate and evaluates it under
accessLock for every replica it touches, so the mark is never dropped in
the first place: a release unmarks after the rebuild, and a quarantine
mark re-applies on the next rebuild, with no window between.

* topology: keep quarantine when a vacuum commit re-adds a replica

SetVolumeAvailable re-adds the replica into the location list; a re-added
slot starts with a cleared read-only flag, so a quarantined replica came
back flag-free and the next ensureCorrectWritables could return a
conflicting volume to the writable list. Pass the quarantine state in and
set the flag back under the same accessLock, with the caller holding
quarantineLock so release cannot interleave between the check and the mark.

* topology: merge quarantine records instead of overwriting them

enforceQuarantine wrote back the state copy it loaded before its remote
push, so a concurrent writer's enforced=true was lost: the server stayed
read-only while the record said unenforced, and a later release skipped
the writable push it owed. The write-back now reloads the record under
quarantineLock and merges -- enforced wins from either side, reported is
taken from the newer pre-enforcement observation only while no push has
landed yet.

All three compensating writable pushes had the same hole in reverse: a
failed un-quarantine left the server read-only with no record to retry
from, and release's restore blindly overwrote any record a fresh flag
pass had created in between. They now go through restoreQuarantineRecord,
which adds the record only when absent and otherwise sets enforced on
the current one, preserving its reported flag.

* topology: hold quarantineLock over registration's quarantine check-and-mark

RegisterVolume reads the quarantine record and marks the replica
read-only under accessLock, but the pair ran without quarantineLock.
A release could push writable, watch the record get re-read, and then
delete it: the replica stayed read-only with no record left to clear,
and unchanged writable heartbeats skip SetReplicaReadOnlyFlag so the
mark never repaired. Holding quarantineLock around the check-and-mark
in all three production callers serializes it against the release
clear-and-delete.

* topology: serialize remote quarantine pushes behind a record recheck

releaseResolvedQuarantine deletes the record under quarantineLock and
sends the writable request after unlock. A conflict flagged while that
request is in flight creates a fresh record and marks the replica
read-only again, and the older writable request can still land last,
leaving the server writable under an active quarantine. The reverse
flap strands a stale read-only mark the same way.

Route every QuarantineVolumeFn call through pushQuarantine, which
holds quarantinePushLock across the remote call and re-checks the
record under quarantineLock right before sending: a read-only push
requires the pair to exist, a writable push requires it to be gone.
A skipped push means the decision already moved on, so the final
server state always follows the newest record.

* topology: serialize quarantine pushes per replica, not globally

A single quarantinePushLock held across the remote mark call let one slow
volume server (up to the 10s client timeout) delay pushes for unrelated
replicas. Keying the lock by (node, vid) keeps pushes to one replica
ordered in record-decision order while pushes to others run in parallel.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: restore the layout read-only mark with a released record

A failed writable push re-armed the quarantine record but left the
replica marked writable in the layout, so PickForWrite could route to a
replica the server still holds read-only.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: wait out async duplicate checks in the release-retry test

TestFailedReleasePushRestoresQuarantineMark asserts layout marks around a
release pass, but the two RegisterVolumeLayout calls each start a
background flagDuplicateVolumeId that can still be mid-flight. A late
mark could land after the final release and fail the writable assertion
even though the code behaved correctly. Track the async passes in a
WaitGroup and drain them before the explicit flag call so the rest of
the test is deterministic.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: keep quarantine when a replica's layout key is unreadable

* topology: keep a failed writable push's restore ordered under the push lock

* topology: retry the quarantine release on every duplicate check

* ci: isolate Java and S3 test servers on persistent runners

* ci: drop the duplicated -ip flags in the lifecycle test server

The bind flags were already on the command; the isolation pass added a
second pair.

* topology: retry a failed quarantine writable push on the next report

* topology: retry pending quarantine releases off the leader pulse

A failed writable push re-arms the quarantine record with a retry that only
enforceQuarantine fires, and that runs only for volumes a heartbeat carries.
Once the server has reported the replica read-only it drops out of the delta
reports, and restoreQuarantineRecord keeps the stored flag matching the
server's so the digests agree and no full list is requested - the pending
release then never retries and the idle volume stays read-only forever. Run
the same release pass from the leader's pulse loop so a pending retry no
longer waits on a report.

* topology: exercise the pulse-driven quarantine release retry

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:00:45 -07:00
Chris Lu f0d2a0d417 Treat co-located volume servers as one fault domain when balancing and allocating (#9854)
* admin/topology: carry the volume server address on DiskInfo

The planning DiskInfo exposed only the node id, which can be an opaque label rather than ip:port. Record the address too so callers can resolve the physical machine a disk sits on.

* ec.balance: spread a volume's shards across machines, not just nodes

Volume servers sharing a host are one fault domain, but the within-rack spread treated them as independent nodes, so one box could end up holding more shards of a volume than EC can afford to lose. Add a machine (host) tier between rack and node: the within-rack pass spreads each volume across machines, and the global load phase no longer re-concentrates a volume onto a machine it already sits on. Host defaults to the node id, so clusters with one server per host are unchanged.

* ec placement: prefer machines holding fewer of a volume's shards

EC allocation and repair picked the least-loaded node in a rack with no regard for which physical machine it sits on, so a volume's shards could pile onto several servers of one box. Rank candidate nodes by their machine's shard count first, then the node's own. The machine is derived from the volume server address carried on DiskInfo, falling back to the node id, matching how the balancer resolves it.

* volume.balance: don't move a replica onto a machine already holding one

isGoodMove only rejected a move onto the same data node, so two replicas could land on two volume servers of one box and a single machine failure would lose both. Reject a target whose host already holds another replica of the volume. Best-effort: balancing simply skips and tries the next target.

* volume allocation: spread same-rack replicas across machines

PickNodesByWeight filled the same-rack replica picks by weight alone, so replicas could co-locate on one box. Prefer candidates on not-yet-used hosts, falling back when too few distinct machines exist. Data-center and rack tiers have no host, so their ordering is unchanged.

* ec.balance: harden machine spread against re-concentration and capped machines

Two cases where the machine-aware spread could still leave a volume badly placed:

- The global load phase could move a shard of a volume onto a machine that
  already held it, raising that machine's count and undoing the within-rack
  spread (a 4/4/3/3 layout could become 3/5/3/3, past parity for 10+4). Limit
  the load-only fallback to same-machine moves, which leave a machine's count
  unchanged; cross-machine concentration is no longer allowed for load alone.

- The within-rack spread chose a destination machine by free slots alone, so if
  that machine's only nodes were already at the SameRackCount cap it skipped the
  move instead of trying another machine. Require a machine to have a node that
  can actually take the shard before selecting it.

* reduce comments across the machine-affinity change

Trim narration down to the non-obvious why; one terse line where a block was overkill.

* ec.balance: gate machine spread on fault-tolerance feasibility

Spreading a volume evenly across machines only helps when there are enough that
each can stay within EC's parity tolerance (numMachines >= ceil(total/parity)).
With fewer -- or wildly unequal -- machines it can't make a machine loss
survivable anyway, and forcing it fights capacity: e.g. a cluster of 12 volume
servers on one host and 2 on another would have half of every volume crammed onto
the 2-server box. So spread across machines only when it's achievable; otherwise
fall back to per-node spread and let capacity/global balancing decide.

The global load phase applies the same test: it protects a volume's machine spread
(no cross-machine move that raises a machine's count past the source's) only where
that spread is achievable, so heterogeneous clusters still level by fullness.

* ec.balance worker: group servers by host when planning

The worker built its planner topology without recording each server's host, so
automated ec.balance treated ports on one machine as independent nodes and could
concentrate a volume's shards on one physical box. Set the host from the volume
server address, matching the shell path.

* volume.balance worker: don't move a replica onto a machine holding one

The worker compared only node ids, and the replica map dropped the server address,
so it could move replicas onto different ports of one machine. Carry the host on
ReplicaLocation (from the server address) and reject a target whose host already
holds another replica of the volume. Best-effort, matching the shell.

* ec.balance: judge machine-spread feasibility by the rack's shards

The within-rack and global feasibility checks compared the whole volume's shard
count against a rack's machine count, so a rack holding only part of a volume after
cross-rack spreading -- e.g. 7 of a 10+4 volume across 2 machines -- was wrongly
judged infeasible and fell back to node spread, which could pile 6 shards onto one
host, past parity. Gate on the rack's own shard count of the volume instead.

* ec.balance: spread a volume's shards across machines by combined count

EC recovers from any loss within parity regardless of shard type, so what bounds a
machine's exposure is its total shards of the volume, not data and parity
separately. Spreading the two independently let each type's remainder land on the
same machine -- ceil(d/M)+ceil(p/M) can exceed ceil(total/M), e.g. a 5/3 split where
4/4 was achievable, past parity. Balance the combined count in one pass; disk-level
data/parity anti-affinity stays in pickBestDiskOnNode.

* ec.balance: don't let the imbalance threshold skip an over-parity machine

The within-rack spread gated on relative skew ((max-min)/avg > threshold), so a
worker threshold of 0.5 skipped an exactly-50%-skewed layout like 5/4/3 for a 10+4
volume, leaving 5 shards -- past parity -- on one machine. The even cap
(ceil(shards/groups)) is the real bound and the move loop already sheds only what
exceeds it, so drop the threshold gate from the within-rack phase (machine and node):
a balanced rack stays a no-op while any over-cap machine is always fixed.

* ec.balance: keep the imbalance threshold for the node fallback

Dropping the threshold from the whole within-rack phase made the node fallback too
eager: it runs only when machine fault tolerance is unachievable, so it is cosmetic
load distribution that should defer to the global utilization phase. Without the
gate it would, for a one-server-per-host 6/4 split at threshold 0.5, schedule a count
move that worsens utilization balance. Restore the threshold there; machine spreading
keeps bypassing it, since that bound is durability, not cosmetic skew.
2026-06-07 14:14:45 -07:00
Chris Lu ecc0390795 fix(master): eagerly remove volume from writable when assign hits limit (#9108)
* fix(master): eagerly remove volume from writable when RecordAssign hits limit

Previously, a volume was only removed from the writable list by the
heartbeat-driven CollectDeadNodeAndFullVolumes pass, which runs every
pulse (5s) after a 5s heartbeat. Under sustained concurrent writes,
fio-style workloads observed in the field grew volumes 8-20x past the
configured 100MB limit (median 530MB, peak 1.98GB) during that
5-15s detection window.

RecordAssign already tracks effective size (reported + pending) on each
/dir/assign. It now also removes the volume from writable the moment
effectiveSize reaches volumeSizeLimit, and mirrors the activeVolumeCount
decrement that Topology.SetVolumeCapacityFull would have done on the
next heartbeat. The heartbeat path remains unchanged and idempotent
(vl.SetVolumeCapacityFull returns false if already removed, so no
double-decrement).

Recovery still works: if a heartbeat later reports size < limit and
the volume is not oversized, EnsureCorrectWritables adds it back.

- weed/topology/volume_layout.go: RecordAssign returns reachedCapacity
  bool; adds AdjustActiveVolumeCountForFull helper.
- weed/topology/topology.go: PickForWrite invokes the decrement on
  eager full transitions.
- TestPickForWrite: pass a 1024-byte hint instead of 0 so the default
  1MB pendingDelta does not immediately bust the test's 32KB limit.
- New TestRecordAssignReachingCapacityRemovesFromWritable covers the
  eager removal, active count accounting, and no-double-accounting.

* fix(master): recover eagerly-removed volume once decay clears pending

After RecordAssign eagerly removes a volume from writables because
effectiveSize reached the limit, decay can later bring effectiveSize
back under the limit (e.g., when a burst of assigns didn't all result
in uploads). Without recovery the volume would stay non-writable until
vacuum or a ReadOnly flip.

UpdateVolumeSize now re-adds the volume to writables once all of the
following hold:

  * RecordAssign is what removed it (tracked via fullSince timestamp)
  * at least capacityRecoveryDelay has elapsed since the removal (30s)
    — this prevents bouncing during a steady stream of assigns near
    the limit
  * effectiveSize has decayed below the crowded threshold (90% of limit)
  * reportedSize is under the limit (actual disk is not over)
  * standard EnsureCorrectWritables preconditions: enough copies, all
    copies writable, not oversized

The caller (SyncDataNodeRegistration) re-increments activeVolumeCount
symmetrically with the decrement done on eager removal.

* review: release VolumeLayout lock before UpAdjustDiskUsageDelta

adjustActiveVolumeCount held vl.accessLock across the tree-climbing
UpAdjustDiskUsageDelta walk. That walk takes per-level DiskUsages
locks and could be re-entered from other call paths that hold a
node-level lock and then acquire vl.accessLock. Copy the node list
under the VolumeLayout lock and release it before the tree walk to
eliminate the lock-ordering hazard.
2026-04-16 12:50:30 -07:00
Chris Lu 10b0bdce02 feat: pass expected_data_size from clients for size-aware assignment (#9032)
* feat: pass expected_data_size from clients for size-aware assignment

Add expected_data_size field to AssignRequest (master proto) and
AssignVolumeRequest (filer proto) so clients can hint how large the
data will be. The master uses this instead of the 1MB default when
tracking pending volume sizes for weighted assignment.

- Add expected_data_size to master.proto AssignRequest
- Add expected_data_size to filer.proto AssignVolumeRequest
- Wire through filer AssignVolume handler
- Wire through HTTP submit handler (uses actual upload size)
- Add ExpectedDataSize to VolumeAssignRequest in operation package
- Topology.PickForWrite accepts optional expectedDataSize parameter

* fix: guard integer conversions in expected_data_size path

- common.go: clamp OriginalDataSize to non-negative before uint64 cast
- topology.go: cap expectedDataSize at math.MaxInt64 before int64 cast

* fix: parse dataSize hint in HTTP /dir/assign and test non-zero expectedDataSize

- HTTP /dir/assign now parses optional "dataSize" query parameter
  and passes it to PickForWrite instead of hardcoded 0
- Add test assertion for PickForWrite with non-zero expectedDataSize
2026-04-11 11:30:47 -07:00
Chris Luandgemini-code-assist[bot] 7acebf11ea Master: volume assignment concurrency (#7159)
* volume assginment concurrency

* accurate tests

* ensure uniqness

* reserve atomically

* address comments

* atomic

* ReserveOneVolumeForReservation

* duplicated

* Update weed/topology/node.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update weed/topology/node.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* atomic counter

* dedup

* select the appropriate functions based on the useReservations flag

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-08-23 21:02:30 -07:00
chrislu d2be5822a1 refactoring 2025-06-16 22:25:22 -07:00
chrislu 35fd1e1c9a optimize memory usage for large number of volumes
1. unwrap the map to avoid extra map object creation
2. fix ec shard counting in UpdateEcShards
2024-10-10 10:00:30 -07:00
chrislu 55976ae04a avoid repeated calls to heavy-weighted viper 2024-04-18 09:09:45 -07:00
Konstantin Lebedev 9c1e0f5811 [master] grow volumes if no writable volumes in current dataCenter (#5434)
* grow volumes if no writable volumes in current dataCenter
https://github.com/seaweedfs/seaweedfs/issues/3886

* fix tests with volume grow

* automatic volume grow one volume

* add ErrorChunkAssign metrics
2024-03-29 00:38:27 -07:00
Konstantin Lebedev dc9568fc0d [master] add test for PickForWrite add metrics for volume layout (#5413) 2024-03-22 07:39:11 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Chris Lu 68775d29e3 fix tests 2021-02-16 10:51:03 -08:00
James Hartig eae3f27c80 Added treat_replication_as_minimums master toml option 2020-04-01 19:08:48 -04:00
Chris Lu 1ae83c2938 go fmt 2020-03-07 06:12:57 -08:00
Chris Lu 564629444b format 2020-03-07 06:12:20 -08:00
zhangsong 40f70481cd schedule new volume by free volume number of nodes 2020-03-07 21:41:42 +08:00
Chris Lu 45156cc2fe add test cases for replication=011 2020-02-20 01:21:11 -08:00
Chris Lu 09ca936c78 shell: add ec.decode command 2019-12-23 12:48:20 -08:00
Chris Lu e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu 0bd7ced7c2 fix test 2019-04-04 20:11:59 -07:00
Chris Lu 30f138a538 fix tests 2017-02-12 22:18:48 -08:00
Chris Lu 5ce6bbf076 directory structure change to work with glide
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
2016-06-02 18:09:14 -07:00