Files
seaweedfs/weed/storage
646549eee1 volume growth: spread writable volumes of a layout over servers and disks (#12749)
* volume growth: spread writable volumes of a layout over servers and disks

Volume growth picked every server, and the volume server picked every disk,
independently: the server weighted by free slots, the disk by the lowest
total volume count. A grow request therefore stacks several writable volumes
of one collection on some disks while most disks get none (a Poisson spread:
with 160 replicas on 479 disks about 20 disks get two or three), and their
writes share those disks.

Data centers and racks are still picked weighted by free slots. Inside a
rack the server is now one of those holding the fewest writable volumes of
the layout per disk, weighted by free slots among them; this covers the main
server, same-rack replicas and replicas walked into other racks and data
centers. On the volume server the disk is the one with the fewest writable
volumes of the collection, then the lowest volume count as before.

A grow request registers each volume before placing the next, so a request
smaller than the number of disks lands on distinct disks.

Fixes #12746

* volume growth: scan the layout load once per grow request

Placing each volume rescanned all writable volumes of the layout under its
read lock. RecordAssign takes that lock exclusively on every assign, so with
thousands of writables (0.8 ms for 10000) every volume of a grow request
stalled assigns once more.

GrowByCountAndType now scans once and findAndGrow adds every volume it
creates to the counts. Per volume, placement costs about 130-160 us on 480
servers whatever the number of writables, against 120 us before the spread
change; the single scan is 0.17 ms for 2000 writables and 0.8 ms for 10000.

* volume growth: address review of the spread placement

- Count only disks that report volume slots in a server's load. An
  unavailable disk, or one low on space with no volumes, is still listed in
  DiskTags with no slots; counting it understated the load of a server with
  failed disks and piled volumes onto its remaining ones.
- On the volume server, judge a volume full by its .dat size, which the
  master compares with the limit, not by ContentSize.
- A server that refuses an allocation ranks last for the rest of the grow
  request, and the request retries the volume up to 3 times. The least
  loaded server is picked first deterministically, so without this one server
  that cannot fit a preallocated volume stopped the layout from growing.
- Scan the layout load after warmup, so a grow request issued right after a
  leader change sees the writable volumes the returning servers report.

* volume growth: keep the request-local placement load ahead of the request fields

The JSON request fields stay together at the end of VolumeGrowOption, and
the comment now says the load lives for the whole grow request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): place a new volume on the disk with the fewest writable volumes of its collection

Store::add_volume picked the disk with the most free slots, so the Rust
volume server ignored the collection spread rule the Go server now applies.
find_free_location ranks disks by the writable volumes of the collection
first (not read-only, .dat below the volume size limit), then by free slots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume growth: leave a refusing server out of every placement tier

A server that refused an allocation only ranked last among the servers of
its rack, so the weighted data center and rack picks could still choose its
rack on every retry while other racks had room. It now offers no free slots
for the rest of the grow request, which takes its slots out of the rack and
data center weights too. If only refusing servers are left, they are tried
again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume growth: require the full same-rack count only in the main rack

The data center check asked every rack for SameRackCount+1 free servers,
but only the main rack holds the same-rack copies; each other rack takes one.
With a refused server left out of its rack, the check rejected a feasible
placement, the retry put the refused server back, and it was picked again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:51:13 -08:00
..