* ecbalancer: cap total shards per rack in Plan
Plan caps data and parity per rack separately (ceil(data/racks) and
ceil(parity/racks)), so with 10+4 over 8 racks a rack can legally hold
2 data + 1 parity. When a rack is one disk, losing two such racks loses
6 of 14 shards and the volume can't be read.
The cross-rack phase now also caps each rack's TOTAL shards of a
volume, sized with Place's rackTotalCap: ceil(shards/racks) unless the
racks lack room, counting a rack's own shards of the volume as room
since Plan can move them.
- A rack above the cap sheds parity until it fits. Those shards may go
to a data-bearing rack, and when no rack is under the parity cap
they fall back to a rack under the total cap. #11438's non-overflow
candidates keep moving only to data-free racks.
- No cross-rack move lands on a rack at the cap.
- A rack above the cap triggers balancing regardless of the imbalance
threshold.
- The fallback applies only while the source rack is above the cap;
otherwise the next Plan moves the shard back.
- Options.RackTotalCapRaised reports volumes whose cap had to be raised
above the even share; the worker and the shell log it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* ecbalancer: finish the rack cap in one Plan, count SameRackCount room
Review follow-ups.
- The data pass can run out of destinations before the parity pass frees
slots elsewhere, which left a data-heavy rack above the cap after one
Plan (a one-shot shell balance stops there). The cross-rack phase now
repeats while a rack is above the cap and the last round moved
something. A shard moves at most once per plan, since each move runs
as its own task.
- planRackTotalCap bounds each node's room by what SameRackCount still
allows, as Place does. Counting the raw free slots sized the cap too
low, so RackTotalCapRaised missed volumes it should report.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* ec: let the encode's balance see a migrating volume's shards across disk-type buckets
Shard generation writes beside the source .dat, so a cross-tier encode
(source on hdd, -diskType=ssd) leaves the fresh shards in the source
disk-type bucket. The encode's internal balance ingested only the target
bucket, saw no shards, and planned no moves; the spread guard then
correctly aborted the encode (and before that guard existed, the shards
silently stayed clumped on the generation host in the wrong tier).
EcBalance now takes the encode batch as migratingVolumeIds and ingests
those volumes' shards from every bucket, while everything else keeps the
bucket filter so a plain ec.balance never drags deliberately tiered
shards onto another disk type. The in-memory model delete also becomes
bucket-agnostic: a node holds a given shard in exactly one bucket, and a
bucket-scoped delete missed cross-bucket moves in the dry-run model.
* volume: decode reads shard 0 from its resolved path, not the EC volume's base dir
On a multi-disk server a volume's shards can sit on several disks; the
store registers each shard with its own path and CollectEcShards resolves
them, but FindDatFileSize derived the .ec00 path from the EcVolume's base
directory. When shard 0 lived on a sibling disk, VolumeEcShardsToVolume
failed with 'open ...ec00: no such file or directory' and ec.decode
aborted.
* ec: decode re-copies shards the topology claims but the target does not hold
An interrupted earlier decode or balance can leave the master believing
the decode target holds a shard whose file never landed: the mount
registered but the partial copy was cleaned, or the file was swept. The
collect step took the topology's word for it, excluded the shard from
the copy set, and the decode failed with 'missing shard'. Probe the
target's live inventory (VolumeEcShardsInfo) and treat anything it
cannot serve as still-to-copy.
* ec: decode discovers shards across disk-type buckets
Shards sit wherever encode generation and balance left them: a
cross-tier encode leaves them in the source disk-type bucket, a partial
migration straddles buckets. ec.decode scoped its shard discovery to the
-diskType bucket and reported a decodable volume as having no shards at
all. Union across buckets, the way the encode's shard verification
already does.
* test: EC chaos lifecycle harness
Randomized, seeded sequences of the EC lifecycle against a live cluster
in the production-shaped layout: multiple data disks per server, a
separate -dir.idx directory so .ecx/.ecj sidecars are shared across
disks, and a tagged ssd tier. Operations cover encode (hdd and ssd
targets), balance, shard damage plus rebuild, decode, re-encode,
deletes, scrub, tier moves, crash-restarts, sidecar fault injections
(a data-dir .vif pushed into the shared idx dir; a stale-generation
shard planted beside a newer encode), and interruptions: a real weed
shell subprocess killed mid-encode, mid-decode, and mid-balance, with
the recovery re-run required to converge.
One invariant holds after every step: every stored byte reads back
identical and every deleted needle stays deleted. EC_CHAOS_SEED and
EC_CHAOS_STEPS make runs reproducible and scalable.
A known gap is tolerated and logged rather than fixed here: a shard
mounted on two disks of one node (orphan adoption after an interrupted
copy) is invisible to ec.balance's dedup and unaddressable by
ec.shard.unmount's shard@address form, so no cleanup path exists yet.
* test: fail payload-corruption checks on the test goroutine
t.Fatalf inside require.Eventually's condition runs on the poller's
goroutine, where Goexit kills only that goroutine and the corruption
message can be lost behind a generic timeout. Record the mismatch, end
the polling, and fail on the test goroutine. Also assert the full shard
count in the cross-bucket decode-discovery test.
* shell: move ErrorWaitGroup to weed/util
* shell: remove unused CandidateEcNode and EcRack types
* ec: extract EC orchestration logic from weed/shell into weed/ec
Move the EC node/topology model, balance engine, encode pipeline, decode
pipeline, and rebuild engine into a new weed/ec package so shell commands
and maintenance workers can share the logic. Shell commands keep flag
parsing and delegate through a small ec.Env (dial option, topology fetch,
volume locations, lock check). Tests move along with the code.
* shell: remove unused proportional-rebalance type stubs
* ec: move scrub, replication check, and shard unmount engines into weed/ec
* worker: share the EC generation-aware shard counter from weed/ec
* ec: gofmt
* shell: drop EC aliases with no remaining callers
* ec: guard a missing topology hook and nil disk entries in topology helpers
* ec: drop trailing newlines from decode error strings
* ec: re-check the shell lock before applying shard unmounts
* shell: trim -node entries in ec.scrub
