Github-Mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-15 19:10:48 +02:00
Code Issues Packages Projects Releases Wiki Activity

test/ec: fix EC interruption matrix slot exhaustion (#11295)

Browse Source 
* test/ec: fix EC interruption matrix slot exhaustion

The EC integration test cluster (test/erasure_coding/chaos_lifecycle_test.go)
configured each disk with -max 4 and the seedAndSpread spread loop fired
volume.grow -count 4 every 2 s with no per-server cap. Because the master
topology lags the volume.grow writes, the loop re-fired before the prior
grow was visible, over-filling disks to capacity. A full disk leaves zero
free EC shard slots (failing the cluster-wide capacity check with "no
free ec shard slots") and drops the source disk below the encode's
FreeVolumeCount >= 2 health check (failing with "no healthy replicas"),
which aborted ec.encode before any phase marker printed and made every
encode scenario in TestECInterruptionMatrix fail.

Three changes to the test cluster:

1. Raise -max from 4 to 8 per disk so the source disk always retains
   FreeVolumeCount >= 2 for ec.encode's 14-shard generation (2 volume-slot
   equivalents) even after the spread loop and multiple encodes.

2. Switch the spread loop from -count 4 to -count 1 so each grow lands
   exactly one volume on the volume server's least-loaded disk, giving
   deterministic cross-disk spreading instead of relying on a single
   multi-volume grow to fan out.

3. Cap grows per server at 4 so heartbeat lag cannot run away and
   over-fill disks before the master registers the prior grow.

4. Pass -minFreeSpace 0 so the test is not falsely gated by the physical
   disk's free-space percentage on the host running CI (the EC shard slot
   calculation separately enforces a 90 % disk-usage cap via
   balancer.DiskTooFullAfter, which already guards against an over-set
   maxVolumeCount on a physically full disk).

Verified locally by running TestECInterruptionMatrix twice (all encode,
decode, and balance scenarios pass, including the previously failing
encode@Deletingoriginalvolumes).

* test/ec: only count successful grows toward the spread cap

A failed volume.grow (e.g. a transient collectTopologyInfo or VolumeGrow
RPC error) would otherwise consume one of the four permitted attempts
without creating any volume, exhausting the retry budget and leaving the
loop to only poll until the Eventually timeout. Increment the per-server
counter only when commandGrow.Do returns nil.
This commit is contained in:
Chris Lu
2026-09-13 14:41:45 -07:00
committed by GitHub
parent bea10e269f
commit 5d8a463b3e
1 changed files with 31 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/erasure_coding/chaos_lifecycle_test.go
+31 -3
View File
@@ -726,6 +726,20 @@ func (r *chaosRun) seedAndSpread() {
require.GreaterOrEqual(r.t, len(r.volumes), 2, "seeding should produce at least two volumes")
time.Sleep(3 * time.Second)
// Cap the grows per server so heartbeat lag cannot run away: the spread
// check reads the master topology, which lags the volume.grow writes, so
// an uncapped loop re-fires every tick and fills every disk to capacity
// before the master registers the prior grow. A full disk has no free EC
// shard slots and the source disk drops below the encode's FreeVolumeCount
// >= 2 health check, failing ec.encode with "no healthy replicas" or "no
// free ec shard slots".
//
// Use -count 1 so each grow lands exactly one volume on the volume
// server's least-loaded disk (deterministic spreading), and cap the
// total grows per server well below the per-disk max so the source disk
// always retains FreeVolumeCount >= 2 for ec.encode's shard generation.
growsPerServer := make(map[string]int)
const maxGrowsPerServer = 4
require.Eventually(r.t, func() bool {
spread := nodeVolumeDiskCounts(r.t, r.env)
if len(spread) == chaosServerCount && allAtLeast(spread, 2) {
@@ -733,9 +747,16 @@ func (r *chaosRun) seedAndSpread() {
}
for i := 0; i < chaosServerCount; i++ {
server := "127.0.0.1:" + chaosVolumePort(i)
if spread[server] < 2 {
if spread[server] < 2 && growsPerServer[server] < maxGrowsPerServer {
out, gerr := captureCommandOutput(r.t, shell.Commands[findCommandIndex("volume.grow")],
[]string{"-collection", chaosCollection, "-dataNode", server, "-count", "4"}, r.env)
[]string{"-collection", chaosCollection, "-dataNode", server, "-count", "1"}, r.env)
// Only count successful grows toward the cap: a transient
// collectTopologyInfo or VolumeGrow RPC error would otherwise
// exhaust the retry budget without creating any volume, and
// the loop would then only poll until the Eventually timeout.
if gerr == nil {
growsPerServer[server]++
}
r.t.Logf("volume.grow on %s: err=%v output:\n%s", server, gerr, out)
}
}
@@ -1159,7 +1180,13 @@ func (c *chaosCluster) startVolumeServer(ctx context.Context, i int, logName str
return nil, err
}
diskDirs = append(diskDirs, dir)
maxVolumes = append(maxVolumes, "4")
// ec.encode generates 14 shards on the source disk (2 volume-slot
// equivalents) and refuses a source whose disk has FreeVolumeCount < 2,
// and the cluster-wide capacity check needs at least one free EC shard
// slot. seedAndSpread's spread loop grows 4 volumes per tick and can
// over-grow before the master's topology catches up, so leave enough
// headroom that a near-full disk never starves the encode.
maxVolumes = append(maxVolumes, "8")
if d == chaosDisksPerNode-1 {
diskTypes = append(diskTypes, "ssd")
} else {
@@ -1176,6 +1203,7 @@ func (c *chaosCluster) startVolumeServer(ctx context.Context, i int, logName str
"-dir.idx", idxDir,
"-disk", strings.Join(diskTypes, ","),
"-max", strings.Join(maxVolumes, ","),
"-minFreeSpace", "0",
"-master", chaosMasterAddr,
"-ip", "127.0.0.1",
"-dataCenter", "dc1",