Files
seaweedfs/weed/topology/volume_layout_capacity_recovery_test.go
T
Chris LuandDevin 3e5aa8c3b3 topology: mark replica quarantined before recording the pair (#11983)
* topology: serialize quarantine records with their layout marks

A quarantine publishes a (node, vid) record and marks the replica read-only at the layout level; release and heartbeat flag updates must observe that pair as one state. Store-then-mark let a concurrent release delete the record before the mark landed, stranding the surviving replica read-only; mark-then-store let a heartbeat clear the mark before the record was visible, returning a conflicting replica to the writable list.

quarantineLock now covers every record<->mark pair: flagDuplicateVolumeId's quarantine, releaseResolvedQuarantine's undo, the isQuarantined-gated marks on registration, and the quarantine-folded read-only flag writes on heartbeat paths. Remote server pushes stay outside the lock; a push that lands after the record was released is undone, and a failed release push re-adds the record for a later retry.

The release test now waits for in-flight async flags to settle instead of asserting on a single read.

* topology: fold quarantine into RegisterVolume's flag rebuild

RegisterVolume rewrites every replica's location read-only flag from what
its node last reported, and reported flags deliberately exclude quarantine.
Any registration of a shared vid -- a reconnected replica, a new copy, the
self-heal and delta-heartbeat re-register paths -- therefore cleared the
quarantine mark on every replica in the location, while the gate afterwards
only re-marked the registering node. Two call sites had no gate at all.

RegisterVolume now takes an isQuarantined predicate and evaluates it under
accessLock for every replica it touches, so the mark is never dropped in
the first place: a release unmarks after the rebuild, and a quarantine
mark re-applies on the next rebuild, with no window between.

* topology: keep quarantine when a vacuum commit re-adds a replica

SetVolumeAvailable re-adds the replica into the location list; a re-added
slot starts with a cleared read-only flag, so a quarantined replica came
back flag-free and the next ensureCorrectWritables could return a
conflicting volume to the writable list. Pass the quarantine state in and
set the flag back under the same accessLock, with the caller holding
quarantineLock so release cannot interleave between the check and the mark.

* topology: merge quarantine records instead of overwriting them

enforceQuarantine wrote back the state copy it loaded before its remote
push, so a concurrent writer's enforced=true was lost: the server stayed
read-only while the record said unenforced, and a later release skipped
the writable push it owed. The write-back now reloads the record under
quarantineLock and merges -- enforced wins from either side, reported is
taken from the newer pre-enforcement observation only while no push has
landed yet.

All three compensating writable pushes had the same hole in reverse: a
failed un-quarantine left the server read-only with no record to retry
from, and release's restore blindly overwrote any record a fresh flag
pass had created in between. They now go through restoreQuarantineRecord,
which adds the record only when absent and otherwise sets enforced on
the current one, preserving its reported flag.

* topology: hold quarantineLock over registration's quarantine check-and-mark

RegisterVolume reads the quarantine record and marks the replica
read-only under accessLock, but the pair ran without quarantineLock.
A release could push writable, watch the record get re-read, and then
delete it: the replica stayed read-only with no record left to clear,
and unchanged writable heartbeats skip SetReplicaReadOnlyFlag so the
mark never repaired. Holding quarantineLock around the check-and-mark
in all three production callers serializes it against the release
clear-and-delete.

* topology: serialize remote quarantine pushes behind a record recheck

releaseResolvedQuarantine deletes the record under quarantineLock and
sends the writable request after unlock. A conflict flagged while that
request is in flight creates a fresh record and marks the replica
read-only again, and the older writable request can still land last,
leaving the server writable under an active quarantine. The reverse
flap strands a stale read-only mark the same way.

Route every QuarantineVolumeFn call through pushQuarantine, which
holds quarantinePushLock across the remote call and re-checks the
record under quarantineLock right before sending: a read-only push
requires the pair to exist, a writable push requires it to be gone.
A skipped push means the decision already moved on, so the final
server state always follows the newest record.

* topology: serialize quarantine pushes per replica, not globally

A single quarantinePushLock held across the remote mark call let one slow
volume server (up to the 10s client timeout) delay pushes for unrelated
replicas. Keying the lock by (node, vid) keeps pushes to one replica
ordered in record-decision order while pushes to others run in parallel.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: restore the layout read-only mark with a released record

A failed writable push re-armed the quarantine record but left the
replica marked writable in the layout, so PickForWrite could route to a
replica the server still holds read-only.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: wait out async duplicate checks in the release-retry test

TestFailedReleasePushRestoresQuarantineMark asserts layout marks around a
release pass, but the two RegisterVolumeLayout calls each start a
background flagDuplicateVolumeId that can still be mid-flight. A late
mark could land after the final release and fail the writable assertion
even though the code behaved correctly. Track the async passes in a
WaitGroup and drain them before the explicit flag call so the rest of
the test is deterministic.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: keep quarantine when a replica's layout key is unreadable

* topology: keep a failed writable push's restore ordered under the push lock

* topology: retry the quarantine release on every duplicate check

* ci: isolate Java and S3 test servers on persistent runners

* ci: drop the duplicated -ip flags in the lifecycle test server

The bind flags were already on the command; the isolation pass added a
second pair.

* topology: retry a failed quarantine writable push on the next report

* topology: retry pending quarantine releases off the leader pulse

A failed writable push re-arms the quarantine record with a retry that only
enforceQuarantine fires, and that runs only for volumes a heartbeat carries.
Once the server has reported the replica read-only it drops out of the delta
reports, and restoreQuarantineRecord keeps the stored flag matching the
server's so the digests agree and no full list is requested - the pending
release then never retries and the idle volume stays read-only forever. Run
the same release pass from the leader's pulse loop so a pending retry no
longer waits on a report.

* topology: exercise the pulse-driven quarantine release retry

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:00:45 -07:00

333 lines
10 KiB
Go

package topology
import (
"testing"
"time"
"github.com/seaweedfs/seaweedfs/weed/storage/types"
)
// A volume removed by the heartbeat capacity path (not RecordAssign) must
// still recover once it shrinks — only works if SetVolumeCapacityFull stamps
// fullSince.
func TestSetVolumeCapacityFullStampsFullSinceAndRecovers(t *testing.T) {
layout := `
{
"dc1":{
"rack1":{
"server1":{
"volumes":[
{"id":1, "size":4000, "replication":"000"}
],
"limit":10
}
}
}
}
`
topo, vl := setupPickTest(t, layout, 10000)
initialActive := topo.diskUsages.usages[types.HardDriveType].activeVolumeCount
initialWritables, _ := vl.GetWritableVolumeCount()
if initialWritables != 1 {
t.Fatalf("expected 1 writable volume initially, got %d", initialWritables)
}
if !vl.SetVolumeCapacityFull(1) {
t.Fatalf("SetVolumeCapacityFull should report the volume was writable")
}
vl.AdjustActiveVolumeCountForFull(1)
if w, _ := vl.GetWritableVolumeCount(); w != 0 {
t.Fatalf("expected 0 writable after capacity full, got %d", w)
}
if vl.sizeTracking[1].fullSince.IsZero() {
t.Fatalf("expected SetVolumeCapacityFull to stamp fullSince")
}
// No recovery before capacityRecoveryDelay.
advanceSizeTrackingClock(vl, 1, 3*time.Second)
if vl.UpdateVolumeSize(1, 4000, 0, true) {
t.Fatalf("recovery should not fire before capacityRecoveryDelay")
}
// After the delay, a smaller size restores it.
advanceSizeTrackingClock(vl, 1, capacityRecoveryDelay)
if !vl.UpdateVolumeSize(1, 4000, 0, true) {
t.Fatalf("expected volume to recover to writable after shrinking")
}
vl.AdjustActiveVolumeCountAfterRecovery(1)
if w, _ := vl.GetWritableVolumeCount(); w != initialWritables {
t.Fatalf("expected %d writable after recovery, got %d", initialWritables, w)
}
if got := topo.diskUsages.usages[types.HardDriveType].activeVolumeCount; got != initialActive {
t.Fatalf("expected activeVolumeCount restored to %d, got %d", initialActive, got)
}
if !vl.sizeTracking[1].fullSince.IsZero() {
t.Fatalf("expected fullSince cleared after recovery")
}
}
func TestSetVolumeAvailableRestoresActiveCountForCapacityFullVolume(t *testing.T) {
layout := `
{
"dc1":{
"rack1":{
"server1":{
"volumes":[
{"id":1, "size":4000, "replication":"000"}
],
"limit":10
}
}
}
}
`
topo, vl := setupPickTest(t, layout, 10000)
initialActive := topo.diskUsages.usages[types.HardDriveType].activeVolumeCount
if !vl.SetVolumeCapacityFull(1) {
t.Fatalf("SetVolumeCapacityFull should report the volume was writable")
}
vl.AdjustActiveVolumeCountForFull(1)
if got := topo.diskUsages.usages[types.HardDriveType].activeVolumeCount; got != initialActive-1 {
t.Fatalf("expected activeVolumeCount decremented to %d, got %d", initialActive-1, got)
}
vl.accessLock.RLock()
dn := vl.vid2location[1].list[0]
vl.accessLock.RUnlock()
if !vl.SetVolumeAvailable(dn, 1, false, false, false) {
t.Fatalf("SetVolumeAvailable should report the volume became writable")
}
if got := topo.diskUsages.usages[types.HardDriveType].activeVolumeCount; got != initialActive {
t.Fatalf("expected SetVolumeAvailable to restore activeVolumeCount to %d, got %d", initialActive, got)
}
if !vl.sizeTracking[1].fullSince.IsZero() {
t.Fatalf("expected SetVolumeAvailable to clear fullSince after restoring activeVolumeCount")
}
advanceSizeTrackingClock(vl, 1, capacityRecoveryDelay+time.Second)
if vl.UpdateVolumeSize(1, 4000, 0, true) {
t.Fatalf("heartbeat recovery should not fire after SetVolumeAvailable already restored the volume")
}
if got := topo.diskUsages.usages[types.HardDriveType].activeVolumeCount; got != initialActive {
t.Fatalf("expected activeVolumeCount to remain %d, got %d", initialActive, got)
}
}
// A volume marked full by pending estimates takes no writes, so no heartbeat
// reports it again and only the periodic decay can bring it back.
func TestDecayQuietVolumeSizesRecoversPhantomFullVolume(t *testing.T) {
layout := `
{
"dc1":{
"rack1":{
"server1":{
"volumes":[
{"id":1, "size":4000, "replication":"000"}
],
"limit":10
}
}
}
}
`
topo, vl := setupPickTest(t, layout, 10000)
VolumeGrowStrategy.Threshold = 0.9
initialActive := topo.diskUsages.usages[types.HardDriveType].activeVolumeCount
if !vl.RecordAssign(1, 20000) {
t.Fatalf("expected RecordAssign to remove the volume from writable")
}
vl.AdjustActiveVolumeCountForFull(1)
if w, _ := vl.GetWritableVolumeCount(); w != 0 {
t.Fatalf("expected 0 writable after the pending estimate filled the volume, got %d", w)
}
recovered := false
for i := 0; i < 10 && !recovered; i++ {
advanceSizeTrackingClock(vl, 1, capacityRecoveryDelay+time.Second)
topo.DecayQuietVolumeSizes()
w, _ := vl.GetWritableVolumeCount()
recovered = w == 1
}
if !recovered {
t.Fatalf("the periodic decay never returned the volume to the writable list")
}
if got := topo.diskUsages.usages[types.HardDriveType].activeVolumeCount; got != initialActive {
t.Fatalf("expected activeVolumeCount restored to %d, got %d", initialActive, got)
}
if !vl.sizeTracking[1].fullSince.IsZero() {
t.Fatalf("expected fullSince cleared after recovery")
}
}
// The decay must not consume the replica-dedup window: a real report landing
// right behind it carries a size no volume server will send again, since only
// a volume whose content changed is reported at all.
func TestDecayQuietVolumeSizesKeepsTheNextHeartbeat(t *testing.T) {
layout := `
{
"dc1":{
"rack1":{
"server1":{
"volumes":[
{"id":1, "size":4000, "replication":"000"}
],
"limit":10
}
}
}
}
`
topo, vl := setupPickTest(t, layout, 10000)
vl.RecordAssign(1, 2000)
advanceSizeTrackingClock(vl, 1, 30*time.Second)
topo.DecayQuietVolumeSizes()
vl.UpdateVolumeSize(1, 6000, 7, true)
vl.accessLock.RLock()
defer vl.accessLock.RUnlock()
if got := vl.sizeTracking[1].reportedSize; got != 6000 {
t.Errorf("the heartbeat after a decay left reportedSize at %d, want the reported 6000", got)
}
if got := vl.sizeTracking[1].compactRevision; got != 7 {
t.Errorf("the heartbeat after a decay left compactRevision at %d, want the reported 7", got)
}
}
// The decay picks its volumes under a read lock and replays them under a write
// one. A compaction heartbeat landing in that gap must survive, so the replay
// takes the record as it stands rather than the size the pass set out with —
// the arguments below stand in for that older snapshot.
func TestDecayQuietVolumeSizesDoesNotRollBackAHeartbeat(t *testing.T) {
layout := `
{
"dc1":{
"rack1":{
"server1":{
"volumes":[
{"id":1, "size":9000, "replication":"000"}
],
"limit":10
}
}
}
}
`
_, vl := setupPickTest(t, layout, 10000)
vl.RecordAssign(1, 500)
// The heartbeat that wins the race: a compaction shrank the volume.
vl.UpdateVolumeSize(1, 2000, 3, true)
// Far enough behind that the dedup window no longer covers for the replay,
// which is the only case where reading the record under the lock is what
// saves the report.
advanceSizeTrackingClock(vl, 1, 3*time.Second)
vl.UpdateVolumeSize(1, 9000, 0, false)
vl.accessLock.RLock()
defer vl.accessLock.RUnlock()
if got := vl.sizeTracking[1].reportedSize; got != 2000 {
t.Errorf("the decay rolled reportedSize back to %d, want the compacted 2000", got)
}
if got := vl.sizeTracking[1].compactRevision; got != 3 {
t.Errorf("the decay rolled compactRevision back to %d, want the reported 3", got)
}
if got := vl.sizeTracking[1].effectiveSize; got > 2000 {
t.Errorf("effectiveSize %d outgrew the compacted size the heartbeat reported", got)
}
}
// The decay stands in for a report that never came, so a heartbeat arriving
// after the pass chose the volume takes its place rather than adding to it:
// halving twice in one cycle forgets pending bytes the volume has yet to write.
func TestDecayQuietVolumeSizesYieldsToAHeartbeatItRaced(t *testing.T) {
layout := `
{
"dc1":{
"rack1":{
"server1":{
"volumes":[
{"id":1, "size":4000, "replication":"000"}
],
"limit":10
}
}
}
}
`
_, vl := setupPickTest(t, layout, 10000)
vl.RecordAssign(1, 4000)
advanceSizeTrackingClock(vl, 1, 30*time.Second)
// The heartbeat wins the race and does the one decay this cycle owes.
vl.UpdateVolumeSize(1, 4000, 0, true)
vl.accessLock.RLock()
afterHeartbeat := vl.sizeTracking[1].effectiveSize
vl.accessLock.RUnlock()
if afterHeartbeat != 6000 {
t.Fatalf("the heartbeat left effectiveSize at %d, want 6000", afterHeartbeat)
}
vl.UpdateVolumeSize(1, 0, 0, false)
vl.accessLock.RLock()
defer vl.accessLock.RUnlock()
if got := vl.sizeTracking[1].effectiveSize; got != afterHeartbeat {
t.Errorf("the decay halved again to %d, want the heartbeat's %d left alone", got, afterHeartbeat)
}
}
// A volume the disk really did fill can never recover through the decay, so it
// must leave the candidate set instead of costing a write lock every pulse for
// the rest of its life.
func TestDecayQuietVolumeSizesSkipsAGenuinelyFullVolume(t *testing.T) {
layout := `
{
"dc1":{
"rack1":{
"server1":{
"volumes":[
{"id":1, "size":10000, "replication":"000"},
{"id":2, "size":4000, "replication":"000"}
],
"limit":10
}
}
}
}
`
_, vl := setupPickTest(t, layout, 10000)
// Volume 1 is full on disk; volume 2 only looks full to the estimate.
vl.SetVolumeCapacityFull(1)
vl.RecordAssign(2, 6000)
advanceSizeTrackingClock(vl, 1, 30*time.Second)
advanceSizeTrackingClock(vl, 2, 30*time.Second)
candidates := vl.quietDecayCandidates(10 * time.Second)
for _, vid := range candidates {
if vid == 1 {
t.Errorf("the decay keeps taking the write lock for volume 1, which the disk really did fill")
}
}
if len(candidates) != 1 || candidates[0] != 2 {
t.Errorf("candidates %v, want only the phantom-full volume 2", candidates)
}
vl.DecayQuietVolumeSizes(10 * time.Second)
vl.accessLock.RLock()
defer vl.accessLock.RUnlock()
if got := vl.sizeTracking[2].effectiveSize; got != 7000 {
t.Errorf("the phantom-full volume was left at %d, want it decayed to 7000", got)
}
}