mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-16 08:45:22 +01:00
* topology: serialize quarantine records with their layout marks A quarantine publishes a (node, vid) record and marks the replica read-only at the layout level; release and heartbeat flag updates must observe that pair as one state. Store-then-mark let a concurrent release delete the record before the mark landed, stranding the surviving replica read-only; mark-then-store let a heartbeat clear the mark before the record was visible, returning a conflicting replica to the writable list. quarantineLock now covers every record<->mark pair: flagDuplicateVolumeId's quarantine, releaseResolvedQuarantine's undo, the isQuarantined-gated marks on registration, and the quarantine-folded read-only flag writes on heartbeat paths. Remote server pushes stay outside the lock; a push that lands after the record was released is undone, and a failed release push re-adds the record for a later retry. The release test now waits for in-flight async flags to settle instead of asserting on a single read. * topology: fold quarantine into RegisterVolume's flag rebuild RegisterVolume rewrites every replica's location read-only flag from what its node last reported, and reported flags deliberately exclude quarantine. Any registration of a shared vid -- a reconnected replica, a new copy, the self-heal and delta-heartbeat re-register paths -- therefore cleared the quarantine mark on every replica in the location, while the gate afterwards only re-marked the registering node. Two call sites had no gate at all. RegisterVolume now takes an isQuarantined predicate and evaluates it under accessLock for every replica it touches, so the mark is never dropped in the first place: a release unmarks after the rebuild, and a quarantine mark re-applies on the next rebuild, with no window between. * topology: keep quarantine when a vacuum commit re-adds a replica SetVolumeAvailable re-adds the replica into the location list; a re-added slot starts with a cleared read-only flag, so a quarantined replica came back flag-free and the next ensureCorrectWritables could return a conflicting volume to the writable list. Pass the quarantine state in and set the flag back under the same accessLock, with the caller holding quarantineLock so release cannot interleave between the check and the mark. * topology: merge quarantine records instead of overwriting them enforceQuarantine wrote back the state copy it loaded before its remote push, so a concurrent writer's enforced=true was lost: the server stayed read-only while the record said unenforced, and a later release skipped the writable push it owed. The write-back now reloads the record under quarantineLock and merges -- enforced wins from either side, reported is taken from the newer pre-enforcement observation only while no push has landed yet. All three compensating writable pushes had the same hole in reverse: a failed un-quarantine left the server read-only with no record to retry from, and release's restore blindly overwrote any record a fresh flag pass had created in between. They now go through restoreQuarantineRecord, which adds the record only when absent and otherwise sets enforced on the current one, preserving its reported flag. * topology: hold quarantineLock over registration's quarantine check-and-mark RegisterVolume reads the quarantine record and marks the replica read-only under accessLock, but the pair ran without quarantineLock. A release could push writable, watch the record get re-read, and then delete it: the replica stayed read-only with no record left to clear, and unchanged writable heartbeats skip SetReplicaReadOnlyFlag so the mark never repaired. Holding quarantineLock around the check-and-mark in all three production callers serializes it against the release clear-and-delete. * topology: serialize remote quarantine pushes behind a record recheck releaseResolvedQuarantine deletes the record under quarantineLock and sends the writable request after unlock. A conflict flagged while that request is in flight creates a fresh record and marks the replica read-only again, and the older writable request can still land last, leaving the server writable under an active quarantine. The reverse flap strands a stale read-only mark the same way. Route every QuarantineVolumeFn call through pushQuarantine, which holds quarantinePushLock across the remote call and re-checks the record under quarantineLock right before sending: a read-only push requires the pair to exist, a writable push requires it to be gone. A skipped push means the decision already moved on, so the final server state always follows the newest record. * topology: serialize quarantine pushes per replica, not globally A single quarantinePushLock held across the remote mark call let one slow volume server (up to the 10s client timeout) delay pushes for unrelated replicas. Keying the lock by (node, vid) keeps pushes to one replica ordered in record-decision order while pushes to others run in parallel. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: restore the layout read-only mark with a released record A failed writable push re-armed the quarantine record but left the replica marked writable in the layout, so PickForWrite could route to a replica the server still holds read-only. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: wait out async duplicate checks in the release-retry test TestFailedReleasePushRestoresQuarantineMark asserts layout marks around a release pass, but the two RegisterVolumeLayout calls each start a background flagDuplicateVolumeId that can still be mid-flight. A late mark could land after the final release and fail the writable assertion even though the code behaved correctly. Track the async passes in a WaitGroup and drain them before the explicit flag call so the rest of the test is deterministic. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: keep quarantine when a replica's layout key is unreadable * topology: keep a failed writable push's restore ordered under the push lock * topology: retry the quarantine release on every duplicate check * ci: isolate Java and S3 test servers on persistent runners * ci: drop the duplicated -ip flags in the lifecycle test server The bind flags were already on the command; the isolation pass added a second pair. * topology: retry a failed quarantine writable push on the next report * topology: retry pending quarantine releases off the leader pulse A failed writable push re-arms the quarantine record with a retry that only enforceQuarantine fires, and that runs only for volumes a heartbeat carries. Once the server has reported the replica read-only it drops out of the delta reports, and restoreQuarantineRecord keeps the stored flag matching the server's so the digests agree and no full list is requested - the pending release then never retries and the idle volume stays read-only forever. Run the same release pass from the leader's pulse loop so a pending retry no longer waits on a report. * topology: exercise the pulse-driven quarantine release retry --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
333 lines
10 KiB
Go
333 lines
10 KiB
Go
package topology
|
|
|
|
import (
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/storage/types"
|
|
)
|
|
|
|
// A volume removed by the heartbeat capacity path (not RecordAssign) must
|
|
// still recover once it shrinks — only works if SetVolumeCapacityFull stamps
|
|
// fullSince.
|
|
func TestSetVolumeCapacityFullStampsFullSinceAndRecovers(t *testing.T) {
|
|
layout := `
|
|
{
|
|
"dc1":{
|
|
"rack1":{
|
|
"server1":{
|
|
"volumes":[
|
|
{"id":1, "size":4000, "replication":"000"}
|
|
],
|
|
"limit":10
|
|
}
|
|
}
|
|
}
|
|
}
|
|
`
|
|
topo, vl := setupPickTest(t, layout, 10000)
|
|
|
|
initialActive := topo.diskUsages.usages[types.HardDriveType].activeVolumeCount
|
|
initialWritables, _ := vl.GetWritableVolumeCount()
|
|
if initialWritables != 1 {
|
|
t.Fatalf("expected 1 writable volume initially, got %d", initialWritables)
|
|
}
|
|
|
|
if !vl.SetVolumeCapacityFull(1) {
|
|
t.Fatalf("SetVolumeCapacityFull should report the volume was writable")
|
|
}
|
|
vl.AdjustActiveVolumeCountForFull(1)
|
|
if w, _ := vl.GetWritableVolumeCount(); w != 0 {
|
|
t.Fatalf("expected 0 writable after capacity full, got %d", w)
|
|
}
|
|
if vl.sizeTracking[1].fullSince.IsZero() {
|
|
t.Fatalf("expected SetVolumeCapacityFull to stamp fullSince")
|
|
}
|
|
|
|
// No recovery before capacityRecoveryDelay.
|
|
advanceSizeTrackingClock(vl, 1, 3*time.Second)
|
|
if vl.UpdateVolumeSize(1, 4000, 0, true) {
|
|
t.Fatalf("recovery should not fire before capacityRecoveryDelay")
|
|
}
|
|
|
|
// After the delay, a smaller size restores it.
|
|
advanceSizeTrackingClock(vl, 1, capacityRecoveryDelay)
|
|
if !vl.UpdateVolumeSize(1, 4000, 0, true) {
|
|
t.Fatalf("expected volume to recover to writable after shrinking")
|
|
}
|
|
vl.AdjustActiveVolumeCountAfterRecovery(1)
|
|
|
|
if w, _ := vl.GetWritableVolumeCount(); w != initialWritables {
|
|
t.Fatalf("expected %d writable after recovery, got %d", initialWritables, w)
|
|
}
|
|
if got := topo.diskUsages.usages[types.HardDriveType].activeVolumeCount; got != initialActive {
|
|
t.Fatalf("expected activeVolumeCount restored to %d, got %d", initialActive, got)
|
|
}
|
|
if !vl.sizeTracking[1].fullSince.IsZero() {
|
|
t.Fatalf("expected fullSince cleared after recovery")
|
|
}
|
|
}
|
|
|
|
func TestSetVolumeAvailableRestoresActiveCountForCapacityFullVolume(t *testing.T) {
|
|
layout := `
|
|
{
|
|
"dc1":{
|
|
"rack1":{
|
|
"server1":{
|
|
"volumes":[
|
|
{"id":1, "size":4000, "replication":"000"}
|
|
],
|
|
"limit":10
|
|
}
|
|
}
|
|
}
|
|
}
|
|
`
|
|
topo, vl := setupPickTest(t, layout, 10000)
|
|
|
|
initialActive := topo.diskUsages.usages[types.HardDriveType].activeVolumeCount
|
|
if !vl.SetVolumeCapacityFull(1) {
|
|
t.Fatalf("SetVolumeCapacityFull should report the volume was writable")
|
|
}
|
|
vl.AdjustActiveVolumeCountForFull(1)
|
|
if got := topo.diskUsages.usages[types.HardDriveType].activeVolumeCount; got != initialActive-1 {
|
|
t.Fatalf("expected activeVolumeCount decremented to %d, got %d", initialActive-1, got)
|
|
}
|
|
|
|
vl.accessLock.RLock()
|
|
dn := vl.vid2location[1].list[0]
|
|
vl.accessLock.RUnlock()
|
|
|
|
if !vl.SetVolumeAvailable(dn, 1, false, false, false) {
|
|
t.Fatalf("SetVolumeAvailable should report the volume became writable")
|
|
}
|
|
if got := topo.diskUsages.usages[types.HardDriveType].activeVolumeCount; got != initialActive {
|
|
t.Fatalf("expected SetVolumeAvailable to restore activeVolumeCount to %d, got %d", initialActive, got)
|
|
}
|
|
if !vl.sizeTracking[1].fullSince.IsZero() {
|
|
t.Fatalf("expected SetVolumeAvailable to clear fullSince after restoring activeVolumeCount")
|
|
}
|
|
|
|
advanceSizeTrackingClock(vl, 1, capacityRecoveryDelay+time.Second)
|
|
if vl.UpdateVolumeSize(1, 4000, 0, true) {
|
|
t.Fatalf("heartbeat recovery should not fire after SetVolumeAvailable already restored the volume")
|
|
}
|
|
if got := topo.diskUsages.usages[types.HardDriveType].activeVolumeCount; got != initialActive {
|
|
t.Fatalf("expected activeVolumeCount to remain %d, got %d", initialActive, got)
|
|
}
|
|
}
|
|
|
|
// A volume marked full by pending estimates takes no writes, so no heartbeat
|
|
// reports it again and only the periodic decay can bring it back.
|
|
func TestDecayQuietVolumeSizesRecoversPhantomFullVolume(t *testing.T) {
|
|
layout := `
|
|
{
|
|
"dc1":{
|
|
"rack1":{
|
|
"server1":{
|
|
"volumes":[
|
|
{"id":1, "size":4000, "replication":"000"}
|
|
],
|
|
"limit":10
|
|
}
|
|
}
|
|
}
|
|
}
|
|
`
|
|
topo, vl := setupPickTest(t, layout, 10000)
|
|
VolumeGrowStrategy.Threshold = 0.9
|
|
|
|
initialActive := topo.diskUsages.usages[types.HardDriveType].activeVolumeCount
|
|
if !vl.RecordAssign(1, 20000) {
|
|
t.Fatalf("expected RecordAssign to remove the volume from writable")
|
|
}
|
|
vl.AdjustActiveVolumeCountForFull(1)
|
|
if w, _ := vl.GetWritableVolumeCount(); w != 0 {
|
|
t.Fatalf("expected 0 writable after the pending estimate filled the volume, got %d", w)
|
|
}
|
|
|
|
recovered := false
|
|
for i := 0; i < 10 && !recovered; i++ {
|
|
advanceSizeTrackingClock(vl, 1, capacityRecoveryDelay+time.Second)
|
|
topo.DecayQuietVolumeSizes()
|
|
w, _ := vl.GetWritableVolumeCount()
|
|
recovered = w == 1
|
|
}
|
|
if !recovered {
|
|
t.Fatalf("the periodic decay never returned the volume to the writable list")
|
|
}
|
|
if got := topo.diskUsages.usages[types.HardDriveType].activeVolumeCount; got != initialActive {
|
|
t.Fatalf("expected activeVolumeCount restored to %d, got %d", initialActive, got)
|
|
}
|
|
if !vl.sizeTracking[1].fullSince.IsZero() {
|
|
t.Fatalf("expected fullSince cleared after recovery")
|
|
}
|
|
}
|
|
|
|
// The decay must not consume the replica-dedup window: a real report landing
|
|
// right behind it carries a size no volume server will send again, since only
|
|
// a volume whose content changed is reported at all.
|
|
func TestDecayQuietVolumeSizesKeepsTheNextHeartbeat(t *testing.T) {
|
|
layout := `
|
|
{
|
|
"dc1":{
|
|
"rack1":{
|
|
"server1":{
|
|
"volumes":[
|
|
{"id":1, "size":4000, "replication":"000"}
|
|
],
|
|
"limit":10
|
|
}
|
|
}
|
|
}
|
|
}
|
|
`
|
|
topo, vl := setupPickTest(t, layout, 10000)
|
|
|
|
vl.RecordAssign(1, 2000)
|
|
advanceSizeTrackingClock(vl, 1, 30*time.Second)
|
|
topo.DecayQuietVolumeSizes()
|
|
|
|
vl.UpdateVolumeSize(1, 6000, 7, true)
|
|
|
|
vl.accessLock.RLock()
|
|
defer vl.accessLock.RUnlock()
|
|
if got := vl.sizeTracking[1].reportedSize; got != 6000 {
|
|
t.Errorf("the heartbeat after a decay left reportedSize at %d, want the reported 6000", got)
|
|
}
|
|
if got := vl.sizeTracking[1].compactRevision; got != 7 {
|
|
t.Errorf("the heartbeat after a decay left compactRevision at %d, want the reported 7", got)
|
|
}
|
|
}
|
|
|
|
// The decay picks its volumes under a read lock and replays them under a write
|
|
// one. A compaction heartbeat landing in that gap must survive, so the replay
|
|
// takes the record as it stands rather than the size the pass set out with —
|
|
// the arguments below stand in for that older snapshot.
|
|
func TestDecayQuietVolumeSizesDoesNotRollBackAHeartbeat(t *testing.T) {
|
|
layout := `
|
|
{
|
|
"dc1":{
|
|
"rack1":{
|
|
"server1":{
|
|
"volumes":[
|
|
{"id":1, "size":9000, "replication":"000"}
|
|
],
|
|
"limit":10
|
|
}
|
|
}
|
|
}
|
|
}
|
|
`
|
|
_, vl := setupPickTest(t, layout, 10000)
|
|
|
|
vl.RecordAssign(1, 500)
|
|
// The heartbeat that wins the race: a compaction shrank the volume.
|
|
vl.UpdateVolumeSize(1, 2000, 3, true)
|
|
// Far enough behind that the dedup window no longer covers for the replay,
|
|
// which is the only case where reading the record under the lock is what
|
|
// saves the report.
|
|
advanceSizeTrackingClock(vl, 1, 3*time.Second)
|
|
|
|
vl.UpdateVolumeSize(1, 9000, 0, false)
|
|
|
|
vl.accessLock.RLock()
|
|
defer vl.accessLock.RUnlock()
|
|
if got := vl.sizeTracking[1].reportedSize; got != 2000 {
|
|
t.Errorf("the decay rolled reportedSize back to %d, want the compacted 2000", got)
|
|
}
|
|
if got := vl.sizeTracking[1].compactRevision; got != 3 {
|
|
t.Errorf("the decay rolled compactRevision back to %d, want the reported 3", got)
|
|
}
|
|
if got := vl.sizeTracking[1].effectiveSize; got > 2000 {
|
|
t.Errorf("effectiveSize %d outgrew the compacted size the heartbeat reported", got)
|
|
}
|
|
}
|
|
|
|
// The decay stands in for a report that never came, so a heartbeat arriving
|
|
// after the pass chose the volume takes its place rather than adding to it:
|
|
// halving twice in one cycle forgets pending bytes the volume has yet to write.
|
|
func TestDecayQuietVolumeSizesYieldsToAHeartbeatItRaced(t *testing.T) {
|
|
layout := `
|
|
{
|
|
"dc1":{
|
|
"rack1":{
|
|
"server1":{
|
|
"volumes":[
|
|
{"id":1, "size":4000, "replication":"000"}
|
|
],
|
|
"limit":10
|
|
}
|
|
}
|
|
}
|
|
}
|
|
`
|
|
_, vl := setupPickTest(t, layout, 10000)
|
|
|
|
vl.RecordAssign(1, 4000)
|
|
advanceSizeTrackingClock(vl, 1, 30*time.Second)
|
|
|
|
// The heartbeat wins the race and does the one decay this cycle owes.
|
|
vl.UpdateVolumeSize(1, 4000, 0, true)
|
|
vl.accessLock.RLock()
|
|
afterHeartbeat := vl.sizeTracking[1].effectiveSize
|
|
vl.accessLock.RUnlock()
|
|
if afterHeartbeat != 6000 {
|
|
t.Fatalf("the heartbeat left effectiveSize at %d, want 6000", afterHeartbeat)
|
|
}
|
|
|
|
vl.UpdateVolumeSize(1, 0, 0, false)
|
|
|
|
vl.accessLock.RLock()
|
|
defer vl.accessLock.RUnlock()
|
|
if got := vl.sizeTracking[1].effectiveSize; got != afterHeartbeat {
|
|
t.Errorf("the decay halved again to %d, want the heartbeat's %d left alone", got, afterHeartbeat)
|
|
}
|
|
}
|
|
|
|
// A volume the disk really did fill can never recover through the decay, so it
|
|
// must leave the candidate set instead of costing a write lock every pulse for
|
|
// the rest of its life.
|
|
func TestDecayQuietVolumeSizesSkipsAGenuinelyFullVolume(t *testing.T) {
|
|
layout := `
|
|
{
|
|
"dc1":{
|
|
"rack1":{
|
|
"server1":{
|
|
"volumes":[
|
|
{"id":1, "size":10000, "replication":"000"},
|
|
{"id":2, "size":4000, "replication":"000"}
|
|
],
|
|
"limit":10
|
|
}
|
|
}
|
|
}
|
|
}
|
|
`
|
|
_, vl := setupPickTest(t, layout, 10000)
|
|
|
|
// Volume 1 is full on disk; volume 2 only looks full to the estimate.
|
|
vl.SetVolumeCapacityFull(1)
|
|
vl.RecordAssign(2, 6000)
|
|
advanceSizeTrackingClock(vl, 1, 30*time.Second)
|
|
advanceSizeTrackingClock(vl, 2, 30*time.Second)
|
|
|
|
candidates := vl.quietDecayCandidates(10 * time.Second)
|
|
for _, vid := range candidates {
|
|
if vid == 1 {
|
|
t.Errorf("the decay keeps taking the write lock for volume 1, which the disk really did fill")
|
|
}
|
|
}
|
|
if len(candidates) != 1 || candidates[0] != 2 {
|
|
t.Errorf("candidates %v, want only the phantom-full volume 2", candidates)
|
|
}
|
|
|
|
vl.DecayQuietVolumeSizes(10 * time.Second)
|
|
|
|
vl.accessLock.RLock()
|
|
defer vl.accessLock.RUnlock()
|
|
if got := vl.sizeTracking[2].effectiveSize; got != 7000 {
|
|
t.Errorf("the phantom-full volume was left at %d, want it decayed to 7000", got)
|
|
}
|
|
}
|