mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 15:41:15 +02:00
Chris LuandDevin
cc281dabc9
* master: keep new volumes and writes off servers in maintenance mode The master recorded a volume server's maintenance flag from the heartbeat but never consulted it. A server in maintenance (#7977) is being drained, yet the master kept creating volumes on it whenever it had free slots and kept handing out its volumes for writes. Nothing on the volume server blocks plain HTTP uploads either, so "read-only mode" was only a name. Volume growth: a data node in maintenance mode reports zero free slots through AvailableSpaceFor, which takes it out of every candidate list, feasibility count and capacity reservation. Its slots still roll up into its rack and data center, so the random offset drawn from those totals for an other-rack or other-DC replica could land in space the walk then skips and fail with "No free volume slot found!" while siblings had room; the walk now folds the offset into the space that is actually eligible. This also covers the pre-existing case of an over-committed sibling. Assignment: a replica on a server in maintenance mode is treated like a read-only replica in isAllWritable, so its volume leaves the writable list and returns when the flag clears. Topology.SetDataNodeMaintenanceMode re-evaluates the node's volumes on every change, since heartbeats are digest-based and a full volume list may not follow for a long time. Reads and lookups are untouched. The flag moves to an atomic so the assign and growth paths can read it without the node lock. Heartbeat: the Go volume server sent its state only when it changed, so a master elected while a server sat in maintenance never learned about it. The state now rides along on every heartbeat, as the Rust server already does; the master's compare is an atomic swap, and only a change does work. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: hold maintenance mode through vacuum commit and mark-writable SetVolumeAvailable and SetVolumeWritable put a volume back on the writable list on the replica count alone. A vacuum that started before the server entered maintenance, or a vacuum worker's mark-writable arriving after it, handed the volume back to assignment with a replica on the draining server. Heartbeats carry only changed volumes, so nothing re-evaluated it until the volume itself changed. Apply isAllWritable on both paths, the same test EnsureCorrectWritables uses. Also pin that re-evaluating a volume a concurrent disconnect already removed from its layout is a no-op. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: record a server's read-only notification on its node before judging the volume A volume server notifies the master the moment it flips a volume between read-only and writable, ahead of the heartbeat that repeats the flag. The layout only set its per-location flag, so isAllWritable, which reads the node's heartbeat copy, still saw the old value: a mark-writable was withheld until the next heartbeat, and a re-evaluation landing between a mark-readonly and its heartbeat put the volume back on the writable list. Record the flag on the node's volume first. AddOrUpdateVolume keeps the digest and the active volume count in step, so the heartbeat that follows finds nothing to change. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: a read-only mark does not confirm a provisional volume DataNode.SetVolumeReadOnly went through Disk.AddOrUpdateVolume, which treats its input as a server report and so ended the grace period that keeps a just-grown volume safe from a full report collected before the grow. A volume marked read-only before its first report could then be removed by that stale report. Give Disk a SetVolumeReadOnly that flips the flag and keeps the digest and active volume count in step without touching volumeAddedAt. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
69 lines
2.5 KiB
Go
69 lines
2.5 KiB
Go
package topology
|
|
|
|
import (
|
|
"sync/atomic"
|
|
"testing"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/storage"
|
|
"github.com/seaweedfs/seaweedfs/weed/storage/types"
|
|
)
|
|
|
|
// A provisional (grow-time) registration carries no disk id; when it lands
|
|
// after the server's own report -- the server pushes its report during the
|
|
// AllocateVolume RPC, so either order happens -- it must not erase the disk id
|
|
// the report recorded.
|
|
func TestProvisionalUpdateKeepsReportedDiskId(t *testing.T) {
|
|
disk := NewDisk(types.HardDriveType.String())
|
|
|
|
disk.AddOrUpdateVolume(storage.VolumeInfo{Id: 1, DiskId: 2})
|
|
disk.AddProvisionalVolume(storage.VolumeInfo{Id: 1})
|
|
|
|
v, err := disk.GetVolumesById(1)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if v.DiskId != 2 {
|
|
t.Fatalf("DiskId = %d, want 2 (provisional update clobbered the reported value)", v.DiskId)
|
|
}
|
|
|
|
// A later server report naming a different disk still wins.
|
|
disk.AddOrUpdateVolume(storage.VolumeInfo{Id: 1, DiskId: 1})
|
|
if v, _ = disk.GetVolumesById(1); v.DiskId != 1 {
|
|
t.Fatalf("DiskId = %d, want 1 (a real report must override)", v.DiskId)
|
|
}
|
|
}
|
|
|
|
// A read-only mark from the server is not a volume report: it must not end the
|
|
// grace period that keeps a just-grown volume safe from a full report collected
|
|
// before the grow, while still keeping the digest and active count in step.
|
|
func TestSetVolumeReadOnlyKeepsProvisionalProtection(t *testing.T) {
|
|
dn := NewDataNode("dn1")
|
|
vi := storage.VolumeInfo{Id: 1, DiskType: types.HardDriveType.String()}
|
|
dn.AddProvisionalVolume(vi)
|
|
|
|
dn.SetVolumeReadOnly(1, true)
|
|
|
|
readOnly := vi
|
|
readOnly.ReadOnly = true
|
|
if got, want := dn.VolumeDigest(), readOnly.ReportHash(); got != want {
|
|
t.Errorf("digest = %x, want %x, the hash of the volume as the server will report it", got, want)
|
|
}
|
|
if got := atomic.LoadInt64(&dn.GetDiskUsages().getOrCreateDisk(types.HardDriveType).activeVolumeCount); got != 0 {
|
|
t.Errorf("activeVolumeCount = %d, want 0 after the read-only mark", got)
|
|
}
|
|
|
|
// the stale report, collected before the grow, does not name the volume
|
|
if _, deleted, _ := dn.UpdateVolumes(nil); len(deleted) != 0 {
|
|
t.Fatalf("a report that raced the grow removed the volume: %v", deleted)
|
|
}
|
|
if v, err := dn.GetVolumesById(1); err != nil || !v.ReadOnly {
|
|
t.Fatalf("GetVolumesById = %+v, %v; want the volume kept, read-only", v, err)
|
|
}
|
|
|
|
// the server's own report confirms it, and from then on absence counts
|
|
dn.UpdateVolumes([]storage.VolumeInfo{readOnly})
|
|
if _, deleted, _ := dn.UpdateVolumes(nil); len(deleted) != 1 {
|
|
t.Fatalf("a report after confirmation should remove the volume, got %v", deleted)
|
|
}
|
|
}
|