Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-10-11 08:47:46 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
cd7c8ae784ce0394dfd19f8f4d28cd41d853e8cd
seaweedfs/weed/worker/tasks/balance/replica_placement_test.go
T
Chris Lu f0d2a0d417 Treat co-located volume servers as one fault domain when balancing and allocating (#9854) 
* admin/topology: carry the volume server address on DiskInfo

The planning DiskInfo exposed only the node id, which can be an opaque label rather than ip:port. Record the address too so callers can resolve the physical machine a disk sits on.

* ec.balance: spread a volume's shards across machines, not just nodes

Volume servers sharing a host are one fault domain, but the within-rack spread treated them as independent nodes, so one box could end up holding more shards of a volume than EC can afford to lose. Add a machine (host) tier between rack and node: the within-rack pass spreads each volume across machines, and the global load phase no longer re-concentrates a volume onto a machine it already sits on. Host defaults to the node id, so clusters with one server per host are unchanged.

* ec placement: prefer machines holding fewer of a volume's shards

EC allocation and repair picked the least-loaded node in a rack with no regard for which physical machine it sits on, so a volume's shards could pile onto several servers of one box. Rank candidate nodes by their machine's shard count first, then the node's own. The machine is derived from the volume server address carried on DiskInfo, falling back to the node id, matching how the balancer resolves it.

* volume.balance: don't move a replica onto a machine already holding one

isGoodMove only rejected a move onto the same data node, so two replicas could land on two volume servers of one box and a single machine failure would lose both. Reject a target whose host already holds another replica of the volume. Best-effort: balancing simply skips and tries the next target.

* volume allocation: spread same-rack replicas across machines

PickNodesByWeight filled the same-rack replica picks by weight alone, so replicas could co-locate on one box. Prefer candidates on not-yet-used hosts, falling back when too few distinct machines exist. Data-center and rack tiers have no host, so their ordering is unchanged.

* ec.balance: harden machine spread against re-concentration and capped machines

Two cases where the machine-aware spread could still leave a volume badly placed:

- The global load phase could move a shard of a volume onto a machine that
  already held it, raising that machine's count and undoing the within-rack
  spread (a 4/4/3/3 layout could become 3/5/3/3, past parity for 10+4). Limit
  the load-only fallback to same-machine moves, which leave a machine's count
  unchanged; cross-machine concentration is no longer allowed for load alone.

- The within-rack spread chose a destination machine by free slots alone, so if
  that machine's only nodes were already at the SameRackCount cap it skipped the
  move instead of trying another machine. Require a machine to have a node that
  can actually take the shard before selecting it.

* reduce comments across the machine-affinity change

Trim narration down to the non-obvious why; one terse line where a block was overkill.

* ec.balance: gate machine spread on fault-tolerance feasibility

Spreading a volume evenly across machines only helps when there are enough that
each can stay within EC's parity tolerance (numMachines >= ceil(total/parity)).
With fewer -- or wildly unequal -- machines it can't make a machine loss
survivable anyway, and forcing it fights capacity: e.g. a cluster of 12 volume
servers on one host and 2 on another would have half of every volume crammed onto
the 2-server box. So spread across machines only when it's achievable; otherwise
fall back to per-node spread and let capacity/global balancing decide.

The global load phase applies the same test: it protects a volume's machine spread
(no cross-machine move that raises a machine's count past the source's) only where
that spread is achievable, so heterogeneous clusters still level by fullness.

* ec.balance worker: group servers by host when planning

The worker built its planner topology without recording each server's host, so
automated ec.balance treated ports on one machine as independent nodes and could
concentrate a volume's shards on one physical box. Set the host from the volume
server address, matching the shell path.

* volume.balance worker: don't move a replica onto a machine holding one

The worker compared only node ids, and the replica map dropped the server address,
so it could move replicas onto different ports of one machine. Carry the host on
ReplicaLocation (from the server address) and reject a target whose host already
holds another replica of the volume. Best-effort, matching the shell.

* ec.balance: judge machine-spread feasibility by the rack's shards

The within-rack and global feasibility checks compared the whole volume's shard
count against a rack's machine count, so a rack holding only part of a volume after
cross-rack spreading -- e.g. 7 of a 10+4 volume across 2 machines -- was wrongly
judged infeasible and fell back to node spread, which could pile 6 shards onto one
host, past parity. Gate on the rack's own shard count of the volume instead.

* ec.balance: spread a volume's shards across machines by combined count

EC recovers from any loss within parity regardless of shard type, so what bounds a
machine's exposure is its total shards of the volume, not data and parity
separately. Spreading the two independently let each type's remainder land on the
same machine -- ceil(d/M)+ceil(p/M) can exceed ceil(total/M), e.g. a 5/3 split where
4/4 was achievable, past parity. Balance the combined count in one pass; disk-level
data/parity anti-affinity stays in pickBestDiskOnNode.

* ec.balance: don't let the imbalance threshold skip an over-parity machine

The within-rack spread gated on relative skew ((max-min)/avg > threshold), so a
worker threshold of 0.5 skipped an exactly-50%-skewed layout like 5/4/3 for a 10+4
volume, leaving 5 shards -- past parity -- on one machine. The even cap
(ceil(shards/groups)) is the real bound and the move loop already sheds only what
exceeds it, so drop the threshold gate from the within-rack phase (machine and node):
a balanced rack stays a no-op while any over-cap machine is always fixed.

* ec.balance: keep the imbalance threshold for the node fallback

Dropping the threshold from the whole within-rack phase made the node fallback too
eager: it runs only when machine fault tolerance is unachievable, so it is cosmetic
load distribution that should defer to the global utilization phase. Without the
gate it would, for a one-server-per-host 6/4 split at threshold 0.5, schedule a count
move that worsens utilization balance. Restore the threshold there; machine spreading
keeps bypassing it, since that bound is durability, not cosmetic skew.
2026-06-07 14:14:45 -07:00

257 lines
9.5 KiB
Go
Raw Blame History

package balance
import (
"testing"
"github.com/seaweedfs/seaweedfs/weed/admin/topology"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
"github.com/seaweedfs/seaweedfs/weed/storage/super_block"
"github.com/seaweedfs/seaweedfs/weed/worker/types"
)
func rp(t *testing.T, code string) *super_block.ReplicaPlacement {
t.Helper()
r, err := super_block.NewReplicaPlacementFromString(code)
if err != nil {
t.Fatalf("invalid replica placement code %q: %v", code, err)
}
return r
}
func loc(dc, rack, node string) types.ReplicaLocation {
return types.ReplicaLocation{DataCenter: dc, Rack: rack, NodeID: node}
}
func TestIsGoodMove_NoReplication(t *testing.T) {
// 000 = no replication. Any move is fine.
if !IsGoodMove(rp(t, "000"), []types.ReplicaLocation{loc("dc1", "r1", "n1")}, "n1", loc("dc1", "r1", "n2")) {
t.Error("000: any move should be allowed")
}
}
func TestIsGoodMove_MachineAntiAffinity(t *testing.T) {
// rep 001 allows two copies in one rack, so replica placement alone would permit
// moving onto another port of a host that already holds a replica; machine
// anti-affinity must reject that and allow a distinct host.
existing := []types.ReplicaLocation{
{DataCenter: "dc1", Rack: "r1", NodeID: "10.0.0.1:8080", Host: "10.0.0.1"},
{DataCenter: "dc1", Rack: "r1", NodeID: "10.0.0.2:8080", Host: "10.0.0.2"},
}
onSameMachine := types.ReplicaLocation{DataCenter: "dc1", Rack: "r1", NodeID: "10.0.0.1:8081", Host: "10.0.0.1"}
if IsGoodMove(rp(t, "001"), existing, "10.0.0.2:8080", onSameMachine) {
t.Error("move onto a machine already holding a replica should be rejected")
}
onOtherMachine := types.ReplicaLocation{DataCenter: "dc1", Rack: "r1", NodeID: "10.0.0.3:8080", Host: "10.0.0.3"}
if !IsGoodMove(rp(t, "001"), existing, "10.0.0.2:8080", onOtherMachine) {
t.Error("move onto a distinct machine should be allowed")
}
}
func TestIsGoodMove_001_SameRack(t *testing.T) {
// 001 = 1 replica on same rack (2 total on same rack)
existing := []types.ReplicaLocation{
loc("dc1", "r1", "n1"),
loc("dc1", "r1", "n2"),
}
// Move n1 -> n3 on same rack: good
if !IsGoodMove(rp(t, "001"), existing, "n1", loc("dc1", "r1", "n3")) {
t.Error("001: move to same rack should be allowed")
}
// Move n1 -> n3 on different rack: bad (would leave only 1 on r1, need 2)
if IsGoodMove(rp(t, "001"), existing, "n1", loc("dc1", "r2", "n3")) {
t.Error("001: move to different rack should not be allowed when it breaks same-rack count")
}
}
func TestIsGoodMove_010_DiffRack(t *testing.T) {
// 010 = 1 replica on different rack (2 racks total)
existing := []types.ReplicaLocation{
loc("dc1", "r1", "n1"),
loc("dc1", "r2", "n2"),
}
// Move n1 -> n3 on r2: bad (both replicas on same rack)
if IsGoodMove(rp(t, "010"), existing, "n1", loc("dc1", "r2", "n3")) {
t.Error("010: move to same rack as other replica should not be allowed")
}
// Move n1 -> n3 on r3: good (still 2 different racks)
if !IsGoodMove(rp(t, "010"), existing, "n1", loc("dc1", "r3", "n3")) {
t.Error("010: move to different rack should be allowed")
}
}
func TestIsGoodMove_100_DiffDC(t *testing.T) {
// 100 = 1 replica in different DC
existing := []types.ReplicaLocation{
loc("dc1", "r1", "n1"),
loc("dc2", "r1", "n2"),
}
// Move n1 -> n3 in dc2: bad (both in same DC)
if IsGoodMove(rp(t, "100"), existing, "n1", loc("dc2", "r1", "n3")) {
t.Error("100: move to same DC as other replica should not be allowed")
}
// Move n1 -> n3 in dc3: good (different DCs)
if !IsGoodMove(rp(t, "100"), existing, "n1", loc("dc3", "r1", "n3")) {
t.Error("100: move to different DC should be allowed")
}
}
func TestIsGoodMove_SameNode(t *testing.T) {
// Moving to the same node as an existing replica should always be rejected
existing := []types.ReplicaLocation{
loc("dc1", "r1", "n1"),
loc("dc1", "r2", "n2"),
}
if IsGoodMove(rp(t, "010"), existing, "n1", loc("dc1", "r2", "n2")) {
t.Error("should reject move to same node as existing replica")
}
}
func TestIsGoodMove_011_Composite(t *testing.T) {
// 011 = 1 same-rack + 1 different-rack (3 replicas: 2 on same rack, 1 on different)
existing := []types.ReplicaLocation{
loc("dc1", "r1", "n1"),
loc("dc1", "r1", "n2"),
loc("dc1", "r2", "n3"),
}
// Move n1 -> n4 on r1: good (maintains 2 on r1, 1 on r2)
if !IsGoodMove(rp(t, "011"), existing, "n1", loc("dc1", "r1", "n4")) {
t.Error("011: move within same rack should be allowed")
}
// Move n3 -> n4 on r1: bad (would have 3 on r1, 0 on different rack)
if IsGoodMove(rp(t, "011"), existing, "n3", loc("dc1", "r1", "n4")) {
t.Error("011: move that eliminates different-rack replica should not be allowed")
}
}
func TestIsGoodMove_110_Composite(t *testing.T) {
// 110 = 1 different-rack + 1 different-DC (3 replicas across 2 DCs and 2 racks)
existing := []types.ReplicaLocation{
loc("dc1", "r1", "n1"),
loc("dc1", "r2", "n2"),
loc("dc2", "r1", "n3"),
}
// Move n1 -> n4 in dc1/r3: good (dc1 still has r2+r3, dc2 has r1)
if !IsGoodMove(rp(t, "110"), existing, "n1", loc("dc1", "r3", "n4")) {
t.Error("110: move to new rack in same DC should be allowed")
}
// Move n3 -> n4 in dc1/r1: bad (would lose the different-DC replica)
if IsGoodMove(rp(t, "110"), existing, "n3", loc("dc1", "r1", "n4")) {
t.Error("110: move that eliminates different-DC replica should not be allowed")
}
}
func TestIsGoodMove_NilReplicaPlacement(t *testing.T) {
if !IsGoodMove(nil, []types.ReplicaLocation{loc("dc1", "r1", "n1")}, "n1", loc("dc1", "r1", "n2")) {
t.Error("nil replica placement should allow any move")
}
}
func TestCalculateBalanceScore_ReplicationAware(t *testing.T) {
disk := func(dc, rack string) *topology.DiskInfo {
return &topology.DiskInfo{
DataCenter: dc,
Rack: rack,
DiskInfo: &master_pb.DiskInfo{MaxVolumeCount: 100, VolumeCount: 50},
}
}
// 001: same-rack replication — should prefer same rack and same DC
rp001 := rp(t, "001")
sameRack := calculateBalanceScore(disk("dc1", "r1"), "r1", "dc1", 0, rp001)
diffRack := calculateBalanceScore(disk("dc1", "r2"), "r1", "dc1", 0, rp001)
diffDC := calculateBalanceScore(disk("dc2", "r2"), "r1", "dc1", 0, rp001)
if sameRack <= diffRack {
t.Errorf("001: same-rack score (%v) should exceed different-rack score (%v)", sameRack, diffRack)
}
if sameRack <= diffDC {
t.Errorf("001: same-rack score (%v) should exceed different-DC score (%v)", sameRack, diffDC)
}
// 010: different-rack replication — should prefer different rack, same DC
rp010 := rp(t, "010")
sameRack = calculateBalanceScore(disk("dc1", "r1"), "r1", "dc1", 0, rp010)
diffRack = calculateBalanceScore(disk("dc1", "r2"), "r1", "dc1", 0, rp010)
if diffRack <= sameRack {
t.Errorf("010: different-rack score (%v) should exceed same-rack score (%v)", diffRack, sameRack)
}
// 100: different-DC replication — should prefer different DC
rp100 := rp(t, "100")
sameDC := calculateBalanceScore(disk("dc1", "r2"), "r1", "dc1", 0, rp100)
diffDCScore := calculateBalanceScore(disk("dc2", "r2"), "r1", "dc1", 0, rp100)
if diffDCScore <= sameDC {
t.Errorf("100: different-DC score (%v) should exceed same-DC score (%v)", diffDCScore, sameDC)
}
// nil rp: should prefer cross-rack/DC (default behavior)
sameRack = calculateBalanceScore(disk("dc1", "r1"), "r1", "dc1", 0, nil)
diffRack = calculateBalanceScore(disk("dc1", "r2"), "r1", "dc1", 0, nil)
if diffRack <= sameRack {
t.Errorf("nil rp: different-rack score (%v) should exceed same-rack score (%v)", diffRack, sameRack)
}
}
func TestPlanBalanceDestination_ChoosesBestValidCompositeDestination(t *testing.T) {
servers := []serverSpec{
{id: "node-a", diskType: "hdd", diskID: 1, dc: "dc1", rack: "rack1"},
{id: "node-b", diskType: "hdd", diskID: 2, dc: "dc1", rack: "rack3"},
{id: "node-c", diskType: "hdd", diskID: 3, dc: "dc1", rack: "rack1"},
}
volumes := makeVolumesWith("node-a", "hdd", "dc1", "rack1", "c1", 1, 1, withReplicas(11))
replicas := []types.ReplicaLocation{
loc("dc1", "rack1", "node-a"),
loc("dc1", "rack1", "node-d"),
loc("dc1", "rack2", "node-e"),
}
plan, err := planBalanceDestination(buildTopology(servers, volumes), volumes[0], rp(t, "011"), replicas, map[string]int{
"node-b": 0,
"node-c": 0,
})
if err != nil {
t.Fatalf("planBalanceDestination failed: %v", err)
}
if plan.TargetNode != "node-c" {
t.Fatalf("expected valid same-rack destination node-c, got %s", plan.TargetNode)
}
if plan.TargetRack != "rack1" {
t.Fatalf("expected rack1 destination, got %s", plan.TargetRack)
}
}
func TestCreateBalanceTask_FallbackSelectsValidCompositeDestination(t *testing.T) {
servers := []serverSpec{
{id: "node-a", diskType: "hdd", diskID: 1, dc: "dc1", rack: "rack1"},
{id: "node-b", diskType: "hdd", diskID: 2, dc: "dc1", rack: "rack3"},
{id: "node-c", diskType: "hdd", diskID: 3, dc: "dc1", rack: "rack1"},
}
volumes := makeVolumesWith("node-a", "hdd", "dc1", "rack1", "c1", 1, 1, withReplicas(11))
clusterInfo := &types.ClusterInfo{
ActiveTopology: buildTopology(servers, volumes),
VolumeReplicaMap: map[uint32][]types.ReplicaLocation{
1: {
loc("dc1", "rack1", "node-a"),
loc("dc1", "rack1", "node-d"),
loc("dc1", "rack2", "node-e"),
},
},
}
task, destination := createBalanceTask("hdd", volumes[0], clusterInfo, "node-b", map[string]int{
"node-b": 0,
"node-c": 0,
})
if task == nil {
t.Fatal("expected a balance task")
}
if destination != "node-c" {
t.Fatalf("expected fallback destination node-c, got %s", destination)
}
if len(task.TypedParams.Targets) != 1 {
t.Fatalf("expected 1 target, got %d", len(task.TypedParams.Targets))
}
if got := task.TypedParams.Targets[0].Node; got != "node-c:8080" {
t.Fatalf("expected target node-c:8080, got %s", got)
}
}
Reference in new issue View Git Blame Copy Permalink