Files
seaweedfs/weed/filer/mount_peer_registry_test.go
T
Chris Lu e24a443b17 peer chunk sharing 2/8: filer mount registry (#9131)
* proto: define MountRegister/MountList and MountPeer service

Adds the wire types for peer chunk sharing between weed mount clients:

* filer.proto: MountRegister / MountList RPCs so each mount can heartbeat
  its peer-serve address into a filer-hosted registry, and refresh the
  list of peers. Tiny payload; the filer stores only O(fleet_size) state.

* mount_peer.proto (new): ChunkAnnounce / ChunkLookup RPCs for the
  mount-to-mount chunk directory. Each fid's directory entry lives on
  an HRW-assigned mount; announces and lookups route to that mount.

No behavior yet — later PRs wire the RPCs into the filer and mount.
See design-weed-mount-peer-chunk-sharing.md for the full design.

* filer: add mount-server registry behind -peer.registry.enable

Implements tier 1 of the peer chunk sharing design: an in-memory registry
of live weed mount servers, keyed by peer address, refreshed by
MountRegister heartbeats and served by MountList.

* weed/filer/peer_registry.go: thread-safe map with TTL eviction; lazy
  sweep on List plus a background sweeper goroutine for bounded memory.

* weed/server/filer_grpc_server_peer.go: MountRegister / MountList RPC
  handlers. When -peer.registry.enable is false (the default), both RPCs
  are silent no-ops so probing older filers is harmless.

* -peer.registry.enable flag on weed filer; FilerOption.PeerRegistryEnabled
  wires it through.

Phase 1 is single-filer (no cross-filer replication of the registry);
mounts that fail over to another filer will re-register on the next
heartbeat, so the registry self-heals within one TTL cycle.

Part of the peer-chunk-sharing design; no behavior change at runtime
until a later PR enables the flag on both filer and mount.

* filer: nil-safe peerRegistryEnable + registry hardening

Addresses review feedback on PR #9131.

* Fix: nil pointer deref in the mini cluster. FilerOptions instances
  constructed outside weed/command/filer.go (e.g. miniFilerOptions in
  mini.go) do not populate peerRegistryEnable, so dereferencing the
  pointer panics at Filer startup. Use the same
  `nil && deref` idiom already used for distributedLock / writebackCache.

* Hardening (gemini review): registry now enforces three invariants:
  - empty peer_addr is silently rejected (no client-controlled sentinel
    mass-inserts)
  - TTL is capped at 1 hour so a runaway client cannot pin entries
  - new-entry count is capped at 10000 to bound memory; renewals of
    existing entries are always honored, so a full registry still
    heartbeats its existing members correctly

Covered by new unit tests.

* filer: rename -peer.registry.enable flag to -mount.p2p

Per review feedback: the old name "peer.registry.enable" leaked
the implementation ("registry") into the CLI surface. "mount.p2p"
is shorter and describes what it actually controls — whether this
filer participates in mount-to-mount peer chunk sharing.

Flag renames (all three keep default=true, idle cost is near-zero):
  -peer.registry.enable        ->  -mount.p2p         (weed filer)
  -filer.peer.registry.enable  ->  -filer.mount.p2p   (weed mini, weed server)

Internal variable names (mountPeerRegistryEnable, MountPeerRegistry)
keep their longer form — they describe the component, not the knob.

* filer: MountList returns DataCenter + List uses RLock

Two review follow-ups on the mount peer registry:

* weed/server/filer_grpc_server_mount_peer.go: MountList was dropping
  the DataCenter on the wire. The whole point of carrying DC separately
  from Rack is letting the mount-side fetcher re-rank peers by the
  two-level locality hierarchy (same-rack > same-DC > cross-DC); without
  DC in the response every remote peer collapsed to "unknown locality."

* weed/filer/mount_peer_registry.go: List() was taking a write lock so
  it could lazy-delete expired entries inline. But MountList is a
  read-heavy RPC hit on every mount's 30 s refresh loop, and Sweep is
  already wired as the sole reclamation path (same pattern as the
  mount-side PeerDirectory). Switch List to RLock + filter, let Sweep
  do the map mutation, so concurrent MountList callers don't serialize
  on each other.

Test updated to reflect the new contract (List no longer mutates the
map; Sweep is what drops expired entries).
2026-04-18 20:03:23 -07:00

170 lines
5.2 KiB
Go

package filer
import (
"fmt"
"sort"
"testing"
"time"
)
func testClock(t time.Time) func() time.Time {
return func() time.Time { return t }
}
func TestMountPeerRegistry_RegisterAndList(t *testing.T) {
start := time.Unix(1000, 0)
current := start
r := newMountPeerRegistryWithClock(func() time.Time { return current })
r.Register("mount-a:18080", "", "rack1", 30*time.Second)
r.Register("mount-b:18080", "", "rack2", 30*time.Second)
list := r.List()
if len(list) != 2 {
t.Fatalf("expected 2 entries, got %d", len(list))
}
sort.Slice(list, func(i, j int) bool { return list[i].PeerAddr < list[j].PeerAddr })
if list[0].PeerAddr != "mount-a:18080" || list[0].Rack != "rack1" {
t.Errorf("entry 0 unexpected: %+v", list[0])
}
if list[1].PeerAddr != "mount-b:18080" || list[1].Rack != "rack2" {
t.Errorf("entry 1 unexpected: %+v", list[1])
}
}
func TestMountPeerRegistry_RenewExtendsExpiry(t *testing.T) {
current := time.Unix(1000, 0)
r := newMountPeerRegistryWithClock(func() time.Time { return current })
r.Register("mount-a:18080", "", "rack1", 30*time.Second)
// Advance time past original expiry.
current = current.Add(25 * time.Second)
// Renew — should push expiry to now+30.
r.Register("mount-a:18080", "", "rack1-updated", 30*time.Second)
// Advance to where original expiry would have triggered eviction.
current = current.Add(10 * time.Second)
list := r.List()
if len(list) != 1 {
t.Fatalf("expected 1 entry after renew, got %d", len(list))
}
if list[0].Rack != "rack1-updated" {
t.Errorf("rack not updated on renew: %q", list[0].Rack)
}
}
func TestMountPeerRegistry_ExpirationDropsEntry(t *testing.T) {
current := time.Unix(1000, 0)
r := newMountPeerRegistryWithClock(func() time.Time { return current })
r.Register("mount-a:18080", "", "", 10*time.Second)
if n := r.Len(); n != 1 {
t.Fatalf("expected 1 entry, got %d", n)
}
current = current.Add(15 * time.Second)
list := r.List()
if len(list) != 0 {
t.Errorf("expected 0 entries after expiry, got %d", len(list))
}
// List no longer deletes — it's RLock-only so concurrent callers can
// proceed in parallel. Sweep is the sole reclamation path.
if n := r.Len(); n != 1 {
t.Errorf("List should not delete expired entries; Len=%d want 1", n)
}
if evicted := r.Sweep(); evicted != 1 {
t.Errorf("Sweep should have evicted the expired entry; got %d", evicted)
}
if n := r.Len(); n != 0 {
t.Errorf("after Sweep, Len=%d want 0", n)
}
}
func TestMountPeerRegistry_SweepCountsEvictions(t *testing.T) {
current := time.Unix(1000, 0)
r := newMountPeerRegistryWithClock(func() time.Time { return current })
r.Register("mount-a:18080", "", "", 10*time.Second)
r.Register("mount-b:18080", "", "", 60*time.Second)
current = current.Add(30 * time.Second)
got := r.Sweep()
if got != 1 {
t.Errorf("expected 1 eviction, got %d", got)
}
if n := r.Len(); n != 1 {
t.Errorf("expected 1 surviving entry, got %d", n)
}
}
func TestMountPeerRegistry_NegativeTTLFallsBackToDefault(t *testing.T) {
current := time.Unix(1000, 0)
r := newMountPeerRegistryWithClock(func() time.Time { return current })
r.Register("mount-a:18080", "", "", -5*time.Second)
if n := r.Len(); n != 1 {
t.Fatalf("expected 1 entry even with bad ttl, got %d", n)
}
// Advance past the default (60s) and confirm it expires.
current = current.Add(61 * time.Second)
list := r.List()
if len(list) != 0 {
t.Errorf("expected entry to expire after default ttl, got %d entries", len(list))
}
}
func TestMountPeerRegistry_EmptyList(t *testing.T) {
r := NewMountPeerRegistry()
list := r.List()
if len(list) != 0 {
t.Errorf("expected empty list, got %d", len(list))
}
_ = testClock // keep helper exported for future tests
}
func TestMountPeerRegistry_EmptyPeerAddrRejected(t *testing.T) {
r := NewMountPeerRegistry()
r.Register("", "", "rack", 30*time.Second)
if n := r.Len(); n != 0 {
t.Errorf("empty peer_addr should not insert; Len=%d", n)
}
}
func TestMountPeerRegistry_TTLCapped(t *testing.T) {
current := time.Unix(1000, 0)
r := newMountPeerRegistryWithClock(func() time.Time { return current })
// Request a ridiculous TTL; should be capped to maxMountPeerRegistryTTL.
r.Register("mount-a:18080", "", "", 24*time.Hour)
// Advance past the cap; entry should now be expired.
current = current.Add(maxMountPeerRegistryTTL + time.Second)
if list := r.List(); len(list) != 0 {
t.Errorf("expected entry to expire after maxMountPeerRegistryTTL, got %d", len(list))
}
}
func TestMountPeerRegistry_CapacityLimit(t *testing.T) {
r := NewMountPeerRegistry()
// Fill to capacity.
for i := 0; i < maxMountPeerRegistryEntries; i++ {
r.Register(fmt.Sprintf("mount-%d:18080", i), "", "", 60*time.Second)
}
if n := r.Len(); n != maxMountPeerRegistryEntries {
t.Fatalf("expected %d entries, got %d", maxMountPeerRegistryEntries, n)
}
// A brand-new address beyond the cap is rejected.
r.Register("new-mount:18080", "", "", 60*time.Second)
if n := r.Len(); n != maxMountPeerRegistryEntries {
t.Errorf("new entry past cap should be rejected; Len=%d", n)
}
// A renewal of an existing entry still succeeds.
r.Register("mount-0:18080", "", "rack-renewed", 60*time.Second)
if n := r.Len(); n != maxMountPeerRegistryEntries {
t.Errorf("renewal should not increase size; Len=%d", n)
}
}