mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-16 11:30:44 +02:00
Chris Lu
e24a443b17
* proto: define MountRegister/MountList and MountPeer service Adds the wire types for peer chunk sharing between weed mount clients: * filer.proto: MountRegister / MountList RPCs so each mount can heartbeat its peer-serve address into a filer-hosted registry, and refresh the list of peers. Tiny payload; the filer stores only O(fleet_size) state. * mount_peer.proto (new): ChunkAnnounce / ChunkLookup RPCs for the mount-to-mount chunk directory. Each fid's directory entry lives on an HRW-assigned mount; announces and lookups route to that mount. No behavior yet — later PRs wire the RPCs into the filer and mount. See design-weed-mount-peer-chunk-sharing.md for the full design. * filer: add mount-server registry behind -peer.registry.enable Implements tier 1 of the peer chunk sharing design: an in-memory registry of live weed mount servers, keyed by peer address, refreshed by MountRegister heartbeats and served by MountList. * weed/filer/peer_registry.go: thread-safe map with TTL eviction; lazy sweep on List plus a background sweeper goroutine for bounded memory. * weed/server/filer_grpc_server_peer.go: MountRegister / MountList RPC handlers. When -peer.registry.enable is false (the default), both RPCs are silent no-ops so probing older filers is harmless. * -peer.registry.enable flag on weed filer; FilerOption.PeerRegistryEnabled wires it through. Phase 1 is single-filer (no cross-filer replication of the registry); mounts that fail over to another filer will re-register on the next heartbeat, so the registry self-heals within one TTL cycle. Part of the peer-chunk-sharing design; no behavior change at runtime until a later PR enables the flag on both filer and mount. * filer: nil-safe peerRegistryEnable + registry hardening Addresses review feedback on PR #9131. * Fix: nil pointer deref in the mini cluster. FilerOptions instances constructed outside weed/command/filer.go (e.g. miniFilerOptions in mini.go) do not populate peerRegistryEnable, so dereferencing the pointer panics at Filer startup. Use the same `nil && deref` idiom already used for distributedLock / writebackCache. * Hardening (gemini review): registry now enforces three invariants: - empty peer_addr is silently rejected (no client-controlled sentinel mass-inserts) - TTL is capped at 1 hour so a runaway client cannot pin entries - new-entry count is capped at 10000 to bound memory; renewals of existing entries are always honored, so a full registry still heartbeats its existing members correctly Covered by new unit tests. * filer: rename -peer.registry.enable flag to -mount.p2p Per review feedback: the old name "peer.registry.enable" leaked the implementation ("registry") into the CLI surface. "mount.p2p" is shorter and describes what it actually controls — whether this filer participates in mount-to-mount peer chunk sharing. Flag renames (all three keep default=true, idle cost is near-zero): -peer.registry.enable -> -mount.p2p (weed filer) -filer.peer.registry.enable -> -filer.mount.p2p (weed mini, weed server) Internal variable names (mountPeerRegistryEnable, MountPeerRegistry) keep their longer form — they describe the component, not the knob. * filer: MountList returns DataCenter + List uses RLock Two review follow-ups on the mount peer registry: * weed/server/filer_grpc_server_mount_peer.go: MountList was dropping the DataCenter on the wire. The whole point of carrying DC separately from Rack is letting the mount-side fetcher re-rank peers by the two-level locality hierarchy (same-rack > same-DC > cross-DC); without DC in the response every remote peer collapsed to "unknown locality." * weed/filer/mount_peer_registry.go: List() was taking a write lock so it could lazy-delete expired entries inline. But MountList is a read-heavy RPC hit on every mount's 30 s refresh loop, and Sweep is already wired as the sole reclamation path (same pattern as the mount-side PeerDirectory). Switch List to RLock + filter, let Sweep do the map mutation, so concurrent MountList callers don't serialize on each other. Test updated to reflect the new contract (List no longer mutates the map; Sweep is what drops expired entries).
66 lines
2.2 KiB
Go
66 lines
2.2 KiB
Go
package weed_server
|
|
|
|
import (
|
|
"context"
|
|
"time"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/glog"
|
|
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
|
)
|
|
|
|
// mountPeerRegistrySweepInterval is how often the filer evicts expired mount
|
|
// registry entries. Eviction is also done lazily on List; the sweep keeps
|
|
// memory bounded on long-running filers with high churn.
|
|
const mountPeerRegistrySweepInterval = 60 * time.Second
|
|
|
|
// runMountPeerRegistrySweeper runs for the lifetime of the FilerServer when
|
|
// peer registry is enabled.
|
|
func (fs *FilerServer) runMountPeerRegistrySweeper() {
|
|
ticker := time.NewTicker(mountPeerRegistrySweepInterval)
|
|
defer ticker.Stop()
|
|
for range ticker.C {
|
|
if fs.mountPeerRegistry == nil {
|
|
return
|
|
}
|
|
if evicted := fs.mountPeerRegistry.Sweep(); evicted > 0 {
|
|
glog.V(2).Infof("peer registry: evicted %d stale entries", evicted)
|
|
}
|
|
}
|
|
}
|
|
|
|
// MountRegister records (or refreshes) the caller as a live mount server in
|
|
// the filer's peer registry. Returns an empty response; the caller is
|
|
// expected to heartbeat before the TTL expires.
|
|
//
|
|
// Requests are silently dropped when the registry is disabled (default), so
|
|
// clients can safely probe without breaking older filers.
|
|
func (fs *FilerServer) MountRegister(ctx context.Context, req *filer_pb.MountRegisterRequest) (*filer_pb.MountRegisterResponse, error) {
|
|
if fs.mountPeerRegistry == nil {
|
|
return &filer_pb.MountRegisterResponse{}, nil
|
|
}
|
|
ttl := time.Duration(req.TtlSeconds) * time.Second
|
|
fs.mountPeerRegistry.Register(req.PeerAddr, req.DataCenter, req.Rack, ttl)
|
|
return &filer_pb.MountRegisterResponse{}, nil
|
|
}
|
|
|
|
// MountList returns the current set of live mounts for callers building
|
|
// their HRW seed view.
|
|
func (fs *FilerServer) MountList(ctx context.Context, req *filer_pb.MountListRequest) (*filer_pb.MountListResponse, error) {
|
|
if fs.mountPeerRegistry == nil {
|
|
return &filer_pb.MountListResponse{}, nil
|
|
}
|
|
entries := fs.mountPeerRegistry.List()
|
|
resp := &filer_pb.MountListResponse{
|
|
Mounts: make([]*filer_pb.MountInfo, 0, len(entries)),
|
|
}
|
|
for _, e := range entries {
|
|
resp.Mounts = append(resp.Mounts, &filer_pb.MountInfo{
|
|
PeerAddr: e.PeerAddr,
|
|
DataCenter: e.DataCenter,
|
|
Rack: e.Rack,
|
|
LastSeenNs: e.LastSeenNs,
|
|
})
|
|
}
|
|
return resp, nil
|
|
}
|