master: bind heartbeat claims to the connecting peer (#9443)

SendHeartbeat used to accept whatever Ip/Port/Volumes the caller put on
the wire. Three changes tighten that:

- Reject heartbeats whose Ip does not match the gRPC peer's source
  address. Loopback peers are still trusted; operators behind a proxy
  can opt out with -master.allowUntrustedHeartbeat.
- Track which (ip, port) first claimed a volume id or an ec shard slot
  and drop foreign re-claims. Non-EC volume claims are bounded by the
  replica copy count so legitimate replicas still register. EC
  ownership is keyed by (vid, shard_id) so the same vid can legitimately
  be split across many peers as long as their EcIndexBits are disjoint;
  rejected bits are cleared from the bitmap and the parallel ShardSizes
  array is compacted in lock-step.
- Maintain reverse indexes owner -> volumes and owner -> ec shard slots
  so disconnect cleanup is O(M) in what that peer held rather than O(N)
  over the whole map.

Bindings are also released when a heartbeat reports that the peer no
longer holds an id, either via explicit Deleted{Volumes,EcShards}
entries or by omitting it from a full snapshot. Without this, a planned
rebalance that moved a vid or an ec shard from peer A to peer B would
leave B's heartbeats permanently filtered out until A disconnected,
breaking ec encode/decode flows that delete shards on the source as
soon as the move completes.

The (vid -> owners) binding still does not track which replica slot
each peer occupies, so the first N claims under the copy count win;
strict per-slot mapping is a follow-up.
This commit is contained in:
Chris Lu authored and GitHub committed 2026-05-12 15:38:52 -07:00
1 parent 10cc06333b
commit f28c7ce6df
5 files changed
+1091

No files matched your search

+6
View File
@@ -74,6 +74,10 @@ type MasterOptions struct {
telemetryEnabled *bool
debug *bool
debugPort *int
// allowUntrustedHeartbeat disables the source-ip binding check on
// SendHeartbeat. Required for deployments where a NAT or proxy sits
// between volume servers and the master.
allowUntrustedHeartbeat *bool
// shutdownCtx, when non-nil, tells startMaster to shut down once the ctx
// is cancelled. Used by integration tests and by weed mini; nil for
// standalone weed master.
@@ -109,6 +113,7 @@ func init() {
m.telemetryEnabled = cmdMaster.Flag.Bool("telemetry", false, "enable telemetry reporting")
m.debug = cmdMaster.Flag.Bool("debug", false, "serves runtime profiling data via pprof on the port specified by -debug.port")
m.debugPort = cmdMaster.Flag.Int("debug.port", 6060, "http port for debugging")
m.allowUntrustedHeartbeat = cmdMaster.Flag.Bool("master.allowUntrustedHeartbeat", false, "accept volume server heartbeats whose advertised ip does not match the gRPC peer address (required behind NAT/proxy)")
}
var cmdMaster = &Command{
@@ -463,5 +468,6 @@ func (m *MasterOptions) toMasterOption(whiteList []string) *weed_server.MasterOp
MetricsIntervalSec: *m.metricsIntervalSec,
TelemetryUrl: *m.telemetryUrl,
TelemetryEnabled: *m.telemetryEnabled,
AllowUntrustedHeartbeat: m.allowUntrustedHeartbeat != nil && *m.allowUntrustedHeartbeat,
}
}