* master: keep the hostname out of raft metric names
go-metrics prefixes gauge names with the hostname by default, which gives
every master its own metric family, e.g. SeaweedFS_<host>_raft_peers.
Prometheus already tells masters apart by the scrape target.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: send Hashicorp raft metrics through the go-metrics global raft uses
hashicorp/raft and raft-boltdb report through github.com/hashicorp/go-metrics,
but the master set up its prometheus sink on the armon/go-metrics global,
so a -raftHashicorp master exported no raft metric families. The sink is now
registered on the hashicorp/go-metrics global, and armon/go-metrics is no
longer a dependency.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart raft commit time and leader contact
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: keep the cluster in the raft panels
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: average raft commit time and leader contact over the rate window
The go-metrics summaries keep quantiles for 10s, so a p99 scraped every
30s misses most observations. The _sum/_count rates count every sample.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: leave raft panels empty when nothing was observed
Clamping the count rate plotted 0 ms for windows without commits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: let the raft metrics test use its own registry
Registering on stats.Gather made go test -count=2 fail with a duplicate
collector registration.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: register raft metrics before opening the raft stores
raft.NewRaft starts its run loop before returning, so registering the
sink afterwards could drop the first state and boltdb samples.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: answer with the leader raft already knows
Topo.Leader() backs off for up to 20 seconds waiting for an election.
Callers that a health probe or a client is blocked on cannot afford that:
/cluster/status, /cluster/healthz and /readyz all sit past the probe
timeout of both the helm chart and the operator, so a master that is
still joining looks dead rather than joining, and the kubelet restarts
it. informNewLeader and SendHeartbeat hold the client on a master that
cannot serve it, exactly when it should move on to find the one that can.
Answer these from MaybeLeader instead, which reports what raft knows
right now. MaybeLeader takes over the "am I the leader myself" fallback
that Leader() used to apply on top of it, so one non-blocking call is
still correct; Leader() keeps the backoff for callers that must wait.
* master: let the leader admit a master that starts with no raft state
Neither raft implementation lets a server outside the configuration
campaign: goraft's promotable() requires a non-empty log, and hashicorp
rejects vote requests from a candidate that is not in its configuration.
A master that comes up with fresh state therefore cannot elect itself in
— the leader has to pull it in. Nothing did.
The peer list is static, rendered from the replica count, so scaling it
up leaves the sitting leader running the old list with no idea the new
masters exist. Under goraft they wait forever. Under hashicorp they are
worse off: each bootstraps a cluster of its own from the new list, and
two of them form a quorum next to the live leader, with their own
TopologyId. That is the split brain SetTopologyId kills a master over.
Admit the peer where it registers instead. Only the leader gets past the
IsLeader check in KeepConnected, and a joining master's client lands
there, so that is the moment it joins. The broadcast OnPeerUpdate rides
on is not enough on its own: it only reaches masters already connected,
which is why a leader that came up first missed both newcomers.
RaftAddServer grew a goraft branch on the way, so cluster.raft.add stops
silently doing nothing on the default raft, and RaftRemoveServer with it.
Bootstrapping is now one call for both implementations, made only after
the peers confirm nobody has a leader, and retried until this master is
in rather than checked once and dropped.
* master: do not evict a peer that is still in -peers
The hashicorp leader drops a master from the raft configuration as soon
as it stops answering pings. A master that is merely restarting answers
nothing, so an ordinary bounce shrinks the quorum behind the operator's
back — and then races its own return: the master comes back, registers,
gets re-admitted, and the eviction lands after it.
A randomized start/stop walk lands on it. Two of three masters running,
the leader evicts the one that just went down, the restart re-adds it,
the removal commits late and takes the leader's own leadership with it.
What is left is a two-server configuration whose other half is down, and
a running master that nobody will ask for a vote — no quorum, no way
back until the third master returns.
-peers is what declares membership. updatePeers already reconciles the
configuration against it on every leadership change, and an operator who
really means to drop a master can say so with cluster.raft.remove, so
keep the eviction for masters that are no longer listed at all.
* test: bounce masters at random and hold the election to it
Twelve rounds of stopping or starting a random master, on both raft
implementations, checking the two things an election must never get
wrong: two masters claiming leadership at once, and a quorum that comes
back without agreeing on one. The cluster's identity has to survive the
whole walk, since a master that re-mints a TopologyId is the split brain
SetTopologyId kills its peers over. The seed is random and logged, so a
failure names the walk that reproduces it.
Below a quorum the walk moves straight on. A master that has lost its
quorum cannot commit anything, and goraft only checks whether it still
has one on an election-timeout ticker, after its peers have been quiet
for a full timeout — measured taking over 30 seconds to step down. That
direction belongs to TestTwoMastersDownAndRestart, which was giving it
ten seconds and would have started failing on a slower machine; it now
waits on that behaviour explicitly rather than sleeping twice and hoping.
WaitForTopologyId returns the id it waited for. Reading it separately
raced the leader applying the raft entry that carries it, which shows up
as an empty id right after an election rather than as a wrong one.
