master: let the leader admit a master that starts with no raft state (#10865)

* master: answer with the leader raft already knows

Topo.Leader() backs off for up to 20 seconds waiting for an election.
Callers that a health probe or a client is blocked on cannot afford that:
/cluster/status, /cluster/healthz and /readyz all sit past the probe
timeout of both the helm chart and the operator, so a master that is
still joining looks dead rather than joining, and the kubelet restarts
it. informNewLeader and SendHeartbeat hold the client on a master that
cannot serve it, exactly when it should move on to find the one that can.

Answer these from MaybeLeader instead, which reports what raft knows
right now. MaybeLeader takes over the "am I the leader myself" fallback
that Leader() used to apply on top of it, so one non-blocking call is
still correct; Leader() keeps the backoff for callers that must wait.

* master: let the leader admit a master that starts with no raft state

Neither raft implementation lets a server outside the configuration
campaign: goraft's promotable() requires a non-empty log, and hashicorp
rejects vote requests from a candidate that is not in its configuration.
A master that comes up with fresh state therefore cannot elect itself in
— the leader has to pull it in. Nothing did.

The peer list is static, rendered from the replica count, so scaling it
up leaves the sitting leader running the old list with no idea the new
masters exist. Under goraft they wait forever. Under hashicorp they are
worse off: each bootstraps a cluster of its own from the new list, and
two of them form a quorum next to the live leader, with their own
TopologyId. That is the split brain SetTopologyId kills a master over.

Admit the peer where it registers instead. Only the leader gets past the
IsLeader check in KeepConnected, and a joining master's client lands
there, so that is the moment it joins. The broadcast OnPeerUpdate rides
on is not enough on its own: it only reaches masters already connected,
which is why a leader that came up first missed both newcomers.

RaftAddServer grew a goraft branch on the way, so cluster.raft.add stops
silently doing nothing on the default raft, and RaftRemoveServer with it.
Bootstrapping is now one call for both implementations, made only after
the peers confirm nobody has a leader, and retried until this master is
in rather than checked once and dropped.

* master: do not evict a peer that is still in -peers

The hashicorp leader drops a master from the raft configuration as soon
as it stops answering pings. A master that is merely restarting answers
nothing, so an ordinary bounce shrinks the quorum behind the operator's
back — and then races its own return: the master comes back, registers,
gets re-admitted, and the eviction lands after it.

A randomized start/stop walk lands on it. Two of three masters running,
the leader evicts the one that just went down, the restart re-adds it,
the removal commits late and takes the leader's own leadership with it.
What is left is a two-server configuration whose other half is down, and
a running master that nobody will ask for a vote — no quorum, no way
back until the third master returns.

-peers is what declares membership. updatePeers already reconciles the
configuration against it on every leadership change, and an operator who
really means to drop a master can say so with cluster.raft.remove, so
keep the eviction for masters that are no longer listed at all.

* test: bounce masters at random and hold the election to it

Twelve rounds of stopping or starting a random master, on both raft
implementations, checking the two things an election must never get
wrong: two masters claiming leadership at once, and a quorum that comes
back without agreeing on one. The cluster's identity has to survive the
whole walk, since a master that re-mints a TopologyId is the split brain
SetTopologyId kills its peers over. The seed is random and logged, so a
failure names the walk that reproduces it.

Below a quorum the walk moves straight on. A master that has lost its
quorum cannot commit anything, and goraft only checks whether it still
has one on an election-timeout ticker, after its peers have been quiet
for a full timeout — measured taking over 30 seconds to step down. That
direction belongs to TestTwoMastersDownAndRestart, which was giving it
ten seconds and would have started failing on a slower machine; it now
waits on that behaviour explicitly rather than sleeping twice and hoping.

WaitForTopologyId returns the id it waited for. Reading it separately
raced the leader applying the raft entry that carries it, which shows up
as an empty id right after an election rather than as a wrong one.

weed/command/master.go

@@ -234,15 +234,17 @@ func startMaster(masterOption MasterOptions, masterWhiteList []string) {
 		if raftServer == nil {
 			glog.Fatalf("please verify %s is writable, see https://github.com/seaweedfs/seaweedfs/issues/717: %s", *masterOption.metaFolder, err)
 		}
-		// For single-master mode with a fresh log, initialize cluster immediately.
-		// When resuming with existing state, the server is already a member and
-		// will self-elect via fastResume — sending another JoinCommand would block
-		// because goraft's setCommitIndex returns early on JoinCommand entries,
-		// preventing the new entry's event from being notified when old uncommitted
-		// JoinCommands exist in the log.
-		if isSingleMaster && !raftServer.HasExistingState() {
-			glog.V(0).Infof("Single-master mode: initializing cluster immediately")
-			raftServer.DoJoinCommand()
+	}
+	// For single-master mode with a fresh log, initialize cluster immediately.
+	// When resuming with existing state, the server is already a member and
+	// will self-elect via fastResume — sending another JoinCommand would block
+	// because goraft's setCommitIndex returns early on JoinCommand entries,
+	// preventing the new entry's event from being notified when old uncommitted
+	// JoinCommands exist in the log.
+	if isSingleMaster && !raftServer.HasExistingState() {
+		glog.V(0).Infof("Single-master mode: initializing cluster immediately")
+		if err := raftServer.Bootstrap(); err != nil {
+			glog.Errorf("fail to bootstrap the cluster: %v", err)
 		}
 	}
 	ms.SetRaftServer(raftServer)
@@ -272,8 +274,13 @@ func startMaster(masterOption MasterOptions, masterWhiteList []string) {
 	go grpcS.Serve(grpcL)
 	pb.ServeGrpcOnLocalSocket(grpcS, grpcPort)
 
-	// For multi-master mode with non-Hashicorp raft, wait and check if we should join
-	if !*masterOption.raftHashicorp && !isSingleMaster {
+	// A master that starts with no raft state cannot elect on its own — neither
+	// raft implementation lets a server outside the configuration campaign — so
+	// it has to be pulled in by a leader. Keep asking the peers who the leader is
+	// until we are in: the leader admits us once our master client registers, and
+	// only when nobody has one does the first peer mint a new cluster. Restarting
+	// a master alone, or scaling the peer list up, both land here.
+	if !isSingleMaster {
 		go func() {
 			// Stagger bootstrap by peer index so masters don't all check
 			// simultaneously. Peer 0 waits ~1.5s, peer 1 ~3s, etc.
@@ -282,22 +289,24 @@ func startMaster(masterOption MasterOptions, masterWhiteList []string) {
 			glog.V(0).Infof("bootstrap check in %v (peer index %d of %d)", delay, idx, len(peers))
 			time.Sleep(delay)
 
-			ms.Topo.RaftServerAccessLock.RLock()
-			isEmptyMaster := ms.Topo.RaftServer.Leader() == "" && ms.Topo.RaftServer.IsLogEmpty()
-			isFirst := idx == 0
-			if isEmptyMaster && isFirst {
-				existingLeader := ms.MasterClient.FindLeaderFromOtherPeers(myMasterAddress)
-				if existingLeader == "" {
-					raftServer.DoJoinCommand()
-				} else {
-					glog.V(0).Infof("skip bootstrap: existing leader %s found from peers", existingLeader)
+			for {
+				if raftServer.HasExistingState() {
+					return
 				}
-			} else if !isEmptyMaster {
-				glog.V(0).Infof("skip bootstrap: leader=%q logEmpty=%v", ms.Topo.RaftServer.Leader(), ms.Topo.RaftServer.IsLogEmpty())
-			} else {
-				glog.V(0).Infof("skip bootstrap: %v is not the first master in peers (index %d)", myMasterAddress, idx)
+				if leader, err := ms.Topo.MaybeLeader(); err == nil && leader != "" {
+					return
+				}
+				if existingLeader := ms.MasterClient.FindLeaderFromOtherPeers(myMasterAddress); existingLeader != "" {
+					glog.V(0).Infof("waiting to be admitted by existing leader %s", existingLeader)
+				} else if idx == 0 {
+					if err := raftServer.Bootstrap(); err != nil {
+						glog.Errorf("fail to bootstrap the cluster: %v", err)
+					}
+				} else {
+					glog.V(0).Infof("skip bootstrap: %v is not the first master in peers (index %d)", myMasterAddress, idx)
+				}
+				time.Sleep(raftJoinCheckDelay)
 			}
-			ms.Topo.RaftServerAccessLock.RUnlock()
 		}()
 	}