mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-26 15:00:48 +01:00
6dd6ad512cd494d3bfa7f86bf74cdec8dff1b3eb
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandDevin
|
da1d7444b8 |
master: shorten the first-write hold after a fresh topology id (#12059)
* master: release the post-mint allocation hold after a quiet pulse DeferAllocation armed a fixed warmup window whenever the leader minted a fresh topology id, so the first write on a brand-new cluster blocked for the whole window even though no prior state could exist. The window still has to cover a rebuilt master's veteran servers, which report on their first post-reconnect heartbeat. Track the last heartbeat that could carry prior cluster state — a server joining, held volumes or shards, a nonzero id bound, or a foreign topology id — and lift the hold one pulse after it goes quiet, capped by the original window. * master: make the post-mint allocation hold configurable master.volume_growth.allocation_deferral overrides the warmup window a leader holds volume id allocation for after minting a fresh topology id. Unset keeps the existing window; "0s" disables the hold for setups that know the cluster is fresh (test harnesses, single-node labs); any other duration replaces the cap while the quiet-pulse early release still applies. * master: re-check the allocation deadline while it is held NextVolumeId slept for the whole computed delay, so a heartbeat landing mid-sleep that pushed the quiet deadline out was ignored — the pending allocation could still run ahead of a straggler's report. Wait in short slices, re-evaluating the deadline and the growth freeze each round. * master: count only ids above the issued bound as prior-state evidence Heartbeat echoes of volumes and shards this master allocated during the hold refreshed the quiet window, so each growth allocation re-extended the deferral until the hard cap. Evidence now requires an id beyond the bound this leader has issued, a higher reported bound, or a foreign topology id; a joining server (dn == nil) still counts once. * master: reject an unparseable allocation_deferral instead of disabling the hold Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: do not count a skipped heartbeat as a joining server Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: assert the allocation hold extends past the original quiet deadline Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: release the post-mint hold after two quiet pulses One pulse proved short for a veteran that reconnects on a staggered restart: with only five seconds of quiet, allocation could resume and reuse ids before a slower server landed its first heartbeat. Two pulses still halves the fresh-cluster first-write stall versus the full warmup cap, and known-fresh setups keep the "0s" override. * master: pick once more when assign times out as growth concludes The pick loop's final attempt can predate the growth it triggered concluding: growth registers its volumes in the layout before clearing the pending flag, so a timeout landing on that boundary used to shed a hard 406 over volumes that had just arrived. Pick once more after the loop in both the http and grpc assign paths before deciding how to shed. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
2874bd05fb |
Freeze volume growth on topology id mismatch (#11763) (#11820)
* pb: ResumeVolumeGrowth rpc for operator-driven growth resume (#11763) * master: freeze volume growth on topology id mismatch (#11763) * shell: volume.growth.resume command (#11763) * master: replicate the growth freeze and gate it on veteran heartbeats (#11763) * master: block allocation while the topology id is being minted DeferAllocation only armed after the topology id persisted, so growth requests arriving during init ran ahead of the warmup window. Gate NextVolumeId on an init-pending flag spanning the whole mint. * master: stash reported topology ids until the local id is minted (#11763) * drain stashed topology id reports when the id arrives outside the mint path * persist the growth-freeze flag in the mirror file and raft snapshots * log hashicorp raft marshal and apply errors in applyGrowthFreeze |
||
|
Chris Lu
|
15f4a97029 |
fix: improve raft leader election reliability and failover speed (#8692)
* fix: clear raft vote state file on non-resume startup The seaweedfs/raft library v1.1.7 added a persistent `state` file for currentTerm and votedFor. When RaftResumeState=false (the default), the log, conf, and snapshot directories are cleared but this state file was not. On repeated restarts, different masters accumulate divergent terms, causing AppendEntries rejections and preventing leader election. Fixes #8690 * fix: recover TopologyId from snapshot before clearing raft state When RaftResumeState=false clears log/conf/snapshot, the TopologyId (used for license validation) was lost. Now extract it from the latest snapshot before cleanup and restore it on the topology. Both seaweedfs/raft and hashicorp/raft paths are handled, with a shared recoverTopologyIdFromState helper in raft_common.go. * fix: stagger multi-master bootstrap delay by peer index Previously all masters used a fixed 1500ms delay before the bootstrap check. Now the delay is proportional to the peer's sorted index with randomization (matching the hashicorp raft path), giving the designated bootstrap node (peer 0) a head start while later peers wait for gRPC servers to be ready. Also adds diagnostic logging showing why DoJoinCommand was or wasn't called, making leader election issues easier to diagnose from logs. * fix: skip unreachable masters during leader reconnection When a master leader goes down, non-leader masters still redirect clients to the stale leader address. The masterClient would follow these redirects, fail, and retry — wasting round-trips each cycle. Now tryAllMasters tracks which masters failed within a cycle and skips redirects pointing to them, reducing log spam and connection overhead during leader failover. * fix: take snapshot after TopologyId generation for recovery After generating a new TopologyId on the leader, immediately take a raft snapshot so the ID can be recovered from the snapshot on future restarts with RaftResumeState=false. Without this, short-lived clusters would lose the TopologyId on restart since no automatic snapshot had been taken yet. * test: add multi-master raft failover integration tests Integration test framework and 5 test scenarios for 3-node master clusters: - TestLeaderConsistencyAcrossNodes: all nodes agree on leader and TopologyId - TestLeaderDownAndRecoverQuickly: leader stops, new leader elected, old leader rejoins as follower - TestLeaderDownSlowRecover: leader gone for extended period, cluster continues with 2/3 quorum - TestTwoMastersDownAndRestart: quorum lost (2/3 down), recovered when both restart - TestAllMastersDownAndRestart: full cluster restart, leader elected, all nodes agree on TopologyId * fix: address PR review comments - peerIndex: return -1 (not 0) when self not found, add warning log - recoverTopologyIdFromSnapshot: defer dir.Close() - tests: check GetTopologyId errors instead of discarding them * fix: address review comments on failover tests - Assert no leader after quorum loss (was only logging) - Verify follower cs.Leader matches expected leader via ServerAddress.ToHttpAddress() comparison - Check GetTopologyId error in TestTwoMastersDownAndRestart |
||
|
Chris Lu
|
753e1db096 |
Prevent split-brain: Persistent ClusterID and Join Validation (#8022)
* Prevent split-brain: Persistent ClusterID and Join Validation - Persist ClusterId in Raft store to survive restarts. - Validate ClusterId on Raft command application (piggybacked on MaxVolumeId). - Prevent masters with conflicting ClusterIds from joining/operating together. - Update Telemetry to report the persistent ClusterId. * Refine ClusterID validation based on feedback - Improved error message in cluster_commands.go. - Added ClusterId mismatch check in RaftServer.Recovery. * Handle Raft errors and support Hashicorp Raft for ClusterId - Check for errors when persisting ClusterId in legacy Raft. - Implement ClusterId generation and persistence for Hashicorp Raft leader changes. - Ensure consistent error logging. * Refactor ClusterId validation - Centralize ClusterId mismatch check in Topology.SetClusterId. - Simplify MaxVolumeIdCommand.Apply and RaftServer.Recovery to rely on SetClusterId. * Fix goroutine leak and add timeout - Handle channel closure in Hashicorp Raft leader listener. - Add timeout to Raft Apply call to prevent blocking. * Fix deadlock in legacy Raft listener - Wrap ClusterId generation/persistence in a goroutine to avoid blocking the Raft event loop (deadlock). * Rename ClusterId to SystemId - Renamed ClusterId to SystemId across the codebase (protobuf, topology, server, telemetry). - Regenerated telemetry.pb.go with new field. * Rename SystemId to TopologyId - Rename to SystemId was intermediate step. - Final name is TopologyId for the persistent cluster identifier. - Updated protobuf, topology, raft server, master server, and telemetry. * Optimize Hashicorp Raft listener - Integrated TopologyId generation into existing monitorLeaderLoop. - Removed extra goroutine in master_server.go. * Fix optimistic TopologyId update - Removed premature local state update of TopologyId in master_server.go and raft_hashicorp.go. - State is now solely updated via the Raft state machine Apply/Restore methods after consensus. * Add explicit log for recovered TopologyId - Added glog.V(0) info log in RaftServer.Recovery to print the recovered TopologyId on startup. * Add Raft barrier to prevent TopologyId race condition - Implement ensureTopologyId helper method - Send no-op MaxVolumeIdCommand to sync Raft log before checking TopologyId - Ensures persisted TopologyId is recovered before generating new one - Prevents race where generation happens during log replay * Serialize TopologyId generation with mutex - Add topologyIdGenLock mutex to MasterServer struct - Wrap ensureTopologyId method with lock to prevent concurrent generation - Fixes race where event listener and manual leadership check both generate IDs - Second caller waits for first to complete and sees the generated ID * Add TopologyId recovery logging to Apply method - Change log level from V(1) to V(0) for visibility - Log 'Recovered TopologyId' when applying from Raft log - Ensures recovery is visible whether from snapshot or log replay - Matches Recovery() method logging for consistency * Fix Raft barrier timing issue - Add 100ms delay after barrier command to ensure log application completes - Add debug logging to track barrier execution and TopologyId state - Return early if barrier command fails - Prevents TopologyId generation before old logs are fully applied * ensure leader * address comments * address comments * redundant * clean up * double check * refactoring * comment |