5 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandDevin	18691e7b5b	server: keep the first write on a fresh weed server from stalling (#12149) ... * master: end the post-mint allocation hold on the embedded reporter's clean inventory DeferAllocation armed a warmup-length hold whenever the leader minted a fresh topology id, and on weed server the hold ran the whole window: the embedded volume server's join kept refreshing the quiet evidence clock, so the two-pulse early release still cost the first write 10-15s on every fresh start. weed mini already releases on its single pinned reporter's first inventory, but pinning refuses outside volume servers, which weed server must keep admitting. Add a non-exclusive variant: the topology names one expected reporter whose first clean full inventory lifts the hold, while other reporters stay admitted and still feed prior-state evidence and the topology-id freeze. A single-master weed server points it at its embedded volume server — the one reporter guaranteed to arrive quickly; a veteran's heartbeat carries prior state and keeps the window through the regular checks, so a wiped -dir with surviving external volume servers still trips the freeze on first report. Restarted clusters are untouched: a recovered topology id means no mint and no hold. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: announce on the startup peer scan for embedded single-master setups A fresh filer whose startup peer scan came back empty re-polled for the full confirm window before registering on the master, so the lock ring had no member and routed traffic such as S3 PUTs spun on "no lock server found" for ~10s on every fresh weed server / weed mini start. For setups that start with exactly one filer the scan is already final: a single-master weed server and weed mini announce as soon as it completes. A late external filer still joins and replays through the normal path — the confirm window only ever caught peers arriving inside it, and the residual risk mirrors the accepted volume-side trade-off: an embedded filer announcing a few seconds earlier can own the lock ring while a surviving external filer's backlog is still on its way. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: retry a failed heartbeat promptly until first registration A heartbeat stream refused before the leader is known — the common race when a volume server starts while an election is still running — always fell back to sleeping out the full pulse, so a fresh volume's first registration could wait ~5s behind a leader elected a second later. The peer refresh already identifies the leader during the wait, so cap the retry pause at a second once a leader is named; while no leader is known the same short retry applies until the first stream is confirmed, since an unregistered server has nothing else to do. Once the server was registered the pulse backoff is unchanged, so a lost leader still paces reconnects the way it did before. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: retry a failed heartbeat promptly until first registration (rust) Mirror the Go heartbeat retry: cap the post-refresh wait at a second when a leader is named, and keep the short retry until the first stream is confirmed so a fresh volume registers promptly when it races an election. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: leave admitted reporters a confirm pulse when the embedded report ends the hold The single-master weed server releases its post-mint allocation hold on the embedded volume server's clean report, but outside volume servers stay admitted and one may still carry prior cluster state — an inventory from the embedded server alone cannot prove none is due. Clearing the hold outright let a rebuilt master re-issue a volume id a reconnecting veteran still owned. Collapse the released hold to one heartbeat interval instead: late reporters land their evidence or a topology-id freeze inside the pulse, and a re-mint re-arms that pulse rather than the full warmup. The pinned single-reporter release (weed mini) still drops the hold outright, since no other reporter can be admitted. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: keep the empty-membership confirm on embedded single-master weed server A single-master weed server still admits external filers, so an empty startup peer scan is not final: after a master restart the existing filers simply have not reconnected yet. Announcing on the scan let the embedded filer take lock-ring ownership and serve writes that a peer's replayed backlog could then overwrite. Only weed mini keeps the skip, where the embedded filer is the only member its dev cluster can have. The confirm window only delays the master listing for a fresh store; the filer serves while it runs. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: tolerate an expired heartbeat retry deadline (rust) refresh_leader_from_peers can consume the whole pulse budget, leaving retry_at in the past; subtracting a later Instant panics and kills the heartbeat task instead of retrying. Match the Go side, where time.Until of a past deadline just sleeps zero. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: retry the seed list without a second pulse sleep (rust) The error path already slept its retry delay; an extra full pulse after losing the round delayed the next attempt past what Go's heartbeat() does, which paces retries only inside the per-master attempt. * master: keep a configured allocation hold shorter than the confirm pulse Once the embedded report lands, the collapsed hold replaced the configured deferral with a full pulse even when the operator set a shorter one, including zero (disabled). Cap at the pulse instead. * filer: end the empty-membership confirm early on a fresh-minted embedded master On a fresh single-master weed server the announce gate still held the filer for the full confirm window, so the master's lock ring stayed empty and the first write retried 'no lock server found' for ~7s on top of the collapsed allocation pulse. A fresh TopologyId mint proves the master's persisted state was empty, making the empty peer scan final; skip the rest of the confirm only then, and keep it whenever the master recovers a prior id. * filer: keep a short reconnect window on a fresh-minted embedded master A fresh TopologyId mint proves the master's persisted state was empty, not that no external filer survived its loss — one can still be mid-reconnect on its keepalive cycle when the confirm starts. Ending the empty-scan confirm on the first poll let the fresh filer register and serve routed reads before that peer was found and backfilled, so existing files could read as missing. The fresh-mint path now shortens the confirm to a few keepalive cycles instead of skipping it: a reconnecting filer still lands in time, while a genuinely fresh cluster announces after ~3s rather than ~10s. * volume: skip the heartbeat loop when no seed masters are configured An empty seed list made the outer loop spin with no await between passes and no path to observe the stop flag. A masterless volume server has nothing to register with, so the heartbeat exits after a warning. * volume: skip the heartbeat loop when no master addresses are given (rust) Mirrors the Go heartbeat: an empty seed list spun the outer loop with no await and no shutdown checks. Exit after a warning so library callers get the same behavior as the binary path that validates the list up front. * filer: re-check the fresh-mint hint on each confirm pass The topology id is minted after raft election, which can land after the empty-scan confirm has already started — a one-time check at entry missed the mint and held the full window. Evaluate the hint on every poll and shorten the confirm from the first observed mint, still capped by the normal confirm deadline. * master: mark a fresh topology mint only when the volume bound is zero Recovery restores the max volume id unconditionally but the topology id only when present, so an upgraded or partially rebuilt master can mint a new id while still holding prior volume state. Marking fresh there would let the embedded filer shortcut its peer confirm on a cluster that is not empty; only mark the mint fresh when no bound survived. * filer: drop the fresh-mint shortcut once a peer has been observed --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 08:08:26 -07:00
Chris LuandDevin	da1d7444b8	master: shorten the first-write hold after a fresh topology id (#12059) ... * master: release the post-mint allocation hold after a quiet pulse DeferAllocation armed a fixed warmup window whenever the leader minted a fresh topology id, so the first write on a brand-new cluster blocked for the whole window even though no prior state could exist. The window still has to cover a rebuilt master's veteran servers, which report on their first post-reconnect heartbeat. Track the last heartbeat that could carry prior cluster state — a server joining, held volumes or shards, a nonzero id bound, or a foreign topology id — and lift the hold one pulse after it goes quiet, capped by the original window. * master: make the post-mint allocation hold configurable master.volume_growth.allocation_deferral overrides the warmup window a leader holds volume id allocation for after minting a fresh topology id. Unset keeps the existing window; "0s" disables the hold for setups that know the cluster is fresh (test harnesses, single-node labs); any other duration replaces the cap while the quiet-pulse early release still applies. * master: re-check the allocation deadline while it is held NextVolumeId slept for the whole computed delay, so a heartbeat landing mid-sleep that pushed the quiet deadline out was ignored — the pending allocation could still run ahead of a straggler's report. Wait in short slices, re-evaluating the deadline and the growth freeze each round. * master: count only ids above the issued bound as prior-state evidence Heartbeat echoes of volumes and shards this master allocated during the hold refreshed the quiet window, so each growth allocation re-extended the deferral until the hard cap. Evidence now requires an id beyond the bound this leader has issued, a higher reported bound, or a foreign topology id; a joining server (dn == nil) still counts once. * master: reject an unparseable allocation_deferral instead of disabling the hold Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: do not count a skipped heartbeat as a joining server Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: assert the allocation hold extends past the original quiet deadline Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: release the post-mint hold after two quiet pulses One pulse proved short for a veteran that reconnects on a staggered restart: with only five seconds of quiet, allocation could resume and reuse ids before a slower server landed its first heartbeat. Two pulses still halves the fresh-cluster first-write stall versus the full warmup cap, and known-fresh setups keep the "0s" override. * master: pick once more when assign times out as growth concludes The pick loop's final attempt can predate the growth it triggered concluding: growth registers its volumes in the layout before clearing the pending flag, so a timeout landing on that boundary used to shed a hard 406 over volumes that had just arrived. Pick once more after the loop in both the http and grpc assign paths before deciding how to shed. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-25 06:17:42 -07:00
Chris Lu	2874bd05fb	Freeze volume growth on topology id mismatch (#11763) (#11820) ... * pb: ResumeVolumeGrowth rpc for operator-driven growth resume (#11763) * master: freeze volume growth on topology id mismatch (#11763) * shell: volume.growth.resume command (#11763) * master: replicate the growth freeze and gate it on veteran heartbeats (#11763) * master: block allocation while the topology id is being minted DeferAllocation only armed after the topology id persisted, so growth requests arriving during init ran ahead of the warmup window. Gate NextVolumeId on an init-pending flag spanning the whole mint. * master: stash reported topology ids until the local id is minted (#11763) * drain stashed topology id reports when the id arrives outside the mint path * persist the growth-freeze flag in the mirror file and raft snapshots * log hashicorp raft marshal and apply errors in applyGrowthFreeze	2026-10-20 07:34:31 -07:00
Chris Lu	15f4a97029	fix: improve raft leader election reliability and failover speed (#8692) ... * fix: clear raft vote state file on non-resume startup The seaweedfs/raft library v1.1.7 added a persistent `state` file for currentTerm and votedFor. When RaftResumeState=false (the default), the log, conf, and snapshot directories are cleared but this state file was not. On repeated restarts, different masters accumulate divergent terms, causing AppendEntries rejections and preventing leader election. Fixes #8690 * fix: recover TopologyId from snapshot before clearing raft state When RaftResumeState=false clears log/conf/snapshot, the TopologyId (used for license validation) was lost. Now extract it from the latest snapshot before cleanup and restore it on the topology. Both seaweedfs/raft and hashicorp/raft paths are handled, with a shared recoverTopologyIdFromState helper in raft_common.go. * fix: stagger multi-master bootstrap delay by peer index Previously all masters used a fixed 1500ms delay before the bootstrap check. Now the delay is proportional to the peer's sorted index with randomization (matching the hashicorp raft path), giving the designated bootstrap node (peer 0) a head start while later peers wait for gRPC servers to be ready. Also adds diagnostic logging showing why DoJoinCommand was or wasn't called, making leader election issues easier to diagnose from logs. * fix: skip unreachable masters during leader reconnection When a master leader goes down, non-leader masters still redirect clients to the stale leader address. The masterClient would follow these redirects, fail, and retry — wasting round-trips each cycle. Now tryAllMasters tracks which masters failed within a cycle and skips redirects pointing to them, reducing log spam and connection overhead during leader failover. * fix: take snapshot after TopologyId generation for recovery After generating a new TopologyId on the leader, immediately take a raft snapshot so the ID can be recovered from the snapshot on future restarts with RaftResumeState=false. Without this, short-lived clusters would lose the TopologyId on restart since no automatic snapshot had been taken yet. * test: add multi-master raft failover integration tests Integration test framework and 5 test scenarios for 3-node master clusters: - TestLeaderConsistencyAcrossNodes: all nodes agree on leader and TopologyId - TestLeaderDownAndRecoverQuickly: leader stops, new leader elected, old leader rejoins as follower - TestLeaderDownSlowRecover: leader gone for extended period, cluster continues with 2/3 quorum - TestTwoMastersDownAndRestart: quorum lost (2/3 down), recovered when both restart - TestAllMastersDownAndRestart: full cluster restart, leader elected, all nodes agree on TopologyId * fix: address PR review comments - peerIndex: return -1 (not 0) when self not found, add warning log - recoverTopologyIdFromSnapshot: defer dir.Close() - tests: check GetTopologyId errors instead of discarding them * fix: address review comments on failover tests - Assert no leader after quorum loss (was only logging) - Verify follower cs.Leader matches expected leader via ServerAddress.ToHttpAddress() comparison - Check GetTopologyId error in TestTwoMastersDownAndRestart	2026-03-18 23:28:07 -07:00
Chris Lu	753e1db096	Prevent split-brain: Persistent ClusterID and Join Validation (#8022) ... * Prevent split-brain: Persistent ClusterID and Join Validation - Persist ClusterId in Raft store to survive restarts. - Validate ClusterId on Raft command application (piggybacked on MaxVolumeId). - Prevent masters with conflicting ClusterIds from joining/operating together. - Update Telemetry to report the persistent ClusterId. * Refine ClusterID validation based on feedback - Improved error message in cluster_commands.go. - Added ClusterId mismatch check in RaftServer.Recovery. * Handle Raft errors and support Hashicorp Raft for ClusterId - Check for errors when persisting ClusterId in legacy Raft. - Implement ClusterId generation and persistence for Hashicorp Raft leader changes. - Ensure consistent error logging. * Refactor ClusterId validation - Centralize ClusterId mismatch check in Topology.SetClusterId. - Simplify MaxVolumeIdCommand.Apply and RaftServer.Recovery to rely on SetClusterId. * Fix goroutine leak and add timeout - Handle channel closure in Hashicorp Raft leader listener. - Add timeout to Raft Apply call to prevent blocking. * Fix deadlock in legacy Raft listener - Wrap ClusterId generation/persistence in a goroutine to avoid blocking the Raft event loop (deadlock). * Rename ClusterId to SystemId - Renamed ClusterId to SystemId across the codebase (protobuf, topology, server, telemetry). - Regenerated telemetry.pb.go with new field. * Rename SystemId to TopologyId - Rename to SystemId was intermediate step. - Final name is TopologyId for the persistent cluster identifier. - Updated protobuf, topology, raft server, master server, and telemetry. * Optimize Hashicorp Raft listener - Integrated TopologyId generation into existing monitorLeaderLoop. - Removed extra goroutine in master_server.go. * Fix optimistic TopologyId update - Removed premature local state update of TopologyId in master_server.go and raft_hashicorp.go. - State is now solely updated via the Raft state machine Apply/Restore methods after consensus. * Add explicit log for recovered TopologyId - Added glog.V(0) info log in RaftServer.Recovery to print the recovered TopologyId on startup. * Add Raft barrier to prevent TopologyId race condition - Implement ensureTopologyId helper method - Send no-op MaxVolumeIdCommand to sync Raft log before checking TopologyId - Ensures persisted TopologyId is recovered before generating new one - Prevents race where generation happens during log replay * Serialize TopologyId generation with mutex - Add topologyIdGenLock mutex to MasterServer struct - Wrap ensureTopologyId method with lock to prevent concurrent generation - Fixes race where event listener and manual leadership check both generate IDs - Second caller waits for first to complete and sees the generated ID * Add TopologyId recovery logging to Apply method - Change log level from V(1) to V(0) for visibility - Log 'Recovered TopologyId' when applying from Raft log - Ensures recovery is visible whether from snapshot or log replay - Matches Recovery() method logging for consistency * Fix Raft barrier timing issue - Add 100ms delay after barrier command to ensure log application completes - Add debug logging to track barrier execution and TopologyId state - Return early if barrier command fails - Prevents TopologyId generation before old logs are fully applied * ensure leader * address comments * address comments * redundant * clean up * double check * refactoring * comment	2026-01-18 14:02:34 -08:00