* pb: ResumeVolumeGrowth rpc for operator-driven growth resume (#11763)
* master: freeze volume growth on topology id mismatch (#11763)
* shell: volume.growth.resume command (#11763)
* master: replicate the growth freeze and gate it on veteran heartbeats (#11763)
* master: block allocation while the topology id is being minted
DeferAllocation only armed after the topology id persisted, so growth
requests arriving during init ran ahead of the warmup window. Gate
NextVolumeId on an init-pending flag spanning the whole mint.
* master: stash reported topology ids until the local id is minted (#11763)
* drain stashed topology id reports when the id arrives outside the mint path
* persist the growth-freeze flag in the mirror file and raft snapshots
* log hashicorp raft marshal and apply errors in applyGrowthFreeze
* design: durable volume-id allocation boundary (#11763)
* master: mirror max volume id to a file outside raft (#11763)
The allocation bound lived only in raft state, so a master that lost it
rebuilt the bound from whatever volume servers were connected and could
reissue ids held by offline servers.
NextVolumeId now persists the bound to <mdir>/max_volume_id before the
raft apply, and both raft implementations persist on apply and recovery
so follower mirrors stay current. The file also carries the topology id,
which survives raft state cleanup the same way the snapshot recovery did.
* master: serialize and fully sync the max volume id mirror (#11763)
* master: rewrite the mirror when the topology id changes at the same bound
A same-bound raft apply carrying a new topology id was skipped by the
lastPersistedMaxVolumeId check, leaving the mirrored topology id empty
until the next allocation. Track the mirrored topology id too and
rewrite when either value differs.
* drop the design doc from the PR stack
* Prevent split-brain: Persistent ClusterID and Join Validation
- Persist ClusterId in Raft store to survive restarts.
- Validate ClusterId on Raft command application (piggybacked on MaxVolumeId).
- Prevent masters with conflicting ClusterIds from joining/operating together.
- Update Telemetry to report the persistent ClusterId.
* Refine ClusterID validation based on feedback
- Improved error message in cluster_commands.go.
- Added ClusterId mismatch check in RaftServer.Recovery.
* Handle Raft errors and support Hashicorp Raft for ClusterId
- Check for errors when persisting ClusterId in legacy Raft.
- Implement ClusterId generation and persistence for Hashicorp Raft leader changes.
- Ensure consistent error logging.
* Refactor ClusterId validation
- Centralize ClusterId mismatch check in Topology.SetClusterId.
- Simplify MaxVolumeIdCommand.Apply and RaftServer.Recovery to rely on SetClusterId.
* Fix goroutine leak and add timeout
- Handle channel closure in Hashicorp Raft leader listener.
- Add timeout to Raft Apply call to prevent blocking.
* Fix deadlock in legacy Raft listener
- Wrap ClusterId generation/persistence in a goroutine to avoid blocking the Raft event loop (deadlock).
* Rename ClusterId to SystemId
- Renamed ClusterId to SystemId across the codebase (protobuf, topology, server, telemetry).
- Regenerated telemetry.pb.go with new field.
* Rename SystemId to TopologyId
- Rename to SystemId was intermediate step.
- Final name is TopologyId for the persistent cluster identifier.
- Updated protobuf, topology, raft server, master server, and telemetry.
* Optimize Hashicorp Raft listener
- Integrated TopologyId generation into existing monitorLeaderLoop.
- Removed extra goroutine in master_server.go.
* Fix optimistic TopologyId update
- Removed premature local state update of TopologyId in master_server.go and raft_hashicorp.go.
- State is now solely updated via the Raft state machine Apply/Restore methods after consensus.
* Add explicit log for recovered TopologyId
- Added glog.V(0) info log in RaftServer.Recovery to print the recovered TopologyId on startup.
* Add Raft barrier to prevent TopologyId race condition
- Implement ensureTopologyId helper method
- Send no-op MaxVolumeIdCommand to sync Raft log before checking TopologyId
- Ensures persisted TopologyId is recovered before generating new one
- Prevents race where generation happens during log replay
* Serialize TopologyId generation with mutex
- Add topologyIdGenLock mutex to MasterServer struct
- Wrap ensureTopologyId method with lock to prevent concurrent generation
- Fixes race where event listener and manual leadership check both generate IDs
- Second caller waits for first to complete and sees the generated ID
* Add TopologyId recovery logging to Apply method
- Change log level from V(1) to V(0) for visibility
- Log 'Recovered TopologyId' when applying from Raft log
- Ensures recovery is visible whether from snapshot or log replay
- Matches Recovery() method logging for consistency
* Fix Raft barrier timing issue
- Add 100ms delay after barrier command to ensure log application completes
- Add debug logging to track barrier execution and TopologyId state
- Return early if barrier command fails
- Prevents TopologyId generation before old logs are fully applied
* ensure leader
* address comments
* address comments
* redundant
* clean up
* double check
* refactoring
* comment
Chris Lu