* wdclient: bound the wait for a master leader by the caller's context
WithClient waited on GetMaster with context.Background(), so a caller that
arrived while no master leader was known parked in a 200ms poll loop until one
appeared, whatever deadline it had already set on the RPC. Each retry above it
then left another goroutine in the same wait.
Take the context in WithClient and WithClientCustomGetMaster and hand it to
GetMaster, and stop the retry loop once it is done. The dial keeps
context.Background(): fn brings its own RPC context, so a cancellation seen
here cannot be attributed to the shared connection.
Call sites pass whatever they hold: the request context in the filer's
CollectionList, DeleteCollection and Statistics handlers and in the credential
store's propagation, the operation context in the shell's s3.bucket.delete and
the kafka gateway's broker and filer discovery, and context.Background() where
there is none - the shell commands, the admin dashboard wrapper, and the
exclusive locker's initial lease. The locker's release keeps its own
uncancelled context so a slow unlock cannot turn into a ghost lock.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: test that WithClient gives up with the caller's context
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: cut the master retry backoff short when the caller gives up
util.Retry sleeps unconditionally between attempts, so a transient error
arriving just before the caller's deadline still cost it a full backoff step.
Use the context-aware util.RetryWithBackoff, the same helper the volume lookup
in this file already uses.
Two call sites went with it: the shell's lock-holder lookup builds its three
second bound before WithClient so it also covers finding the leader, as its
comment already promised, and the filer's post-delete collection cleanup goes
back to an uncancelled context - the entry is already gone, so a caller that
hung up must not leave the collection behind.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: test that a cancel during backoff ends the retry
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* proto: add RaftLeadershipTransfer RPC for forced leader change
Add new gRPC RPC and messages for leadership transfer:
- RaftLeadershipTransferRequest: optional target_id and target_address
- RaftLeadershipTransferResponse: previous_leader and new_leader
This enables graceful leadership transfer before master maintenance,
reducing errors in filers during planned maintenance windows.
Ref: https://github.com/seaweedfs/seaweedfs/issues/7527
* proto: regenerate Go files for RaftLeadershipTransfer
Generated from master.proto changes.
* master: implement RaftLeadershipTransfer gRPC handler
Add gRPC handler for leadership transfer with support for:
- Transfer to any eligible follower (when target_id is empty)
- Transfer to a specific server (when target_id and target_address are provided)
Uses hashicorp/raft LeadershipTransfer() and LeadershipTransferToServer() APIs.
Returns the previous and new leader in the response.
* shell: add cluster.raft.leader.transfer command
Add weed shell command for graceful leadership transfer:
- Displays current cluster status before transfer
- Supports auto-selection of target (any eligible follower)
- Supports targeted transfer with -id and -address flags
- Provides clear feedback on success/failure with troubleshooting tips
Usage:
cluster.raft.leader.transfer
cluster.raft.leader.transfer -id <server_id> -address <grpc_address>
* master: add unit tests for raft gRPC handlers
Add tests covering:
- RaftLeadershipTransfer with no raft initialized
- RaftLeadershipTransfer with target_id but no address
- RaftListClusterServers with no raft initialized
- RaftAddServer with no raft initialized
- RaftRemoveServer with no raft initialized
These tests verify error handling when raft is not configured.
* shell: add tests for cluster.raft.leader.transfer command
Add tests covering:
- Command name and help text validation
- HasTag returns false for ResourceHeavy
- Validation of -id without -address
- Argument parsing with unknown flags
* master: clarify that leadership transfer requires -raftHashicorp
The default raft implementation (seaweedfs/raft, a goraft fork) does not
support graceful leadership transfer. This feature is only available when
using hashicorp raft (-raftHashicorp=true).
Update error messages and help text to make this requirement clear:
- gRPC handler returns specific error for goraft users
- Shell command help text notes the requirement
- Added test for goraft case
* test: use strings.Contains instead of custom helper
Replace custom contains/containsHelper functions with the standard
library strings.Contains for better maintainability.
* shell: return flag parsing errors instead of swallowing them
- Return the error from flag.Parse() instead of returning nil
- Update test to explicitly assert error for unknown flags
* test: document integration test scenarios for Raft leadership transfer
Add comments explaining:
- Why these unit tests only cover 'Raft not initialized' scenarios
- What integration tests should cover (with multi-master cluster)
- hashicorp/raft uses concrete types that cannot be easily mocked
* fix: address reviewer feedback on tests and leader routing
- Remove misleading tests that couldn't properly validate their
documented behavior without a real Raft cluster:
- TestRaftLeadershipTransfer_GoraftNotSupported
- TestRaftLeadershipTransfer_ValidationTargetIdWithoutAddress
- Change WithClient(false) to WithClient(true) for RaftLeadershipTransfer
RPC to ensure the request is routed to the current leader
* Improve cluster.raft.transferLeader command
- Rename command from cluster.raft.leader.transfer to cluster.raft.transferLeader
- Add symmetric validation: -id and -address must be specified together
- Handle case where same leader is re-elected after transfer
- Add test for -address without -id validation
- Add docker compose file for 5-master raft cluster testing
Chris Lu