* mount: bound each filer call so a wedged filer cannot hold the failover walk
WithFilerClient callers run their RPCs on context.Background(), so a filer
whose transport stalls without refusing (stopped peer, black-holed
connection) blocks the call forever and the walk never reaches the
remaining filer addresses. Run each attempt under a timeout; on expiry
drop the cached channel so the abandoned call is unblocked and the next
address is tried.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: time out wedged mutation streams and rotate to the next filer
doUnary and Rename wait on stream Send/Recv under context.Background(); a
dead transport never delivers an error, so writes hang without ever
reaching the unary fallback. Bound the waits and tear the stream down on
timeout; teardown also advances the sticky filer index so the next stream
opens against a different filer instead of redialing the dead one.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: cover filer failover past a wedged filer
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: cancel filer attempts by context and unwind them before retry
Review follow-ups on the filer failover:
- callFiler derives a per-attempt context instead of only abandoning the
wait, so a timed-out attempt's RPCs die on their own without tearing
down the shared cached channel. The walk now also waits for the
attempt's callback to unwind before trying the next filer, so two runs
of the same closure cannot race on the caller's captured results.
Callbacks that ignore the attempt context are still forced out by
dropping the channel, as a last resort after a grace period.
- teardownStream captures the dying stream's filer index and rotates it
while holding m.mu, so it can never rotate a successor generation's
filer by mistake.
- The unary rename fallback bounds silence on its dedicated stream the
way the mux does, instead of trusting Recv on a wedged filer.
- downloadRemoteEntry opts out of the metadata-call bound: a healthy
whole-file download can legitimately outlast it.
* mount: count only wire silence against the rename timeout
Applying a rename event can outlast the silence bound on a healthy
filer — a directory rename migrates an open child's DLM lock under
another writer. Stop the mux's response timer for the local work and
re-arm it before the next wait; the fallback stream's watchdog defers
while the handler runs instead of cancelling a live stream.
* mount: restart the rename fallback's silence bound after each event
The bound covers the wait for the next response, not the rename's whole
lifetime — a healthy rename that keeps sending events can outlast it.
Re-arm after each handled event; the defer-while-handling stays for
local work that outlasts the interval on its own.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A Canceled/DeadlineExceeded from the caller's per-request context was
treated like a dead channel: it closed the shared cached ClientConn and
cancelled every other in-flight RPC on it with "the client connection is
closing". Under a burst of concurrent chunk assigns (e.g. a large S3
multipart upload) one slow assign hitting its 10s attempt timeout could
poison the connection for all the rest, cascading into a flood of 500s.
Thread the caller's context into shouldInvalidateConnection and only
invalidate on Canceled/DeadlineExceeded while that context is still live,
which isolates the genuine stale-channel signal (a peer restart behind a
k8s Service VIP). To carry the context, add a ctx parameter to the
existing WithGrpcClient, WithMasterClient, and WithMasterServerClient; the
master assign and volume-lookup paths pass their per-attempt context and
every other caller passes context.Background().
* remove old raft servers if they don't answer to pings for too long
add ping durations as options
rename ping fields
fix some todos
get masters through masterclient
raft remove server from leader
use raft servers to ping them
CheckMastersAlive for hashicorp raft only
* prepare blocking ping
* pass waitForReady as param
* pass waitForReady through all functions
* waitForReady works
* refactor
* remove unneeded params
* rollback unneeded changes
* fix
When multiple filer requests are in-flight and the current filer
disappears and a new one is selected by the first goroutine, then
there can be a lot of race conditions while retrieving the current
filer.
Therefore, load/save the current filer index atomically.
