* wdclient: bound the wait for a master leader by the caller's context
WithClient waited on GetMaster with context.Background(), so a caller that
arrived while no master leader was known parked in a 200ms poll loop until one
appeared, whatever deadline it had already set on the RPC. Each retry above it
then left another goroutine in the same wait.
Take the context in WithClient and WithClientCustomGetMaster and hand it to
GetMaster, and stop the retry loop once it is done. The dial keeps
context.Background(): fn brings its own RPC context, so a cancellation seen
here cannot be attributed to the shared connection.
Call sites pass whatever they hold: the request context in the filer's
CollectionList, DeleteCollection and Statistics handlers and in the credential
store's propagation, the operation context in the shell's s3.bucket.delete and
the kafka gateway's broker and filer discovery, and context.Background() where
there is none - the shell commands, the admin dashboard wrapper, and the
exclusive locker's initial lease. The locker's release keeps its own
uncancelled context so a slow unlock cannot turn into a ghost lock.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: test that WithClient gives up with the caller's context
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: cut the master retry backoff short when the caller gives up
util.Retry sleeps unconditionally between attempts, so a transient error
arriving just before the caller's deadline still cost it a full backoff step.
Use the context-aware util.RetryWithBackoff, the same helper the volume lookup
in this file already uses.
Two call sites went with it: the shell's lock-holder lookup builds its three
second bound before WithClient so it also covers finding the leader, as its
comment already promised, and the filer's post-delete collection cleanup goes
back to an uncancelled context - the entry is already gone, so a caller that
hung up must not leave the collection behind.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: test that a cancel during backoff ends the retry
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
The existing multi-disk EC integration test asserts on shard counts. Counting
cannot tell a healthy volume from one a repair reassembled out of the wrong
inputs — both have fourteen shards. This drives the real shell commands
(ec.encode, ec.balance, ec.rebuild) against a live three-node, four-disk
cluster and reads the stored bytes back after every step, so a rebuild that
produced fourteen plausible-but-wrong shards fails here.
An 8 KB random payload is stored, then encoded, balanced, damaged (two shard
files removed and the servers restarted so the master relearns the reduced
set from disk), and rebuilt. The rebuild output matches the shape of the
support case that motivated this — "rebuildOneEcVolume", "missing shard N.0",
"copied N.1 from ..." — and the payload is verified identical after each of
upload, encode, balance, shard loss, and rebuild.
Two ordering facts the test pins, both of which cost real debugging time:
ec.rebuild is driven by the master's topology, not disk truth, so shards must
be relearned (via restart) before a repair can target the right set; and the
shell lock is dropped when the restart disconnects the master, so it has to be
retaken before the rebuild.
Chris Lu