* master: scope the startup capacity shed to a truly empty topology
The retryable "no volume server capacity registered yet" shed checked
capacity for the requested disk type, so a cluster serving only other
media -- where that capacity will never register -- shed every assign
until the client's deadline instead of failing fast. An unsteered write
to such a cluster hung for its full HTTP deadline and surfaced "context
deadline exceeded" in place of "No writable volumes". Shed only while
no disk type has any registered capacity, and name the unserved medium
in the fast failure.
Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R
* master: name the unserved medium for every fail-fast caller
The diagnostic sat in the growth-initiator block, so a follower joining
an in-flight growth and a growth-disabled master failed the same way
with only the generic pick error. Wrap at the fail-fast break instead,
which every caller reaches, and cover all three paths in the test.
Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R
An assign arriving before any volume server has heartbeated saw zero
available space and failed outright with a plain error no client retries,
so the first write to a fresh bucket answered 500 while the cluster was
still starting. Distinguish a topology with no registered capacity from a
genuinely full one: fail fast only when registered capacity is exhausted,
and shed ResourceExhausted otherwise so the client's retry budget rides
out the startup window.
Claude-Session: https://claude.ai/code/session_018G9kWFgy8BaBAEkYV3YL9n
* fix(master): let the growth initiator wait for the growth it triggered
The growth-in-flight shed also fired on the request that initiated the
growth: it sets the pending flag right before the shed check, so a
cold-start assign enqueued growth and immediately failed itself with
"volume growth in progress". With no concurrent assigns around to pick
up the freshly grown volume, a single writer against an empty cluster
never completes a write despite ample free space.
Claim the pending flag with a compare-and-swap so exactly one request
becomes the initiator, triggering growth at most once, and let it wait
for that growth to land. Everyone else still sheds retryably instead of
pinning a goroutine: followers behind an in-flight growth, an initiator
whose growth concluded without yielding a writable volume, and an
initiator whose growth outlives the 10s wait budget, which previously
surfaced a non-retryable error (gRPC Unknown, HTTP 406) even though a
retry would have succeeded moments later.
* fix(master): stop assign waits when the request is cancelled
The assign retry loops slept through client cancellation, keeping a
goroutine spinning for the rest of the 10s budget after the caller had
gone; StreamAssign also ran assigns on a background context detached
from the stream. Wait on the request context and pass the stream
context through.
* topology: drop the unconditional grow-request setter
Growth is only claimed through AddGrowRequestIfAbsent's compare-and-swap
now; keeping the raw Store(true) around invites the check-then-set race
back.
* test: cover cold-start first write with a real cluster
Boot a fresh master plus three empty volume servers and require the very
first assign - HTTP and gRPC, each on a cold volume layout, no client
retries - to complete a write. The assign that triggers volume growth
must wait for it rather than answering "volume growth in progress";
unit tests stub the topology, so only a real cluster exercises the
assign-grow-wait path end to end.
Under a herd of concurrent assigns with no writable volume, Assign spun
PickForWrite for the full 10s timeout, pinning a goroutine per request and
starving the master of the cycles it needs to process growth and answer
heartbeats. When growth is the relevant remedy and already in flight, stop
spinning: if free space exists, shed with a fast retryable error so clients
back off and retry once growth lands; if the cluster is out of space, fail fast
with the real out-of-space error instead of masking it as retryable.
The gRPC shed uses ResourceExhausted, not Unavailable: operation.Assign retries
it, but the client connection layer doesn't treat it as a dead channel, so a
per-request shed across a herd doesn't tear down the shared master connection
and cancel every other in-flight assign. The HTTP dirAssignHandler sheds with
503 + Retry-After.
Chris Lu