* Replay metadata log chunks the way the mount reads every other chunk
The subscription's log-chunk replay built its own lookup, which always
resolves volume server addresses. A mount started with
-volumeServerAccess=filerProxy cannot reach those, so every fresh
subscription failed on the previous minute's persisted segment and
resubscribed a second later, forever. Take the lookup from the caller
instead; the mount hands over the one it uses for file reads, which also
keeps publicUrl and the bounded location cache in play.
Claude-Session: https://claude.ai/code/session_01NGqrxYj7cHUpSrL249n3Z6
* Keep a log chunk read failure off the filer connection
A metadata subscriber reads persisted log chunks over HTTP from volume
servers and hands whatever went wrong back as the subscription's error.
"connection refused" from a volume server then matched the transport
patterns that decide a gRPC channel is dead, so every failed replay
closed the shared filer ClientConn and cancelled the assign and upload
RPCs riding on it with "the client connection is closing". Mark those
read failures so they are judged for what they are.
Claude-Session: https://claude.ai/code/session_01NGqrxYj7cHUpSrL249n3Z6
* grpc: a non-cancellable context is no evidence of a stale channel
shouldInvalidateConnection only invalidates on Canceled/DeadlineExceeded
while the context handed to WithGrpcClient is still live, so that an RPC
timing out on its own does not close the shared cached ClientConn and
cancel every other in-flight RPC on it. context.Background()/TODO never
expire, so Err() stays nil forever and that guard always answered
"invalidate" - and Background is what almost every caller passes, the S3
gateway included.
One S3 request whose RPC rode an abandoned HTTP request context therefore
closed the shared filer connection, and every multipart part in flight
died with "the client connection is closing", surfacing to the client as
400 InvalidRequest.
Only a cancellable context bounds an RPC attempt, so require one before
reading it. A genuinely stale channel (a peer restart behind a stable L4
endpoint) surfaces as Unavailable, which invalidates on its own branch.
* grpc: a bystander of a connection teardown is not a stale-channel witness
gRPC raises ErrClientConnClosing locally, before an RPC reaches the wire,
when this process has already closed the ClientConn. Every caller that
touches a channel during another goroutine's teardown gets it, so reading
it as a stale-channel signal lets one teardown re-arm itself across the
whole herd of callers it just cancelled.
The cached-connection version check keeps those callers from closing a
replacement channel, but the streaming path invalidates by address alone
and has no such guard.
* grpc: end a stream without dropping the peer connection under it
A streaming caller gets its own ClientConn, but on any error it also drops
the cached non-streaming ClientConn every request handler shares with that
peer, to recover a peer restart hidden behind a stable L4 endpoint. Any
error includes the ordinary ones: a metadata subscription that reached its
stop point, a follow callback that refused an event, a caller that gave up.
The S3 gateway follows filer metadata on such a stream and reconnects
forever, so each ordinary end of it cancelled every S3 request in flight
against the filer. Drop the shared channel only for errors that say the
peer went away, which is what invalidation is for.
* test: close the connections the cascade tests leave cached
Each test swaps in a fresh connection cache and restores the previous one,
dropping its own entries without closing them, so the ClientConn's
transport and reconnect goroutines outlive the fake filer they dialed.
* grpc: say why ErrClientConnClosing's deprecation notice does not apply
It points at codes.Canceled, which is the code this function exists to
disambiguate. Only the message distinguishes a teardown a caller merely
walked into, so the sentinel stays.
Chris Lu