grpc: don't tear down the shared master connection on a caller's own timeout (#9775)

A Canceled/DeadlineExceeded from the caller's per-request context was
treated like a dead channel: it closed the shared cached ClientConn and
cancelled every other in-flight RPC on it with "the client connection is
closing". Under a burst of concurrent chunk assigns (e.g. a large S3
multipart upload) one slow assign hitting its 10s attempt timeout could
poison the connection for all the rest, cascading into a flood of 500s.

Thread the caller's context into shouldInvalidateConnection and only
invalidate on Canceled/DeadlineExceeded while that context is still live,
which isolates the genuine stale-channel signal (a peer restart behind a
k8s Service VIP). To carry the context, add a ctx parameter to the
existing WithGrpcClient, WithMasterClient, and WithMasterServerClient; the
master assign and volume-lookup paths pass their per-attempt context and
every other caller passes context.Background().
This commit is contained in:
Chris Lu authored and GitHub committed 2026-06-01 15:11:02 -07:00
1 parent dfa86b4313
commit 2386fa550a
31 files changed
+129 -63

No files matched your search

+1 -1
View File
@@ -24,7 +24,7 @@ const iamRequestTimeout = 30 * time.Second
// applied — callers can derive child contexts but should not need their own
// timeout boilerplate.
func (ce *CommandEnv) withIamClient(fn func(ctx context.Context, client iam_pb.SeaweedIdentityAccessManagementClient) error) error {
return pb.WithGrpcClient(false, 0, func(conn *grpc.ClientConn) error {
return pb.WithGrpcClient(context.Background(), false, 0, func(conn *grpc.ClientConn) error {
ctx, cancel := context.WithTimeout(iamAdminAuthContext(context.Background()), iamRequestTimeout)
defer cancel()
return fn(ctx, iam_pb.NewSeaweedIdentityAccessManagementClient(conn))