From 911d4e5f318d0ccc5681c32cc41338cb803c1244 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Tue, 18 Aug 2026 17:45:19 -0700 Subject: [PATCH] Pin grpc-go to v1.82.0 to avoid a Unix-socket transport deadlock grpc-go v1.82.1 broadened the control-buffer throttle to count registerStream and cleanupStream. Those are per-RPC bookkeeping items that produce no wire traffic, so any connection carrying enough concurrent RPCs latches throttling in both directions at once: both peers stop reading, both loopyWriters block writing to a full socket, and the connection never recovers. Local gRPC goes over Unix sockets, whose send buffers are small and do not autotune, so loopyWriter blocks readily and the deadlock is easy to reach. TCP loopback does not trigger it. weed mini wedges at roughly 320 concurrent S3 PUTs: throughput collapses to tens of ops/s with 65s stalls and no filer handlers running. Pinned back until upstream is fixed. --- go.mod | 16 ++++++++++++++++ go.sum | 2 ++ 2 files changed, 18 insertions(+) diff --git a/go.mod b/go.mod index c652f977e..1485a6081 100644 --- a/go.mod +++ b/go.mod @@ -527,3 +527,19 @@ replace github.com/apache/thrift => github.com/apache/thrift v0.23.1-0.202604291 // through rclone's internxt backend, which calls IsMnemonicValid and NewSeed. // cosmos/go-bip39 is a maintained, API-compatible fork. replace github.com/tyler-smith/go-bip39 => github.com/cosmos/go-bip39 v1.0.0 + +// grpc-go v1.82.1 started counting registerStream/cleanupStream toward the +// control-buffer throttle limit. Those are per-RPC bookkeeping items that emit +// no wire traffic, so a connection carrying enough concurrent RPCs latches +// throttling in both directions at once and deadlocks for good: both peers stop +// reading, both loopyWriters block writing to a full socket. +// +// It triggers whenever the socket send buffer is too small to absorb the burst +// loopyWriter emits for the in-flight streams, so it is about buffer size, not +// transport: TCP forced to a 64KB buffer deadlocks, a Unix socket with a 2MB +// buffer does not. Local gRPC rides Unix sockets, whose buffers are small and +// fixed (208KB on Linux, 8KB on macOS), which is why weed mini wedges under +// concurrent S3 writes; a remote filer over TCP is exposed too wherever +// send buffers are constrained. Pin to the last unaffected release until +// grpc/grpc-go#9330 is fixed. +replace google.golang.org/grpc => google.golang.org/grpc v1.82.0 diff --git a/go.sum b/go.sum index 83731e42a..7fd559443 100644 --- a/go.sum +++ b/go.sum @@ -2838,6 +2838,8 @@ google.golang.org/grpc v1.51.0/go.mod h1:wgNDFcnuBGmxLKI/qn4T+m5BtEBYXJPvibbUPsA google.golang.org/grpc v1.52.0/go.mod h1:pu6fVzoFb+NBYNAvQL08ic+lvB2IojljRYuun5vorUY= google.golang.org/grpc v1.53.0/go.mod h1:OnIrk0ipVdj4N5d9IUoFUx72/VlD7+jUsHwZgwSMQpw= google.golang.org/grpc v1.55.0/go.mod h1:iYEXKGkEBhg1PjZQvoYEVPTDkHo1/bjTnfwTeGONTY8= +google.golang.org/grpc v1.82.0 h1:vguDnZUPjE26w09A63VoxZPnvPjB5Riyc0mkXPFmAIU= +google.golang.org/grpc v1.82.0/go.mod h1:yzTZ1TB1Z3SG+LIYaI+WiE8D5+PZ3ArnrSp8zF3+/ZA= google.golang.org/grpc v1.84.0-dev.0.20260723093437-b6eac429d7b6 h1:HfjjkdGIa8u9sP9EW5WCygy0kQDuTI/Tax4j//t24Fo= google.golang.org/grpc v1.84.0-dev.0.20260723093437-b6eac429d7b6/go.mod h1:ljCht0DrxQrXBDRTZp52Qxh3Ffk8CdYm2sj4O2QN2C0= google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0/go.mod h1:6Kw0yEErY5E/yWrBtf03jp27GLLJujG4z/JK95pnjjw=