Files
seaweedfs/weed/pb/grpc_client_server.go
T
a3b2fd722d mount: report opens for reading with TouchAccessTime (-accessReport) (#12721)
* mount: report opens for reading with TouchAccessTime (-accessReport)

An open answered from the metadata cache never reaches the filer, so a
filer that tracks access cannot see that a cached file is still being
read. With -accessReport, the mount sends one TouchAccessTime per open
for reading, with the open time, whether the open was answered from the
cache or not. Write-only opens are not reported.

Reports go out in the background and never hold up the open: a report
that does not fit in the queue is dropped, and drops and failures are
logged once a minute. A filer that answers Unimplemented pauses
reporting for 10 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: address review of the access report

- Drop reports still queued when reporting pauses; the filer that answered
  Unimplemented would answer them the same way.
- Bound each filer attempt to 2s, so a filer that accepts the connection
  but never answers leaves time for the next one, and keep using the filer
  that answered.
- Fail over only when a filer cannot be reached. Any answer, Unimplemented
  included, ends the walk.
- Report an O_CREAT open of an existing file with read access, which comes
  through Create instead of Open. A new file has nothing to read and is not
  reported.
- Stop the reporter on unmount: stop ends the workers and cancels the
  report in flight.
- Say in the flag help that it has no effect unless the filer implements
  TouchAccessTime.
- Tests no longer wait a fixed time to see that nothing else was sent; they
  look at the queue of a reporter that is not started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fail an access report over only when the filer did not answer

A status the filer sent ends the walk even when its message contains a
word the transient classifier matches, so an Unimplemented answer still
pauses reporting and a rejected request is not resent elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a filer selection made while an access report failed over

A report that reached a later filer moves the mount there only if the
selection it started from is still current, so a slower report cannot
undo another call's failover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: share an access report's time with the filers not yet tried

Each attempt is capped by an equal share of the report time left, so a
row of stalled filers cannot use up the report before a healthy one later
in the list is tried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report the path an open for reading came through

A create that opens an existing hard link reports the name it was
given rather than the inode's first known name. Open reads the inode's
path from the inode table rather than the handle's saved path, which
other handle operations may rewrite concurrently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check the access report stop and failover tests without races

The post-stop report is checked on a reporter without workers, so no
worker can take an accepted report before the check, and the failover
test asserts that the stalled filer was tried first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* pb: keep the shared channel when the peer answers with its own status

gRPC never raises InvalidArgument, NotFound, AlreadyExists,
FailedPrecondition, OutOfRange or DataLoss itself, so such a status is
the peer's answer. It fell through to the string matching, and a message
naming a path like /transport closed the cached channel and cancelled
every other call on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: say which filer selection an access report keeps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: bound the in-flight call in the rejected-report channel test

The call held in flight went through touchAccessTime, whose 2 s attempt
cap could expire on a slow runner and fail the test on a healthy channel.
It now uses callFiler with the test deadline. The wait for it to reach
the filer also ends if it returns early or the deadline passes, and the
blocked handler is released on every exit path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report each file at most once per -accessReport.interval

Every read open used to send a TouchAccessTime, adding a filer metadata
call per open. A path reported within the interval (default 1h) is now
skipped; the last report times live in a bounded LRU. 0 keeps reporting
every open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report an open again once the file was modified since its last report

Within -accessReport.interval a read open was skipped even after a write,
so the filer kept an atime older than the mtime until the interval ran
out. relatime records that first read after a write; do the same by
comparing the entry's mtime with the last report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep the report time apart from the mtime it saw

Recording max(now, mtime) as the last report let a file dated a day ahead
suppress reports for a day plus the interval. Record when the report was
sent and the mtime the entry had, and treat any change of that mtime as a
modification, so neither clock can stretch or shorten the interval.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 14:11:11 -08:00

719 lines
27 KiB
Go

package pb
import (
"context"
"errors"
"fmt"
"math/rand/v2"
"net"
"net/http"
"os"
"runtime"
"strconv"
"strings"
"sync"
"time"
"github.com/google/uuid"
"github.com/seaweedfs/seaweedfs/weed/util/request_id"
"google.golang.org/grpc/metadata"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
"github.com/seaweedfs/seaweedfs/weed/util"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/keepalive"
"google.golang.org/grpc/status"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/mq_pb"
)
const (
Max_Message_Size = 1 << 30 // 1 GB
// gRPC keepalive settings - must be consistent between client and server
GrpcKeepAliveTime = 60 * time.Second // ping interval when no activity
GrpcKeepAliveTimeout = 20 * time.Second // ping timeout
GrpcKeepAliveMinimumTime = 20 * time.Second // minimum interval between client pings (enforcement)
)
var (
// cache grpc connections
grpcClients = make(map[string]*versionedGrpcClient)
grpcClientsLock sync.Mutex
// localGrpcSockets maps gRPC port numbers to Unix socket paths for
// gRPC services running in this process. Used by both the server side
// (to start serving on the socket) and the client side (to dial it
// instead of TCP for same-host RPCs).
localGrpcSockets = make(map[int]string)
// localGrpcHosts maps gRPC port → set of host strings that count as
// "this machine" for that port. Only dials whose target host is in this
// set are routed through the Unix socket; dials to other hosts that
// happen to use the same port number go over TCP as normal.
//
// Without this host check the hijack was keyed purely on port, so a
// remote server reusing one of our local gRPC ports (e.g. a standalone
// `weed volume -port=7334` defaulting `port.grpc=17334` against a
// `weed server` whose in-process volume socket is also on 17334) had
// its inbound RPCs silently rerouted to our local Unix socket — see
// issue #9254.
localGrpcHosts = make(map[int]map[string]struct{})
localGrpcSocketsLock sync.RWMutex
)
type versionedGrpcClient struct {
*grpc.ClientConn
version int
errCount int
}
func init() {
t := http.DefaultTransport.(*http.Transport)
t.MaxIdleConnsPerHost = 1024
t.MaxIdleConns = 1024
// Bind outbound HTTP to the -ip.bind source address. Reads the setting
// per dial, so it applies regardless of when SetOutboundLocalIP runs
// relative to this init.
t.DialContext = util.OutboundDialContext
}
// RegisterLocalGrpcSocket registers a Unix socket path for a gRPC service
// running on host:grpcPort in this process. After this is called, any
// GrpcDial to host:grpcPort (or to a loopback alias of host on the same
// port) is routed through the Unix socket. Dials to any other host on the
// same port still go over TCP.
//
// No-op on Windows: the /tmp/...sock paths callers pass are POSIX-only and
// the listen/dial would fail at runtime, taking gRPC down with it (#9430).
// Skipping registration leaves the maps empty, so ServeGrpcOnLocalSocket
// and resolveLocalGrpcSocket short-circuit and same-host RPCs go over TCP.
func RegisterLocalGrpcSocket(host string, grpcPort int, socketPath string) {
if runtime.GOOS == "windows" {
return
}
localGrpcSocketsLock.Lock()
defer localGrpcSocketsLock.Unlock()
localGrpcSockets[grpcPort] = socketPath
hosts, ok := localGrpcHosts[grpcPort]
if !ok {
hosts = make(map[string]struct{})
localGrpcHosts[grpcPort] = hosts
}
// The advertised host plus the well-known loopback aliases all refer
// to "this machine"; the empty entry covers SplitHostPort outputs like
// ":17334".
for _, h := range []string{host, "", "localhost", "127.0.0.1", "::1"} {
hosts[h] = struct{}{}
}
}
// GetLocalGrpcSocket returns the Unix socket path for a gRPC port, or empty if not registered.
func GetLocalGrpcSocket(grpcPort int) string {
localGrpcSocketsLock.RLock()
defer localGrpcSocketsLock.RUnlock()
return localGrpcSockets[grpcPort]
}
// resolveLocalGrpcSocket returns the Unix socket path to use for an outgoing
// gRPC dial, or empty if the dial should go over TCP. It matches both the
// host and the port: a remote peer that happens to reuse a local service's
// gRPC port must not be rerouted to the local socket (issue #9254).
func resolveLocalGrpcSocket(address string) string {
host, portStr, err := net.SplitHostPort(address)
if err != nil {
return ""
}
port, err := strconv.Atoi(portStr)
if err != nil {
return ""
}
localGrpcSocketsLock.RLock()
defer localGrpcSocketsLock.RUnlock()
if _, ok := localGrpcHosts[port][host]; !ok {
return ""
}
return localGrpcSockets[port]
}
// ListenOnLocalSocket binds a unix socket listener for a local fast path
// next to the TCP listener. A stale socket file is removed first; a path
// that cannot be cleared or bound is skipped, since the socket is optional.
func ListenOnLocalSocket(socketPath string) net.Listener {
if err := os.Remove(socketPath); err != nil && !os.IsNotExist(err) {
glog.Warningf("Failed to remove old socket %s: %v", socketPath, err)
}
lc := net.ListenConfig{Control: setLocalSocketBuffers}
listener, err := lc.Listen(context.Background(), "unix", socketPath)
if err != nil {
glog.Errorf("Failed to listen on Unix socket %s: %v", socketPath, err)
return nil
}
return listener
}
// ServeGrpcOnLocalSocket starts serving a gRPC server on a Unix socket
// if one is registered for the given port.
func ServeGrpcOnLocalSocket(grpcServer *grpc.Server, grpcPort int) {
socketPath := GetLocalGrpcSocket(grpcPort)
if socketPath == "" {
return
}
listener := ListenOnLocalSocket(socketPath)
if listener == nil {
return
}
glog.V(0).Infof("gRPC also listening on Unix socket %s", socketPath)
go serveGrpcOnLocalListener(grpcServer, socketPath, listener)
}
// localSocketCheckInterval is how often a served socket path is checked for
// removal by something else, such as a temp cleaner aging /tmp.
var localSocketCheckInterval = time.Minute
// serveGrpcOnLocalListener serves the socket until the gRPC server stops,
// rebinding it when the socket file is removed underneath the listener.
func serveGrpcOnLocalListener(grpcServer *grpc.Server, socketPath string, listener net.Listener) {
served := make(chan error, 1)
serve := func(l net.Listener) {
// A replaced listener must not unlink the path its successor binds;
// every exit below removes the socket file itself.
if unixListener, ok := l.(*net.UnixListener); ok {
unixListener.SetUnlinkOnClose(false)
}
go func() { served <- grpcServer.Serve(&localSocketListener{Listener: l}) }()
}
serve(listener)
ticker := time.NewTicker(localSocketCheckInterval)
defer ticker.Stop()
for {
select {
case err := <-served:
if err != nil && err != grpc.ErrServerStopped {
glog.Errorf("gRPC Unix socket server error on %s: %v", socketPath, err)
}
os.Remove(socketPath)
return
case <-ticker.C:
if _, err := os.Lstat(socketPath); !os.IsNotExist(err) {
continue
}
// Bind before closing the old listener, so a failed bind is
// retried on the next check and shutdown still ends this loop.
replacement := ListenOnLocalSocket(socketPath)
if replacement == nil {
continue
}
glog.Warningf("gRPC Unix socket %s was removed and has been recreated", socketPath)
listener.Close()
if err := <-served; err == nil || err == grpc.ErrServerStopped {
replacement.Close()
os.Remove(socketPath)
return
}
listener = replacement
serve(listener)
}
}
}
// localSocketListener re-applies the buffer sizes to every accepted connection.
type localSocketListener struct {
net.Listener
}
func (l *localSocketListener) Accept() (net.Conn, error) {
c, err := l.Listener.Accept()
if err != nil {
return nil, err
}
applyLocalSocketBuffers(c)
return c, nil
}
func NewGrpcServer(opts ...grpc.ServerOption) *grpc.Server {
var options []grpc.ServerOption
options = append(options,
grpc.KeepaliveParams(keepalive.ServerParameters{
Time: GrpcKeepAliveTime, // server pings client if no activity for this long
Timeout: GrpcKeepAliveTimeout, // ping timeout
}),
grpc.KeepaliveEnforcementPolicy(keepalive.EnforcementPolicy{
MinTime: GrpcKeepAliveMinimumTime, // min time a client should wait before sending a ping
PermitWithoutStream: true,
}),
grpc.MaxRecvMsgSize(Max_Message_Size),
grpc.MaxSendMsgSize(Max_Message_Size),
grpc.MaxConcurrentStreams(1000), // Allow more concurrent streams
grpc.InitialWindowSize(16*1024*1024), // 16MB initial window
grpc.InitialConnWindowSize(16*1024*1024), // 16MB connection window
grpc.MaxHeaderListSize(8*1024*1024), // 8MB header list limit
grpc.UnaryInterceptor(requestIDUnaryInterceptor()),
)
for _, opt := range opts {
if opt != nil {
options = append(options, opt)
}
}
return grpc.NewServer(options...)
}
// GrpcDial establishes a gRPC connection.
// IMPORTANT: This function intentionally uses the deprecated grpc.DialContext/grpc.Dial behavior
// to preserve the "passthrough" resolver semantics required for Kubernetes ndots/search-domain DNS behavior.
// This allows kube DNS suffixes to be correctly appended by the OS resolver.
//
// Switching to grpc.NewClient (which defaults to the "dns" resolver) would break this behavior
// in environments with ndots:5 and many-dot hostnames.
//
// Safe alternatives if switching to grpc.NewClient:
// 1. Prefix the target with "passthrough:///" (e.g., "passthrough:///my-service:8080"). This is the recommended primary migration path.
// 2. Call resolver.SetDefaultScheme("passthrough") exactly once during init().
// WARNING: This is NOT thread-safe, and mutates global resolver state affecting all grpc.NewClient calls in the process.
func GrpcDial(ctx context.Context, address string, waitForReady bool, opts ...grpc.DialOption) (*grpc.ClientConn, error) {
var options []grpc.DialOption
// Route through Unix socket if one is registered for this address's port.
// The socket file can vanish under a running server (temp cleaners age
// /tmp), so a failed socket dial falls back to TCP.
if socketPath := resolveLocalGrpcSocket(address); socketPath != "" {
options = append(options, grpc.WithContextDialer(func(ctx context.Context, addr string) (net.Conn, error) {
d := net.Dialer{Control: setLocalSocketBuffers}
conn, err := d.DialContext(ctx, "unix", socketPath)
if err != nil && ctx.Err() == nil {
glog.V(1).Infof("dial unix %s: %v, dialing %s over TCP", socketPath, err, addr)
return dialGrpcTcp(ctx, addr)
}
return conn, err
}))
} else {
options = append(options, grpc.WithContextDialer(dialGrpcTcp))
}
options = append(options,
grpc.WithDefaultCallOptions(
grpc.MaxCallSendMsgSize(Max_Message_Size),
grpc.MaxCallRecvMsgSize(Max_Message_Size),
grpc.WaitForReady(waitForReady),
),
grpc.WithKeepaliveParams(keepalive.ClientParameters{
Time: GrpcKeepAliveTime, // client ping server if no activity for this long
Timeout: GrpcKeepAliveTimeout, // ping timeout
// Disable pings when there are no active streams to avoid triggering
// server enforcement for too-frequent pings from idle clients.
PermitWithoutStream: false,
}))
for _, opt := range opts {
if opt != nil {
options = append(options, opt)
}
}
return grpc.DialContext(ctx, address, options...)
}
// dialGrpcTcp is installed even before SetOutboundLocalIP so a cached conn
// binds once it is set; the stock net.Dialer until then preserves gRPC's
// default dial behavior.
func dialGrpcTcp(ctx context.Context, addr string) (net.Conn, error) {
if util.OutboundLocalAddr() == nil {
var d net.Dialer
return d.DialContext(ctx, "tcp", addr)
}
return util.OutboundDialContext(ctx, "tcp", addr)
}
func getOrCreateConnection(address string, waitForReady bool, opts ...grpc.DialOption) (*versionedGrpcClient, error) {
grpcClientsLock.Lock()
defer grpcClientsLock.Unlock()
existingConnection, found := grpcClients[address]
if found {
glog.V(4).Infof("gRPC cache hit for %s (version %d)", address, existingConnection.version)
return existingConnection, nil
}
glog.V(2).Infof("Creating new gRPC connection to %s", address)
ctx := context.Background()
grpcConnection, err := GrpcDial(ctx, address, waitForReady, opts...)
if err != nil {
return nil, fmt.Errorf("fail to dial %s: %v", address, err)
}
vgc := &versionedGrpcClient{
grpcConnection,
rand.Int(),
0,
}
grpcClients[address] = vgc
glog.V(2).Infof("New gRPC connection established to %s (version %d)", address, vgc.version)
return vgc, nil
}
func requestIDUnaryInterceptor() grpc.UnaryServerInterceptor {
return func(
ctx context.Context,
req interface{},
info *grpc.UnaryServerInfo,
handler grpc.UnaryHandler,
) (interface{}, error) {
// Get request ID from incoming metadata
var reqID string
if incomingMd, ok := metadata.FromIncomingContext(ctx); ok {
if idList := incomingMd.Get(request_id.AmzRequestIDHeader); len(idList) > 0 {
reqID = idList[0]
}
}
if reqID == "" {
reqID = uuid.New().String()
}
// Store request ID in context for handlers to access
ctx = request_id.Set(ctx, reqID)
// Also set outgoing context so handlers making downstream gRPC calls
// will automatically propagate the request ID
ctx = metadata.AppendToOutgoingContext(ctx, request_id.AmzRequestIDHeader, reqID)
// Set trailer with request ID for response
grpc.SetTrailer(ctx, metadata.Pairs(request_id.AmzRequestIDHeader, reqID))
return handler(ctx, req)
}
}
// InvalidateGrpcConnection drops any cached gRPC ClientConn for the given
// address. Use when a higher-level signal (for example a streaming master
// connection detecting its peer has died) indicates the cached channel is
// stale, even though gRPC itself may still believe the channel is healthy.
// Silently returns if no cached connection exists.
func InvalidateGrpcConnection(address string) {
grpcClientsLock.Lock()
vgc, ok := grpcClients[address]
if ok {
delete(grpcClients, address)
}
grpcClientsLock.Unlock()
if ok {
glog.V(1).Infof("Invalidating cached gRPC connection to %s", address)
vgc.Close()
}
}
// grpcMarshalErrorPrefix is the library-owned prefix gRPC prepends to every
// client-side proto marshal failure; see grpc-go rpc_util.go encode():
//
// status.Errorf(codes.Internal, "grpc: error while marshaling: %v", ...)
//
// The "grpc:" token is reserved for gRPC internal diagnostics and will not
// collide with user-produced Internal statuses.
const grpcMarshalErrorPrefix = "grpc: error while marshaling"
// grpcStatusError is the minimal interface every gRPC status error satisfies.
// Using it with errors.As lets us reach through arbitrary fmt.Errorf("...: %w")
// wrapping to pull out the *original* gRPC Status — important because
// status.FromError rewrites Status.Message with the outermost err.Error()
// whenever it has to unwrap, which would defeat a prefix check on the
// library-owned message.
type grpcStatusError interface{ GRPCStatus() *status.Status }
// isClientSideMarshalError reports whether err originates from gRPC failing to
// marshal the outgoing request on the client side. These errors are surfaced
// with codes.Internal (because gRPC has no better code for them) but do NOT
// reflect a problem with the TCP/HTTP2 connection: the request never left the
// process. Tearing down the shared cached ClientConn in response would cancel
// every other in-flight RPC on that connection — which is exactly the cascade
// seen in seaweedfs#9139 when a single file with invalid-UTF-8 bytes in its
// name produces an avalanche of "connection is closing" errors on unrelated
// LookupEntry / ReadDirAll / UpdateEntry calls.
//
// errors.Is against a proto-level sentinel is not viable: gRPC's encode()
// collapses the inner proto error with "%v" before wrapping it in a Status,
// so the original error type does not survive. The structural signal that
// *does* survive is the Status itself — we recover it via errors.As and
// match the library-owned "grpc:" prefix to disambiguate from a server-side
// application Internal status that genuinely warrants invalidation.
func isClientSideMarshalError(err error) bool {
var gse grpcStatusError
if !errors.As(err, &gse) {
return false
}
s := gse.GRPCStatus()
if s == nil {
return false
}
return s.Code() == codes.Internal && strings.HasPrefix(s.Message(), grpcMarshalErrorPrefix)
}
// shouldInvalidateConnection checks if an error indicates the cached connection
// should be invalidated. ctx must be the context that bounds this RPC attempt; it
// disambiguates a genuinely broken channel from the RPC hitting its own deadline
// or cancellation. A nil or non-cancellable ctx carries no such evidence.
func shouldInvalidateConnection(ctx context.Context, err error) bool {
if err == nil {
return false
}
// Client-side marshal errors (e.g. invalid UTF-8 in a proto string field)
// are per-request bugs, not connection failures. Do not poison the shared
// ClientConn because of them.
if isClientSideMarshalError(err) {
return false
}
// A metadata subscriber reads log chunks over HTTP from volume servers and
// returns what went wrong there through the stream. Those failures read like
// transport failures below, and would drop a filer channel that is fine.
if errors.Is(err, ErrLogFileRead) {
return false
}
// gRPC raises this locally, before the RPC reaches the wire, when this
// process already closed the ClientConn. The caller is a bystander of
// someone else's teardown, and knows nothing about the peer. Its message
// is the only thing separating it from any other Canceled, so the plain
// codes.Canceled its deprecation notice recommends cannot stand in: that
// is the very code this function has to keep telling apart below.
if errors.Is(err, grpc.ErrClientConnClosing) {
return false
}
// Check gRPC status codes first (more reliable)
if s, ok := status.FromError(err); ok {
code := s.Code()
switch code {
case codes.Unavailable, codes.Aborted, codes.Internal:
return true
case codes.ResourceExhausted:
// Server-side backpressure on a healthy channel (e.g. the master's
// assign shed while volume growth is in flight), not a dead connection.
// Invalidating would Close() the shared conn and cancel every other
// in-flight RPC with "the client connection is closing"; keep it and let
// the caller retry.
return false
case codes.InvalidArgument, codes.NotFound, codes.AlreadyExists,
codes.FailedPrecondition, codes.OutOfRange, codes.DataLoss:
// gRPC never raises these itself, so the peer answered. Its
// message can name a path such as /transport, which must not
// reach the string matching below.
return false
case codes.Canceled, codes.DeadlineExceeded:
// Ambiguous: a stale cached channel rejects RPCs this way (a peer
// restart behind a k8s Service VIP), and so does the RPC's own context
// expiring. Tearing the channel down for the latter cancels every other
// in-flight RPC on it with "the client connection is closing", turning
// one abandoned request into a flood of failures. Only a cancellable
// context bounds an RPC attempt and can tell the two apart:
// Background/TODO never expires, so its Err() would answer
// "stale channel" for every caller with no deadline to honor.
return ctx != nil && ctx.Done() != nil && ctx.Err() == nil
}
}
// Fall back to string matching for transport-level errors not captured by gRPC codes
errStr := err.Error()
errLower := strings.ToLower(errStr)
return strings.Contains(errLower, "transport") ||
strings.Contains(errLower, "connection closed") ||
strings.Contains(errLower, "dns") ||
strings.Contains(errLower, "connection refused") ||
strings.Contains(errLower, "no route to host") ||
strings.Contains(errLower, "network is unreachable") ||
strings.Contains(errLower, "connection reset")
}
// WithGrpcClient In streamingMode, always use a fresh connection. Otherwise, try to reuse an existing connection.
// ctx must be the context that bounds the RPC fn issues, so that a
// Canceled/DeadlineExceeded fired by ctx itself does not tear down the shared
// cached ClientConn out from under other concurrent callers. Pass
// context.Background() when fn picks its own context: a Canceled/DeadlineExceeded
// then never invalidates, since nothing here can attribute it to the channel.
func WithGrpcClient(ctx context.Context, streamingMode bool, signature int32, fn func(*grpc.ClientConn) error, address string, waitForReady bool, opts ...grpc.DialOption) error {
if !streamingMode {
vgc, err := getOrCreateConnection(address, waitForReady, opts...)
if err != nil {
return fmt.Errorf("getOrCreateConnection %s: %v", address, err)
}
executionErr := fn(vgc.ClientConn)
if executionErr != nil && shouldInvalidateConnection(ctx, executionErr) {
grpcClientsLock.Lock()
t, ok := grpcClients[address]
shouldClose := ok && t.version == vgc.version
if shouldClose {
delete(grpcClients, address)
}
grpcClientsLock.Unlock()
if shouldClose {
glog.V(1).Infof("Removing cached gRPC connection to %s due to error: %v", address, executionErr)
vgc.Close()
}
}
return executionErr
}
// Streaming mode: dedicate a fresh ClientConn to this call.
dialCtx := context.Background()
if signature != 0 {
// Optimize: Use AppendToOutgoingContext instead of creating new map
dialCtx = metadata.AppendToOutgoingContext(dialCtx, "sw-client-id", fmt.Sprintf("%d", signature))
}
grpcConnection, err := GrpcDial(dialCtx, address, waitForReady, opts...)
if err != nil {
return fmt.Errorf("fail to dial %s: %v", address, err)
}
defer grpcConnection.Close()
executionErr := fn(grpcConnection)
if executionErr != nil && shouldInvalidateConnection(ctx, executionErr) {
// The streaming channel is dedicated to this caller, but unrelated
// request-path callers share a cached non-streaming ClientConn to the
// same peer. When the stream dies of a peer that went away, drop that
// cached channel so the next caller dials fresh: this recovers cases
// where a stable L4 endpoint (k8s Service VIP, external LB) hides a
// peer restart from the transport layer, leaving the cached ClientConn
// healthy-looking but silently cancelling RPCs. A stream that merely
// ended, or that its own caller gave up on, says nothing about the peer.
InvalidateGrpcConnection(address)
}
return executionErr
}
func hostAndPort(address string) (host string, port uint64, err error) {
colonIndex := strings.LastIndex(address, ":")
if colonIndex < 0 {
return "", 0, fmt.Errorf("server should have hostname:port format: %v", address)
}
dotIndex := strings.LastIndex(address, ".")
if dotIndex > colonIndex {
// port format is "port.grpcPort"
port, err = strconv.ParseUint(address[colonIndex+1:dotIndex], 10, 64)
if err != nil {
return "", 0, fmt.Errorf("server port parse error: %w", err)
}
return address[:colonIndex], port, err
}
port, err = strconv.ParseUint(address[colonIndex+1:], 10, 64)
if err != nil {
return "", 0, fmt.Errorf("server port parse error: %w", err)
}
return address[:colonIndex], port, err
}
func ServerToGrpcAddress(server string) (serverGrpcAddress string) {
colonIndex := strings.LastIndex(server, ":")
if colonIndex >= 0 {
if dotIndex := strings.LastIndex(server, "."); dotIndex > colonIndex {
// port format is "port.grpcPort"
// return the host:grpcPort
host := server[:colonIndex]
grpcPort := server[dotIndex+1:]
if _, err := strconv.ParseUint(grpcPort, 10, 64); err == nil {
return util.JoinHostPort(host, int(0+util.ParseInt(grpcPort, 0)))
}
}
}
host, port, parseErr := hostAndPort(server)
if parseErr != nil {
glog.Errorf("server address %s parse error: %v", server, parseErr)
return server
}
grpcPort := int(port) + 10000
return util.JoinHostPort(host, grpcPort)
}
func GrpcAddressToServerAddress(grpcAddress string) (serverAddress string) {
host, grpcPort, parseErr := hostAndPort(grpcAddress)
if parseErr != nil {
glog.Errorf("server grpc address %s parse error: %v", grpcAddress, parseErr)
return ""
}
port := int(grpcPort) - 10000
return util.JoinHostPort(host, port)
}
// WithMasterClient threads the context bounding this RPC attempt into the
// connection-invalidation decision, so a Canceled/DeadlineExceeded from the
// attempt's own timeout does not invalidate the shared cached master connection.
// Pass context.Background() when fn picks its own context.
func WithMasterClient(ctx context.Context, streamingMode bool, master ServerAddress, grpcDialOption grpc.DialOption, waitForReady bool, fn func(client master_pb.SeaweedClient) error) error {
return WithGrpcClient(ctx, streamingMode, 0, func(grpcConnection *grpc.ClientConn) error {
client := master_pb.NewSeaweedClient(grpcConnection)
return fn(client)
}, master.ToGrpcAddress(), waitForReady, grpcDialOption)
}
func WithVolumeServerClient(streamingMode bool, volumeServer ServerAddress, grpcDialOption grpc.DialOption, fn func(client volume_server_pb.VolumeServerClient) error) error {
return WithGrpcClient(context.Background(), streamingMode, 0, func(grpcConnection *grpc.ClientConn) error {
client := volume_server_pb.NewVolumeServerClient(grpcConnection)
return fn(client)
}, volumeServer.ToGrpcAddress(), false, grpcDialOption)
}
func WithOneOfGrpcMasterClients(streamingMode bool, masterGrpcAddresses map[string]ServerAddress, grpcDialOption grpc.DialOption, fn func(client master_pb.SeaweedClient) error) (err error) {
for _, masterGrpcAddress := range masterGrpcAddresses {
err = WithGrpcClient(context.Background(), streamingMode, 0, func(grpcConnection *grpc.ClientConn) error {
client := master_pb.NewSeaweedClient(grpcConnection)
return fn(client)
}, masterGrpcAddress.ToGrpcAddress(), false, grpcDialOption)
if err == nil {
return nil
}
}
return err
}
func WithBrokerGrpcClient(streamingMode bool, brokerGrpcAddress string, grpcDialOption grpc.DialOption, fn func(client mq_pb.SeaweedMessagingClient) error) error {
return WithGrpcClient(context.Background(), streamingMode, 0, func(grpcConnection *grpc.ClientConn) error {
client := mq_pb.NewSeaweedMessagingClient(grpcConnection)
return fn(client)
}, brokerGrpcAddress, false, grpcDialOption)
}
func WithFilerClient(streamingMode bool, signature int32, filer ServerAddress, grpcDialOption grpc.DialOption, fn func(client filer_pb.SeaweedFilerClient) error) error {
return WithGrpcFilerClient(streamingMode, signature, filer, grpcDialOption, fn)
}
func WithGrpcFilerClient(streamingMode bool, signature int32, filerAddress ServerAddress, grpcDialOption grpc.DialOption, fn func(client filer_pb.SeaweedFilerClient) error) error {
return WithGrpcClient(context.Background(), streamingMode, signature, func(grpcConnection *grpc.ClientConn) error {
client := filer_pb.NewSeaweedFilerClient(grpcConnection)
return fn(client)
}, filerAddress.ToGrpcAddress(), false, grpcDialOption)
}
func WithOneOfGrpcFilerClients(streamingMode bool, filerAddresses []ServerAddress, grpcDialOption grpc.DialOption, fn func(client filer_pb.SeaweedFilerClient) error) (err error) {
for _, filerAddress := range filerAddresses {
err = WithGrpcClient(context.Background(), streamingMode, 0, func(grpcConnection *grpc.ClientConn) error {
client := filer_pb.NewSeaweedFilerClient(grpcConnection)
return fn(client)
}, filerAddress.ToGrpcAddress(), false, grpcDialOption)
if err == nil {
return nil
}
}
return err
}