Files
seaweedfs/weed/command/grpc_local_socket.go
T
b74971b6d4 server: add -grpc.localSocketDir for the local gRPC sockets (#12693)
* server: add -grpc.localSocketDir for the local gRPC sockets

`weed server` and `weed mini` hard-coded each local gRPC Unix socket
under /tmp. systemd-tmpfiles ages /tmp after 10 days and macOS after 3,
so a long-running server loses its socket, and RegisterLocalGrpcSocket
keeps routing local dials to the missing path: the master can no longer
reach its volume server.

Add -grpc.localSocketDir to both commands, defaulting to /tmp so an
existing setup is unchanged, and build every socket path from it with
filepath.Join. A folder that does not exist or is not a directory is
refused at startup rather than at the first local RPC. A folder whose
socket paths exceed the platform's sockaddr_un limit is refused too,
because every bind then fails with EINVAL while the server keeps running
and each local RPC dials a socket that does not exist.

Refs #12692

* server: refuse a -grpc.localSocketDir the server cannot bind in

An existing folder the server's user cannot write to passed the startup
check, and then every socket bind failed: ListenOnLocalSocket only logs
the error while GrpcDial keeps routing local dials to a socket that was
never created. After the path-length check, create and remove a probe
Unix socket in the folder, and refuse startup with the bind error if
that fails.

* server: give the -grpc.localSocketDir probe a random name

A probe named only by the process id could be precreated by another
user in a shared /tmp, or left by a crashed process whose pid was
reused, and the server cannot remove such a file, so startup failed
although the real socket paths were free. Name the probe with 8 random
bytes, retry a name that is taken, and rely on the listener's close to
unlink it.

* server: keep the -grpc.localSocketDir probe name within the service socket lengths

The probe was 5 bytes longer than the master and volume socket names, so a
folder whose service sockets fit the sockaddr_un limit could still be refused.
A 23-byte probe name is shorter than any service socket name, and a process
with no local sockets skips the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* server: trim the -grpc.localSocketDir comments and drop issue references

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Brent Pease <179639673+brentDSL@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 16:56:15 -08:00

86 lines
2.8 KiB
Go

package command
import (
"crypto/rand"
"errors"
"fmt"
"net"
"os"
"path/filepath"
"runtime"
"syscall"
)
// grpcLocalSocket is a service name and the resolved gRPC port it listens on.
type grpcLocalSocket struct {
name string
port int
}
// grpcLocalSocketPath returns the Unix socket path for local RPCs to a gRPC
// service, inside dir.
func grpcLocalSocketPath(dir, name string, grpcPort int) string {
return filepath.Join(dir, fmt.Sprintf("seaweedfs-%s-grpc-%d.sock", name, grpcPort))
}
// grpcLocalSocketPathLimit returns the longest Unix socket path the platform
// accepts: sockaddr_un's sun_path less the terminating NUL.
func grpcLocalSocketPathLimit() int {
return len(syscall.RawSockaddrUnix{}.Path) - 1
}
// checkGrpcLocalSocketDir rejects a socket folder that does not exist, is not
// a directory, is too long for one of its socket paths, or refuses a socket
// bind, since local RPCs keep dialing a registered socket that was never bound.
//
// The folder is unused, and not checked, on Windows (see
// pb.RegisterLocalGrpcSocket) or when no service registers a socket.
func checkGrpcLocalSocketDir(dir string, sockets []grpcLocalSocket) error {
if runtime.GOOS == "windows" || len(sockets) == 0 {
return nil
}
info, err := os.Stat(dir)
if err != nil {
return err
}
if !info.IsDir() {
return errors.New("not a directory")
}
limit := grpcLocalSocketPathLimit()
for _, socket := range sockets {
path := grpcLocalSocketPath(dir, socket.name, socket.port)
if len(path) > limit {
return fmt.Errorf("socket path %q is %d bytes and does not fit the platform's %d-byte Unix socket path limit; use a shorter -grpc.localSocketDir", path, len(path), limit)
}
}
return probeGrpcLocalSocketDir(dir)
}
// probeGrpcLocalSocketDir binds and removes a randomly named Unix socket in
// dir, so a file another user left in a shared folder cannot block startup.
// The 23-byte name is shorter than any service socket name.
func probeGrpcLocalSocketDir(dir string) error {
const attempts = 5
var lastErr error
for i := 0; i < attempts; i++ {
suffix := make([]byte, 4)
if _, err := rand.Read(suffix); err != nil {
return fmt.Errorf("cannot pick a Unix socket probe name: %v", err)
}
probePath := filepath.Join(dir, fmt.Sprintf("seaweedfs-%x.sock", suffix))
listener, err := net.Listen("unix", probePath)
if errors.Is(err, syscall.EADDRINUSE) {
lastErr = err
continue
}
if err != nil {
return fmt.Errorf("cannot create a Unix socket in it: %v; give the server's user write access or choose another -grpc.localSocketDir", err)
}
if err := listener.Close(); err != nil {
return fmt.Errorf("cannot close the Unix socket probe %q: %v", probePath, err)
}
return nil
}
return fmt.Errorf("cannot create a Unix socket in it: %d random probe names were all taken, last: %v", attempts, lastErr)
}