admin: default to port 12535, keep 23646 as a best-effort compat listener (#11816)

* admin: default to port 12535, keep 23646 as a best-effort compat listener

The default admin HTTP port 23646 derives a worker gRPC port of 33646
(port + GrpcPortOffset), inside the Linux default ephemeral range
(32768+). One of our own startup dials can claim 33646 before the admin
binds it, killing startup. This is the same failure #10209 fixed for
the test port allocator, still reachable via the mini default (#10838)
and via standalone `weed admin` defaults.

Move the default admin port to 12535 (gRPC 22535) everywhere, keeping
the http+10000 convention so worker -admin derivation, the Rust worker
and host:http.grpc addressing keep working:

- weed admin -port, weed mini -admin.port
- weed worker -admin, the plugin worker fallback, Rust worker and
  weed-lance-worker defaults
- help text, README, helm values, docker compose, tests

So old workers, bookmarks and scripts keep working, admin also serves
the same handler and the worker gRPC server on the previous default
(-port.legacy / -admin.port.legacy, default 23646, 0 disables). The
legacy gRPC port binds best-effort: it lives in the ephemeral range, so
a failed bind only warns instead of reintroducing the startup failure.
weed mini reserves it early like the primary port and keeps it out of
its own port relocation. grpc.Server tracks every listener handed to
Serve, so the compat listener stops with the primary.

Also warn when an explicitly configured admin port puts the derived
gRPC port at or above the ephemeral floor, the other fix suggested in
the issue.

Supersedes the pinned-grpc approach in #11814: pinning -admin.port.grpc
off the +10000 convention breaks external workers that derive the admin
gRPC port from the HTTP address. ClickHouse/ClickHouse#115512 pins
-admin.port on their side; this makes that pin unnecessary.

Fixes #10838

* admin: let services win the legacy ports, close compat listener on failure

Review findings on the compat listeners:

- Do not put the legacy ports in mini's reservedPorts: an explicit
  -s3.port=23646 or -master.port.grpc=33646 must be honored, not rejected
  or stolen by the early reservation. Mini now checks the finalized port
  set and skips only the conflicting compat listener, so HTTP and gRPC
  compat degrade independently and the service keeps its port.
- Also skip the legacy HTTP listener when a mini service owns the port:
  a wildcard-bound service can still leave loopback bindable, which would
  let the compat listener shadow it for loopback clients.
- Close legacyServer's raw listener in the deferred cleanup: Serve may
  not have registered it yet when startAdminServer returns early.
- Helm: add admin.legacyPort (default 23646) and expose the legacy
  http/grpc ports through the admin Service, container ports and
  NetworkPolicy so older workers resolved via the Service keep working.

* helm: only pass -port.legacy when it differs from the binary default

ct install deploys the released image with the new chart, and 4.48 does
not know -port.legacy, so the admin pod crash-loops. Omitting the flag at
the default value is behavior-identical on new binaries (their default
is also 23646) and keeps older releases installable; legacyPort: 0 or a
custom port still renders the flag.
This commit is contained in:
Chris Lu authored and GitHub committed 2026-10-18 21:21:05 -07:00
1 parent d3ee817b2e
commit 8ac5133abe
33 files changed
+316 -101

No files matched your search

+95 -7
View File
@@ -5,6 +5,7 @@ import (
"context"
"crypto/rand"
"crypto/tls"
"errors"
"fmt"
"log"
"net"
@@ -42,6 +43,7 @@ var (
type AdminOptions struct {
port *int
grpcPort *int
legacyPort *int
ip *string
master *string
masters *string // deprecated, for backward compatibility
@@ -68,6 +70,23 @@ type AdminOptions struct {
// binds it only after every other service is up.
workerGrpcListener net.Listener
// workerGrpcLegacyListener is the same reservation for the legacy
// compat gRPC port. Its bind failure is tolerated: the legacy port lives
// in the ephemeral range, so best-effort compat must not take startup down.
workerGrpcLegacyListener net.Listener
// workerGrpcLegacySkip disables just the legacy gRPC listener. `weed
// mini` sets it when another mini service already owns the legacy gRPC
// port, so the compat bind can neither steal the port nor warn about a
// failure that is expected.
workerGrpcLegacySkip bool
// adminHttpLegacySkip is the same for the legacy HTTP port. Skipping
// matters even when the bind would succeed: a service bound on a
// different address family or wildcard can leave loopback free, and the
// compat listener would then shadow it for loopback clients.
adminHttpLegacySkip bool
// workerGrpcBindIp, when non-empty, is the address the worker gRPC
// listener binds to. It is separate from ip because the worker gRPC has
// no password auth (its mTLS comes from grpc.admin, not https.admin), so
@@ -83,8 +102,9 @@ type AdminOptions struct {
func init() {
cmdAdmin.Run = runAdmin // break init cycle
a.port = cmdAdmin.Flag.Int("port", 23646, "admin server port")
a.port = cmdAdmin.Flag.Int("port", defaultAdminHTTPPort, "admin server port")
a.grpcPort = cmdAdmin.Flag.Int("port.grpc", 0, "gRPC server port for worker connections (default: http port + 10000)")
a.legacyPort = cmdAdmin.Flag.Int("port.legacy", defaultAdminHTTPLegacyPort, "also serve admin http and gRPC (port + 10000) on this port for backward compatibility (0 to disable)")
a.ip = cmdAdmin.Flag.String("ip", "127.0.0.1", "ip address to listen on. Defaults to loopback when auth is disabled, or 0.0.0.0 when -adminPassword or [https.admin] mTLS is configured. Set explicitly to override.")
a.master = cmdAdmin.Flag.String("master", "localhost:9333", "comma-separated master servers")
a.masters = cmdAdmin.Flag.String("masters", "", "comma-separated master servers (deprecated, use -master instead)")
@@ -108,7 +128,7 @@ func init() {
}
var cmdAdmin = &Command{
UsageLine: "admin -port=23646 -master=localhost:9333 [-filerGroup=group] [-port.grpc=33646] [-dataDir=/path/to/data]",
UsageLine: "admin -port=12535 -master=localhost:9333 [-filerGroup=group] [-port.grpc=22535] [-dataDir=/path/to/data]",
Short: "start SeaweedFS web admin interface",
Long: `Start a web admin interface for SeaweedFS cluster management.
@@ -124,12 +144,12 @@ var cmdAdmin = &Command{
A gRPC server for worker connections runs on the configured gRPC port (default: HTTP port + 10000).
Example Usage:
weed admin -port=23646 -master="master1:9333,master2:9333"
weed admin -port=23646 -master="localhost:9333" -filerGroup="tenant-a"
weed admin -port=23646 -master="localhost:9333" -dataDir="/var/lib/seaweedfs-admin"
weed admin -port=23646 -port.grpc=33646 -master="localhost:9333" -dataDir="~/seaweedfs-admin"
weed admin -port=12535 -master="master1:9333,master2:9333"
weed admin -port=12535 -master="localhost:9333" -filerGroup="tenant-a"
weed admin -port=12535 -master="localhost:9333" -dataDir="/var/lib/seaweedfs-admin"
weed admin -port=12535 -port.grpc=22535 -master="localhost:9333" -dataDir="~/seaweedfs-admin"
weed admin -port=9900 -port.grpc=19900 -master="localhost:9333"
weed admin -port=23646 -master="localhost:9333" -urlPrefix="/seaweedfs"
weed admin -port=12535 -master="localhost:9333" -urlPrefix="/seaweedfs"
Data Directory:
- If dataDir is specified, admin configuration and maintenance data is persisted
@@ -295,6 +315,9 @@ func runAdmin(cmd *Command, args []string) bool {
if *a.grpcPort == 0 {
*a.grpcPort = *a.port + 10000
}
if *a.grpcPort >= linuxEphemeralPortFloor {
glog.Warningf("Admin gRPC port %d is at or above the Linux ephemeral floor %d; an outbound connection may claim it before Admin binds it. Pick a lower -port.", *a.grpcPort, linuxEphemeralPortFloor)
}
hasMTLS := viper.GetString("https.admin.key") != "" && viper.GetString("https.admin.ca") != ""
@@ -504,6 +527,31 @@ func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool,
}
warnInsecureWorkerGrpcBind(workerGrpcIp, *options.grpcPort, adminServer.WorkerGrpcMTLSEnabled())
// Legacy compat listeners keep the previous default admin port reachable
// for old workers, bookmarks and scripts while the deployment moves to the
// new default. Best-effort only: the legacy gRPC port sits inside the
// Linux ephemeral range, so it may already be taken and must never fail
// startup.
legacyPort := 0
if options.legacyPort != nil && *options.legacyPort > 0 && *options.legacyPort != *options.port {
legacyPort = *options.legacyPort
}
if legacyPort > 0 && !options.workerGrpcLegacySkip {
legacyGrpcPort := legacyPort + GrpcPortOffset
legacyGrpcListener := options.workerGrpcLegacyListener
if legacyGrpcListener == nil {
if l, listenErr := net.Listen("tcp", util.JoinHostPort(workerGrpcIp, legacyGrpcPort)); listenErr == nil {
legacyGrpcListener = l
} else {
glog.Warningf("Legacy admin gRPC port %d unavailable, skipping compat listener: %v", legacyGrpcPort, listenErr)
}
}
if legacyGrpcListener != nil {
adminServer.ServeWorkerGrpcOn(legacyGrpcListener)
glog.Infof("Worker gRPC compat listener on %s", legacyGrpcListener.Addr())
}
}
// Set up cleanup for gRPC server
defer func() {
if stopErr := adminServer.StopWorkerGrpcServer(); stopErr != nil {
@@ -578,6 +626,43 @@ func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool,
server.TLSConfig.GetCertificate = getCert
}
// Serve the same handler on the legacy port so old clients keep working.
// Best-effort like the gRPC compat listener above.
var legacyServer *http.Server
if legacyPort > 0 && !options.adminHttpLegacySkip {
legacyAddr := util.JoinHostPort(*options.ip, legacyPort)
if legacyListener, listenErr := net.Listen("tcp", legacyAddr); listenErr != nil {
glog.Warningf("Legacy admin HTTP port %d unavailable, skipping compat listener: %v", legacyPort, listenErr)
} else {
legacyServer = &http.Server{
Addr: legacyAddr,
Handler: server.Handler,
TLSConfig: server.TLSConfig,
}
// Close on every exit path so a failed primary startup cannot
// leave the compat port held by a zombie admin. On the normal
// path the graceful Shutdown below runs first; this Close then
// no-ops. Close the raw listener too: Serve may not have
// registered it with legacyServer yet when this runs.
defer func() {
_ = legacyListener.Close()
_ = legacyServer.Close()
}()
go func() {
glog.Infof("Admin compat listener on %s", legacyAddr)
var serveErr error
if useTLS {
serveErr = legacyServer.ServeTLS(legacyListener, "", "")
} else {
serveErr = legacyServer.Serve(legacyListener)
}
if serveErr != nil && serveErr != http.ErrServerClosed && !errors.Is(serveErr, net.ErrClosed) {
glog.Warningf("Admin compat listener on %s stopped: %v", legacyAddr, serveErr)
}
}()
}
}
// Start server. Surface immediate failures (e.g. bind error) via
// serveErrCh so the caller doesn't block on ctx.Done() while no server
// is actually listening. http.ErrServerClosed after Shutdown is normal
@@ -612,6 +697,9 @@ func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool,
if err := server.Shutdown(shutdownCtx); err != nil {
return fmt.Errorf("admin server forced to shutdown: %w", err)
}
if legacyServer != nil {
_ = legacyServer.Shutdown(shutdownCtx)
}
adminServer.Shutdown()
+57 -12
View File
@@ -49,13 +49,16 @@ type MiniOptions struct {
}
const (
bytesPerMB = 1024 * 1024 // Bytes per MB
miniVolumeMinFreeSpace = "1" // 1% minimum free space
minVolumeSizeMB = 64 // Minimum volume size in MB
defaultMiniVolumeSizeMB = 128 // Default volume size for mini mode
maxVolumeSizeMB = 1024 // Maximum volume size in MB (1GB)
GrpcPortOffset = 10000 // Offset used to calculate gRPC port from HTTP port
defaultMiniPluginJobTypes = "all"
bytesPerMB = 1024 * 1024 // Bytes per MB
miniVolumeMinFreeSpace = "1" // 1% minimum free space
minVolumeSizeMB = 64 // Minimum volume size in MB
defaultMiniVolumeSizeMB = 128 // Default volume size for mini mode
maxVolumeSizeMB = 1024 // Maximum volume size in MB (1GB)
GrpcPortOffset = 10000 // Offset used to calculate gRPC port from HTTP port
linuxEphemeralPortFloor = 32768 // Default net.ipv4.ip_local_port_range floor on Linux
defaultAdminHTTPPort = 12535 // Keeps the derived admin gRPC port (port+GrpcPortOffset) below linuxEphemeralPortFloor
defaultAdminHTTPLegacyPort = 23646 // Pre-12535 default; served alongside the new port for backward compatibility
defaultMiniPluginJobTypes = "all"
)
var (
@@ -371,12 +374,12 @@ After starting, you can access:
- S3 Endpoint: http://localhost:8333
- Iceberg Catalog: http://localhost:8181
- WebDAV: http://localhost:7333
- Admin UI: http://localhost:23646
- Admin UI: http://localhost:12535
S3 Access:
The S3 endpoint is available at http://localhost:8333. For client
configuration and IAM setup, see the project documentation or use the
Admin UI (http://localhost:23646) to manage users and policies.
Admin UI (http://localhost:12535) to manage users and policies.
Notes:
- Values mini computes itself (currently master.volumeSizeLimitMB) are
@@ -574,8 +577,9 @@ func initMiniWebDAVFlags() {
// initMiniAdminFlags initializes Admin server flag options
func initMiniAdminFlags() {
miniAdminOptions.port = cmdMini.Flag.Int("admin.port", 23646, "admin server http listen port")
miniAdminOptions.port = cmdMini.Flag.Int("admin.port", defaultAdminHTTPPort, "admin server http listen port")
miniAdminOptions.grpcPort = cmdMini.Flag.Int("admin.port.grpc", 0, "admin server grpc listen port (default: admin http port + GrpcPortOffset)")
miniAdminOptions.legacyPort = cmdMini.Flag.Int("admin.port.legacy", defaultAdminHTTPLegacyPort, "also serve admin http and grpc (port + GrpcPortOffset) on this port for backward compatibility (0 to disable)")
miniAdminOptions.master = cmdMini.Flag.String("admin.master", "", "master server address (automatically set)")
// admin discovery must query the same group the co-located filer registers under
miniAdminOptions.filerGroup = miniFilerOptions.filerGroup
@@ -966,7 +970,7 @@ func ensureAllPortsAvailableOnIP(bindIp string) error {
// Initialize all gRPC ports before services start
// This ensures they won't be recalculated and cause conflicts
// All gRPC port handling (calculation, validation, and assignment) is performed exclusively in initializeGrpcPortsOnIP
initializeGrpcPortsOnIP(bindIp)
allocatedPorts := initializeGrpcPortsOnIP(bindIp)
// Every other service binds within a moment of this check, but the admin
// waits for all of them first. The admin gRPC port sits inside the Linux
@@ -986,6 +990,33 @@ func ensureAllPortsAvailableOnIP(bindIp string) error {
miniAdminOptions.workerGrpcListener = listener
}
// Same early reservation for the legacy compat gRPC port — but never take
// a port a mini service already owns: compat is a convenience, not a
// startup requirement. Failure is only a warning — the legacy port sits
// inside the ephemeral range. Without a reservation the admin's late
// best-effort bind simply fails harmlessly if the port is taken by then.
miniAdminOptions.workerGrpcLegacyListener = nil
miniAdminOptions.workerGrpcLegacySkip = false
miniAdminOptions.adminHttpLegacySkip = false
if *miniAdminOptions.legacyPort > 0 && *miniAdminOptions.legacyPort != *miniAdminOptions.port {
if allocatedPorts[*miniAdminOptions.legacyPort] {
glog.Warningf("Legacy Admin HTTP port %d is used by another mini service, skipping compat listener", *miniAdminOptions.legacyPort)
miniAdminOptions.adminHttpLegacySkip = true
}
legacyGrpcPort := *miniAdminOptions.legacyPort + GrpcPortOffset
switch {
case allocatedPorts[legacyGrpcPort]:
glog.Warningf("Legacy Admin gRPC port %d is used by another mini service, skipping compat listener", legacyGrpcPort)
miniAdminOptions.workerGrpcLegacySkip = true
default:
if listener, err := listenMiniAdminWorker(workerGrpcBindIP, legacyGrpcPort); err != nil {
glog.Warningf("Could not reserve legacy Admin gRPC port %d: %v", legacyGrpcPort, err)
} else {
miniAdminOptions.workerGrpcLegacyListener = listener
}
}
}
// Log the final port configuration
icebergPortStr := "disabled"
if miniS3Options.portIceberg != nil && *miniS3Options.portIceberg > 0 {
@@ -1016,7 +1047,9 @@ func listenMiniAdminWorker(bindIP string, port int) (net.Listener, error) {
// initializeGrpcPortsOnIP initializes all gRPC ports based on their HTTP ports on a specific IP
// If a gRPC port is 0, it will be set to httpPort + GrpcPortOffset
// This must be called after HTTP ports are finalized and before services start
func initializeGrpcPortsOnIP(bindIp string) {
// Returns the set of ports allocated to mini services, for callers that must
// keep optional listeners from stealing them.
func initializeGrpcPortsOnIP(bindIp string) map[int]bool {
// Track all ports allocated (both HTTP and gRPC) to prevent collisions.
// We must reserve HTTP ports so that gRPC fallback allocation never picks
// a port already assigned to an HTTP service (which hasn't bound yet).
@@ -1086,6 +1119,15 @@ func initializeGrpcPortsOnIP(bindIp string) {
allocatedPorts[*config.grpcPort] = true
glog.V(1).Infof("%s gRPC port set to %d", config.name, *config.grpcPort)
}
// The Linux kernel draws outbound source ports from the ephemeral range.
// A gRPC port inside it can be claimed by one of mini's own startup dials
// before the service binds it.
if *miniAdminOptions.grpcPort >= linuxEphemeralPortFloor {
glog.Warningf("Admin gRPC port %d is at or above the Linux ephemeral floor %d; an outbound connection may claim it before Admin binds it. Pick a lower -admin.port.", *miniAdminOptions.grpcPort, linuxEphemeralPortFloor)
}
return allocatedPorts
}
// miniOptionsFilePath returns the path of the mini.options file under the
@@ -1880,6 +1922,9 @@ func startMiniAdminWithWorker(allServicesReady chan struct{}) {
if adminOptions.workerGrpcListener != nil {
adminOptions.workerGrpcListener.Close()
}
if adminOptions.workerGrpcLegacyListener != nil {
adminOptions.workerGrpcLegacyListener.Close()
}
}()
// Wait for admin server's HTTP port to be ready before launching worker
+30
View File
@@ -2,6 +2,7 @@ package command
import (
"net"
"strconv"
"testing"
"github.com/seaweedfs/seaweedfs/weed/pb"
@@ -254,3 +255,32 @@ func TestMiniAdminAdvertisedIP(t *testing.T) {
})
}
}
func TestDefaultAdminGrpcPortBelowEphemeralFloor(t *testing.T) {
if grpcPort := defaultAdminHTTPPort + GrpcPortOffset; grpcPort >= linuxEphemeralPortFloor {
t.Fatalf("default admin HTTP port %d derives gRPC port %d, which must stay below ephemeral floor %d",
defaultAdminHTTPPort, grpcPort, linuxEphemeralPortFloor)
}
for _, f := range []struct {
name string
flagName string
def string
}{
{"mini", "admin.port", strconv.Itoa(defaultAdminHTTPPort)},
{"mini", "admin.port.legacy", strconv.Itoa(defaultAdminHTTPLegacyPort)},
{"admin", "port", strconv.Itoa(defaultAdminHTTPPort)},
{"admin", "port.legacy", strconv.Itoa(defaultAdminHTTPLegacyPort)},
} {
cmd := cmdMini
if f.name == "admin" {
cmd = cmdAdmin
}
flag := cmd.Flag.Lookup(f.flagName)
if flag == nil {
t.Fatalf("%s: flag %s not registered", f.name, f.flagName)
}
if flag.DefValue != f.def {
t.Fatalf("%s -%s default = %s, want %s so its gRPC port stays below the ephemeral floor", f.name, f.flagName, flag.DefValue, f.def)
}
}
}
+12 -11
View File
@@ -1,6 +1,7 @@
package command
import (
"fmt"
"time"
"github.com/seaweedfs/seaweedfs/weed/util"
@@ -25,21 +26,21 @@ Behavior:
- Use -metricsPort/-metricsIp to expose /health, /ready, and /metrics
Examples:
weed worker -admin=localhost:23646
weed worker -admin=localhost:23646 -jobType=all
weed worker -admin=localhost:23646 -jobType=default
weed worker -admin=localhost:23646 -jobType=heavy
weed worker -admin=localhost:23646 -jobType=default,iceberg
weed worker -admin=localhost:23646 -jobType=vacuum,volume_balance
weed worker -admin=localhost:23646 -jobType=erasure_coding
weed worker -admin=admin.example.com:23646 -id=plugin-vacuum-a -heartbeat=10s
weed worker -admin=localhost:23646 -workingDir=/var/lib/seaweedfs-plugin
weed worker -admin=localhost:23646 -metricsPort=9327 -metricsIp=0.0.0.0
weed worker -admin=localhost:12535
weed worker -admin=localhost:12535 -jobType=all
weed worker -admin=localhost:12535 -jobType=default
weed worker -admin=localhost:12535 -jobType=heavy
weed worker -admin=localhost:12535 -jobType=default,iceberg
weed worker -admin=localhost:12535 -jobType=vacuum,volume_balance
weed worker -admin=localhost:12535 -jobType=erasure_coding
weed worker -admin=admin.example.com:12535 -id=plugin-vacuum-a -heartbeat=10s
weed worker -admin=localhost:12535 -workingDir=/var/lib/seaweedfs-plugin
weed worker -admin=localhost:12535 -metricsPort=9327 -metricsIp=0.0.0.0
`,
}
var (
workerAdminServer = cmdWorker.Flag.String("admin", "localhost:23646", "admin server address")
workerAdminServer = cmdWorker.Flag.String("admin", fmt.Sprintf("localhost:%d", defaultAdminHTTPPort), "admin server address")
workerID = cmdWorker.Flag.String("id", "", "worker ID (auto-generated when empty)")
workerWorkingDir = cmdWorker.Flag.String("workingDir", "", "working directory for persistent worker state")
workerJobType = cmdWorker.Flag.String("jobType", defaultPluginWorkerJobTypes, "job types or categories to serve: all, default, heavy, or explicit names/aliases such as ec, balance, iceberg (comma-separated)")
+1 -1
View File
@@ -45,7 +45,7 @@ func runPluginWorkerWithOptions(options pluginWorkerRunOptions) bool {
options.AdminServer = strings.TrimSpace(options.AdminServer)
if options.AdminServer == "" {
options.AdminServer = "localhost:23646"
options.AdminServer = fmt.Sprintf("localhost:%d", defaultAdminHTTPPort)
}
options.JobTypes = strings.TrimSpace(options.JobTypes)