admin: default to port 12535, keep 23646 as a best-effort compat listener (#11816)

* admin: default to port 12535, keep 23646 as a best-effort compat listener

The default admin HTTP port 23646 derives a worker gRPC port of 33646
(port + GrpcPortOffset), inside the Linux default ephemeral range
(32768+). One of our own startup dials can claim 33646 before the admin
binds it, killing startup. This is the same failure #10209 fixed for
the test port allocator, still reachable via the mini default (#10838)
and via standalone `weed admin` defaults.

Move the default admin port to 12535 (gRPC 22535) everywhere, keeping
the http+10000 convention so worker -admin derivation, the Rust worker
and host:http.grpc addressing keep working:

- weed admin -port, weed mini -admin.port
- weed worker -admin, the plugin worker fallback, Rust worker and
  weed-lance-worker defaults
- help text, README, helm values, docker compose, tests

So old workers, bookmarks and scripts keep working, admin also serves
the same handler and the worker gRPC server on the previous default
(-port.legacy / -admin.port.legacy, default 23646, 0 disables). The
legacy gRPC port binds best-effort: it lives in the ephemeral range, so
a failed bind only warns instead of reintroducing the startup failure.
weed mini reserves it early like the primary port and keeps it out of
its own port relocation. grpc.Server tracks every listener handed to
Serve, so the compat listener stops with the primary.

Also warn when an explicitly configured admin port puts the derived
gRPC port at or above the ephemeral floor, the other fix suggested in
the issue.

Supersedes the pinned-grpc approach in #11814: pinning -admin.port.grpc
off the +10000 convention breaks external workers that derive the admin
gRPC port from the HTTP address. ClickHouse/ClickHouse#115512 pins
-admin.port on their side; this makes that pin unnecessary.

Fixes #10838

* admin: let services win the legacy ports, close compat listener on failure

Review findings on the compat listeners:

- Do not put the legacy ports in mini's reservedPorts: an explicit
  -s3.port=23646 or -master.port.grpc=33646 must be honored, not rejected
  or stolen by the early reservation. Mini now checks the finalized port
  set and skips only the conflicting compat listener, so HTTP and gRPC
  compat degrade independently and the service keeps its port.
- Also skip the legacy HTTP listener when a mini service owns the port:
  a wildcard-bound service can still leave loopback bindable, which would
  let the compat listener shadow it for loopback clients.
- Close legacyServer's raw listener in the deferred cleanup: Serve may
  not have registered it yet when startAdminServer returns early.
- Helm: add admin.legacyPort (default 23646) and expose the legacy
  http/grpc ports through the admin Service, container ports and
  NetworkPolicy so older workers resolved via the Service keep working.

* helm: only pass -port.legacy when it differs from the binary default

ct install deploys the released image with the new chart, and 4.48 does
not know -port.legacy, so the admin pod crash-loops. Omitting the flag at
the default value is behavior-identical on new binaries (their default
is also 23646) and keeps older releases installable; legacyPort: 0 or a
custom port still renders the flag.
This commit is contained in:
Chris Lu authored and GitHub committed 2026-10-18 21:21:05 -07:00
1 parent d3ee817b2e
commit 8ac5133abe
33 files changed
+316 -101

No files matched your search

+1 -1
View File
@@ -543,7 +543,7 @@ a language-agnostic gRPC stream that external maintenance workers connect on: th
opens `WorkerStream`, sends `WorkerHello` with the job types it can `detect` and `execute`,
answers `RequestConfigSchema` with a `JobTypeDescriptor`, replies to `RunDetectionRequest`
with `JobProposal`s and to `ExecuteJobRequest` with `JobProgressUpdate`s and `JobCompleted`.
`weed worker -admin=host:23646` is the Go reference implementation of exactly that contract,
`weed worker -admin=host:12535` is the Go reference implementation of exactly that contract,
from outside the admin process.
Nothing in it is Go-specific, and the Rust toolchain is already in the tree.
@@ -21,7 +21,7 @@ The testing environment uses **official SeaweedFS commands** and includes:
- **1 Master Server** (port 9333) - Coordinates the cluster with 50MB volume size limit
- **6 Volume Servers** (ports 8080-8085) - Distributed across 2 data centers and 3 racks for diversity
- **1 Filer** (port 8888) - Provides file system interface
- **1 Admin Server** (port 23646) - Detects volumes needing EC and manages workers using official `admin` command
- **1 Admin Server** (port 12535) - Detects volumes needing EC and manages workers using official `admin` command
- **3 EC Workers** - Execute erasure coding tasks using official `worker` command with task-specific working directories
- **1 Load Generator** - Continuously writes and deletes files using SeaweedFS shell commands
- **1 Monitor** - Tracks cluster health and EC progress using shell scripts
@@ -47,7 +47,7 @@ This provides:
- Docker and Docker Compose installed
- GNU Make installed
- At least 4GB RAM available for containers
- Ports 8080-8085, 8888, 9333, 23646 available
- Ports 8080-8085, 8888, 9333, 12535 available
### Start the Environment
@@ -127,7 +127,7 @@ Run `make help` to see all available targets:
|---------|-----|-------------|
| Master UI | http://localhost:9333 | Cluster status and topology |
| Filer | http://localhost:8888 | File operations |
| Admin Server | http://localhost:23646/ | Task management |
| Admin Server | http://localhost:12535/ | Task management |
| Monitor | http://localhost:9999/status | Complete cluster monitoring |
| Volume Servers | http://localhost:8080-8085/status | Individual volume server stats |
+15 -15
View File
@@ -26,7 +26,7 @@ build-and-restart: build ## Build with latest changes and restart services
@echo "2️⃣ Recreating workers to reconnect..."
@docker-compose -f $(COMPOSE_FILE) up -d worker1 worker2 worker3
@echo "✅ All services recreated with latest changes!"
@echo "🌐 Admin UI: http://localhost:23646/"
@echo "🌐 Admin UI: http://localhost:12535/"
@echo "💡 Workers will reconnect to the new admin server"
restart-workers: ## Restart all workers to reconnect to admin server
@@ -64,7 +64,7 @@ start: ## Start the complete SeaweedFS cluster with admin and workers
@echo "✅ Cluster started!"
@echo ""
@echo "📊 Access points:"
@echo " • Admin UI: http://localhost:23646/"
@echo " • Admin UI: http://localhost:12535/"
@echo " • Master UI: http://localhost:9333/"
@echo " • Filer: http://localhost:8888/"
@echo ""
@@ -107,7 +107,7 @@ start-staged: ## Start services in proper order with delays
@echo "✅ All services started!"
@echo ""
@echo "📊 Access points:"
@echo " • Admin UI: http://localhost:23646/"
@echo " • Admin UI: http://localhost:12535/"
@echo " • Master UI: http://localhost:9333/"
@echo " • Filer: http://localhost:8888/"
@echo ""
@@ -136,7 +136,7 @@ status: ## Check the status of all services
@echo "Master:"
@curl -s http://localhost:9333/cluster/status | jq '.IsLeader' 2>/dev/null || echo " ❌ Master not ready"
@echo "Admin:"
@curl -s http://localhost:23646/ | grep -q "Admin" && echo " ✅ Admin ready" || echo " ❌ Admin not ready"
@curl -s http://localhost:12535/ | grep -q "Admin" && echo " ✅ Admin ready" || echo " ❌ Admin not ready"
logs: ## Show logs from all services
@echo "📜 Following logs from all services..."
@@ -157,7 +157,7 @@ master-logs: ## Show logs from master server
admin-ui: ## Open admin UI in browser (macOS)
@echo "🌐 Opening admin UI in browser..."
@open http://localhost:23646/ || echo "💡 Manually open: http://localhost:23646/"
@open http://localhost:12535/ || echo "💡 Manually open: http://localhost:12535/"
test: ## Run integration test to verify task assignment and completion
@echo "🧪 Running Admin-Worker Integration Test"
@@ -166,11 +166,11 @@ test: ## Run integration test to verify task assignment and completion
@echo "1️⃣ Checking cluster health..."
@sleep 5
@curl -s http://localhost:9333/cluster/status | jq '.IsLeader' > /dev/null && echo "✅ Master healthy" || echo "❌ Master not ready"
@curl -s http://localhost:23646/ | grep -q "Admin" && echo "✅ Admin healthy" || echo "❌ Admin not ready"
@curl -s http://localhost:12535/ | grep -q "Admin" && echo "✅ Admin healthy" || echo "❌ Admin not ready"
@echo ""
@echo "2️⃣ Checking worker registration..."
@sleep 10
@echo "💡 Check admin UI for connected workers: http://localhost:23646/"
@echo "💡 Check admin UI for connected workers: http://localhost:12535/"
@echo ""
@echo "3️⃣ Generating load to trigger EC tasks..."
@echo "📝 Creating test files to fill volumes..."
@@ -193,13 +193,13 @@ test: ## Run integration test to verify task assignment and completion
@echo " • EC shards being distributed"
@echo ""
@echo "✅ Integration test setup complete!"
@echo "📊 Monitor progress at: http://localhost:23646/"
@echo "📊 Monitor progress at: http://localhost:12535/"
quick-test: ## Quick verification that core services are running
@echo "⚡ Quick Health Check"
@echo "===================="
@echo "Master: $$(curl -s http://localhost:9333/cluster/status | jq -r '.IsLeader // "not ready"')"
@echo "Admin: $$(curl -s http://localhost:23646/ | grep -q "Admin" && echo "ready" || echo "not ready")"
@echo "Admin: $$(curl -s http://localhost:12535/ | grep -q "Admin" && echo "ready" || echo "not ready")"
@echo "Workers: $$(docker-compose -f $(COMPOSE_FILE) ps worker1 worker2 worker3 | grep -c Up) running"
validate: ## Validate integration test configuration
@@ -223,7 +223,7 @@ demo: start ## Start cluster and run demonstration
@echo " • Load generator (creates files continuously)"
@echo ""
@echo "📊 Watch the process:"
@echo " 1. Visit: http://localhost:23646/"
@echo " 1. Visit: http://localhost:12535/"
@echo " 2. Observe workers connecting"
@echo " 3. Watch tasks being created and assigned"
@echo " 4. See tasks progress from pending → completed"
@@ -244,13 +244,13 @@ vacuum-test: ## Create test data with garbage and verify vacuum detection
@echo ""
@echo "1️⃣ Checking cluster health..."
@curl -s http://localhost:9333/cluster/status | jq '.IsLeader' > /dev/null && echo "✅ Master ready" || (echo "❌ Master not ready. Run 'make start' first." && exit 1)
@curl -s http://localhost:23646/ | grep -q "Admin" && echo "✅ Admin ready" || (echo "❌ Admin not ready. Run 'make start' first." && exit 1)
@curl -s http://localhost:12535/ | grep -q "Admin" && echo "✅ Admin ready" || (echo "❌ Admin not ready. Run 'make start' first." && exit 1)
@echo ""
@echo "2️⃣ Creating test data with garbage..."
@docker-compose -f $(COMPOSE_FILE) exec vacuum-tester go run create_vacuum_test_data.go -files=25 -delete=0.5 -size=200
@echo ""
@echo "3️⃣ Configuration Instructions:"
@echo " Visit: http://localhost:23646/maintenance/config/vacuum"
@echo " Visit: http://localhost:12535/maintenance/config/vacuum"
@echo " Set for testing:"
@echo " • Enable Vacuum Tasks: ✅ Checked"
@echo " • Garbage Threshold: 0.20 (20%)"
@@ -258,7 +258,7 @@ vacuum-test: ## Create test data with garbage and verify vacuum detection
@echo " • Min Volume Age: [0] [Minutes]"
@echo " • Max Concurrent: 2"
@echo ""
@echo "4️⃣ Monitor vacuum tasks at: http://localhost:23646/maintenance"
@echo "4️⃣ Monitor vacuum tasks at: http://localhost:12535/maintenance"
@echo ""
@echo "💡 Use 'make vacuum-status' to check volume garbage ratios"
@@ -335,8 +335,8 @@ vacuum-help: ## Show vacuum testing help and examples
@echo " make vacuum-demo # Full guided demonstration"
@echo ""
@echo "Configuration:"
@echo " Visit: http://localhost:23646/maintenance/config/vacuum"
@echo " Monitor: http://localhost:23646/maintenance"
@echo " Visit: http://localhost:12535/maintenance/config/vacuum"
@echo " Monitor: http://localhost:12535/maintenance"
@echo ""
@echo "Custom Parameters:"
@echo " make vacuum-data FILES=50 DELETE=0.8 SIZE=200"
@@ -243,7 +243,7 @@ func printTestingInstructions() {
fmt.Println("🧪 Testing Instructions:")
fmt.Println()
fmt.Println("1. Configure Vacuum for Testing:")
fmt.Println(" Visit: http://localhost:23646/maintenance/config/vacuum")
fmt.Println(" Visit: http://localhost:12535/maintenance/config/vacuum")
fmt.Println(" Set:")
fmt.Printf(" - Garbage Percentage Threshold: 20 (20%% - lower than default 30)\n")
fmt.Printf(" - Scan Interval: [30] [Seconds] (faster than default)\n")
@@ -253,7 +253,7 @@ func printTestingInstructions() {
fmt.Println()
fmt.Println("2. Monitor Vacuum Tasks:")
fmt.Println(" Visit: http://localhost:23646/maintenance")
fmt.Println(" Visit: http://localhost:12535/maintenance")
fmt.Println(" Watch for vacuum tasks to appear in the queue")
fmt.Println()
@@ -7,7 +7,7 @@ echo ""
# Check if SeaweedFS is running
echo "📋 Checking SeaweedFS status..."
MASTER_URL="${MASTER_HOST:-master:9333}"
ADMIN_URL="${ADMIN_HOST:-admin:23646}"
ADMIN_URL="${ADMIN_HOST:-admin:12535}"
if ! curl -s http://$MASTER_URL/cluster/status > /dev/null; then
echo "❌ SeaweedFS master not running at $MASTER_URL"
@@ -42,7 +42,7 @@ echo ""
# Phase 3: Configure vacuum
echo "⚙️ Phase 3: Vacuum configuration instructions..."
echo " 1. Visit: http://localhost:23646/maintenance/config/vacuum"
echo " 1. Visit: http://localhost:12535/maintenance/config/vacuum"
echo " 2. Set these values for testing:"
echo " - Enable Vacuum Tasks: ✅ Checked"
echo " - Garbage Threshold: 0.30"
@@ -57,7 +57,7 @@ echo ""
# Phase 4: Monitor tasks
echo "🎯 Phase 4: Monitoring vacuum tasks..."
echo " Visit: http://localhost:23646/maintenance"
echo " Visit: http://localhost:12535/maintenance"
echo " You should see vacuum tasks appear within 30 seconds"
echo ""
@@ -93,9 +93,9 @@ echo ""
echo "🎉 Demo Complete!"
echo ""
echo "🔍 Things to check:"
echo " 1. Maintenance Queue: http://localhost:23646/maintenance"
echo " 1. Maintenance Queue: http://localhost:12535/maintenance"
echo " 2. Volume Status: http://localhost:9333/vol/status"
echo " 3. Admin Dashboard: http://localhost:23646"
echo " 3. Admin Dashboard: http://localhost:12535"
echo ""
echo "💡 Next Steps:"
echo " - Try different garbage thresholds (0.10, 0.50, 0.80)"
@@ -114,9 +114,9 @@ services:
admin:
image: chrislusf/seaweedfs:local
ports:
- "23646:23646" # HTTP admin interface (default port)
- "33646:33646" # gRPC worker communication (23646 + 10000)
command: "-v=2 admin -port=23646 -masters=master:9333 -dataDir=/data"
- "12535:12535" # HTTP admin interface (default port)
- "22535:22535" # gRPC worker communication (12535 + 10000)
command: "-v=2 admin -port=12535 -masters=master:9333 -dataDir=/data"
depends_on:
- master
- filer
@@ -127,7 +127,7 @@ services:
worker1:
image: chrislusf/seaweedfs:local
command: "-v=2 worker -admin=admin:23646 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
command: "-v=2 worker -admin=admin:12535 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
depends_on:
- admin
volumes:
@@ -139,7 +139,7 @@ services:
worker2:
image: chrislusf/seaweedfs:local
command: "-v=2 worker -admin=admin:23646 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
command: "-v=2 worker -admin=admin:12535 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
depends_on:
- admin
volumes:
@@ -151,7 +151,7 @@ services:
worker3:
image: chrislusf/seaweedfs:local
command: "-v=2 worker -admin=admin:23646 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
command: "-v=2 worker -admin=admin:12535 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
depends_on:
- admin
volumes:
@@ -199,7 +199,7 @@ services:
curl -s http://master:9333/cluster/status | jq '.IsLeader, .Peers' || echo 'Master not ready';
echo;
echo 'Admin status:';
curl -s http://admin:23646/ | grep -o 'Admin.*Interface' || echo 'Admin not ready';
curl -s http://admin:12535/ | grep -o 'Admin.*Interface' || echo 'Admin not ready';
echo;
echo 'Volume count by server:';
curl -s http://master:9333/vol/status | jq '.Volumes | length' || echo 'Volumes not ready';
@@ -237,4 +237,4 @@ services:
- seaweed_net
environment:
- MASTER_HOST=master:9333
- ADMIN_HOST=admin:23646
- ADMIN_HOST=admin:12535
+2 -2
View File
@@ -23,7 +23,7 @@ else
fi
echo -e "${BLUE}2. Checking if required ports are available...${NC}"
for port in 9333 8080 8081 8082 8083 8084 8085 8888 23646; do
for port in 9333 8080 8081 8082 8083 8084 8085 8888 12535; do
if lsof -i :$port > /dev/null 2>&1; then
echo -e "${YELLOW}⚠️ Port $port is in use${NC}"
else
@@ -41,7 +41,7 @@ else
fi
echo -e "${BLUE}4. Verifying admin server configuration...${NC}"
if docker-compose -f docker-compose-ec-test.yml config | grep -q "admin:23646"; then
if docker-compose -f docker-compose-ec-test.yml config | grep -q "admin:12535"; then
echo -e "${GREEN}✅ Admin server port configuration is correct${NC}"
else
echo -e "${RED}❌ Admin server port configuration is incorrect${NC}"
+1 -1
View File
@@ -45,7 +45,7 @@ services:
admin:
image: chrislusf/seaweedfs:dev # use a remote dev image
ports:
- 23646:23646
- 12535:12535
command: 'admin -master=master:9333'
depends_on:
- master
+1 -1
View File
@@ -350,7 +350,7 @@ main() {
info " weed volume -mserver=localhost:9333 # Start Go volume server"
info " weed-volume -mserver localhost:9333 # Start Rust volume server"
if [ "$WORKER_INSTALLED" = true ]; then
info " weed-worker --admin localhost:23646 # Start the Rust maintenance worker"
info " weed-worker --admin localhost:12535 # Start the Rust maintenance worker"
fi
}
+11 -8
View File
@@ -345,8 +345,9 @@ To enable the admin interface, add the following to your values.yaml:
```yaml
admin:
enabled: true
port: 23646
grpcPort: 33646 # For worker connections
port: 12535
grpcPort: 22535 # For worker connections
legacyPort: 23646 # Compat: also serve http + gRPC (+10000) for pre-12535 clients
secret:
adminUser: "admin"
adminPassword: "your-secure-password" # Leave empty to disable auth
@@ -364,7 +365,9 @@ admin:
className: "nginx"
```
The admin interface will be available at `http://<admin-service>:23646` (or via ingress). Workers connect to the admin server via gRPC on port `33646`.
The admin interface will be available at `http://<admin-service>:12535` (or via ingress). Workers connect to the admin server via gRPC on port `22535`.
For backward compatibility with older clients and workers, admin also serves the previous default ports: HTTP on `23646` and gRPC on `33646` (legacyPort + 10000). The Service and NetworkPolicy expose both. Set `admin.legacyPort: 0` to disable.
### Admin Authentication
@@ -436,7 +439,7 @@ worker:
# Auto-discovery only works if the admin is in the same namespace and same Helm release.
# For cross-namespace or separate release scenarios, explicitly set this value.
# Example: If main SeaweedFS is deployed in "production" namespace:
# adminServer: "seaweedfs-admin.production.svc:33646"
# adminServer: "seaweedfs-admin.production.svc:22535"
adminServer: ""
# Workers need storage for task execution
@@ -502,8 +505,8 @@ worker:
# REQUIRED: Point to the admin service of your main SeaweedFS release
# Replace <namespace> with the namespace where your main seaweedfs is deployed
# Example: If deploying in namespace "production":
# adminServer: "seaweedfs-admin.production.svc:33646"
adminServer: "seaweedfs-admin.<namespace>.svc:33646"
# adminServer: "seaweedfs-admin.production.svc:22535"
adminServer: "seaweedfs-admin.<namespace>.svc:22535"
```
**values-worker-balance.yaml** (for balance operations):
@@ -528,8 +531,8 @@ worker:
# REQUIRED: Point to the admin service of your main SeaweedFS release
# Replace <namespace> with the namespace where your main seaweedfs is deployed
# Example: If deploying in namespace "production":
# adminServer: "seaweedfs-admin.production.svc:33646"
adminServer: "seaweedfs-admin.<namespace>.svc:33646"
# adminServer: "seaweedfs-admin.production.svc:22535"
adminServer: "seaweedfs-admin.<namespace>.svc:22535"
```
Deploy the specialized workers as separate releases:
@@ -37,6 +37,16 @@ spec:
nodePort: {{ $nodePorts.grpc }}
{{- end }}
protocol: TCP
{{- if gt (int .Values.admin.legacyPort) 0 }}
- name: "legacy-http"
port: {{ .Values.admin.legacyPort }}
targetPort: {{ .Values.admin.legacyPort }}
protocol: TCP
- name: "legacy-grpc"
port: {{ add (int .Values.admin.legacyPort) 10000 }}
targetPort: {{ add (int .Values.admin.legacyPort) 10000 }}
protocol: TCP
{{- end }}
selector:
app.kubernetes.io/name: {{ template "seaweedfs.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
@@ -170,6 +170,9 @@ spec:
-ip={{ .Values.admin.ip | default "0.0.0.0" }} \
-port={{ .Values.admin.port }} \
-port.grpc={{ .Values.admin.grpcPort }} \
{{- if ne (int .Values.admin.legacyPort) 23646 }}
-port.legacy={{ .Values.admin.legacyPort }} \
{{- end }}
{{- if or (eq .Values.admin.data.type "hostPath") (eq .Values.admin.data.type "persistentVolumeClaim") (eq .Values.admin.data.type "emptyDir") (eq .Values.admin.data.type "existingClaim") }}
-dataDir=/data \
{{- else if .Values.admin.dataDir }}
@@ -233,6 +236,12 @@ spec:
name: http
- containerPort: {{ .Values.admin.grpcPort }}
name: grpc
{{- if gt (int .Values.admin.legacyPort) 0 }}
- containerPort: {{ .Values.admin.legacyPort }}
name: legacy-http
- containerPort: {{ add (int .Values.admin.legacyPort) 10000 }}
name: legacy-grpc
{{- end }}
{{- if .Values.admin.readinessProbe.enabled }}
readinessProbe:
httpGet:
@@ -93,7 +93,11 @@
ClusterRole under global.seaweedfs.createClusterRole is a leftover from a
migration and is not exercised by any component. */}}
{{- if .Values.admin.enabled }}
{{- $targets = append $targets (dict "component" "admin" "ports" (list .Values.admin.port .Values.admin.grpcPort)) }}
{{- $ports := list .Values.admin.port .Values.admin.grpcPort }}
{{- if gt (int .Values.admin.legacyPort) 0 }}
{{- $ports = concat $ports (list .Values.admin.legacyPort (add (int .Values.admin.legacyPort) 10000)) }}
{{- end }}
{{- $targets = append $targets (dict "component" "admin" "ports" $ports) }}
{{- end }}
{{- if .Values.worker.enabled }}
{{- $ports := list }}
+8 -2
View File
@@ -1511,8 +1511,14 @@ admin:
imageOverride: null
restartPolicy: null
replicas: 1
port: 23646 # Default admin port
grpcPort: 33646 # Default gRPC port for worker connections
port: 12535 # Default admin port
grpcPort: 22535 # Default gRPC port for worker connections
# Legacy compat port: admin also serves HTTP and worker gRPC (legacyPort +
# 10000) on this port so workers and clients from before the 23646 -> 12535
# default change keep connecting during migration. 0 disables. The flag is
# only passed to the binary when the value differs from its 23646 default,
# so the chart keeps installing on releases that predate -port.legacy.
legacyPort: 23646
# IP address the admin server listens on. Since `weed admin` 4.46 defaults to
# loopback (127.0.0.1), the chart must bind a non-loopback address for the
# kubelet's httpGet readiness/liveness probes (which dial the pod IP) to ever
+4 -4
View File
@@ -2,7 +2,7 @@
`weed/pb/plugin.proto` is a language-agnostic contract: a maintenance worker
connects out to admin, announces the job types it can detect and execute, and
answers requests on that one stream. `weed worker -admin=host:23646` is the Go
answers requests on that one stream. `weed worker -admin=host:12535` is the Go
implementation of it from outside the admin process. This workspace is the Rust
one.
@@ -35,7 +35,7 @@ binary for the runner's platform, resolved from the version in `Cargo.lock`.
## Running
cargo run -p weed-lance-worker -- --admin 127.0.0.1:23646
cargo run -p weed-lance-worker -- --admin 127.0.0.1:12535
The admin's *HTTP* address is what an operator has; the gRPC port is derived
from it the way the Go side does. Dialling the HTTP port fails as "frame with
@@ -49,7 +49,7 @@ Released builds do not need a toolchain. The worker ships inside the SeaweedFS
image, beside the Rust volume server, under the verb that mirrors
`volume-rust`:
docker run chrislusf/seaweedfs worker-rust --admin admin:23646
docker run chrislusf/seaweedfs worker-rust --admin admin:12535
and as `weed-worker_linux_{amd64,arm64}.tar.gz` on each GitHub release. Both are
linux amd64/arm64 only — lance, arrow and datafusion make every extra target an
@@ -59,7 +59,7 @@ entrypoint says so rather than failing as "not found".
## Metrics
cargo run -p weed-lance-worker -- --admin 127.0.0.1:23646 --metrics-port 9328
cargo run -p weed-lance-worker -- --admin 127.0.0.1:12535 --metrics-port 9328
Serves `/health`, `/ready` and `/metrics` on that port, the same three the Go
worker serves under `weed worker -metricsPort`, so one scrape config covers
+2 -2
View File
@@ -3,7 +3,7 @@ use std::time::Duration;
/// How one worker process connects and how much work it will take on.
#[derive(Clone, Debug)]
pub struct WorkerOptions {
/// Admin gRPC address, e.g. "localhost:23646".
/// Admin gRPC address, e.g. "localhost:12535".
pub admin_address: String,
pub worker_id: String,
pub worker_version: String,
@@ -44,7 +44,7 @@ pub struct TlsOptions {
impl Default for WorkerOptions {
fn default() -> Self {
Self {
admin_address: "localhost:23646".to_string(),
admin_address: "localhost:12535".to_string(),
worker_id: String::new(),
worker_version: env!("CARGO_PKG_VERSION").to_string(),
worker_address: String::new(),
+1 -1
View File
@@ -2,7 +2,7 @@
//!
//! `weed/pb/plugin.proto` is a language-agnostic gRPC stream: a worker connects
//! out to admin, announces the job types it can detect and execute, and then
//! answers requests on that one stream. `weed worker -admin=host:23646` is the
//! answers requests on that one stream. `weed worker -admin=host:12535` is the
//! Go implementation of the same contract from outside the admin process; this
//! is the Rust one, and it knows nothing about any particular job.
+1 -1
View File
@@ -15,7 +15,7 @@ use weed_lance_worker::metrics::LanceMetrics;
#[command(name = "weed-worker", about = "SeaweedFS maintenance worker")]
struct Args {
/// Admin server gRPC address.
#[arg(long, default_value = "localhost:23646", env = "WEED_ADMIN")]
#[arg(long, default_value = "localhost:12535", env = "WEED_ADMIN")]
admin: String,
/// Worker identity reported to admin.
+1 -1
View File
@@ -1,6 +1,6 @@
# Cluster security group + the SeaweedFS port matrix.
# Intra-cluster: all TCP allowed within the SG (master 9333/19333, volume
# 8080/18080, filer 8888/18888, admin 33646, ...). Client-facing: only S3 (8333)
# 8080/18080, filer 8888/18888, admin 22535, ...). Client-facing: only S3 (8333)
# and filer (8888). Metrics ports (9327) are never opened to clients.
resource "aws_security_group" "cluster" {
@@ -18,7 +18,7 @@ import (
)
const (
AdminUrl = "http://localhost:23646"
AdminUrl = "http://localhost:12535"
MasterUrl = "http://localhost:9333"
FilerUrl = "http://localhost:8888"
)
@@ -147,13 +147,13 @@ func ensureEnvironment(t *testing.T) {
debugPort := 6060 + i
dir, _ := filepath.Abs(filepath.Join("tmp", workerName))
os.MkdirAll(dir, 0755)
startWeed(t, workerName, "worker", "-admin=localhost:23646", "-workingDir="+dir, fmt.Sprintf("-metricsPort=%d", metricsPort), fmt.Sprintf("-debug.port=%d", debugPort))
startWeed(t, workerName, "worker", "-admin=localhost:12535", "-workingDir="+dir, fmt.Sprintf("-metricsPort=%d", metricsPort), fmt.Sprintf("-debug.port=%d", debugPort))
}
// 6. Start Admin
os.RemoveAll(filepath.Join("tmp", "admin"))
os.MkdirAll(filepath.Join("tmp", "admin"), 0755)
startWeed(t, "admin", "admin", "-master=localhost:9333", "-port=23646", "-dataDir=./tmp/admin")
startWeed(t, "admin", "admin", "-master=localhost:9333", "-port=12535", "-dataDir=./tmp/admin")
// Admin is started after master, 14 volume servers, filer and 2 workers,
// so under cold CI conditions the wait here has to absorb the tail of
// every earlier subprocess coming up. 60s is too tight and has flaked;
+1 -1
View File
@@ -12,7 +12,7 @@ FILER_PORT := 8888
# Pin the filer gRPC port off the FILER_PORT+10000 convention so any
# code path that assumes the offset breaks here, not in production.
FILER_GRPC_PORT := 18890
ADMIN_PORT := 23646
ADMIN_PORT := 12535
ACCESS_KEY ?= some_access_key1
SECRET_KEY ?= some_secret_key1
TEST_TIMEOUT := 10m
+1 -1
View File
@@ -41,7 +41,7 @@ const (
defaultS3Endpoint = "http://localhost:8333"
defaultS3GrpcEndpoint = "localhost:18333"
defaultMasterEndpt = "http://localhost:9333"
defaultAdminEndpoint = "http://localhost:23646"
defaultAdminEndpoint = "http://localhost:12535"
// Pinned off the FILER_PORT+10000 convention; see Makefile.
defaultFilerGRPC = "localhost:18890"
bucketLifecycleXMLKey = "s3-bucket-lifecycle-configuration-xml"
+2 -2
View File
@@ -705,10 +705,10 @@ func uniqueName(prefix string) string {
func startMiniCluster(t *testing.T) (*TestCluster, error) {
// Allocate two extra ports for admin HTTP + gRPC. Without explicit
// values mini falls back to 23646/33646, which collide across the
// values mini falls back to 12535/22535, which collide across the
// sequential subtests in this package: each subtest spins up a fresh
// mini in a goroutine, but the previous mini's admin goroutine is
// still binding 23646 — so the next test's admin can never become
// still binding 12535 — so the next test's admin can never become
// ready and mini.go fatals on its readiness check.
ports := testutil.MustAllocatePorts(t, 10)
masterPort, masterGrpcPort := ports[0], ports[1]
+1 -1
View File
@@ -40,7 +40,7 @@ var miniDefaultPorts = []int{
8181, // s3.port.iceberg
9101, // s3.port.lance
7333, // webdav.port
23646, // admin.port
12535, // admin.port
}
func reservedMiniPorts() map[int]bool {
+1 -1
View File
@@ -4,7 +4,7 @@ import "testing"
// AllocateMiniPorts must never hand out a port that weed mini will reserve
// for one of its default services (or that default's gRPC offset). A real
// failure: Filer was given 33646 (Admin default 23646 + GrpcPortOffset),
// failure: Filer was given 22535 (Admin default 12535 + GrpcPortOffset),
// which mini then refused as "reserved for gRPC calculation".
func TestAllocateMiniPortsAvoidsMiniDefaults(t *testing.T) {
reserved := reservedMiniPorts()
+8
View File
@@ -1642,6 +1642,14 @@ func (s *AdminServer) WorkerGrpcMTLSEnabled() bool {
return s.workerGrpcServer != nil && s.workerGrpcServer.mtlsEnabled
}
// ServeWorkerGrpcOn serves the worker gRPC API on an additional already-bound
// listener, e.g. a legacy compat port kept for backward compatibility.
func (s *AdminServer) ServeWorkerGrpcOn(listener net.Listener) {
if s.workerGrpcServer != nil {
s.workerGrpcServer.ServeOn(listener)
}
}
// StopWorkerGrpcServer stops the worker gRPC server
func (s *AdminServer) StopWorkerGrpcServer() error {
if s.workerGrpcServer != nil {
+11
View File
@@ -133,6 +133,17 @@ func (s *WorkerGrpcServer) StartWithTLS(bindIp string, port int, listener net.Li
return nil
}
// ServeOn serves the worker gRPC API on an additional already-bound listener,
// e.g. a legacy compat port kept alongside the primary one. grpc.Server tracks
// every listener passed to Serve, so Stop closes it together with the primary.
func (s *WorkerGrpcServer) ServeOn(listener net.Listener) {
go func() {
if err := s.grpcServer.Serve(listener); err != nil && s.running {
glog.Warningf("Worker gRPC compat listener on %s stopped: %v", listener.Addr(), err)
}
}()
}
// ListenPort returns the currently bound worker gRPC listen port.
func (s *WorkerGrpcServer) ListenPort() int {
if s == nil || s.listener == nil {
+3 -3
View File
@@ -180,9 +180,9 @@ Phase 4 status (starter)
Run example:
- Start admin: `weed admin -master=localhost:9333`
- Start worker: `weed worker -admin=localhost:23646`
- Optional explicit job type: `weed worker -admin=localhost:23646 -jobType=vacuum`
- Optional stable worker ID persistence: `weed worker -admin=localhost:23646 -workingDir=/var/lib/seaweedfs-plugin`
- Start worker: `weed worker -admin=localhost:12535`
- Optional explicit job type: `weed worker -admin=localhost:12535 -jobType=vacuum`
- Optional stable worker ID persistence: `weed worker -admin=localhost:12535 -workingDir=/var/lib/seaweedfs-plugin`
Phase 5
- Migrate remaining job types and deprecate old mechanism.
+95 -7
View File
@@ -5,6 +5,7 @@ import (
"context"
"crypto/rand"
"crypto/tls"
"errors"
"fmt"
"log"
"net"
@@ -42,6 +43,7 @@ var (
type AdminOptions struct {
port *int
grpcPort *int
legacyPort *int
ip *string
master *string
masters *string // deprecated, for backward compatibility
@@ -68,6 +70,23 @@ type AdminOptions struct {
// binds it only after every other service is up.
workerGrpcListener net.Listener
// workerGrpcLegacyListener is the same reservation for the legacy
// compat gRPC port. Its bind failure is tolerated: the legacy port lives
// in the ephemeral range, so best-effort compat must not take startup down.
workerGrpcLegacyListener net.Listener
// workerGrpcLegacySkip disables just the legacy gRPC listener. `weed
// mini` sets it when another mini service already owns the legacy gRPC
// port, so the compat bind can neither steal the port nor warn about a
// failure that is expected.
workerGrpcLegacySkip bool
// adminHttpLegacySkip is the same for the legacy HTTP port. Skipping
// matters even when the bind would succeed: a service bound on a
// different address family or wildcard can leave loopback free, and the
// compat listener would then shadow it for loopback clients.
adminHttpLegacySkip bool
// workerGrpcBindIp, when non-empty, is the address the worker gRPC
// listener binds to. It is separate from ip because the worker gRPC has
// no password auth (its mTLS comes from grpc.admin, not https.admin), so
@@ -83,8 +102,9 @@ type AdminOptions struct {
func init() {
cmdAdmin.Run = runAdmin // break init cycle
a.port = cmdAdmin.Flag.Int("port", 23646, "admin server port")
a.port = cmdAdmin.Flag.Int("port", defaultAdminHTTPPort, "admin server port")
a.grpcPort = cmdAdmin.Flag.Int("port.grpc", 0, "gRPC server port for worker connections (default: http port + 10000)")
a.legacyPort = cmdAdmin.Flag.Int("port.legacy", defaultAdminHTTPLegacyPort, "also serve admin http and gRPC (port + 10000) on this port for backward compatibility (0 to disable)")
a.ip = cmdAdmin.Flag.String("ip", "127.0.0.1", "ip address to listen on. Defaults to loopback when auth is disabled, or 0.0.0.0 when -adminPassword or [https.admin] mTLS is configured. Set explicitly to override.")
a.master = cmdAdmin.Flag.String("master", "localhost:9333", "comma-separated master servers")
a.masters = cmdAdmin.Flag.String("masters", "", "comma-separated master servers (deprecated, use -master instead)")
@@ -108,7 +128,7 @@ func init() {
}
var cmdAdmin = &Command{
UsageLine: "admin -port=23646 -master=localhost:9333 [-filerGroup=group] [-port.grpc=33646] [-dataDir=/path/to/data]",
UsageLine: "admin -port=12535 -master=localhost:9333 [-filerGroup=group] [-port.grpc=22535] [-dataDir=/path/to/data]",
Short: "start SeaweedFS web admin interface",
Long: `Start a web admin interface for SeaweedFS cluster management.
@@ -124,12 +144,12 @@ var cmdAdmin = &Command{
A gRPC server for worker connections runs on the configured gRPC port (default: HTTP port + 10000).
Example Usage:
weed admin -port=23646 -master="master1:9333,master2:9333"
weed admin -port=23646 -master="localhost:9333" -filerGroup="tenant-a"
weed admin -port=23646 -master="localhost:9333" -dataDir="/var/lib/seaweedfs-admin"
weed admin -port=23646 -port.grpc=33646 -master="localhost:9333" -dataDir="~/seaweedfs-admin"
weed admin -port=12535 -master="master1:9333,master2:9333"
weed admin -port=12535 -master="localhost:9333" -filerGroup="tenant-a"
weed admin -port=12535 -master="localhost:9333" -dataDir="/var/lib/seaweedfs-admin"
weed admin -port=12535 -port.grpc=22535 -master="localhost:9333" -dataDir="~/seaweedfs-admin"
weed admin -port=9900 -port.grpc=19900 -master="localhost:9333"
weed admin -port=23646 -master="localhost:9333" -urlPrefix="/seaweedfs"
weed admin -port=12535 -master="localhost:9333" -urlPrefix="/seaweedfs"
Data Directory:
- If dataDir is specified, admin configuration and maintenance data is persisted
@@ -295,6 +315,9 @@ func runAdmin(cmd *Command, args []string) bool {
if *a.grpcPort == 0 {
*a.grpcPort = *a.port + 10000
}
if *a.grpcPort >= linuxEphemeralPortFloor {
glog.Warningf("Admin gRPC port %d is at or above the Linux ephemeral floor %d; an outbound connection may claim it before Admin binds it. Pick a lower -port.", *a.grpcPort, linuxEphemeralPortFloor)
}
hasMTLS := viper.GetString("https.admin.key") != "" && viper.GetString("https.admin.ca") != ""
@@ -504,6 +527,31 @@ func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool,
}
warnInsecureWorkerGrpcBind(workerGrpcIp, *options.grpcPort, adminServer.WorkerGrpcMTLSEnabled())
// Legacy compat listeners keep the previous default admin port reachable
// for old workers, bookmarks and scripts while the deployment moves to the
// new default. Best-effort only: the legacy gRPC port sits inside the
// Linux ephemeral range, so it may already be taken and must never fail
// startup.
legacyPort := 0
if options.legacyPort != nil && *options.legacyPort > 0 && *options.legacyPort != *options.port {
legacyPort = *options.legacyPort
}
if legacyPort > 0 && !options.workerGrpcLegacySkip {
legacyGrpcPort := legacyPort + GrpcPortOffset
legacyGrpcListener := options.workerGrpcLegacyListener
if legacyGrpcListener == nil {
if l, listenErr := net.Listen("tcp", util.JoinHostPort(workerGrpcIp, legacyGrpcPort)); listenErr == nil {
legacyGrpcListener = l
} else {
glog.Warningf("Legacy admin gRPC port %d unavailable, skipping compat listener: %v", legacyGrpcPort, listenErr)
}
}
if legacyGrpcListener != nil {
adminServer.ServeWorkerGrpcOn(legacyGrpcListener)
glog.Infof("Worker gRPC compat listener on %s", legacyGrpcListener.Addr())
}
}
// Set up cleanup for gRPC server
defer func() {
if stopErr := adminServer.StopWorkerGrpcServer(); stopErr != nil {
@@ -578,6 +626,43 @@ func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool,
server.TLSConfig.GetCertificate = getCert
}
// Serve the same handler on the legacy port so old clients keep working.
// Best-effort like the gRPC compat listener above.
var legacyServer *http.Server
if legacyPort > 0 && !options.adminHttpLegacySkip {
legacyAddr := util.JoinHostPort(*options.ip, legacyPort)
if legacyListener, listenErr := net.Listen("tcp", legacyAddr); listenErr != nil {
glog.Warningf("Legacy admin HTTP port %d unavailable, skipping compat listener: %v", legacyPort, listenErr)
} else {
legacyServer = &http.Server{
Addr: legacyAddr,
Handler: server.Handler,
TLSConfig: server.TLSConfig,
}
// Close on every exit path so a failed primary startup cannot
// leave the compat port held by a zombie admin. On the normal
// path the graceful Shutdown below runs first; this Close then
// no-ops. Close the raw listener too: Serve may not have
// registered it with legacyServer yet when this runs.
defer func() {
_ = legacyListener.Close()
_ = legacyServer.Close()
}()
go func() {
glog.Infof("Admin compat listener on %s", legacyAddr)
var serveErr error
if useTLS {
serveErr = legacyServer.ServeTLS(legacyListener, "", "")
} else {
serveErr = legacyServer.Serve(legacyListener)
}
if serveErr != nil && serveErr != http.ErrServerClosed && !errors.Is(serveErr, net.ErrClosed) {
glog.Warningf("Admin compat listener on %s stopped: %v", legacyAddr, serveErr)
}
}()
}
}
// Start server. Surface immediate failures (e.g. bind error) via
// serveErrCh so the caller doesn't block on ctx.Done() while no server
// is actually listening. http.ErrServerClosed after Shutdown is normal
@@ -612,6 +697,9 @@ func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool,
if err := server.Shutdown(shutdownCtx); err != nil {
return fmt.Errorf("admin server forced to shutdown: %w", err)
}
if legacyServer != nil {
_ = legacyServer.Shutdown(shutdownCtx)
}
adminServer.Shutdown()
+57 -12
View File
@@ -49,13 +49,16 @@ type MiniOptions struct {
}
const (
bytesPerMB = 1024 * 1024 // Bytes per MB
miniVolumeMinFreeSpace = "1" // 1% minimum free space
minVolumeSizeMB = 64 // Minimum volume size in MB
defaultMiniVolumeSizeMB = 128 // Default volume size for mini mode
maxVolumeSizeMB = 1024 // Maximum volume size in MB (1GB)
GrpcPortOffset = 10000 // Offset used to calculate gRPC port from HTTP port
defaultMiniPluginJobTypes = "all"
bytesPerMB = 1024 * 1024 // Bytes per MB
miniVolumeMinFreeSpace = "1" // 1% minimum free space
minVolumeSizeMB = 64 // Minimum volume size in MB
defaultMiniVolumeSizeMB = 128 // Default volume size for mini mode
maxVolumeSizeMB = 1024 // Maximum volume size in MB (1GB)
GrpcPortOffset = 10000 // Offset used to calculate gRPC port from HTTP port
linuxEphemeralPortFloor = 32768 // Default net.ipv4.ip_local_port_range floor on Linux
defaultAdminHTTPPort = 12535 // Keeps the derived admin gRPC port (port+GrpcPortOffset) below linuxEphemeralPortFloor
defaultAdminHTTPLegacyPort = 23646 // Pre-12535 default; served alongside the new port for backward compatibility
defaultMiniPluginJobTypes = "all"
)
var (
@@ -371,12 +374,12 @@ After starting, you can access:
- S3 Endpoint: http://localhost:8333
- Iceberg Catalog: http://localhost:8181
- WebDAV: http://localhost:7333
- Admin UI: http://localhost:23646
- Admin UI: http://localhost:12535
S3 Access:
The S3 endpoint is available at http://localhost:8333. For client
configuration and IAM setup, see the project documentation or use the
Admin UI (http://localhost:23646) to manage users and policies.
Admin UI (http://localhost:12535) to manage users and policies.
Notes:
- Values mini computes itself (currently master.volumeSizeLimitMB) are
@@ -574,8 +577,9 @@ func initMiniWebDAVFlags() {
// initMiniAdminFlags initializes Admin server flag options
func initMiniAdminFlags() {
miniAdminOptions.port = cmdMini.Flag.Int("admin.port", 23646, "admin server http listen port")
miniAdminOptions.port = cmdMini.Flag.Int("admin.port", defaultAdminHTTPPort, "admin server http listen port")
miniAdminOptions.grpcPort = cmdMini.Flag.Int("admin.port.grpc", 0, "admin server grpc listen port (default: admin http port + GrpcPortOffset)")
miniAdminOptions.legacyPort = cmdMini.Flag.Int("admin.port.legacy", defaultAdminHTTPLegacyPort, "also serve admin http and grpc (port + GrpcPortOffset) on this port for backward compatibility (0 to disable)")
miniAdminOptions.master = cmdMini.Flag.String("admin.master", "", "master server address (automatically set)")
// admin discovery must query the same group the co-located filer registers under
miniAdminOptions.filerGroup = miniFilerOptions.filerGroup
@@ -966,7 +970,7 @@ func ensureAllPortsAvailableOnIP(bindIp string) error {
// Initialize all gRPC ports before services start
// This ensures they won't be recalculated and cause conflicts
// All gRPC port handling (calculation, validation, and assignment) is performed exclusively in initializeGrpcPortsOnIP
initializeGrpcPortsOnIP(bindIp)
allocatedPorts := initializeGrpcPortsOnIP(bindIp)
// Every other service binds within a moment of this check, but the admin
// waits for all of them first. The admin gRPC port sits inside the Linux
@@ -986,6 +990,33 @@ func ensureAllPortsAvailableOnIP(bindIp string) error {
miniAdminOptions.workerGrpcListener = listener
}
// Same early reservation for the legacy compat gRPC port — but never take
// a port a mini service already owns: compat is a convenience, not a
// startup requirement. Failure is only a warning — the legacy port sits
// inside the ephemeral range. Without a reservation the admin's late
// best-effort bind simply fails harmlessly if the port is taken by then.
miniAdminOptions.workerGrpcLegacyListener = nil
miniAdminOptions.workerGrpcLegacySkip = false
miniAdminOptions.adminHttpLegacySkip = false
if *miniAdminOptions.legacyPort > 0 && *miniAdminOptions.legacyPort != *miniAdminOptions.port {
if allocatedPorts[*miniAdminOptions.legacyPort] {
glog.Warningf("Legacy Admin HTTP port %d is used by another mini service, skipping compat listener", *miniAdminOptions.legacyPort)
miniAdminOptions.adminHttpLegacySkip = true
}
legacyGrpcPort := *miniAdminOptions.legacyPort + GrpcPortOffset
switch {
case allocatedPorts[legacyGrpcPort]:
glog.Warningf("Legacy Admin gRPC port %d is used by another mini service, skipping compat listener", legacyGrpcPort)
miniAdminOptions.workerGrpcLegacySkip = true
default:
if listener, err := listenMiniAdminWorker(workerGrpcBindIP, legacyGrpcPort); err != nil {
glog.Warningf("Could not reserve legacy Admin gRPC port %d: %v", legacyGrpcPort, err)
} else {
miniAdminOptions.workerGrpcLegacyListener = listener
}
}
}
// Log the final port configuration
icebergPortStr := "disabled"
if miniS3Options.portIceberg != nil && *miniS3Options.portIceberg > 0 {
@@ -1016,7 +1047,9 @@ func listenMiniAdminWorker(bindIP string, port int) (net.Listener, error) {
// initializeGrpcPortsOnIP initializes all gRPC ports based on their HTTP ports on a specific IP
// If a gRPC port is 0, it will be set to httpPort + GrpcPortOffset
// This must be called after HTTP ports are finalized and before services start
func initializeGrpcPortsOnIP(bindIp string) {
// Returns the set of ports allocated to mini services, for callers that must
// keep optional listeners from stealing them.
func initializeGrpcPortsOnIP(bindIp string) map[int]bool {
// Track all ports allocated (both HTTP and gRPC) to prevent collisions.
// We must reserve HTTP ports so that gRPC fallback allocation never picks
// a port already assigned to an HTTP service (which hasn't bound yet).
@@ -1086,6 +1119,15 @@ func initializeGrpcPortsOnIP(bindIp string) {
allocatedPorts[*config.grpcPort] = true
glog.V(1).Infof("%s gRPC port set to %d", config.name, *config.grpcPort)
}
// The Linux kernel draws outbound source ports from the ephemeral range.
// A gRPC port inside it can be claimed by one of mini's own startup dials
// before the service binds it.
if *miniAdminOptions.grpcPort >= linuxEphemeralPortFloor {
glog.Warningf("Admin gRPC port %d is at or above the Linux ephemeral floor %d; an outbound connection may claim it before Admin binds it. Pick a lower -admin.port.", *miniAdminOptions.grpcPort, linuxEphemeralPortFloor)
}
return allocatedPorts
}
// miniOptionsFilePath returns the path of the mini.options file under the
@@ -1880,6 +1922,9 @@ func startMiniAdminWithWorker(allServicesReady chan struct{}) {
if adminOptions.workerGrpcListener != nil {
adminOptions.workerGrpcListener.Close()
}
if adminOptions.workerGrpcLegacyListener != nil {
adminOptions.workerGrpcLegacyListener.Close()
}
}()
// Wait for admin server's HTTP port to be ready before launching worker
+30
View File
@@ -2,6 +2,7 @@ package command
import (
"net"
"strconv"
"testing"
"github.com/seaweedfs/seaweedfs/weed/pb"
@@ -254,3 +255,32 @@ func TestMiniAdminAdvertisedIP(t *testing.T) {
})
}
}
func TestDefaultAdminGrpcPortBelowEphemeralFloor(t *testing.T) {
if grpcPort := defaultAdminHTTPPort + GrpcPortOffset; grpcPort >= linuxEphemeralPortFloor {
t.Fatalf("default admin HTTP port %d derives gRPC port %d, which must stay below ephemeral floor %d",
defaultAdminHTTPPort, grpcPort, linuxEphemeralPortFloor)
}
for _, f := range []struct {
name string
flagName string
def string
}{
{"mini", "admin.port", strconv.Itoa(defaultAdminHTTPPort)},
{"mini", "admin.port.legacy", strconv.Itoa(defaultAdminHTTPLegacyPort)},
{"admin", "port", strconv.Itoa(defaultAdminHTTPPort)},
{"admin", "port.legacy", strconv.Itoa(defaultAdminHTTPLegacyPort)},
} {
cmd := cmdMini
if f.name == "admin" {
cmd = cmdAdmin
}
flag := cmd.Flag.Lookup(f.flagName)
if flag == nil {
t.Fatalf("%s: flag %s not registered", f.name, f.flagName)
}
if flag.DefValue != f.def {
t.Fatalf("%s -%s default = %s, want %s so its gRPC port stays below the ephemeral floor", f.name, f.flagName, flag.DefValue, f.def)
}
}
}
+12 -11
View File
@@ -1,6 +1,7 @@
package command
import (
"fmt"
"time"
"github.com/seaweedfs/seaweedfs/weed/util"
@@ -25,21 +26,21 @@ Behavior:
- Use -metricsPort/-metricsIp to expose /health, /ready, and /metrics
Examples:
weed worker -admin=localhost:23646
weed worker -admin=localhost:23646 -jobType=all
weed worker -admin=localhost:23646 -jobType=default
weed worker -admin=localhost:23646 -jobType=heavy
weed worker -admin=localhost:23646 -jobType=default,iceberg
weed worker -admin=localhost:23646 -jobType=vacuum,volume_balance
weed worker -admin=localhost:23646 -jobType=erasure_coding
weed worker -admin=admin.example.com:23646 -id=plugin-vacuum-a -heartbeat=10s
weed worker -admin=localhost:23646 -workingDir=/var/lib/seaweedfs-plugin
weed worker -admin=localhost:23646 -metricsPort=9327 -metricsIp=0.0.0.0
weed worker -admin=localhost:12535
weed worker -admin=localhost:12535 -jobType=all
weed worker -admin=localhost:12535 -jobType=default
weed worker -admin=localhost:12535 -jobType=heavy
weed worker -admin=localhost:12535 -jobType=default,iceberg
weed worker -admin=localhost:12535 -jobType=vacuum,volume_balance
weed worker -admin=localhost:12535 -jobType=erasure_coding
weed worker -admin=admin.example.com:12535 -id=plugin-vacuum-a -heartbeat=10s
weed worker -admin=localhost:12535 -workingDir=/var/lib/seaweedfs-plugin
weed worker -admin=localhost:12535 -metricsPort=9327 -metricsIp=0.0.0.0
`,
}
var (
workerAdminServer = cmdWorker.Flag.String("admin", "localhost:23646", "admin server address")
workerAdminServer = cmdWorker.Flag.String("admin", fmt.Sprintf("localhost:%d", defaultAdminHTTPPort), "admin server address")
workerID = cmdWorker.Flag.String("id", "", "worker ID (auto-generated when empty)")
workerWorkingDir = cmdWorker.Flag.String("workingDir", "", "working directory for persistent worker state")
workerJobType = cmdWorker.Flag.String("jobType", defaultPluginWorkerJobTypes, "job types or categories to serve: all, default, heavy, or explicit names/aliases such as ec, balance, iceberg (comma-separated)")
+1 -1
View File
@@ -45,7 +45,7 @@ func runPluginWorkerWithOptions(options pluginWorkerRunOptions) bool {
options.AdminServer = strings.TrimSpace(options.AdminServer)
if options.AdminServer == "" {
options.AdminServer = "localhost:23646"
options.AdminServer = fmt.Sprintf("localhost:%d", defaultAdminHTTPPort)
}
options.JobTypes = strings.TrimSpace(options.JobTypes)