0 Fork 0

mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-10-19 20:44:28 +02:00

Files

T

Chris Lu 8ac5133abe admin: default to port 12535, keep 23646 as a best-effort compat listener (#11816 )

* admin: default to port 12535, keep 23646 as a best-effort compat listener

The default admin HTTP port 23646 derives a worker gRPC port of 33646
(port + GrpcPortOffset), inside the Linux default ephemeral range
(32768+). One of our own startup dials can claim 33646 before the admin
binds it, killing startup. This is the same failure #10209 fixed for
the test port allocator, still reachable via the mini default (#10838)
and via standalone `weed admin` defaults.

Move the default admin port to 12535 (gRPC 22535) everywhere, keeping
the http+10000 convention so worker -admin derivation, the Rust worker
and host:http.grpc addressing keep working:

- weed admin -port, weed mini -admin.port
- weed worker -admin, the plugin worker fallback, Rust worker and
  weed-lance-worker defaults
- help text, README, helm values, docker compose, tests

So old workers, bookmarks and scripts keep working, admin also serves
the same handler and the worker gRPC server on the previous default
(-port.legacy / -admin.port.legacy, default 23646, 0 disables). The
legacy gRPC port binds best-effort: it lives in the ephemeral range, so
a failed bind only warns instead of reintroducing the startup failure.
weed mini reserves it early like the primary port and keeps it out of
its own port relocation. grpc.Server tracks every listener handed to
Serve, so the compat listener stops with the primary.

Also warn when an explicitly configured admin port puts the derived
gRPC port at or above the ephemeral floor, the other fix suggested in
the issue.

Supersedes the pinned-grpc approach in #11814: pinning -admin.port.grpc
off the +10000 convention breaks external workers that derive the admin
gRPC port from the HTTP address. ClickHouse/ClickHouse#115512 pins
-admin.port on their side; this makes that pin unnecessary.

Fixes #10838

* admin: let services win the legacy ports, close compat listener on failure

Review findings on the compat listeners:

- Do not put the legacy ports in mini's reservedPorts: an explicit
  -s3.port=23646 or -master.port.grpc=33646 must be honored, not rejected
  or stolen by the early reservation. Mini now checks the finalized port
  set and skips only the conflicting compat listener, so HTTP and gRPC
  compat degrade independently and the service keeps its port.
- Also skip the legacy HTTP listener when a mini service owns the port:
  a wildcard-bound service can still leave loopback bindable, which would
  let the compat listener shadow it for loopback clients.
- Close legacyServer's raw listener in the deferred cleanup: Serve may
  not have registered it yet when startAdminServer returns early.
- Helm: add admin.legacyPort (default 23646) and expose the legacy
  http/grpc ports through the admin Service, container ports and
  NetworkPolicy so older workers resolved via the Service keep working.

* helm: only pass -port.legacy when it differs from the binary default

ct install deploys the released image with the new chart, and 4.48 does
not know -port.legacy, so the admin pod crash-loops. Omitting the flag at
the default value is behavior-identical on new binaries (their default
is also 23646) and keeps older releases installable; legacyPort: 0 or a
custom port still renders the flag.

2026-10-18 21:21:05 -07:00

7.0 KiB

Raw Blame History

Admin Worker Plugin System (Design)

This document describes the plugin system for admin-managed workers, implemented in parallel with the current maintenance/worker mechanism.

Scope

Add a new plugin protocol and runtime model for multi-language workers.
Keep all current admin + worker code paths untouched.
Use gRPC for all admin-worker communication.
Let workers describe job configuration UI declaratively via protobuf.
Persist all job type configuration under admin server data directory.
Support detector workers and executor workers per job type.
Add end-to-end workflow observability (activities, active jobs, progress).

New Contract

Proto file: weed/pb/plugin.proto
gRPC service: PluginControlService.WorkerStream
Connection model: worker-initiated long-lived bidirectional stream.

Why this model:

Works for workers in any language with gRPC support.
Avoids admin dialing constraints in NAT/private networks.
Allows command/response, progress streaming, and heartbeat over one channel.

Core Runtime Components (Admin Side)

PluginRegistry

Tracks connected workers and their per-job-type capabilities.
Maintains liveness via heartbeat timeout.

SchemaCoordinator

For each job type, asks one capable worker for JobTypeDescriptor.
Caches descriptor version and refresh timestamp.

ConfigStore

Persists descriptor + saved config values in dataDir.
Stores both:
- Admin-owned runtime config (detection interval, dispatch concurrency, retry).
- Worker-owned config values (plugin-specific detection/execution knobs).

DetectorScheduler

Per job type, chooses one detector worker (can_detect=true).
Sends RunDetectionRequest with saved configs + cluster context.
Accepts DetectionProposals, dedupes by dedupe_key, inserts jobs.

JobDispatcher

Chooses executor worker (can_execute=true) for each pending job.
Sends ExecuteJobRequest.
Consumes JobProgressUpdate and JobCompleted.

WorkflowMonitor

Builds live counters and timeline from events:
- activities per job type,
- active jobs,
- per-job progress/state,
- worker health/load.

Worker Responsibilities

Register capabilities on connect (WorkerHello).
Expose job type descriptor (ConfigSchemaResponse) including UI schemas:

admin config form,
worker config form,
defaults.

Run detection on demand (RunDetectionRequest) and return proposals.
Execute assigned jobs (ExecuteJobRequest) and stream progress.
Heartbeat regularly with slot usage and running work.
Handle cancellation requests (CancelRequest) for in-flight detection/execution.

Declarative UI Model

UI is fully derived from protobuf schema:

ConfigForm
ConfigSection
ConfigField
ConfigOption
ValidationRule
ConfigValue (typed scalar/list/map/object value container)

Result:

Admin can render forms without hardcoded task structs.
New job types can ship UI schema from worker binary alone.
Worker language is irrelevant as long as it can emit protobuf messages.

Detection and Dispatch Flow

Worker connects and registers capabilities.
Admin requests descriptor per job type.
Admin persists descriptor and editable config values.
On detection interval (admin-owned setting):

Admin chooses one detector worker for that job type.
Sends RunDetectionRequest with:
- AdminRuntimeConfig,
- admin_config_values,
- worker_config_values,
- ClusterContext (master/filer/volume grpc locations, metadata).

Detector emits DetectionProposals and DetectionComplete.
Admin dedupes and enqueues jobs.
Dispatcher assigns jobs to any eligible executor worker.
Executor emits JobProgressUpdate and JobCompleted.
Monitor updates workflow UI in near-real-time.

Persistence Layout (Admin Data Dir)

Current layout under <admin-data-dir>/plugin/:

job_types/<job_type>/descriptor.pb
job_types/<job_type>/descriptor.json
job_types/<job_type>/config.pb
job_types/<job_type>/config.json
job_types/<job_type>/runs.json
jobs/tracked_jobs.json
activities/activities.json

config.pb should use PersistedJobTypeConfig from plugin.proto.

Admin UI

Route: /plugin
Includes:
- runtime status,
- workers/capabilities,
- declarative descriptor-driven config forms,
- run history (last 10 success + last 10 errors),
- tracked jobs and activity stream,
- manual actions for schema refresh, detection, and detect+execute workflow.

Scheduling Policy (Initial)

Detector selection per job type:

only workers with can_detect=true.
prefer healthy worker with highest free detection slots.
lease ends when heartbeat timeout or stream drop.

Execution dispatch:

only workers with can_execute=true.
select by available execution slots and least active jobs.
retry on failure using admin runtime retry config.

Safety and Reliability

Idempotency: dedupe proposals by (job_type, dedupe_key).
Backpressure: enforce max jobs per detection run.
Timeouts: detection and execution timeout from admin runtime config.
Replay-safe persistence: write job state changes before emitting UI events.
Heartbeat-based failover for detector/executor reassignment.

Backward Compatibility

Legacy worker.proto runtime remains internally available where still referenced.
External CLI worker path is moved to plugin runtime behavior.
Runtime is enabled by default on admin worker gRPC server.

Incremental Rollout Plan

Phase 1

Introduce protocol and storage models only.

Phase 2

Build admin registry/scheduler/dispatcher behind feature flag.

Phase 3

Add dedicated plugin UI pages and metrics.

Phase 4

Port one existing job type (e.g. vacuum) as external worker plugin.

Phase 4 status (starter)

Added weed worker command as an external plugin.proto worker process.
Initial handler implements vacuum job type with:
- declarative descriptor/config form response (ConfigSchemaResponse),
- detection via master topology scan (RunDetectionRequest),
- execution via existing vacuum task logic (ExecuteJobRequest),
- heartbeat/load reporting for monitor UI.
Legacy maintenance-worker-specific CLI path is removed.

Run example:

Start admin: weed admin -master=localhost:9333
Start worker: weed worker -admin=localhost:12535
Optional explicit job type: weed worker -admin=localhost:12535 -jobType=vacuum
Optional stable worker ID persistence: weed worker -admin=localhost:12535 -workingDir=/var/lib/seaweedfs-plugin

Phase 5

Migrate remaining job types and deprecate old mechanism.

Agreed Defaults

Detector multiplicity

Exactly one detector worker per job type at a time. Admin selects one worker and runs detection there.

Secret handling

No encryption at rest required for plugin config in this phase.

Schema compatibility

No migration policy required yet; this is a new system.

Execution ownership

Same worker is allowed to do both detection and execution.

Retention

Keep last 10 successful runs and last 10 error runs per job type.

7.0 KiB Raw Blame History