mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-20 13:04:25 +02:00
Chris Lu
8ac5133abe
* admin: default to port 12535, keep 23646 as a best-effort compat listener The default admin HTTP port 23646 derives a worker gRPC port of 33646 (port + GrpcPortOffset), inside the Linux default ephemeral range (32768+). One of our own startup dials can claim 33646 before the admin binds it, killing startup. This is the same failure #10209 fixed for the test port allocator, still reachable via the mini default (#10838) and via standalone `weed admin` defaults. Move the default admin port to 12535 (gRPC 22535) everywhere, keeping the http+10000 convention so worker -admin derivation, the Rust worker and host:http.grpc addressing keep working: - weed admin -port, weed mini -admin.port - weed worker -admin, the plugin worker fallback, Rust worker and weed-lance-worker defaults - help text, README, helm values, docker compose, tests So old workers, bookmarks and scripts keep working, admin also serves the same handler and the worker gRPC server on the previous default (-port.legacy / -admin.port.legacy, default 23646, 0 disables). The legacy gRPC port binds best-effort: it lives in the ephemeral range, so a failed bind only warns instead of reintroducing the startup failure. weed mini reserves it early like the primary port and keeps it out of its own port relocation. grpc.Server tracks every listener handed to Serve, so the compat listener stops with the primary. Also warn when an explicitly configured admin port puts the derived gRPC port at or above the ephemeral floor, the other fix suggested in the issue. Supersedes the pinned-grpc approach in #11814: pinning -admin.port.grpc off the +10000 convention breaks external workers that derive the admin gRPC port from the HTTP address. ClickHouse/ClickHouse#115512 pins -admin.port on their side; this makes that pin unnecessary. Fixes #10838 * admin: let services win the legacy ports, close compat listener on failure Review findings on the compat listeners: - Do not put the legacy ports in mini's reservedPorts: an explicit -s3.port=23646 or -master.port.grpc=33646 must be honored, not rejected or stolen by the early reservation. Mini now checks the finalized port set and skips only the conflicting compat listener, so HTTP and gRPC compat degrade independently and the service keeps its port. - Also skip the legacy HTTP listener when a mini service owns the port: a wildcard-bound service can still leave loopback bindable, which would let the compat listener shadow it for loopback clients. - Close legacyServer's raw listener in the deferred cleanup: Serve may not have registered it yet when startAdminServer returns early. - Helm: add admin.legacyPort (default 23646) and expose the legacy http/grpc ports through the admin Service, container ports and NetworkPolicy so older workers resolved via the Service keep working. * helm: only pass -port.legacy when it differs from the binary default ct install deploys the released image with the new chart, and 4.48 does not know -port.legacy, so the admin pod crash-loops. Omitting the flag at the default value is behavior-identical on new binaries (their default is also 23646) and keeps older releases installable; legacyPort: 0 or a custom port still renders the flag.
115 lines
5.3 KiB
Markdown
115 lines
5.3 KiB
Markdown
# SeaweedFS Rust workers
|
|
|
|
`weed/pb/plugin.proto` is a language-agnostic contract: a maintenance worker
|
|
connects out to admin, announces the job types it can detect and execute, and
|
|
answers requests on that one stream. `weed worker -admin=host:12535` is the Go
|
|
implementation of it from outside the admin process. This workspace is the Rust
|
|
one.
|
|
|
|
crates/core the contract: stream, handshake, heartbeat, registry, config forms
|
|
crates/lance maintenance jobs for Lance tables, and a binary
|
|
|
|
It also depends on `../seaweed-common`, a small crate outside this workspace
|
|
holding the few helpers the Rust volume server needs identically — the
|
|
HTTP<->gRPC address rule and the rustls provider choice.
|
|
|
|
`core` knows nothing about any job. A second worker is a new crate beside
|
|
`lance` that depends on it, not a fork of the protocol.
|
|
|
|
## Building
|
|
|
|
Requires Rust 1.94.1+ (2024 edition), matching `rust-version` in `Cargo.toml`.
|
|
The patch release matters: 1.94.0 does not build. The edition itself only needs
|
|
1.85; the higher floor comes from the dependency tree — lance's `aws` feature
|
|
pulls in the AWS SDK — so it moves with those crates. CI builds on the latest
|
|
stable.
|
|
|
|
`core` compiles `plugin.proto` with the protoc that protoc-bin-vendored ships,
|
|
the way seaweed-volume does, so it needs no system install.
|
|
|
|
The lance crates compile protos of their own, in their own build-script
|
|
processes, which nothing our build script sets can reach. They need a protoc of
|
|
their own: either one on PATH — `brew install protobuf`, `apt install
|
|
protobuf-compiler` — or `PROTOC` naming one. CI points it at the vendored
|
|
binary for the runner's platform, resolved from the version in `Cargo.lock`.
|
|
|
|
## Running
|
|
|
|
cargo run -p weed-lance-worker -- --admin 127.0.0.1:12535
|
|
|
|
The admin's *HTTP* address is what an operator has; the gRPC port is derived
|
|
from it the way the Go side does. Dialling the HTTP port fails as "frame with
|
|
invalid size", which reads like a protocol bug rather than a wrong port.
|
|
|
|
The binary is `weed-worker`, not `weed-lance-worker`: it is the Rust side of
|
|
`weed worker`, and lance is the first family of jobs it carries rather than the
|
|
only one it ever will.
|
|
|
|
Released builds do not need a toolchain. The worker ships inside the SeaweedFS
|
|
image, beside the Rust volume server, under the verb that mirrors
|
|
`volume-rust`:
|
|
|
|
docker run chrislusf/seaweedfs worker-rust --admin admin:12535
|
|
|
|
and as `weed-worker_linux_{amd64,arm64}.tar.gz` on each GitHub release. Both are
|
|
linux amd64/arm64 only — lance, arrow and datafusion make every extra target an
|
|
expensive build, and the worker runs beside the cluster it maintains. On an
|
|
architecture without a build the image carries an empty placeholder and the
|
|
entrypoint says so rather than failing as "not found".
|
|
|
|
## Metrics
|
|
|
|
cargo run -p weed-lance-worker -- --admin 127.0.0.1:12535 --metrics-port 9328
|
|
|
|
Serves `/health`, `/ready` and `/metrics` on that port, the same three the Go
|
|
worker serves under `weed worker -metricsPort`, so one scrape config covers
|
|
workers in either language. Off by default, and bound to loopback unless
|
|
`--metrics-ip` says otherwise, because the endpoint is unauthenticated. 9328
|
|
continues the series the other components use (master 9324, volume 9325, filer
|
|
9326, s3 9327); an IPv6 address works with or without brackets.
|
|
|
|
Grafana: the "Plugin Workers" row of `other/metrics/grafana_seaweedfs.json`
|
|
graphs these. Its panels filter on `$cluster`, which comes from the scrape job's
|
|
labels, so scrape the worker the way the rest of the cluster is scraped or the
|
|
row stays empty.
|
|
|
|
Names are `SeaweedFS_worker_*`, matching the Go side's convention. The pair
|
|
worth alerting on is `objects_seen_total` and `objects_skipped_total`: a sweep
|
|
that proposes nothing and a sweep that could read nothing look identical from
|
|
`proposals_total` alone.
|
|
|
|
SeaweedFS_worker_connected 1
|
|
SeaweedFS_worker_objects_seen_total{job_type="lance_compact"} 7
|
|
SeaweedFS_worker_proposals_total{job_type="lance_compact"} 2
|
|
SeaweedFS_worker_jobs_total{job_type="lance_compact",result="ok"} 2
|
|
SeaweedFS_worker_lance_fragments_removed_total 25
|
|
|
|
`/ready` follows the control stream: a worker whose admin has gone away is
|
|
running but is not going to do anything.
|
|
|
|
## Credentials
|
|
|
|
The worker holds none. It asks the namespace to describe a table with
|
|
`vend_credentials` and hands the `storage_options` that come back to lance. A
|
|
gateway without STS configured vends no credentials at all, so `--access-key`
|
|
and `--secret-key` supply a fallback; anything the namespace does vend wins over
|
|
them.
|
|
|
|
## State
|
|
|
|
All three jobs are implemented and tested end to end against a live gateway:
|
|
|
|
compaction result: 12 fragments became 1
|
|
reindex result: 512 uncovered rows became 0
|
|
cleanup result: removed 14 versions and 24272 bytes
|
|
|
|
`cargo test -p weed-lance-worker` runs them when `WEED_LANCE_NAMESPACE` names a
|
|
live namespace and skips otherwise, the way the Go integration tests skip
|
|
without Docker. Each test seeds the table it needs, including building a vector
|
|
index and then appending rows outside it, so a run does not depend on what the
|
|
previous one left behind — the first version of these did, and quietly stopped
|
|
testing anything once it had done its job.
|
|
|
|
The handshake, descriptor exchange and heartbeat work against a live admin,
|
|
which logs the worker connecting and prefetches all three descriptors.
|