seaweedfs/seaweed-worker/README.md

# SeaweedFS Rust workers

`weed/pb/plugin.proto` is a language-agnostic contract: a maintenance worker
connects out to admin, announces the job types it can detect and execute, and
answers requests on that one stream. `weed worker -admin=host:12535` is the Go
implementation of it from outside the admin process. This workspace is the Rust
one.

    crates/core     the contract: stream, handshake, heartbeat, registry, config forms
    crates/lance    maintenance jobs for Lance tables, and a binary

It also depends on `../seaweed-common`, a small crate outside this workspace
holding the few helpers the Rust volume server needs identically — the
HTTP<->gRPC address rule and the rustls provider choice.

`core` knows nothing about any job. A second worker is a new crate beside
`lance` that depends on it, not a fork of the protocol.

## Building

Requires Rust 1.94.1+ (2024 edition), matching `rust-version` in `Cargo.toml`.
The patch release matters: 1.94.0 does not build. The edition itself only needs
1.85; the higher floor comes from the dependency tree — lance's `aws` feature
pulls in the AWS SDK — so it moves with those crates. CI builds on the latest
stable.

`core` compiles `plugin.proto` with the protoc that protoc-bin-vendored ships,
the way seaweed-volume does, so it needs no system install.

The lance crates compile protos of their own, in their own build-script
processes, which nothing our build script sets can reach. They need a protoc of
their own: either one on PATH — `brew install protobuf`, `apt install
protobuf-compiler` — or `PROTOC` naming one. CI points it at the vendored
binary for the runner's platform, resolved from the version in `Cargo.lock`.

## Running

    cargo run -p weed-lance-worker -- --admin 127.0.0.1:12535 --masters 127.0.0.1:9333

The admin's *HTTP* address is what an operator has; the gRPC port is derived
from it the way the Go side does. Dialling the HTTP port fails as "frame with
invalid size", which reads like a protocol bug rather than a wrong port.

The binary is `weed-worker`, not `weed-lance-worker`: it is the Rust side of
`weed worker`, and lance is the first family of jobs it carries rather than the
only one it ever will.

Released builds do not need a toolchain. The worker ships inside the SeaweedFS
image, beside the Rust volume server, under the verb that mirrors
`volume-rust`:

    docker run chrislusf/seaweedfs worker-rust --admin admin:12535 --masters master:9333

and as `weed-worker_linux_{amd64,arm64}.tar.gz` on each GitHub release. Both are
linux amd64/arm64 only — lance, arrow and datafusion make every extra target an
expensive build, and the worker runs beside the cluster it maintains. On an
architecture without a build the image carries an empty placeholder and the
entrypoint says so rather than failing as "not found".

## Metrics

    cargo run -p weed-lance-worker -- --admin 127.0.0.1:12535 --masters 127.0.0.1:9333 --metrics-port 9328

Serves `/health`, `/ready` and `/metrics` on that port, the same three the Go
worker serves under `weed worker -metricsPort`, so one scrape config covers
workers in either language. Off by default, and bound to loopback unless
`--metrics-ip` says otherwise, because the endpoint is unauthenticated. 9328
continues the series the other components use (master 9324, volume 9325, filer
9326, s3 9327); an IPv6 address works with or without brackets.

Grafana: the "Plugin Workers" row of `other/metrics/grafana_seaweedfs.json`
graphs these. Its panels filter on `$cluster`, which comes from the scrape job's
labels, so scrape the worker the way the rest of the cluster is scraped or the
row stays empty.

Names are `SeaweedFS_worker_*`, matching the Go side's convention. The pair
worth alerting on is `objects_seen_total` and `objects_skipped_total`: a sweep
that proposes nothing and a sweep that could read nothing look identical from
`proposals_total` alone.

    SeaweedFS_worker_connected 1
    SeaweedFS_worker_objects_seen_total{job_type="lance_compact"} 7
    SeaweedFS_worker_proposals_total{job_type="lance_compact"} 2
    SeaweedFS_worker_jobs_total{job_type="lance_compact",result="ok"} 2
    SeaweedFS_worker_lance_fragments_removed_total 25

`/ready` follows the control stream: a worker whose admin has gone away is
running but is not going to do anything.

## Credentials

The worker holds no S3 credentials. The namespace tells it where a table is;
its files are read and written on the cluster's filers and volume servers,
which `--masters` names the masters of (`--filer-group` picks the filers). What
that takes comes from the same security.toml the weed processes use:
`grpc.client` for the filers, `https.client` for volume servers under HTTPS,
`jwt.signing.read.key` where volume reads are signed, and `s3.sse.kek` or
`s3.sse.key` where a bucket encrypts with SSE-S3. `--encryptVolumeData` encrypts
every chunk the worker writes for the volume servers, as the gateway's flag does.

The namespace authenticates the worker by the cluster token it derives from
`jwt.filer_signing.key`; the same key signs the admin stream when admin requires
it. The worker reads it from where `weed worker` does: `WEED_JWT_FILER_SIGNING_KEY`,
or `[jwt.filer_signing]` in the security.toml found in the working directory,
`~/.seaweedfs`, `/usr/local/etc/seaweedfs` or `/etc/seaweedfs`.
`--filer-signing-key` overrides both.

## State

All three jobs are implemented and tested end to end against a live gateway:

    compaction result: 12 fragments became 1
    reindex result:    512 uncovered rows became 0
    cleanup result:    removed 14 versions and 24272 bytes

`cargo test -p weed-lance-worker` runs them when `WEED_LANCE_NAMESPACE` names a
live namespace and skips otherwise, the way the Go integration tests skip
without Docker. A live namespace needs `WEED_MASTERS` naming its cluster's
masters, and `WEED_JWT_FILER_SIGNING_KEY` (or security.toml) where the
namespace authenticates cluster processes; the tests fail without the first
and are refused without the second. Each test seeds the table it needs, including building a vector
index and then appending rows outside it, so a run does not depend on what the
previous one left behind — the first version of these did, and quietly stopped
testing anything once it had done its job.

The handshake, descriptor exchange and heartbeat work against a live admin,
which logs the worker connecting and prefetches all three descriptors.