Files
seaweedfs/seaweed-worker
aceb089bc8 lance: per-table cleanup retention and on-demand purge of deleted rows (#12739)
* lance: per-table cleanup retention and on-demand purge of deleted rows

A vector deleted from an index stayed in storage until compaction
rewrote its fragment (only above 10% deletions) and the cleanup job's
168 h retention, set once per worker, aged out every version that
referenced it.

Let a table override retain_hours and min_versions in its Lance config
(seaweedfs.cleanup.retain_hours, seaweedfs.cleanup.min_versions), and
let it request a purge (seaweedfs.purge.requested). The cleanup job
proposes a table with a pending purge regardless of the version floor
and then rewrites every fragment with deletions, retrains the indices
(a remapped IVF_FLAT index still holds the deleted vector), keeps only
the latest version, and records the request as served. A version pinned
by a tag fails the purge and leaves the request pending.

The test scans every file of a local table for a deleted vector's key,
metadata and values, and finds none after the purge.

* lance: refuse to purge a sharded vector index

A retrain builds every index name over the whole table, so the purge
turned shards of one vector index (one name per fragment set) into
names that each covered every fragment, indexing every row once per
shard. Check before compacting and fail with the request left pending,
and retrain only the vector indices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: rebuild scalar indices in a purge

Compaction only remaps scalar indices, and a retrain leaves them alone,
so a purge reported success while a bitmap index still listed the
deleted row's value, and under stable row ids a BTree index still held
the deleted key. Build each scalar index again from the remaining rows
with the parameters it was built with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: purge through the cleanup job's retention policy

The purge built its own cleanup policy, so a deployment that has the
cleanup job keep more than the floor and the window (versions an
unleased reader may still be on) would have lost them to a purge. Clean
up through retention_policy, the one the job itself uses, with the
latest version kept.

Mark the request served only once the oldest version left is one the
rebuild wrote, and record the rebuild so a purge that has to wait for
its cleanup (a tag, or that policy) retries only the cleanup instead of
compacting and retraining the indices on every pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: lease the version every list page reads

A list page read its version without a lease unless it was a segment
that read data and metadata twice. A purge removes every version but
the latest, and only a lease tag stops it, so a listing that opened the
version before the purge's compaction could lose its files mid-read.
Pin it as get_vectors and query_vectors do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: give the cleanup job a build's execution time

A purge compacts the table and retrains its indices inside a cleanup
job, but the job declared no runtime defaults, so the admin held it to
its 90 s default and could expire a purge mid-rebuild. Declare the same
budget the indices job uses for a retrain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:24:28 -08:00
..

SeaweedFS Rust workers

weed/pb/plugin.proto is a language-agnostic contract: a maintenance worker connects out to admin, announces the job types it can detect and execute, and answers requests on that one stream. weed worker -admin=host:12535 is the Go implementation of it from outside the admin process. This workspace is the Rust one.

crates/core     the contract: stream, handshake, heartbeat, registry, config forms
crates/lance    maintenance jobs for Lance tables, and a binary

It also depends on ../seaweed-common, a small crate outside this workspace holding the few helpers the Rust volume server needs identically — the HTTP<->gRPC address rule and the rustls provider choice.

core knows nothing about any job. A second worker is a new crate beside lance that depends on it, not a fork of the protocol.

Building

Requires Rust 1.94.1+ (2024 edition), matching rust-version in Cargo.toml. The patch release matters: 1.94.0 does not build. The edition itself only needs 1.85; the higher floor comes from the dependency tree — lance's aws feature pulls in the AWS SDK — so it moves with those crates. CI builds on the latest stable.

core compiles plugin.proto with the protoc that protoc-bin-vendored ships, the way seaweed-volume does, so it needs no system install.

The lance crates compile protos of their own, in their own build-script processes, which nothing our build script sets can reach. They need a protoc of their own: either one on PATH — brew install protobuf, apt install protobuf-compiler — or PROTOC naming one. CI points it at the vendored binary for the runner's platform, resolved from the version in Cargo.lock.

Running

cargo run -p weed-lance-worker -- --admin 127.0.0.1:12535 --masters 127.0.0.1:9333

The admin's HTTP address is what an operator has; the gRPC port is derived from it the way the Go side does. Dialling the HTTP port fails as "frame with invalid size", which reads like a protocol bug rather than a wrong port.

The binary is weed-worker, not weed-lance-worker: it is the Rust side of weed worker, and lance is the first family of jobs it carries rather than the only one it ever will.

Released builds do not need a toolchain. The worker ships inside the SeaweedFS image, beside the Rust volume server, under the verb that mirrors volume-rust:

docker run chrislusf/seaweedfs worker-rust --admin admin:12535 --masters master:9333

and as weed-worker_linux_{amd64,arm64}.tar.gz on each GitHub release. Both are linux amd64/arm64 only — lance, arrow and datafusion make every extra target an expensive build, and the worker runs beside the cluster it maintains. On an architecture without a build the image carries an empty placeholder and the entrypoint says so rather than failing as "not found".

Metrics

cargo run -p weed-lance-worker -- --admin 127.0.0.1:12535 --masters 127.0.0.1:9333 --metrics-port 9328

Serves /health, /ready and /metrics on that port, the same three the Go worker serves under weed worker -metricsPort, so one scrape config covers workers in either language. Off by default, and bound to loopback unless --metrics-ip says otherwise, because the endpoint is unauthenticated. 9328 continues the series the other components use (master 9324, volume 9325, filer 9326, s3 9327); an IPv6 address works with or without brackets.

Grafana: the "Plugin Workers" row of other/metrics/grafana_seaweedfs.json graphs these. Its panels filter on $cluster, which comes from the scrape job's labels, so scrape the worker the way the rest of the cluster is scraped or the row stays empty.

Names are SeaweedFS_worker_*, matching the Go side's convention. The pair worth alerting on is objects_seen_total and objects_skipped_total: a sweep that proposes nothing and a sweep that could read nothing look identical from proposals_total alone.

SeaweedFS_worker_connected 1
SeaweedFS_worker_objects_seen_total{job_type="lance_compact"} 7
SeaweedFS_worker_proposals_total{job_type="lance_compact"} 2
SeaweedFS_worker_jobs_total{job_type="lance_compact",result="ok"} 2
SeaweedFS_worker_lance_fragments_removed_total 25

/ready follows the control stream: a worker whose admin has gone away is running but is not going to do anything.

Credentials

The worker holds no S3 credentials. The namespace tells it where a table is; its files are read and written on the cluster's filers and volume servers, which --masters names the masters of (--filer-group picks the filers). What that takes comes from the same security.toml the weed processes use: grpc.client for the filers, https.client for volume servers under HTTPS, jwt.signing.read.key where volume reads are signed, and s3.sse.kek or s3.sse.key where a bucket encrypts with SSE-S3. Chunks are encrypted for the volume servers when the filer is (weed filer -encryptVolumeData), as the S3 gateway's are.

The namespace authenticates the worker by the cluster token it derives from jwt.filer_signing.key; the same key signs the admin stream when admin requires it. The worker reads it from where weed worker does: WEED_JWT_FILER_SIGNING_KEY, or [jwt.filer_signing] in the security.toml found in the working directory, ~/.seaweedfs, /usr/local/etc/seaweedfs or /etc/seaweedfs. --filer-signing-key overrides both.

State

All three jobs are implemented and tested end to end against a live gateway:

compaction result: 12 fragments became 1
reindex result:    512 uncovered rows became 0
cleanup result:    removed 14 versions and 24272 bytes

cargo test -p weed-lance-worker runs them when WEED_LANCE_NAMESPACE names a live namespace and skips otherwise, the way the Go integration tests skip without Docker. A live namespace needs WEED_MASTERS naming its cluster's masters, and WEED_JWT_FILER_SIGNING_KEY (or security.toml) where the namespace authenticates cluster processes; the tests fail without the first and are refused without the second. Each test seeds the table it needs, including building a vector index and then appending rows outside it, so a run does not depend on what the previous one left behind — the first version of these did, and quietly stopped testing anything once it had done its job.

The handshake, descriptor exchange and heartbeat work against a live admin, which logs the worker connecting and prefetches all three descriptors.