Chris Lu
9e34426a56
* lance: a maintenance job that sorts a table by its declared fields Lance appends fragments in write order and has no notion of a sorted table, so nothing but a rewrite establishes one, and nothing but another rewrite restores it once rows have been appended. lance_sort reads the order from the dataset's own configuration, falls back to the worker's, and rewrites the table in it. The spec and the marker live in crates/sort rather than in the job, because weed/worker/tasks/iceberg sorts too: two jobs that disagreed about what "id desc nulls-first" means would be two features wearing one name. The sort spills. lance builds its DataFusion runtime with a FairSpillPool and a disk manager, but only when LanceExecutionOptions::use_spilling is set, and that struct derives Default over a plain bool — so Scanner::try_into_stream, which fills its options with ..Default::default(), is precisely the path that does not spill. The job builds the plan with create_plan and executes it with spilling on and the operator's memory budget. The marker rides in the same commit as the data: Operation::Overwrite is the one operation carrying config values alongside fragments, so a sorted table and the record of its sorting cannot disagree. It records the version the sort read, not the one it wrote, which is not knowable while the marker is being assembled. Detection treats anything committed after the sort's own commit as data the sort did not produce — row counts alone cannot see a rewrite that leaves the count where it was, and such a table would look sorted forever. Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y * lance: identify a sorted table by the files it wrote, not its version Review found three ways the version-based marker misjudges a table, and they share a cause: the version a sort produces is not knowable while the marker is being assembled, so the marker recorded the version it read and detection inferred the rest. A commit that rebases past a conflict lands on a different number, and the inference then reads a rewrite into an ordinary table — a full re-sort, and its indices, for nothing. Data file names do not have that problem. They are chosen before the commit, so the commit can carry them, and they do not change with the version it lands on. The marker now records how many files the sort wrote and a digest of their names, and detection asks whether the table still holds them: the same files means untouched, the same files followed by more means appended, anything else means the data was replaced. That also closes the hole the row threshold left. A replacement that grew the table by fewer rows than min_unsorted_rows read as sorted, however many rows had actually moved; the threshold now applies only where the sorted files are still in place, which is what it was for. A marker without a row count is stale rather than a zero to compare against, and deletes stop forcing a re-sort — they write a deletion file beside the data rather than rewriting it, and removing rows does not unsort the ones that remain. Sort fields are also compared exactly rather than case-folded. Arrow schemas are case-sensitive, so `id` and `ID` are two columns, and folding them together rejected a valid order. Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y * lance: count the rows appended after a sort, not the table's net growth Review found that rows deleted from the sorted fragments hide appended rows one for one: the threshold compared the live row count against the count recorded at sort time, so 800 deletions and 300 appends read as a table that shrank, and a table where deletions keep pace with appends stays "sorted" with an unsorted tail forever. The fragments say it directly. The marker already records how many fragments the sort wrote, so the ones after that prefix are exactly what arrived since, and the manifest carries each fragment's live row count — physical rows less its deletions. Counting those is the arithmetic the threshold was always meant to do, and it needs no row count from the marker at all. A fragment whose length the manifest does not record cannot be counted, and a table that cannot be judged is one to sort rather than one to leave alone forever, so an uncountable appended fragment reads as stale. Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y
SeaweedFS Rust workers
weed/pb/plugin.proto is a language-agnostic contract: a maintenance worker
connects out to admin, announces the job types it can detect and execute, and
answers requests on that one stream. weed worker -admin=host:23646 is the Go
implementation of it from outside the admin process. This workspace is the Rust
one.
crates/core the contract: stream, handshake, heartbeat, registry, config forms
crates/lance maintenance jobs for Lance tables, and a binary
core knows nothing about any job. A second worker is a new crate beside
lance that depends on it, not a fork of the protocol.
Building
core compiles plugin.proto with the protoc that protoc-bin-vendored ships,
the way seaweed-volume does, so it needs no system install.
The lance crates compile protos of their own, in their own build-script
processes, which nothing our build script sets can reach. They need a protoc of
their own: either one on PATH — brew install protobuf, apt install protobuf-compiler — or PROTOC naming one. CI points it at the vendored
binary for the runner's platform, resolved from the version in Cargo.lock.
Running
cargo run -p weed-lance-worker -- --admin 127.0.0.1:23646
The admin's HTTP address is what an operator has; the gRPC port is derived from it the way the Go side does. Dialling the HTTP port fails as "frame with invalid size", which reads like a protocol bug rather than a wrong port.
The binary is weed-worker, not weed-lance-worker: it is the Rust side of
weed worker, and lance is the first family of jobs it carries rather than the
only one it ever will.
Released builds do not need a toolchain. The worker ships inside the SeaweedFS
image, beside the Rust volume server, under the verb that mirrors
volume-rust:
docker run chrislusf/seaweedfs worker-rust --admin admin:23646
and as weed-worker_linux_{amd64,arm64}.tar.gz on each GitHub release. Both are
linux amd64/arm64 only — lance, arrow and datafusion make every extra target an
expensive build, and the worker runs beside the cluster it maintains. On an
architecture without a build the image carries an empty placeholder and the
entrypoint says so rather than failing as "not found".
Metrics
cargo run -p weed-lance-worker -- --admin 127.0.0.1:23646 --metrics-port 9328
Serves /health, /ready and /metrics on that port, the same three the Go
worker serves under weed worker -metricsPort, so one scrape config covers
workers in either language. Off by default, and bound to loopback unless
--metrics-ip says otherwise, because the endpoint is unauthenticated. 9328
continues the series the other components use (master 9324, volume 9325, filer
9326, s3 9327); an IPv6 address works with or without brackets.
Grafana: the "Plugin Workers" row of other/metrics/grafana_seaweedfs.json
graphs these. Its panels filter on $cluster, which comes from the scrape job's
labels, so scrape the worker the way the rest of the cluster is scraped or the
row stays empty.
Names are SeaweedFS_worker_*, matching the Go side's convention. The pair
worth alerting on is objects_seen_total and objects_skipped_total: a sweep
that proposes nothing and a sweep that could read nothing look identical from
proposals_total alone.
SeaweedFS_worker_connected 1
SeaweedFS_worker_objects_seen_total{job_type="lance_compact"} 7
SeaweedFS_worker_proposals_total{job_type="lance_compact"} 2
SeaweedFS_worker_jobs_total{job_type="lance_compact",result="ok"} 2
SeaweedFS_worker_lance_fragments_removed_total 25
/ready follows the control stream: a worker whose admin has gone away is
running but is not going to do anything.
Credentials
The worker holds none. It asks the namespace to describe a table with
vend_credentials and hands the storage_options that come back to lance. A
gateway without STS configured vends no credentials at all, so --access-key
and --secret-key supply a fallback; anything the namespace does vend wins over
them.
State
All three jobs are implemented and tested end to end against a live gateway:
compaction result: 12 fragments became 1
reindex result: 512 uncovered rows became 0
cleanup result: removed 14 versions and 24272 bytes
cargo test -p weed-lance-worker runs them when WEED_LANCE_NAMESPACE names a
live namespace and skips otherwise, the way the Go integration tests skip
without Docker. Each test seeds the table it needs, including building a vector
index and then appending rows outside it, so a run does not depend on what the
previous one left behind — the first version of these did, and quietly stopped
testing anything once it had done its job.
The handshake, descriptor exchange and heartbeat work against a live admin, which logs the worker connecting and prefetches all three descriptors.