Say where the Lance worker comes from now that it ships

It is in the SeaweedFS image as worker-rust and on each release as
weed-worker_linux_{amd64,arm64}.tar.gz, so a Rust toolchain is no longer the
only way in. The binary is weed-worker, not weed-lance-worker: the crate keeps
the lance name, the binary does not.

Also records what the build needs from protoc - core vendors its own, lance's
crates need one of their own.
Chris Lu
2026-08-22 11:19:42 -07:00
parent b9475cdf7f
commit 9eeaa8e3cf
4 changed files with 47 additions and 6 deletions
+39 -4
@@ -4,12 +4,40 @@ A Lance table needs upkeep the Iceberg worker cannot do, because reading the Lan
It is **not a sidecar**: `PluginControlService` is a language-agnostic gRPC contract for external maintenance workers, and `weed worker` is the Go reference implementation of it. Everything the Go worker gets from the protocol — scheduling, retries, dedupe, progress, concurrency limits, and a settings page rendered in the admin UI from the worker's own descriptor — the Rust worker gets too.
Source: `seaweed-worker/` in the SeaweedFS repository.
Source: `seaweed-worker/` in the SeaweedFS repository. The binary is
`weed-worker`: it is the Rust side of `weed worker`, the way `weed-volume` is
the Rust side of `weed volume`, and the Lance jobs are the first family it
carries rather than the only one it ever will.
## Getting it
### Docker
```bash
docker run chrislusf/seaweedfs worker-rust \
--admin admin:23646 --namespace http://s3:9101
```
The standard image carries it at `/usr/bin/weed-worker` on `amd64` and `arm64`,
beside the Rust volume server. The verb mirrors `volume-rust`, so plain `worker`
still runs the Go one. On `arm` and `386` the image says the worker is not
available for the platform rather than failing as "not found".
### From a release
`weed-worker_linux_amd64.tar.gz` and `weed-worker_linux_arm64.tar.gz`, with
md5s, on each GitHub release. Linux only: the worker runs beside the cluster it
maintains, and lance, arrow and datafusion make every extra target an expensive
build.
### From source
A Rust toolchain and a protoc; see [Building it](#building-it) below.
## Running it
```bash
weed-lance-worker \
weed-worker \
--admin localhost:23646 \
--namespace http://localhost:9101
```
@@ -90,10 +118,17 @@ Note the port convention: master 9324, volume 9325, filer 9326, s3 9327 — so a
```bash
cd seaweed-worker
cargo build --release -p weed-lance-worker
cargo build --release -p weed-lance-worker # binary at target/release/weed-worker
```
Needs a Rust toolchain and `protoc`; the crates compile the protocol straight out of `weed/pb/plugin.proto`. A cold build pulls in lance and DataFusion and takes a while.
The package keeps the lance name; the binary it produces does not.
`seaweed-worker-core` compiles the protocol straight out of `weed/pb/plugin.proto`
with a protoc it vendors, so it needs no system install. The lance crates compile
protos of their own, in build scripts nothing SeaweedFS sets can reach, so they
need a protoc on PATH (`brew install protobuf`, `apt install protobuf-compiler`)
or `PROTOC` naming one. A cold build pulls in lance and DataFusion and takes a
while — the binary is around 200MB.
## See also
+4
@@ -29,6 +29,10 @@ docker run -d chrislusf/seaweedfs volume-rust -mserver=localhost:9333
The Rust binary is included in the standard Docker image for `amd64` and `arm64` platforms. On `arm` and `386`, only the Go volume server is available.
The image carries one other Rust binary on the same terms: `weed-worker`, the
maintenance worker for Lance tables, under the verb `worker-rust`. See
[[Lance Maintenance Worker]].
### Build from Source
```bash
+1 -1
@@ -130,7 +130,7 @@ opens the same data with no catalog in the path. duckdb, pandas, Polars and Data
## Maintenance
A Lance table needs upkeep that no Iceberg worker can do — in particular, rows written after an index was built are not covered by it, so a vector search silently misses them. See [[Lance Maintenance Worker]].
A Lance table needs upkeep that no Iceberg worker can do — in particular, rows written after an index was built are not covered by it, so a vector search silently misses them. The worker ships in the SeaweedFS image (`docker run chrislusf/seaweedfs worker-rust --admin admin:23646`) and on each release; see [[Lance Maintenance Worker]].
## Admin UI
+3 -1
@@ -22,7 +22,9 @@ a first-class worker: it registers capabilities, answers detection and execution
and gets its settings page rendered from its own descriptor. The Lance jobs
(`lance_compact`, `lance_optimize_indices`, `lance_cleanup_versions`) come from
one — see [[Lance Maintenance Worker]] — because reading the Lance format in Go
is not implemented.
is not implemented. That worker ships in the same image, as
`docker run chrislusf/seaweedfs worker-rust`, and on each release as
`weed-worker_linux_{amd64,arm64}.tar.gz`.
## Usage