diff --git a/Lance-Maintenance-Worker.md b/Lance-Maintenance-Worker.md index 85ac5f2..a6422ab 100644 --- a/Lance-Maintenance-Worker.md +++ b/Lance-Maintenance-Worker.md @@ -4,12 +4,40 @@ A Lance table needs upkeep the Iceberg worker cannot do, because reading the Lan It is **not a sidecar**: `PluginControlService` is a language-agnostic gRPC contract for external maintenance workers, and `weed worker` is the Go reference implementation of it. Everything the Go worker gets from the protocol — scheduling, retries, dedupe, progress, concurrency limits, and a settings page rendered in the admin UI from the worker's own descriptor — the Rust worker gets too. -Source: `seaweed-worker/` in the SeaweedFS repository. +Source: `seaweed-worker/` in the SeaweedFS repository. The binary is +`weed-worker`: it is the Rust side of `weed worker`, the way `weed-volume` is +the Rust side of `weed volume`, and the Lance jobs are the first family it +carries rather than the only one it ever will. + +## Getting it + +### Docker + +```bash +docker run chrislusf/seaweedfs worker-rust \ + --admin admin:23646 --namespace http://s3:9101 +``` + +The standard image carries it at `/usr/bin/weed-worker` on `amd64` and `arm64`, +beside the Rust volume server. The verb mirrors `volume-rust`, so plain `worker` +still runs the Go one. On `arm` and `386` the image says the worker is not +available for the platform rather than failing as "not found". + +### From a release + +`weed-worker_linux_amd64.tar.gz` and `weed-worker_linux_arm64.tar.gz`, with +md5s, on each GitHub release. Linux only: the worker runs beside the cluster it +maintains, and lance, arrow and datafusion make every extra target an expensive +build. + +### From source + +A Rust toolchain and a protoc; see [Building it](#building-it) below. ## Running it ```bash -weed-lance-worker \ +weed-worker \ --admin localhost:23646 \ --namespace http://localhost:9101 ``` @@ -90,10 +118,17 @@ Note the port convention: master 9324, volume 9325, filer 9326, s3 9327 — so a ```bash cd seaweed-worker -cargo build --release -p weed-lance-worker +cargo build --release -p weed-lance-worker # binary at target/release/weed-worker ``` -Needs a Rust toolchain and `protoc`; the crates compile the protocol straight out of `weed/pb/plugin.proto`. A cold build pulls in lance and DataFusion and takes a while. +The package keeps the lance name; the binary it produces does not. + +`seaweed-worker-core` compiles the protocol straight out of `weed/pb/plugin.proto` +with a protoc it vendors, so it needs no system install. The lance crates compile +protos of their own, in build scripts nothing SeaweedFS sets can reach, so they +need a protoc on PATH (`brew install protobuf`, `apt install protobuf-compiler`) +or `PROTOC` naming one. A cold build pulls in lance and DataFusion and takes a +while — the binary is around 200MB. ## See also diff --git a/Rust-Volume-Server.md b/Rust-Volume-Server.md index 1bcc3ee..056fdd8 100644 --- a/Rust-Volume-Server.md +++ b/Rust-Volume-Server.md @@ -29,6 +29,10 @@ docker run -d chrislusf/seaweedfs volume-rust -mserver=localhost:9333 The Rust binary is included in the standard Docker image for `amd64` and `arm64` platforms. On `arm` and `386`, only the Go volume server is available. +The image carries one other Rust binary on the same terms: `weed-worker`, the +maintenance worker for Lance tables, under the verb `worker-rust`. See +[[Lance Maintenance Worker]]. + ### Build from Source ```bash diff --git a/SeaweedFS-Lance-Catalog.md b/SeaweedFS-Lance-Catalog.md index 56c23cf..e126a15 100644 --- a/SeaweedFS-Lance-Catalog.md +++ b/SeaweedFS-Lance-Catalog.md @@ -130,7 +130,7 @@ opens the same data with no catalog in the path. duckdb, pandas, Polars and Data ## Maintenance -A Lance table needs upkeep that no Iceberg worker can do — in particular, rows written after an index was built are not covered by it, so a vector search silently misses them. See [[Lance Maintenance Worker]]. +A Lance table needs upkeep that no Iceberg worker can do — in particular, rows written after an index was built are not covered by it, so a vector search silently misses them. The worker ships in the SeaweedFS image (`docker run chrislusf/seaweedfs worker-rust --admin admin:23646`) and on each release; see [[Lance Maintenance Worker]]. ## Admin UI diff --git a/Worker.md b/Worker.md index 8b360a6..57f5a15 100644 --- a/Worker.md +++ b/Worker.md @@ -22,7 +22,9 @@ a first-class worker: it registers capabilities, answers detection and execution and gets its settings page rendered from its own descriptor. The Lance jobs (`lance_compact`, `lance_optimize_indices`, `lance_cleanup_versions`) come from one — see [[Lance Maintenance Worker]] — because reading the Lance format in Go -is not implemented. +is not implemented. That worker ships in the same image, as +`docker run chrislusf/seaweedfs worker-rust`, and on each release as +`weed-worker_linux_{amd64,arm64}.tar.gz`. ## Usage