mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-20 13:30:46 +02:00
Say where the Lance worker comes from now that it ships
It is in the SeaweedFS image as worker-rust and on each release as
weed-worker_linux_{amd64,arm64}.tar.gz, so a Rust toolchain is no longer the
only way in. The binary is weed-worker, not weed-lance-worker: the crate keeps
the lance name, the binary does not.
Also records what the build needs from protoc - core vendors its own, lance's
crates need one of their own.
+39
-4
@@ -4,12 +4,40 @@ A Lance table needs upkeep the Iceberg worker cannot do, because reading the Lan
|
||||
|
||||
It is **not a sidecar**: `PluginControlService` is a language-agnostic gRPC contract for external maintenance workers, and `weed worker` is the Go reference implementation of it. Everything the Go worker gets from the protocol — scheduling, retries, dedupe, progress, concurrency limits, and a settings page rendered in the admin UI from the worker's own descriptor — the Rust worker gets too.
|
||||
|
||||
Source: `seaweed-worker/` in the SeaweedFS repository.
|
||||
Source: `seaweed-worker/` in the SeaweedFS repository. The binary is
|
||||
`weed-worker`: it is the Rust side of `weed worker`, the way `weed-volume` is
|
||||
the Rust side of `weed volume`, and the Lance jobs are the first family it
|
||||
carries rather than the only one it ever will.
|
||||
|
||||
## Getting it
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
docker run chrislusf/seaweedfs worker-rust \
|
||||
--admin admin:23646 --namespace http://s3:9101
|
||||
```
|
||||
|
||||
The standard image carries it at `/usr/bin/weed-worker` on `amd64` and `arm64`,
|
||||
beside the Rust volume server. The verb mirrors `volume-rust`, so plain `worker`
|
||||
still runs the Go one. On `arm` and `386` the image says the worker is not
|
||||
available for the platform rather than failing as "not found".
|
||||
|
||||
### From a release
|
||||
|
||||
`weed-worker_linux_amd64.tar.gz` and `weed-worker_linux_arm64.tar.gz`, with
|
||||
md5s, on each GitHub release. Linux only: the worker runs beside the cluster it
|
||||
maintains, and lance, arrow and datafusion make every extra target an expensive
|
||||
build.
|
||||
|
||||
### From source
|
||||
|
||||
A Rust toolchain and a protoc; see [Building it](#building-it) below.
|
||||
|
||||
## Running it
|
||||
|
||||
```bash
|
||||
weed-lance-worker \
|
||||
weed-worker \
|
||||
--admin localhost:23646 \
|
||||
--namespace http://localhost:9101
|
||||
```
|
||||
@@ -90,10 +118,17 @@ Note the port convention: master 9324, volume 9325, filer 9326, s3 9327 — so a
|
||||
|
||||
```bash
|
||||
cd seaweed-worker
|
||||
cargo build --release -p weed-lance-worker
|
||||
cargo build --release -p weed-lance-worker # binary at target/release/weed-worker
|
||||
```
|
||||
|
||||
Needs a Rust toolchain and `protoc`; the crates compile the protocol straight out of `weed/pb/plugin.proto`. A cold build pulls in lance and DataFusion and takes a while.
|
||||
The package keeps the lance name; the binary it produces does not.
|
||||
|
||||
`seaweed-worker-core` compiles the protocol straight out of `weed/pb/plugin.proto`
|
||||
with a protoc it vendors, so it needs no system install. The lance crates compile
|
||||
protos of their own, in build scripts nothing SeaweedFS sets can reach, so they
|
||||
need a protoc on PATH (`brew install protobuf`, `apt install protobuf-compiler`)
|
||||
or `PROTOC` naming one. A cold build pulls in lance and DataFusion and takes a
|
||||
while — the binary is around 200MB.
|
||||
|
||||
## See also
|
||||
|
||||
|
||||
@@ -29,6 +29,10 @@ docker run -d chrislusf/seaweedfs volume-rust -mserver=localhost:9333
|
||||
|
||||
The Rust binary is included in the standard Docker image for `amd64` and `arm64` platforms. On `arm` and `386`, only the Go volume server is available.
|
||||
|
||||
The image carries one other Rust binary on the same terms: `weed-worker`, the
|
||||
maintenance worker for Lance tables, under the verb `worker-rust`. See
|
||||
[[Lance Maintenance Worker]].
|
||||
|
||||
### Build from Source
|
||||
|
||||
```bash
|
||||
|
||||
@@ -130,7 +130,7 @@ opens the same data with no catalog in the path. duckdb, pandas, Polars and Data
|
||||
|
||||
## Maintenance
|
||||
|
||||
A Lance table needs upkeep that no Iceberg worker can do — in particular, rows written after an index was built are not covered by it, so a vector search silently misses them. See [[Lance Maintenance Worker]].
|
||||
A Lance table needs upkeep that no Iceberg worker can do — in particular, rows written after an index was built are not covered by it, so a vector search silently misses them. The worker ships in the SeaweedFS image (`docker run chrislusf/seaweedfs worker-rust --admin admin:23646`) and on each release; see [[Lance Maintenance Worker]].
|
||||
|
||||
## Admin UI
|
||||
|
||||
|
||||
+3
-1
@@ -22,7 +22,9 @@ a first-class worker: it registers capabilities, answers detection and execution
|
||||
and gets its settings page rendered from its own descriptor. The Lance jobs
|
||||
(`lance_compact`, `lance_optimize_indices`, `lance_cleanup_versions`) come from
|
||||
one — see [[Lance Maintenance Worker]] — because reading the Lance format in Go
|
||||
is not implemented.
|
||||
is not implemented. That worker ships in the same image, as
|
||||
`docker run chrislusf/seaweedfs worker-rust`, and on each release as
|
||||
`weed-worker_linux_{amd64,arm64}.tar.gz`.
|
||||
|
||||
## Usage
|
||||
|
||||
|
||||
Reference in New Issue
Block a user