Github-Mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-08 23:50:43 +02:00
Code Issues Packages Projects Releases Wiki Activity

Document the Lance catalog, its worker, and worker metrics 

Lance appeared nowhere in the wiki, and two pages said a table bucket is
Iceberg. New: SeaweedFS Lance Catalog, LanceDB Integration, Lance
Maintenance Worker. Updated: table bucket pages for the format
declaration, Worker and System Metrics for what a plugin worker publishes,
Plugin Worker Scheduling for the lance job types and lanes, and Iceberg
Table Maintenance for the foreign-format skip that keeps orphan cleanup
away from Lance fragments.
Chris Lu
2026-08-21 10:51:39 -07:00
parent 400fb54280
commit b4a474bb7a
10 changed files with 435 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Iceberg-Table-Maintenance.md
+7
@@ -141,6 +141,13 @@ Set `apply_deletes=false` to revert to the previous behavior of skipping tables
#### Skip Conditions
A table whose recorded format is not `ICEBERG` is skipped by every job here,
before any metadata is read. This matters more than it sounds: a Lance table
registered through the Iceberg REST adapter keeps its fragments under `data/`,
where they are unreferenced by any Iceberg snapshot by construction, and orphan
cleanup would otherwise delete them. See [[SeaweedFS Lance Catalog]] for the
catalog that serves those tables properly.
| Condition | Result |
|:---|:---|
| All files >= `target_file_size_mb` | `"no files eligible for compaction"` |
Lance-Maintenance-Worker.md
+103
@@ -0,0 +1,103 @@
# Lance Maintenance Worker
A Lance table needs upkeep the Iceberg worker cannot do, because reading the Lance format in Go is not implemented. The Lance maintenance worker is a separate process, written in Rust, that connects to the admin server over the same plugin protocol `weed worker` uses.
It is **not a sidecar**: `PluginControlService` is a language-agnostic gRPC contract for external maintenance workers, and `weed worker` is the Go reference implementation of it. Everything the Go worker gets from the protocol — scheduling, retries, dedupe, progress, concurrency limits, and a settings page rendered in the admin UI from the worker's own descriptor — the Rust worker gets too.
Source: `seaweed-worker/` in the SeaweedFS repository.
## Running it
```bash
weed-lance-worker \
--admin localhost:23646 \
--namespace http://localhost:9101
```
`--admin` takes the admin server's **HTTP** address; the gRPC port is derived from it the same way the Go worker does. Dialling the gRPC port directly fails with "frame with invalid size", which reads like a protocol bug rather than a wrong port.
| Flag | Default | Meaning |
|------|---------|---------|
| `--admin` | `localhost:23646` | admin server's HTTP address |
| `--namespace` | | Lance Namespace URL (`WEED_LANCE_NAMESPACE`) |
| `--id` | hostname-derived | worker id; **two workers sharing one id evict each other** |
| `--heartbeat-seconds` | `10` | |
| `--max-concurrency` | `1` | detection and execution slots |
| `--access-key`, `--secret-key` | | storage credentials for a gateway that vends none |
| `--tls-ca`, `--tls-cert`, `--tls-key` | | mTLS for the control stream; all three together |
| `--tls-server-name` | | when admin's certificate does not name the address dialled |
| `--metrics-port` | `0` (off) | serves `/health`, `/ready`, `/metrics` |
| `--metrics-ip` | `127.0.0.1` | the endpoint is unauthenticated |
### Credentials
The worker holds none of its own. It asks the namespace to describe a table with `vend_credentials` and hands the `storage_options` to lance. A gateway without STS vends no credentials at all, so `--access-key` and `--secret-key` are a fallback; anything the namespace does vend wins over them.
If detection reports nothing at all, credentials are the first thing to check: a table the worker cannot open is skipped with a warning, and a sweep that could open nothing looks exactly like a cluster with no work to do. The `objects_skipped_total` metric below exists to tell those apart.
### TLS
`--tls-ca`, `--tls-cert` and `--tls-key` take the same certificates the Go worker reads from the `[grpc.worker]` section of `security.toml`, and must be given together — a CA on its own would quietly mean one-way TLS, which a mutual setup rejects anyway. Without them the stream is plaintext, which is the Go worker's behaviour too when nothing is configured.
## The jobs
| Job type | What it does | Detected from |
|----------|--------------|---------------|
| `lance_compact` | merges small fragments | fragment count |
| `lance_optimize_indices` | extends indices to cover rows written after they were built | rows no index covers |
| `lance_cleanup_versions` | removes old versions and the files only they referenced | version count and age |
**`lance_optimize_indices` is the one that matters most**, and has no Iceberg equivalent. Rows written after an index was built are not covered by it, so a vector search silently misses them. It is a correctness problem wearing a performance problem's clothes.
Each job type has a settings form in the admin UI, rendered from the worker's descriptor:
| Setting | Job | Default |
|---------|-----|---------|
| `min_fragments` | compact | 8 |
| `target_rows_per_fragment` | compact | 1048576 |
| `max_unindexed_rows` | optimize indices | 100000 |
| `retain_hours` | cleanup | 168 |
| `min_versions_to_keep` | cleanup | 5 |
`min_versions_to_keep` is a floor applied when the job runs, not only when it is proposed: versions that age past the retention window between proposal and execution do not take the table below it.
### Job types start disabled
Like every plugin job type, these are `enabled=false` until an operator turns them on in the admin UI (or via `PUT /api/plugin/job-types/{jobType}/config`). A connected worker with nothing running is usually this.
## What the worker tells the admin UI
SeaweedFS cannot read a Lance table, so the admin UI would otherwise show a location and nothing else. The worker fills that in:
- **Observations** — while detection opens a dataset to decide whether it needs work, it reports the schema, row count, fragment count and version count. Admin caches the last one per table and serves it back with the time it was taken and which worker took it. Nothing is scheduled from it; it is a cache with its staleness on the label. Also available at `GET /api/plugin/observations`.
- **Sample rows** — fetched from a worker when the Browse Data page is opened, never cached, because rows are the table's data rather than a description of it.
## Metrics
With `--metrics-port`, the worker serves the same three endpoints as `weed worker -metricsPort`:
| Path | |
|------|--|
| `/health` | process is alive |
| `/ready` | the control stream is up — a worker whose admin has gone away is running but will do nothing |
| `/metrics` | Prometheus |
Names are `SeaweedFS_worker_*`; see [[System Metrics]] for the list. The pair worth alerting on is `objects_seen_total` against `objects_skipped_total`.
Note the port convention: master 9324, volume 9325, filer 9326, s3 9327 — so a worker on the same host wants 9328.
## Building it
```bash
cd seaweed-worker
cargo build --release -p weed-lance-worker
```
Needs a Rust toolchain and `protoc`; the crates compile the protocol straight out of `weed/pb/plugin.proto`. A cold build pulls in lance and DataFusion and takes a while.
## See also
- [[SeaweedFS Lance Catalog]]
- [[Worker]] — the Go plugin worker
- [[Plugin Worker Scheduling]]
- [[System Metrics]]
LanceDB-Integration.md
+110
@@ -0,0 +1,110 @@
# LanceDB Integration
[LanceDB](https://lancedb.com) connects to the [[SeaweedFS Lance Catalog]] over the Lance Namespace REST protocol, so a SeaweedFS table bucket works as a LanceDB catalog with no other service in the way.
Verified against `lancedb 0.37.1`, `lance-namespace 0.8.6` and `pylance 10.0.0` by the integration suite in `test/s3tables/catalog_lancedb/`.
## 1. Start SeaweedFS
```bash
weed server -dir=/data -s3 -s3.port.lance=9101
```
## 2. Create a Lance table bucket
```bash
weed shell
> s3tables.bucket -create -name vectors -format LANCE -account 000000000000
```
## 3. Connect
```python
import lancedb
db = lancedb.connect_namespace(
"rest",
{"uri": "http://localhost:9101"},
storage_options={
"aws_endpoint": "http://localhost:8333",
"allow_http": "true",
"aws_access_key_id": "...",
"aws_secret_access_key": "...",
"aws_region": "us-east-1",
},
)
```
`connect_namespace` takes the properties dict as `Dict[str, str]`; `storage_options` is a separate keyword argument, and passing it inside the properties fails with `'dict' object is not an instance of 'str'`.
## 4. Use it
A table bucket is the first level of the namespace path, so `namespace_path` is `[bucket, namespace]`:
```python
# create - LanceDB declares the table through the catalog and writes the data
table = db.create_table(
"embeddings", data=rows, namespace_path=["vectors", "ml"])
# list
db.table_names(namespace_path=["vectors", "ml"])
# -> ['vectors$ml$embeddings']
# open and query
table = db.open_table("embeddings", namespace_path=["vectors", "ml"])
table.count_rows()
table.search([0.1] * 8).limit(5).to_list()
table.search().where("id < 5").limit(10).to_list()
# index
table.create_index(metric="l2", vector_column_name="vector",
index_type="IVF_PQ", num_partitions=1, num_sub_vectors=4)
```
## Credentials
**This is the one that catches people.** A gateway without STS configured vends `storage_options` carrying an endpoint and a region but no credentials, and LanceDB uses what the namespace vends on some paths — so a client configured only through `connect_namespace(storage_options=...)` can still end up with none, failing inside lance with:
```
Failed to get AWS credentials: CredentialsNotLoaded(... "no providers in chain provided credentials")
```
Two ways out, and you can use both:
1. **Configure credential vending** so the namespace hands out real, scoped, expiring credentials:
```bash
weed server -s3 -s3.port.lance=9101 -s3.iceberg.credentialRole=arn:aws:iam::…:role/…
```
2. **Give the client credentials the ordinary way**, in the environment, so lance's provider chain finds them whichever path is taken:
```bash
export AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=…
export AWS_ENDPOINT_URL=http://localhost:8333
export AWS_ALLOW_HTTP=true
```
## What is not served
`create_table` works because LanceDB declares through the catalog and writes the data itself. Asking LanceDB to push the operation to the server instead —
```python
lancedb.connect_namespace(..., namespace_client_pushdown_operations=["CreateTable"])
```
— calls the namespace's own `CreateTable`, which carries Arrow data and is not implemented here; see [[SeaweedFS Lance Catalog]] for the full list. The client falls back to declare-and-write, so the table still lands.
## Without the catalog
The same dataset opens by URI, which is worth knowing before you build anything that assumes the catalog is always reachable:
```python
import lance
lance.dataset("s3://vectors/ml/embeddings", storage_options=opts)
```
## See also
- [[SeaweedFS Lance Catalog]]
- [[Lance Maintenance Worker]] — compaction, index optimization, version cleanup
Plugin-Worker-Scheduling.md
+17
@@ -111,6 +111,23 @@ Each job type has its own `detection_interval_seconds` that controls how often t
| `erasure_coding` | 5 minutes |
| `admin_script` | configurable |
| `iceberg_maintenance` | 1 hour |
| `lance_compact` | from the worker's descriptor |
| `lance_optimize_indices` | from the worker's descriptor |
| `lance_cleanup_versions` | from the worker's descriptor |
The Lance job types come from an external worker (see [[Lance Maintenance Worker]]),
so their defaults arrive with the worker's descriptor rather than being compiled in.
### Lanes
Job types are grouped into lanes, and a lane runs one group at a time. `default`
takes the cluster admin lock, so anything in it is serialised behind vacuum and
balancing; `iceberg`, `lifecycle` and `lance` do not, because they act on table
data rather than on volumes.
Lane assignment lives in the admin server, not in the worker: a new job type from
an external worker lands in `default` until it is mapped, which silently serialises
it behind cluster maintenance.
On each iteration, the scheduler:
S3-Table-Bucket-Commands.md
+18
@@ -34,6 +34,24 @@ curl -X POST $S3_ENDPOINT/ \
-d '{"name": "my-test-bucket"}'
```
> [!NOTE]
> A table bucket holds **one table format**. `format` is a SeaweedFS extension to
> `CreateTableBucket` and accepts `ICEBERG` (the default, and what AWS S3 Tables
> serves) or `LANCE`. Creating a table of another format in a declared bucket is
> refused with 409.
>
> ```bash
> curl -X POST $S3_ENDPOINT/ \
> -H "X-Amz-Target: S3Tables.CreateTableBucket" \
> -H "Content-Type: application/x-amz-json-1.1" \
> -d '{"name": "vectors", "format": "LANCE"}'
> ```
>
> From the shell: `s3tables.bucket -create -name vectors -format LANCE -account 000000000000`
>
> Buckets created before this field existed carry no declaration and keep
> accepting either format. See [[SeaweedFS Lance Catalog]].
---
### List Table Buckets
S3-Table-Bucket.md
+1 -1
@@ -2,7 +2,7 @@
## Introduction
SeaweedFS supports **Amazon S3 Tables**, providing a dedicated interface for managing structured datasets using the **Apache Iceberg** table format. Unlike standard S3 buckets that store unstructured objects, S3 Table Buckets are optimized for analytics workloads, offering a hierarchical structure of **Namespaces** and **Tables**.
SeaweedFS supports **Amazon S3 Tables**, providing a dedicated interface for managing structured datasets. A table bucket holds one table format and declares which when it is created: **Apache Iceberg**, served by the [[SeaweedFS Iceberg Catalog]], or **Lance**, served by the [[SeaweedFS Lance Catalog]]. Unlike standard S3 buckets that store unstructured objects, S3 Table Buckets are optimized for analytics workloads, offering a hierarchical structure of **Namespaces** and **Tables**.
This feature implements the **Iceberg REST Catalog API**, allowing direct integration with analytics engines like Apache Spark, Trino, Dremio, DuckDB, and RisingWave without needing an external catalog service.
SeaweedFS-Lance-Catalog.md
+121
@@ -0,0 +1,121 @@
# SeaweedFS Lance Catalog
SeaweedFS provides a built-in [Lance Namespace](https://lancedb.github.io/lance-namespace/) REST catalog, beside the [[SeaweedFS Iceberg Catalog]] and over the same table buckets.
[Lance](https://lancedb.com) is a columnar format built for multimodal and vector data: random access is orders of magnitude faster than Parquet, and a dataset carries its own vector and scalar indices.
## Architecture
- **Lance Namespace API**: a dedicated port (default `9101`)
- **S3 data access**: the S3 port (default `8333`)
- **Authentication**: the same SigV4 identities as the rest of the S3 stack, with optional credential vending
A **table bucket is a catalog**, exactly as it is for Iceberg. Which protocol serves it depends on the format the bucket declares.
| Format | Served by | Default port |
|--------|-----------|--------------|
| `ICEBERG` | Iceberg REST Catalog | 8181 |
| `LANCE` | Lance Namespace API | 9101 |
Identifiers are lists, joined on the wire by `$` (configurable per request), and map onto what already exists:
```
["vectors"] -> table bucket "vectors"
["vectors", "ml"] -> namespace "ml" in that bucket
["vectors", "ml", "embeddings"] -> table "embeddings"
```
The root (`/v1/namespace/%24/list`, or just the delimiter) lists table buckets.
## Quick Start
### 1. Start SeaweedFS
```bash
weed server -s3 -s3.port.lance=9101
```
or `weed mini`, which starts the Lance namespace along with everything else.
### 2. Create a Lance table bucket
```bash
weed shell
> s3tables.bucket -create -name vectors -format LANCE -account 000000000000
```
A bucket holds one format. Declaring `LANCE` means the catalog refuses an Iceberg table in it, and the reverse. Buckets created before formats were declared carry none and keep accepting either.
### 3. Use it
```python
import lance, lance_namespace as ln
ns = ln.connect("rest", {"uri": "http://localhost:9101"})
ns.create_namespace(ln.CreateNamespaceRequest(id=["vectors", "ml"]))
declared = ns.declare_table(ln.DeclareTableRequest(id=["vectors", "ml", "embeddings"]))
lance.write_dataset(table, declared.location, storage_options=declared.storage_options)
```
See [[LanceDB Integration]] for the same thing through LanceDB.
## What the catalog serves
| Operation | Route | Notes |
|-----------|-------|-------|
| Create / list / describe / drop / exists namespace | `/v1/namespace/{id}/…` | root lists table buckets |
| List tables | `GET /v1/namespace/{id}/table/list` | |
| List every table | `GET /v1/table` | across the catalog |
| Declare table | `POST /v1/table/{id}/declare` | reserves the name, returns the location |
| Describe table | `POST /v1/table/{id}/describe` | location, and credentials on request |
| Register / deregister | `POST /v1/table/{id}/register`, `/deregister` | see below |
| Drop / rename | `POST /v1/table/{id}/drop`, `/rename` | |
Everything else — `create`, `insert`, `query`, `count_rows`, index, tag, branch and version operations — answers the spec's `Unsupported` (HTTP 501). Those carry Arrow data and would mean reading and writing the Lance file format inside the gateway, which is not implemented. Clients do the reading and writing themselves against the location the catalog hands back, which is the split the format is designed for.
`managed_versioning` is `false` and will stay false: a Lance commit is a conditional PUT, and SeaweedFS answers `If-None-Match: *` atomically at the object's owner filer, so the dataset can own its own version history.
### Deregister is not a drop
`deregister` hides a table from the catalog and **keeps the data**. `drop` removes both. A deregistered table can be brought back with `register`, or read straight off its location by any client.
## Credential vending
The Lance spec carries this in the protocol. Ask for it on a describe:
```python
described = ns.describe_table(ln.DescribeTableRequest(
id=["vectors", "ml", "embeddings"], vend_credentials=True))
dataset = lance.dataset(described.location, storage_options=described.storage_options)
```
The response's `storage_options` carry object_store's own key names — `aws_access_key_id`, `aws_secret_access_key`, `aws_session_token`, `aws_region`, `aws_endpoint`, `allow_http` — plus `expires_at_millis`. It reuses the same STS path as the Iceberg catalog; configure it with `-s3.iceberg.credentialRole`.
Without STS configured, the namespace vends an endpoint and a region but **no credentials**, and the client must bring its own. See [[LanceDB Integration]] for what that looks like in practice.
## The catalog is optional
Tables are laid out so that a namespace prefix is also a valid Lance directory:
```python
lance.dataset("s3://vectors/ml/embeddings", storage_options=opts)
```
opens the same data with no catalog in the path. duckdb, pandas, Polars and DataFusion do not speak the namespace protocol; this is what keeps them able to read your tables.
## Maintenance
A Lance table needs upkeep that no Iceberg worker can do — in particular, rows written after an index was built are not covered by it, so a vector search silently misses them. See [[Lance Maintenance Worker]].
## Admin UI
Table buckets show the format they hold and the endpoint that serves them. For a Lance table, the schema, row count and version history come from a maintenance worker rather than from the gateway, which cannot read the format; sample rows are fetched from a worker when the page is opened. Both are labelled with which worker answered and when.
## See also
- [[LanceDB Integration]]
- [[Lance Maintenance Worker]]
- [[S3 Table Bucket]]
- [[SeaweedFS Iceberg Catalog]]
System-Metrics.md
+40
@@ -40,6 +40,46 @@ Note: All server should be running on different ports for accepting prometheus m
And then you can configure your Prometheus to crawl them periodically.
# Plugin Worker Metrics
Maintenance workers publish their own metrics when started with a metrics port —
`weed worker -metricsPort=9328` for the Go worker, `--metrics-port 9328` for the
Rust one (see [[Lance Maintenance Worker]]). Both serve `/health`, `/ready` and
`/metrics` on it, so one scrape config covers either.
Pick a port that is free: master 9324, volume 9325, filer 9326 and s3 9327 are
the convention, so a worker on the same host wants 9328.
| Metric | Type | Labels |
|--------|------|--------|
| `SeaweedFS_worker_build_info` | gauge | `worker_id`, `version` |
| `SeaweedFS_worker_connected` | gauge | 1 while the control stream is up |
| `SeaweedFS_worker_stream_events_total` | counter | `event` = connected, closed, failed, shutdown |
| `SeaweedFS_worker_slots_used` / `_slots_total` | gauge | `lane` = detection, execution |
| `SeaweedFS_worker_detections_total` | counter | `job_type`, `result` |
| `SeaweedFS_worker_detection_seconds` | histogram | `job_type` |
| `SeaweedFS_worker_proposals_total` | counter | `job_type` |
| `SeaweedFS_worker_objects_seen_total` | counter | `job_type` |
| `SeaweedFS_worker_objects_skipped_total` | counter | `job_type`, `reason` |
| `SeaweedFS_worker_jobs_total` | counter | `job_type`, `result` |
| `SeaweedFS_worker_job_seconds` | histogram | `job_type` |
| `SeaweedFS_worker_previews_total` | counter | `result` |
The Lance worker adds what it reclaimed: `SeaweedFS_worker_lance_fragments_removed_total`,
`_lance_rows_indexed_total`, `_lance_versions_removed_total`, `_lance_bytes_reclaimed_total`.
**The pair worth alerting on is `objects_seen_total` against `objects_skipped_total`.**
A sweep that proposes nothing because there is nothing to do and a sweep that
proposes nothing because it could read nothing produce the same
`proposals_total`; skips are what tells them apart, and the usual cause is
credentials.
> [!NOTE]
> `SeaweedFS_admin_workers_connected`, `SeaweedFS_admin_worker_slots` and
> `SeaweedFS_admin_worker_events_total` are fed by the older maintenance-worker
> queue, **not** the plugin system. Plugin workers — Go or Rust — do not appear
> in them. Use the `SeaweedFS_worker_*` family above for those.
# Dashboard
The dashboard is shared at https://github.com/seaweedfs/seaweedfs/blob/master/other/metrics/grafana_seaweedfs.json
Worker.md
+13 -1
@@ -16,6 +16,14 @@ Built-in job types:
| `erasure_coding` | heavy | Convert volumes to erasure-coded format for storage efficiency |
| `iceberg_maintenance` | heavy | Compact, expire snapshots, remove orphans for Iceberg tables |
`weed worker` is the Go implementation of the plugin protocol, not the only one.
`PluginControlService` is language-agnostic, and a worker in another language is
a first-class worker: it registers capabilities, answers detection and execution,
and gets its settings page rendered from its own descriptor. The Lance jobs
(`lance_compact`, `lance_optimize_indices`, `lance_cleanup_versions`) come from
one — see [[Lance Maintenance Worker]] — because reading the Lance format in Go
is not implemented.
## Usage
```bash
@@ -117,12 +125,16 @@ weed worker -admin=localhost:23646 -id=ec-worker-1 -jobType=erasure_coding
```bash
# Enable Prometheus /metrics, /health, /ready endpoints
weed worker -admin=localhost:23646 -metricsPort=9327
weed worker -admin=localhost:23646 -metricsPort=9328
# Debug with pprof
weed worker -admin=localhost:23646 -debug -debug.port=6060
```
The metric names are listed in [[System Metrics]]. Note that 9327 is the S3
gateway's metrics port in the sample configurations, so a worker sharing a host
with one wants 9328.
## Worker Architecture
### Worker Lifecycle
_Sidebar.md
+5
@@ -112,6 +112,8 @@
* [[SeaweedFS Iceberg Catalog]]
* [[Iceberg REST Catalog API]]
* [[Iceberg Table Maintenance]]
* [[SeaweedFS Lance Catalog]]
* [[Lance Maintenance Worker]]
### Iceberg Integrations
* [[Spark Iceberg Integration]]
@@ -122,6 +124,9 @@
* [[RisingWave Iceberg Integration]]
* [[Lakekeeper Iceberg Integration]]
### Lance Integrations
* [[LanceDB Integration]]
### S3 Authentication & IAM
* [[S3 Configuration]] - Start Here
* [[S3 Credentials]] (`-s3.config`)