Files
seaweedfs/seaweed-vector
aceb089bc8 lance: per-table cleanup retention and on-demand purge of deleted rows (#12739)
* lance: per-table cleanup retention and on-demand purge of deleted rows

A vector deleted from an index stayed in storage until compaction
rewrote its fragment (only above 10% deletions) and the cleanup job's
168 h retention, set once per worker, aged out every version that
referenced it.

Let a table override retain_hours and min_versions in its Lance config
(seaweedfs.cleanup.retain_hours, seaweedfs.cleanup.min_versions), and
let it request a purge (seaweedfs.purge.requested). The cleanup job
proposes a table with a pending purge regardless of the version floor
and then rewrites every fragment with deletions, retrains the indices
(a remapped IVF_FLAT index still holds the deleted vector), keeps only
the latest version, and records the request as served. A version pinned
by a tag fails the purge and leaves the request pending.

The test scans every file of a local table for a deleted vector's key,
metadata and values, and finds none after the purge.

* lance: refuse to purge a sharded vector index

A retrain builds every index name over the whole table, so the purge
turned shards of one vector index (one name per fragment set) into
names that each covered every fragment, indexing every row once per
shard. Check before compacting and fail with the request left pending,
and retrain only the vector indices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: rebuild scalar indices in a purge

Compaction only remaps scalar indices, and a retrain leaves them alone,
so a purge reported success while a bitmap index still listed the
deleted row's value, and under stable row ids a BTree index still held
the deleted key. Build each scalar index again from the remaining rows
with the parameters it was built with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: purge through the cleanup job's retention policy

The purge built its own cleanup policy, so a deployment that has the
cleanup job keep more than the floor and the window (versions an
unleased reader may still be on) would have lost them to a purge. Clean
up through retention_policy, the one the job itself uses, with the
latest version kept.

Mark the request served only once the oldest version left is one the
rebuild wrote, and record the rebuild so a purge that has to wait for
its cleanup (a tag, or that policy) retries only the cleanup instead of
compacting and retraining the indices on every pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: lease the version every list page reads

A list page read its version without a lease unless it was a segment
that read data and metadata twice. A purge removes every version but
the latest, and only a lease tag stops it, so a listing that opened the
version before the purge's compaction could lose its files mid-read.
Pin it as get_vectors and query_vectors do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: give the cleanup job a build's execution time

A purge compacts the table and retrains its indices inside a cleanup
job, but the job declared no runtime defaults, so the admin held it to
its 90 s default and could expire a purge mid-rebuild. Declare the same
budget the indices job uses for a retrain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:24:28 -08:00
..

SeaweedFS vector engine

weed-vector serves the S3 Vectors data plane through the S3 gateway. Indexes use Lance tables on the cluster's filers and volume servers. Go and Rust volume servers use the same engine and API.

With a local weed mini cluster running:

weed-vector --masters=localhost:9333 --metrics-port=9329

Point an AWS S3 Vectors client at the S3 endpoint, normally http://localhost:8333. The engine registers with the masters automatically. --listen defaults to 127.0.0.1:8090; use an address reachable from the gateway when running on another host or in a container.

Setting Default Purpose
--masters Required Comma-separated master addresses
--listen 127.0.0.1:8090 gRPC listener and registered address
--filer-group Empty Filer group used for table storage
--read-cache-mb 512 Immutable data/index read cache; zero disables it
--nprobes Automatic ANN partitions searched per query
--metrics-port 0 HTTP metrics and health listener; zero disables it
--metrics-ip 127.0.0.1 HTTP listener address
--security-toml Discovered Cluster certificates, signing keys and encryption keys

The HTTP listener provides /health, /ready, and /metrics. Readiness requires an active registration with the master. The gRPC listener also implements the standard health service for vector_pb.VectorService and the whole server. These health statuses follow master registration.

Metrics use the SeaweedFS_vector_ prefix:

  • connected: active master registration.
  • request_total{type,code}, request_seconds{type}, and in_flight_requests{type}: engine requests and their gRPC outcomes.
  • query_results_total: vectors returned by queries.
  • cache_requests_total{result}, cache_bytes, and cache_entries: immutable read cache activity and occupancy.
  • storage_read_bytes_total and storage_read_seconds: storage fetches for cacheable immutable reads.

The S3 gateway separately publishes SeaweedFS_s3_ request metrics with type="s3vectors.<Operation>". Scrape both processes to distinguish gateway latency from engine work. Metrics and HTTP probes are unauthenticated; bind them to an internal address or restrict their network access.

For mutual TLS, configure [grpc.vector] for the engine's server certificate, [grpc.client] for its master/filer connections, and the cluster CA. HTTPS volume reads also use [https.client]. The engine follows the same signing and encryption configuration as the other cluster clients.

Run the Lance maintenance worker against the same masters and Lance namespace to build and update ANN indexes. Queries can scan data that has not been indexed yet. See the maintenance worker documentation and Helm chart for deployment settings.

The integration suite at test/s3vectors runs the same API and observability checks against both Go and Rust volume servers, including mutual TLS, signed volume reads, and encrypted volume data.