* lance: per-table cleanup retention and on-demand purge of deleted rows A vector deleted from an index stayed in storage until compaction rewrote its fragment (only above 10% deletions) and the cleanup job's 168 h retention, set once per worker, aged out every version that referenced it. Let a table override retain_hours and min_versions in its Lance config (seaweedfs.cleanup.retain_hours, seaweedfs.cleanup.min_versions), and let it request a purge (seaweedfs.purge.requested). The cleanup job proposes a table with a pending purge regardless of the version floor and then rewrites every fragment with deletions, retrains the indices (a remapped IVF_FLAT index still holds the deleted vector), keeps only the latest version, and records the request as served. A version pinned by a tag fails the purge and leaves the request pending. The test scans every file of a local table for a deleted vector's key, metadata and values, and finds none after the purge. * lance: refuse to purge a sharded vector index A retrain builds every index name over the whole table, so the purge turned shards of one vector index (one name per fragment set) into names that each covered every fragment, indexing every row once per shard. Check before compacting and fail with the request left pending, and retrain only the vector indices. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: rebuild scalar indices in a purge Compaction only remaps scalar indices, and a retrain leaves them alone, so a purge reported success while a bitmap index still listed the deleted row's value, and under stable row ids a BTree index still held the deleted key. Build each scalar index again from the remaining rows with the parameters it was built with. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: purge through the cleanup job's retention policy The purge built its own cleanup policy, so a deployment that has the cleanup job keep more than the floor and the window (versions an unleased reader may still be on) would have lost them to a purge. Clean up through retention_policy, the one the job itself uses, with the latest version kept. Mark the request served only once the oldest version left is one the rebuild wrote, and record the rebuild so a purge that has to wait for its cleanup (a tag, or that policy) retries only the cleanup instead of compacting and retraining the indices on every pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: lease the version every list page reads A list page read its version without a lease unless it was a segment that read data and metadata twice. A purge removes every version but the latest, and only a lease tag stops it, so a listing that opened the version before the purge's compaction could lose its files mid-read. Pin it as get_vectors and query_vectors do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: give the cleanup job a build's execution time A purge compacts the table and retrains its indices inside a cleanup job, but the job declared no runtime defaults, so the admin held it to its 90 s default and could expire a purge mid-rebuild. Declare the same budget the indices job uses for a retrain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
SeaweedFS vector engine
weed-vector serves the S3 Vectors data plane through the S3 gateway. Indexes use Lance tables on the cluster's filers and volume servers. Go and Rust volume servers use the same engine and API.
With a local weed mini cluster running:
weed-vector --masters=localhost:9333 --metrics-port=9329
Point an AWS S3 Vectors client at the S3 endpoint, normally http://localhost:8333. The engine registers with the masters automatically. --listen defaults to 127.0.0.1:8090; use an address reachable from the gateway when running on another host or in a container.
| Setting | Default | Purpose |
|---|---|---|
--masters |
Required | Comma-separated master addresses |
--listen |
127.0.0.1:8090 |
gRPC listener and registered address |
--filer-group |
Empty | Filer group used for table storage |
--read-cache-mb |
512 |
Immutable data/index read cache; zero disables it |
--nprobes |
Automatic | ANN partitions searched per query |
--metrics-port |
0 |
HTTP metrics and health listener; zero disables it |
--metrics-ip |
127.0.0.1 |
HTTP listener address |
--security-toml |
Discovered | Cluster certificates, signing keys and encryption keys |
The HTTP listener provides /health, /ready, and /metrics. Readiness requires an active registration with the master. The gRPC listener also implements the standard health service for vector_pb.VectorService and the whole server. These health statuses follow master registration.
Metrics use the SeaweedFS_vector_ prefix:
connected: active master registration.request_total{type,code},request_seconds{type}, andin_flight_requests{type}: engine requests and their gRPC outcomes.query_results_total: vectors returned by queries.cache_requests_total{result},cache_bytes, andcache_entries: immutable read cache activity and occupancy.storage_read_bytes_totalandstorage_read_seconds: storage fetches for cacheable immutable reads.
The S3 gateway separately publishes SeaweedFS_s3_ request metrics with type="s3vectors.<Operation>". Scrape both processes to distinguish gateway latency from engine work. Metrics and HTTP probes are unauthenticated; bind them to an internal address or restrict their network access.
For mutual TLS, configure [grpc.vector] for the engine's server certificate, [grpc.client] for its master/filer connections, and the cluster CA. HTTPS volume reads also use [https.client]. The engine follows the same signing and encryption configuration as the other cluster clients.
Run the Lance maintenance worker against the same masters and Lance namespace to build and update ANN indexes. Queries can scan data that has not been indexed yet. See the maintenance worker documentation and Helm chart for deployment settings.
The integration suite at test/s3vectors runs the same API and observability checks against both Go and Rust volume servers, including mutual TLS, signed volume reads, and encrypted volume data.