From 8350f297af50111e1595a9baea5036da5015065b Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Wed, 4 Nov 2026 19:25:35 -0800 Subject: [PATCH] seaweed-vector: dev plan for ANN search over S3 Vectors indexes (#12408) * seaweed-vector: dev plan for ANN search over S3 Vectors indexes Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: one probe count per query, a budget that can shrink Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: smaller deltas are searched in full only up to the probe count Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: drop the private issue reference Co-Authored-By: Claude Opus 5.5 * seaweed-vector: dev plan: phase 1 probes use the Design count Co-Authored-By: Claude Opus 5.5 * seaweed-vector: dev plan: status after phases 0 to 3, and what the measurements changed Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: name the Enterprise repository, not the private one Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: the Enterprise repository by that name Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: a filtered query widens to four times the probes, not without bound Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: throughput through the index at 50k and 200k Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: union coverage over shard names, and an explicit merge at the delta limit Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: dev plan: status table names branches, not pull requests Co-Authored-By: Claude Opus 5.5 * seaweed-vector: dev plan: maximum_nprobes is always four times the count Lance reads maximum_nprobes only under a prefilter, and the phase 1 policy sets it the same way for every query. Co-Authored-By: Claude Opus 5.5 * seaweed-vector: dev plan: the retrain follows the partition count, not a recorded row count Co-Authored-By: Claude Opus 5.5 * seaweed-vector: dev plan: phase 3 is open like the other phases Co-Authored-By: Claude Opus 5.5 * seaweed-vector: dev plan: the retrain stops past 2,048 partitions Co-Authored-By: Claude Opus 5.5 * seaweed-vector: dev plan: the merge at the delta limit takes the name's delta count Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Fable 5.1 --- seaweed-vector/DEV_PLAN.md | 339 +++++++++++++++++++++++++++++++++++++ 1 file changed, 339 insertions(+) create mode 100644 seaweed-vector/DEV_PLAN.md diff --git a/seaweed-vector/DEV_PLAN.md b/seaweed-vector/DEV_PLAN.md new file mode 100644 index 0000000000..2407c0b3bc --- /dev/null +++ b/seaweed-vector/DEV_PLAN.md @@ -0,0 +1,339 @@ +# S3 Vectors ANN — Dev Plan + +## Goal + +Approximate nearest-neighbour search for S3 Vectors indexes, built on Lance +vector indexes, so that query latency stops growing linearly with the index. +Target: 90%+ recall@10 (what AWS states for S3 Vectors), exact results below +a size threshold and for rows written since the last index build, and one +code path that serves both the open-source single engine and the Enterprise +distributed query layer. Writes stay unchanged. + +## Status (2026-11-04) + +| Phase | Branch | State | +|---|---|---| +| 0 Build and measure | `vector-ann-phase0` | open: `vector_index` builder, `weed-vector-cli index`, `PERF_INDEX`, `PERF_CLUSTERS` | +| 1 Probe policy | `vector-ann-phase1` | open: policy in `query::scanner`, `--nprobes`, one Lance session per engine | +| 2 Build job | `vector-ann-phase2` | open: `lance_vector_index` in `weed-worker`, admin-driven end-to-end test, worker CI legs | +| 3 Keep it covering | `vector-ann-phase3` | open: relative budget, retrain rule, end-to-end steps | +| 4 Docs and gates | — | after the above merge | +| 5 Enterprise sharded builds | — | the Enterprise repository | + +What the measurements changed: + +- **IVF needs structure in the data.** On independent Gaussian vectors, the + perf suite's default, recall through an IVF index tracks the fraction of + partitions probed: at 50k × 384 with 224 partitions, 8 probes gave 0.13 and + 64 probes 0.56, and latency did not fall with fewer probes. On vectors drawn + around 100 shared centres, 8 probes gave recall 1.0; with eight clients + the index served 145 queries/s against 52 exact at 50k and 70 against 11 + at 200k, and the 10% filtered query at 200k fell from 445 to 265 ms, + while one client's latency stayed at the request path's floor on the + shared host. Embeddings cluster; the perf suite grows `PERF_CLUSTERS` + and the end-to-end test draws its rows around centres. The probe rule + stands as written. +- **The engine opened every table in a fresh Lance session**, so the index + cache died with each request. Phase 1 gives the engine one session, sized + by `--index-cache-mb`. +- **Compaction keeps the index covering** (Lance 10 remaps it), so the + compaction risk below did not materialise. + +## Where things stand (2026-11-04) + +What already exists, with the files that hold it: + +- **The engine is index-aware; nothing builds an index.** `QueryVectors` in + `crates/server/src/service.rs` always calls `weed_vector_core::query::ann`, + which runs a Lance scan with `use_index(true)`: ANN on fragments an index + covers, a flat pass on the rest, decided per fragment by Lance + (`crates/core/src/query.rs`). `nprobes` is `None`. `CreateIndex` writes an + empty Lance dataset and no vector index, so today every fragment is "tail" + and every query is an exact scan. +- **The maintenance worker only extends indexes that exist.** The + `lance_optimize_indices` job in `seaweed-worker/crates/lance/src/jobs/indices.rs` + skips a table whose `load_indices()` is empty and otherwise calls + `optimize_indices(&OptimizeOptions::default())` once `num_unindexed_rows` + exceeds its `max_unindexed_rows` budget (default 100,000). +- **Coverage and segments are modelled.** `crates/core/src/coverage.rs` reports, + per index name, the fragments covered and the uncovered rows; it requires the + fragment bitmap each Lance `IndexMetadata` delta carries. + `crates/core/src/segments.rs` lists physical segments (one per delta) with + their names and bitmaps, and partitions them across workers. `weed-vector-cli + coverage` and `weed-vector-cli query --flat` expose both. +- **Filters are applied before ranking.** `query.rs` sets `prefilter(true)` + unless the request asks for post-filtering, which S3 Vectors never does. + Lance's IVF search honours the prefilter, so this is already the behaviour + AWS calls an `ENHANCED` index. +- **Enterprise consumes exactly this model.** In the Enterprise repository, + `enterprise/weed/pb/vector_coord.proto` defines `SegmentInfo{id, + resident_bytes, rows, fragment_ids}` and `SegmentList{tail_fragment_ids, + tail_bytes}`; `VectorQuery` already carries `nprobes`; a `Task` is a set of + segment ids plus fragment ids. `sw-vector-distributed/src/plan.rs` assigns + a fragment covered by several segments to the lowest segment and never mixes + index names in one scan; `resident_bytes` is the index files' size. + `datasets.rs` warms a slot with `prewarm_index_segments` per index name and + warms the tail by scanning it. `sw-vector-distributed/src/index.rs` holds + `plan_index` (shard the uncovered fragments across workers), + `build_index_task` (one delta per shard, named + `-w-v`, built with `create_index_builder(..) + .fragments(..)`) and `optimize_policy` (append deltas until `MAX_DELTAS = 8`, + then Lance's default options, which promise no merge; see phase 3). + Nothing calls these three yet; `tests/index.rs`, + `tests/plan_search.rs` and `tests/engine_query.rs` exercise them with + `ivf_flat(4)` indexes. +- **Lance 10.0.0 offers** `VectorIndexParams::ivf_flat`, `ivf_pq`, + `with_ivf_sq_params`, `with_ivf_rq_params`, `ivf_hnsw` and HNSW variants; + scanner knobs `nprobes`, `minimum_nprobes`, `maximum_nprobes`, `ef`, + `refine`; `OptimizeOptions{num_indices_to_merge, index_names, retrain}`. +- **Measured today** (384-d, cosine, top-10, one 16-core host, exact scans): + one engine answers in 42 ms at 50k vectors and 82 ms at 200k; filtered + queries take 81 and 259–336 ms. Four Enterprise engines split four ways + bring 200k to 58 ms and the filtered query to 73 ms. Recall is 1.000. +- **Tests assume exactness only on tiny tables.** `test/s3vectors/s3vectors_test.py` + asserts exact result order on indexes of a handful of vectors; the perf + suite asserts recall@10 ≥ 0.9 against a client-side exact search. +- **AWS**: "90%+ average recall for most datasets"; `CLASSIC` and `ENHANCED` + index modes, the latter pre-filtering; no algorithm named. + +## Design + +### The contract every consumer relies on + +An S3 Vectors index gains a Lance vector index on its `vector` column, and +everything downstream keeps working because it only ever sees: + +1. **Segments**: Lance index deltas with fragment bitmaps. Every build and + every merge must leave `coverage_known == true`; a delta without a bitmap + turns the whole index into "rebuild" for the Enterprise planner. +2. **Names**: one logical index name per shard, never two shards under one + name within a build. Scans and warm-ups group by name. +3. **Tail**: fragments no segment covers, searched exactly. Rows written + after a build land here, so recall never drops to zero between builds. +4. **Bytes**: `resident_bytes` of a segment is its index files' size. The + controller's W sizing (`W_mem`, `W_scan`) and the engine's + `--capacity-bytes` budget use it unchanged. +5. **Versions**: a build or merge commits a new dataset version. Routers cache + segment lists per version, plans pin versions with the lease tags in + `crates/core/src/maintenance.rs`, and cleanup respects those leases. No + new mechanism is needed, but the build job must go through the same + commit path as any Lance write. + +### Index type and parameters + +- **IVF_FLAT first.** Distances inside probed partitions are exact, so recall + is governed by `nprobes` alone; the only training is k-means over the + centroids; resident bytes stay close to the raw vectors, which keeps the + Enterprise W sizing in the same range as today. `IVF_PQ` becomes an + operator option later for indexes whose raw size is the constraint. HNSW + variants are out of scope: their build cost and memory do not fit the + maintenance-worker model. +- **Metric from the table.** The vector field's `distance_metric` metadata + (`COSINE`, `DOT_PRODUCT`, else L2) is the metric the index is built with, + exactly as `service.rs` resolves it for queries. +- **Partitions**: start at `clamp(round(sqrt(rows)), 1, 4096)` for the whole + table, and for a shard, `sqrt` of the shard's rows. Phase 0 measures + recall and latency against this and adjusts before it becomes a default. +- **Probes**: one count per query, `max(8, ceil(partitions / 10))` clamped to + the partition count, taken from the largest delta of the index. A delta + with no more partitions than that count is searched in full; a larger one + at that count. Lance searches at least that many partitions. Under a + filter it may widen to four times the count until it has k rows, which a + selective filter needs; an open upper bound made Lance prepare every + partition for that and cost as much as the exact scan (201 ms against + 104 ms for the 10% filter at 50k × 384). Without a filter it stops at the + count. + +### Who builds, and when + +The Lance maintenance worker (`weed-worker`) gets a new job type, +`lance_vector_index`, beside `lance_optimize_indices`: + +- **Detection**: tables in a vector bucket's `vector` namespace (ids of the + form `$vector$`) whose vector column has no index and at + least `min_rows` live rows (default 50,000). Below the threshold the index + stays exact, which also keeps the existing small-table tests exact. +- **Execution**: one `create_index` call with IVF_FLAT, the table's metric, + the partition rule above, and a fixed name (`vector_idx`). Progress and + counters follow the pattern of `indices.rs`; a new `indexes_built_total` + metric and an admin observation of index coverage sit beside the row and + fragment counts the worker already reports. +- **Why the worker and not the engine or the gateway.** `CreateIndex` has no + rows to index. The engine serves queries and should not run minute-long + k-means builds. The worker already has the catalog client, credentials, + the admin UI settings form, metrics, retries and dedupe, and plugin jobs + start disabled until an operator turns them on, which is the rollout + switch we want. +- **Enterprise sharded builds** (phase 5): the same job, when Enterprise + engines are registered, splits the uncovered fragments with `plan_index` + and has each engine run `build_index_task` for its shard, producing one + named delta per shard. This is what `index.rs` was written for; the worker + drives it, the engines build, and the shared `optimize_policy` keeps each + name's delta count under eight. + +### Keeping it covering + +`lance_optimize_indices` already does the work once an index exists. Two +changes: + +- The budget becomes relative for vector tables: reindex when uncovered rows + exceed `min(absolute budget, 10% of indexed rows)`, so a 60k-row index + reindexes after 6k new rows instead of waiting for 100k. +- A retrain once the table has grown to about 4× the rows the centroids were + trained on, so partitions do not go stale. No row count is recorded: the + retrain is due when the partition rule for the current rows gives twice + the partitions the index has, and it rebuilds a single IVF_FLAT name + through the builder at that count. Past 2,048 partitions (about 4.2M + rows) the rule cannot double under the 4,096 cap, so those centroids are + not retrained again: that would need the trained row count recorded and a + full rebuild of a table that size, which is left for later. + +### Deletes and upserts + +A delete marks rows in Lance deletion files; an upsert is a delete plus an +append. Deleted rows are excluded through the prefilter on indexed and +unindexed fragments alike, and appended rows are tail until the next +optimize. Phase 1 adds a test that proves both, since this is where ANN +systems silently return stale hits. + +### API surface + +No new S3 Vectors API parameter is needed to reach AWS parity. Two optional +items for later: accept and echo AWS's `indexMode`/`queryMode` (both behave +as `ENHANCED` here, because filters are always applied first), and a +non-AWS per-request `nprobes` for the perf suite and the CLI, carried in the +engine's `QueryVectorsRequest` (the Enterprise `VectorQuery` already has the +field). + +### Enterprise compatibility requirements + +These hold for every phase, and the mono s3vectors legs (go/rust × +single/distributed) are the gate: + +- Builds produce deltas with fragment bitmaps; merges keep them. A test in + `tests/index.rs` asserts `coverage_known` after build, append and merge. +- Index names never mix within a shard set; the router and `plan.rs` group by + name, and `with_index_segments` rejects mixed names. +- Segment count per index name stays under `MAX_DELTAS`: at the limit + `optimize_policy(deltas)` passes `OptimizeOptions::merge(deltas)`, where + `deltas` is the name's delta count, so every delta of that name folds + into one with the new rows; Lance's default options may append another + delta instead of merging. A test runs repeated append-and-optimize cycles + and checks the actual segment counts. +- Coverage is measured as the union over an index's names. After a sharded + build each name misses the other shards' fragments, so a per-name count + would extend every shard over the whole table and duplicate the index. + An optimize extends one name only, the one with the fewest deltas; a test + checks that two disjoint shards stay disjoint through it. +- `resident_bytes` is the index files' size and `tail_bytes` the uncovered + fragments' size; the controller's `max_scan_mb` and memory sizing need no + change for IVF_FLAT, and PQ only makes slots smaller. +- A merge replaces segment UUIDs, which is a new placement epoch. The + existing make-before-break (target groups warm with + `prewarm_index_segments` while serving groups keep answering) covers it; + phase 5 measures how long a merged segment takes to warm and sets + expectations for `warmup_timeout`. +- Workers read index files the way they read data files, directly from the + filers and volume servers, and the files are write-once, + so the read cache and the warm path apply unchanged. +- Query semantics are identical in both modes: each engine applies the same + probe policy to the segments it owns, or the count the router passes in + `VectorQuery.nprobes`, plus a flat pass over its tail fragments; the + gateway merges the partial top-k lists. Each delta is probed on its own + centroids, so the merged answer is never worse than one engine's. +- The open-source `weed-vector` stays the writer and the fallback engine, and + it serves ANN through the same `query::ann` the workers call. + +## Phases + +### Phase 0 — Measure before deciding (small) + +- `weed-vector-cli index build --partitions N --name ..` on a local table, + reusing the code paths the tests already use, so builds can be tried + without the worker. +- Perf suite: `PERF_INDEX=1` builds the index after ingest and reports exact + versus ANN latency and recall@10 at 50k, 200k and 1M × 384, unfiltered and + with the 10% filter, for a grid of partitions and `nprobes`. +- Verify Lance behaviour we depend on: deletions excluded under ANN; + compaction (`lance_compact`) keeps bitmaps and coverage; `optimize_indices` + with `append` versus merge versus `retrain`. +- Exit: chosen defaults for `min_rows`, partitions and probes with numbers + behind them, and a written answer to each verification item. + +### Phase 1 — Engine query path (small) + +- Probe policy when an index exists: `minimum_nprobes` is the per-query + count from the Probes rule above, taken from the largest delta; + `maximum_nprobes` is four times it, which Lance uses only under a + prefilter. +- `nprobes` in `QueryVectorsRequest` for tests and the CLI; zero means the + policy. +- Per-query log line and counters: segments searched, tail rows scanned, + index hit or not. Without these, recall regressions are invisible. +- Tests in `crates/core`: ANN versus flat recall on a 50k synthetic table; + delete and upsert correctness under ANN; filtered query with a selective + filter still returns k results. + +### Phase 2 — Build job in the maintenance worker (medium) + +- `lance_vector_index` job: detection, execution, settings form (`min_rows`, + index type, partition rule), metrics, admin observation of coverage. +- Unit tests after the pattern of the other jobs; an integration test in + `test/s3vectors` that ingests above the threshold, runs the worker, and + asserts coverage through the CLI and recall ≥ 0.9 through the API. +- Exit: a fresh cluster with the job enabled indexes a 100k-vector table on + its own, and the small-table tests stay exact. + +### Phase 3 — Keep it covering (small) + +- Relative budget and `retrain` policy in `lance_optimize_indices` for vector + tables. The retrain rebuilds through the builder with partitions for the + current row count; Lance's `OptimizeOptions::retrain()` does not size them. +- Union-based coverage over an index's names, extension scoped to one name, + and `optimize_policy` with an explicit merge of all deltas at + `MAX_DELTAS`, shared with the Enterprise builds in the worker crate. +- Tests: append and optimize nine times and check the delta counts climb to + the limit and fold; two names on disjoint fragments stay disjoint through + an optimize; the end-to-end test runs the job after a tenth of the index + is appended and again after the table has grown fivefold. + +### Phase 4 — Gateway, docs and gates (small) + +- Optional `indexMode`/`queryMode` acceptance; `GetIndex` echoes the mode. +- Wiki and site docs: "Queries are exact" becomes "exact below the threshold, + ANN above it, with the tail exact", with the recall and latency measured + in phase 0. The performance page gets an ANN column. +- CI: a `test/s3vectors` leg with the worker and a low threshold, gated on + recall. + +### Phase 5 — Enterprise distributed builds and placement (medium) + +- Drive `plan_index` / `build_index_task` from the worker job when engines + are registered; replace the Enterprise copy of `optimize_policy` with the + shared one, which merges at the limit. +- Measure four engines on 1M × 384 with ANN: latency, warm time of a merged + segment, placement epochs per merge. +- mono s3vectors legs with the index enabled, go and rust volume servers, + single and distributed, plus the hardened TLS legs. + +## Risks + +- **Filtered recall.** AWS's weak spot is selective filters. Pre-filtering + keeps recall but costs probes; `minimum_nprobes`/`maximum_nprobes` is the + lever, and phase 0 has to show the latency is acceptable at 1M rows. +- **Memory.** IVF_FLAT resident bytes are about the raw vectors: 1M × 384 × + 4 bytes is 1.5 GB per copy. Enterprise spreads it by `capacity_bytes`; the + single open-source engine needs that much RAM or PQ. +- **Compaction and coverage.** Settled in phase 0: Lance 10 keeps the + bitmaps through `compact_files`, and coverage stays complete. +- **Build cost on the worker.** k-means over 1M × 384 is CPU-bound; execution + concurrency stays at one and the job reports progress so the admin UI shows + it working rather than hung. +- **Placement churn after merges.** Every merge is a new epoch; too frequent + merging makes engines re-warm constantly. `MAX_DELTAS = 8` bounds it; the + phase 5 measurement decides whether to raise it. + +## Out of scope + +HNSW indexes, GPU builds, per-request recall parameters in the AWS API +surface, and building an index at `CreateIndex` time.