seaweed-vector: dev plan for ANN search over S3 Vectors indexes (#12408)

* seaweed-vector: dev plan for ANN search over S3 Vectors indexes

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: one probe count per query, a budget that can shrink

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: smaller deltas are searched in full only up to the probe count

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: drop the private issue reference

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: phase 1 probes use the Design count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: status after phases 0 to 3, and what the measurements changed

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: name the Enterprise repository, not the private one

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: the Enterprise repository by that name

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: a filtered query widens to four times the probes, not without bound

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: throughput through the index at 50k and 200k

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: union coverage over shard names, and an explicit merge at the delta limit

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: status table names branches, not pull requests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: maximum_nprobes is always four times the count

Lance reads maximum_nprobes only under a prefilter, and the phase 1
policy sets it the same way for every query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: the retrain follows the partition count, not a recorded row count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: phase 3 is open like the other phases

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: the retrain stops past 2,048 partitions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: the merge at the delta limit takes the name's delta count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

seaweed-vector/DEV_PLAN.md

@@ -0,0 +1,339 @@
+# S3 Vectors ANN — Dev Plan
+
+## Goal
+
+Approximate nearest-neighbour search for S3 Vectors indexes, built on Lance
+vector indexes, so that query latency stops growing linearly with the index.
+Target: 90%+ recall@10 (what AWS states for S3 Vectors), exact results below
+a size threshold and for rows written since the last index build, and one
+code path that serves both the open-source single engine and the Enterprise
+distributed query layer. Writes stay unchanged.
+
+## Status (2026-11-04)
+
+| Phase | Branch | State |
+|---|---|---|
+| 0 Build and measure | `vector-ann-phase0` | open: `vector_index` builder, `weed-vector-cli index`, `PERF_INDEX`, `PERF_CLUSTERS` |
+| 1 Probe policy | `vector-ann-phase1` | open: policy in `query::scanner`, `--nprobes`, one Lance session per engine |
+| 2 Build job | `vector-ann-phase2` | open: `lance_vector_index` in `weed-worker`, admin-driven end-to-end test, worker CI legs |
+| 3 Keep it covering | `vector-ann-phase3` | open: relative budget, retrain rule, end-to-end steps |
+| 4 Docs and gates | — | after the above merge |
+| 5 Enterprise sharded builds | — | the Enterprise repository |
+
+What the measurements changed:
+
+- **IVF needs structure in the data.** On independent Gaussian vectors, the
+  perf suite's default, recall through an IVF index tracks the fraction of
+  partitions probed: at 50k × 384 with 224 partitions, 8 probes gave 0.13 and
+  64 probes 0.56, and latency did not fall with fewer probes. On vectors drawn
+  around 100 shared centres, 8 probes gave recall 1.0; with eight clients
+  the index served 145 queries/s against 52 exact at 50k and 70 against 11
+  at 200k, and the 10% filtered query at 200k fell from 445 to 265 ms,
+  while one client's latency stayed at the request path's floor on the
+  shared host. Embeddings cluster; the perf suite grows `PERF_CLUSTERS`
+  and the end-to-end test draws its rows around centres. The probe rule
+  stands as written.
+- **The engine opened every table in a fresh Lance session**, so the index
+  cache died with each request. Phase 1 gives the engine one session, sized
+  by `--index-cache-mb`.
+- **Compaction keeps the index covering** (Lance 10 remaps it), so the
+  compaction risk below did not materialise.
+
+## Where things stand (2026-11-04)
+
+What already exists, with the files that hold it:
+
+- **The engine is index-aware; nothing builds an index.** `QueryVectors` in
+  `crates/server/src/service.rs` always calls `weed_vector_core::query::ann`,
+  which runs a Lance scan with `use_index(true)`: ANN on fragments an index
+  covers, a flat pass on the rest, decided per fragment by Lance
+  (`crates/core/src/query.rs`). `nprobes` is `None`. `CreateIndex` writes an
+  empty Lance dataset and no vector index, so today every fragment is "tail"
+  and every query is an exact scan.
+- **The maintenance worker only extends indexes that exist.** The
+  `lance_optimize_indices` job in `seaweed-worker/crates/lance/src/jobs/indices.rs`
+  skips a table whose `load_indices()` is empty and otherwise calls
+  `optimize_indices(&OptimizeOptions::default())` once `num_unindexed_rows`
+  exceeds its `max_unindexed_rows` budget (default 100,000).
+- **Coverage and segments are modelled.** `crates/core/src/coverage.rs` reports,
+  per index name, the fragments covered and the uncovered rows; it requires the
+  fragment bitmap each Lance `IndexMetadata` delta carries.
+  `crates/core/src/segments.rs` lists physical segments (one per delta) with
+  their names and bitmaps, and partitions them across workers. `weed-vector-cli
+  coverage` and `weed-vector-cli query --flat` expose both.
+- **Filters are applied before ranking.** `query.rs` sets `prefilter(true)`
+  unless the request asks for post-filtering, which S3 Vectors never does.
+  Lance's IVF search honours the prefilter, so this is already the behaviour
+  AWS calls an `ENHANCED` index.
+- **Enterprise consumes exactly this model.** In the Enterprise repository,
+  `enterprise/weed/pb/vector_coord.proto` defines `SegmentInfo{id,
+  resident_bytes, rows, fragment_ids}` and `SegmentList{tail_fragment_ids,
+  tail_bytes}`; `VectorQuery` already carries `nprobes`; a `Task` is a set of
+  segment ids plus fragment ids. `sw-vector-distributed/src/plan.rs` assigns
+  a fragment covered by several segments to the lowest segment and never mixes
+  index names in one scan; `resident_bytes` is the index files' size.
+  `datasets.rs` warms a slot with `prewarm_index_segments` per index name and
+  warms the tail by scanning it. `sw-vector-distributed/src/index.rs` holds
+  `plan_index` (shard the uncovered fragments across workers),
+  `build_index_task` (one delta per shard, named
+  `<column>-w<worker>-v<version>`, built with `create_index_builder(..)
+  .fragments(..)`) and `optimize_policy` (append deltas until `MAX_DELTAS = 8`,
+  then Lance's default options, which promise no merge; see phase 3).
+  Nothing calls these three yet; `tests/index.rs`,
+  `tests/plan_search.rs` and `tests/engine_query.rs` exercise them with
+  `ivf_flat(4)` indexes.
+- **Lance 10.0.0 offers** `VectorIndexParams::ivf_flat`, `ivf_pq`,
+  `with_ivf_sq_params`, `with_ivf_rq_params`, `ivf_hnsw` and HNSW variants;
+  scanner knobs `nprobes`, `minimum_nprobes`, `maximum_nprobes`, `ef`,
+  `refine`; `OptimizeOptions{num_indices_to_merge, index_names, retrain}`.
+- **Measured today** (384-d, cosine, top-10, one 16-core host, exact scans):
+  one engine answers in 42 ms at 50k vectors and 82 ms at 200k; filtered
+  queries take 81 and 259–336 ms. Four Enterprise engines split four ways
+  bring 200k to 58 ms and the filtered query to 73 ms. Recall is 1.000.
+- **Tests assume exactness only on tiny tables.** `test/s3vectors/s3vectors_test.py`
+  asserts exact result order on indexes of a handful of vectors; the perf
+  suite asserts recall@10 ≥ 0.9 against a client-side exact search.
+- **AWS**: "90%+ average recall for most datasets"; `CLASSIC` and `ENHANCED`
+  index modes, the latter pre-filtering; no algorithm named.
+
+## Design
+
+### The contract every consumer relies on
+
+An S3 Vectors index gains a Lance vector index on its `vector` column, and
+everything downstream keeps working because it only ever sees:
+
+1. **Segments**: Lance index deltas with fragment bitmaps. Every build and
+   every merge must leave `coverage_known == true`; a delta without a bitmap
+   turns the whole index into "rebuild" for the Enterprise planner.
+2. **Names**: one logical index name per shard, never two shards under one
+   name within a build. Scans and warm-ups group by name.
+3. **Tail**: fragments no segment covers, searched exactly. Rows written
+   after a build land here, so recall never drops to zero between builds.
+4. **Bytes**: `resident_bytes` of a segment is its index files' size. The
+   controller's W sizing (`W_mem`, `W_scan`) and the engine's
+   `--capacity-bytes` budget use it unchanged.
+5. **Versions**: a build or merge commits a new dataset version. Routers cache
+   segment lists per version, plans pin versions with the lease tags in
+   `crates/core/src/maintenance.rs`, and cleanup respects those leases. No
+   new mechanism is needed, but the build job must go through the same
+   commit path as any Lance write.
+
+### Index type and parameters
+
+- **IVF_FLAT first.** Distances inside probed partitions are exact, so recall
+  is governed by `nprobes` alone; the only training is k-means over the
+  centroids; resident bytes stay close to the raw vectors, which keeps the
+  Enterprise W sizing in the same range as today. `IVF_PQ` becomes an
+  operator option later for indexes whose raw size is the constraint. HNSW
+  variants are out of scope: their build cost and memory do not fit the
+  maintenance-worker model.
+- **Metric from the table.** The vector field's `distance_metric` metadata
+  (`COSINE`, `DOT_PRODUCT`, else L2) is the metric the index is built with,
+  exactly as `service.rs` resolves it for queries.
+- **Partitions**: start at `clamp(round(sqrt(rows)), 1, 4096)` for the whole
+  table, and for a shard, `sqrt` of the shard's rows. Phase 0 measures
+  recall and latency against this and adjusts before it becomes a default.
+- **Probes**: one count per query, `max(8, ceil(partitions / 10))` clamped to
+  the partition count, taken from the largest delta of the index. A delta
+  with no more partitions than that count is searched in full; a larger one
+  at that count. Lance searches at least that many partitions. Under a
+  filter it may widen to four times the count until it has k rows, which a
+  selective filter needs; an open upper bound made Lance prepare every
+  partition for that and cost as much as the exact scan (201 ms against
+  104 ms for the 10% filter at 50k × 384). Without a filter it stops at the
+  count.
+
+### Who builds, and when
+
+The Lance maintenance worker (`weed-worker`) gets a new job type,
+`lance_vector_index`, beside `lance_optimize_indices`:
+
+- **Detection**: tables in a vector bucket's `vector` namespace (ids of the
+  form `<bucket>$vector$<index>`) whose vector column has no index and at
+  least `min_rows` live rows (default 50,000). Below the threshold the index
+  stays exact, which also keeps the existing small-table tests exact.
+- **Execution**: one `create_index` call with IVF_FLAT, the table's metric,
+  the partition rule above, and a fixed name (`vector_idx`). Progress and
+  counters follow the pattern of `indices.rs`; a new `indexes_built_total`
+  metric and an admin observation of index coverage sit beside the row and
+  fragment counts the worker already reports.
+- **Why the worker and not the engine or the gateway.** `CreateIndex` has no
+  rows to index. The engine serves queries and should not run minute-long
+  k-means builds. The worker already has the catalog client, credentials,
+  the admin UI settings form, metrics, retries and dedupe, and plugin jobs
+  start disabled until an operator turns them on, which is the rollout
+  switch we want.
+- **Enterprise sharded builds** (phase 5): the same job, when Enterprise
+  engines are registered, splits the uncovered fragments with `plan_index`
+  and has each engine run `build_index_task` for its shard, producing one
+  named delta per shard. This is what `index.rs` was written for; the worker
+  drives it, the engines build, and the shared `optimize_policy` keeps each
+  name's delta count under eight.
+
+### Keeping it covering
+
+`lance_optimize_indices` already does the work once an index exists. Two
+changes:
+
+- The budget becomes relative for vector tables: reindex when uncovered rows
+  exceed `min(absolute budget, 10% of indexed rows)`, so a 60k-row index
+  reindexes after 6k new rows instead of waiting for 100k.
+- A retrain once the table has grown to about 4× the rows the centroids were
+  trained on, so partitions do not go stale. No row count is recorded: the
+  retrain is due when the partition rule for the current rows gives twice
+  the partitions the index has, and it rebuilds a single IVF_FLAT name
+  through the builder at that count. Past 2,048 partitions (about 4.2M
+  rows) the rule cannot double under the 4,096 cap, so those centroids are
+  not retrained again: that would need the trained row count recorded and a
+  full rebuild of a table that size, which is left for later.
+
+### Deletes and upserts
+
+A delete marks rows in Lance deletion files; an upsert is a delete plus an
+append. Deleted rows are excluded through the prefilter on indexed and
+unindexed fragments alike, and appended rows are tail until the next
+optimize. Phase 1 adds a test that proves both, since this is where ANN
+systems silently return stale hits.
+
+### API surface
+
+No new S3 Vectors API parameter is needed to reach AWS parity. Two optional
+items for later: accept and echo AWS's `indexMode`/`queryMode` (both behave
+as `ENHANCED` here, because filters are always applied first), and a
+non-AWS per-request `nprobes` for the perf suite and the CLI, carried in the
+engine's `QueryVectorsRequest` (the Enterprise `VectorQuery` already has the
+field).
+
+### Enterprise compatibility requirements
+
+These hold for every phase, and the mono s3vectors legs (go/rust ×
+single/distributed) are the gate:
+
+- Builds produce deltas with fragment bitmaps; merges keep them. A test in
+  `tests/index.rs` asserts `coverage_known` after build, append and merge.
+- Index names never mix within a shard set; the router and `plan.rs` group by
+  name, and `with_index_segments` rejects mixed names.
+- Segment count per index name stays under `MAX_DELTAS`: at the limit
+  `optimize_policy(deltas)` passes `OptimizeOptions::merge(deltas)`, where
+  `deltas` is the name's delta count, so every delta of that name folds
+  into one with the new rows; Lance's default options may append another
+  delta instead of merging. A test runs repeated append-and-optimize cycles
+  and checks the actual segment counts.
+- Coverage is measured as the union over an index's names. After a sharded
+  build each name misses the other shards' fragments, so a per-name count
+  would extend every shard over the whole table and duplicate the index.
+  An optimize extends one name only, the one with the fewest deltas; a test
+  checks that two disjoint shards stay disjoint through it.
+- `resident_bytes` is the index files' size and `tail_bytes` the uncovered
+  fragments' size; the controller's `max_scan_mb` and memory sizing need no
+  change for IVF_FLAT, and PQ only makes slots smaller.
+- A merge replaces segment UUIDs, which is a new placement epoch. The
+  existing make-before-break (target groups warm with
+  `prewarm_index_segments` while serving groups keep answering) covers it;
+  phase 5 measures how long a merged segment takes to warm and sets
+  expectations for `warmup_timeout`.
+- Workers read index files the way they read data files, directly from the
+  filers and volume servers, and the files are write-once,
+  so the read cache and the warm path apply unchanged.
+- Query semantics are identical in both modes: each engine applies the same
+  probe policy to the segments it owns, or the count the router passes in
+  `VectorQuery.nprobes`, plus a flat pass over its tail fragments; the
+  gateway merges the partial top-k lists. Each delta is probed on its own
+  centroids, so the merged answer is never worse than one engine's.
+- The open-source `weed-vector` stays the writer and the fallback engine, and
+  it serves ANN through the same `query::ann` the workers call.
+
+## Phases
+
+### Phase 0 — Measure before deciding (small)
+
+- `weed-vector-cli index build --partitions N --name ..` on a local table,
+  reusing the code paths the tests already use, so builds can be tried
+  without the worker.
+- Perf suite: `PERF_INDEX=1` builds the index after ingest and reports exact
+  versus ANN latency and recall@10 at 50k, 200k and 1M × 384, unfiltered and
+  with the 10% filter, for a grid of partitions and `nprobes`.
+- Verify Lance behaviour we depend on: deletions excluded under ANN;
+  compaction (`lance_compact`) keeps bitmaps and coverage; `optimize_indices`
+  with `append` versus merge versus `retrain`.
+- Exit: chosen defaults for `min_rows`, partitions and probes with numbers
+  behind them, and a written answer to each verification item.
+
+### Phase 1 — Engine query path (small)
+
+- Probe policy when an index exists: `minimum_nprobes` is the per-query
+  count from the Probes rule above, taken from the largest delta;
+  `maximum_nprobes` is four times it, which Lance uses only under a
+  prefilter.
+- `nprobes` in `QueryVectorsRequest` for tests and the CLI; zero means the
+  policy.
+- Per-query log line and counters: segments searched, tail rows scanned,
+  index hit or not. Without these, recall regressions are invisible.
+- Tests in `crates/core`: ANN versus flat recall on a 50k synthetic table;
+  delete and upsert correctness under ANN; filtered query with a selective
+  filter still returns k results.
+
+### Phase 2 — Build job in the maintenance worker (medium)
+
+- `lance_vector_index` job: detection, execution, settings form (`min_rows`,
+  index type, partition rule), metrics, admin observation of coverage.
+- Unit tests after the pattern of the other jobs; an integration test in
+  `test/s3vectors` that ingests above the threshold, runs the worker, and
+  asserts coverage through the CLI and recall ≥ 0.9 through the API.
+- Exit: a fresh cluster with the job enabled indexes a 100k-vector table on
+  its own, and the small-table tests stay exact.
+
+### Phase 3 — Keep it covering (small)
+
+- Relative budget and `retrain` policy in `lance_optimize_indices` for vector
+  tables. The retrain rebuilds through the builder with partitions for the
+  current row count; Lance's `OptimizeOptions::retrain()` does not size them.
+- Union-based coverage over an index's names, extension scoped to one name,
+  and `optimize_policy` with an explicit merge of all deltas at
+  `MAX_DELTAS`, shared with the Enterprise builds in the worker crate.
+- Tests: append and optimize nine times and check the delta counts climb to
+  the limit and fold; two names on disjoint fragments stay disjoint through
+  an optimize; the end-to-end test runs the job after a tenth of the index
+  is appended and again after the table has grown fivefold.
+
+### Phase 4 — Gateway, docs and gates (small)
+
+- Optional `indexMode`/`queryMode` acceptance; `GetIndex` echoes the mode.
+- Wiki and site docs: "Queries are exact" becomes "exact below the threshold,
+  ANN above it, with the tail exact", with the recall and latency measured
+  in phase 0. The performance page gets an ANN column.
+- CI: a `test/s3vectors` leg with the worker and a low threshold, gated on
+  recall.
+
+### Phase 5 — Enterprise distributed builds and placement (medium)
+
+- Drive `plan_index` / `build_index_task` from the worker job when engines
+  are registered; replace the Enterprise copy of `optimize_policy` with the
+  shared one, which merges at the limit.
+- Measure four engines on 1M × 384 with ANN: latency, warm time of a merged
+  segment, placement epochs per merge.
+- mono s3vectors legs with the index enabled, go and rust volume servers,
+  single and distributed, plus the hardened TLS legs.
+
+## Risks
+
+- **Filtered recall.** AWS's weak spot is selective filters. Pre-filtering
+  keeps recall but costs probes; `minimum_nprobes`/`maximum_nprobes` is the
+  lever, and phase 0 has to show the latency is acceptable at 1M rows.
+- **Memory.** IVF_FLAT resident bytes are about the raw vectors: 1M × 384 ×
+  4 bytes is 1.5 GB per copy. Enterprise spreads it by `capacity_bytes`; the
+  single open-source engine needs that much RAM or PQ.
+- **Compaction and coverage.** Settled in phase 0: Lance 10 keeps the
+  bitmaps through `compact_files`, and coverage stays complete.
+- **Build cost on the worker.** k-means over 1M × 384 is CPU-bound; execution
+  concurrency stays at one and the job reports progress so the admin UI shows
+  it working rather than hung.
+- **Placement churn after merges.** Every merge is a new epoch; too frequent
+  merging makes engines re-warm constantly. `MAX_DELTAS = 8` bounds it; the
+  phase 5 measurement decides whether to raise it.
+
+## Out of scope
+
+HNSW indexes, GPU builds, per-request recall parameters in the AWS API
+surface, and building an index at `CreateIndex` time.