weed-worker: build the vector index of S3 Vectors tables past a row threshold (ANN phase 2) (#12413)

* seaweed-vector: build and extend the vector index

weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector-cli: index build and index optimize

Logs go to stderr so stdout stays the JSON a caller parses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_INDEX builds the index before the timed queries

The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: probe a tenth of the index's partitions, more under a filter

A query that names no probe count gets one from the index's partition
count. Lance searches at least that many partitions, and under a prefilter
keeps going until it has k rows; without one it stops there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: --nprobes overrides the probe policy

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that a build takes the table's metric

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make clean removes weed-vector-cli

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: measure recall with vectors the table does not hold

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep the read cache's doc comment on with_read_cache

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold

Detection keeps the tables a vector bucket created, with at least min_rows
rows (50,000 by default) and no index on their vector column. Execution
builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: optimize only the indices on the vector column

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: resolve the query metric with index::distance_metric

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test the L2 metric under the name CreateIndex stores

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: build the index through the admin server and the Lance worker

VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new
test sets the job's threshold, runs it through the plugin API, checks
coverage and recall, and that a row written after the build is found.
CI gains the worker leg for both volume servers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: open a table inside a caller's Lance session

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep opened indices between queries in one session

Every open used a fresh Lance session, so the index and metadata caches
died with the request and each query loaded the IVF index again.
--index-cache-mb sizes the shared cache, 512 MiB by default.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the index test draws embedding-like vectors

Independent Gaussian vectors have no locality for an IVF index to use, so
recall through one only tracks the fraction of partitions probed; points
around shared centres behave like embeddings. The test must also run
before the one that stops the engine.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that compaction drops the deleted key from tied hits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test that an append-only optimize serves the new row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: record PERF_NOISE in the perf report config

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: a filtered query widens to four times the probes, not without bound

With no upper bound Lance prepares every partition for the late search
even when the minimum probes already hold k rows, and a filtered query
through the index cost as much as the exact scan. A query that names its
own probe count gets exactly that count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: report recall through an index without asserting it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a hit tying the k-th best score toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: lance_vector_index runs in the Lance lane

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: check a table's index before counting its rows

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make build and clean cover the worker leg's binaries

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: allow float32 rounding at the recall cutoff

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a repeated hit key once toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: assert every filtered hit passes the filter, indexed or not

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: forget the shared session when a table was dropped and created again

The session keys index lists and transactions by dataset version, and a
table created again at the same location restarts its versions. An open
that finds an older version than the newest seen at that URI, or the same
version committed later, replaces the session and opens again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: test an open after another engine created the table again

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: replace the session in the same lock that records a recreated table

A second open of the recreated table could see the new version recorded
and the old session still shared, and keep the dropped table's index list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: skip a vector table whose rows cannot be counted

Detection returned on the first table whose count_rows failed, so one
unreadable table kept every other table from being proposed. It now skips
that table like an open or index failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: panel for vector indexes the Lance worker built

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: size default probes from the largest index on the column

A sharded build gives the column one index name per shard. The first
name alone could be a small shard, and its probe count would then be
spent on a larger one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: skip the probe lookup when a query is given no segments

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: open every table in its own Lance session again

One session shared by every open keeps a table's index loaded between
queries, but Lance keys what it caches by location and version, so a
table dropped and created again at the same location could be served
the old table's index list. Telling the two apart from version and
commit time needs a map that only grows, a retry loop and clocks that
agree across engines. A private session per open, as before, has no
such state. Measured at 50k x 384 through an index, eight clients: 164
queries/s with the shared session, 137 without; one client is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: reject --nprobes 0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: read the partition count from the opened index

Lance's index statistics took 3.6 ms a query on a 200k-row table with
447 partitions. Opening the index and asking it takes 0.4 ms, and the
scan that follows finds the index already open.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* ci: give the S3 Vectors worker legs the vendored protoc

The worker workspace has no protoc of its own for lance's build
scripts, so the worker legs failed at their first build.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: give a vector index build an hour before the admin times it out

The job set no execution timeout, so the admin's default of 90 seconds
applied, and a build that trains on a large table would be reported as
failed and retried while it was still running.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: lance_vector_index proposes at most the admin's max_results

Every qualifying table went into one message however many there were.
The detector stops at the limit the request carries and sets has_more,
as the Go detectors do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a vector index job re-checks the table it was proposed for

A table emptied since detection completes without a build, and one
without a vector column fails instead of reporting it already indexed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a vector index run window as long as one build

An attempt is capped at the end of the job type's run window plus the
drain grace, and the default window is 30 minutes, so a build could be
cut off before its own hour was up. The window is an hour too, as the
Go iceberg handler sets both.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

weed/admin/plugin/scheduler_lane.go

@@ -98,6 +98,7 @@ var jobTypeLaneMap = map[string]SchedulerLane{
 	"lance_compact":          LaneLance,
 	"lance_optimize_indices": LaneLance,
 	"lance_cleanup_versions": LaneLance,
+	"lance_vector_index":     LaneLance,
 }
 
 // JobTypeLane returns the scheduler lane for the given job type.