S3 Vector Bucket: building a vector index

S3-Vector-Bucket.md

@@ -49,7 +49,7 @@ The gateway serves all three bucket kinds on the S3 port. A vector bucket is a t
 * **The vector engine** is a separate process, `weed-vector`. It registers with the master, and the gateway finds it through the master on the first request that needs it. Bucket and policy operations work without it; index and vector operations need it. Several can register: the open-source gateway sends its requests to one at a time, and SeaweedFS Enterprise spreads each index and its queries over them (see [[S3 Vector Bucket Performance#beyond-one-engine]]).
 * **The data** is stored as [Lance](https://lancedb.com) datasets. A vector bucket is a table bucket with format `LANCE`, and each index is a Lance table in the bucket's `vector` namespace, stored under `s3://<bucket>/vector/<index>/`. The engine reaches it through the [[SeaweedFS Lance Catalog]] (`-s3.port.lance`, default `9101`, on by default) and the S3 port.
 
-**Queries are exact until a vector index is built.** Without one, every query compares against all vectors of the index, and recall@10 was 1.000 in every such run of the perf suite. The [[Lance Maintenance Worker]]'s `lance_vector_index` job builds an approximate (ANN) index, IVF, once an index holds at least `min_rows` vectors (50,000 by default). From then on a query searches a tenth of the index's partitions, at least 8, and up to four times that many when that leaves it short of `topK` results. It returns only what it finds in those partitions, so a filter that few vectors match can return fewer results than exist ([[S3 Vectors API]] has numbers). Vectors written after the build are still returned, and the worker's `lance_optimize_indices` job brings them into the index. Like every worker job, both start disabled. See [[S3 Vector Bucket Performance#through-a-vector-index]] for measurements.
+**Queries are exact until a vector index is built.** Without one, every query compares against all vectors of the index, and recall@10 was 1.000 in every such run of the perf suite. The [[Lance Maintenance Worker]]'s `lance_vector_index` job builds an approximate (ANN) index, IVF, once an index holds at least `min_rows` vectors (50,000 by default). From then on a query searches a tenth of the index's partitions, at least 8, and up to four times that many when that leaves it short of `topK` results. It returns only what it finds in those partitions, so a filter that few vectors match can return fewer results than exist ([[S3 Vectors API]] has numbers). Vectors written after the build are still returned, and the worker's `lance_optimize_indices` job brings them into the index. Like every worker job, both start disabled. [Building a vector index](#building-a-vector-index) has the steps, and [[S3 Vector Bucket Performance#through-a-vector-index]] the measurements.
 
 ## Getting weed-vector
 
@@ -259,6 +259,74 @@ neuromancer 0.1 {'genre': 'scifi', 'pages': 271, 'summary': 'Cyberspace heist.'}
 
 * **A standby engine.** The gateway sends every request to one registered engine at a time. Start a second engine (with its own `--listen`) and it takes over when the first stops: in our test, queries kept succeeding right after the engine in use was stopped.
 
+## Building a vector index
+
+Without a vector index every query compares against all vectors of the index. The [[Lance Maintenance Worker]] (`weed-worker`) builds the vector index and keeps it covering new vectors; `weed-vector` uses it once it exists. The steps below continue the [quick start](#quick-start), with `weed mini`'s admin server on port `12535`.
+
+**1. Start the worker**, with the engine's identity:
+
+```bash
+weed-worker --admin localhost:12535 --namespace http://localhost:9101 \
+  --access-key admin --secret-key admin-secret
+```
+
+It logs `connected to admin`, and its job types appear in `curl http://localhost:12535/api/plugin/job-types`, among them `lance_vector_index` and `lance_optimize_indices`.
+
+**2. Turn the two jobs on**, in the admin UI at `/plugin` or through its API. They start disabled:
+
+```bash
+for job in lance_vector_index lance_optimize_indices; do
+  curl -X PUT http://localhost:12535/api/plugin/job-types/$job/config \
+    -H 'Content-Type: application/json' -d '{"admin_runtime": {"enabled": true}}'
+done
+```
+
+From then on the admin server asks the worker for indexes that need work every 5 minutes. A second index, loaded with 50,000 vectors after the jobs were turned on, had its vector index within five minutes with no further command.
+
+**3. Or run a job now**, which also works while the job is disabled:
+
+```bash
+curl -X POST http://localhost:12535/api/plugin/job-types/lance_vector_index/run
+```
+
+```json
+{"canceled_count":0,"detected_count":1,"error_count":0,"executed_count":1,"job_type":"lance_vector_index","ready_to_execute_count":1,"skipped_active_count":0,"success_count":1}
+```
+
+`detected_count` is the number of indexes that needed a build and `success_count` the number built. The call waits for the builds, up to 5 minutes; `-d '{"timeout_seconds": 1800}'` raises that to the maximum of 30. An index of 50,000 vectors of 16 dimensions was built in half a second. A second run reports `"detected_count":0`: an index that has a vector index is left alone.
+
+**Which indexes get one.** Those holding at least `min_rows` vectors, 50,000 by default. To change it, send the setting with the same `PUT`:
+
+```bash
+curl -X PUT http://localhost:12535/api/plugin/job-types/lance_vector_index/config \
+  -H 'Content-Type: application/json' \
+  -d '{"admin_runtime": {"enabled": true}, "worker_config_values": {"min_rows": {"int64_value": "10000"}}}'
+```
+
+**Vectors written afterwards** are returned by queries right away, and `lance_optimize_indices` adds them to the vector index once they amount to more than a tenth of the vectors already in it. After 6,000 vectors were added to the 50,000 above, a run of that job reported `"detected_count":1,"success_count":1`, and none were left outside the vector index. See [[Lance Maintenance Worker#the-jobs]] for the rules and settings.
+
+**Checking the result.** `weed-vector-cli`, built from source (`cargo build --release --locked -p weed-vector-core --bin weed-vector-cli` inside `seaweed-vector/`), prints what the vector index of `movies` in bucket `media` covers:
+
+```bash
+weed-vector-cli --access-key admin --secret-key admin-secret coverage 'media$vector$movies'
+```
+
+```json
+[
+  {
+    "index_name": "vector_idx",
+    "segments": 1,
+    "partitions": 224,
+    "coverage_known": true,
+    "indexed_fragments": [0, 1, 2, ...],
+    "unindexed_fragments": [],
+    "unindexed_rows": 0
+  }
+]
+```
+
+An empty list means no vector index has been built. `unindexed_rows` counts the vectors written since the last build or optimize run.
+
 ## Limits
 
 | What | Limit |