From 47476aa7deeed3df0a6dcda049ef964783c0e932 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Thu, 5 Nov 2026 17:43:17 -0800 Subject: [PATCH] S3 Vector Bucket: building a vector index --- S3-Vector-Bucket.md | 70 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 69 insertions(+), 1 deletion(-) diff --git a/S3-Vector-Bucket.md b/S3-Vector-Bucket.md index ffbcf53..381f27b 100644 --- a/S3-Vector-Bucket.md +++ b/S3-Vector-Bucket.md @@ -49,7 +49,7 @@ The gateway serves all three bucket kinds on the S3 port. A vector bucket is a t * **The vector engine** is a separate process, `weed-vector`. It registers with the master, and the gateway finds it through the master on the first request that needs it. Bucket and policy operations work without it; index and vector operations need it. Several can register: the open-source gateway sends its requests to one at a time, and SeaweedFS Enterprise spreads each index and its queries over them (see [[S3 Vector Bucket Performance#beyond-one-engine]]). * **The data** is stored as [Lance](https://lancedb.com) datasets. A vector bucket is a table bucket with format `LANCE`, and each index is a Lance table in the bucket's `vector` namespace, stored under `s3:///vector//`. The engine reaches it through the [[SeaweedFS Lance Catalog]] (`-s3.port.lance`, default `9101`, on by default) and the S3 port. -**Queries are exact until a vector index is built.** Without one, every query compares against all vectors of the index, and recall@10 was 1.000 in every such run of the perf suite. The [[Lance Maintenance Worker]]'s `lance_vector_index` job builds an approximate (ANN) index, IVF, once an index holds at least `min_rows` vectors (50,000 by default). From then on a query searches a tenth of the index's partitions, at least 8, and up to four times that many when that leaves it short of `topK` results. It returns only what it finds in those partitions, so a filter that few vectors match can return fewer results than exist ([[S3 Vectors API]] has numbers). Vectors written after the build are still returned, and the worker's `lance_optimize_indices` job brings them into the index. Like every worker job, both start disabled. See [[S3 Vector Bucket Performance#through-a-vector-index]] for measurements. +**Queries are exact until a vector index is built.** Without one, every query compares against all vectors of the index, and recall@10 was 1.000 in every such run of the perf suite. The [[Lance Maintenance Worker]]'s `lance_vector_index` job builds an approximate (ANN) index, IVF, once an index holds at least `min_rows` vectors (50,000 by default). From then on a query searches a tenth of the index's partitions, at least 8, and up to four times that many when that leaves it short of `topK` results. It returns only what it finds in those partitions, so a filter that few vectors match can return fewer results than exist ([[S3 Vectors API]] has numbers). Vectors written after the build are still returned, and the worker's `lance_optimize_indices` job brings them into the index. Like every worker job, both start disabled. [Building a vector index](#building-a-vector-index) has the steps, and [[S3 Vector Bucket Performance#through-a-vector-index]] the measurements. ## Getting weed-vector @@ -259,6 +259,74 @@ neuromancer 0.1 {'genre': 'scifi', 'pages': 271, 'summary': 'Cyberspace heist.'} * **A standby engine.** The gateway sends every request to one registered engine at a time. Start a second engine (with its own `--listen`) and it takes over when the first stops: in our test, queries kept succeeding right after the engine in use was stopped. +## Building a vector index + +Without a vector index every query compares against all vectors of the index. The [[Lance Maintenance Worker]] (`weed-worker`) builds the vector index and keeps it covering new vectors; `weed-vector` uses it once it exists. The steps below continue the [quick start](#quick-start), with `weed mini`'s admin server on port `12535`. + +**1. Start the worker**, with the engine's identity: + +```bash +weed-worker --admin localhost:12535 --namespace http://localhost:9101 \ + --access-key admin --secret-key admin-secret +``` + +It logs `connected to admin`, and its job types appear in `curl http://localhost:12535/api/plugin/job-types`, among them `lance_vector_index` and `lance_optimize_indices`. + +**2. Turn the two jobs on**, in the admin UI at `/plugin` or through its API. They start disabled: + +```bash +for job in lance_vector_index lance_optimize_indices; do + curl -X PUT http://localhost:12535/api/plugin/job-types/$job/config \ + -H 'Content-Type: application/json' -d '{"admin_runtime": {"enabled": true}}' +done +``` + +From then on the admin server asks the worker for indexes that need work every 5 minutes. A second index, loaded with 50,000 vectors after the jobs were turned on, had its vector index within five minutes with no further command. + +**3. Or run a job now**, which also works while the job is disabled: + +```bash +curl -X POST http://localhost:12535/api/plugin/job-types/lance_vector_index/run +``` + +```json +{"canceled_count":0,"detected_count":1,"error_count":0,"executed_count":1,"job_type":"lance_vector_index","ready_to_execute_count":1,"skipped_active_count":0,"success_count":1} +``` + +`detected_count` is the number of indexes that needed a build and `success_count` the number built. The call waits for the builds, up to 5 minutes; `-d '{"timeout_seconds": 1800}'` raises that to the maximum of 30. An index of 50,000 vectors of 16 dimensions was built in half a second. A second run reports `"detected_count":0`: an index that has a vector index is left alone. + +**Which indexes get one.** Those holding at least `min_rows` vectors, 50,000 by default. To change it, send the setting with the same `PUT`: + +```bash +curl -X PUT http://localhost:12535/api/plugin/job-types/lance_vector_index/config \ + -H 'Content-Type: application/json' \ + -d '{"admin_runtime": {"enabled": true}, "worker_config_values": {"min_rows": {"int64_value": "10000"}}}' +``` + +**Vectors written afterwards** are returned by queries right away, and `lance_optimize_indices` adds them to the vector index once they amount to more than a tenth of the vectors already in it. After 6,000 vectors were added to the 50,000 above, a run of that job reported `"detected_count":1,"success_count":1`, and none were left outside the vector index. See [[Lance Maintenance Worker#the-jobs]] for the rules and settings. + +**Checking the result.** `weed-vector-cli`, built from source (`cargo build --release --locked -p weed-vector-core --bin weed-vector-cli` inside `seaweed-vector/`), prints what the vector index of `movies` in bucket `media` covers: + +```bash +weed-vector-cli --access-key admin --secret-key admin-secret coverage 'media$vector$movies' +``` + +```json +[ + { + "index_name": "vector_idx", + "segments": 1, + "partitions": 224, + "coverage_known": true, + "indexed_fragments": [0, 1, 2, ...], + "unindexed_fragments": [], + "unindexed_rows": 0 + } +] +``` + +An empty list means no vector index has been built. `unindexed_rows` counts the vectors written since the last build or optimize run. + ## Limits | What | Limit |