diff --git a/Lance-Maintenance-Worker.md b/Lance-Maintenance-Worker.md
index 5f8f3b1..a1db6ce 100644
--- a/Lance-Maintenance-Worker.md
+++ b/Lance-Maintenance-Worker.md
@@ -79,10 +79,13 @@ If detection reports nothing at all, credentials are the first thing to check: a
| Job type | What it does | Detected from |
|----------|--------------|---------------|
| `lance_compact` | merges small fragments | fragment count |
+| `lance_vector_index` | builds the vector index of an S3 Vectors table that has none | row count |
| `lance_optimize_indices` | extends indices to cover rows written after they were built | rows no index covers |
| `lance_cleanup_versions` | removes old versions and the files only they referenced | version count and age |
-**`lance_optimize_indices` is the one that matters most**, and has no Iceberg equivalent. Rows written after an index was built are not covered by it, so a vector search silently misses them. It is a correctness problem wearing a performance problem's clothes.
+**`lance_optimize_indices` is the one that matters most**, and has no Iceberg equivalent. Rows written after an index was built are not covered by it. A query through the [[S3 Vector Bucket]] API still returns them, because rows outside the index are searched without it; this job is what brings them into the index.
+
+**S3 Vectors tables** (the tables in a vector bucket's `vector` namespace, one per index) get two more rules. `lance_vector_index` builds a table's IVF index once it holds `min_rows` rows and has no index. `lance_optimize_indices` then extends that index when the uncovered rows exceed a tenth of the indexed rows or `max_unindexed_rows`, whichever is smaller, and trains it again, with partitions for the current row count, once the table has grown to four times the rows it was sized for. Both jobs may run for up to an hour.
Each job type has a settings form in the admin UI, rendered from the worker's descriptor:
@@ -90,6 +93,7 @@ Each job type has a settings form in the admin UI, rendered from the worker's de
|---------|-----|---------|
| `min_fragments` | compact | 8 |
| `target_rows_per_fragment` | compact | 1048576 |
+| `min_rows` | vector index | 50000 |
| `max_unindexed_rows` | optimize indices | 100000 |
| `retain_hours` | cleanup | 168 |
| `min_versions_to_keep` | cleanup | 5 |
diff --git a/S3-Vector-Bucket-Performance.md b/S3-Vector-Bucket-Performance.md
index 99cc33f..5956144 100644
--- a/S3-Vector-Bucket-Performance.md
+++ b/S3-Vector-Bucket-Performance.md
@@ -4,12 +4,12 @@ Measured numbers for [[S3 Vector Bucket]]s, and how to reproduce them with the p
## The perf suite
-`test/s3vectors/perf_test.py` starts a local cluster (master, volume server, filer, S3 gateway and `weed-vector`), loads random vectors through the S3 Vectors API and times every data-plane operation. It checks correctness, not speed: recall is checked against an exact search done in the client, and every listing must return every vector.
+`test/s3vectors/perf_test.py` starts a local cluster (master, volume server, filer, S3 gateway and `weed-vector`), loads random vectors through the S3 Vectors API and times every data-plane operation. It checks correctness, not speed: recall is checked against an exact search done in the client (with `PERF_INDEX` it is reported, not asserted), and every listing must return every vector.
```bash
cd test/s3vectors
pip install -r requirements.txt # boto3, pytest
-make perf # builds weed, weed-vector and weed-volume, then runs the suite
+make perf # builds weed, weed-vector, weed-vector-cli and weed-volume, then runs the suite
# or, with the binaries already in this directory:
VOLUME_SERVER_IMPL=go ./run_perf.sh
```
@@ -25,6 +25,10 @@ The Python packages are `boto3` 1.40 or newer and `pytest` 8 or newer. `make per
| `PERF_TOPK` | 10 | `topK` |
| `PERF_CONCURRENCY` | `1,4,8` | client threads for the throughput scenario |
| `PERF_RECALL` | 20 | queries checked against the exact search |
+| `PERF_INDEX` | unset | `1` builds the vector index after the load, so the queries run through it |
+| `PERF_PARTITIONS` | the square root of the vector count | partitions of that index |
+| `PERF_CLUSTERS` | 0 | draw the vectors around this many shared centres; `0` draws each one independently |
+| `PERF_NOISE` | 0.3 | spread of the vectors around their centre |
| `PERF_REPORT` | `perf-results.json` in the cluster directory | where the JSON report goes |
| `VOLUME_SERVER_IMPL` | `go,rust` | which volume server(s) to run against |
@@ -50,15 +54,45 @@ One 16-core x86_64 host, every process on it, `cosine` metric, top-10, one clien
What the numbers show:
-* **Queries scan every vector.** No approximate index is built, so latency grows with the size of the index: going from 10k × 128 to 50k × 384 (15 times the data) raised the median query from about 22 ms to about 55 ms. Recall is 1.000 throughout.
+* **Queries scan every vector.** These runs built no vector index (see [Through a vector index](#through-a-vector-index) for runs that did), so latency grows with the size of the index: going from 10k × 128 to 50k × 384 (15 times the data) raised the median query from about 22 ms to about 55 ms. Recall is 1.000 throughout.
* **Throughput grows with clients until the index gets large.** At 10k × 128, 8 clients reach about 285 queries/s; at 50k × 384 throughput levels off between 4 and 8 clients at 55 to 65 queries/s.
* **Parallel listing scales.** `ListVectors` with 16 segments read the whole index 5 to 11 times faster than one sequential listing.
* **Ingest is steady.** Batch latency did not grow from the first to the last tenth of the load (for example 283 ms → 272 ms at 50k × 384).
* **No consistent difference between the volume servers.** The largest gap, the filtered query at 50k × 384 (82 ms on Go, 45 ms on Rust), goes the other way at 10k × 128 (40 ms against 51 ms).
+## Through a vector index
+
+The runs above have no vector index, so every query compares against all vectors. With `PERF_INDEX=1` the suite builds the IVF index after the load, with the `weed-vector-cli index build` command, and the queries then go through it. On a running cluster the [[Lance Maintenance Worker]]'s `lance_vector_index` job builds it.
+
+Setup: the same 16-core host, shared with other work (load average 3 to 15), every process on it; Go volume server, `cosine` metric, top-10, 384 dimensions, 100 timed queries per scenario, master at 878979af70. The vectors are drawn around 100 shared centres (`PERF_CLUSTERS=100`), the same data with and without the index. The build sizes the index at the square root of the vector count, 224 partitions at 50k and 447 at 200k, and a query searches a tenth of them, 23 and 45. Each configuration ran twice, alternating, and the cells show both runs:
+
+| 50k × 384 | No index | Indexed |
+|---|---:|---:|
+| Query top-10, p50 / p95 | 36 / 84 ms, 40 / 80 ms | 23 / 61 ms, 45 / 58 ms |
+| Query + metadata, p50 | 44 ms, 58 ms | 44 ms, 52 ms |
+| Query with a 10% filter, p50 | 70 ms, 110 ms | 61 ms, 110 ms |
+| Queries/s with 1 / 4 / 8 clients | 29 / 54 / 59, 20 / 59 / 61 | 45 / 136 / 200, 28 / 129 / 225 |
+| Recall@10, unfiltered and filtered | 1.000 | 1.000 |
+| Index build | | 2.0 s, 2.2 s |
+
+| 200k × 384 | No index | Indexed |
+|---|---:|---:|
+| Query top-10, p50 / p95 | 83 / 121 ms, 81 / 120 ms | 37 / 86 ms, 65 / 89 ms |
+| Query + metadata, p50 | 85 ms, 83 ms | 44 ms, 56 ms |
+| Query with a 10% filter, p50 | 237 ms, 362 ms | 228 ms, 198 ms |
+| Queries/s with 1 / 4 / 8 clients | 12 / 18 / 17, 12 / 18 / 18 | 18 / 82 / 115, 29 / 80 / 108 |
+| Recall@10, unfiltered and filtered | 1.000 | 1.000 |
+| Index build | | 4.6 s, 6.2 s |
+
+* **Throughput with several clients is where the index pays.** With 8 clients it served 200 to 225 queries/s instead of 59 to 61 at 50k × 384, and 108 to 115 instead of 17 to 18 at 200k × 384.
+* **One client gains at 200k.** At 200k × 384 the median query took 37 to 65 ms instead of 81 to 83 ms. At 50k × 384 the two runs disagree (23 and 45 ms, against 36 and 40 ms without the index), so no gain can be read from them.
+* **The 10% filter gains little.** At 50k its medians (61 and 110 ms) were about those without the index (70 and 110 ms), and at 200k it took 198 to 228 ms instead of 237 to 362 ms.
+* **Recall depends on the data.** It was 1.000 here because the vectors cluster, as embeddings do. On vectors drawn independently (`PERF_CLUSTERS=0`) recall@10 followed the share of partitions searched: 0.13 with 8 of 224 partitions and 0.56 with 64 of 224.
+* **Building is quick next to loading.** 2 s at 50k and 5 to 6 s at 200k, on vectors that took 27 to 29 s and about two minutes to ingest.
+
## Beyond one engine
-The open-source gateway sends every query to one `weed-vector` engine at a time, and that engine reads the index's files through the S3 gateway. Every number above comes from a single engine. The [SeaweedFS Enterprise](https://seaweedfs.com/) version adds distributed vector querying: each index is placed on groups of engines, a query goes to one group, each engine in that group searches its share of the index, and the gateway merges the results. The Enterprise engines read the Lance files straight from the filers and volume servers, not through the gateway. When an engine is lost, queries keep returning the same results from the remaining engines.
+The open-source gateway sends every query to one `weed-vector` engine at a time, and that engine reads the index's files through the S3 gateway. Every number above comes from a single engine, and the numbers in this section are measured without a vector index. The [SeaweedFS Enterprise](https://seaweedfs.com/) version adds distributed vector querying: each index is placed on groups of engines, a query goes to one group, each engine in that group searches its share of the index, and the gateway merges the results. The Enterprise engines read the Lance files straight from the filers and volume servers, not through the gateway. When an engine is lost, queries keep returning the same results from the remaining engines.
Setup: one 16-core x86_64 host, shared with other work (load average 9 to 19), every process on it; Go volume server, `cosine` metric, top-10, 100 timed queries per scenario. The open-source build is master at d727c6970f; the Enterprise build is the branch of seaweed-mono#954, which makes every engine read directly. Each configuration ran twice, and the cells show both runs:
diff --git a/S3-Vector-Bucket.md b/S3-Vector-Bucket.md
index 9d45ba6..ffbcf53 100644
--- a/S3-Vector-Bucket.md
+++ b/S3-Vector-Bucket.md
@@ -25,7 +25,7 @@ flowchart TB
vector["vector bucket
a table bucket with format LANCE"]
normal["normal bucket
objects"]
end
- engine["weed-vector
exact search over the index"]
+ engine["weed-vector
similarity search over the index"]
filers[("filers
file metadata")]
volumes[("volume servers
file chunks")]
client -->|"S3, S3 Tables and S3 Vectors requests"| gateway
@@ -49,7 +49,7 @@ The gateway serves all three bucket kinds on the S3 port. A vector bucket is a t
* **The vector engine** is a separate process, `weed-vector`. It registers with the master, and the gateway finds it through the master on the first request that needs it. Bucket and policy operations work without it; index and vector operations need it. Several can register: the open-source gateway sends its requests to one at a time, and SeaweedFS Enterprise spreads each index and its queries over them (see [[S3 Vector Bucket Performance#beyond-one-engine]]).
* **The data** is stored as [Lance](https://lancedb.com) datasets. A vector bucket is a table bucket with format `LANCE`, and each index is a Lance table in the bucket's `vector` namespace, stored under `s3:///vector//`. The engine reaches it through the [[SeaweedFS Lance Catalog]] (`-s3.port.lance`, default `9101`, on by default) and the S3 port.
-**Queries are exact.** No approximate (ANN) index is built, so every query compares against all vectors of the index; recall@10 was 1.000 in every run of the perf suite. Query time grows with the size of the index; see [[S3 Vector Bucket Performance]].
+**Queries are exact until a vector index is built.** Without one, every query compares against all vectors of the index, and recall@10 was 1.000 in every such run of the perf suite. The [[Lance Maintenance Worker]]'s `lance_vector_index` job builds an approximate (ANN) index, IVF, once an index holds at least `min_rows` vectors (50,000 by default). From then on a query searches a tenth of the index's partitions, at least 8, and up to four times that many when that leaves it short of `topK` results. It returns only what it finds in those partitions, so a filter that few vectors match can return fewer results than exist ([[S3 Vectors API]] has numbers). Vectors written after the build are still returned, and the worker's `lance_optimize_indices` job brings them into the index. Like every worker job, both start disabled. See [[S3 Vector Bucket Performance#through-a-vector-index]] for measurements.
## Getting weed-vector
@@ -248,6 +248,7 @@ neuromancer 0.1 {'genre': 'scifi', 'pages': 271, 'summary': 'Cyberspace heist.'}
| `--listen` | `WEED_VECTOR_LISTEN` | `127.0.0.1:8090` | gRPC address. This is the address registered on the master, so the gateway must be able to reach it. |
| `--storage-option key=value` | | none | Repeatable. `aws_access_key_id` and `aws_secret_access_key` are used for S3 data access and for the Lance catalog. Needed when the S3 server has identities. |
| `--read-cache-mb` | `WEED_VECTOR_READ_CACHE_MB` | `512` | Memory for caching reads of the index's (write-once) data files, in MiB. `0` disables it. |
+| `--nprobes` | `WEED_VECTOR_NPROBES` | a tenth of the partitions, at least 8 | Partitions every query searches in a vector index, in place of the default. For measurements; `0` is rejected. |
| `--filer-signing-key` | `WEED_FILER_SIGNING_KEY` | none | The cluster's `jwt.filer_signing.key`. The Lance catalog vends credentials that can write index data only to a caller holding it. |
| `--security-toml` | `WEED_VECTOR_SECURITY_TOML` | found like `weed` does | The cluster's `security.toml`, for gRPC TLS. Without it the engine looks in `--config-dir`, `.`, `~/.seaweedfs`, `/usr/local/etc/seaweedfs` and `/etc/seaweedfs`. |
| `--config-dir` | `WEED_VECTOR_CONFIG_DIR` | none | The directory `weed -config_dir` names, searched first for `security.toml`. `~` is expanded. |
diff --git a/S3-Vectors-API.md b/S3-Vectors-API.md
index 9a66e96..8aa4516 100644
--- a/S3-Vectors-API.md
+++ b/S3-Vectors-API.md
@@ -99,10 +99,10 @@ s3v.query_vectors(
)
```
-* **`topK`**: 1 to 10000. `topK` larger than the index returns every vector.
+* **`topK`**: 1 to 10000. `topK` larger than the index returns every vector, as long as no vector index is built (see the last point).
* The query vector must have the index's dimension.
* `distance` is in the result only with `returnDistance`, and `metadata` only with `returnMetadata`.
-* **Search is exact**: every query compares against every vector in the index; no approximate index is built.
+* **Search is exact until a vector index is built**: without one, every query compares against every vector in the index. Once the [[Lance Maintenance Worker]] has built an approximate index (from 50,000 vectors by default), a query searches part of it (see [[S3 Vector Bucket#how-it-works]]) and returns only what it finds there, so a `topK` near the size of the index, or a filter that few vectors match, can return fewer results than exist. In a test with 4,000 vectors in 63 partitions, `topK` 4000 returned 2,173 vectors, and a filter that 400 vectors matched returned 210 of them for `topK` 500.
## Metadata filters