S3 Vector Bucket Performance: measured distributed querying

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

S3-Vector-Bucket-Performance.md

@@ -58,7 +58,33 @@ What the numbers show:
 
 ## Beyond one engine
 
-The open-source gateway sends every query to one `weed-vector` engine at a time, so every number above comes from a single engine. The [SeaweedFS Enterprise](https://seaweedfs.com/) version adds distributed vector querying: each index is spread over a group of engines, a query goes to one group, each engine in it searches its share of the index, and the gateway merges the results. An index can have several groups to serve more queries, and when an engine is lost, queries keep returning the same results from the remaining engines. The distributed mode has not been measured on this page.
+The open-source gateway sends every query to one `weed-vector` engine at a time, so every number above comes from a single engine. The [SeaweedFS Enterprise](https://seaweedfs.com/) version adds distributed vector querying: each index is placed on groups of engines, a query goes to one group, each engine in that group searches its share of the index, and the gateway merges the results. When an engine is lost, queries keep returning the same results from the remaining engines.
+
+The numbers below were measured with an Enterprise build that includes a fix on its way into the Enterprise version. Before it, each query was searched by a single engine of the group, which re-read the whole index through the S3 gateway on every query.
+
+Setup: one 16-core x86_64 host, shared with other work, with every process on it, including four search engines; Go volume server, `cosine` metric, top-10. Three configurations, each run twice (the cells show both runs):
+
+* **Single engine**: the open-source `weed-vector`.
+* **4 groups of 1**: each query goes to one engine, and queries are spread over the four.
+* **1 group of 4**: each query is split over all four engines, each searching a quarter of the index.
+
+| 50k × 384 | Single engine | 4 groups of 1 | 1 group of 4 |
+|---|---:|---:|---:|
+| Query top-10, p50 / p95 | 66 / 113 ms, 57 / 82 ms | 28 / 80 ms, 40 / 69 ms | 21 / 68 ms, 20 / 64 ms |
+| Query with a 10% filter, p50 | 74 ms, 125 ms | 61 ms, 95 ms | 25 ms, 26 ms |
+| Queries/s with 1 / 4 / 8 clients | 28 / 50 / 56, 19 / 56 / 62 | 35 / 63 / 65, 29 / 69 / 69 | 47 / 57 / 61, 48 / 63 / 65 |
+
+| 200k × 384 | Single engine | 4 groups of 1 | 1 group of 4 |
+|---|---:|---:|---:|
+| Query top-10, p50 / p95 | 81 / 105 ms, 84 / 113 ms | 71 / 110 ms, 70 / 107 ms | 57 / 89 ms, 62 / 88 ms |
+| Query with a 10% filter, p50 | 380 ms, 316 ms | 249 ms, 290 ms | 73 ms, 78 ms |
+| Queries/s with 1 / 4 / 8 clients | 12 / 16 / 17, 10 / 13 / 14 | 14 / 19 / 18, 13 / 20 / 18 | 14 / 17 / 17, 17 / 18 / 17 |
+
+Recall@10 was 1.000 in every run, filtered queries included.
+
+* **Splitting a query cuts its latency.** With one group of four engines, the filtered query took about 75 ms instead of 316 to 380 ms at 200k × 384, and 25 ms instead of 74 to 125 ms at 50k × 384; one client ran about 48 queries/s instead of 19 to 28 at 50k × 384.
+* **Spreading queries keeps up with one engine.** Four groups of one engine matched or beat the single engine's median latency and its throughput at both sizes.
+* **Throughput at 4 to 8 clients is bound by this host's CPU.** All engines shared the same 16 cores, so no configuration went far beyond the single engine with 4 or 8 clients. Engines on separate hosts were not measured.
 
 ## The read cache