Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-11-07 12:37:41 +01:00
Code Issues Packages Projects Releases Wiki Activity

S3 Vector Bucket Performance: direct-reading Enterprise engines against the open-source single engine, 50k and 200k 

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Chris LuandClaude Fable 5.1 committed 2026-11-04 07:11:08 -08:00
1 parent 65139a4842
commit e04a41f42f
1 file changed
+29 -20
Download Patch File Download Diff File Expand all files Collapse all files
S3-Vector-Bucket-Performance.md
+29 -20
@@ -58,33 +58,42 @@ What the numbers show:
## Beyond one engine
The open-source gateway sends every query to one `weed-vector` engine at a time, so every number above comes from a single engine. The [SeaweedFS Enterprise](https://seaweedfs.com/) version adds distributed vector querying: each index is placed on groups of engines, a query goes to one group, each engine in that group searches its share of the index, and the gateway merges the results. When an engine is lost, queries keep returning the same results from the remaining engines.
The open-source gateway sends every query to one `weed-vector` engine at a time, and that engine reads the index's files through the S3 gateway. Every number above comes from a single engine. The [SeaweedFS Enterprise](https://seaweedfs.com/) version adds distributed vector querying: each index is placed on groups of engines, a query goes to one group, each engine in that group searches its share of the index, and the gateway merges the results. The Enterprise engines read the Lance files straight from the filers and volume servers, not through the gateway. When an engine is lost, queries keep returning the same results from the remaining engines.
The numbers below were measured with an Enterprise build that includes a fix on its way into the Enterprise version. Before it, each query was searched by a single engine of the group, which re-read the whole index through the S3 gateway on every query.
Setup: one 16-core x86_64 host, shared with other work, with every process on it, including four search engines; Go volume server, `cosine` metric, top-10. Three configurations, each run twice (the cells show both runs):
Setup: one 16-core x86_64 host, shared with other work (load average 9 to 19), every process on it; Go volume server, `cosine` metric, top-10, 100 timed queries per scenario. The open-source build is master at d727c6970f; the Enterprise build is the branch of seaweed-mono#954, which makes every engine read directly. Each configuration ran twice, and the cells show both runs:
* **Single engine**: the open-source `weed-vector`.
* **4 groups of 1**: each query goes to one engine, and queries are spread over the four.
* **1 group of 4**: each query is split over all four engines, each searching a quarter of the index.
* **1 group of 4**: four Enterprise engines in one group (`min_width = 4`, `min_replicas = 1`), so each query is split four ways.
| 50k × 384 | Single engine | 4 groups of 1 | 1 group of 4 |
|---|---:|---:|---:|
| Query top-10, p50 / p95 | 66 / 113 ms, 57 / 82 ms | 28 / 80 ms, 40 / 69 ms | 21 / 68 ms, 20 / 64 ms |
| Query with a 10% filter, p50 | 74 ms, 125 ms | 61 ms, 95 ms | 25 ms, 26 ms |
| Queries/s with 1 / 4 / 8 clients | 28 / 50 / 56, 19 / 56 / 62 | 35 / 63 / 65, 29 / 69 / 69 | 47 / 57 / 61, 48 / 63 / 65 |
| 50k × 384 | Single engine | 1 group of 4 |
|---|---:|---:|
| Query top-10, p50 / p95 | 42 / 87 ms, 54 / 82 ms | 23 / 70 ms, 21 / 72 ms |
| Query + metadata, p50 | 45 ms, 57 ms | 25 ms, 23 ms |
| Query with a 10% filter, p50 | 84 ms, 116 ms | 29 ms, 31 ms |
| Queries/s with 1 / 4 / 8 clients | 24 / 47 / 52, 20 / 58 / 62 | 41 / 55 / 58, 42 / 62 / 63 |
| GetVectors, 100 keys, p50 | 68 ms, 58 ms | 67 ms, 62 ms |
| Ingest (vectors/s) | 1,733, 1,727 | 1,757, 1,690 |
| 200k × 384 | Single engine | 4 groups of 1 | 1 group of 4 |
|---|---:|---:|---:|
| Query top-10, p50 / p95 | 81 / 105 ms, 84 / 113 ms | 71 / 110 ms, 70 / 107 ms | 57 / 89 ms, 62 / 88 ms |
| Query with a 10% filter, p50 | 380 ms, 316 ms | 249 ms, 290 ms | 73 ms, 78 ms |
| Queries/s with 1 / 4 / 8 clients | 12 / 16 / 17, 10 / 13 / 14 | 14 / 19 / 18, 13 / 20 / 18 | 14 / 17 / 17, 17 / 18 / 17 |
| 200k × 384 | Single engine | 1 group of 4 |
|---|---:|---:|
| Query top-10, p50 / p95 | 273 / 392 ms*, 83 / 121 ms | 58 / 101 ms, 58 / 98 ms |
| Query + metadata, p50 | 104 ms, 85 ms | 63 ms, 63 ms |
| Query with a 10% filter, p50 | 303 ms, 354 ms | 73 ms, 73 ms |
| Queries/s with 1 / 4 / 8 clients | 10 / 12 / 13, 12 / 18 / 18 | 17 / 18 / 18, 17 / 17 / 18 |
| GetVectors, 100 keys, p50 | 138 ms, 112 ms | 135 ms, 120 ms |
| Ingest (vectors/s) | 1,365, 1,673 | 1,634, 1,649 |
Recall@10 was 1.000 in every run, filtered queries included.
\* The first run's unfiltered scenario coincided with a load spike on the host; the scenarios right after it on the same cluster, and the repeat run, were at 83 to 104 ms. The Enterprise build's engine, queried singly on the same day, gave 86 and 82 ms.
* **Splitting a query cuts its latency.** With one group of four engines, the filtered query took about 75 ms instead of 316 to 380 ms at 200k × 384, and 25 ms instead of 74 to 125 ms at 50k × 384; one client ran about 48 queries/s instead of 19 to 28 at 50k × 384.
* **Spreading queries keeps up with one engine.** Four groups of one engine matched or beat the single engine's median latency and its throughput at both sizes.
* **Throughput at 4 to 8 clients is bound by this host's CPU.** All engines shared the same 16 cores, so no configuration went far beyond the single engine with 4 or 8 clients. Engines on separate hosts were not measured.
Recall@10 was 1.000 in every run, filtered queries included, and the gateway routed every timed query to all four engines (none fell back to the single engine).
* **Splitting a query cuts its latency.** With one group of four engines, the filtered query took 73 ms instead of 303 to 354 ms at 200k × 384, and 29 to 31 ms instead of 84 to 116 ms at 50k × 384. Unfiltered queries took 58 ms instead of 83 ms at 200k, and 21 to 23 ms instead of 42 to 54 ms at 50k.
* **One client gets 1.4 to 2 times the throughput.** 41 to 42 queries/s instead of 20 to 24 at 50k × 384, and 17 instead of 10 to 12 at 200k × 384.
* **Throughput at 4 to 8 clients is bound by this host's CPU.** All engines shared the same 16 cores, so with 4 or 8 clients the four engines ended within a few queries/s of the single engine. Engines on separate hosts were not measured.
* **Default placement sits in between.** With the controller's defaults the same four engines were placed as two replica groups instead of one group of four. Medians were 47 ms at 50k and 70 to 80 ms at 200k, filtered 40 ms and 128 to 139 ms. At 200k each query was split over two engines, and the first few hundred queries went to one engine before the index was widened; the timed scenario started inside that window.
* **Ingest does not change.** Vectors are written through the gateway in every configuration, at 1.6k to 1.8k vectors/s here.
These 200k indexes were queried right after the bulk load, as 400 fragments of 500 rows, with no [[Lance Maintenance Worker]] compacting them.
## The read cache