Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-11-07 12:37:41 +01:00
Code Issues Packages Projects Releases Wiki Activity
Page: S3 Vector Bucket Performance
Pages
A typical step‐by‐step example AWS CLI with SeaweedFS AWS IAM CLI Actual Users Admin UI OIDC Admin UI Amazon IAM API Amazon S3 API Applications Async Backup Async Filer Metadata Backup Async Replication to Cloud Async Replication to another Filer Azure Blob Storage Authentication Benchmark SeaweedFS as a GlusterFS replacement Benchmarks from jinleileiking Benchmarks Blob Store Architecture Cache Remote Storage Choosing a Filer Store Client Libraries Cloud Drive Architecture Cloud Drive Benefits Cloud Drive Quick Setup Cloud Monitoring Cloud Tier Cluster Plan Day 2 Operations Cluster Plan Inventory Reference Cluster Plan Workflow Components Configure Remote Storage Cryptography and FIPS Compliance Customize Filer Store Data Backup Data Structure for Large Files Deployment to Kubernetes and Minikube Deployment with seaweed up Directories and Files Distributed POSIX Locks Distributing AI Model Files for Multi GPU Loading Docker Compose for S3 Docker Image Registry with SeaweedFS Doris Iceberg Integration Dremio Iceberg Integration DuckDB Iceberg Integration DuckDB Lance Integration EC Bitrot Detection Environment Variables Erasure Coding for warm storage Error reporting to sentry FAQ FIO benchmark FUSE Mount Failover Master Server File Operations Quick Reference Filer Active Active cross cluster continuous synchronization Filer Cassandra Setup Filer Change Data Capture Filer Commands and Operations Filer Data Encryption Filer JWT Use Filer Metadata Events Filer Notification Webhook Filer Operation Serialization Filer Redis Setup Filer Server API Filer Setup Filer Store Replication Filer Stores Filer as a Key Large Value Store Gateway to Remote Object Storage Getting Started HDFS via S3 connector Hadoop Benchmark Hadoop Compatible File System Hardware Helm Chart Recipes Hobbyest Tinkerer scale on premises tutorial Home Iceberg REST Catalog API Iceberg Table Maintenance Image Gateway Independent Benchmarks Kafka to Kafka Gateway to SMQ to SQL Kubernetes Backups and Recovery with K8up Kubernetes ServiceAccount Authentication Lakekeeper Iceberg Integration Lance Maintenance Worker LanceDB Integration Large File Handling Load Command Line Options from a file Master Server API Migrate Between Filer Stores Migrate Maintenance Scripts to Admin Script Plugin Migrate to Filer Store Mount Remote Storage Mount on Windows Mount over the Internet OIDC Integration OLake Go Iceberg Integration Optimization for Many Small Buckets Optimization P2P reading in weed mount POSIX Compliance Path Specific Configuration Path Specific Filer Store Plugin Worker Scheduling PostgreSQL compatible Server weed db Production Setup Pub Sub to SMQ to SQL Quick Start with weed mini Replication RisingWave Iceberg Integration Run Blob Storage on Public Internet Run Trino on SeaweedFS Rust Volume Server S3 API Audit log S3 API Benchmark S3 API FAQ S3 Bucket Policies S3 Bucket Quota S3 CORS S3 Conditional Operations S3 Configuration S3 Credentials S3 Lifecycle Architecture S3 Lifecycle Monitoring S3 Lifecycle Operator Guide S3 Lifecycle Recipes S3 Lifecycle Troubleshooting S3 Lifecycle vs Volume TTL S3 Lifecycle S3 Nginx Proxy S3 Object Lock and Retention S3 Object Versioning S3 Policy Conditions S3 Policy Variables S3 Rate Limiting S3 RenameObject S3 Table Bucket Commands S3 Table Bucket S3 Tables Security S3 Vector Bucket Performance S3 Vector Bucket Security S3 Vector Bucket S3 Vectors API SFTP Server SQL Queries on Message Queue SQL Quick Reference SRV Service Discovery Seaweed Message Queue SeaweedFS Architecture SeaweedFS Iceberg Catalog SeaweedFS Java Client SeaweedFS Lance Catalog SeaweedFS in Docker Swarm Security Configuration Security Overview Server Side Encryption SSE C Server Side Encryption SSE KMS Server Side Encryption Server Startup via Systemd Simplest S3 Bucket and User Setup Spark Iceberg Integration Spark Lance Integration Store file with a Time To Live Structured Data Lake with SMQ and SQL Super Large Directories Supported APIs vs Minio System Metrics TUS Resumable Uploads TensorFlow with SeaweedFS Tiered Storage Trino Iceberg Integration UrBackup with SeaweedFS Use Cases Volume Files Structure Volume Management Volume Server API WebDAV Words from SeaweedFS Users Worker fstab and systemd mount nodejs with Seaweed S3 rclone with SeaweedFS restic with SeaweedFS run HBase on SeaweedFS run Spark on SeaweedFS s3cmd with SeaweedFS seaweed up Command Reference weed shell
Clone
6
S3 Vector Bucket Performance
Chris Lu edited this page 2026-11-05 12:54:16 -08:00
Unescape Escape
Table of Contents
This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

S3 Vector Bucket Performance

Measured numbers for S3 Vector Buckets, and how to reproduce them with the perf suite that ships in the repository.

The perf suite

test/s3vectors/perf_test.py starts a local cluster (master, volume server, filer, S3 gateway and weed-vector), loads random vectors through the S3 Vectors API and times every data-plane operation. It checks correctness, not speed: recall is checked against an exact search done in the client (with PERF_INDEX it is reported, not asserted), and every listing must return every vector.

cd test/s3vectors
pip install -r requirements.txt          # boto3, pytest
make perf                                # builds weed, weed-vector, weed-vector-cli and weed-volume, then runs the suite
# or, with the binaries already in this directory:
VOLUME_SERVER_IMPL=go ./run_perf.sh

The Python packages are boto3 1.40 or newer and pytest 8 or newer. make perf needs Go and Rust; like weed-vector (see S3 Vector Bucket#getting-weed-vector), it builds without a system protoc.

Variable Default Meaning
PERF_VECTORS 10000 vectors to ingest
PERF_DIM 128 dimension
PERF_BATCH 500 vectors per PutVectors call
PERF_QUERIES 100 timed queries per scenario
PERF_TOPK 10 topK
PERF_CONCURRENCY 1,4,8 client threads for the throughput scenario
PERF_RECALL 20 queries checked against the exact search
PERF_INDEX unset 1 builds the vector index after the load, so the queries run through it
PERF_PARTITIONS the square root of the vector count partitions of that index
PERF_CLUSTERS 0 draw the vectors around this many shared centres; 0 draws each one independently
PERF_NOISE 0.3 spread of the vectors around their centre
PERF_REPORT perf-results.json in the cluster directory where the JSON report goes
VOLUME_SERVER_IMPL go,rust which volume server(s) to run against

Each scenario starts with one untimed call, and the throughput scenario creates and warms its clients before the timer starts. The suite prints a summary and writes all numbers to the JSON report.

Results

One 16-core x86_64 host, every process on it, cosine metric, top-10, one client unless noted. Other work kept the host's load average between 3.4 and 4.9 during the runs, so read differences of a few milliseconds, and the Go/Rust columns against each other, as noise.

Go volume, 10k × 128 Rust volume, 10k × 128 Go volume, 50k × 384 Rust volume, 50k × 384
Ingest (vectors/s) 4,579 4,476 1,805 1,654
PutVectors, 500 per call, p50 109 ms 111 ms 272 ms 282 ms
Upsert 500 existing, p50 111 ms 114 ms 285 ms 270 ms
Query top-10, p50 / p95 23 / 43 ms 21 / 44 ms 56 / 73 ms 53 / 73 ms
Query + metadata, p50 22 ms 29 ms 37 ms 41 ms
Query with a 10% filter, p50 40 ms 51 ms 82 ms 45 ms
Recall@10 1.000 1.000 1.000 1.000
Queries/s with 1 / 4 / 8 clients 33 / 135 / 286 29 / 186 / 285 18 / 55 / 64 25 / 56 / 54
GetVectors, 100 keys, p50 53 ms 52 ms 58 ms 60 ms
DeleteVectors, 100 keys, p50 25 ms 23 ms 44 ms 44 ms
ListVectors (vectors/s), 1 / 4 / 16 segments 18k / 63k / 134k 24k / 69k / 129k 18k / 75k / 169k 16k / 74k / 170k

What the numbers show:

  • Queries scan every vector. These runs built no vector index (see Through a vector index for runs that did), so latency grows with the size of the index: going from 10k × 128 to 50k × 384 (15 times the data) raised the median query from about 22 ms to about 55 ms. Recall is 1.000 throughout.
  • Throughput grows with clients until the index gets large. At 10k × 128, 8 clients reach about 285 queries/s; at 50k × 384 throughput levels off between 4 and 8 clients at 55 to 65 queries/s.
  • Parallel listing scales. ListVectors with 16 segments read the whole index 5 to 11 times faster than one sequential listing.
  • Ingest is steady. Batch latency did not grow from the first to the last tenth of the load (for example 283 ms → 272 ms at 50k × 384).
  • No consistent difference between the volume servers. The largest gap, the filtered query at 50k × 384 (82 ms on Go, 45 ms on Rust), goes the other way at 10k × 128 (40 ms against 51 ms).

Through a vector index

The runs above have no vector index, so every query compares against all vectors. With PERF_INDEX=1 the suite builds the IVF index after the load, with the weed-vector-cli index build command, and the queries then go through it. On a running cluster the Lance Maintenance Worker's lance_vector_index job builds it.

Setup: the same 16-core host, shared with other work (load average 3 to 15), every process on it; Go volume server, cosine metric, top-10, 384 dimensions, 100 timed queries per scenario, master at 878979af70. The vectors are drawn around 100 shared centres (PERF_CLUSTERS=100), the same data with and without the index. The build sizes the index at the square root of the vector count, 224 partitions at 50k and 447 at 200k, and a query searches a tenth of them, 23 and 45. Each configuration ran twice, alternating, and the cells show both runs:

50k × 384 No index Indexed
Query top-10, p50 / p95 36 / 84 ms, 40 / 80 ms 23 / 61 ms, 45 / 58 ms
Query + metadata, p50 44 ms, 58 ms 44 ms, 52 ms
Query with a 10% filter, p50 70 ms, 110 ms 61 ms, 110 ms
Queries/s with 1 / 4 / 8 clients 29 / 54 / 59, 20 / 59 / 61 45 / 136 / 200, 28 / 129 / 225
Recall@10, unfiltered and filtered 1.000 1.000
Index build 2.0 s, 2.2 s
200k × 384 No index Indexed
Query top-10, p50 / p95 83 / 121 ms, 81 / 120 ms 37 / 86 ms, 65 / 89 ms
Query + metadata, p50 85 ms, 83 ms 44 ms, 56 ms
Query with a 10% filter, p50 237 ms, 362 ms 228 ms, 198 ms
Queries/s with 1 / 4 / 8 clients 12 / 18 / 17, 12 / 18 / 18 18 / 82 / 115, 29 / 80 / 108
Recall@10, unfiltered and filtered 1.000 1.000
Index build 4.6 s, 6.2 s
  • Throughput with several clients is where the index pays. With 8 clients it served 200 to 225 queries/s instead of 59 to 61 at 50k × 384, and 108 to 115 instead of 17 to 18 at 200k × 384.
  • One client gains at 200k. At 200k × 384 the median query took 37 to 65 ms instead of 81 to 83 ms. At 50k × 384 the two runs disagree (23 and 45 ms, against 36 and 40 ms without the index), so no gain can be read from them.
  • The 10% filter gains little. At 50k its medians (61 and 110 ms) were about those without the index (70 and 110 ms), and at 200k it took 198 to 228 ms instead of 237 to 362 ms.
  • Recall depends on the data. It was 1.000 here because the vectors cluster, as embeddings do. On vectors drawn independently (PERF_CLUSTERS=0) recall@10 followed the share of partitions searched: 0.13 with 8 of 224 partitions and 0.56 with 64 of 224.
  • Building is quick next to loading. 2 s at 50k and 5 to 6 s at 200k, on vectors that took 27 to 29 s and about two minutes to ingest.

Beyond one engine

The open-source gateway sends every query to one weed-vector engine at a time, and that engine reads the index's files through the S3 gateway. Every number above comes from a single engine, and the numbers in this section are measured without a vector index. The SeaweedFS Enterprise version adds distributed vector querying: each index is placed on groups of engines, a query goes to one group, each engine in that group searches its share of the index, and the gateway merges the results. The Enterprise engines read the Lance files straight from the filers and volume servers, not through the gateway. When an engine is lost, queries keep returning the same results from the remaining engines.

Setup: one 16-core x86_64 host, shared with other work (load average 9 to 19), every process on it; Go volume server, cosine metric, top-10, 100 timed queries per scenario. The open-source build is master at d727c6970f; the Enterprise build is the branch of seaweed-mono#954, which makes every engine read directly. Each configuration ran twice, and the cells show both runs:

  • Single engine: the open-source weed-vector.
  • 1 group of 4: four Enterprise engines in one group (min_width = 4, min_replicas = 1), so each query is split four ways.
50k × 384 Single engine 1 group of 4
Query top-10, p50 / p95 42 / 87 ms, 54 / 82 ms 23 / 70 ms, 21 / 72 ms
Query + metadata, p50 45 ms, 57 ms 25 ms, 23 ms
Query with a 10% filter, p50 84 ms, 116 ms 29 ms, 31 ms
Queries/s with 1 / 4 / 8 clients 24 / 47 / 52, 20 / 58 / 62 41 / 55 / 58, 42 / 62 / 63
GetVectors, 100 keys, p50 68 ms, 58 ms 67 ms, 62 ms
Ingest (vectors/s) 1,733, 1,727 1,757, 1,690
200k × 384 Single engine 1 group of 4
Query top-10, p50 / p95 273 / 392 ms*, 83 / 121 ms 58 / 101 ms, 58 / 98 ms
Query + metadata, p50 104 ms, 85 ms 63 ms, 63 ms
Query with a 10% filter, p50 303 ms, 354 ms 73 ms, 73 ms
Queries/s with 1 / 4 / 8 clients 10 / 12 / 13, 12 / 18 / 18 17 / 18 / 18, 17 / 17 / 18
GetVectors, 100 keys, p50 138 ms, 112 ms 135 ms, 120 ms
Ingest (vectors/s) 1,365, 1,673 1,634, 1,649

* The first run's unfiltered scenario coincided with a load spike on the host; the scenarios right after it on the same cluster, and the repeat run, were at 83 to 104 ms. The Enterprise build's engine, queried singly on the same day, gave 86 and 82 ms.

Recall@10 was 1.000 in every run, filtered queries included, and the gateway routed every timed query to all four engines (none fell back to the single engine).

  • Splitting a query cuts its latency. With one group of four engines, the filtered query took 73 ms instead of 303 to 354 ms at 200k × 384, and 29 to 31 ms instead of 84 to 116 ms at 50k × 384. Unfiltered queries took 58 ms instead of 83 ms at 200k, and 21 to 23 ms instead of 42 to 54 ms at 50k.
  • One client gets 1.4 to 2 times the throughput. 41 to 42 queries/s instead of 20 to 24 at 50k × 384, and 17 instead of 10 to 12 at 200k × 384.
  • Throughput at 4 to 8 clients is bound by this host's CPU. All engines shared the same 16 cores, so with 4 or 8 clients the four engines ended within a few queries/s of the single engine. Engines on separate hosts were not measured.
  • Default placement sits in between. With the controller's defaults the same four engines were placed as two replica groups instead of one group of four. Medians were 47 ms at 50k and 70 to 80 ms at 200k, filtered 40 ms and 128 to 139 ms. At 200k each query was split over two engines, and the first few hundred queries went to one engine before the index was widened; the timed scenario started inside that window.
  • Ingest does not change. Vectors are written through the gateway in every configuration, at 1.6k to 1.8k vectors/s here.

These 200k indexes were queried right after the bulk load, as 400 fragments of 500 rows, with no Lance Maintenance Worker compacting them.

The read cache

weed-vector keeps reads of an index's data files in memory, 512 MiB by default (--read-cache-mb, WEED_VECTOR_READ_CACHE_MB; 0 disables it). The results above ran with the default. The same 50k × 384 index on the Go volume server, with the cache off and on, run alternately twice:

Cache off (0) Cache 512 MiB
Query top-10, p50 127 ms, 127 ms 48 ms, 36 ms
Query with a 10% filter, p50 233 ms, 233 ms 71 ms, 48 ms
GetVectors, 100 keys, p50 332 ms, 319 ms 59 ms, 76 ms
Queries/s with 1 / 4 / 8 clients 7.5 / 11.5 / 12.2, 7.8 / 11.6 / 11.9 19 / 59 / 59, 26 / 50 / 50

Without the cache, every query reads the index's files back through the S3 gateway and the volume server. The cache cut median query latency to about a third and raised throughput about four to five times, even though other work loaded the host more during the cached runs (load average 19 to 21, against 6 to 10).

This index holds about 77 MB of raw vector data (50,000 × 384 × 4 bytes), well inside the default cache. An index larger than the cache was not measured.

Introduction

API

Configuration

Filer

Filer Stores

Management

Advanced Filer Configurations

FUSE Mount

WebDAV

SFTP Server

Cloud Drive

AWS S3 API

S3 Table Bucket

S3 Vector Bucket

Iceberg Integrations

Lance Integrations

S3 Authentication & IAM

Server-Side Encryption

S3 Client Tools

Machine Learning

HDFS

Replication and Backup

Metadata Change Events

Messaging

Use Cases

Operations

Rust Volume Server

Advanced

Security

Misc Use Case Examples