S3 Vector Bucket: diagram shows the filers in the data path and the Enterprise engines' direct reads

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

S3-Vector-Bucket.md

@@ -14,31 +14,40 @@ flowchart LR
         direction TB
         client["<b>Clients</b><br/>AWS CLI, boto3, any AWS SDK"]
         vapi["<b>S3 gateway: S3 Vectors API</b><br/>port 8333, signed for s3vectors"]
-        engines["<b>weed-vector instances</b><br/>gRPC port 8090"]
+        engines["<b>weed-vector</b><br/>gRPC port 8090"]
         sapi["<b>S3 gateway: Lance catalog, S3 API</b><br/>ports 9101 and 8333"]
-        volumes[("<b>volume servers</b>")]
+        filers[("<b>filers</b><br/>bucket, policy and table<br/>records, file metadata")]
+        volumes[("<b>volume servers</b><br/>file chunks")]
         client -->|"S3 Vectors requests"| vapi
         vapi -->|"index and vector operations"| engines
         engines -->|"index tables and Lance files"| sapi
+        sapi -->|"file metadata"| filers
         sapi -->|"file chunks"| volumes
     end
-    subgraph side ["cluster services"]
+    subgraph ent ["SeaweedFS Enterprise: distributed queries"]
         direction TB
+        workers["<b>weed-vector-worker</b> engines<br/>gRPC port 18888, one share of each index"]
+    end
+    vapi -.->|"QueryVectors, one request per engine"| workers
+    workers -.->|"file metadata, read directly"| filers
+    workers -.->|"file chunks, read directly"| volumes
+    subgraph side ["cluster services"]
         master["<b>master</b>, port 9333<br/>engines register here,<br/>the gateway looks them up"]
-        filers[("<b>filers</b><br/>bucket, policy and table<br/>records, file metadata")]
-        master ~~~ filers
     end
     path -.- side
 
     classDef s3server fill:#dbeafe,stroke:#2563eb,color:#0f172a
     classDef other fill:#f1f5f9,stroke:#64748b,color:#0f172a
+    classDef enterprise fill:#fef3c7,stroke:#d97706,color:#0f172a
     class vapi,sapi s3server
     class client,engines,master,filers,volumes other
+    class workers enterprise
     style path fill:none,stroke:#94a3b8
+    style ent fill:none,stroke:#d97706,stroke-dasharray:4 3
     style side fill:none,stroke:#94a3b8
 ```
 
-Both blue boxes run in the S3 server process: the engines call back into an S3 server for the index data. Each index is one Lance dataset under `s3://<bucket>/vector/<index>/`, stored as file chunks on the volume servers.
+Both blue boxes run in the S3 server process: `weed-vector` calls back into an S3 server for the index data, and that S3 server reads the file metadata from the filers and the chunks from the volume servers like any other S3 request. Each index is one Lance dataset under `s3://<bucket>/vector/<index>/`, stored as file chunks on the volume servers. The dashed path is SeaweedFS Enterprise: its search engines (`weed-vector-worker`) answer `QueryVectors` and read the same Lance files straight from the filers and volume servers, skipping the gateway; writes and index management still go through `weed-vector`.
 
 * **The S3 gateway** (`weed s3`, or the S3 server inside `weed server -s3` and `weed mini`) answers S3 Vectors requests on the normal S3 port. It tells them apart by the service name in the SigV4 credential scope: a request signed for `s3vectors` goes to the vectors API, everything else to the S3 API.
 * **The vector engine** is a separate process, `weed-vector`. It registers with the master, and the gateway finds it through the master on the first request that needs it. Bucket and policy operations work without it; index and vector operations need it. Several can register: the open-source gateway sends its requests to one at a time, and SeaweedFS Enterprise spreads each index and its queries over them (see [[S3 Vector Bucket Performance#beyond-one-engine]]).