diff --git a/S3-Vector-Bucket.md b/S3-Vector-Bucket.md index 6afede5..2c5aa96 100644 --- a/S3-Vector-Bucket.md +++ b/S3-Vector-Bucket.md @@ -9,45 +9,45 @@ SeaweedFS serves the [Amazon S3 Vectors](https://docs.aws.amazon.com/AmazonS3/la ## How it works ```mermaid -flowchart LR - subgraph path ["request and data path"] - direction TB - client["Clients
AWS CLI, boto3, any AWS SDK"] - vapi["S3 gateway: S3 Vectors API
port 8333, signed for s3vectors"] - engines["weed-vector
gRPC port 8090"] - sapi["S3 gateway: Lance catalog, S3 API
ports 9101 and 8333"] - filers[("filers
bucket, policy and table
records, file metadata")] +--- +config: + flowchart: + wrappingWidth: 420 + nodeSpacing: 50 + rankSpacing: 50 +--- +flowchart TB + client["Clients
AWS CLI, boto3, any AWS SDK"] + gateway["S3 gateway
S3, S3 Tables and S3 Vectors APIs on the S3 port"] + subgraph buckets ["bucket kinds"] + direction LR + normal["normal bucket
objects"] + table["table bucket
Iceberg or Lance tables"] + vector["vector bucket
a table bucket with format LANCE"] + end + engine["weed-vector
exact search over the index"] + subgraph store ["stored in SeaweedFS, through the gateway's S3 API"] + direction LR + filers[("filers
file metadata")] volumes[("volume servers
file chunks")] - client -->|"S3 Vectors requests"| vapi - vapi -->|"index and vector operations"| engines - engines -->|"index tables and Lance files"| sapi - sapi -->|"file metadata"| filers - sapi -->|"file chunks"| volumes end - subgraph ent ["SeaweedFS Enterprise: distributed queries"] - direction TB - workers["weed-vector-worker engines
gRPC port 18888, one share of each index"] - end - vapi -.->|"QueryVectors, one request per engine"| workers - workers -.->|"file metadata, read directly"| filers - workers -.->|"file chunks, read directly"| volumes - subgraph side ["cluster services"] - master["master, port 9333
engines register here,
the gateway looks them up"] - end - path -.- side + client -->|"S3, S3 Tables and S3 Vectors requests"| gateway + gateway --> normal + gateway --> table + gateway --> vector + vector -->|"index and vector operations"| engine + engine -->|"table and file metadata"| filers + engine -->|"file chunks"| volumes classDef s3server fill:#dbeafe,stroke:#2563eb,color:#0f172a classDef other fill:#f1f5f9,stroke:#64748b,color:#0f172a - classDef enterprise fill:#fef3c7,stroke:#d97706,color:#0f172a - class vapi,sapi s3server - class client,engines,master,filers,volumes other - class workers enterprise - style path fill:none,stroke:#94a3b8 - style ent fill:none,stroke:#d97706,stroke-dasharray:4 3 - style side fill:none,stroke:#94a3b8 + class gateway s3server + class client,normal,table,vector,engine,filers,volumes other + style buckets fill:none,stroke:#94a3b8 + style store fill:none,stroke:#94a3b8 ``` -Both blue boxes run in the S3 server process: `weed-vector` calls back into an S3 server for the index data, and that S3 server reads the file metadata from the filers and the chunks from the volume servers like any other S3 request. Each index is one Lance dataset under `s3:///vector//`, stored as file chunks on the volume servers. The dashed path is SeaweedFS Enterprise: its search engines (`weed-vector-worker`) answer `QueryVectors` and read the same Lance files straight from the filers and volume servers, skipping the gateway; writes and index management still go through `weed-vector`. +The gateway serves all three bucket kinds on the S3 port. A vector bucket is a table bucket with format `LANCE`; each index is one Lance dataset under `s3:///vector//`. `weed-vector` reads and writes it through the gateway's S3 API, and the gateway, as for any S3 request, keeps the file metadata on the filers and the chunks on the volume servers. * **The S3 gateway** (`weed s3`, or the S3 server inside `weed server -s3` and `weed mini`) answers S3 Vectors requests on the normal S3 port. It tells them apart by the service name in the SigV4 credential scope: a request signed for `s3vectors` goes to the vectors API, everything else to the S3 API. * **The vector engine** is a separate process, `weed-vector`. It registers with the master, and the gateway finds it through the master on the first request that needs it. Bucket and policy operations work without it; index and vector operations need it. Several can register: the open-source gateway sends its requests to one at a time, and SeaweedFS Enterprise spreads each index and its queries over them (see [[S3 Vector Bucket Performance#beyond-one-engine]]).