From b33a4b388998770571d1541c954b2932a4cf17ec Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sun, 1 Nov 2026 16:13:39 -0800 Subject: [PATCH] Add S3 Vector Bucket pages S3 Vector Bucket (setup, engine, limits, LanceDB access, troubleshooting, monitoring), S3 Vectors API, S3 Vector Bucket Security and S3 Vector Bucket Performance, linked from the sidebar, S3 Table Bucket and SeaweedFS Lance Catalog. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --- S3-Table-Bucket.md | 1 + S3-Vector-Bucket-Performance.md | 72 ++++++++ S3-Vector-Bucket-Security.md | 109 ++++++++++++ S3-Vector-Bucket.md | 305 ++++++++++++++++++++++++++++++++ S3-Vectors-API.md | 120 +++++++++++++ SeaweedFS-Lance-Catalog.md | 1 + _Sidebar.md | 6 + 7 files changed, 614 insertions(+) create mode 100644 S3-Vector-Bucket-Performance.md create mode 100644 S3-Vector-Bucket-Security.md create mode 100644 S3-Vector-Bucket.md create mode 100644 S3-Vectors-API.md diff --git a/S3-Table-Bucket.md b/S3-Table-Bucket.md index 577b06f..954ca40 100644 --- a/S3-Table-Bucket.md +++ b/S3-Table-Bucket.md @@ -152,3 +152,4 @@ sequenceDiagram - [SeaweedFS Iceberg Catalog](SeaweedFS-Iceberg-Catalog) - Usage with Apache Spark, Trino, Dremio, and other engines. - [Iceberg Table Maintenance](Iceberg-Table-Maintenance) - Automated compaction, snapshot expiration, orphan removal, and manifest rewriting. - [S3 Tables Security](S3-Tables-Security) - IAM permissions and policy reference. +- [S3 Vector Bucket](S3-Vector-Bucket) - The S3 Vectors API, stored as Lance tables in table buckets. diff --git a/S3-Vector-Bucket-Performance.md b/S3-Vector-Bucket-Performance.md new file mode 100644 index 0000000..59848ed --- /dev/null +++ b/S3-Vector-Bucket-Performance.md @@ -0,0 +1,72 @@ +# S3 Vector Bucket Performance + +Measured numbers for [[S3 Vector Bucket]]s, and how to reproduce them with the perf suite that ships in the repository. + +## The perf suite + +`test/s3vectors/perf_test.py` starts a local cluster (master, volume server, filer, S3 gateway and `weed-vector`), loads random vectors through the S3 Vectors API and times every data-plane operation. It checks correctness, not speed: recall is checked against an exact search done in the client, and every listing must return every vector. + +```bash +cd test/s3vectors +pip install -r requirements.txt # boto3, pytest +make perf # builds weed, weed-vector and weed-volume, then runs the suite +# or, with the binaries already in this directory: +VOLUME_SERVER_IMPL=go ./run_perf.sh +``` + +The Python packages are `boto3` 1.40 or newer and `pytest` 8 or newer. `make perf` needs Go, Rust and `protoc`, as for building `weed-vector` (see [[S3 Vector Bucket#getting-weed-vector]]). + +| Variable | Default | Meaning | +|----------|---------|---------| +| `PERF_VECTORS` | 10000 | vectors to ingest | +| `PERF_DIM` | 128 | dimension | +| `PERF_BATCH` | 500 | vectors per `PutVectors` call | +| `PERF_QUERIES` | 100 | timed queries per scenario | +| `PERF_TOPK` | 10 | `topK` | +| `PERF_CONCURRENCY` | `1,4,8` | client threads for the throughput scenario | +| `PERF_RECALL` | 20 | queries checked against the exact search | +| `PERF_REPORT` | `perf-results.json` in the cluster directory | where the JSON report goes | +| `VOLUME_SERVER_IMPL` | `go,rust` | which volume server(s) to run against | + +Each scenario starts with one untimed call, and the throughput scenario creates and warms its clients before the timer starts. The suite prints a summary and writes all numbers to the JSON report. + +## Results + +One 16-core x86_64 host, every process on it, `cosine` metric, top-10, one client unless noted. Other work kept the host's load average between 3.4 and 4.9 during the runs, so read differences of a few milliseconds, and the Go/Rust columns against each other, as noise. + +| | Go volume, 10k × 128 | Rust volume, 10k × 128 | Go volume, 50k × 384 | Rust volume, 50k × 384 | +|---|---:|---:|---:|---:| +| Ingest (vectors/s) | 4,579 | 4,476 | 1,805 | 1,654 | +| PutVectors, 500 per call, p50 | 109 ms | 111 ms | 272 ms | 282 ms | +| Upsert 500 existing, p50 | 111 ms | 114 ms | 285 ms | 270 ms | +| Query top-10, p50 / p95 | 23 / 43 ms | 21 / 44 ms | 56 / 73 ms | 53 / 73 ms | +| Query + metadata, p50 | 22 ms | 29 ms | 37 ms | 41 ms | +| Query with a 10% filter, p50 | 40 ms | 51 ms | 82 ms | 45 ms | +| Recall@10 | 1.000 | 1.000 | 1.000 | 1.000 | +| Queries/s with 1 / 4 / 8 clients | 33 / 135 / 286 | 29 / 186 / 285 | 18 / 55 / 64 | 25 / 56 / 54 | +| GetVectors, 100 keys, p50 | 53 ms | 52 ms | 58 ms | 60 ms | +| DeleteVectors, 100 keys, p50 | 25 ms | 23 ms | 44 ms | 44 ms | +| ListVectors (vectors/s), 1 / 4 / 16 segments | 18k / 63k / 134k | 24k / 69k / 129k | 18k / 75k / 169k | 16k / 74k / 170k | + +What the numbers show: + +* **Queries scan every vector.** No approximate index is built, so latency grows with the size of the index: going from 10k × 128 to 50k × 384 (15 times the data) raised the median query from about 22 ms to about 55 ms. Recall is 1.000 throughout. +* **Throughput grows with clients until the index gets large.** At 10k × 128, 8 clients reach about 285 queries/s; at 50k × 384 throughput levels off between 4 and 8 clients at 55 to 65 queries/s. +* **Parallel listing scales.** `ListVectors` with 16 segments read the whole index 5 to 11 times faster than one sequential listing. +* **Ingest is steady.** Batch latency did not grow from the first to the last tenth of the load (for example 283 ms → 272 ms at 50k × 384). +* **No consistent difference between the volume servers.** The largest gap, the filtered query at 50k × 384 (82 ms on Go, 45 ms on Rust), goes the other way at 10k × 128 (40 ms against 51 ms). + +## The read cache + +`weed-vector` keeps reads of an index's data files in memory, 512 MiB by default (`--read-cache-mb`, `WEED_VECTOR_READ_CACHE_MB`; `0` disables it). The results above ran with the default. The same 50k × 384 index on the Go volume server, with the cache off and on, run alternately twice: + +| | Cache off (`0`) | Cache 512 MiB | +|---|---:|---:| +| Query top-10, p50 | 127 ms, 127 ms | 48 ms, 36 ms | +| Query with a 10% filter, p50 | 233 ms, 233 ms | 71 ms, 48 ms | +| GetVectors, 100 keys, p50 | 332 ms, 319 ms | 59 ms, 76 ms | +| Queries/s with 1 / 4 / 8 clients | 7.5 / 11.5 / 12.2, 7.8 / 11.6 / 11.9 | 19 / 59 / 59, 26 / 50 / 50 | + +Without the cache, every query reads the index's files back through the S3 gateway and the volume server. The cache cut median query latency to about a third and raised throughput about four to five times, even though other work loaded the host more during the cached runs (load average 19 to 21, against 6 to 10). + +This index holds about 77 MB of raw vector data (50,000 × 384 × 4 bytes), well inside the default cache. An index larger than the cache was not measured. diff --git a/S3-Vector-Bucket-Security.md b/S3-Vector-Bucket-Security.md new file mode 100644 index 0000000..93bedaf --- /dev/null +++ b/S3-Vector-Bucket-Security.md @@ -0,0 +1,109 @@ +# S3 Vector Bucket Security + +Who may call the [[S3 Vectors API]], and how to grant access with a vector bucket policy. Vector bucket policies use the same policy evaluation as [[S3 Tables Security]], with `s3vectors:` actions. + +## Authentication + +Requests are signed with SigV4 for the service `s3vectors`, with an identity from the S3 configuration (`-s3.config`, see [[S3 Credentials]]). An unknown access key is refused with `403 AccessDenied: authentication failed`. + +Configure at least one identity: the `weed-vector` engine needs one to reach the data (see [[S3 Vector Bucket#quick-start]]), so vector indexes do not work on a SeaweedFS without identities. + +## Who can do what + +Access comes from two places: the actions listed for the identity in the S3 configuration, and the bucket's policy. + +| Caller | Access | +|--------|--------| +| The identity named `admin` | Everything on every vector bucket. Bucket policies do not apply to it, not even a `Deny`. | +| An identity with the `Admin` action, or with `s3vectors:*` | Can create vector buckets and use every vector bucket, unless a bucket policy denies it. | +| An identity with specific actions such as `s3vectors:QueryVectors` | Those operations on every vector bucket, unless a bucket policy denies them. `s3vectors:CreateVectorBucket` lets it create buckets. | +| The bucket's owner, i.e. the identity that created it | Everything on that bucket, unless its policy denies it. | +| Any other identity | Only what the bucket's policy allows. | + +Identity actions look like this in the `-s3.config` file: + +```json +{ + "identities": [ + {"name": "search", "credentials": [{"accessKey": "search", "secretKey": "search-secret"}], + "actions": ["s3vectors:GetVectors", "s3vectors:QueryVectors"]} + ] +} +``` + +`search` can then call `GetVectors` and `QueryVectors` on every vector bucket, and nothing else. + +* The `Read`, `Write` and `List` actions do **not** grant anything on vector buckets. An identity with only those got `AccessDeniedException` for every vector operation, including `CreateVectorBucket`. +* A bucket-scoped identity action such as `s3vectors:QueryVectors:team-a` did **not** grant access in our test. To give access to one bucket only, use that bucket's policy. +* `ListVectorBuckets` returns only the buckets on which the caller has `s3vectors:ListVectorBuckets` or `s3vectors:GetVectorBucket`. The `search` identity above can query every bucket but lists none. + +## Vector bucket policies + +Attach a policy with `PutVectorBucketPolicy`. The caller needs `s3vectors:PutVectorBucketPolicy`: an admin identity has it, and so does an identity the bucket's current policy grants it. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": {"AWS": "reader"}, + "Action": [ + "s3vectors:GetVectorBucket", + "s3vectors:ListIndexes", + "s3vectors:GetIndex", + "s3vectors:GetVectors", + "s3vectors:ListVectors", + "s3vectors:QueryVectors" + ], + "Resource": [ + "arn:aws:s3vectors:us-east-1:000000000000:bucket/team-a", + "arn:aws:s3vectors:us-east-1:000000000000:bucket/team-a/index/*" + ] + } + ] +} +``` + +```python +s3v.put_vector_bucket_policy(vectorBucketName="team-a", policy=json.dumps(policy)) +``` + +With this policy, identity `reader` can list the bucket, its indexes and their vectors, and query them; `PutVectors` and `DeleteIndex` still get `AccessDeniedException`. + +* **Principal**: an identity name from the S3 configuration, as `{"AWS": "reader"}` or a list of names, or `"*"` for every authenticated identity. +* **Action**: `s3vectors:` followed by the operation name, e.g. `s3vectors:QueryVectors`, or `s3vectors:*`. Each operation of the [[S3 Vectors API]] checks the action of the same name. +* **Resource**: the bucket ARN, an index ARN, or a wildcard such as `arn:aws:s3vectors:us-east-1:000000000000:bucket/team-a/index/*`. ARNs always use region `us-east-1` and account `000000000000`. +* **Deny wins**: a matching `Deny` overrides any `Allow`, the owner's access and the identity's own actions. Only the identity named `admin` is exempt. + +Example: let every identity query one index, but shut out `writer` entirely: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + {"Effect": "Allow", "Principal": "*", "Action": "s3vectors:QueryVectors", + "Resource": "arn:aws:s3vectors:us-east-1:000000000000:bucket/team-a/index/idx"}, + {"Effect": "Deny", "Principal": {"AWS": "writer"}, "Action": "s3vectors:*", "Resource": "*"} + ] +} +``` + +An identity with no vector grants of its own can then call `QueryVectors` on `idx`, but not `GetVectors`; `writer` gets `AccessDeniedException` even for `QueryVectors`. + +`GetVectorBucketPolicy` returns the document as it was stored; with no policy it returns `NotFoundException`. `DeleteVectorBucketPolicy` removes it, and with it every access the policy granted. + +## The engine's identity + +`weed-vector` reads and writes every index's files with the identity passed in its `--storage-option` flags, whoever called the API. Authorization happens in the gateway before the engine is called. In our tests the engine used an identity with the `Admin` action. + +## Rejected policies + +`PutVectorBucketPolicy` refuses, with `ValidationException`, a policy that it could not enforce: + +| Policy content | Message | +|----------------|---------| +| An action of another service, e.g. `s3tables:GetTable` or `s3:GetObject` | `action "s3:GetObject" is not enforceable in a vector bucket policy` | +| A resource ARN of another service, e.g. `arn:aws:s3tables:...` | `resource "arn:aws:s3tables:..." is not enforceable in a vector bucket policy` | +| `NotAction` | `NotAction is not supported` | +| `NotResource` | `NotResource is not supported` | diff --git a/S3-Vector-Bucket.md b/S3-Vector-Bucket.md new file mode 100644 index 0000000..08b740b --- /dev/null +++ b/S3-Vector-Bucket.md @@ -0,0 +1,305 @@ +# S3 Vector Bucket + +SeaweedFS serves the [Amazon S3 Vectors](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors.html) API: vector buckets, vector indexes, and `PutVectors` / `QueryVectors` similarity search. Point the AWS CLI or an AWS SDK's `s3vectors` client at the SeaweedFS S3 endpoint and make the same calls; [[S3 Vectors API]] lists what differs from AWS. + +* [[S3 Vectors API]]: supported operations, distance metrics, metadata filters, limits +* [[S3 Vector Bucket Security]]: who can do what, and vector bucket policies +* [[S3 Vector Bucket Performance]]: the perf suite and measured numbers + +## How it works + +``` + aws s3vectors / boto3 / any AWS SDK + │ SigV4, service name "s3vectors" + ▼ + S3 gateway :8333 ── buckets, indexes and policies live in the filer catalog + │ gRPC, engine address looked up on the master + ▼ + weed-vector ── Lance catalog :9101 + S3 :8333 ──► s3:///vector// + (one Lance dataset per index) +``` + +* **The S3 gateway** (`weed s3`, or the S3 server inside `weed server -s3` and `weed mini`) answers S3 Vectors requests on the normal S3 port. It tells them apart by the service name in the SigV4 credential scope: a request signed for `s3vectors` goes to the vectors API, everything else to the S3 API. +* **The vector engine** is a separate process, `weed-vector`. It registers with the master, and the gateway finds it through the master on the first request that needs it. Bucket and policy operations work without it; index and vector operations need it. +* **The data** is stored as [Lance](https://lancedb.com) datasets. A vector bucket is a table bucket with format `LANCE`, and each index is a Lance table in the bucket's `vector` namespace, stored under `s3:///vector//`. The engine reaches it through the [[SeaweedFS Lance Catalog]] (`-s3.port.lance`, default `9101`, on by default) and the S3 port. + +**Queries are exact.** No approximate (ANN) index is built, so every query compares against all vectors of the index; recall@10 was 1.000 in every run of the perf suite. Query time grows with the size of the index; see [[S3 Vector Bucket Performance]]. + +## Getting weed-vector + +`weed-vector` is not part of the release archives or the Docker images. Build it from source with Rust 1.94.1 or newer and `protoc`, which Lance's build scripts need: + +```bash +sudo apt-get install -y protobuf-compiler # Debian/Ubuntu; provides protoc +git clone https://github.com/seaweedfs/seaweedfs.git +cd seaweedfs +cargo build --release --locked --manifest-path seaweed-vector/Cargo.toml -p weed-vector-server +# the binary is seaweed-vector/target/release/weed-vector +``` + +Without `protoc` the build stops in `lance-encoding` with `Could not find protoc`. + +## Quick start + +**1. Start SeaweedFS with an S3 identity.** `weed mini` turns the two variables into an identity with admin rights: + +```bash +export AWS_ACCESS_KEY_ID=admin +export AWS_SECRET_ACCESS_KEY=admin-secret +weed mini -dir=/data/seaweedfs +``` + +**2. Start the engine in another terminal**, pointing it at the master and at the Lance catalog, and giving it the same identity: + +```bash +weed-vector --masters localhost:9333 --namespace http://localhost:9101 \ + --storage-option aws_access_key_id=admin \ + --storage-option aws_secret_access_key=admin-secret +``` + +The engine needs a real S3 identity: it reads and writes the Lance files over S3 with it, and authenticates to the Lance catalog with it. A SeaweedFS with no identities at all does not work: `CreateIndex` then fails with `InternalFailure`. + +**3. Create a bucket and an index, write vectors, query:** + +```bash +export AWS_ACCESS_KEY_ID=admin +export AWS_SECRET_ACCESS_KEY=admin-secret +export AWS_DEFAULT_REGION=us-east-1 + +aws s3vectors create-vector-bucket --endpoint-url http://localhost:8333 \ + --vector-bucket-name media + +aws s3vectors create-index --endpoint-url http://localhost:8333 \ + --vector-bucket-name media --index-name movies \ + --data-type float32 --dimension 3 --distance-metric cosine + +aws s3vectors put-vectors --endpoint-url http://localhost:8333 \ + --vector-bucket-name media --index-name movies \ + --vectors '[ + {"key": "star-wars", "data": {"float32": [0.1, 0.9, 0.2]}, "metadata": {"genre": "scifi", "year": 1977}}, + {"key": "alien", "data": {"float32": [0.2, 0.8, 0.1]}, "metadata": {"genre": "scifi", "year": 1979}}, + {"key": "amelie", "data": {"float32": [0.9, 0.1, 0.3]}, "metadata": {"genre": "romance", "year": 2001}} + ]' + +aws s3vectors query-vectors --endpoint-url http://localhost:8333 \ + --vector-bucket-name media --index-name movies \ + --query-vector '{"float32": [0.15, 0.85, 0.15]}' --top-k 2 \ + --return-distance --return-metadata +``` + +```json +{ + "vectors": [ + { + "distance": 0.0029967427, + "key": "star-wars", + "metadata": { + "genre": "scifi", + "year": 1977 + } + }, + { + "distance": 0.0037363768, + "key": "alien", + "metadata": { + "genre": "scifi", + "year": 1979 + } + } + ], + "distanceMetric": "cosine" +} +``` + +With a metadata filter, only vectors from 1978 on are candidates: + +```bash +aws s3vectors query-vectors --endpoint-url http://localhost:8333 \ + --vector-bucket-name media --index-name movies \ + --query-vector '{"float32": [0.15, 0.85, 0.15]}' --top-k 2 \ + --filter '{"year": {"$gte": 1978}}' --return-metadata +``` + +```json +{ + "vectors": [ + { + "key": "alien", + "metadata": { + "genre": "scifi", + "year": 1979 + } + }, + { + "key": "amelie", + "metadata": { + "genre": "romance", + "year": 2001 + } + } + ], + "distanceMetric": "cosine" +} +``` + +`get-vectors`, `list-vectors` and `delete-vectors` work the same way. + +**The same with boto3:** + +```python +import boto3 + +s3v = boto3.client( + "s3vectors", + endpoint_url="http://localhost:8333", + aws_access_key_id="admin", + aws_secret_access_key="admin-secret", + region_name="us-east-1", +) + +s3v.create_vector_bucket(vectorBucketName="books") +s3v.create_index( + vectorBucketName="books", indexName="titles", + dataType="float32", dimension=4, distanceMetric="euclidean", + metadataConfiguration={"nonFilterableMetadataKeys": ["summary"]}, +) +s3v.put_vectors( + vectorBucketName="books", indexName="titles", + vectors=[ + {"key": "dune", "data": {"float32": [0.9, 0.1, 0.0, 0.2]}, + "metadata": {"genre": "scifi", "pages": 412, "summary": "Desert planet politics."}}, + {"key": "emma", "data": {"float32": [0.1, 0.9, 0.3, 0.0]}, + "metadata": {"genre": "classic", "pages": 474, "summary": "Matchmaking in Highbury."}}, + {"key": "neuromancer", "data": {"float32": [0.8, 0.2, 0.1, 0.3]}, + "metadata": {"genre": "scifi", "pages": 271, "summary": "Cyberspace heist."}}, + ], +) + +result = s3v.query_vectors( + vectorBucketName="books", indexName="titles", + queryVector={"float32": [0.85, 0.15, 0.05, 0.25]}, topK=3, + filter={"$and": [{"genre": {"$eq": "scifi"}}, {"pages": {"$lt": 400}}]}, + returnDistance=True, returnMetadata=True, +) +for hit in result["vectors"]: + print(hit["key"], round(hit["distance"], 4), hit["metadata"]) +``` + +``` +neuromancer 0.01 {'genre': 'scifi', 'pages': 271, 'summary': 'Cyberspace heist.'} +``` + +`summary` is non-filterable: it is stored and returned, but a filter on it is rejected. + +## Running the engine + +| Flag | Environment variable | Default | Meaning | +|------|----------------------|---------|---------| +| `--masters` | `WEED_VECTOR_MASTERS` | none | Comma-separated master addresses to register with. Without it the gateway never finds the engine. | +| `--namespace` | `WEED_VECTOR_NAMESPACE` | `http://localhost:8888/lance` | The Lance catalog URL. Set it to the S3 server's Lance port, e.g. `http://localhost:9101`; the default does not work with `weed mini` or `weed server`. | +| `--listen` | `WEED_VECTOR_LISTEN` | `127.0.0.1:8080` | gRPC address. This is the address registered on the master, so the gateway must be able to reach it. | +| `--storage-option key=value` | | none | Repeatable. `aws_access_key_id` and `aws_secret_access_key` are used for S3 data access and for the Lance catalog. | +| `--read-cache-mb` | `WEED_VECTOR_READ_CACHE_MB` | `512` | Memory for caching reads of the index's (write-once) data files, in MiB. `0` disables it. | + +* **Port clash with `weed server`.** Its volume server uses `8080` and `18080` (gRPC), so start the engine with another address, e.g. `--listen 127.0.0.1:8090`. With the default it exits with `Address already in use (os error 98)`. `weed mini` puts its volume server on `9340`, so the default is free there. +* **Engine on another host.** With a wildcard address such as `--listen 0.0.0.0:8090`, the engine registers the IP it uses to reach the first master. + +* **A standby engine.** The gateway sends every request to one registered engine at a time. Start a second engine (with its own `--listen`) and it takes over when the first stops: in our test, queries kept succeeding right after the engine in use was stopped. + +## Limits + +| What | Limit | +|------|-------| +| Data type | `float32` only | +| Dimension | 1 to 4096 | +| Distance metric | `euclidean` (default), `cosine`, `dotproduct` | +| Vectors per `PutVectors` | 1 to 500 | +| Keys per `GetVectors` | 1 to 100 | +| Keys per `DeleteVectors` | 1 to 500 | +| `topK` | 1 to 10000 | +| `ListVectors` page | 500 by default, `maxResults` up to 1000 | +| `ListVectors` segments | up to 16 | +| `ListVectorBuckets` / `ListIndexes` `maxResults` | up to 500 | +| Request body | 32 MiB | + +Bucket names are 3 to 63 lowercase letters, digits and hyphens, starting and ending with a letter or digit. Index names use lowercase letters, digits, `-` and `_`, and start with a letter or digit. See [[S3 Vectors API]] for the details. + +## Reading an index with LanceDB + +Each index is an ordinary Lance table in namespace `[, "vector"]`, so Lance tools can read it through the [[SeaweedFS Lance Catalog]]. The catalog takes the S3 credential as an `x-api-key` header of the form `access_key:secret_key`: + +```python +import lancedb + +db = lancedb.connect_namespace( + "rest", + {"uri": "http://localhost:9101", "header.x-api-key": "admin:admin-secret"}, + storage_options={ + "aws_endpoint": "http://localhost:8333", + "allow_http": "true", + "aws_access_key_id": "admin", + "aws_secret_access_key": "admin-secret", + "aws_region": "us-east-1", + }, +) +table = db.open_table("docs", namespace_path=["interop", "vector"]) +print(db.list_tables(namespace_path=["interop", "vector"]).tables) +print(table.schema) +print(table.search([1, 0, 0]).limit(1).to_list()) +``` + +For an index `docs` (dimension 3, `cosine`) in bucket `interop`, holding two vectors: + +``` +['interop$vector$docs'] +key: string not null +vector: fixed_size_list[3] not null + child 0, item: float + -- field metadata -- + distance_metric: 'COSINE' +metadata: large_binary + -- field metadata -- + ARROW:extension:name: 'lance.json' +[{'key': 'a', 'vector': [1.0, 0.0, 0.0], 'metadata': '{"lang":"en"}', '_distance': 0.0}] +``` + +The table has three columns: `key`, `vector` (with the metric in the field metadata) and `metadata` (JSON). + +**Treat it as read-only from Lance.** The catalog refuses to drop, rename, register or deregister tables in the `vector` namespace (`the vector namespace of a vector bucket is read-only; manage its indexes through S3 Vectors`). Appending rows with LanceDB does reach the dataset, and the S3 Vectors API then returns them, but such writes bypass the API and its validation. + +## Troubleshooting + +Start the S3 server (or `weed mini` / `weed server`) with `-v=1` to see the vectors log lines. When the engine registers, the master logs: + +``` +Cluster: vector node 127.0.0.1:8080 added to group '' +``` + +and on the first request that needs the engine, the gateway logs: + +``` +S3 Vectors: discovered weed-vector at 127.0.0.1:8080 via master 172.17.0.3:9333 +``` + +| Symptom | Cause | +|---------|-------| +| `ServiceUnavailableException: vector service unavailable` on `CreateIndex`, `PutVectors` or `QueryVectors` | No engine is registered: it is not running, it has no `--masters`, or it exited (for example on a busy `--listen` port). Bucket operations keep working. | +| `InternalFailure: internal error` on `CreateIndex`, and with `-v=1` the gateway logs `S3 Vectors: engine error: ... 403 Forbidden for url (http://localhost:9101/v1/table/.../declare)` | The engine has no credential for the Lance catalog. Pass `--storage-option aws_access_key_id=...` and `--storage-option aws_secret_access_key=...`. | +| The same, with `401 Unauthorized` | The engine's access key is unknown or its secret is wrong. | +| The same, with `500 Internal Server Error for url (http://localhost:8888/lance/v1/table/...)` | `--namespace` was left at its default; point it at the Lance port, e.g. `http://localhost:9101`. | +| The same, with `Failed to get AWS credentials: CredentialsNotLoaded` | The engine has no storage credential at all (seen on a SeaweedFS without identities). | +| `weed-vector` exits with `Address already in use (os error 98)` | The `--listen` port is taken; see the port clash note above. | +| `PutVectors` fails with `403 AccessDenied: authentication failed` although the credential is right | The request body is over 32 MiB. Send fewer vectors per call. | +| A plain HTTP request to `/ListVectorBuckets` gets `405` | Only requests signed for the `s3vectors` service reach the vectors API. Use an AWS CLI or SDK `s3vectors` client. | + +A failed `CreateIndex` leaves no index behind, so it is safe to retry once the engine is fixed. + +## Monitoring + +S3 Vectors requests are counted in the S3 server's Prometheus metrics, with the operation in the `type` label as `s3vectors.`. Enable them with `-metricsPort`; the listener binds to the server's IP address, not to `localhost`. + +``` +SeaweedFS_s3_request_total{bucket="",code="200",type="s3vectors.QueryVectors"} 1 +SeaweedFS_s3_request_seconds_bucket{bucket="",type="s3vectors.CreateIndex",le="0.0001"} 0 +SeaweedFS_s3_in_flight_requests{type="s3vectors.QueryVectors"} 0 +``` diff --git a/S3-Vectors-API.md b/S3-Vectors-API.md new file mode 100644 index 0000000..e398a1f --- /dev/null +++ b/S3-Vectors-API.md @@ -0,0 +1,120 @@ +# S3 Vectors API + +What the SeaweedFS S3 Vectors API supports, and where it differs from AWS. For setup see [[S3 Vector Bucket]]; for permissions see [[S3 Vector Bucket Security]]. + +## Operations + +Requests go to the S3 port, signed with SigV4 for the service `s3vectors`. Use an AWS CLI or SDK `s3vectors` client with its endpoint set to the SeaweedFS S3 address; every example on these pages was run with the AWS CLI and boto3. + +| Operation | Supported | +|-----------|-----------| +| `CreateVectorBucket`, `GetVectorBucket`, `ListVectorBuckets`, `DeleteVectorBucket` | yes | +| `PutVectorBucketPolicy`, `GetVectorBucketPolicy`, `DeleteVectorBucketPolicy` | yes | +| `CreateIndex`, `GetIndex`, `ListIndexes`, `DeleteIndex` | yes | +| `PutVectors`, `GetVectors`, `ListVectors`, `DeleteVectors`, `QueryVectors` | yes | +| `TagResource`, `UntagResource`, `ListTagsForResource` | no: HTTP 404 | +| `UpdateIndexMode`, `PutVectorBucketDefaultIndexMode` | no: `ValidationException: unknown operation` | + +Every bucket and index can be addressed by name or by ARN. ARNs always use region `us-east-1` and account `000000000000`, e.g. `arn:aws:s3vectors:us-east-1:000000000000:bucket/media/index/movies`; an ARN with another partition, region or account, or a name that disagrees with the ARN, is rejected with `ValidationException`. The region a request is signed for is not checked. + +## Vector buckets + +* **Names**: 3 to 63 characters, lowercase letters, digits and hyphens, starting and ending with a letter or digit. +* **Deleting** a bucket that still has indexes fails with `ConflictException`. +* **Encryption**: `encryptionConfiguration` accepts `{"sseType": "AES256"}` and returns it from `GetVectorBucket`; `aws:kms` is rejected with `unsupported sseType "aws:kms"`. **The setting is only recorded: the index files are written without server-side encryption.** A `HeadObject` on an index data file in an `AES256` bucket shows no `ServerSideEncryption`, exactly as in a bucket without the setting. +* A vector bucket is a table bucket with format `LANCE`, and its objects are visible through the S3 API under `vector//`. + +## Indexes + +```python +s3v.create_index( + vectorBucketName="media", indexName="movies", + dataType="float32", dimension=384, distanceMetric="cosine", + metadataConfiguration={"nonFilterableMetadataKeys": ["description"]}, +) +``` + +* **Names**: lowercase letters, digits, `-` and `_`, starting with a letter or digit. The AWS CLI and SDKs require 3 to 63 characters before sending. +* **`dataType`**: `float32` only (`only float32 vectors are supported`). +* **`dimension`**: 1 to 4096 (`dimension must be 1..=4096`). +* **`nonFilterableMetadataKeys`**: those keys are stored and returned, but a filter that names one is rejected with `metadata key "description" is not filterable`. +* Deleting an index removes its vectors. An index created again under the same name starts empty and may have a different dimension or metric. + +## Distance metrics + +The metric is fixed when the index is created and used by every query; it is returned by `GetIndex` and in each `QueryVectors` response as `distanceMetric`. The metric name is not case-sensitive. + +| `distanceMetric` | `distance` in results | Example: query `[1, 0]` | +|------------------|-----------------------|-------------------------| +| `euclidean` (default when omitted) | **squared** Euclidean distance | `[3, 4]` from `[0, 0]` → `25.0` | +| `cosine` | 1 − cosine similarity | `[10, 0]` → `0.0` | +| `dotproduct` | 1 − dot product | `[2, 0]` → `-1.0`, `[1, 1]` → `0.0`, `[-1, 0]` → `2.0` | + +Results are ordered by ascending `distance`. `dotproduct` is a SeaweedFS addition; AWS offers `euclidean` and `cosine`. Any other name is rejected with `unknown distanceMetric`. + +## Writing vectors + +```python +s3v.put_vectors(vectorBucketName="media", indexName="movies", vectors=[ + {"key": "alien", "data": {"float32": [0.2, 0.8, 0.1]}, + "metadata": {"genre": "scifi", "year": 1979, "cast": {"lead": "Weaver"}, "tags": ["space", "horror"]}}, +]) +``` + +* **`PutVectors` is an upsert by key**: writing an existing key replaces its vector and metadata. +* **A batch is all or nothing**: 1 to 500 vectors, and if any vector has the wrong dimension, or a key appears twice in the batch, the whole request fails with `ValidationException` and nothing is written. +* **Metadata** is any JSON object: strings, numbers, booleans, arrays and nested objects come back unchanged. Keys may contain characters such as `'`. +* **`DeleteVectors`** takes 1 to 500 keys. +* The request body is limited to 32 MiB. A larger `PutVectors` is refused with `403 AccessDenied: authentication failed`, not with a size error. 500 vectors of 4096 dimensions fit (a 9.8 MiB body in our test). + +## Reading vectors + +* **`GetVectors`** takes 1 to 100 keys; keys that do not exist are left out of the result. Data and metadata are returned only when `returnData` / `returnMetadata` are set. +* **`ListVectors`** pages through every vector: 500 per page by default, `maxResults` up to 1000, continued with `nextToken`. An invalid `nextToken` is rejected. +* **Parallel listing**: pass `segmentCount` (up to 16) and a `segmentIndex` from 0 to `segmentCount - 1` to list one disjoint part of the index; the parts together hold every vector exactly once. `segmentIndex` without `segmentCount` is rejected. + +## Querying + +```python +s3v.query_vectors( + vectorBucketName="media", indexName="movies", + queryVector={"float32": [0.15, 0.85, 0.15]}, topK=10, + filter={"$and": [{"genre": "scifi"}, {"year": {"$gte": 1978}}]}, + returnDistance=True, returnMetadata=True, +) +``` + +* **`topK`**: 1 to 10000. `topK` larger than the index returns every vector. +* The query vector must have the index's dimension. +* `distance` is in the result only with `returnDistance`, and `metadata` only with `returnMetadata`. +* **Search is exact**: every query compares against every vector in the index; no approximate index is built. + +## Metadata filters + +| Filter | Matches | +|--------|---------| +| `{"genre": "scifi"}` | `genre` equals `"scifi"` | +| `{"genre": {"$eq": "scifi"}}`, `{"$ne": ...}` | equal / not equal | +| `{"year": {"$gt": 1977}}`, `$gte`, `$lt`, `$lte` | numeric comparison; the operand must be a number (`$gt expects a number`) | +| `{"genre": {"$in": ["drama", "western"]}}`, `$nin` | equal to one of / none of the values | +| `{"tags": {"$exists": true}}` | the key is present (or absent, with `false`) | +| `{"$and": [ ... ]}`, `{"$or": [ ... ]}` | all / any of the sub-filters | +| `{"genre": "scifi", "year": 2014}` | several keys in one object: all must match | + +Behaviour to know: + +* **Types are compared strictly.** `1`, `"1"`, `true` and `"true"` are four different values, for `$eq` as well as `$in` and `$nin`. +* **Only top-level scalar values match.** A filter does not look inside arrays: with `"tags": ["space", "classic"]`, neither `{"tags": "space"}` nor `{"tags": {"$in": ["space"]}}` matches; only `$exists` sees the key. A dotted path such as `{"info.lang": "fr"}` does not reach into a nested object either. +* An unknown operator is rejected (`unsupported operator $regex`), and so is a filter on a non-filterable key. + +## Errors + +| Error | When | +|-------|------| +| `ValidationException` (400) | invalid parameters, limits exceeded, invalid policy | +| `AccessDeniedException` (403) | not allowed; see [[S3 Vector Bucket Security]] | +| `AccessDenied` (403) | the credential is unknown or the signature is wrong | +| `NotFoundException` (404) | bucket, index or policy does not exist | +| `ConflictException` (409) | bucket or index already exists; bucket still has indexes | +| `ServiceUnavailableException` (503) | no `weed-vector` engine is registered | +| `InternalFailure` (500) | an engine error; the gateway logs it with `-v=1` (see [[S3 Vector Bucket#troubleshooting]]) | diff --git a/SeaweedFS-Lance-Catalog.md b/SeaweedFS-Lance-Catalog.md index 80c1dc9..d8f5099 100644 --- a/SeaweedFS-Lance-Catalog.md +++ b/SeaweedFS-Lance-Catalog.md @@ -138,6 +138,7 @@ Table buckets show the format they hold and the endpoint that serves them. For a ## See also +- [[S3 Vector Bucket]]: a vector bucket's indexes are Lance tables in its `vector` namespace; this catalog serves them but refuses to drop, rename or re-register them - [[LanceDB Integration]] - [[Spark Lance Integration]] - [[DuckDB Lance Integration]] diff --git a/_Sidebar.md b/_Sidebar.md index 8f8048d..a78f6fa 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -118,6 +118,12 @@ * [[SeaweedFS Lance Catalog]] * [[Lance Maintenance Worker]] +### S3 Vector Bucket +* [[S3 Vector Bucket]] +* [[S3 Vectors API]] +* [[S3 Vector Bucket Security]] +* [[S3 Vector Bucket Performance]] + ### Iceberg Integrations * [[Spark Iceberg Integration]] * [[Trino Iceberg Integration]]