diff --git a/S3-Table-Bucket.md b/S3-Table-Bucket.md index 577b06f..954ca40 100644 --- a/S3-Table-Bucket.md +++ b/S3-Table-Bucket.md @@ -152,3 +152,4 @@ sequenceDiagram - [SeaweedFS Iceberg Catalog](SeaweedFS-Iceberg-Catalog) - Usage with Apache Spark, Trino, Dremio, and other engines. - [Iceberg Table Maintenance](Iceberg-Table-Maintenance) - Automated compaction, snapshot expiration, orphan removal, and manifest rewriting. - [S3 Tables Security](S3-Tables-Security) - IAM permissions and policy reference. +- [S3 Vector Bucket](S3-Vector-Bucket) - The S3 Vectors API, stored as Lance tables in table buckets. diff --git a/S3-Vector-Bucket-Performance.md b/S3-Vector-Bucket-Performance.md new file mode 100644 index 0000000..59848ed --- /dev/null +++ b/S3-Vector-Bucket-Performance.md @@ -0,0 +1,72 @@ +# S3 Vector Bucket Performance + +Measured numbers for [[S3 Vector Bucket]]s, and how to reproduce them with the perf suite that ships in the repository. + +## The perf suite + +`test/s3vectors/perf_test.py` starts a local cluster (master, volume server, filer, S3 gateway and `weed-vector`), loads random vectors through the S3 Vectors API and times every data-plane operation. It checks correctness, not speed: recall is checked against an exact search done in the client, and every listing must return every vector. + +```bash +cd test/s3vectors +pip install -r requirements.txt # boto3, pytest +make perf # builds weed, weed-vector and weed-volume, then runs the suite +# or, with the binaries already in this directory: +VOLUME_SERVER_IMPL=go ./run_perf.sh +``` + +The Python packages are `boto3` 1.40 or newer and `pytest` 8 or newer. `make perf` needs Go, Rust and `protoc`, as for building `weed-vector` (see [[S3 Vector Bucket#getting-weed-vector]]). + +| Variable | Default | Meaning | +|----------|---------|---------| +| `PERF_VECTORS` | 10000 | vectors to ingest | +| `PERF_DIM` | 128 | dimension | +| `PERF_BATCH` | 500 | vectors per `PutVectors` call | +| `PERF_QUERIES` | 100 | timed queries per scenario | +| `PERF_TOPK` | 10 | `topK` | +| `PERF_CONCURRENCY` | `1,4,8` | client threads for the throughput scenario | +| `PERF_RECALL` | 20 | queries checked against the exact search | +| `PERF_REPORT` | `perf-results.json` in the cluster directory | where the JSON report goes | +| `VOLUME_SERVER_IMPL` | `go,rust` | which volume server(s) to run against | + +Each scenario starts with one untimed call, and the throughput scenario creates and warms its clients before the timer starts. The suite prints a summary and writes all numbers to the JSON report. + +## Results + +One 16-core x86_64 host, every process on it, `cosine` metric, top-10, one client unless noted. Other work kept the host's load average between 3.4 and 4.9 during the runs, so read differences of a few milliseconds, and the Go/Rust columns against each other, as noise. + +| | Go volume, 10k × 128 | Rust volume, 10k × 128 | Go volume, 50k × 384 | Rust volume, 50k × 384 | +|---|---:|---:|---:|---:| +| Ingest (vectors/s) | 4,579 | 4,476 | 1,805 | 1,654 | +| PutVectors, 500 per call, p50 | 109 ms | 111 ms | 272 ms | 282 ms | +| Upsert 500 existing, p50 | 111 ms | 114 ms | 285 ms | 270 ms | +| Query top-10, p50 / p95 | 23 / 43 ms | 21 / 44 ms | 56 / 73 ms | 53 / 73 ms | +| Query + metadata, p50 | 22 ms | 29 ms | 37 ms | 41 ms | +| Query with a 10% filter, p50 | 40 ms | 51 ms | 82 ms | 45 ms | +| Recall@10 | 1.000 | 1.000 | 1.000 | 1.000 | +| Queries/s with 1 / 4 / 8 clients | 33 / 135 / 286 | 29 / 186 / 285 | 18 / 55 / 64 | 25 / 56 / 54 | +| GetVectors, 100 keys, p50 | 53 ms | 52 ms | 58 ms | 60 ms | +| DeleteVectors, 100 keys, p50 | 25 ms | 23 ms | 44 ms | 44 ms | +| ListVectors (vectors/s), 1 / 4 / 16 segments | 18k / 63k / 134k | 24k / 69k / 129k | 18k / 75k / 169k | 16k / 74k / 170k | + +What the numbers show: + +* **Queries scan every vector.** No approximate index is built, so latency grows with the size of the index: going from 10k × 128 to 50k × 384 (15 times the data) raised the median query from about 22 ms to about 55 ms. Recall is 1.000 throughout. +* **Throughput grows with clients until the index gets large.** At 10k × 128, 8 clients reach about 285 queries/s; at 50k × 384 throughput levels off between 4 and 8 clients at 55 to 65 queries/s. +* **Parallel listing scales.** `ListVectors` with 16 segments read the whole index 5 to 11 times faster than one sequential listing. +* **Ingest is steady.** Batch latency did not grow from the first to the last tenth of the load (for example 283 ms → 272 ms at 50k × 384). +* **No consistent difference between the volume servers.** The largest gap, the filtered query at 50k × 384 (82 ms on Go, 45 ms on Rust), goes the other way at 10k × 128 (40 ms against 51 ms). + +## The read cache + +`weed-vector` keeps reads of an index's data files in memory, 512 MiB by default (`--read-cache-mb`, `WEED_VECTOR_READ_CACHE_MB`; `0` disables it). The results above ran with the default. The same 50k × 384 index on the Go volume server, with the cache off and on, run alternately twice: + +| | Cache off (`0`) | Cache 512 MiB | +|---|---:|---:| +| Query top-10, p50 | 127 ms, 127 ms | 48 ms, 36 ms | +| Query with a 10% filter, p50 | 233 ms, 233 ms | 71 ms, 48 ms | +| GetVectors, 100 keys, p50 | 332 ms, 319 ms | 59 ms, 76 ms | +| Queries/s with 1 / 4 / 8 clients | 7.5 / 11.5 / 12.2, 7.8 / 11.6 / 11.9 | 19 / 59 / 59, 26 / 50 / 50 | + +Without the cache, every query reads the index's files back through the S3 gateway and the volume server. The cache cut median query latency to about a third and raised throughput about four to five times, even though other work loaded the host more during the cached runs (load average 19 to 21, against 6 to 10). + +This index holds about 77 MB of raw vector data (50,000 × 384 × 4 bytes), well inside the default cache. An index larger than the cache was not measured. diff --git a/S3-Vector-Bucket-Security.md b/S3-Vector-Bucket-Security.md new file mode 100644 index 0000000..93bedaf --- /dev/null +++ b/S3-Vector-Bucket-Security.md @@ -0,0 +1,109 @@ +# S3 Vector Bucket Security + +Who may call the [[S3 Vectors API]], and how to grant access with a vector bucket policy. Vector bucket policies use the same policy evaluation as [[S3 Tables Security]], with `s3vectors:` actions. + +## Authentication + +Requests are signed with SigV4 for the service `s3vectors`, with an identity from the S3 configuration (`-s3.config`, see [[S3 Credentials]]). An unknown access key is refused with `403 AccessDenied: authentication failed`. + +Configure at least one identity: the `weed-vector` engine needs one to reach the data (see [[S3 Vector Bucket#quick-start]]), so vector indexes do not work on a SeaweedFS without identities. + +## Who can do what + +Access comes from two places: the actions listed for the identity in the S3 configuration, and the bucket's policy. + +| Caller | Access | +|--------|--------| +| The identity named `admin` | Everything on every vector bucket. Bucket policies do not apply to it, not even a `Deny`. | +| An identity with the `Admin` action, or with `s3vectors:*` | Can create vector buckets and use every vector bucket, unless a bucket policy denies it. | +| An identity with specific actions such as `s3vectors:QueryVectors` | Those operations on every vector bucket, unless a bucket policy denies them. `s3vectors:CreateVectorBucket` lets it create buckets. | +| The bucket's owner, i.e. the identity that created it | Everything on that bucket, unless its policy denies it. | +| Any other identity | Only what the bucket's policy allows. | + +Identity actions look like this in the `-s3.config` file: + +```json +{ + "identities": [ + {"name": "search", "credentials": [{"accessKey": "search", "secretKey": "search-secret"}], + "actions": ["s3vectors:GetVectors", "s3vectors:QueryVectors"]} + ] +} +``` + +`search` can then call `GetVectors` and `QueryVectors` on every vector bucket, and nothing else. + +* The `Read`, `Write` and `List` actions do **not** grant anything on vector buckets. An identity with only those got `AccessDeniedException` for every vector operation, including `CreateVectorBucket`. +* A bucket-scoped identity action such as `s3vectors:QueryVectors:team-a` did **not** grant access in our test. To give access to one bucket only, use that bucket's policy. +* `ListVectorBuckets` returns only the buckets on which the caller has `s3vectors:ListVectorBuckets` or `s3vectors:GetVectorBucket`. The `search` identity above can query every bucket but lists none. + +## Vector bucket policies + +Attach a policy with `PutVectorBucketPolicy`. The caller needs `s3vectors:PutVectorBucketPolicy`: an admin identity has it, and so does an identity the bucket's current policy grants it. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": {"AWS": "reader"}, + "Action": [ + "s3vectors:GetVectorBucket", + "s3vectors:ListIndexes", + "s3vectors:GetIndex", + "s3vectors:GetVectors", + "s3vectors:ListVectors", + "s3vectors:QueryVectors" + ], + "Resource": [ + "arn:aws:s3vectors:us-east-1:000000000000:bucket/team-a", + "arn:aws:s3vectors:us-east-1:000000000000:bucket/team-a/index/*" + ] + } + ] +} +``` + +```python +s3v.put_vector_bucket_policy(vectorBucketName="team-a", policy=json.dumps(policy)) +``` + +With this policy, identity `reader` can list the bucket, its indexes and their vectors, and query them; `PutVectors` and `DeleteIndex` still get `AccessDeniedException`. + +* **Principal**: an identity name from the S3 configuration, as `{"AWS": "reader"}` or a list of names, or `"*"` for every authenticated identity. +* **Action**: `s3vectors:` followed by the operation name, e.g. `s3vectors:QueryVectors`, or `s3vectors:*`. Each operation of the [[S3 Vectors API]] checks the action of the same name. +* **Resource**: the bucket ARN, an index ARN, or a wildcard such as `arn:aws:s3vectors:us-east-1:000000000000:bucket/team-a/index/*`. ARNs always use region `us-east-1` and account `000000000000`. +* **Deny wins**: a matching `Deny` overrides any `Allow`, the owner's access and the identity's own actions. Only the identity named `admin` is exempt. + +Example: let every identity query one index, but shut out `writer` entirely: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + {"Effect": "Allow", "Principal": "*", "Action": "s3vectors:QueryVectors", + "Resource": "arn:aws:s3vectors:us-east-1:000000000000:bucket/team-a/index/idx"}, + {"Effect": "Deny", "Principal": {"AWS": "writer"}, "Action": "s3vectors:*", "Resource": "*"} + ] +} +``` + +An identity with no vector grants of its own can then call `QueryVectors` on `idx`, but not `GetVectors`; `writer` gets `AccessDeniedException` even for `QueryVectors`. + +`GetVectorBucketPolicy` returns the document as it was stored; with no policy it returns `NotFoundException`. `DeleteVectorBucketPolicy` removes it, and with it every access the policy granted. + +## The engine's identity + +`weed-vector` reads and writes every index's files with the identity passed in its `--storage-option` flags, whoever called the API. Authorization happens in the gateway before the engine is called. In our tests the engine used an identity with the `Admin` action. + +## Rejected policies + +`PutVectorBucketPolicy` refuses, with `ValidationException`, a policy that it could not enforce: + +| Policy content | Message | +|----------------|---------| +| An action of another service, e.g. `s3tables:GetTable` or `s3:GetObject` | `action "s3:GetObject" is not enforceable in a vector bucket policy` | +| A resource ARN of another service, e.g. `arn:aws:s3tables:...` | `resource "arn:aws:s3tables:..." is not enforceable in a vector bucket policy` | +| `NotAction` | `NotAction is not supported` | +| `NotResource` | `NotResource is not supported` | diff --git a/S3-Vector-Bucket.md b/S3-Vector-Bucket.md new file mode 100644 index 0000000..08b740b --- /dev/null +++ b/S3-Vector-Bucket.md @@ -0,0 +1,305 @@ +# S3 Vector Bucket + +SeaweedFS serves the [Amazon S3 Vectors](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors.html) API: vector buckets, vector indexes, and `PutVectors` / `QueryVectors` similarity search. Point the AWS CLI or an AWS SDK's `s3vectors` client at the SeaweedFS S3 endpoint and make the same calls; [[S3 Vectors API]] lists what differs from AWS. + +* [[S3 Vectors API]]: supported operations, distance metrics, metadata filters, limits +* [[S3 Vector Bucket Security]]: who can do what, and vector bucket policies +* [[S3 Vector Bucket Performance]]: the perf suite and measured numbers + +## How it works + +``` + aws s3vectors / boto3 / any AWS SDK + │ SigV4, service name "s3vectors" + ▼ + S3 gateway :8333 ── buckets, indexes and policies live in the filer catalog + │ gRPC, engine address looked up on the master + ▼ + weed-vector ── Lance catalog :9101 + S3 :8333 ──► s3:///vector// + (one Lance dataset per index) +``` + +* **The S3 gateway** (`weed s3`, or the S3 server inside `weed server -s3` and `weed mini`) answers S3 Vectors requests on the normal S3 port. It tells them apart by the service name in the SigV4 credential scope: a request signed for `s3vectors` goes to the vectors API, everything else to the S3 API. +* **The vector engine** is a separate process, `weed-vector`. It registers with the master, and the gateway finds it through the master on the first request that needs it. Bucket and policy operations work without it; index and vector operations need it. +* **The data** is stored as [Lance](https://lancedb.com) datasets. A vector bucket is a table bucket with format `LANCE`, and each index is a Lance table in the bucket's `vector` namespace, stored under `s3:///vector//`. The engine reaches it through the [[SeaweedFS Lance Catalog]] (`-s3.port.lance`, default `9101`, on by default) and the S3 port. + +**Queries are exact.** No approximate (ANN) index is built, so every query compares against all vectors of the index; recall@10 was 1.000 in every run of the perf suite. Query time grows with the size of the index; see [[S3 Vector Bucket Performance]]. + +## Getting weed-vector + +`weed-vector` is not part of the release archives or the Docker images. Build it from source with Rust 1.94.1 or newer and `protoc`, which Lance's build scripts need: + +```bash +sudo apt-get install -y protobuf-compiler # Debian/Ubuntu; provides protoc +git clone https://github.com/seaweedfs/seaweedfs.git +cd seaweedfs +cargo build --release --locked --manifest-path seaweed-vector/Cargo.toml -p weed-vector-server +# the binary is seaweed-vector/target/release/weed-vector +``` + +Without `protoc` the build stops in `lance-encoding` with `Could not find protoc`. + +## Quick start + +**1. Start SeaweedFS with an S3 identity.** `weed mini` turns the two variables into an identity with admin rights: + +```bash +export AWS_ACCESS_KEY_ID=admin +export AWS_SECRET_ACCESS_KEY=admin-secret +weed mini -dir=/data/seaweedfs +``` + +**2. Start the engine in another terminal**, pointing it at the master and at the Lance catalog, and giving it the same identity: + +```bash +weed-vector --masters localhost:9333 --namespace http://localhost:9101 \ + --storage-option aws_access_key_id=admin \ + --storage-option aws_secret_access_key=admin-secret +``` + +The engine needs a real S3 identity: it reads and writes the Lance files over S3 with it, and authenticates to the Lance catalog with it. A SeaweedFS with no identities at all does not work: `CreateIndex` then fails with `InternalFailure`. + +**3. Create a bucket and an index, write vectors, query:** + +```bash +export AWS_ACCESS_KEY_ID=admin +export AWS_SECRET_ACCESS_KEY=admin-secret +export AWS_DEFAULT_REGION=us-east-1 + +aws s3vectors create-vector-bucket --endpoint-url http://localhost:8333 \ + --vector-bucket-name media + +aws s3vectors create-index --endpoint-url http://localhost:8333 \ + --vector-bucket-name media --index-name movies \ + --data-type float32 --dimension 3 --distance-metric cosine + +aws s3vectors put-vectors --endpoint-url http://localhost:8333 \ + --vector-bucket-name media --index-name movies \ + --vectors '[ + {"key": "star-wars", "data": {"float32": [0.1, 0.9, 0.2]}, "metadata": {"genre": "scifi", "year": 1977}}, + {"key": "alien", "data": {"float32": [0.2, 0.8, 0.1]}, "metadata": {"genre": "scifi", "year": 1979}}, + {"key": "amelie", "data": {"float32": [0.9, 0.1, 0.3]}, "metadata": {"genre": "romance", "year": 2001}} + ]' + +aws s3vectors query-vectors --endpoint-url http://localhost:8333 \ + --vector-bucket-name media --index-name movies \ + --query-vector '{"float32": [0.15, 0.85, 0.15]}' --top-k 2 \ + --return-distance --return-metadata +``` + +```json +{ + "vectors": [ + { + "distance": 0.0029967427, + "key": "star-wars", + "metadata": { + "genre": "scifi", + "year": 1977 + } + }, + { + "distance": 0.0037363768, + "key": "alien", + "metadata": { + "genre": "scifi", + "year": 1979 + } + } + ], + "distanceMetric": "cosine" +} +``` + +With a metadata filter, only vectors from 1978 on are candidates: + +```bash +aws s3vectors query-vectors --endpoint-url http://localhost:8333 \ + --vector-bucket-name media --index-name movies \ + --query-vector '{"float32": [0.15, 0.85, 0.15]}' --top-k 2 \ + --filter '{"year": {"$gte": 1978}}' --return-metadata +``` + +```json +{ + "vectors": [ + { + "key": "alien", + "metadata": { + "genre": "scifi", + "year": 1979 + } + }, + { + "key": "amelie", + "metadata": { + "genre": "romance", + "year": 2001 + } + } + ], + "distanceMetric": "cosine" +} +``` + +`get-vectors`, `list-vectors` and `delete-vectors` work the same way. + +**The same with boto3:** + +```python +import boto3 + +s3v = boto3.client( + "s3vectors", + endpoint_url="http://localhost:8333", + aws_access_key_id="admin", + aws_secret_access_key="admin-secret", + region_name="us-east-1", +) + +s3v.create_vector_bucket(vectorBucketName="books") +s3v.create_index( + vectorBucketName="books", indexName="titles", + dataType="float32", dimension=4, distanceMetric="euclidean", + metadataConfiguration={"nonFilterableMetadataKeys": ["summary"]}, +) +s3v.put_vectors( + vectorBucketName="books", indexName="titles", + vectors=[ + {"key": "dune", "data": {"float32": [0.9, 0.1, 0.0, 0.2]}, + "metadata": {"genre": "scifi", "pages": 412, "summary": "Desert planet politics."}}, + {"key": "emma", "data": {"float32": [0.1, 0.9, 0.3, 0.0]}, + "metadata": {"genre": "classic", "pages": 474, "summary": "Matchmaking in Highbury."}}, + {"key": "neuromancer", "data": {"float32": [0.8, 0.2, 0.1, 0.3]}, + "metadata": {"genre": "scifi", "pages": 271, "summary": "Cyberspace heist."}}, + ], +) + +result = s3v.query_vectors( + vectorBucketName="books", indexName="titles", + queryVector={"float32": [0.85, 0.15, 0.05, 0.25]}, topK=3, + filter={"$and": [{"genre": {"$eq": "scifi"}}, {"pages": {"$lt": 400}}]}, + returnDistance=True, returnMetadata=True, +) +for hit in result["vectors"]: + print(hit["key"], round(hit["distance"], 4), hit["metadata"]) +``` + +``` +neuromancer 0.01 {'genre': 'scifi', 'pages': 271, 'summary': 'Cyberspace heist.'} +``` + +`summary` is non-filterable: it is stored and returned, but a filter on it is rejected. + +## Running the engine + +| Flag | Environment variable | Default | Meaning | +|------|----------------------|---------|---------| +| `--masters` | `WEED_VECTOR_MASTERS` | none | Comma-separated master addresses to register with. Without it the gateway never finds the engine. | +| `--namespace` | `WEED_VECTOR_NAMESPACE` | `http://localhost:8888/lance` | The Lance catalog URL. Set it to the S3 server's Lance port, e.g. `http://localhost:9101`; the default does not work with `weed mini` or `weed server`. | +| `--listen` | `WEED_VECTOR_LISTEN` | `127.0.0.1:8080` | gRPC address. This is the address registered on the master, so the gateway must be able to reach it. | +| `--storage-option key=value` | | none | Repeatable. `aws_access_key_id` and `aws_secret_access_key` are used for S3 data access and for the Lance catalog. | +| `--read-cache-mb` | `WEED_VECTOR_READ_CACHE_MB` | `512` | Memory for caching reads of the index's (write-once) data files, in MiB. `0` disables it. | + +* **Port clash with `weed server`.** Its volume server uses `8080` and `18080` (gRPC), so start the engine with another address, e.g. `--listen 127.0.0.1:8090`. With the default it exits with `Address already in use (os error 98)`. `weed mini` puts its volume server on `9340`, so the default is free there. +* **Engine on another host.** With a wildcard address such as `--listen 0.0.0.0:8090`, the engine registers the IP it uses to reach the first master. + +* **A standby engine.** The gateway sends every request to one registered engine at a time. Start a second engine (with its own `--listen`) and it takes over when the first stops: in our test, queries kept succeeding right after the engine in use was stopped. + +## Limits + +| What | Limit | +|------|-------| +| Data type | `float32` only | +| Dimension | 1 to 4096 | +| Distance metric | `euclidean` (default), `cosine`, `dotproduct` | +| Vectors per `PutVectors` | 1 to 500 | +| Keys per `GetVectors` | 1 to 100 | +| Keys per `DeleteVectors` | 1 to 500 | +| `topK` | 1 to 10000 | +| `ListVectors` page | 500 by default, `maxResults` up to 1000 | +| `ListVectors` segments | up to 16 | +| `ListVectorBuckets` / `ListIndexes` `maxResults` | up to 500 | +| Request body | 32 MiB | + +Bucket names are 3 to 63 lowercase letters, digits and hyphens, starting and ending with a letter or digit. Index names use lowercase letters, digits, `-` and `_`, and start with a letter or digit. See [[S3 Vectors API]] for the details. + +## Reading an index with LanceDB + +Each index is an ordinary Lance table in namespace `[, "vector"]`, so Lance tools can read it through the [[SeaweedFS Lance Catalog]]. The catalog takes the S3 credential as an `x-api-key` header of the form `access_key:secret_key`: + +```python +import lancedb + +db = lancedb.connect_namespace( + "rest", + {"uri": "http://localhost:9101", "header.x-api-key": "admin:admin-secret"}, + storage_options={ + "aws_endpoint": "http://localhost:8333", + "allow_http": "true", + "aws_access_key_id": "admin", + "aws_secret_access_key": "admin-secret", + "aws_region": "us-east-1", + }, +) +table = db.open_table("docs", namespace_path=["interop", "vector"]) +print(db.list_tables(namespace_path=["interop", "vector"]).tables) +print(table.schema) +print(table.search([1, 0, 0]).limit(1).to_list()) +``` + +For an index `docs` (dimension 3, `cosine`) in bucket `interop`, holding two vectors: + +``` +['interop$vector$docs'] +key: string not null +vector: fixed_size_list[3] not null + child 0, item: float + -- field metadata -- + distance_metric: 'COSINE' +metadata: large_binary + -- field metadata -- + ARROW:extension:name: 'lance.json' +[{'key': 'a', 'vector': [1.0, 0.0, 0.0], 'metadata': '{"lang":"en"}', '_distance': 0.0}] +``` + +The table has three columns: `key`, `vector` (with the metric in the field metadata) and `metadata` (JSON). + +**Treat it as read-only from Lance.** The catalog refuses to drop, rename, register or deregister tables in the `vector` namespace (`the vector namespace of a vector bucket is read-only; manage its indexes through S3 Vectors`). Appending rows with LanceDB does reach the dataset, and the S3 Vectors API then returns them, but such writes bypass the API and its validation. + +## Troubleshooting + +Start the S3 server (or `weed mini` / `weed server`) with `-v=1` to see the vectors log lines. When the engine registers, the master logs: + +``` +Cluster: vector node 127.0.0.1:8080 added to group '' +``` + +and on the first request that needs the engine, the gateway logs: + +``` +S3 Vectors: discovered weed-vector at 127.0.0.1:8080 via master 172.17.0.3:9333 +``` + +| Symptom | Cause | +|---------|-------| +| `ServiceUnavailableException: vector service unavailable` on `CreateIndex`, `PutVectors` or `QueryVectors` | No engine is registered: it is not running, it has no `--masters`, or it exited (for example on a busy `--listen` port). Bucket operations keep working. | +| `InternalFailure: internal error` on `CreateIndex`, and with `-v=1` the gateway logs `S3 Vectors: engine error: ... 403 Forbidden for url (http://localhost:9101/v1/table/.../declare)` | The engine has no credential for the Lance catalog. Pass `--storage-option aws_access_key_id=...` and `--storage-option aws_secret_access_key=...`. | +| The same, with `401 Unauthorized` | The engine's access key is unknown or its secret is wrong. | +| The same, with `500 Internal Server Error for url (http://localhost:8888/lance/v1/table/...)` | `--namespace` was left at its default; point it at the Lance port, e.g. `http://localhost:9101`. | +| The same, with `Failed to get AWS credentials: CredentialsNotLoaded` | The engine has no storage credential at all (seen on a SeaweedFS without identities). | +| `weed-vector` exits with `Address already in use (os error 98)` | The `--listen` port is taken; see the port clash note above. | +| `PutVectors` fails with `403 AccessDenied: authentication failed` although the credential is right | The request body is over 32 MiB. Send fewer vectors per call. | +| A plain HTTP request to `/ListVectorBuckets` gets `405` | Only requests signed for the `s3vectors` service reach the vectors API. Use an AWS CLI or SDK `s3vectors` client. | + +A failed `CreateIndex` leaves no index behind, so it is safe to retry once the engine is fixed. + +## Monitoring + +S3 Vectors requests are counted in the S3 server's Prometheus metrics, with the operation in the `type` label as `s3vectors.`. Enable them with `-metricsPort`; the listener binds to the server's IP address, not to `localhost`. + +``` +SeaweedFS_s3_request_total{bucket="",code="200",type="s3vectors.QueryVectors"} 1 +SeaweedFS_s3_request_seconds_bucket{bucket="",type="s3vectors.CreateIndex",le="0.0001"} 0 +SeaweedFS_s3_in_flight_requests{type="s3vectors.QueryVectors"} 0 +``` diff --git a/S3-Vectors-API.md b/S3-Vectors-API.md new file mode 100644 index 0000000..e398a1f --- /dev/null +++ b/S3-Vectors-API.md @@ -0,0 +1,120 @@ +# S3 Vectors API + +What the SeaweedFS S3 Vectors API supports, and where it differs from AWS. For setup see [[S3 Vector Bucket]]; for permissions see [[S3 Vector Bucket Security]]. + +## Operations + +Requests go to the S3 port, signed with SigV4 for the service `s3vectors`. Use an AWS CLI or SDK `s3vectors` client with its endpoint set to the SeaweedFS S3 address; every example on these pages was run with the AWS CLI and boto3. + +| Operation | Supported | +|-----------|-----------| +| `CreateVectorBucket`, `GetVectorBucket`, `ListVectorBuckets`, `DeleteVectorBucket` | yes | +| `PutVectorBucketPolicy`, `GetVectorBucketPolicy`, `DeleteVectorBucketPolicy` | yes | +| `CreateIndex`, `GetIndex`, `ListIndexes`, `DeleteIndex` | yes | +| `PutVectors`, `GetVectors`, `ListVectors`, `DeleteVectors`, `QueryVectors` | yes | +| `TagResource`, `UntagResource`, `ListTagsForResource` | no: HTTP 404 | +| `UpdateIndexMode`, `PutVectorBucketDefaultIndexMode` | no: `ValidationException: unknown operation` | + +Every bucket and index can be addressed by name or by ARN. ARNs always use region `us-east-1` and account `000000000000`, e.g. `arn:aws:s3vectors:us-east-1:000000000000:bucket/media/index/movies`; an ARN with another partition, region or account, or a name that disagrees with the ARN, is rejected with `ValidationException`. The region a request is signed for is not checked. + +## Vector buckets + +* **Names**: 3 to 63 characters, lowercase letters, digits and hyphens, starting and ending with a letter or digit. +* **Deleting** a bucket that still has indexes fails with `ConflictException`. +* **Encryption**: `encryptionConfiguration` accepts `{"sseType": "AES256"}` and returns it from `GetVectorBucket`; `aws:kms` is rejected with `unsupported sseType "aws:kms"`. **The setting is only recorded: the index files are written without server-side encryption.** A `HeadObject` on an index data file in an `AES256` bucket shows no `ServerSideEncryption`, exactly as in a bucket without the setting. +* A vector bucket is a table bucket with format `LANCE`, and its objects are visible through the S3 API under `vector//`. + +## Indexes + +```python +s3v.create_index( + vectorBucketName="media", indexName="movies", + dataType="float32", dimension=384, distanceMetric="cosine", + metadataConfiguration={"nonFilterableMetadataKeys": ["description"]}, +) +``` + +* **Names**: lowercase letters, digits, `-` and `_`, starting with a letter or digit. The AWS CLI and SDKs require 3 to 63 characters before sending. +* **`dataType`**: `float32` only (`only float32 vectors are supported`). +* **`dimension`**: 1 to 4096 (`dimension must be 1..=4096`). +* **`nonFilterableMetadataKeys`**: those keys are stored and returned, but a filter that names one is rejected with `metadata key "description" is not filterable`. +* Deleting an index removes its vectors. An index created again under the same name starts empty and may have a different dimension or metric. + +## Distance metrics + +The metric is fixed when the index is created and used by every query; it is returned by `GetIndex` and in each `QueryVectors` response as `distanceMetric`. The metric name is not case-sensitive. + +| `distanceMetric` | `distance` in results | Example: query `[1, 0]` | +|------------------|-----------------------|-------------------------| +| `euclidean` (default when omitted) | **squared** Euclidean distance | `[3, 4]` from `[0, 0]` → `25.0` | +| `cosine` | 1 − cosine similarity | `[10, 0]` → `0.0` | +| `dotproduct` | 1 − dot product | `[2, 0]` → `-1.0`, `[1, 1]` → `0.0`, `[-1, 0]` → `2.0` | + +Results are ordered by ascending `distance`. `dotproduct` is a SeaweedFS addition; AWS offers `euclidean` and `cosine`. Any other name is rejected with `unknown distanceMetric`. + +## Writing vectors + +```python +s3v.put_vectors(vectorBucketName="media", indexName="movies", vectors=[ + {"key": "alien", "data": {"float32": [0.2, 0.8, 0.1]}, + "metadata": {"genre": "scifi", "year": 1979, "cast": {"lead": "Weaver"}, "tags": ["space", "horror"]}}, +]) +``` + +* **`PutVectors` is an upsert by key**: writing an existing key replaces its vector and metadata. +* **A batch is all or nothing**: 1 to 500 vectors, and if any vector has the wrong dimension, or a key appears twice in the batch, the whole request fails with `ValidationException` and nothing is written. +* **Metadata** is any JSON object: strings, numbers, booleans, arrays and nested objects come back unchanged. Keys may contain characters such as `'`. +* **`DeleteVectors`** takes 1 to 500 keys. +* The request body is limited to 32 MiB. A larger `PutVectors` is refused with `403 AccessDenied: authentication failed`, not with a size error. 500 vectors of 4096 dimensions fit (a 9.8 MiB body in our test). + +## Reading vectors + +* **`GetVectors`** takes 1 to 100 keys; keys that do not exist are left out of the result. Data and metadata are returned only when `returnData` / `returnMetadata` are set. +* **`ListVectors`** pages through every vector: 500 per page by default, `maxResults` up to 1000, continued with `nextToken`. An invalid `nextToken` is rejected. +* **Parallel listing**: pass `segmentCount` (up to 16) and a `segmentIndex` from 0 to `segmentCount - 1` to list one disjoint part of the index; the parts together hold every vector exactly once. `segmentIndex` without `segmentCount` is rejected. + +## Querying + +```python +s3v.query_vectors( + vectorBucketName="media", indexName="movies", + queryVector={"float32": [0.15, 0.85, 0.15]}, topK=10, + filter={"$and": [{"genre": "scifi"}, {"year": {"$gte": 1978}}]}, + returnDistance=True, returnMetadata=True, +) +``` + +* **`topK`**: 1 to 10000. `topK` larger than the index returns every vector. +* The query vector must have the index's dimension. +* `distance` is in the result only with `returnDistance`, and `metadata` only with `returnMetadata`. +* **Search is exact**: every query compares against every vector in the index; no approximate index is built. + +## Metadata filters + +| Filter | Matches | +|--------|---------| +| `{"genre": "scifi"}` | `genre` equals `"scifi"` | +| `{"genre": {"$eq": "scifi"}}`, `{"$ne": ...}` | equal / not equal | +| `{"year": {"$gt": 1977}}`, `$gte`, `$lt`, `$lte` | numeric comparison; the operand must be a number (`$gt expects a number`) | +| `{"genre": {"$in": ["drama", "western"]}}`, `$nin` | equal to one of / none of the values | +| `{"tags": {"$exists": true}}` | the key is present (or absent, with `false`) | +| `{"$and": [ ... ]}`, `{"$or": [ ... ]}` | all / any of the sub-filters | +| `{"genre": "scifi", "year": 2014}` | several keys in one object: all must match | + +Behaviour to know: + +* **Types are compared strictly.** `1`, `"1"`, `true` and `"true"` are four different values, for `$eq` as well as `$in` and `$nin`. +* **Only top-level scalar values match.** A filter does not look inside arrays: with `"tags": ["space", "classic"]`, neither `{"tags": "space"}` nor `{"tags": {"$in": ["space"]}}` matches; only `$exists` sees the key. A dotted path such as `{"info.lang": "fr"}` does not reach into a nested object either. +* An unknown operator is rejected (`unsupported operator $regex`), and so is a filter on a non-filterable key. + +## Errors + +| Error | When | +|-------|------| +| `ValidationException` (400) | invalid parameters, limits exceeded, invalid policy | +| `AccessDeniedException` (403) | not allowed; see [[S3 Vector Bucket Security]] | +| `AccessDenied` (403) | the credential is unknown or the signature is wrong | +| `NotFoundException` (404) | bucket, index or policy does not exist | +| `ConflictException` (409) | bucket or index already exists; bucket still has indexes | +| `ServiceUnavailableException` (503) | no `weed-vector` engine is registered | +| `InternalFailure` (500) | an engine error; the gateway logs it with `-v=1` (see [[S3 Vector Bucket#troubleshooting]]) | diff --git a/SeaweedFS-Lance-Catalog.md b/SeaweedFS-Lance-Catalog.md index 80c1dc9..d8f5099 100644 --- a/SeaweedFS-Lance-Catalog.md +++ b/SeaweedFS-Lance-Catalog.md @@ -138,6 +138,7 @@ Table buckets show the format they hold and the endpoint that serves them. For a ## See also +- [[S3 Vector Bucket]]: a vector bucket's indexes are Lance tables in its `vector` namespace; this catalog serves them but refuses to drop, rename or re-register them - [[LanceDB Integration]] - [[Spark Lance Integration]] - [[DuckDB Lance Integration]] diff --git a/_Sidebar.md b/_Sidebar.md index 8f8048d..a78f6fa 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -118,6 +118,12 @@ * [[SeaweedFS Lance Catalog]] * [[Lance Maintenance Worker]] +### S3 Vector Bucket +* [[S3 Vector Bucket]] +* [[S3 Vectors API]] +* [[S3 Vector Bucket Security]] +* [[S3 Vector Bucket Performance]] + ### Iceberg Integrations * [[Spark Iceberg Integration]] * [[Trino Iceberg Integration]]