From 1b8d34092869b4d39f9f36598c2fd66392aadf4c Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 7 Nov 2026 18:10:53 -0800 Subject: [PATCH] S3 Vectors: the engine, worker and CLI keep tables on the filers and volume servers After seaweedfs#12469 and #12470: weed-vector, weed-worker and weed-vector-cli take --masters and hold no S3 credentials (--storage-option, --access-key and --secret-key are refused); the worker and CLI authenticate to the Lance catalog with the cluster key; an SSE-S3 bucket needs the gateway's key on the engine too, including weed mini's generated one. The performance page compares the engine through the gateway with the direct one. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --- Lance-Maintenance-Worker.md | 17 ++++++--- S3-Vector-Bucket-Performance.md | 33 ++++++++++++++++-- S3-Vector-Bucket-Security.md | 8 +++-- S3-Vector-Bucket.md | 61 +++++++++++++++++++-------------- S3-Vectors-API.md | 2 +- 5 files changed, 83 insertions(+), 38 deletions(-) diff --git a/Lance-Maintenance-Worker.md b/Lance-Maintenance-Worker.md index a1db6ce..96e868a 100644 --- a/Lance-Maintenance-Worker.md +++ b/Lance-Maintenance-Worker.md @@ -15,7 +15,7 @@ carries rather than the only one it ever will. ```bash docker run chrislusf/seaweedfs worker-rust \ - --admin admin:12535 --namespace http://s3:9101 + --admin admin:12535 --masters master:9333 --namespace http://s3:9101 ``` The standard image carries it at `/usr/bin/weed-worker` on `amd64` and `arm64`, @@ -46,6 +46,7 @@ A Rust toolchain and a protoc; see [Building it](#building-it) below. ```bash weed-worker \ --admin localhost:12535 \ + --masters localhost:9333 \ --namespace http://localhost:9101 ``` @@ -54,11 +55,15 @@ weed-worker \ | Flag | Default | Meaning | |------|---------|---------| | `--admin` | `localhost:12535` | admin server's HTTP address | -| `--namespace` | | Lance Namespace URL (`WEED_LANCE_NAMESPACE`) | +| `--namespace` | `http://localhost:9101` | Lance Namespace URL (`WEED_LANCE_NAMESPACE`) | +| `--masters` | none; required | masters whose filers the tables are kept through (`WEED_MASTERS`) | +| `--filer-group` | empty | filer group whose filers it uses (`WEED_FILER_GROUP`) | +| `--encryptVolumeData` | off | encrypt every chunk it writes for the volume servers, as `weed s3 -encryptVolumeData` does | +| `--security-toml` | found like `weed` does | the cluster's `security.toml` (`WEED_WORKER_SECURITY_TOML`) | +| `--filer-signing-key` | `jwt.filer_signing.key` | cluster key for the namespace and the admin stream (`WEED_FILER_SIGNING_KEY`); see [Credentials](#credentials) | | `--id` | hostname-derived | worker id; **two workers sharing one id evict each other** | | `--heartbeat-seconds` | `10` | | | `--max-concurrency` | `1` | detection and execution slots | -| `--access-key`, `--secret-key` | | storage credentials for a gateway that vends none | | `--tls-ca`, `--tls-cert`, `--tls-key` | | mTLS for the control stream; all three together | | `--tls-server-name` | | when admin's certificate does not name the address dialled | | `--metrics-port` | `0` (off) | serves `/health`, `/ready`, `/metrics` | @@ -66,9 +71,11 @@ weed-worker \ ### Credentials -The worker holds none of its own. It asks the namespace to describe a table with `vend_credentials` and hands the `storage_options` to lance. A gateway without STS vends no credentials at all, so `--access-key` and `--secret-key` are a fallback; anything the namespace does vend wins over them. +The worker holds no S3 credentials. The namespace tells it where a table is; it reads and writes the table's files on the filers and volume servers the masters list, with what the cluster's `security.toml` gives it, as `weed-vector` does: `grpc.client` for the masters and filers, `https.client` for volume servers under HTTPS, `jwt.signing.read.key` where volume reads are signed, and `s3.sse.kek` or `s3.sse.key` for buckets with SSE-S3 encryption. It writes a table as its bucket declares, read from the bucket's record on the filer. -If detection reports nothing at all, credentials are the first thing to check: a table the worker cannot open is skipped with a warning, and a sweep that could open nothing looks exactly like a cluster with no work to do. The `objects_skipped_total` metric below exists to tell those apart. +The namespace knows the worker by a token derived from the cluster's `jwt.filer_signing.key`, the same key that signs the admin stream when admin requires it. The worker takes it from `--filer-signing-key`, else `WEED_JWT_FILER_SIGNING_KEY`, else `[jwt.filer_signing]` in `security.toml`. A worker without it is listed no tables. `--access-key` and `--secret-key` are gone (`error: unexpected argument '--access-key' found`). + +If detection reports nothing at all, the cluster key is the first thing to check: a table the worker cannot open is skipped with a warning, and a sweep that could open nothing looks exactly like a cluster with no work to do. The `objects_skipped_total` metric below exists to tell those apart. ### TLS diff --git a/S3-Vector-Bucket-Performance.md b/S3-Vector-Bucket-Performance.md index 5956144..fb67baf 100644 --- a/S3-Vector-Bucket-Performance.md +++ b/S3-Vector-Bucket-Performance.md @@ -36,6 +36,8 @@ Each scenario starts with one untimed call, and the throughput scenario creates ## Results +The sections up to [The read cache](#the-read-cache) were measured with an open-source engine that read and wrote the index files through the S3 gateway, as releases up to 4.52 do. [Direct storage](#direct-storage) compares that with the engine that keeps them on the filers and volume servers itself. + One 16-core x86_64 host, every process on it, `cosine` metric, top-10, one client unless noted. Other work kept the host's load average between 3.4 and 4.9 during the runs, so read differences of a few milliseconds, and the Go/Rust columns against each other, as noise. | | Go volume, 10k × 128 | Rust volume, 10k × 128 | Go volume, 50k × 384 | Rust volume, 50k × 384 | @@ -92,7 +94,7 @@ Setup: the same 16-core host, shared with other work (load average 3 to 15), eve ## Beyond one engine -The open-source gateway sends every query to one `weed-vector` engine at a time, and that engine reads the index's files through the S3 gateway. Every number above comes from a single engine, and the numbers in this section are measured without a vector index. The [SeaweedFS Enterprise](https://seaweedfs.com/) version adds distributed vector querying: each index is placed on groups of engines, a query goes to one group, each engine in that group searches its share of the index, and the gateway merges the results. The Enterprise engines read the Lance files straight from the filers and volume servers, not through the gateway. When an engine is lost, queries keep returning the same results from the remaining engines. +The open-source gateway sends every query to one `weed-vector` engine at a time. Every number above comes from a single engine, and the numbers in this section are measured without a vector index. The [SeaweedFS Enterprise](https://seaweedfs.com/) version adds distributed vector querying: each index is placed on groups of engines, a query goes to one group, each engine in that group searches its share of the index, and the gateway merges the results. The Enterprise engines read the Lance files straight from the filers and volume servers; at the time of these runs the open-source engine still went through the gateway. When an engine is lost, queries keep returning the same results from the remaining engines. Setup: one 16-core x86_64 host, shared with other work (load average 9 to 19), every process on it; Go volume server, `cosine` metric, top-10, 100 timed queries per scenario. The open-source build is master at d727c6970f; the Enterprise build is the branch of seaweed-mono#954, which makes every engine read directly. Each configuration ran twice, and the cells show both runs: @@ -125,7 +127,7 @@ Recall@10 was 1.000 in every run, filtered queries included, and the gateway rou * **One client gets 1.4 to 2 times the throughput.** 41 to 42 queries/s instead of 20 to 24 at 50k × 384, and 17 instead of 10 to 12 at 200k × 384. * **Throughput at 4 to 8 clients is bound by this host's CPU.** All engines shared the same 16 cores, so with 4 or 8 clients the four engines ended within a few queries/s of the single engine. Engines on separate hosts were not measured. * **Default placement sits in between.** With the controller's defaults the same four engines were placed as two replica groups instead of one group of four. Medians were 47 ms at 50k and 70 to 80 ms at 200k, filtered 40 ms and 128 to 139 ms. At 200k each query was split over two engines, and the first few hundred queries went to one engine before the index was widened; the timed scenario started inside that window. -* **Ingest does not change.** Vectors are written through the gateway in every configuration, at 1.6k to 1.8k vectors/s here. +* **Ingest does not change.** Vectors were written by the single open-source engine, through the gateway, in every configuration, at 1.6k to 1.8k vectors/s here. These 200k indexes were queried right after the bulk load, as 400 fragments of 500 rows, with no [[Lance Maintenance Worker]] compacting them. @@ -140,6 +142,31 @@ These 200k indexes were queried right after the bulk load, as 400 fragments of 5 | GetVectors, 100 keys, p50 | 332 ms, 319 ms | 59 ms, 76 ms | | Queries/s with 1 / 4 / 8 clients | 7.5 / 11.5 / 12.2, 7.8 / 11.6 / 11.9 | 19 / 59 / 59, 26 / 50 / 50 | -Without the cache, every query reads the index's files back through the S3 gateway and the volume server. The cache cut median query latency to about a third and raised throughput about four to five times, even though other work loaded the host more during the cached runs (load average 19 to 21, against 6 to 10). +Without the cache, every query read the index's files back through the S3 gateway and the volume server. The cache cut median query latency to about a third and raised throughput about four to five times, even though other work loaded the host more during the cached runs (load average 19 to 21, against 6 to 10). This index holds about 77 MB of raw vector data (50,000 × 384 × 4 bytes), well inside the default cache. An index larger than the cache was not measured. + +## Direct storage + +With [#12469](https://github.com/seaweedfs/seaweedfs/pull/12469) and [#12470](https://github.com/seaweedfs/seaweedfs/pull/12470) the open-source engine reads and writes the index files on the filers and volume servers itself instead of through the S3 gateway. To see what that changes, master before [#12468](https://github.com/seaweedfs/seaweedfs/pull/12468) (every byte through the gateway) and master after [#12470](https://github.com/seaweedfs/seaweedfs/pull/12470) ran the perf suite alternately, twice each, on one 16-core x86_64 host shared with other work: release builds, Go volume server, `cosine`, top-10, 50,000 vectors, no vector index. The cells show both runs. + +| 50k × 128, read cache on | Through the gateway | Direct | +|---|---:|---:| +| Ingest (vectors/s) | 4,208, 4,265 | 3,947, 4,458 | +| Query top-10, p50 | 26.5 ms, 28.0 ms | 24.7 ms, 22.4 ms | +| Query with a 10% filter, p50 | 76.8 ms, 69.0 ms | 83.7 ms, 101.7 ms | +| Queries/s with 1 / 4 / 8 clients | 29 / 94 / 132, 35 / 91 / 116 | 32 / 115 / 126, 30 / 113 / 141 | +| GetVectors, 100 keys, p50 | 46.8 ms, 43.7 ms | 38.5 ms, 36.8 ms | + +| 50k × 1024, read cache off (`--read-cache-mb 0`) | Through the gateway | Direct | +|---|---:|---:| +| Ingest (vectors/s) | 701, 657 | 642, 579 | +| Query top-10, p50 | 205 ms, 440 ms | 232 ms, 240 ms | +| GetVectors, 100 keys, p50 | 342 ms, 377 ms | 226 ms, 152 ms | +| DeleteVectors, 100 keys, p50 | 182 ms, 312 ms | 63 ms, 150 ms | + +Recall@10 was 1.000 in every run. The host's load average was about 5 when the 1024-dimension runs started and between 31 and 48 at the start of the later ones; the two runs through the gateway differ by more than two times there. + +* **Reads by key were faster direct.** `GetVectors` of 100 keys took about 17% less time at 128 dimensions with the cache, and about half the time without it, in every run. +* **Queries moved in no single direction.** Unfiltered queries were a little faster direct with the cache; without it the runs overlap (205 and 440 ms through the gateway, 232 and 240 ms direct). Queries with a 10% filter were slower direct in both cached runs. With two runs each on a shared host, none of this is firm. +* **Ingest costs the same either way.** A `PutVectors` call of 500 vectors took about 110 ms at 128 dimensions and about 700 ms at 1,024 in both builds, which the S3 hop the direct engine saves does not move. diff --git a/S3-Vector-Bucket-Security.md b/S3-Vector-Bucket-Security.md index 5d788b0..d81ba57 100644 --- a/S3-Vector-Bucket-Security.md +++ b/S3-Vector-Bucket-Security.md @@ -6,7 +6,7 @@ Who may call the [[S3 Vectors API]], and how to grant access with a vector bucke Requests are signed with SigV4 for the service `s3vectors`, with an identity from the S3 configuration (`-s3.config`, see [[S3 Credentials]]). An unknown access key is refused with `403 AccessDenied: authentication failed`. -On a SeaweedFS without identities, every request is accepted whatever its access key, and `weed-vector` needs no credentials (see [[S3 Vector Bucket#quick-start]]). Configure identities for anything beyond a trial. +On a SeaweedFS without identities, every request is accepted whatever its access key (see [[S3 Vector Bucket#quick-start]]). Configure identities for anything beyond a trial. `weed-vector` needs no S3 credentials either way; see [The engine, the worker and the CLI](#the-engine-the-worker-and-the-cli). ## Who can do what @@ -145,9 +145,11 @@ A `Deny` with a tag condition protects the resources carrying the tag. This stat * The operators are the ones in [[S3 Policy Conditions#supported-condition-operators]], including the `ForAllValues:` and `ForAnyValue:` forms. The `IfExists` forms are rejected. * Condition keys are case-sensitive: `aws:resourcetag/env` is rejected. -## The engine's identity +## The engine, the worker and the CLI -`weed-vector` reads and writes every index's files with the identity passed in its `--storage-option` flags, whoever called the API. Authorization happens in the gateway before the engine is called. In our tests the engine used an identity with the `Admin` action. On a SeaweedFS without identities it needs none. +`weed-vector` holds no S3 identity. Authorization happens in the gateway before the engine is called: the gateway checks the caller, looks the index up in the Lance catalog, and sends the engine the index's location with the request. The engine accepts only an index's own location, `s3:///vector//`, and reads and writes its files on the filers and volume servers directly, with what the cluster's `security.toml` gives it: `grpc.client` for the masters and filers, `https.client` for volume servers under HTTPS, `jwt.signing.read.key` where volume reads are signed. In a bucket created with SSE-S3 (`sseType` `AES256`) it encrypts the files as the gateway does, so it needs the gateway's `s3.sse.kek` or `s3.sse.key` (see [[S3 Vector Bucket#running-the-engine]]); `--encryptVolumeData` encrypts every chunk it writes for the volume servers, as `weed s3 -encryptVolumeData` does. + +The [[Lance Maintenance Worker]] and `weed-vector-cli` keep the files the same way, and authenticate to the Lance catalog with a token derived from the cluster's `jwt.filer_signing.key`. A request carrying only that token runs as a trusted cluster process and is vended no storage credentials. Without the key the catalog refuses them: the CLI stops with `403 Forbidden`, and the worker is listed no tables. ## Rejected policies diff --git a/S3-Vector-Bucket.md b/S3-Vector-Bucket.md index 381f27b..5bd99eb 100644 --- a/S3-Vector-Bucket.md +++ b/S3-Vector-Bucket.md @@ -43,11 +43,11 @@ flowchart TB style buckets fill:none,stroke:#94a3b8 ``` -The gateway serves all three bucket kinds on the S3 port. A vector bucket is a table bucket with format `LANCE`; each index is one Lance dataset under `s3:///vector//`. `weed-vector` reads and writes it through the gateway's S3 API, and the gateway, as for any S3 request, keeps the file metadata on the filers and the chunks on the volume servers. +The gateway serves all three bucket kinds on the S3 port. A vector bucket is a table bucket with format `LANCE`; each index is one Lance dataset under `s3:///vector//`. `weed-vector` reads and writes its files on the filers and volume servers itself, in the layout the gateway gives any S3 object, so the same files are visible through the S3 API. * **The S3 gateway** (`weed s3`, or the S3 server inside `weed server -s3` and `weed mini`) answers S3 Vectors requests on the normal S3 port. It tells them apart by the service name in the SigV4 credential scope: a request signed for `s3vectors` goes to the vectors API, everything else to the S3 API. * **The vector engine** is a separate process, `weed-vector`. It registers with the master, and the gateway finds it through the master on the first request that needs it. Bucket and policy operations work without it; index and vector operations need it. Several can register: the open-source gateway sends its requests to one at a time, and SeaweedFS Enterprise spreads each index and its queries over them (see [[S3 Vector Bucket Performance#beyond-one-engine]]). -* **The data** is stored as [Lance](https://lancedb.com) datasets. A vector bucket is a table bucket with format `LANCE`, and each index is a Lance table in the bucket's `vector` namespace, stored under `s3:///vector//`. The engine reaches it through the [[SeaweedFS Lance Catalog]] (`-s3.port.lance`, default `9101`, on by default) and the S3 port. +* **The data** is stored as [Lance](https://lancedb.com) datasets. A vector bucket is a table bucket with format `LANCE`, and each index is a Lance table in the bucket's `vector` namespace, stored under `s3:///vector//`. The gateway looks each index up in the [[SeaweedFS Lance Catalog]] (`-s3.port.lance`, default `9101`, on by default) and sends its location with the request. The engine asks the masters for the filers and keeps the files through them and the volume servers, with no S3 credentials. **Queries are exact until a vector index is built.** Without one, every query compares against all vectors of the index, and recall@10 was 1.000 in every such run of the perf suite. The [[Lance Maintenance Worker]]'s `lance_vector_index` job builds an approximate (ANN) index, IVF, once an index holds at least `min_rows` vectors (50,000 by default). From then on a query searches a tenth of the index's partitions, at least 8, and up to four times that many when that leaves it short of `topK` results. It returns only what it finds in those partitions, so a filter that few vectors match can return fewer results than exist ([[S3 Vectors API]] has numbers). Vectors written after the build are still returned, and the worker's `lance_optimize_indices` job brings them into the index. Like every worker job, both start disabled. [Building a vector index](#building-a-vector-index) has the steps, and [[S3 Vector Bucket Performance#through-a-vector-index]] the measurements. @@ -56,13 +56,13 @@ The gateway serves all three bucket kinds on the S3 port. A vector bucket is a t ### Docker ```bash -docker run chrislusf/seaweedfs vector --masters master:9333 \ - --storage-option aws_access_key_id=admin \ - --storage-option aws_secret_access_key=admin-secret +docker run chrislusf/seaweedfs vector --masters master:9333 ``` Since release 4.52 the standard image carries the engine at `/usr/bin/weed-vector` on `amd64` and `arm64`, beside the Rust volume server and the Lance maintenance worker. The `vector` verb listens on `0.0.0.0:8090` unless `--listen` or `WEED_VECTOR_LISTEN` says otherwise, because the gateway dials it from another container. On `arm` and `386` the verb says the engine is not available for the platform. +**Release 4.52 and earlier:** that engine read and wrote the index files through the S3 gateway and took the gateway's credentials as `--storage-option aws_access_key_id=…` and `--storage-option aws_secret_access_key=…`. Builds that include [#12470](https://github.com/seaweedfs/seaweedfs/pull/12470) refuse that flag (`error: unexpected argument '--storage-option' found`) and need only `--masters`, plus the cluster's `security.toml` where the cluster has one (see [Running the engine](#running-the-engine)). + ### From a release ```bash @@ -87,25 +87,24 @@ Building from another directory with `--manifest-path` skips that setting. The b ## Quick start -**1. Start SeaweedFS with an S3 identity.** `weed mini` turns the two variables into an identity with admin rights: +**1. Start SeaweedFS with an S3 identity.** `weed mini` turns the two `AWS_` variables into an identity with admin rights. `WEED_JWT_FILER_SIGNING_KEY` is the cluster key (`jwt.filer_signing.key`); the engine does not need it, but the [[Lance Maintenance Worker]] and `weed-vector-cli` authenticate to the Lance catalog with it (see [Building a vector index](#building-a-vector-index)). Any secret string works; the processes that use it must have the same one: ```bash export AWS_ACCESS_KEY_ID=admin export AWS_SECRET_ACCESS_KEY=admin-secret +export WEED_JWT_FILER_SIGNING_KEY=change-me-cluster-key weed mini -dir=/data/seaweedfs ``` -**2. Start the engine in another terminal**, pointing it at the master and giving it the same identity: +**2. Start the engine in another terminal**, pointing it at the master: ```bash -weed-vector --masters localhost:9333 \ - --storage-option aws_access_key_id=admin \ - --storage-option aws_secret_access_key=admin-secret +weed-vector --masters localhost:9333 ``` -The engine reads and writes the Lance files over S3 with this identity, and authenticates with it to the Lance catalog, which it finds at `http://localhost:9101` by default. It listens on `127.0.0.1:8090`. +The engine registers with the master, asks it for the filers, and reads and writes the index files on the filers and volume servers itself. It needs no S3 credentials. It listens on `127.0.0.1:8090`. -**Trying it without identities:** start `weed mini` without the two variables, and the engine without `--storage-option` (`weed-vector --masters localhost:9333`). The S3 port then accepts any access key, so step 3 also works with `AWS_ACCESS_KEY_ID=any` and `AWS_SECRET_ACCESS_KEY=any`. +**Trying it without identities:** start `weed mini` without the two `AWS_` variables. The S3 port then accepts any access key, so step 3 also works with `AWS_ACCESS_KEY_ID=any` and `AWS_SECRET_ACCESS_KEY=any`. The engine is started the same way. **3. Create a bucket and an index, write vectors, query:** @@ -243,18 +242,27 @@ neuromancer 0.1 {'genre': 'scifi', 'pages': 271, 'summary': 'Cyberspace heist.'} | Flag | Environment variable | Default | Meaning | |------|----------------------|---------|---------| -| `--masters` | `WEED_VECTOR_MASTERS` | none | Comma-separated master addresses to register with. Without it the gateway never finds the engine. | -| `--namespace` | `WEED_VECTOR_NAMESPACE` | `http://localhost:9101` | The Lance catalog URL: the S3 server's Lance port (`-s3.port.lance`). | +| `--masters` | `WEED_VECTOR_MASTERS` | none; required | Comma-separated master addresses. The engine registers with them, so the gateway finds it, and asks them for the filers. Without it the engine does not start (`the following required arguments were not provided: --masters `). | +| `--filer-group` | `WEED_VECTOR_FILER_GROUP` | empty | The filer group whose filers the engine uses. | | `--listen` | `WEED_VECTOR_LISTEN` | `127.0.0.1:8090` | gRPC address. This is the address registered on the master, so the gateway must be able to reach it. | -| `--storage-option key=value` | | none | Repeatable. `aws_access_key_id` and `aws_secret_access_key` are used for S3 data access and for the Lance catalog. Needed when the S3 server has identities. | +| `--encryptVolumeData` | `WEED_VECTOR_ENCRYPT_VOLUME_DATA` | off | Encrypt every chunk the engine writes for the volume servers, as `weed s3 -encryptVolumeData` does. A filer configured to encrypt has the same effect. | | `--read-cache-mb` | `WEED_VECTOR_READ_CACHE_MB` | `512` | Memory for caching reads of the index's (write-once) data files, in MiB. `0` disables it. | | `--nprobes` | `WEED_VECTOR_NPROBES` | a tenth of the partitions, at least 8 | Partitions every query searches in a vector index, in place of the default. For measurements; `0` is rejected. | -| `--filer-signing-key` | `WEED_FILER_SIGNING_KEY` | none | The cluster's `jwt.filer_signing.key`. The Lance catalog vends credentials that can write index data only to a caller holding it. | -| `--security-toml` | `WEED_VECTOR_SECURITY_TOML` | found like `weed` does | The cluster's `security.toml`, for gRPC TLS. Without it the engine looks in `--config-dir`, `.`, `~/.seaweedfs`, `/usr/local/etc/seaweedfs` and `/etc/seaweedfs`. | +| `--security-toml` | `WEED_VECTOR_SECURITY_TOML` | found like `weed` does | The cluster's `security.toml`. Without it the engine looks in `--config-dir`, `.`, `~/.seaweedfs`, `/usr/local/etc/seaweedfs` and `/etc/seaweedfs`. | | `--config-dir` | `WEED_VECTOR_CONFIG_DIR` | none | The directory `weed -config_dir` names, searched first for `security.toml`. `~` is expanded. | +`--namespace` and `--filer-signing-key`, which earlier builds took, are still accepted and ignored, so an existing command line starts. `--storage-option` is refused. + * **Ports.** The default `8090` is free next to `weed mini` and `weed server`, whose volume server uses `8080`. If the port is taken, `weed-vector` exits with `Error: listen on 127.0.0.1:8090` and `Address already in use (os error 98)`; pass another `--listen`. * **Engine on another host.** With a wildcard address such as `--listen 0.0.0.0:8090`, the engine registers the IP it uses to reach the first master. +* **What it reads from `security.toml`.** The same settings the weed processes use, with `WEED_*` environment variables over the file: `[grpc.vector]` for its own listener and `[grpc.client]` for the masters and the filers, `https.client` for volume servers under HTTPS, `jwt.signing.read.key` where volume reads are signed, and `s3.sse.kek` or `s3.sse.key` for vector buckets with SSE-S3 encryption (a KEK the filer holds as `/etc/s3/sse_kek` is read too). +* **Encrypted vector buckets.** The engine writes the index files itself, so for a bucket created with `encryptionConfiguration` `{"sseType": "AES256"}` it needs the gateway's SSE-S3 key. Without it, `CreateIndex` on such a bucket fails with `InternalFailure`, and the S3 gateway logs `the bucket encrypts with SSE-S3, which needs s3.sse.kek or s3.sse.key for the engine as for the S3 gateway`. `weed mini` generates its key into `.mini_sse_kek` in its data directory and gives it only to its own S3 server, so start the engine with it: + + ```bash + WEED_S3_SSE_KEK=$(cat /data/seaweedfs/.mini_sse_kek) weed-vector --masters localhost:9333 + ``` + + The index files are then stored with SSE-S3: `HeadObject` on one shows `ServerSideEncryption: AES256`, and a `GetObject` returns the plain Lance file. * **gRPC TLS.** With TLS configured in `security.toml`, the engine serves with `[grpc.vector]` and dials masters with `[grpc.client]`. The master's certificate must chain to `grpc.ca`, and its common name or a DNS SAN must pass `grpc.client.master_allowed_server_commonNames` (else `grpc.client.allowed_server_commonNames`, else `grpc.allowed_server_commonNames`) when one is set, as for `weed` clients. `WEED_GRPC_*` environment variables override the file. * **A standby engine.** The gateway sends every request to one registered engine at a time. Start a second engine (with its own `--listen`) and it takes over when the first stops: in our test, queries kept succeeding right after the engine in use was stopped. @@ -263,14 +271,14 @@ neuromancer 0.1 {'genre': 'scifi', 'pages': 271, 'summary': 'Cyberspace heist.'} Without a vector index every query compares against all vectors of the index. The [[Lance Maintenance Worker]] (`weed-worker`) builds the vector index and keeps it covering new vectors; `weed-vector` uses it once it exists. The steps below continue the [quick start](#quick-start), with `weed mini`'s admin server on port `12535`. -**1. Start the worker**, with the engine's identity: +**1. Start the worker**, with the cluster key `weed mini` was started with: ```bash -weed-worker --admin localhost:12535 --namespace http://localhost:9101 \ - --access-key admin --secret-key admin-secret +export WEED_JWT_FILER_SIGNING_KEY=change-me-cluster-key +weed-worker --admin localhost:12535 --masters localhost:9333 --namespace http://localhost:9101 ``` -It logs `connected to admin`, and its job types appear in `curl http://localhost:12535/api/plugin/job-types`, among them `lance_vector_index` and `lance_optimize_indices`. +Like the engine, it reads and writes the index files on the filers and volume servers and needs no S3 credentials. The Lance catalog knows it by the token it derives from the cluster key; a worker without the key is listed no tables, so detection finds nothing to do. It logs `connected to admin`, and its job types appear in `curl http://localhost:12535/api/plugin/job-types`, among them `lance_vector_index` and `lance_optimize_indices`. **2. Turn the two jobs on**, in the admin UI at `/plugin` or through its API. They start disabled: @@ -305,10 +313,11 @@ curl -X PUT http://localhost:12535/api/plugin/job-types/lance_vector_index/confi **Vectors written afterwards** are returned by queries right away, and `lance_optimize_indices` adds them to the vector index once they amount to more than a tenth of the vectors already in it. After 6,000 vectors were added to the 50,000 above, a run of that job reported `"detected_count":1,"success_count":1`, and none were left outside the vector index. See [[Lance Maintenance Worker#the-jobs]] for the rules and settings. -**Checking the result.** `weed-vector-cli`, built from source (`cargo build --release --locked -p weed-vector-core --bin weed-vector-cli` inside `seaweed-vector/`), prints what the vector index of `movies` in bucket `media` covers: +**Checking the result.** `weed-vector-cli`, built from source (`cargo build --release --locked -p weed-vector-core --bin weed-vector-cli` inside `seaweed-vector/`), prints what the vector index of `movies` in bucket `media` covers. Like the worker, it takes `--masters` and the cluster key: ```bash -weed-vector-cli --access-key admin --secret-key admin-secret coverage 'media$vector$movies' +export WEED_JWT_FILER_SIGNING_KEY=change-me-cluster-key +weed-vector-cli --masters localhost:9333 coverage 'media$vector$movies' ``` ```json @@ -325,7 +334,7 @@ weed-vector-cli --access-key admin --secret-key admin-secret coverage 'media$vec ] ``` -An empty list means no vector index has been built. `unindexed_rows` counts the vectors written since the last build or optimize run. +An empty list means no vector index has been built. `unindexed_rows` counts the vectors written since the last build or optimize run. Without the cluster key the CLI stops with `HTTP status client error (403 Forbidden) for url (http://localhost:9101/v1/table/media$vector$movies/describe)`. ## Limits @@ -412,8 +421,8 @@ S3 Vectors: discovered weed-vector at 127.0.0.1:8090 via master 172.17.0.3:9333 | Symptom | Cause | |---------|-------| | `ServiceUnavailableException: vector service unavailable` on `CreateIndex`, `PutVectors` or `QueryVectors`, and the gateway logs `no weed-vector registered on master` | No engine is registered: it is not running, it has no `--masters`, or it exited (for example on a busy `--listen` port). Bucket operations keep working. | -| `InternalFailure: internal error, see the S3 gateway log` on `CreateIndex`, and the gateway logs `(403 Forbidden) for url (http://localhost:9101/v1/table/.../describe)` | The S3 server has identities and the engine has no credential. Pass `--storage-option aws_access_key_id=...` and `--storage-option aws_secret_access_key=...`. | -| The same, with `401 Unauthorized` | The engine's access key is unknown or its secret is wrong. | +| `InternalFailure: internal error, see the S3 gateway log` on `CreateIndex` in an encrypted bucket, and the gateway logs `the bucket encrypts with SSE-S3, which needs s3.sse.kek or s3.sse.key for the engine as for the S3 gateway` | The engine has no SSE-S3 key. Give it the gateway's; see [Running the engine](#running-the-engine). | +| `weed-vector` exits with `error: unexpected argument '--storage-option' found` | A command line written for release 4.52 or earlier. Drop the `--storage-option` flags; the engine needs no S3 credentials. | | `weed-vector` exits with `Address already in use (os error 98)` | The `--listen` port is taken; see the ports note above. | | `PutVectors` fails with `ValidationException: request body exceeds 32 MiB` | Send fewer vectors per call. In our tests 500 vectors of 4096 dimensions took 9.8 to 33.2 MiB as JSON, depending on the digits per value. | | A plain HTTP request to `/ListVectorBuckets` gets `405` | Only requests signed for the `s3vectors` service reach the vectors API. Use an AWS CLI or SDK `s3vectors` client. | diff --git a/S3-Vectors-API.md b/S3-Vectors-API.md index 8aa4516..a336555 100644 --- a/S3-Vectors-API.md +++ b/S3-Vectors-API.md @@ -20,7 +20,7 @@ Every bucket and index can be addressed by name or by ARN. ARNs always use regio * **Names**: 3 to 63 characters, lowercase letters, digits and hyphens, starting and ending with a letter or digit. * **Deleting** a bucket that still has indexes fails with `ConflictException`. -* **Encryption**: with `encryptionConfiguration` `{"sseType": "AES256"}`, every object of the bucket's indexes is written with SSE-S3: `HeadObject` on them shows `ServerSideEncryption: AES256`, and queries work as in an unencrypted bucket. `GetVectorBucket` returns the setting. It needs an SSE-S3 key on the S3 gateway (`s3.sse.kek` or `s3.sse.key` in `security.toml`; `weed mini` creates one); without a key, `CreateVectorBucket` is refused with `AES256 encryption needs an SSE-S3 key on the S3 gateway: set s3.sse.kek or s3.sse.key in security.toml`. `aws:kms` is rejected with `unsupported sseType "aws:kms"`. +* **Encryption**: with `encryptionConfiguration` `{"sseType": "AES256"}`, every object of the bucket's indexes is written with SSE-S3: `HeadObject` on them shows `ServerSideEncryption: AES256`, and queries work as in an unencrypted bucket. `GetVectorBucket` returns the setting. It needs an SSE-S3 key on the S3 gateway (`s3.sse.kek` or `s3.sse.key` in `security.toml`; `weed mini` creates one); without a key, `CreateVectorBucket` is refused with `AES256 encryption needs an SSE-S3 key on the S3 gateway: set s3.sse.kek or s3.sse.key in security.toml`. `weed-vector` writes the index files itself and needs the same key, or `CreateIndex` fails with `InternalFailure`; with `weed mini`, pass it the generated one (see [[S3 Vector Bucket#running-the-engine]]). `aws:kms` is rejected with `unsupported sseType "aws:kms"`. * A vector bucket is a table bucket with format `LANCE`, and its objects are visible through the S3 API under `vector//`. ## Indexes