* test: align vector bucket policy coverage with vector actions
* s3vectors: validate ARN service and resource shape before resolving indexes
* test: exercise signed vector bucket policies and catalog lifecycle
* ci: require the expanded vector integration matrix and all gateway tests
* test: reject index ARNs in bucket-level API requests
* s3vectors: scope ARN resolution to the gateway namespace
* s3vectors: reject name fields that conflict with resolved ARNs
bucketARNs and resolveIndexARNs already reject an indexArn that names a
different bucket than vectorBucketArn, but a vectorBucketName or
indexName disagreeing with the ARNs was silently dropped. Validate the
plain names against the resolved ARN values the same way.
* test: cover conflicting vector names and ARNs in live requests
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: reject empty boolean-operator lists and sub-objects in filters
An empty $and/$or list or a nested empty object rendered '()'' — invalid
SQL surfaced as an engine error instead of a clean ValidationException.
* s3vectors: e2e conformance for vector bucket policy evaluation
The handler-level test covers statements; this runs the whole stack a
client sees — HTTP dispatch, authentication, policy evaluation, catalog,
engine — and walks the full lifecycle: a non-owner is denied without a
grant, an Allow on the bucket ARN grants every index under it, a Deny
vetoes an Allow on the same action, and deleting the policy restores
owner-only access.
* s3vectors: prove deny precedence and grant removal in policy e2e
* s3vectors: reject field operator objects with no operators
* test: use an s3vectors action in the vector policy round-trip
* s3vectors: reuse shared authenticated identity context
* s3vectors: encode empty results as arrays
* test: cover S3 Vectors with Go and Rust volume servers
* ci: require both volume implementations in vector integration tests
* s3vectors: keep identityOf as a shim over the shared identity context
Concurrent branches add identityOf call sites; deleting the helper would
break the build when they merge in either order. Route it through
GetIdentityNameFromContext so every caller resolves the same identity.
* test: wait for the vector engine to register before running
The gateway resolves weed-vector lazily through the master cluster list,
so a listening port does not mean the engine is reachable yet; the first
vector write or query could race KeepConnected and fail. Restore the
cluster.ps registration wait the previous harness had, and ignore the
pytest cache directory the suite creates.
* test: wait for master registration across OSS and enterprise
Use the exact engine registration event after AddClusterNode completes. The current enterprise cluster.ps does not list vector nodes, so waiting on its output prevents the shared fixture from starting there.
* s3vectors: keep contextWithIdentity as a shim alongside identityOf
In-flight callers still write the caller name through it; route the
write through SetIdentityNameInContext so both helpers agree on the
shared key whichever lands first.
* util: do not bind outbound dials to a loopback -ip.bind
`weed server -ip.bind=127.0.0.1` keeps every listener local, but the
same address became the source of outbound connections, so the filer
dialled its remote mount from 127.0.0.1 and failed with "connect:
invalid argument". A loopback source cannot reach off the host; leave
the source to the OS, as for a wildcard bind.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: put, get and delete vector bucket policies
The AWS S3 Vectors policy operations had no handlers; the policy document
is stored on the shared table bucket, so the handlers delegate to the
catalog's PutTableBucketPolicy, GetTableBucketPolicy and
DeleteTableBucketPolicy and keep its guard.
* s3vectors: treat ICEBERG buckets as absent in policy handlers
GetVectorBucketPolicy/PutVectorBucketPolicy/DeleteVectorBucketPolicy
delegated straight to the table bucket, so a bucket created for ICEBERG
tables — invisible to every other vectors operation — could still have
its policy read and written through the vectors API. Gate all three on
lanceBucket, the same format check DeleteVectorBucket and CreateIndex
already apply.
* test/vectors: cover vector bucket policy put/get/delete
Round-trips a policy document through the new ops and asserts a get on
the deleted policy surfaces NotFoundException, so the boto3 conformance
run exercises the added handlers end to end.
* s3vectors: validate the bucket ARN and policy document
Bucket-level operations now reject an index ARN and an ARN minted for
another service, and PutVectorBucketPolicy refuses a document carrying
s3vectors actions — the evaluator only honors s3tables actions, so such a
rule would be a dead grant.
* s3vectors: validate the bucket ARN on the remaining bucket-level operations
* s3vectors: cover bucket ARN and policy document validation
TestBucketARNs pins the accepted forms and the index/foreign-service
rejections; TestValidatePolicyDocument covers a valid document, a
malformed one, and the dead s3vectors action namespace.
* s3vectors test: use an enforceable action in the bucket policy test
The catalog refuses s3vectors actions in a stored policy, so the put call
rejected the document before the get and delete assertions ran.
---------
Co-authored-by: Ólafur Bjarki Bogason <olafur@genkiinstruments.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The vectors gateway backs off discovery for 5s between resolutions, but
the backoff also suppressed rediscovery after an empty lookup. A worker
that landed its KeepConnected registration a moment after the first
query stayed invisible for the whole window — longer than a client's
retry budget — so the S3 Vectors e2e flaked: CreateIndex raced the
registration, got InternalFailure, and every boto retry fell inside the
same 5s fast-fail window.
With no cached backend, recheck every 500ms instead; the 5s interval now
only bounds membership refreshes against a live client, and an absent
worker still cannot flood the master with lookups. The empty-membership
error also becomes codes.Unavailable so the API answers 503 rather than
InternalFailure.
cluster.ps lists registered vector nodes like the other node types, and
the e2e harness polls it for the engine's registration instead of only
checking the gRPC port — a listening engine is not discoverable until
the registration lands.
* vector: weed-vector registers with the weed master
weed-vector-server --masters holds a KeepConnected stream to the weed
master, the same mechanism filer and s3 use via wdclient — the server
shows up in ListClusterNodes(client_type=vector). Resends registration
and rediscovers on failure.
* vector: s3 discovers weed-vector through the master
With -vectors.grpc unset the vectors gateway resolves one registered
weed-vector via ListClusterNodes on the masters it already derives
from filer config — discovery replaces the static dial for the
single-worker OSS path. The conformance harness registers the engine
to mini's master and drops -vectors.grpc.
* vector: default-fill registration fields
..Default::default() keeps the source compiling against proto variants
that carry extra fields, so the same file works in either tree.
* vector: serve the Vectors API on the main s3 port
The vectors router sits on the main -port listener behind a dispatch
that checks the SigV4 credential scope for /s3vectors/ — no separate
port needed. -port.vectors stays for explicit dedicated-port
deployments; the conformance harness now hits vectors through :8334.
* vector: drop -port.vectors entirely
It was new — no compat to preserve. The main -port listener serves the
API whenever a weed-vector backend is reachable.
* vector: drop -vectors.grpc; discovery is the only locator
weed-vector always registers to master, the gateway always resolves it
through ListClusterNodes — a static address just forks membership.
* vector: binary is weed-vector
Matches the enterprise binary; package and lib names are unchanged.
* vector: review fixes — cluster type, grpc port, lazy resolve, leader hint, advertise
- cluster: register 'vector' as a node type so ListClusterNodes returns
weed-vector members
- s3: dial masters via ToGrpcAddress (HTTP port + 10000)
- s3: backend resolves lazily through vectorDiscovery — workers joining
after gateway startup work on first query, and address changes re-dial
- master_client: convert master HTTP addr to grpc +10000, honor http(s)
scheme, follow the leader hint in KeepConnectedResponse, and
--advertise covers wildcard/unreachable listen addresses
* vector: no --advertise flag; derive the outbound address
A wildcard listen resolves to the source IP the master sees via a
route lookup, so registration always advertises something reachable
without another flag.
* vector: review fixes — plaintext dispatch, conn leaks, discovery backoff
The two plaintext http servers now serve the vectors dispatch too —
signed s3vectors calls on the main port reach the right router.
discoverVectorWorker closes each master conn and bounds lookups to 5s;
a dead or absent worker no longer re-hits master on every RPC — the
resolver backs off for 5s between rediscovery attempts.
* vector: 1 GiB grpc cap, dead-worker invalidation, CLI rename
Valid PutVectors batches exceed the tonic 4 MiB default — match the
gateway's 1 GiB cap. A replaced worker no longer sticks: any RPC error
resets the cached address and the next call rediscovers through the
master. The core CLI bin renames to weed-vector-cli so the two crates
do not collide on the same output path.
* vector: retry-only-transport reset, periodic re-resolve, cli rename
- reset() takes the failed client and runs only on transport errors —
an InvalidArgument on a healthy conn no longer interrupts unrelated
requests, and a concurrent re-resolve wins the race.
- cli() re-resolves once the interval passes even with a live client,
so a worker that moves addresses is picked up.
- verify.rs spike docs use weed-vector-cli after the bin rename.
* vector: only Unavailable counts as a transport failure
A request deadline or an application error can fire on a healthy
connection — resetting the shared conn on those churned reconnects.
* test/vectors: give the vector server static storage credentials
The e2e harness ran weed-vector-server without credentials, so lance's
AWS object store fell back to IMDS and CreateIndex died on the runner's
411 from the token endpoint. Point it at the test credentials the s3
gateway now reads from s3.json.
* vector: take the list item field from the schema when batching vectors
Lance stores the fixed-size list item as nullable even when the write
schema declared it non-null, so PutVectors built batches that no longer
matched: expected FixedSizeList(4 x Float32) but found FixedSizeList(4 x
non-null Float32).
* vector: store index metadata as lance jsonb so filters resolve
The metadata field was plain Utf8, but the filter functions lance's
planner registers (json_get_string, json_get_float, json_exists) take a
LargeBinary jsonb column, so QueryVectors with a metadata filter failed
to plan. Declare the field with the arrow.json extension — lance encodes
it to jsonb on write and back on read — and encode the batch's strings
when the stored column is LargeBinary. Also rename the filter's
json_get_str call to json_get_string, the name lance registers.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: boto3 s3vectors conformance harness
test/vectors spins a mini cluster, an s3 gateway with -port.vectors and
-vectors.grpc, and a weed-vector-server, then exercises the AWS client
end to end: bucket and index lifecycle, put/get/list/delete vectors,
and a filtered query asserting distances and metadata.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run the s3vectors conformance subset
An e2e job in rust-vector-tests: builds weed and weed-vector-server,
then runs test/vectors against a mini cluster plus the vectors gateway.
The workflow now also watches vector.proto, its stubs, and test/vectors.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: fix e2e protoc and codespell findings
The e2e job needed the same vendored-protoc step as the unit-test job —
lance's build scripts locate protoc themselves. Codespell flags the AWS
`$nin` operator literal and the `lits` variable: exempt `nin` in
.codespellrc like the existing word exemptions, rename the variable,
and fold the negation into the IN/NOT IN symbol directly.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: classify catalog errors by code, not message text
* s3: drop the implicit vector namespace on vector-bucket delete
* test: point the vector engine at the gateway lance port
* test: bind the vectors gateway listeners to loopback
* test: make the filter and distance assertions load-bearing
* test: rebuild the harness binaries on every run
* ci: fix the vectors end-to-end job
* ci: run the vector tests on gateway and command changes
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
