* vector: weed-vector registers with the weed master
weed-vector-server --masters holds a KeepConnected stream to the weed
master, the same mechanism filer and s3 use via wdclient — the server
shows up in ListClusterNodes(client_type=vector). Resends registration
and rediscovers on failure.
* vector: s3 discovers weed-vector through the master
With -vectors.grpc unset the vectors gateway resolves one registered
weed-vector via ListClusterNodes on the masters it already derives
from filer config — discovery replaces the static dial for the
single-worker OSS path. The conformance harness registers the engine
to mini's master and drops -vectors.grpc.
* vector: default-fill registration fields
..Default::default() keeps the source compiling against proto variants
that carry extra fields, so the same file works in either tree.
* vector: serve the Vectors API on the main s3 port
The vectors router sits on the main -port listener behind a dispatch
that checks the SigV4 credential scope for /s3vectors/ — no separate
port needed. -port.vectors stays for explicit dedicated-port
deployments; the conformance harness now hits vectors through :8334.
* vector: drop -port.vectors entirely
It was new — no compat to preserve. The main -port listener serves the
API whenever a weed-vector backend is reachable.
* vector: drop -vectors.grpc; discovery is the only locator
weed-vector always registers to master, the gateway always resolves it
through ListClusterNodes — a static address just forks membership.
* vector: binary is weed-vector
Matches the enterprise binary; package and lib names are unchanged.
* vector: review fixes — cluster type, grpc port, lazy resolve, leader hint, advertise
- cluster: register 'vector' as a node type so ListClusterNodes returns
weed-vector members
- s3: dial masters via ToGrpcAddress (HTTP port + 10000)
- s3: backend resolves lazily through vectorDiscovery — workers joining
after gateway startup work on first query, and address changes re-dial
- master_client: convert master HTTP addr to grpc +10000, honor http(s)
scheme, follow the leader hint in KeepConnectedResponse, and
--advertise covers wildcard/unreachable listen addresses
* vector: no --advertise flag; derive the outbound address
A wildcard listen resolves to the source IP the master sees via a
route lookup, so registration always advertises something reachable
without another flag.
* vector: review fixes — plaintext dispatch, conn leaks, discovery backoff
The two plaintext http servers now serve the vectors dispatch too —
signed s3vectors calls on the main port reach the right router.
discoverVectorWorker closes each master conn and bounds lookups to 5s;
a dead or absent worker no longer re-hits master on every RPC — the
resolver backs off for 5s between rediscovery attempts.
* vector: 1 GiB grpc cap, dead-worker invalidation, CLI rename
Valid PutVectors batches exceed the tonic 4 MiB default — match the
gateway's 1 GiB cap. A replaced worker no longer sticks: any RPC error
resets the cached address and the next call rediscovers through the
master. The core CLI bin renames to weed-vector-cli so the two crates
do not collide on the same output path.
* vector: retry-only-transport reset, periodic re-resolve, cli rename
- reset() takes the failed client and runs only on transport errors —
an InvalidArgument on a healthy conn no longer interrupts unrelated
requests, and a concurrent re-resolve wins the race.
- cli() re-resolves once the interval passes even with a live client,
so a worker that moves addresses is picked up.
- verify.rs spike docs use weed-vector-cli after the bin rename.
* vector: only Unavailable counts as a transport failure
A request deadline or an application error can fire on a healthy
connection — resetting the shared conn on those churned reconnects.
* test/vectors: give the vector server static storage credentials
The e2e harness ran weed-vector-server without credentials, so lance's
AWS object store fell back to IMDS and CreateIndex died on the runner's
411 from the token endpoint. Point it at the test credentials the s3
gateway now reads from s3.json.
* vector: take the list item field from the schema when batching vectors
Lance stores the fixed-size list item as nullable even when the write
schema declared it non-null, so PutVectors built batches that no longer
matched: expected FixedSizeList(4 x Float32) but found FixedSizeList(4 x
non-null Float32).
* vector: store index metadata as lance jsonb so filters resolve
The metadata field was plain Utf8, but the filter functions lance's
planner registers (json_get_string, json_get_float, json_exists) take a
LargeBinary jsonb column, so QueryVectors with a metadata filter failed
to plan. Declare the field with the arrow.json extension — lance encodes
it to jsonb on write and back on read — and encode the batch's strings
when the stored column is LargeBinary. Also rename the filter's
json_get_str call to json_get_string, the name lance registers.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: boto3 s3vectors conformance harness
test/vectors spins a mini cluster, an s3 gateway with -port.vectors and
-vectors.grpc, and a weed-vector-server, then exercises the AWS client
end to end: bucket and index lifecycle, put/get/list/delete vectors,
and a filtered query asserting distances and metadata.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run the s3vectors conformance subset
An e2e job in rust-vector-tests: builds weed and weed-vector-server,
then runs test/vectors against a mini cluster plus the vectors gateway.
The workflow now also watches vector.proto, its stubs, and test/vectors.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: fix e2e protoc and codespell findings
The e2e job needed the same vendored-protoc step as the unit-test job —
lance's build scripts locate protoc themselves. Codespell flags the AWS
`$nin` operator literal and the `lits` variable: exempt `nin` in
.codespellrc like the existing word exemptions, rename the variable,
and fold the negation into the IN/NOT IN symbol directly.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: classify catalog errors by code, not message text
* s3: drop the implicit vector namespace on vector-bucket delete
* test: point the vector engine at the gateway lance port
* test: bind the vectors gateway listeners to loopback
* test: make the filter and distance assertions load-bearing
* test: rebuild the harness binaries on every run
* ci: fix the vectors end-to-end job
* ci: run the vector tests on gateway and command changes
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Chris Lu