* vector: compare json_array_contains elements as typed JSON values
Lance's json_array_contains compares each element's text with the value, or
with the value wrapped in quotes, so 1 matches "1", true matches "true", and
1.0 misses a stored 1. The filter swap now replaces it like json_extract and
json_exists: the value is parsed as JSON once per batch and compared with
jsonb's typed equality, and only literal arguments are swapped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: match a filter value against the elements of an array
A filter never looked inside an array: {"tags": "space"} and
{"tags": {"$in": ["space"]}} both missed "tags": ["space", "classic"].
AWS documents that $eq, and so a bare value, matches an array metadata value
when any element equals the input. $eq, a bare value and each $in member now
also match through json_array_contains; $ne and $nin remain their negations.
Range operators, $exists and nested objects are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test(s3vectors): add an opt-in end-to-end perf suite
perf_test.py runs on the same cluster fixture as the integration suite
(Go or Rust volume, single or distributed queries) and times the data
plane through the S3 Vectors API: PutVectors ingest and upsert, QueryVectors
latency with and without metadata and a 10% filter, query throughput
across client threads, GetVectors, segmented ListVectors, and
DeleteVectors. Recall is checked against an exact client-side search and
listings must be complete; timings only report. Sizes come from PERF_*
variables, and results go to a JSON report and a printed summary.
It runs through run_perf.sh or `make perf`. run_test.sh does not collect
it, so CI is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): check every listed key in the perf suite, not just the count
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): cap perf samples at PERF_VECTORS so small runs work
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): warm perf worker clients before starting throughput timers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): report the key count of perf GetVectors and DeleteVectors calls
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: parse a filter's JSONPath once per batch instead of per row
Lance's json_extract and json_exists parse their JSONPath argument again
for every row they see. Every translated S3 Vectors filter goes through
them, so a flat scan paid about 1 µs per row before the predicate
compared anything, and a 10%-selective {"category": {"$eq": "c3"}} ran
about 50% slower than the same query unfiltered.
The path is always a literal, so the scanner now parses the filter as
before and swaps in copies of the two functions that parse it once per
batch. They call the same jsonb selector and serializer, so results and
errors do not change. Calls whose path is not a literal keep Lance's
functions. The filter SQL is unchanged, so the gateway, existing tables
and the distributed workers (which run query::ann) need no change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: read the field once per extractor in $in and $nin
$in and $nin rendered one IN per member, each with its own json_extract,
so an N-member list parsed the row's metadata N times. Members that share
an extractor (strings and booleans, or numbers) now go into one IN list,
so a list reads the field at most twice. Matching is unchanged: each
group still only sees its own JSON type and still coalesces NULL before
$nin negates it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: rename the $in literal map so codespell passes
Codespell reads "lits" as a misspelling of "list".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bound ListVectors maxResults and segment parameters
segmentCount had no upper bound, so segmentCount=1000000,
segmentIndex=999999 made the segment emulation scan the whole index to
return nothing. Apply the AWS ListVectors windows before authorization
and any engine call: maxResults 1..1000, segmentCount 1..16,
segmentIndex in [0, segmentCount), and segmentCount and segmentIndex
given together. The fields become pointers so an explicit 0 is rejected
while an omitted one still takes the default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: store and return distanceMetric in lowercase
CreateIndex stored distanceMetric exactly as sent, so an index created
with "COSINE" came back as "COSINE" from GetIndex and QueryVectors.
AWS returns the lowercase enum values. Lowercase the metric when the
index is created, and again when the record is read so indexes already
stored in mixed case render the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: data-plane lifecycle integration test
TestVectorDataPlaneLifecycle spawns a real weed-vector against the lance
listener, mirrors the pytest cluster's flags, waits for the gateway to
discover the registered engine, and walks the full lifecycle: create
bucket and index, put/get/list vectors, a filtered query that excludes the
nearer non-matching vector, delete vectors, index, and bucket. The test
skips when no engine binary resolves (VECTOR_BINARY, ./weed-vector, PATH).
* test/s3vectors: fail fast when the engine dies before listening
* test/s3vectors: harden engine launch and tighten lifecycle assertions
* test/s3vectors: wait on the engine's exit once through a broadcast channel
* test: move the s3vectors pytest suite under test/s3vectors
The vectors API tests lived in test/vectors next to the new s3vectors
integration home; the rust-vector-tests workflow's paths and working
directory move with it.
* gitignore: cover the binaries the s3vectors Makefile builds
* test/s3vectors: integration test design and catalog-surface suite
The design doc lays out a two-tier suite: catalog/policy ops that need no
vector engine run against a real weed server over HTTP, while the data
plane skips itself when no engine is reachable. The first test brings up
the server and exercises the bucket/policy lifecycle, encryption-config
round-trip, and pre-engine validation boundaries.
* test/s3vectors: sign API calls so requests reach the vectors router
The gateway selects the vectors router off the SigV4 credential scope —
unsigned requests with only X-Amz-Target fall through to the plain S3
router, so every call in the suite failed before dispatch. Sign each
request with the s3vectors service name and point the spawned server at
a written identity file so the signature is actually verified.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/s3vectors: bound the spawned server's ports, listeners, and clients
weed server derives each gRPC listener as HTTP+10000, so ephemeral
allocation could hand back a port whose pair is invalid or taken; the
-s3 defaults also bind the Iceberg and Lance listeners. Allocate the
HTTP/gRPC pairs together, pin -ip/-ip.bind, disable the extra listeners,
and give the readiness probe and API client deadlines.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/s3vectors: verify policy round-trip and gate the engine skip on 503
The stored policy check accepted any JSON body, and a delete that left
the document behind could not be caught — compare the returned document
verbatim and assert a follow-up Get returns 404. The engine probe never
created its bucket, so a healthy engine still produced a skip; create it
first, skip only on a confirmed 503, and fail on any other error, then
exercise the catalog-side index ops when an engine answers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
