* test/vectors: give the vector server static storage credentials
The e2e harness ran weed-vector-server without credentials, so lance's
AWS object store fell back to IMDS and CreateIndex died on the runner's
411 from the token endpoint. Point it at the test credentials the s3
gateway now reads from s3.json.
* vector: take the list item field from the schema when batching vectors
Lance stores the fixed-size list item as nullable even when the write
schema declared it non-null, so PutVectors built batches that no longer
matched: expected FixedSizeList(4 x Float32) but found FixedSizeList(4 x
non-null Float32).
* vector: store index metadata as lance jsonb so filters resolve
The metadata field was plain Utf8, but the filter functions lance's
planner registers (json_get_string, json_get_float, json_exists) take a
LargeBinary jsonb column, so QueryVectors with a metadata filter failed
to plan. Declare the field with the arrow.json extension — lance encodes
it to jsonb on write and back on read — and encode the batch's strings
when the stored column is LargeBinary. Also rename the filter's
json_get_str call to json_get_string, the name lance registers.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: boto3 s3vectors conformance harness
test/vectors spins a mini cluster, an s3 gateway with -port.vectors and
-vectors.grpc, and a weed-vector-server, then exercises the AWS client
end to end: bucket and index lifecycle, put/get/list/delete vectors,
and a filtered query asserting distances and metadata.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run the s3vectors conformance subset
An e2e job in rust-vector-tests: builds weed and weed-vector-server,
then runs test/vectors against a mini cluster plus the vectors gateway.
The workflow now also watches vector.proto, its stubs, and test/vectors.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: fix e2e protoc and codespell findings
The e2e job needed the same vendored-protoc step as the unit-test job —
lance's build scripts locate protoc themselves. Codespell flags the AWS
`$nin` operator literal and the `lits` variable: exempt `nin` in
.codespellrc like the existing word exemptions, rename the variable,
and fold the negation into the IN/NOT IN symbol directly.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: classify catalog errors by code, not message text
* s3: drop the implicit vector namespace on vector-bucket delete
* test: point the vector engine at the gateway lance port
* test: bind the vectors gateway listeners to loopback
* test: make the filter and distance assertions load-bearing
* test: rebuild the harness binaries on every run
* ci: fix the vectors end-to-end job
* ci: run the vector tests on gateway and command changes
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: nil-check metadataConfiguration before reading its keys
* s3: honor maxResults/nextToken in segmented ListVectors
* s3: exclude non-Lance buckets from ListVectorBuckets
* s3: test segmented ListVectors paging token math
* ci: silence codespell on the "$nin" operator and rename lits
* vector: S3 Vectors API gateway over VectorService
weed/s3api/s3vectors serves the AWS S3 Vectors JSON API: bucket CRUD
maps to S3 Tables table buckets, index CRUD to Lance table records plus
the VectorService CreateIndex/DropIndex RPCs, and the data plane
(Put/Get/List/Delete/QueryVectors) calls the shared service. The AWS
filter document translates to a Lance SQL predicate over the metadata
JSON column; unsupported operators fail loudly rather than overmatch.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: -port.vectors and -vectors.grpc on weed s3
Mirror the Lance server startup: the S3 Vectors API binds its own port
and dials the weed-vector-server gRPC address; both disabled by default.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: resolve index ARNs, authorize index ops, fix catalog fields
- vector buckets declare FormatLance so a new bucket accepts index
tables instead of defaulting to ICEBERG.
- ListIndexes decodes the `tables` response field; `tableSummaries`
hid every index.
- resolveIndexARNs honors indexArn alone; handlers no longer treat
indexName as the only carrier.
- data-plane RPCs fetch the index record first, so callers authorize
against the index resource and a missing index 404s before touching
the engine; QueryVectors reports the index's configured metric.
- IndexArn added to the index-level request types.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: serve the vectors port over TLS when configured
When -key.file/-cert.file are set, the vectors listener mirrors the
main API and serves TLS rather than a second plaintext endpoint.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: harden the s3vectors gateway handlers
- $ne no longer shadows the operator: exclusion filters emit <>
instead of equality.
- a null element in a PutVectors list fails the request instead of
panicking.
- requests are capped at 32 MiB; distances return via *float32 so an
exact match still reports 0 when returnDistance is set.
- CreateIndex declares the `vector` namespace first and validates
dimension (1..=4096) before the catalog write; a failed
VectorService.CreateIndex rolls the table record back.
- non-filterable metadata keys persist as JSON rather than a
comma-joined string, and index records carry their createdAt.
- ListVectors honors segmentIndex/segmentCount by slicing the full
listing in the gateway.
- catalog errors map to AccessDeniedException/ConflictException instead
of a blanket ValidationException, and malformed ARNs 400 instead of
slicing.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: /{Operation} routes and vectors-port TLS, conn cleanup
Path-dispatched clients (POST /QueryVectors) reach the same handlers as
X-Amz-Target calls. A failed listener bind now closes the vector gRPC
connection instead of leaving it open with the endpoint disabled.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
