* vector: scan only the fragments a ListVectors page covers
A page pushed its offset into the scan, but the scan still loaded every
fragment first, reading each one's deletion file: about 105 S3 GETs per
page at 50k rows against 25 at 10k. The page's fragments now come from the
live row counts in the manifest, so a page reads only its own. Tokens and
row order are unchanged; a manifest without the counts keeps the old scan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: list a ListVectors segment in the engine
The gateway paged through the whole index for every segment and kept one
row in segmentCount, so a segment cost segmentCount engine pages and 16
segments listed slower than one. The engine now takes segment_count and
segment_index: it reads keys across the stride, keeps the segment's rows
and takes their data and metadata alone, one call per page.
A segment is still the rows whose position modulo segmentCount is
segmentIndex, and its token still the position to resume from. An engine
that predates the fields does not set segmented, and the gateway then
pages through it as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: read only the key column across the table in GetVectors
The key lookup scanned every column. On S3, lance reads a column during
the scan when a row of it is under 1000 bytes, so a 128-dimension vector
was read for every row of the table to return 100 of them, and data and
metadata were read even when the request did not ask for them. The scan
now projects the requested columns and materializes everything but the
key late, for the matching rows only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: start the legacy segment fallback from the page already read
With an engine that ignores segment_count, the gateway dropped the page
it had just read and fetched it again. The fallback now consumes that
page first, so a one-row segment page still costs one engine call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: check metadata on segmented ListVectors pages
The segment test listed data only. Its rows now carry metadata on two in
three keys, and every segmented row, read through take_rows, must equal
the same row in a plain listing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: lease the version while a segment takes its data
A segmented page with data or metadata reads keys first and then takes
the other columns from the same version, so maintenance could reclaim
its files in between. That path now holds a lease like GetVectors. A
keys-only page is one scan and stays unleased, as plain pages are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: leave GetVectors materialization to lance
Forcing every column late made a 100-key GetVectors with data slower at
10k x 128 in the perf cluster: 54.7 ms against 46.3 ms. The take issued
a read per row, where lance's heuristic reads a narrow vector column in
one pass per fragment. The projection keeps the gain on its own: with
neither data nor metadata requested, the call went from 38.6 ms to
25.3 ms and from 150 to 89 S3 GETs. At 50k x 384 the vector is already
late under the heuristic, and a full lookup is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: VectorService protobuf contract
The shared RPC boundary between the Go S3 gateway and the Rust vector
engine from the vector design doc: CreateIndex/DropIndex, Put/Get/
List/Delete/Query vectors. A Table reference carries the Lance table id
and an optional pinned version, matching the snapshot discipline queries
are required to keep.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: generated Go stubs for vector.proto
protoc output via weed/pb/Makefile.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: carry return_data through QueryVectors
S3 Vectors QueryVectors honours returnData, but the contract had no
way to ask for or carry vector data in a query response, so the
gateway would need a second GetVectors round-trip per query. Add
return_data to the request and data to QueryResult now, while the
contract is still new.
* vector: pin metadata numbers to double precision in the contract
Metadata arrives as JSON, where numbers carry double precision; say
so on the field so neither side promises integer fidelity past 2^53.
* vector: clarify the query filter is raw Lance SQL over one metadata column
The doc said 'metadata columns' but the table stores all metadata in a
single JSON column named metadata; the gateway translates S3 Vectors
filter expressions onto it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
