5 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandDevin	87f15e4d6c	vector: weed-vector-server, the VectorService gRPC (P1) (#12102) ... * vector: VectorService protobuf contract The shared RPC boundary between the Go S3 gateway and the Rust vector engine from the vector design doc: CreateIndex/DropIndex, Put/Get/ List/Delete/Query vectors. A Table reference carries the Lance table id and an optional pinned version, matching the snapshot discipline queries are required to keep. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: generated Go stubs for vector.proto protoc output via weed/pb/Makefile. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: carry return_data through QueryVectors S3 Vectors QueryVectors honours returnData, but the contract had no way to ask for or carry vector data in a query response, so the gateway would need a second GetVectors round-trip per query. Add return_data to the request and data to QueryResult now, while the contract is still new. * vector: pin metadata numbers to double precision in the contract Metadata arrives as JSON, where numbers carry double precision; say so on the field so neither side promises integer fidelity past 2^53. * vector: declare_table on the lance namespace client CreateIndex needs a way to ask the catalog for a new table's location; expose POST /v1/table/{id}/declare. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: cargo fmt coverage.rs Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: weed-vector-server, the VectorService gRPC A tonic server over weed-vector-core: CreateIndex declares the table through the namespace and writes an empty {key, vector, metadata} dataset; PutVectors upserts by key (delete + append until merge_insert is exercised); Get/List/Delete/Query route through the engine's resolved table, honoring pinned versions and the prefilter default. Metadata travels as a Utf8 JSON column, proto Struct on the wire. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: stamp the metric in the schema, tighten put validation CreateIndex now bounds dimension at the S3 Vectors 4096 maximum and records the distance metric in the vector field's Arrow metadata, so a later reader recovers the index's configured metric from the table itself. batch_of checks every vector's data length individually before flattening — equal totals could still let a short vector borrow floats from its neighbor — and writes against the dataset's own schema. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: upsert vectors in a single merge commit PutVectors ran delete-then-append: a failed append left the old keys deleted, and two concurrent puts could both append a replacement. Use lance's merge_insert on the key column — one commit, no delete window, conflicts fail the request instead of duplicating keys. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: read the index's metric; fetch metadata in one take QueryVectors hardcoded L2 — it now reads the metric the index was created with from the vector field's metadata. return_metadata fetched each hit's row one at a time; collect the row ids into a single take_rows call and join metadata back on the hit key. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: honor return_data in query results QueryResult grew a data field but query_vectors never populated it. Fetch the vector column in the same batched take_rows as metadata and return each hit's stored embedding. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: reject a malformed query as invalid_argument A zero top_k silently became one result and a dim-mismatched query vector surfaced as an internal scan error; reject both before querying. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: reject a malformed list next_token A token that failed to parse silently restarted the listing from offset zero; surface it as invalid_argument instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: answer an empty put without a commit and cover ragged input An empty vectors list would still run a merge commit; return the current version instead. Pin the ragged-vector check with a test. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: keep non_filterable_metadata_keys on the table schema create_index dropped the declared filter policy. Persist it as JSON on the metadata field so the index carries which keys a filter may touch; enforcing the list belongs to the gateway that translates the request filter into Lance SQL. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: reject duplicate keys in one put A repeated key for a new row committed both copies, and for an existing key the merge failure surfaced as internal. Reject duplicates up front as invalid_argument. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: report open failures as not_found or unavailable A catalog 404 or a missing dataset/version is not a client mistake, and a catalog outage is retryable: mapping every open error to invalid_argument made both indistinguishable from a bad request. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: page list_vectors when max_results is unset A proto3 client that omits max_results sends 0, which used to limit the scan to zero rows and return no token. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: drop stray empty service.rs.tmp * vector: rename datas map to data_map codespell flags datas as a misspelling of data; use a distinct map name. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-26 13:32:34 -07:00
Chris Lu	aff18b2336	vector: restore spike CLI flags lost in the #12098 rebase (#12099) ... * vector: --vector takes hyphenated floats again * vector: --vector-column/--key-column on the get subcommand * vector: --vector-column/--key-column on describe and coverage * vector: surface describe resolve errors when column flags are set * vector: cover the restored CLI flags with parse tests	2026-10-26 12:24:01 -07:00
Chris LuandDevin	8498ad716b	vector: spike CLI, reference-comparison suite, CI (P0 phase 3) (#12098) ... * vector: project _distance and _rowid explicitly Lance warns that a knn projection without _distance will stop auto-including it; spell both scoring columns out. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: reference-comparison tests The spike's exit gate: assert the engine's answers equal a bare scan().nearest() on the same pinned version — over a fresh index, over unindexed fragments, over a two-segment index after optimize_indices, and after a delete. A version-pinned handle querying through maintenance sees only its own version. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: weed-vector spike CLI describe / coverage / query / get against a namespace-registered table or a plain dataset URI, so the spike runs against a real SeaweedFS Lance table bucket and against a local fixture alike. Flag and env names mirror weed-worker. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: rust-vector-tests workflow Build and test seaweed-vector in release on pushes and PRs touching it or its path-dependency trees, mirroring rust-worker-tests. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: derive Serialize for the spike CLI's JSON output Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: --postfilter on the spike CLI Track the Query field rename; the flag stays an explicit opt-out of prefiltering for comparison runs. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: assert the pinned query's answers The maintenance test passed even with zero pinned hits: an empty result satisfies all() trivially. Assert the hit count and compare the pinned answers against the reference on the pinned handle. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: vector tests also watch plugin.proto weed-vector-core path-depends on weed-lance-worker, which compiles weed/pb/plugin.proto through seaweed-worker-core; a proto-only change would skip these checks and could break this workspace. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: describe prints layout and coverage The describe doc comment promised the vector layout and index coverage; print both, and propagate count_rows errors instead of printing 0. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-26 11:48:46 -07:00
3 people	eda4b9efdd	vector: ANN coverage introspection + prefilter (P0 phase 2) (#12097) ... * vector: share the test dataset builder The coverage tests need the same tempdir Lance datasets the query tests build; move them into a cfg(test) helper module. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: prefilter the metadata filter prefilter=false post-filters the top-K rows, which can return fewer than k results and never sees unindexed rows beyond the nearest cut. Expose the choice so the S3 Vectors layer can honour AWS filter semantics. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: index coverage introspection coverage() unions each index's fragment bitmaps against the manifest's fragment list to report which fragments a vector index covers, which it has not reached, and how many rows the flat pass still has to scan. Tests pin the behaviour the spike is for: rows appended after an index build stay in the top-K, and a version-pinned dataset never sees the next commit. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: default filtered queries to prefilter postfilter=false is now the normal path: the metadata filter cuts the eligible rows before ranking, so a query returns k hits whenever k eligible rows exist. Lance's post-filter remains reachable through Query::postfilter for comparison runs. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: intersect index bitmaps with the live manifest A bitmap records fragments the index covered when built; a fragment the manifest dropped afterwards stayed in the reported coverage. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: keep bitmap-less indices in coverage An index whose metadata has no fragment bitmap was dropped from the report entirely, indistinguishable from no index at all. Count its segments and mark its coverage unknown instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: count live rows in unindexed coverage physical_rows counts deleted rows and can be absent in the manifest; sum FileFragment::count_rows for the unindexed fragments instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: error on a null key in a selected key column * vector: compare filtered ANN and flat hits as sets * vector: coverage reports the checked-out version * vector: project _distance and _rowid explicitly --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chris@seaweedfs.com>	2026-10-26 11:45:07 -07:00
Chris LuandDevin	7388d0d4fc	vector: seaweed-vector workspace + flat/indexed top-K engine (P0 phase 1) (#12096) ... * vector: scaffold the seaweed-vector workspace Standalone cargo workspace for the vector query engine, mirroring the seaweed-worker layout: own lockfile, same lint policy, path dependencies across the two trees since there is no common root manifest. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: resolve a table's vector layout VectorTable::resolve finds the vector column (explicit name, else the column named vector, else the sole fixed-size float list), the optional key column, and the metadata columns a query can project. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: flat and indexed top-k, key lookup query::flat runs an exact scan (use_index(false)); query::ann lets Lance split the query between index segments and a flat pass over fragments no index covers yet, so rows written after the last index build stay eligible. Both honour a metadata filter and return the raw _distance in the query's metric. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: reject a non-utf8 or non-identifier key column An explicit --key-column naming a non-UTF8 field made hits() silently report row ids as keys, and get() interpolated the name into a SQL filter verbatim. Require Utf8 at resolve and in hits(), and require a plain identifier before building the IN predicate. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: fall back to row id on a null key keys.value() on a null slot fabricates an empty-string key; a row with no key reports its row id instead, matching the no-key-column case. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: quote the key column in the IN filter A Utf8 key column named e.g. document-key made the predicate parse as subtraction, and an identifier interpolated unquoted is injectable. Backtick-quote it and double its inner backticks; double quotes are string literals in the filter dialect, not identifiers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: project only the columns a hit needs The knn projection pulled every metadata column — including other vector columns — although Hit returns none of them. Filters still evaluate off the scan path, so restrict the output to the key and the scoring columns. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-26 10:13:05 -07:00