2 Commits

Include renames

Author	SHA1	Message	Date
3 people	eda4b9efdd	vector: ANN coverage introspection + prefilter (P0 phase 2) (#12097) ... * vector: share the test dataset builder The coverage tests need the same tempdir Lance datasets the query tests build; move them into a cfg(test) helper module. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: prefilter the metadata filter prefilter=false post-filters the top-K rows, which can return fewer than k results and never sees unindexed rows beyond the nearest cut. Expose the choice so the S3 Vectors layer can honour AWS filter semantics. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: index coverage introspection coverage() unions each index's fragment bitmaps against the manifest's fragment list to report which fragments a vector index covers, which it has not reached, and how many rows the flat pass still has to scan. Tests pin the behaviour the spike is for: rows appended after an index build stay in the top-K, and a version-pinned dataset never sees the next commit. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: default filtered queries to prefilter postfilter=false is now the normal path: the metadata filter cuts the eligible rows before ranking, so a query returns k hits whenever k eligible rows exist. Lance's post-filter remains reachable through Query::postfilter for comparison runs. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: intersect index bitmaps with the live manifest A bitmap records fragments the index covered when built; a fragment the manifest dropped afterwards stayed in the reported coverage. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: keep bitmap-less indices in coverage An index whose metadata has no fragment bitmap was dropped from the report entirely, indistinguishable from no index at all. Count its segments and mark its coverage unknown instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: count live rows in unindexed coverage physical_rows counts deleted rows and can be absent in the manifest; sum FileFragment::count_rows for the unindexed fragments instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: error on a null key in a selected key column * vector: compare filtered ANN and flat hits as sets * vector: coverage reports the checked-out version * vector: project _distance and _rowid explicitly --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chris@seaweedfs.com>	2026-10-26 11:45:07 -07:00
Chris LuandDevin	7388d0d4fc	vector: seaweed-vector workspace + flat/indexed top-K engine (P0 phase 1) (#12096) ... * vector: scaffold the seaweed-vector workspace Standalone cargo workspace for the vector query engine, mirroring the seaweed-worker layout: own lockfile, same lint policy, path dependencies across the two trees since there is no common root manifest. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: resolve a table's vector layout VectorTable::resolve finds the vector column (explicit name, else the column named vector, else the sole fixed-size float list), the optional key column, and the metadata columns a query can project. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: flat and indexed top-k, key lookup query::flat runs an exact scan (use_index(false)); query::ann lets Lance split the query between index segments and a flat pass over fragments no index covers yet, so rows written after the last index build stay eligible. Both honour a metadata filter and return the raw _distance in the query's metric. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: reject a non-utf8 or non-identifier key column An explicit --key-column naming a non-UTF8 field made hits() silently report row ids as keys, and get() interpolated the name into a SQL filter verbatim. Require Utf8 at resolve and in hits(), and require a plain identifier before building the IN predicate. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: fall back to row id on a null key keys.value() on a null slot fabricates an empty-string key; a row with no key reports its row id instead, matching the no-key-column case. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: quote the key column in the IN filter A Utf8 key column named e.g. document-key made the predicate parse as subtraction, and an identifier interpolated unquoted is injectable. Backtick-quote it and double its inner backticks; double quotes are string literals in the filter dialect, not identifiers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: project only the columns a hit needs The knn projection pulled every metadata column — including other vector columns — although Hit returns none of them. Filters still evaluate off the scan path, so restrict the output to the key and the scoring columns. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-26 10:13:05 -07:00