Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-10-28 16:00:37 +01:00
Code Issues Packages Projects Releases Wiki Activity
15,802 Commits 226 Branches 364 Tags
ee315527cba08dda648d4a3be32dff7c1d93cb49
Commit Graph
7 Commits
Author SHA1 Message Date
Chris LuandDevin ee315527cb vector: address remaining s3vectors review findings (#12115) 
* vector: address remaining s3vectors review findings

Catalog and permission model:
- Vector buckets are declared Lance table buckets end to end: create sets
  format=lance, get/list/delete reject or skip ICEBERG buckets, and the
  format lookup runs on an internal trusted catalog so callers need no
  GetTableBucket grant.
- The implicit "vector" index namespace is created through the same
  internal manager, so a CreateTable grant alone authorizes CreateIndex;
  typed S3TablesError codes replace string matching on err.Error().
- CreateIndex pre-checks existence after the authorize, tags the record
  with a create nonce, and rolls back on engine failure under
  context.WithoutCancel with a versionToken CAS, so a retry never deletes
  a table it did not create.
- DeleteIndex drops the catalog record directly; DropIndex is
  unimplemented on the shared VectorService.
- DeleteVectorBucket removes the empty internal namespace on
  BucketNotEmpty and retries once.
- PutVectors/DeleteVectors require the update grant, and engine gRPC
  errors map to typed API errors instead of generic 500s.

API correctness:
- CreateVectorBucket/CreateIndex return their resource ARNs.
- ARNs reject empty bucket segments; vectorBucketArn must be an ARN.
- ListIndexes returns summaries built only from catalog fields.
- ListVectors honors maxResults/nextToken inside a segment via
  "seg:<pageStart>:<nextIndex>" resume tokens and validates
  segmentCount/segmentIndex.
- QueryVectors requires positive topK; PutVectors rejects empty keys.
- CreateVectorBucket rejects a supplied encryptionConfiguration rather
  than silently dropping it.
- Metadata filters emit Lance's real JSONB UDFs: json_get_string/
  json_get_float/json_get_bool picked by operand type, json_exists with a
  bracketed JSONPath, and per-member accessors for $in/$nin.

Engine:
- weed-vector-server stores the metadata column as Lance JSONB
  (lance.json extension) so the json_* UDFs apply; reads accept legacy
  Utf8 JSON columns.

Vectors port TLS:
- The TLS config is built before binding listeners, loads -cacert.file
  into ClientCAs when client-cert verification is on, fails startup on an
  empty/invalid CA file, and closes the gRPC conn on every failure path.

* s3vectors: authorize index create before writing the implicit namespace

A denied caller could still leave a persistent namespace behind because
ensureIndexNamespace wrote through the trusted catalog before the
caller's create grant ran. AuthorizeCreateTable now tolerates the
not-yet-declared implicit namespace (bucket policy alone decides) and
runs before any write; real table creates keep requiring the namespace.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3vectors: keep engine error details out of the API response

Unavailable now maps to 503 ServiceUnavailableException and unclassified
codes return a generic message; the real gRPC error stays in the log.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3vectors: exact-type metadata filters + NULL-safe $nin

The jsonb extractors convert numeric-looking strings through
json_get_float and 'true'/'false' through json_get_bool, so a stored '3'
could satisfy a filter for the number 3. Numeric and boolean comparisons
now also require the stored value not be a string, and each $in/$nin
group coalesces NULL so NOT cannot invert NULL into NULL.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: validate the vectors TLS key pair before binding

The reloading certificate provider reads the files asynchronously, so a
missing or mismatched pair would only surface as failed handshakes on an
announced HTTPS port. Load it once up front.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: make a token-checked table delete atomic

The version token was read in one filer call and enforced in another, so
a delete+recreate slipping between them let the delete erase the
replacement. When a token is supplied the name path now carries the
observed mtime as an if_not_modified_after precondition (evaluated under
the path lock), and a decoupled delete strips the catalog marker under a
compare of the captured attributes before touching the data path.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3vectors: compare filter operands against the stored JSON type

jsonb's strict converters make json_get_string return text for numbers
and booleans too, so a get_string IS NULL guard is never true for scalar
metadata — numeric and boolean comparisons silently matched nothing.
Numbers and booleans now compare through TRY_CAST of the serialized
json_extract text, which yields NULL on any other stored type; string
comparisons stay on json_get_string behind a CASE guard so the extractor
only runs on rows storing a JSON string.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: allocate implicit namespace metadata before use

A first table create in an undeclared namespace took the
allowMissingNamespace branch with a nil record and panicked on the
owner assignment.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: read the delete target's metadata and snapshot in one lookup

Two separate lookups let a delete+recreate in between validate the
caller token against the old record while the conditional delete aimed
at the replacement.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: assert the captured record before a conditional table delete

An xattr update keeps Mtime, so the IfNotModifiedAfterNs guard alone
could not see a metadata write that had already rotated the version
token, and a recreate preserving the mtime could slip the mtime check.
The delete now first proves the entry still carries the captured
attributes under the filer's path lock, then keeps the mtime
precondition.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3vectors: match only JSON boolean literals in metadata filters

* filer,s3tables: delete versioned tables under one attribute precondition

* s3vectors: compare serialized JSON for string filter operands

json_get_string converts numeric-looking strings and booleans, so it
cannot serve exact-typed string comparisons even behind a guard. String
operands now compare the serialized json_extract text directly, which is
exact-typed and never errors; a Rust test pins the semantics against a
mixed-type dataset.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3vectors: compare boolean filter operands as serialized text

Arrow's boolean cast accepts '1' and '0', so a TRY_CAST column let a
stored number satisfy a boolean filter. Booleans now compare the
serialized json_extract text like strings do — a JSON boolean only ever
serializes as 'true' or 'false'.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: claim a catalog name's markers in one compare-checked write

A create wrote its markers attribute-by-attribute after checking the
name was free. A delete's restore landing in the check-to-write gap put
the old markers back first, and the create then overwrote only the ones
it sets — leaving the old tags, policy, and maintenance settings on the
new table. CreateTable, RegisterTable, and CreateView now write all
their markers in a single update that refuses to run while any catalog
marker is present, so the second writer loses with a conflict instead
of inheriting them.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: test the attribute-preconditioned table delete and name claim

Adds a BeforeDelete hook plus ExpectedExtended evaluation to the
in-memory filer so tests can land a competing write in a handler's
check-to-delete window, and covers two losing races: a token delete
whose record changed since the read, and a create colliding with a
delete's marker restore.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: reserve a catalog record under one compare-checked claim

claimCatalogRecord writes a deleting marker onto the record while it
still holds exactly the caller's captured attributes, so a delete or
rename can hold the record through a multi-step operation and a losing
writer backs out before any data or destination entry is written. The
claim key joins the markers a name-claim asserts absent, so the name
cannot be reused while the reservation stands.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: claim the record before purging or publishing past it

A token delete of a decoupled table purged the data first and checked
the marker afterwards, so a rename that finished in between left the
moved record pointing at purged data while the delete reported a
conflict. The delete now claims the record under the captured
attributes before any data is removed and releases the claim if the
purge fails, keeping the record intact for a retry.

The rename claims the source before publishing the destination, so a
losing rename backs out while no destination entry exists instead of
leaving one behind pointing at data a delete went on to remove.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: test the record claim across delete and rename races

A rename completing in the token delete's check-to-claim window turns
the delete into a conflict with the data intact, and a delete's claim
on the source turns the rename into a conflict before the destination
entry is published.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-26 20:49:44 -07:00
Chris LuandDevin 8379ee9251 vector: snapshot lease + maintenance safety tests (P2) (#12107) 
* vector: version-lease helpers + maintenance safety tests

Pinned queries must survive Lance maintenance. maintenance::pin tags the
current dataset version so cleanup refuses to reclaim it; release drops
the tag. Tests prove the P2 gate: a pinned query returns identical hits
across compact_files, a tagged version survives cleanup_old_versions
(error_if_tagged protects it, the non-strict run skips it), and a
missing data file fails the query instead of returning partial hits.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: lease the query's dataset version in the VectorService

query_vectors pins the resolved version with a query tag for the
duration of the RPC and releases it after, so concurrent maintenance
cleanup cannot reclaim the files out from under an in-flight read.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: release the version lease on every query exit path

The lease tag was only released on query_vectors' success path — get_vectors
never released and query_vectors leaked its tag on early returns. pin now
returns a guard whose drop deletes the tag, so every path frees the version.

* vector: unique lease tag names + reap abandoned leases

Nanos-since-epoch tag names can collide across concurrent readers and
processes, so each lease now carries a unique suffix. pin() also drops
query tags older than an hour, since a killed or cancelled request
leaves its lease behind and would otherwise block cleanup forever.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: keep live lease tags + throttle abandoned-lease scans

An expired tag is only reaped once no live lease in this process owns
it, so a query running past LEASE_TTL keeps its pin. Tag scans run at
most once per 60s per dataset instead of on every request.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: answer read-only queries without a version lease

A namespace may hand the server credentials that can read a table but
not update it, so the tag write in pin() fails and the whole query
errors out. try_pin() degrades a denied tag write to an unleased query
instead of failing the request.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: fail a lease whose pinned version is already gone

Cleanup lists tags once at the start, so a tag created mid-cleanup does
not protect the version it names. After the tag lands, pin now verifies
the version still exists and drops the lease when it does not, failing
the read at pin time instead of mid-query.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-26 20:11:58 -07:00
Chris LuandDevin 87f15e4d6c vector: weed-vector-server, the VectorService gRPC (P1) (#12102) 
* vector: VectorService protobuf contract

The shared RPC boundary between the Go S3 gateway and the Rust vector
engine from the vector design doc: CreateIndex/DropIndex, Put/Get/
List/Delete/Query vectors. A Table reference carries the Lance table id
and an optional pinned version, matching the snapshot discipline queries
are required to keep.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: generated Go stubs for vector.proto

protoc output via weed/pb/Makefile.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: carry return_data through QueryVectors

S3 Vectors QueryVectors honours returnData, but the contract had no
way to ask for or carry vector data in a query response, so the
gateway would need a second GetVectors round-trip per query. Add
return_data to the request and data to QueryResult now, while the
contract is still new.

* vector: pin metadata numbers to double precision in the contract

Metadata arrives as JSON, where numbers carry double precision; say
so on the field so neither side promises integer fidelity past 2^53.

* vector: declare_table on the lance namespace client

CreateIndex needs a way to ask the catalog for a new table's location;
expose POST /v1/table/{id}/declare.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: cargo fmt coverage.rs

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: weed-vector-server, the VectorService gRPC

A tonic server over weed-vector-core: CreateIndex declares the table
through the namespace and writes an empty {key, vector, metadata}
dataset; PutVectors upserts by key (delete + append until merge_insert
is exercised); Get/List/Delete/Query route through the engine's
resolved table, honoring pinned versions and the prefilter default.
Metadata travels as a Utf8 JSON column, proto Struct on the wire.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: stamp the metric in the schema, tighten put validation

CreateIndex now bounds dimension at the S3 Vectors 4096 maximum and
records the distance metric in the vector field's Arrow metadata, so a
later reader recovers the index's configured metric from the table
itself. batch_of checks every vector's data length individually before
flattening — equal totals could still let a short vector borrow floats
from its neighbor — and writes against the dataset's own schema.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: upsert vectors in a single merge commit

PutVectors ran delete-then-append: a failed append left the old keys
deleted, and two concurrent puts could both append a replacement. Use
lance's merge_insert on the key column — one commit, no delete window,
conflicts fail the request instead of duplicating keys.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: read the index's metric; fetch metadata in one take

QueryVectors hardcoded L2 — it now reads the metric the index was
created with from the vector field's metadata. return_metadata fetched
each hit's row one at a time; collect the row ids into a single
take_rows call and join metadata back on the hit key.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: honor return_data in query results

QueryResult grew a data field but query_vectors never populated it.
Fetch the vector column in the same batched take_rows as metadata and
return each hit's stored embedding.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: reject a malformed query as invalid_argument

A zero top_k silently became one result and a dim-mismatched query
vector surfaced as an internal scan error; reject both before querying.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: reject a malformed list next_token

A token that failed to parse silently restarted the listing from
offset zero; surface it as invalid_argument instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: answer an empty put without a commit and cover ragged input

An empty vectors list would still run a merge commit; return the
current version instead. Pin the ragged-vector check with a test.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: keep non_filterable_metadata_keys on the table schema

create_index dropped the declared filter policy. Persist it as JSON on
the metadata field so the index carries which keys a filter may touch;
enforcing the list belongs to the gateway that translates the request
filter into Lance SQL.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: reject duplicate keys in one put

A repeated key for a new row committed both copies, and for an existing
key the merge failure surfaced as internal. Reject duplicates up front
as invalid_argument.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: report open failures as not_found or unavailable

A catalog 404 or a missing dataset/version is not a client mistake, and a
catalog outage is retryable: mapping every open error to invalid_argument
made both indistinguishable from a bad request.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: page list_vectors when max_results is unset

A proto3 client that omits max_results sends 0, which used to limit the
scan to zero rows and return no token.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: drop stray empty service.rs.tmp

* vector: rename datas map to data_map

codespell flags datas as a misspelling of data; use a distinct map name.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-26 13:32:34 -07:00
Chris Lu aff18b2336 vector: restore spike CLI flags lost in the #12098 rebase (#12099) 
* vector: --vector takes hyphenated floats again

* vector: --vector-column/--key-column on the get subcommand

* vector: --vector-column/--key-column on describe and coverage

* vector: surface describe resolve errors when column flags are set

* vector: cover the restored CLI flags with parse tests
 2026-10-26 12:24:01 -07:00
Chris LuandDevin 8498ad716b vector: spike CLI, reference-comparison suite, CI (P0 phase 3) (#12098) 
* vector: project _distance and _rowid explicitly

Lance warns that a knn projection without _distance will stop
auto-including it; spell both scoring columns out.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: reference-comparison tests

The spike's exit gate: assert the engine's answers equal a bare
scan().nearest() on the same pinned version — over a fresh index, over
unindexed fragments, over a two-segment index after optimize_indices,
and after a delete. A version-pinned handle querying through
maintenance sees only its own version.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: weed-vector spike CLI

describe / coverage / query / get against a namespace-registered table
or a plain dataset URI, so the spike runs against a real SeaweedFS Lance
table bucket and against a local fixture alike. Flag and env names
mirror weed-worker.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: rust-vector-tests workflow

Build and test seaweed-vector in release on pushes and PRs touching it
or its path-dependency trees, mirroring rust-worker-tests.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: derive Serialize for the spike CLI's JSON output

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: --postfilter on the spike CLI

Track the Query field rename; the flag stays an explicit opt-out of
prefiltering for comparison runs.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: assert the pinned query's answers

The maintenance test passed even with zero pinned hits: an empty result
satisfies all() trivially. Assert the hit count and compare the pinned
answers against the reference on the pinned handle.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: vector tests also watch plugin.proto

weed-vector-core path-depends on weed-lance-worker, which compiles
weed/pb/plugin.proto through seaweed-worker-core; a proto-only change
would skip these checks and could break this workspace.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: describe prints layout and coverage

The describe doc comment promised the vector layout and index coverage;
print both, and propagate count_rows errors instead of printing 0.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-26 11:48:46 -07:00
Chris LuDevinchrislusf
 		eda4b9efdd vector: ANN coverage introspection + prefilter (P0 phase 2) (#12097) 
* vector: share the test dataset builder

The coverage tests need the same tempdir Lance datasets the query tests
build; move them into a cfg(test) helper module.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: prefilter the metadata filter

prefilter=false post-filters the top-K rows, which can return fewer than
k results and never sees unindexed rows beyond the nearest cut. Expose
the choice so the S3 Vectors layer can honour AWS filter semantics.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: index coverage introspection

coverage() unions each index's fragment bitmaps against the manifest's
fragment list to report which fragments a vector index covers, which it
has not reached, and how many rows the flat pass still has to scan.
Tests pin the behaviour the spike is for: rows appended after an index
build stay in the top-K, and a version-pinned dataset never sees the
next commit.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: default filtered queries to prefilter

postfilter=false is now the normal path: the metadata filter cuts the
eligible rows before ranking, so a query returns k hits whenever k
eligible rows exist. Lance's post-filter remains reachable through
Query::postfilter for comparison runs.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: intersect index bitmaps with the live manifest

A bitmap records fragments the index covered when built; a fragment the
manifest dropped afterwards stayed in the reported coverage.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: keep bitmap-less indices in coverage

An index whose metadata has no fragment bitmap was dropped from the
report entirely, indistinguishable from no index at all. Count its
segments and mark its coverage unknown instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: count live rows in unindexed coverage

physical_rows counts deleted rows and can be absent in the manifest;
sum FileFragment::count_rows for the unindexed fragments instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: error on a null key in a selected key column

* vector: compare filtered ANN and flat hits as sets

* vector: coverage reports the checked-out version

* vector: project _distance and _rowid explicitly

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris@seaweedfs.com>
 2026-10-26 11:45:07 -07:00
Chris LuandDevin 7388d0d4fc vector: seaweed-vector workspace + flat/indexed top-K engine (P0 phase 1) (#12096) 
* vector: scaffold the seaweed-vector workspace

Standalone cargo workspace for the vector query engine, mirroring the
seaweed-worker layout: own lockfile, same lint policy, path dependencies
across the two trees since there is no common root manifest.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: resolve a table's vector layout

VectorTable::resolve finds the vector column (explicit name, else the
column named vector, else the sole fixed-size float list), the optional
key column, and the metadata columns a query can project.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: flat and indexed top-k, key lookup

query::flat runs an exact scan (use_index(false)); query::ann lets Lance
split the query between index segments and a flat pass over fragments no
index covers yet, so rows written after the last index build stay
eligible. Both honour a metadata filter and return the raw _distance in
the query's metric.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: reject a non-utf8 or non-identifier key column

An explicit --key-column naming a non-UTF8 field made hits() silently
report row ids as keys, and get() interpolated the name into a SQL
filter verbatim. Require Utf8 at resolve and in hits(), and require a
plain identifier before building the IN predicate.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: fall back to row id on a null key

keys.value() on a null slot fabricates an empty-string key; a row with
no key reports its row id instead, matching the no-key-column case.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: quote the key column in the IN filter

A Utf8 key column named e.g. document-key made the predicate parse as
subtraction, and an identifier interpolated unquoted is injectable.
Backtick-quote it and double its inner backticks; double quotes are
string literals in the filter dialect, not identifiers.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: project only the columns a hit needs

The knn projection pulled every metadata column — including other
vector columns — although Hit returns none of them. Filters still
evaluate off the scan path, so restrict the output to the key and the
scoring columns.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-10-26 10:13:05 -07:00