vector: indexed metadata keys, engine side (derive, rewrite, exact planner, filter counter) (#12813)

* vector: derive the indexed metadata key columns

An indexed metadata key is a hidden `_mk_<n>` column of List<Utf8>: the
strings an S3 `$eq` on a string matches, that is the value when it is a
string or the string elements of an array, and nothing for any other type
or a missing key. The field metadata names the key and its state
(`building` or `ready`).

The module derives the column from metadata given as JSON (the write
path) or as stored (jsonb or JSON text, for a backfill), and reads the
indexed keys from a schema. It lives in the worker crate so the engine and
the worker share it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: fill the indexed metadata key columns on PutVectors

PutVectors builds its batch against the table's current schema. Once a
table has indexed metadata key columns, the three-column batch no longer
matches that schema and the write fails. Fill each `_mk_*` column the
schema has from the row's metadata, whatever the key's state; a column
the engine does not know still fails the request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: answer string $eq and $in on a ready indexed key from its column

The gateway renders a string `$eq` or `$in` on a key as an OR of
`json_extract(metadata, P) = '"v"'` (or IN) and
`json_array_contains(metadata, P, '"v"')` over the same values. On the
parsed filter, a top-level conjunct of that shape on a key whose column is
`ready` becomes `array_has` / `array_has_any` on the column, which its
LABEL_LIST index answers. The two halves must name the same key and the
same strings, the scalar half in the text json_extract returns for them;
anything else, including `$ne`, `$nin`, `$exists`, numbers and booleans,
stays on the JSON path, and the other conjuncts still apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: score few indexed matches exactly, per scope

With a ready indexed key in the filter, `ann` first counts the rows of its
scope that the key's conjuncts match, through the scalar index. When they
are at most the limit it scores them exactly over that scope, the way
the short-result fallback does; above it, the vector index searches with
the indexed prefilter, and the fallback still applies. The scope is the
one `exact_in_scope` scores, so per-scope exact answers merge into the
global exact top-k.

The limit is `Query::exact_match_limit` (None takes 256; 0 never counts),
set by the engine's --exact-match-limit (WEED_VECTOR_EXACT_MATCH_LIMIT).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: count filtered queries by metadata key

For each index, the engine counts its QueryVectors calls by the metadata
keys of the parsed filter, with their total latency and the operand types
each key was compared with, over the last hour in 5-minute buckets. At most
512 indexes and 32 keys per index are kept (an idle key makes room;
otherwise the index is marked truncated), and the stats are not exported
as metrics labels.

The new FilterStats RPC returns an index's stats; the gateway's discovery
client forwards it like the other calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: report the metadata columns capability in gRPC health

An engine whose writes fill the indexed metadata key columns reports the
gRPC health service `weed.vector.capability.metadata_columns` as serving,
so an operator action can refuse to index a key while an older engine
could still write the table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: expire filter stats' operand kinds and truncation with the window

The operand kinds a key was compared with, and the mark that a key went
uncounted, outlived the queries that set them. Keep the kinds per
5-minute bucket and the truncation as the bucket it last happened in, so
both leave the hour window with the counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: fill indexed key columns a write left empty racing their add

A PutVectors that opened the table before an indexed key's column was
added, and commits after the add, is rebased by Lance onto the wider
schema: it commits, and the rows it wrote hold NULL in the new column, so
the column's index misses them (reproduced in the test: 100 of 150 rows).

After its commit, PutVectors now looks for indexed-key columns the batch
did not carry, reads those of its keys whose column is still NULL, derives
the values from their stored metadata and patches just those columns with
a partial-schema merge that updates only rows still NULL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: keep filter stats' truncation mark at the later bucket

record() reads the clock before it takes the lock, so a call that waited
on the lock across a bucket boundary could move the truncation mark (and
the index's last-used bucket) back to its earlier bucket, and the mark
then left the window one bucket early.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: derive a raced write's indexed key columns from current metadata

A PutVectors that opened the table before an indexed key's column was
added is retried by Lance against the wider schema: rows it inserts hold
NULL there, and rows it updates keep the value derived from their old
metadata, which the NULL-only heal never looked at. The heal now derives
the columns for all of the write's keys.

Its merge also ran with Lance's retry, which joins the values derived
before the conflict onto the rows a later write of the same keys left.
It now fails on a conflict and is redone from the latest version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: check the raced write's keys keep the later write's rows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chris LuandClaude Opus 5.5 authored and GitHub committed 2026-11-19 23:45:30 -08:00
1 parent 817338c911
commit e109f91c6f
27 files changed
+2461 -77

No files matched your search

+3
View File
@@ -7143,6 +7143,7 @@ dependencies = [
"aes",
"aes-gcm",
"anyhow",
"arrow-array",
"arrow-cast",
"arrow-schema",
"async-trait",
@@ -7158,6 +7159,7 @@ dependencies = [
"hmac 0.12.1",
"http",
"hyper-util",
"jsonb",
"jsonwebtoken",
"lance",
"lance-datafusion",
@@ -7236,6 +7238,7 @@ dependencies = [
"futures",
"lance",
"lance-arrow",
"lance-index",
"lance-linalg",
"moka",
"object_store",
@@ -139,6 +139,7 @@ mod tests {
nprobes: Some(4),
index_segments: None,
fragments: None,
exact_match_limit: None,
},
)
.await
@@ -167,6 +168,7 @@ mod tests {
nprobes: Some(4),
index_segments: None,
fragments: None,
exact_match_limit: None,
};
let filtered = ann(&dataset, &table, &query).await.unwrap();
let exact = flat(&dataset, &table, &query).await.unwrap();
@@ -204,6 +206,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
},
)
.await
+17 -7
View File
@@ -4,6 +4,7 @@
//! compares the elements as typed JSON values: Lance's compares their text, so
//! 1 would match "1".
use std::collections::HashMap;
use std::sync::Arc;
use arrow_array::builder::{BooleanBuilder, StringBuilder};
@@ -19,13 +20,21 @@ use lance::deps::datafusion::logical_expr::{
ColumnarValue, Expr, ScalarFunctionArgs, ScalarUDF, ScalarUDFImpl, Signature, Volatility,
};
/// Set `filter` on the scanner with the JSONPath functions swapped.
pub(crate) fn apply(scanner: &mut Scanner, filter: &str) -> anyhow::Result<()> {
/// Set `filter` on the scanner with the JSONPath functions swapped and the
/// conjuncts on `ready` indexed keys answered by their columns. Returns the
/// conjunction of those.
pub(crate) fn apply(
scanner: &mut Scanner,
filter: &str,
ready: &HashMap<String, String>,
) -> anyhow::Result<Option<Expr>> {
scanner.filter(filter)?;
if let Some(expr) = scanner.get_expr_filter()? {
scanner.filter_expr(swap(expr)?);
}
Ok(())
let Some(expr) = scanner.get_expr_filter()? else {
return Ok(None);
};
let (expr, indexed) = crate::indexed_filter::rewrite(expr, ready);
scanner.filter_expr(swap(expr)?);
Ok(indexed)
}
fn swap(expr: Expr) -> Result<Expr> {
@@ -268,7 +277,7 @@ mod tests {
async fn keys(dataset: &Dataset, filter: &str, swapped: bool) -> anyhow::Result<Vec<String>> {
let mut scanner = dataset.scan();
if swapped {
apply(&mut scanner, filter)?;
apply(&mut scanner, filter, &HashMap::new())?;
} else {
scanner.filter(filter)?;
}
@@ -395,6 +404,7 @@ mod tests {
apply(
&mut scanner,
r#"json_extract(`metadata`, '$["k"]') = '"v"' AND json_exists(`metadata`, '$["k"]') AND json_get_string(`metadata`, 'k') = 'v' AND json_array_contains(`metadata`, '$["k"]', '"v"') AND NOT json_array_contains(`metadata`, '$["k"]', key)"#,
&HashMap::new(),
)
.unwrap();
let mut swapped = Vec::new();
+1
View File
@@ -104,6 +104,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
}
}
@@ -0,0 +1,599 @@
//! Filters on indexed metadata keys (see `weed_lance_worker::metadata_index`).
//!
//! The gateway renders a string `$eq` or `$in` on a key as an OR of
//! `json_extract(metadata, P) = '"v"'` (or `IN`) and
//! `json_array_contains(metadata, P, '"v"')` over the same values. In the
//! parsed filter, such a top-level conjunct on a ready key is replaced by
//! `array_has` / `array_has_any` on the key's column, which its LABEL_LIST
//! index answers. Everything else, including `$ne` and `$nin`, stays on the
//! JSON path.
use std::collections::{BTreeMap, BTreeSet, HashMap};
use anyhow::Result;
use jsonb::OwnedJsonb;
use lance::Dataset;
use lance::deps::datafusion::common::{Column, ScalarValue};
use lance::deps::datafusion::functions_nested::expr_fn::{array_has, array_has_any};
use lance::deps::datafusion::logical_expr::utils::split_conjunction_owned;
use lance::deps::datafusion::logical_expr::{BinaryExpr, Expr, Operator};
use lance::deps::datafusion::prelude::lit;
use weed_lance_worker::metadata_index::{self, State};
const METADATA_COLUMN: &str = "metadata";
/// The column of each ready indexed key of `dataset`, by metadata key.
pub(crate) fn ready_columns(dataset: &Dataset) -> HashMap<String, String> {
let schema: arrow_schema::Schema = dataset.schema().into();
metadata_index::indexed_keys(&schema)
.into_iter()
.filter(|k| k.state == Some(State::Ready))
.map(|k| (k.key, k.column))
.collect()
}
/// `filter` with each top-level conjunct that is a string `$eq` or `$in` on
/// a key in `ready` answered by the key's column instead, and the
/// conjunction of those replacements.
pub(crate) fn rewrite(filter: Expr, ready: &HashMap<String, String>) -> (Expr, Option<Expr>) {
if ready.is_empty() {
return (filter, None);
}
let mut indexed: Option<Expr> = None;
let mut rewritten: Option<Expr> = None;
for conjunct in split_conjunction_owned(filter) {
let replacement = string_membership(&conjunct).and_then(|(key, values)| {
let column = Expr::Column(Column::from_name(ready.get(&key)?));
Some(match values.as_slice() {
[one] => array_has(column, lit(one.as_str())),
_ => {
let values: Vec<ScalarValue> = values
.iter()
.map(|v| ScalarValue::Utf8(Some(v.clone())))
.collect();
let list = ScalarValue::new_list(&values, &arrow_schema::DataType::Utf8, true);
array_has_any(column, Expr::Literal(ScalarValue::List(list), None))
}
})
});
let part = match replacement {
Some(expr) => {
indexed = Some(and(indexed, expr.clone()));
expr
}
None => conjunct,
};
rewritten = Some(and(rewritten, part));
}
(rewritten.unwrap_or(lit(true)), indexed)
}
fn and(left: Option<Expr>, right: Expr) -> Expr {
match left {
Some(left) => Expr::BinaryExpr(BinaryExpr::new(
Box::new(left),
Operator::And,
Box::new(right),
)),
None => right,
}
}
/// The key and string values of a conjunct meaning "the key holds one of
/// these strings, or an array with one of them": the OR of scalar matches
/// and array-member matches over the same values and key.
fn string_membership(expr: &Expr) -> Option<(String, Vec<String>)> {
let mut disjuncts = Vec::new();
split_or(expr, &mut disjuncts);
let mut path: Option<&str> = None;
let mut scalars = BTreeSet::new();
let mut elements = BTreeSet::new();
for disjunct in disjuncts {
let (p, kind) = match unwrap_false_coalesce(disjunct) {
Expr::BinaryExpr(BinaryExpr {
left,
op: Operator::Eq,
right,
}) => (
json_function(left, "json_extract")?,
Member::Scalar(vec![utf8(right)?]),
),
Expr::InList(list) if !list.negated => (
json_function(&list.expr, "json_extract")?,
Member::Scalar(list.list.iter().map(utf8).collect::<Option<_>>()?),
),
Expr::ScalarFunction(f) if f.func.name() == "json_array_contains" => {
let [data, p, value] = f.args.as_slice() else {
return None;
};
if !is_metadata(data) {
return None;
}
let element: serde_json::Value = serde_json::from_str(utf8(value)?).ok()?;
(utf8(p)?, Member::Element(element.as_str()?.to_string()))
}
_ => return None,
};
if path.is_some_and(|q| q != p) {
return None;
}
path = Some(p);
match kind {
Member::Scalar(values) => scalars.extend(values.into_iter().map(str::to_string)),
Member::Element(value) => {
elements.insert(value);
}
}
}
let key = json_path_key(path?)?;
let rendered: BTreeSet<String> = elements
.iter()
.map(|s| jsonb_text(s))
.collect::<Option<_>>()?;
(!elements.is_empty() && rendered == scalars).then(|| (key, elements.into_iter().collect()))
}
enum Member<'a> {
Scalar(Vec<&'a str>),
Element(String),
}
fn split_or<'a>(expr: &'a Expr, out: &mut Vec<&'a Expr>) {
match expr {
Expr::BinaryExpr(BinaryExpr {
left,
op: Operator::Or,
right,
}) => {
split_or(left, out);
split_or(right, out);
}
Expr::Alias(alias) => split_or(&alias.expr, out),
other => out.push(other),
}
}
/// `x` for `COALESCE(x, false)` or its simplified `x IS NOT NULL AND x`;
/// either is false where `x` is null, which a top-level conjunct treats alike.
fn unwrap_false_coalesce(expr: &Expr) -> &Expr {
match expr {
Expr::ScalarFunction(f) if f.func.name() == "coalesce" => match f.args.as_slice() {
[x, Expr::Literal(ScalarValue::Boolean(Some(false)), _)] => x,
_ => expr,
},
Expr::BinaryExpr(BinaryExpr {
left,
op: Operator::And,
right,
}) => match left.as_ref() {
Expr::IsNotNull(x) if x == right => right,
_ => expr,
},
_ => expr,
}
}
fn utf8(expr: &Expr) -> Option<&str> {
match expr {
Expr::Literal(ScalarValue::Utf8(Some(s)) | ScalarValue::LargeUtf8(Some(s)), _) => {
Some(s.as_str())
}
_ => None,
}
}
fn is_metadata(expr: &Expr) -> bool {
matches!(expr, Expr::Column(c) if c.name == METADATA_COLUMN)
}
/// The JSONPath literal of `name(metadata, path)`.
fn json_function<'a>(expr: &'a Expr, name: &str) -> Option<&'a str> {
match expr {
Expr::ScalarFunction(f) if f.func.name() == name => match f.args.as_slice() {
[data, path] if is_metadata(data) => utf8(path),
_ => None,
},
_ => None,
}
}
/// The key of a `$["key"]` path, with `\\` and `\"` unescaped.
fn json_path_key(path: &str) -> Option<String> {
let inner = path.strip_prefix("$[\"")?.strip_suffix("\"]")?;
let mut key = String::with_capacity(inner.len());
let mut chars = inner.chars();
while let Some(c) = chars.next() {
match c {
'\\' => match chars.next()? {
c @ ('\\' | '"') => key.push(c),
_ => return None,
},
'"' => return None,
c => key.push(c),
}
}
Some(key)
}
/// What json_extract returns for a stored string `s`.
fn jsonb_text(s: &str) -> Option<String> {
let value: OwnedJsonb = serde_json::to_string(s).ok()?.parse().ok()?;
Some(value.to_string())
}
/// How a filter compares a key.
pub mod operand {
pub const STRING: u8 = 1;
pub const NUMBER: u8 = 2;
pub const BOOL: u8 = 4;
pub const EXISTS: u8 = 8;
}
/// The metadata keys `filter` reads, each with the [`operand`] kinds it is
/// compared with.
pub fn filter_keys(dataset: &Dataset, filter: &str) -> Result<BTreeMap<String, u8>> {
let mut scanner = dataset.scan();
scanner.filter(filter)?;
let mut keys = BTreeMap::new();
if let Some(expr) = scanner.get_expr_filter()? {
collect_keys(&expr, &mut keys);
}
Ok(keys)
}
fn literal_kind(text: &str) -> u8 {
match serde_json::from_str::<serde_json::Value>(text) {
Ok(serde_json::Value::String(_)) => operand::STRING,
Ok(serde_json::Value::Number(_)) => operand::NUMBER,
Ok(serde_json::Value::Bool(_)) => operand::BOOL,
// json_extract's text of a string with control characters is not
// JSON, but it is quoted.
_ if text.starts_with('"') => operand::STRING,
_ => 0,
}
}
fn collect_keys(expr: &Expr, keys: &mut BTreeMap<String, u8>) {
let mut note = |path: &str, kind: u8| {
if let Some(key) = json_path_key(path) {
*keys.entry(key).or_default() |= kind;
}
};
match expr {
Expr::TryCast(cast) if extract(&cast.expr).is_some() => {
note(extract(&cast.expr).unwrap_or_default(), operand::NUMBER);
return;
}
Expr::BinaryExpr(BinaryExpr { left, right, .. }) => {
if let (Some(path), Some(value)) = (extract(left), utf8(right)) {
note(path, literal_kind(value));
return;
}
}
Expr::InList(list) => {
if let Some(path) = extract(&list.expr) {
let kind = list
.list
.iter()
.filter_map(utf8)
.fold(0, |k, v| k | literal_kind(v));
note(path, kind);
return;
}
}
Expr::ScalarFunction(f) if is_json_function(f.func.name()) => {
if let [data, path, rest @ ..] = f.args.as_slice()
&& is_metadata(data)
&& let Some(path) = utf8(path)
{
let kind = match (f.func.name(), rest) {
("json_exists", _) => operand::EXISTS,
(_, [value]) => utf8(value).map_or(0, literal_kind),
_ => 0,
};
note(path, kind);
return;
}
}
_ => {}
}
for child in children(expr) {
collect_keys(child, keys);
}
}
fn extract(expr: &Expr) -> Option<&str> {
json_function(expr, "json_extract")
}
fn is_json_function(name: &str) -> bool {
matches!(name, "json_extract" | "json_exists" | "json_array_contains")
}
fn children(expr: &Expr) -> Vec<&Expr> {
use lance::deps::datafusion::common::tree_node::TreeNode;
let mut out = Vec::new();
let _ = expr.apply_children(|child| {
out.push(child);
Ok(lance::deps::datafusion::common::tree_node::TreeNodeRecursion::Continue)
});
out
}
#[cfg(test)]
mod tests {
use super::*;
use crate::query::{self, Query};
use crate::table::VectorTable;
use arrow_array::{FixedSizeListArray, RecordBatch, RecordBatchIterator, StringArray};
use arrow_schema::{DataType, Field, Schema};
use lance::index::DatasetIndexExt;
use lance_index::IndexType;
use lance_index::scalar::{BuiltinIndexType, ScalarIndexParams};
use lance_linalg::distance::DistanceType;
use std::sync::Arc;
use tempfile::TempDir;
const DIM: usize = 4;
/// The gateway's text for `{"author": "<v>"}`.
fn eq(field: &str, json: &str) -> String {
let json = json.replace('\'', "''");
format!(
"(COALESCE(json_extract(`metadata`, '$[\"{field}\"]') = '{json}', false) OR COALESCE(json_array_contains(`metadata`, '$[\"{field}\"]', '{json}'), false))"
)
}
/// The gateway's text for `{"author": {"$in": [<strings>]}}`.
fn among(values: &[&str]) -> String {
let scalars: Vec<String> = values.iter().map(|v| format!("'\"{v}\"'")).collect();
let mut groups = vec![format!(
"COALESCE(json_extract(`metadata`, '$[\"author\"]') in ({}), false)",
scalars.join(", ")
)];
for v in values {
groups.push(format!(
"COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"{v}\"'), false)"
));
}
format!("({})", groups.join(" OR "))
}
/// Rows `u0..u<n>` by author, plus a few others, in fragments of 100,
/// with the author column ready and indexed and a vector index.
async fn dataset(dir: &TempDir, rows: usize) -> Dataset {
let metadata = Field::new("metadata", DataType::Utf8, true)
.with_metadata([("ARROW:extension:name".to_string(), "arrow.json".to_string())].into());
let schema = Arc::new(Schema::new(vec![
Field::new("key", DataType::Utf8, false),
Field::new(
"vector",
DataType::FixedSizeList(
Arc::new(Field::new("item", DataType::Float32, true)),
DIM as i32,
),
false,
),
metadata,
metadata_index::field("_mk_0", "author", State::Ready),
]));
let metas: Vec<serde_json::Value> = (0..rows)
.map(|i| match i % 10 {
0 => serde_json::json!({"author": "rare"}),
1 => serde_json::json!({"author": ["many", "rare2"]}),
_ => serde_json::json!({"author": "many", "n": i}),
})
.collect();
let batches: Vec<RecordBatch> = (0..rows)
.step_by(100)
.map(|start| {
let range = start..(start + 100).min(rows);
RecordBatch::try_new(
schema.clone(),
vec![
Arc::new(StringArray::from(
range.clone().map(|i| format!("k{i}")).collect::<Vec<_>>(),
)),
Arc::new(FixedSizeListArray::from_iter_primitive::<
arrow_array::types::Float32Type,
_,
_,
>(
range.clone().map(|i| {
Some((0..DIM).map(move |j| Some(((i * 31 + j * 17) % 97) as f32)))
}),
DIM as i32,
)),
Arc::new(StringArray::from(
range
.clone()
.map(|i| Some(metas[i].to_string()))
.collect::<Vec<_>>(),
)),
metadata_index::column_from_json(range.map(|i| Some(&metas[i])), "author"),
],
)
.unwrap()
})
.collect();
let uri = dir.path().to_str().unwrap();
let mut dataset = None::<Dataset>;
for batch in batches {
let reader = RecordBatchIterator::new(vec![Ok(batch)], schema.clone());
dataset = Some(match dataset {
None => Dataset::write(reader, uri, None).await.unwrap(),
Some(mut d) => {
d.append(reader, None).await.unwrap();
d
}
});
}
let mut dataset = dataset.unwrap();
dataset
.create_index(
&["_mk_0"],
IndexType::LabelList,
None,
&ScalarIndexParams::for_builtin(BuiltinIndexType::LabelList),
true,
)
.await
.unwrap();
crate::index::build(&mut dataset, "vector", crate::index::INDEX_NAME, 4, false)
.await
.unwrap();
dataset
}
fn rewritten(dataset: &Dataset, filter: &str) -> (String, Option<String>) {
let mut scanner = dataset.scan();
let indexed = crate::filter::apply(&mut scanner, filter, &ready_columns(dataset)).unwrap();
(
scanner.get_expr_filter().unwrap().unwrap().to_string(),
indexed.map(|e| e.to_string()),
)
}
#[tokio::test]
async fn string_eq_and_in_on_a_ready_key_use_its_column() {
let dir = TempDir::new().unwrap();
let dataset = dataset(&dir, 200).await;
assert_eq!(
ready_columns(&dataset),
HashMap::from([("author".to_string(), "_mk_0".to_string())])
);
for (filter, want) in [
(eq("author", "\"rare\""), Some("array_has(_mk_0, Utf8(\"rare\"))")),
(
among(&["rare", "many"]),
Some("array_has_any(_mk_0, List([many, rare]))"),
),
(
format!("{} AND TRY_CAST(json_extract(`metadata`, '$[\"n\"]') AS DOUBLE) > 3", eq("author", "\"rare\"")),
Some("array_has(_mk_0, Utf8(\"rare\"))"),
),
(eq("author", "\"o'neil \\\"q\\\"\""), Some("array_has(_mk_0, Utf8(\"o'neil \"q\"\"))")),
(
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"c\\u0001d\\n\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"c\\u0001d\\n\"'), false))".to_string(),
Some("array_has(_mk_0, Utf8(\"c\u{1}d\n\"))"),
),
// The scalar half's text is not what json_extract returns for
// the array half's string, so the two do not agree.
(
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"c\\u{1}d\\n\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"c\\u0001d\\n\"'), false))".to_string(),
None,
),
// Not a string, not this key, or not every value on both sides.
(eq("author", "true"), None),
(eq("other", "\"rare\""), None),
(format!("NOT {}", eq("author", "\"rare\"")), None),
(format!("NOT {}", among(&["rare", "many"])), None),
(
"COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"rare\"', false)".to_string(),
None,
),
(
among(&["rare", "many"]).replace("'\"many\"'), false))", "'\"other\"'), false))"),
None,
),
("json_exists(`metadata`, '$[\"author\"]')".to_string(), None),
(format!("({} OR json_exists(`metadata`, '$[\"n\"]'))", eq("author", "\"rare\"")), None),
] {
let (expr, indexed) = rewritten(&dataset, &filter);
assert_eq!(indexed.as_deref(), want, "{filter}");
assert_eq!(expr.contains("json_"), want.is_none() || filter.contains(" AND "), "{filter}: {expr}");
}
}
/// The planner scores few matches exactly and leaves many to the vector
/// index, per scope.
#[tokio::test]
async fn few_indexed_matches_are_scored_exactly() {
let dir = TempDir::new().unwrap();
let dataset = dataset(&dir, 1000).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
let query = |filter: &str, fragments: Option<Vec<u64>>| Query {
vector: vec![1.0, 2.0, 3.0, 4.0],
k: 10,
filter: Some(filter.to_string()),
postfilter: false,
distance: DistanceType::L2,
nprobes: None,
index_segments: None,
fragments,
exact_match_limit: None,
};
let rare = eq("author", "\"rare\"");
let many = eq("author", "\"many\"");
let few = |q: Query| {
let (dataset, table) = (&dataset, &table);
async move {
query::few_indexed_matches(dataset, table, &q)
.await
.unwrap()
}
};
// 100 and 900 matches, and 90 in the first two fragments.
assert!(few(query(&rare, None)).await);
assert!(!few(query(&many, None)).await);
let first = Some(vec![0, 1]);
assert!(few(query(&many, first.clone())).await);
assert!(
!few(Query {
exact_match_limit: Some(0),
..query(&rare, None)
})
.await
);
// A JSON-only filter is not counted.
assert!(!few(query("json_exists(`metadata`, '$[\"n\"]')", None)).await);
for (filter, fragments) in [(&rare, None), (&many, first)] {
let q = query(filter, fragments);
let exact = query::flat(&dataset, &table, &q).await.unwrap();
let got = query::ann(&dataset, &table, &q).await.unwrap();
assert_eq!(got.len(), 10);
for (g, e) in got.iter().zip(&exact) {
assert!(
(g.distance - e.distance).abs() < 1e-4,
"{got:?} vs {exact:?}"
);
}
}
}
#[tokio::test]
async fn the_plan_reads_the_scalar_index() {
let dir = TempDir::new().unwrap();
let dataset = dataset(&dir, 300).await;
let mut scanner = dataset.scan();
crate::filter::apply(
&mut scanner,
&eq("author", "\"rare\""),
&ready_columns(&dataset),
)
.unwrap();
let plan = scanner.explain_plan(true).await.unwrap();
assert!(plan.contains("LabelList"), "{plan}");
assert!(!plan.contains("json_"), "{plan}");
}
#[tokio::test]
async fn filter_keys_note_how_each_key_is_compared() {
let dir = TempDir::new().unwrap();
let dataset = dataset(&dir, 100).await;
let filter = format!(
"{} AND TRY_CAST(json_extract(`metadata`, '$[\"n\"]') AS DOUBLE) > 3 AND NOT json_exists(`metadata`, '$[\"a.b\\\"c\"]') AND COALESCE(json_extract(`metadata`, '$[\"f\"]') = 'true', false)",
eq("author", "\"rare\"")
);
let keys = filter_keys(&dataset, &filter).unwrap();
assert_eq!(
keys,
BTreeMap::from([
("a.b\"c".to_string(), operand::EXISTS),
("author".to_string(), operand::STRING),
("f".to_string(), operand::BOOL),
("n".to_string(), operand::NUMBER),
])
);
}
}
+1
View File
@@ -8,6 +8,7 @@
pub mod coverage;
mod filter;
pub mod index;
pub mod indexed_filter;
pub mod maintenance;
pub mod query;
pub mod segments;
+1
View File
@@ -309,6 +309,7 @@ async fn main() -> Result<()> {
nprobes,
index_segments: None,
fragments: None,
exact_match_limit: None,
};
let hits = if flat {
query::flat(&dataset, &vector_table, &query).await?
@@ -312,6 +312,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
};
let mut ids: Vec<u64> = query::flat(&pinned, &resolved, &q)
.await
@@ -734,6 +735,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
};
assert!(query::flat(&dataset, &resolved, &q).await.is_err());
}
+76 -7
View File
@@ -39,8 +39,15 @@ pub struct Query {
/// a worker the unindexed-fragment sweep. Listed ids that no longer exist
/// scan nothing.
pub fragments: Option<Vec<u64>>,
/// Score the matches exactly instead of searching the vector index when
/// the filter's indexed metadata keys match at most this many rows of the
/// scope; 0 never does. None takes [`DEFAULT_EXACT_MATCH_LIMIT`].
pub exact_match_limit: Option<usize>,
}
/// Up to this many matches of an indexed metadata key are scored exactly.
pub const DEFAULT_EXACT_MATCH_LIMIT: usize = 256;
/// One row of a top-K answer.
#[derive(Debug, Clone, serde::Serialize)]
pub struct Hit {
@@ -63,7 +70,14 @@ pub async fn flat(dataset: &Dataset, table: &VectorTable, query: &Query) -> Resu
/// A prefiltered search sees only the probed partitions: it can come back
/// short of k, or with unscored rows (infinite distance) when at most k rows
/// match. Either way it is answered again by an exact scan of the same scope.
///
/// When the filter's indexed metadata keys match few rows of that scope,
/// they are scored exactly to begin with. The decision is per scope, so
/// exact answers of several scopes merge into the exact answer.
pub async fn ann(dataset: &Dataset, table: &VectorTable, query: &Query) -> Result<Vec<Hit>> {
if few_indexed_matches(dataset, table, query).await? {
return exact_in_scope(dataset, table, query).await;
}
let (scanner, indexed) = scanner(dataset, table, query, true).await?;
let hits = run(scanner, table).await?;
if indexed
@@ -79,7 +93,17 @@ pub async fn ann(dataset: &Dataset, table: &VectorTable, query: &Query) -> Resul
/// [`flat`] over what [`ann`] searches for `query`: the listed fragments plus
/// the live coverage of the listed index segments, or the whole table.
async fn exact_in_scope(dataset: &Dataset, table: &VectorTable, query: &Query) -> Result<Vec<Hit>> {
let fragments = match &query.index_segments {
let exact = Query {
index_segments: None,
fragments: scope(dataset, table, query).await?,
..query.clone()
};
run(scanner(dataset, table, &exact, false).await?.0, table).await
}
/// The fragments [`ann`] searches for `query`, or None for the whole table.
async fn scope(dataset: &Dataset, table: &VectorTable, query: &Query) -> Result<Option<Vec<u64>>> {
Ok(match &query.index_segments {
Some(segments) if !segments.is_empty() => {
let mut ids = query.fragments.clone().unwrap_or_default();
ids.extend(
@@ -88,13 +112,51 @@ async fn exact_in_scope(dataset: &Dataset, table: &VectorTable, query: &Query) -
Some(ids)
}
_ => query.fragments.clone(),
})
}
/// Whether the rows of [`ann`]'s scope matching the filter's indexed
/// metadata keys, counted through their scalar indexes, are at most the
/// query's limit. Only a search through the vector index asks.
pub(crate) async fn few_indexed_matches(
dataset: &Dataset,
table: &VectorTable,
query: &Query,
) -> Result<bool> {
let limit = query.exact_match_limit.unwrap_or(DEFAULT_EXACT_MATCH_LIMIT);
let Some(filter) = &query.filter else {
return Ok(false);
};
let exact = Query {
index_segments: None,
fragments,
..query.clone()
if limit == 0 || query.postfilter || query.index_segments.as_ref().is_some_and(|s| s.is_empty())
{
return Ok(false);
}
let ready = crate::indexed_filter::ready_columns(dataset);
if ready.is_empty()
|| crate::index::index_names(dataset, &table.vector_column)
.await?
.is_empty()
{
return Ok(false);
}
let Some(indexed) = crate::filter::apply(&mut dataset.scan(), filter, &ready)? else {
return Ok(false);
};
run(scanner(dataset, table, &exact, false).await?.0, table).await
let mut counter = dataset.scan();
counter.filter_expr(indexed);
if let Some(ids) = scope(dataset, table, query).await? {
counter.with_fragments(
dataset
.fragments()
.iter()
.filter(|f| ids.contains(&f.id))
.cloned()
.collect(),
);
}
let matches = counter.count_rows().await?;
tracing::debug!(matches, limit, "indexed metadata filter");
Ok(matches <= limit as u64)
}
/// Fetch rows by key, projected to `columns` (all of them when None).
@@ -151,7 +213,8 @@ async fn scanner(
.use_index(use_index)
.with_row_id();
if let Some(filter) = &query.filter {
crate::filter::apply(&mut scanner, filter)?;
let ready = crate::indexed_filter::ready_columns(dataset);
crate::filter::apply(&mut scanner, filter, &ready)?;
scanner.prefilter(!query.postfilter);
}
// An explicitly empty segment list searches no index, so there is
@@ -300,6 +363,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
};
let hits = flat(&dataset, &table, &query).await.unwrap();
assert_eq!(hits.len(), 2);
@@ -341,6 +405,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
},
)
.await
@@ -420,6 +485,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
},
)
.await
@@ -523,6 +589,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
};
assert!(flat(&dataset, &table, &query).await.is_err());
}
@@ -546,6 +613,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
};
let indexed = ann(&dataset, &table, &query).await.unwrap();
let exact = flat(&dataset, &table, &query).await.unwrap();
@@ -572,6 +640,7 @@ mod tests {
nprobes: Some(4),
index_segments: None,
fragments: None,
exact_match_limit: None,
};
assert!(!scanner(&dataset, &table, &query, true).await.unwrap().1);
crate::index::build(
@@ -139,6 +139,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
};
let want: Vec<(u64, u32)> = query::ann(&dataset, &resolved, &base)
.await
@@ -239,6 +240,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: Some(vec![u64::MAX]),
exact_match_limit: None,
};
assert!(
query::ann(&dataset, &resolved, &q)
+1
View File
@@ -44,6 +44,7 @@ fn query_for(vector: Vec<f32>, k: usize) -> Query {
nprobes: Some(4),
index_segments: None,
fragments: None,
exact_match_limit: None,
}
}
+1
View File
@@ -46,6 +46,7 @@ tracing.workspace = true
tracing-subscriber.workspace = true
[dev-dependencies]
lance-index = "10"
tempfile = "3"
[build-dependencies]
@@ -0,0 +1,290 @@
//! How often, and how slowly, each index's queries filter on each metadata
//! key over the last hour. Keys come from the parsed filter; the number of
//! indexes and of keys per index is bounded, so arbitrary keys cannot grow
//! memory without limit. Not exported as metrics: a key is no label.
use std::collections::{BTreeMap, HashMap};
use std::sync::Mutex;
use std::time::{Duration, Instant};
use weed_vector_core::indexed_filter::operand;
use crate::pb::vector as pb;
const BUCKETS: usize = 12;
const BUCKET: Duration = Duration::from_secs(300);
pub const WINDOW: Duration = Duration::from_secs(BUCKET.as_secs() * BUCKETS as u64);
pub const MAX_INDEXES: usize = 512;
pub const MAX_KEYS_PER_INDEX: usize = 32;
#[derive(Default, Clone, Copy)]
struct Bucket {
/// The bucket's number since the stats started.
at: u64,
queries: u64,
seconds: f64,
/// The operand kinds the bucket's queries compared the key with.
kinds: u8,
}
#[derive(Default)]
struct KeyStats {
buckets: [Bucket; BUCKETS],
}
#[derive(Default)]
struct IndexStats {
keys: HashMap<String, KeyStats>,
/// The last bucket a key was left uncounted in.
truncated: Option<u64>,
last: u64,
}
/// Whether bucket `at` is still in the window of bucket `now`.
fn live(at: u64, now: u64) -> bool {
at + BUCKETS as u64 > now
}
pub struct FilterStats {
start: Instant,
indexes: Mutex<HashMap<String, IndexStats>>,
}
impl Default for FilterStats {
fn default() -> Self {
Self::new()
}
}
impl FilterStats {
pub fn new() -> Self {
Self {
start: Instant::now(),
indexes: Mutex::new(HashMap::new()),
}
}
fn bucket(&self, now: Instant) -> u64 {
(now.duration_since(self.start).as_secs() / BUCKET.as_secs()) + 1
}
/// One query of `index` that took `elapsed` and filtered on `keys`.
pub fn record(&self, index: &str, keys: &BTreeMap<String, u8>, elapsed: Duration) {
self.record_at(index, keys, elapsed, Instant::now());
}
fn record_at(&self, index: &str, keys: &BTreeMap<String, u8>, elapsed: Duration, now: Instant) {
if keys.is_empty() {
return;
}
let at = self.bucket(now);
let mut indexes = self.indexes.lock().unwrap();
if !indexes.contains_key(index) && indexes.len() >= MAX_INDEXES {
let Some(oldest) = indexes
.iter()
.min_by_key(|(_, stats)| stats.last)
.map(|(name, _)| name.clone())
else {
return;
};
indexes.remove(&oldest);
}
let stats = indexes.entry(index.to_string()).or_default();
stats.last = stats.last.max(at);
for (key, kinds) in keys {
if !stats.keys.contains_key(key) && stats.keys.len() >= MAX_KEYS_PER_INDEX {
// Make room only by dropping a key no query used in the window.
let idle = stats
.keys
.iter()
.find(|(_, k)| k.buckets.iter().all(|b| !live(b.at, at)))
.map(|(name, _)| name.clone());
match idle {
Some(idle) => {
stats.keys.remove(&idle);
}
None => {
stats.truncated = stats.truncated.max(Some(at));
continue;
}
}
}
let entry = stats.keys.entry(key.clone()).or_default();
let bucket = &mut entry.buckets[(at % BUCKETS as u64) as usize];
if bucket.at != at {
*bucket = Bucket {
at,
..Default::default()
};
}
bucket.queries += 1;
bucket.seconds += elapsed.as_secs_f64();
bucket.kinds |= kinds;
}
}
/// The stats of `index` over the window.
pub fn snapshot(&self, index: &str) -> pb::FilterStatsResponse {
self.snapshot_at(index, Instant::now())
}
fn snapshot_at(&self, index: &str, now: Instant) -> pb::FilterStatsResponse {
let at = self.bucket(now);
let window_seconds = now.duration_since(self.start).min(WINDOW).as_secs() as u32;
let indexes = self.indexes.lock().unwrap();
let Some(stats) = indexes.get(index) else {
return pb::FilterStatsResponse {
window_seconds,
..Default::default()
};
};
let mut keys: Vec<pb::FilterKeyStats> = stats
.keys
.iter()
.filter_map(|(key, s)| {
let (queries, seconds, kinds) = s
.buckets
.iter()
.filter(|b| live(b.at, at))
.fold((0, 0.0, 0), |(q, t, k), b| {
(q + b.queries, t + b.seconds, k | b.kinds)
});
(queries > 0).then(|| pb::FilterKeyStats {
key: key.clone(),
queries,
seconds,
string_operand: kinds & operand::STRING != 0,
number_operand: kinds & operand::NUMBER != 0,
bool_operand: kinds & operand::BOOL != 0,
})
})
.collect();
keys.sort_by(|a, b| b.queries.cmp(&a.queries).then(a.key.cmp(&b.key)));
pb::FilterStatsResponse {
keys,
window_seconds,
truncated: stats.truncated.is_some_and(|t| live(t, at)),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn keys(pairs: &[(&str, u8)]) -> BTreeMap<String, u8> {
pairs.iter().map(|(k, v)| (k.to_string(), *v)).collect()
}
#[test]
fn counts_queries_and_latency_per_key_within_the_window() {
let stats = FilterStats::new();
let t0 = stats.start;
let ms = Duration::from_millis;
stats.record_at("b/i", &keys(&[("author", operand::STRING)]), ms(10), t0);
stats.record_at(
"b/i",
&keys(&[("author", operand::STRING), ("year", operand::NUMBER)]),
ms(30),
t0 + Duration::from_secs(600),
);
stats.record_at("b/j", &keys(&[("x", operand::BOOL)]), ms(5), t0);
let got = stats.snapshot_at("b/i", t0 + Duration::from_secs(900));
assert_eq!(got.window_seconds, 900);
assert_eq!(got.keys.len(), 2);
assert_eq!(got.keys[0].key, "author");
assert_eq!(got.keys[0].queries, 2);
assert!((got.keys[0].seconds - 0.04).abs() < 1e-9);
assert!(got.keys[0].string_operand && !got.keys[0].number_operand);
assert_eq!(got.keys[1].key, "year");
assert!(got.keys[1].number_operand);
// The first query leaves the window an hour after its bucket.
let later = stats.snapshot_at("b/i", t0 + WINDOW + Duration::from_secs(1));
assert_eq!(later.window_seconds, WINDOW.as_secs() as u32);
assert_eq!(later.keys[0].queries, 1);
let year = later.keys.iter().find(|k| k.key == "year").unwrap();
assert!(year.number_operand);
// A string query that left the window no longer marks the key.
stats.record_at(
"b/i",
&keys(&[("author", operand::NUMBER)]),
ms(1),
t0 + WINDOW + Duration::from_secs(2),
);
let mixed = stats.snapshot_at("b/i", t0 + WINDOW + Duration::from_secs(3));
let author = mixed.keys.iter().find(|k| k.key == "author").unwrap();
assert!(author.string_operand && author.number_operand);
let numeric = stats.snapshot_at("b/i", t0 + WINDOW + Duration::from_secs(700));
let author = numeric.keys.iter().find(|k| k.key == "author").unwrap();
assert!(!author.string_operand && author.number_operand);
let gone = stats.snapshot_at("b/i", t0 + 2 * WINDOW + Duration::from_secs(900));
assert!(gone.keys.is_empty());
assert!(stats.snapshot_at("other", t0).keys.is_empty());
}
#[test]
fn indexes_and_keys_are_bounded() {
let stats = FilterStats::new();
let t0 = stats.start;
for i in 0..MAX_KEYS_PER_INDEX + 5 {
stats.record_at(
"b/i",
&keys(&[(&format!("k{i}"), operand::STRING)]),
Duration::ZERO,
t0,
);
}
let got = stats.snapshot_at("b/i", t0);
assert_eq!(got.keys.len(), MAX_KEYS_PER_INDEX);
assert!(got.truncated);
// A key idle for the whole window makes room for a new one.
let later = t0 + WINDOW + BUCKET;
stats.record_at(
"b/i",
&keys(&[("new", operand::STRING)]),
Duration::ZERO,
later,
);
let got = stats.snapshot_at("b/i", later);
assert!(!got.truncated, "the uncounted keys left the window");
assert_eq!(got.keys.len(), 1);
assert_eq!(got.keys[0].key, "new");
for i in 0..MAX_INDEXES + 3 {
stats.record_at(
&format!("b/{i}"),
&keys(&[("k", operand::STRING)]),
Duration::ZERO,
later,
);
}
assert_eq!(stats.indexes.lock().unwrap().len(), MAX_INDEXES);
}
#[test]
fn a_record_from_an_earlier_bucket_keeps_the_later_truncation() {
let stats = FilterStats::new();
let t0 = stats.start;
for i in 0..=MAX_KEYS_PER_INDEX {
stats.record_at(
"b/i",
&keys(&[(&format!("k{i}"), operand::STRING)]),
Duration::ZERO,
t0 + BUCKET,
);
}
// Took its time before the lock while the later bucket recorded.
stats.record_at(
"b/i",
&keys(&[("late", operand::STRING)]),
Duration::ZERO,
t0,
);
let got = stats.snapshot_at("b/i", t0 + WINDOW);
assert!(
got.truncated,
"the later bucket's uncounted key is in the window"
);
assert_eq!(stats.indexes.lock().unwrap()["b/i"].last, 2);
}
}
@@ -0,0 +1,627 @@
//! Indexed metadata keys through the VectorService: PutVectors derives the
//! columns, and a query answers the same with or without them.
use std::sync::Arc;
use arrow_array::RecordBatch;
use arrow_schema::Schema;
use lance::Dataset;
use lance::dataset::{BatchUDF, NewColumnTransform};
use lance::index::DatasetIndexExt;
use lance_index::IndexType;
use lance_index::scalar::{BuiltinIndexType, ScalarIndexParams};
use tonic::Request;
use weed_lance_worker::metadata_index::{self, STATE_FIELD_META, State};
use weed_vector_core::query::{self, Query};
use weed_vector_core::table::VectorTable;
use crate::pb::vector::{self as pb, vector_service_server::VectorService};
use crate::service::VectorServer;
const DIM: usize = 8;
const ROWS: usize = 3000;
const COLUMN: &str = "_mk_0";
/// How the engine answers a filter once `author` is indexed.
#[derive(Debug, Clone, Copy, PartialEq)]
enum Path {
/// Through the column, at most the exact-match limit of rows: exact.
Exact,
/// Through the column, more rows: the vector index with the column's
/// prefilter.
Ivf,
/// Not through the column.
Json,
}
/// S3 Vectors filters, the gateway's translation of each, and its path.
const FILTERS: [(&str, &str, Path); 22] = [
(
"{\"author\":\"alice\"}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"alice\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"alice\"'), false))",
Path::Ivf,
),
(
"{\"author\":{\"$eq\":\"bob\"}}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"bob\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"bob\"'), false))",
Path::Exact,
),
(
"{\"author\":\"carol\"}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"carol\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"carol\"'), false))",
Path::Exact,
),
(
"{\"author\":\"x\"}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"x\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"x\"'), false))",
Path::Exact,
),
(
"{\"author\":\"7\"}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"7\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"7\"'), false))",
Path::Exact,
),
(
"{\"author\":7}",
"(COALESCE(TRY_CAST(json_extract(`metadata`, '$[\"author\"]') AS DOUBLE) = 7, false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '7'), false))",
Path::Json,
),
(
"{\"author\":true}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = 'true', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', 'true'), false))",
Path::Json,
),
(
"{\"author\":{\"$in\":[\"alice\",\"bob\",\"x\"]}}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') in ('\"alice\"', '\"bob\"', '\"x\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"alice\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"bob\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"x\"'), false))",
Path::Ivf,
),
(
"{\"author\":{\"$in\":[\"alice\",7]}}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') in ('\"alice\"'), false) OR COALESCE(TRY_CAST(json_extract(`metadata`, '$[\"author\"]') AS DOUBLE) in (7), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"alice\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '7'), false))",
Path::Json,
),
(
"{\"author\":{\"$ne\":\"alice\"}}",
"NOT (COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"alice\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"alice\"'), false))",
Path::Json,
),
(
"{\"author\":{\"$nin\":[\"alice\",\"bob\"]}}",
"NOT (COALESCE(json_extract(`metadata`, '$[\"author\"]') in ('\"alice\"', '\"bob\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"alice\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"bob\"'), false))",
Path::Json,
),
(
"{\"author\":{\"$exists\":false}}",
"NOT json_exists(`metadata`, '$[\"author\"]')",
Path::Json,
),
(
"{\"$and\":[{\"author\":\"alice\"},{\"genre\":\"g1\"}]}",
"((COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"alice\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"alice\"'), false)) AND (COALESCE(json_extract(`metadata`, '$[\"genre\"]') = '\"g1\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"genre\"]', '\"g1\"'), false)))",
Path::Ivf,
),
(
"{\"$and\":[{\"author\":\"bob\"},{\"year\":{\"$gt\":2005}}]}",
"((COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"bob\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"bob\"'), false)) AND TRY_CAST(json_extract(`metadata`, '$[\"year\"]') AS DOUBLE) > 2005)",
Path::Exact,
),
(
"{\"$or\":[{\"author\":\"bob\"},{\"genre\":\"g2\"}]}",
"((COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"bob\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"bob\"'), false)) OR (COALESCE(json_extract(`metadata`, '$[\"genre\"]') = '\"g2\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"genre\"]', '\"g2\"'), false)))",
Path::Json,
),
(
"{\"author\":\"o'neil \\\"q\\\" \\\\ \\n\"}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"o''neil \\\"q\\\" \\\\ \\n\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"o''neil \\\"q\\\" \\\\ \\n\"'), false))",
Path::Exact,
),
(
"{\"author\":\"nobody\"}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"nobody\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"nobody\"'), false))",
Path::Exact,
),
("{\"author\":{\"$in\":[]}}", "false", Path::Json),
(
"{\"genre\":\"g1\"}",
"(COALESCE(json_extract(`metadata`, '$[\"genre\"]') = '\"g1\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"genre\"]', '\"g1\"'), false))",
Path::Json,
),
(
"{\"author\":\"u5\"}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"u5\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"u5\"'), false))",
Path::Exact,
),
(
"{\"author\":\"ctl\\u0001x\"}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"ctl\\u{1}x\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"ctl\\u0001x\"'), false))",
Path::Json,
),
(
"{\"author\":{\"$in\":[\"u5\",\"u6\"]}}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') in ('\"u5\"', '\"u6\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"u5\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"u6\"'), false))",
Path::Exact,
),
];
fn rng(seed: &mut u64) -> f32 {
*seed = seed
.wrapping_mul(6364136223846793005)
.wrapping_add(1442695040888963407);
((*seed >> 33) as f32 / (1u64 << 31) as f32) - 0.5
}
fn vector_of(i: usize) -> Vec<f32> {
let mut centre = (i % 16) as u64 + 1;
let mut noise = i as u64 * 7919 + 13;
(0..DIM)
.map(|_| rng(&mut centre) * 4.0 + rng(&mut noise))
.collect()
}
fn author(i: usize) -> Option<serde_json::Value> {
use serde_json::json;
Some(match i % 100 {
0..=9 => json!("alice"),
10 => json!("bob"),
11 => json!(["carol", "x"]),
12 => json!(7),
13 => json!("7"),
14 => serde_json::Value::Null,
15 => return None,
16 => json!(true),
17 => json!("o'neil \"q\" \\ \n"),
18 => json!("ctl\u{1}x"),
19 => json!(["alice", 3]),
_ => json!(format!("u{}", i % 37)),
})
}
fn row(i: usize, author: Option<serde_json::Value>) -> pb::Vector {
let mut meta = serde_json::json!({"genre": format!("g{}", i % 3), "year": 2000 + i % 10});
if let Some(author) = author {
meta["author"] = author;
}
let serde_json::Value::Object(meta) = meta else {
unreachable!()
};
pb::Vector {
key: format!("k{i}"),
data: vector_of(i),
metadata: Some(crate::service::json_to_struct(meta)),
}
}
struct Fixture {
server: VectorServer,
exact_off: VectorServer,
table: pb::Table,
}
impl Fixture {
async fn new() -> Self {
let bucket = format!(
"mk-{:x}",
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
);
let table = pb::Table {
id: vec![bucket.clone(), "vector".into(), "x".into()],
location: format!("shared-memory://{bucket}/vector/x"),
..Default::default()
};
let fixture = Self {
server: VectorServer::new().with_schemes(&["shared-memory"]),
exact_off: VectorServer::new()
.with_schemes(&["shared-memory"])
.with_exact_match_limit(Some(0)),
table,
};
fixture
.server
.create_index(Request::new(pb::CreateIndexRequest {
table: Some(fixture.table.clone()),
dimension: DIM as u32,
distance_metric: pb::DistanceMetric::L2 as i32,
..Default::default()
}))
.await
.unwrap();
fixture
}
async fn put(&self, vectors: Vec<pb::Vector>) {
self.server
.put_vectors(Request::new(pb::PutVectorsRequest {
table: Some(self.table.clone()),
vectors,
}))
.await
.unwrap();
}
async fn dataset(&self, version: u64) -> Dataset {
let dataset = Dataset::open(&self.table.location).await.unwrap();
if version == 0 {
dataset
} else {
dataset.checkout_version(version).await.unwrap()
}
}
/// Adds `author`'s column, indexes it and marks it ready, returning the
/// version before it was ready.
async fn index_author(&self) -> u64 {
let mut dataset = self.dataset(0).await;
let output = Arc::new(Schema::new(vec![metadata_index::field(
COLUMN,
"author",
State::Building,
)]));
let schema = output.clone();
dataset
.add_columns(
NewColumnTransform::BatchUDF(BatchUDF {
mapper: Box::new(move |batch: &RecordBatch| {
let column = metadata_index::column_from_metadata(
batch.column_by_name("metadata").unwrap().as_ref(),
"author",
)
.map_err(|e| lance::Error::invalid_input(e.to_string()))?;
Ok(RecordBatch::try_new(schema.clone(), vec![column])?)
}),
output_schema: output,
result_checkpoint: None,
}),
Some(vec!["metadata".to_string()]),
None,
)
.await
.unwrap();
dataset
.create_index(
&[COLUMN],
IndexType::LabelList,
Some(format!("{COLUMN}_idx")),
&ScalarIndexParams::for_builtin(BuiltinIndexType::LabelList),
true,
)
.await
.unwrap();
let building = dataset.version().version;
self.set_state(State::Ready).await;
building
}
async fn set_state(&self, state: State) -> u64 {
let mut dataset = self.dataset(0).await;
dataset
.update_field_metadata()
.update(COLUMN, [(STATE_FIELD_META, state.as_str())])
.unwrap()
.await
.unwrap();
dataset.version().version
}
async fn query(
&self,
server: &VectorServer,
version: u64,
filter: &str,
k: u32,
v: &[f32],
) -> Vec<(String, f32)> {
let mut table = self.table.clone();
table.version = version;
server
.query_vectors(Request::new(pb::QueryVectorsRequest {
table: Some(table),
query_vector: v.to_vec(),
top_k: k,
filter: filter.to_string(),
return_distance: true,
..Default::default()
}))
.await
.unwrap()
.into_inner()
.vectors
.into_iter()
.map(|r| (r.key, r.distance))
.collect()
}
/// Exact filtered top-k at `version`.
async fn flat(&self, version: u64, filter: &str, k: u32, v: &[f32]) -> Vec<(String, f32)> {
let dataset = self.dataset(version).await;
let table = VectorTable::resolve(&dataset, None, Some("key")).unwrap();
query::flat(
&dataset,
&table,
&Query {
vector: v.to_vec(),
k: k as usize,
filter: Some(filter.to_string()),
postfilter: false,
distance: lance_linalg::distance::DistanceType::L2,
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
},
)
.await
.unwrap()
.into_iter()
.map(|h| (h.key, h.distance))
.collect()
}
/// Every filter answers through the column as through the JSON at
/// `json` (the same rows and indexes, the column not ready), and an
/// exact answer as the exact filtered top-k.
async fn compare(&self, json: u64, ready: u64) {
for q in [7, 1234] {
let v = vector_of(q);
for (s3, filter, path) in FILTERS {
for k in [10, 400] {
let what = format!("{s3} k={k} q={q}");
let exact = self.flat(json, filter, k, &v).await;
assert_same(&self.flat(ready, filter, k, &v).await, &exact, &what);
let json_ann = self.query(&self.server, json, filter, k, &v).await;
let off = self.query(&self.exact_off, ready, filter, k, &v).await;
assert_same(&off, &json_ann, &format!("{what}: indexed prefilter"));
let got = self.query(&self.server, ready, filter, k, &v).await;
let want = if path == Path::Exact {
&exact
} else {
&json_ann
};
assert_same(&got, want, &format!("{what}: {path:?}"));
}
}
}
}
}
fn assert_same(got: &[(String, f32)], want: &[(String, f32)], what: &str) {
assert_eq!(got.len(), want.len(), "{what}: {got:?} vs {want:?}");
for (g, w) in got.iter().zip(want) {
let same = g.0 == w.0 || (g.1 - w.1).abs() < 1e-4;
assert!(
same && ((g.1 - w.1).abs() < 1e-4 || (g.1.is_infinite() && w.1.is_infinite())),
"{what}: {got:?} vs {want:?}"
);
}
}
/// Indexed and JSON answers agree on mixed types, missing keys, arrays,
/// null, untouched operators, and after updates, deletes and rows no
/// scalar or vector index covers yet.
#[tokio::test]
async fn indexed_answers_match_the_json_path() {
let fixture = Fixture::new().await;
for start in (0..ROWS).step_by(300) {
fixture
.put((start..start + 300).map(|i| row(i, author(i))).collect())
.await;
}
let mut dataset = fixture.dataset(0).await;
weed_lance_worker::vector_index::build(&mut dataset, "vector", "vector_idx", 16, false)
.await
.unwrap();
let json = fixture.index_author().await;
let schema: Schema = fixture.dataset(0).await.schema().into();
assert_eq!(
metadata_index::indexed_keys(&schema)[0].state,
Some(State::Ready)
);
fixture.compare(json, 0).await;
// Updates move rows between values, deletes drop some, and new rows land
// in fragments neither index covers; PutVectors fills the column.
fixture
.put(
(0..60)
.map(|i| row(i, Some(serde_json::json!(["moved", "bob"]))))
.chain((ROWS..ROWS + 200).map(|i| row(i, author(i))))
.collect(),
)
.await;
fixture
.server
.delete_vectors(Request::new(pb::DeleteVectorsRequest {
table: Some(fixture.table.clone()),
keys: (100..130).map(|i| format!("k{i}")).collect(),
}))
.await
.unwrap();
let json = fixture.set_state(State::Building).await;
let ready = fixture.set_state(State::Ready).await;
fixture.compare(json, ready).await;
let moved = r#"(COALESCE(json_extract(`metadata`, '$["author"]') = '"moved"', false) OR COALESCE(json_array_contains(`metadata`, '$["author"]', '"moved"'), false))"#;
let hits = fixture
.query(&fixture.server, ready, moved, 100, &vector_of(0))
.await;
assert_eq!(hits.len(), 60);
}
/// The planner counts and scores per scope: an exact answer over a fragment
/// subset is that subset's exact top-k.
#[tokio::test]
async fn a_scoped_query_is_exact_within_its_fragments() {
let fixture = Fixture::new().await;
for start in (0..ROWS).step_by(300) {
fixture
.put((start..start + 300).map(|i| row(i, author(i))).collect())
.await;
}
let mut dataset = fixture.dataset(0).await;
weed_lance_worker::vector_index::build(&mut dataset, "vector", "vector_idx", 16, false)
.await
.unwrap();
let json = fixture.index_author().await;
let ready = fixture.dataset(0).await;
let table = VectorTable::resolve(&ready, None, Some("key")).unwrap();
let ids: Vec<u64> = ready.fragments().iter().map(|f| f.id).take(4).collect();
let filter = FILTERS[1].1;
let scoped = || Query {
vector: vector_of(5),
k: 10,
filter: Some(filter.to_string()),
postfilter: false,
distance: lance_linalg::distance::DistanceType::L2,
nprobes: None,
index_segments: None,
fragments: Some(ids.clone()),
exact_match_limit: None,
};
let got = query::ann(&ready, &table, &scoped()).await.unwrap();
let old = fixture.dataset(json).await;
let want = query::flat(&old, &table, &scoped()).await.unwrap();
assert_eq!(want.len(), 10);
let pairs = |hits: Vec<weed_vector_core::Hit>| {
hits.into_iter()
.map(|h| (h.key, h.distance))
.collect::<Vec<_>>()
};
assert_same(&pairs(got), &pairs(want), "scoped bob");
}
/// The engine counts each index's filtered queries by key.
#[tokio::test]
async fn queries_are_counted_by_filter_key() {
let fixture = Fixture::new().await;
fixture
.put((0..50).map(|i| row(i, author(i))).collect())
.await;
for (_, filter, _) in &FILTERS[..3] {
fixture
.query(&fixture.server, 0, filter, 5, &vector_of(1))
.await;
}
let year = FILTERS[13].1;
fixture
.query(&fixture.server, 0, year, 5, &vector_of(1))
.await;
let stats = fixture
.server
.filter_stats(Request::new(pb::FilterStatsRequest {
table: Some(fixture.table.clone()),
}))
.await
.unwrap()
.into_inner();
let author = stats.keys.iter().find(|k| k.key == "author").unwrap();
assert_eq!(author.queries, 4);
assert!(author.string_operand && !author.number_operand);
let year = stats.keys.iter().find(|k| k.key == "year").unwrap();
assert_eq!(year.queries, 1);
assert!(year.number_operand && !year.string_operand);
}
/// A PutVectors that opened the table before its key column was added and
/// commits after: Lance rebases the write and its rows hold NULL in the new
/// column, which the write then fills from the rows' metadata.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_write_that_raced_the_column_add_fills_the_column() {
let fixture = Fixture::new().await;
fixture
.put((0..300).map(|i| row(i, author(i))).collect())
.await;
let unhealed = fixture.dataset(0).await;
let healed = fixture.dataset(0).await;
fixture.index_author().await;
let nulls = |dataset: Dataset| async move {
dataset
.count_rows(Some(format!("{COLUMN} IS NULL")))
.await
.unwrap()
};
let rows: Vec<pb::Vector> = (250..400).map(|i| row(i, author(i))).collect();
let written = crate::service::merge_vectors(unhealed, &rows)
.await
.unwrap();
assert!(written.schema().field(COLUMN).is_some());
assert!(nulls(written).await >= 100, "the race is reproduced");
let written = crate::service::write_vectors(healed, &rows).await.unwrap();
assert_eq!(nulls(written.clone()).await, 0);
let filter = FILTERS[1].1;
let latest = written.version().version;
let v = vector_of(5);
assert_same(
&fixture
.query(&fixture.server, latest, filter, 400, &v)
.await,
&fixture.flat(latest, filter, 400, &v).await,
"bob after the race",
);
assert_eq!(fixture.flat(latest, filter, 400, &v).await.len(), 4);
}
/// Two writes updating the same keys raced the column add, and the first
/// one's fill commits last: each write's rows get the values of its own
/// metadata, not the ones derived before it or from the other write.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_write_that_raced_the_column_add_derives_its_updated_rows() {
let fixture = Fixture::new().await;
fixture
.put((0..300).map(|i| row(i, author(i))).collect())
.await;
let first = fixture.dataset(0).await;
let second = fixture.dataset(0).await;
let written: Schema = first.schema().into();
fixture.index_author().await;
let rows = |name: &str| -> Vec<pb::Vector> {
(250..300)
.map(|i| row(i, Some(serde_json::json!(name))))
.collect()
};
let keys: Vec<String> = (250..300).map(|i| format!("k{i}")).collect();
let first = crate::service::merge_vectors(first, &rows("alice"))
.await
.unwrap();
let second = crate::service::merge_vectors(second, &rows("bob"))
.await
.unwrap();
crate::service::heal_columns(second, &written, &keys)
.await
.unwrap();
crate::service::heal_columns(first, &written, &keys)
.await
.unwrap();
let latest = fixture.dataset(0).await;
let count = |filter: String| {
let latest = latest.clone();
async move { latest.count_rows(Some(filter)).await.unwrap() }
};
assert_eq!(count(format!("{COLUMN} IS NULL")).await, 0);
for (name, filter) in [("alice", FILTERS[0].1), ("bob", FILTERS[1].1)] {
assert_eq!(
count(format!("array_has({COLUMN}, '{name}')")).await,
count(filter.to_string()).await,
"{name}"
);
}
let mut got = fixture
.server
.get_vectors(Request::new(pb::GetVectorsRequest {
table: Some(fixture.table.clone()),
keys: keys.clone(),
return_data: true,
return_metadata: true,
}))
.await
.unwrap()
.into_inner()
.vectors;
got.sort_by_key(|v| v.key[1..].parse::<usize>().unwrap());
assert_eq!(got, rows("bob"));
}
+7
View File
@@ -1,5 +1,12 @@
//! The VectorService gRPC server the S3 gateway talks to.
// PutVectors awaits two Lance merge_insert futures; proving the handler
// future Send takes deeper trait recursion than the default.
#![recursion_limit = "256"]
pub mod filter_stats;
#[cfg(test)]
mod indexed_keys_tests;
pub mod metrics;
pub mod observed;
pub mod read_cache;
+6
View File
@@ -52,6 +52,11 @@ struct Args {
/// tenth of the index's partitions, at least 8; more under a filter.
#[arg(long, env = "WEED_VECTOR_NPROBES")]
nprobes: Option<std::num::NonZeroUsize>,
/// A filtered query whose indexed metadata keys match at most this many
/// rows scores them exactly instead of searching the vector index; 0
/// never does.
#[arg(long, env = "WEED_VECTOR_EXACT_MATCH_LIMIT", default_value_t = weed_vector_core::query::DEFAULT_EXACT_MATCH_LIMIT)]
exact_match_limit: usize,
/// The cluster's security.toml: [grpc.vector] for this server,
/// [grpc.client] for the masters and filers, https.client for volume
/// servers under HTTPS, jwt.signing.read.key for signed volume reads,
@@ -170,6 +175,7 @@ async fn main() -> Result<()> {
.then(|| ReadCache::new(args.read_cache_mb << 20).with_metrics(metrics.clone()));
let mut service = VectorServer::new()
.with_nprobes(args.nprobes.map(std::num::NonZeroUsize::get))
.with_exact_match_limit(Some(args.exact_match_limit))
.with_lance_caches(args.index_cache_mb << 20, args.metadata_cache_mb << 20)
.with_storage(reads);
if let Some(cache) = &cache {
@@ -10,6 +10,10 @@ use tonic_health::{ServingStatus, server::HealthReporter};
use crate::read_cache::ReadCache;
/// The gRPC health service an engine reports serving when its writes derive
/// the indexed metadata key columns.
pub const METADATA_COLUMNS_CAPABILITY: &str = "weed.vector.capability.metadata_columns";
#[derive(Clone)]
pub struct Metrics {
pub registry: Registry,
@@ -120,6 +124,11 @@ impl Metrics {
health
.set_service_status("vector_pb.VectorService", status)
.await;
// A capability, not readiness: the admin refuses to index a
// metadata key while an engine without it could write the table.
health
.set_service_status(METADATA_COLUMNS_CAPABILITY, ServingStatus::Serving)
.await;
}
}
@@ -78,6 +78,15 @@ impl VectorService for ObservedService {
}
result
}
async fn filter_stats(
&self,
request: Request<pb::FilterStatsRequest>,
) -> Result<Response<pb::FilterStatsResponse>, Status> {
let mut in_flight = self.metrics.start("FilterStats");
let result = self.inner.filter_stats(request).await;
in_flight.complete(result.as_ref().err().map(|e| e.code()));
result
}
}
#[cfg(test)]
+180 -35
View File
@@ -19,7 +19,9 @@ use weed_lance_worker::catalog::TableDescription;
use weed_lance_worker::dataset;
use weed_lance_worker::direct_read::{DirectReads, Wrappers};
use weed_lance_worker::jobs::vector_index::is_vector_table;
use weed_lance_worker::metadata_index;
use weed_vector_core::index;
use weed_vector_core::indexed_filter;
use weed_vector_core::maintenance::SharedLeases;
use weed_vector_core::query::{self, Query};
use weed_vector_core::table::VectorTable;
@@ -167,14 +169,13 @@ fn batch_of(vectors: &[pb::Vector], dimension: usize, schema: Arc<Schema>) -> Re
};
let vectors_array =
FixedSizeListArray::try_new(item_field.clone(), dimension as i32, values, None)?;
let metas = vectors
let json: Vec<Option<serde_json::Value>> = vectors
.iter()
.map(|v| {
v.metadata
.as_ref()
.map(|m| serde_json::to_string(&struct_to_json(m)))
.transpose()
})
.map(|v| v.metadata.as_ref().map(struct_to_json))
.collect();
let metas = json
.iter()
.map(|m| m.as_ref().map(serde_json::to_string).transpose())
.collect::<Result<Vec<_>, _>>()?;
// The stored column is lance's jsonb (LargeBinary) once the field carries
// the arrow.json extension; encode the strings, else keep the Utf8 batch.
@@ -187,11 +188,144 @@ fn batch_of(vectors: &[pb::Vector], dimension: usize, schema: Arc<Schema>) -> Re
}
_ => Arc::new(StringArray::from(metas)),
};
RecordBatch::try_new(
schema,
vec![Arc::new(keys), Arc::new(vectors_array), metadata_array],
let (keys, vectors_array): (ArrayRef, ArrayRef) = (Arc::new(keys), Arc::new(vectors_array));
// An indexed metadata key's column is derived from the row's metadata,
// whatever the key's state.
let columns = schema
.fields()
.iter()
.map(|field| match field.name().as_str() {
KEY_COLUMN => Ok(keys.clone()),
VECTOR_COLUMN => Ok(vectors_array.clone()),
METADATA_COLUMN => Ok(metadata_array.clone()),
name => match metadata_index::indexed_key(field) {
Some(indexed) => Ok(metadata_index::column_from_json(
json.iter().map(Option::as_ref),
&indexed.key,
)),
None => bail!("the table has a column {name} that PutVectors cannot fill"),
},
})
.collect::<Result<Vec<_>>>()?;
RecordBatch::try_new(schema, columns).map_err(Into::into)
}
/// Upserts `vectors` by key into `dataset`, the version the request opened,
/// then fills the indexed-key columns a concurrent column add left empty.
pub(crate) async fn write_vectors(
dataset: Dataset,
vectors: &[pb::Vector],
) -> Result<Dataset, Status> {
let written = Arc::new(Into::<arrow_schema::Schema>::into(dataset.schema()));
let dataset = merge_vectors(dataset, vectors).await?;
let keys: Vec<String> = vectors.iter().map(|v| v.key.clone()).collect();
heal_columns(dataset, &written, &keys)
.await
.map_err(internal)
}
/// The upsert itself, in the schema of the version `dataset` is at.
pub(crate) async fn merge_vectors(
dataset: Dataset,
vectors: &[pb::Vector],
) -> Result<Dataset, Status> {
let dimension = VectorTable::resolve(&dataset, None, Some(KEY_COLUMN))
.map_err(invalid)?
.dimension;
let batch = batch_of(
vectors,
dimension,
Arc::new(Into::<arrow_schema::Schema>::into(dataset.schema())),
)
.map_err(Into::into)
.map_err(invalid)?;
// Upsert by key in one merge commit: no window where the old rows are
// already deleted, and a losing writer fails the request instead of
// leaving duplicate keys.
let mut builder = MergeInsertBuilder::try_new(Arc::new(dataset), vec![KEY_COLUMN.to_string()])
.map_err(internal)?;
builder.when_matched(WhenMatched::UpdateAll);
builder.when_not_matched(WhenNotMatched::InsertAll);
let (dataset, _stats) = builder
.try_build()
.map_err(internal)?
.execute_batches(vec![batch])
.await
.map_err(internal)?;
Ok(Arc::unwrap_or_clone(dataset))
}
/// Attempts [`heal_columns`] makes against writes committing meanwhile.
const HEAL_ATTEMPTS: usize = 10;
/// Lance rebases a write onto a column add committed after the write opened
/// its version: the rows it inserted hold NULL in the new column and the
/// rows it updated keep the value derived from their old metadata. Derive
/// such columns again for the rows of `keys` from their stored metadata.
pub(crate) async fn heal_columns(
mut dataset: Dataset,
written: &Schema,
keys: &[String],
) -> Result<Dataset> {
for _ in 0..HEAL_ATTEMPTS {
let schema: Schema = dataset.schema().into();
let missing: Vec<metadata_index::IndexedKey> = metadata_index::indexed_keys(&schema)
.into_iter()
.filter(|k| written.field_with_name(&k.column).is_err())
.collect();
if missing.is_empty() || keys.is_empty() {
return Ok(dataset);
}
let mut scanner = dataset.scan();
scanner
.filter(&key_list(keys))?
.project(&[KEY_COLUMN, METADATA_COLUMN])?;
let rows: Vec<RecordBatch> = scanner.try_into_stream().await?.try_collect().await?;
let mut fields = vec![schema.field_with_name(KEY_COLUMN)?.clone()];
for k in &missing {
fields.push(schema.field_with_name(&k.column)?.clone());
}
let patch_schema = Arc::new(Schema::new(fields));
let mut patches = Vec::new();
for batch in rows.iter().filter(|b| b.num_rows() > 0) {
let (Some(keys), Some(metadata)) = (
batch.column_by_name(KEY_COLUMN),
batch.column_by_name(METADATA_COLUMN),
) else {
bail!("the scan did not return keys and metadata");
};
let mut columns = vec![keys.clone()];
for k in &missing {
columns.push(metadata_index::column_from_metadata(
metadata.as_ref(),
&k.key,
)?);
}
patches.push(RecordBatch::try_new(patch_schema.clone(), columns)?);
}
if patches.is_empty() {
return Ok(dataset);
}
// Lance's own retry would join these values onto rows a later write
// replaced; derive them again from the version that write left.
let mut builder =
MergeInsertBuilder::try_new(Arc::new(dataset.clone()), vec![KEY_COLUMN.to_string()])?;
builder
.when_matched(WhenMatched::UpdateAll)
.when_not_matched(WhenNotMatched::DoNothing)
.conflict_retries(0);
match builder.try_build()?.execute_batches(patches).await {
Ok((healed, _stats)) => {
tracing::info!(
rows = rows.iter().map(|b| b.num_rows()).sum::<usize>(),
"derived indexed metadata key columns a concurrent column add left stale"
);
return Ok(Arc::unwrap_or_clone(healed));
}
Err(lance::Error::TooMuchWriteContention { .. }) => dataset.checkout_latest().await?,
Err(e) => return Err(e.into()),
}
}
bail!("writes kept replacing the rows while their indexed key columns were derived")
}
fn struct_to_json(s: &Struct) -> serde_json::Value {
@@ -230,7 +364,7 @@ fn json_to_value(v: serde_json::Value) -> Value {
Value { kind: Some(kind) }
}
fn json_to_struct(map: serde_json::Map<String, serde_json::Value>) -> Struct {
pub(crate) fn json_to_struct(map: serde_json::Map<String, serde_json::Value>) -> Struct {
Struct {
fields: map
.into_iter()
@@ -528,6 +662,8 @@ pub struct VectorServer {
lance_cache_bytes: (usize, usize),
leases: SharedLeases,
nprobes: Option<usize>,
exact_match_limit: Option<usize>,
filter_stats: crate::filter_stats::FilterStats,
/// URI schemes a table's location may have.
schemes: &'static [&'static str],
}
@@ -551,6 +687,8 @@ impl VectorServer {
handles: moka::sync::Cache::new(MAX_HANDLES),
leases: SharedLeases::default(),
nprobes: None,
exact_match_limit: None,
filter_stats: Default::default(),
schemes: &["s3"],
}
}
@@ -568,6 +706,13 @@ impl VectorServer {
self
}
/// Matches of a filter's indexed metadata keys scored exactly instead of
/// searched through the vector index, at most; None keeps the default.
pub fn with_exact_match_limit(mut self, limit: Option<usize>) -> Self {
self.exact_match_limit = limit;
self
}
/// Keep tables on the filers and volume servers, not through the gateway.
/// Unset only for tests on an in-process store.
pub fn with_storage(mut self, storage: DirectReads) -> Self {
@@ -808,29 +953,7 @@ impl VectorService for VectorServer {
version: dataset.version().version,
}));
}
let dimension = VectorTable::resolve(&dataset, None, Some(KEY_COLUMN))
.map_err(invalid)?
.dimension;
let batch = batch_of(
&req.vectors,
dimension,
Arc::new(Into::<arrow_schema::Schema>::into(dataset.schema())),
)
.map_err(invalid)?;
// Upsert by key in one merge commit: no window where the old rows are
// already deleted, and a losing writer fails the request instead of
// leaving duplicate keys.
let mut builder =
MergeInsertBuilder::try_new(Arc::new(dataset), vec![KEY_COLUMN.to_string()])
.map_err(internal)?;
builder.when_matched(WhenMatched::UpdateAll);
builder.when_not_matched(WhenNotMatched::InsertAll);
let (dataset, _stats) = builder
.try_build()
.map_err(internal)?
.execute_batches(vec![batch])
.await
.map_err(internal)?;
let dataset = write_vectors(dataset, &req.vectors).await?;
Ok(Response::new(pb::PutVectorsResponse {
version: dataset.version().version,
}))
@@ -948,6 +1071,7 @@ impl VectorService for VectorServer {
&self,
request: Request<pb::QueryVectorsRequest>,
) -> Result<Response<pb::QueryVectorsResponse>, Status> {
let started = std::time::Instant::now();
let req = request.into_inner();
let table = req
.table
@@ -957,6 +1081,11 @@ impl VectorService for VectorServer {
// while this query runs against it.
let _lease = self.leases.try_pin(&dataset).await.map_err(internal)?;
let resolved = VectorTable::resolve(&dataset, None, Some(KEY_COLUMN)).map_err(invalid)?;
let filter_keys = if req.filter.is_empty() {
Default::default()
} else {
indexed_filter::filter_keys(&dataset, &req.filter).unwrap_or_default()
};
if req.query_vector.len() != resolved.dimension {
return Err(Status::invalid_argument(format!(
"query vector carries {} floats, the index dimension is {}",
@@ -981,6 +1110,7 @@ impl VectorService for VectorServer {
nprobes: self.nprobes,
index_segments: None,
fragments: None,
exact_match_limit: self.exact_match_limit,
};
let hits = query::ann(&dataset, &resolved, &query)
.await
@@ -1056,8 +1186,23 @@ impl VectorService for VectorServer {
data: data_map.remove(&hit.key).unwrap_or_default(),
});
}
self.filter_stats
.record(&table.id.join("/"), &filter_keys, started.elapsed());
Ok(Response::new(pb::QueryVectorsResponse { vectors }))
}
async fn filter_stats(
&self,
request: Request<pb::FilterStatsRequest>,
) -> Result<Response<pb::FilterStatsResponse>, Status> {
let table = request
.into_inner()
.table
.ok_or_else(|| Status::invalid_argument("missing table"))?;
Ok(Response::new(
self.filter_stats.snapshot(&table.id.join("/")),
))
}
}
#[cfg(test)]
+2
View File
@@ -7170,8 +7170,10 @@ dependencies = [
"hmac 0.12.1",
"http 1.5.0",
"hyper-util",
"jsonb",
"jsonwebtoken",
"lance",
"lance-arrow",
"lance-datafusion",
"lance-index",
"lance-linalg",
+5 -1
View File
@@ -35,8 +35,12 @@ lance-table = "10"
# For the execution options the sort needs; the scanner's own stream helper
# takes the default, which has spilling off.
lance-datafusion = "10"
arrow-array = "58"
arrow-schema = "58"
arrow-cast = "58"
# Lance's stored form of the metadata JSON, read when deriving the
# indexed-key columns.
jsonb = { version = "0.5", default-features = false }
chrono = "0.4"
futures = "0.3"
anyhow.workspace = true
@@ -88,7 +92,7 @@ protoc-bin-vendored = "3"
[dev-dependencies]
tokio = { workspace = true, features = ["test-util"] }
tonic.workspace = true
arrow-array = "58"
lance-arrow = "10"
arrow-schema = "58"
arrow-cast = "58"
lance-linalg = "10"
+1
View File
@@ -11,6 +11,7 @@ pub mod dataset;
pub mod direct_read;
pub mod grpc_tls;
pub mod jobs;
pub mod metadata_index;
pub mod metrics;
pub mod preview;
pub mod security;
@@ -0,0 +1,285 @@
//! Metadata keys an S3 Vectors index keeps a scalar index for.
//!
//! Each indexed key is a hidden `_mk_<n>` column of `List<Utf8>`: the strings
//! an S3 `$eq` on a string matches, that is the value when it is a string or
//! the string elements of an array, and nothing for any other type or a
//! missing key. The field metadata names the key and its state. The
//! `metadata` JSON column stays the source of truth: every writer derives the
//! columns from it, and queries use one only once its state is `ready`.
use std::sync::Arc;
use anyhow::{Result, bail};
use arrow_array::builder::{ListBuilder, StringBuilder};
use arrow_array::{Array, ArrayRef, LargeBinaryArray, StringArray};
use arrow_schema::{DataType, Field, Schema};
pub const COLUMN_PREFIX: &str = "_mk_";
/// Field metadata naming the metadata key a column holds.
pub const KEY_FIELD_META: &str = "weed:metadata_key";
/// Field metadata holding the column's [`State`].
pub const STATE_FIELD_META: &str = "weed:metadata_key_state";
/// Indexed keys an index may have.
pub const MAX_KEYS: usize = 8;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum State {
/// The column exists; its scalar index may not.
Building,
/// The column and its scalar index exist; queries may use them.
Ready,
}
impl State {
pub fn as_str(self) -> &'static str {
match self {
State::Building => "building",
State::Ready => "ready",
}
}
pub fn parse(s: &str) -> Option<Self> {
match s {
"building" => Some(State::Building),
"ready" => Some(State::Ready),
_ => None,
}
}
}
/// One indexed key, as the table's schema records it.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct IndexedKey {
pub column: String,
pub key: String,
/// None for a state this version does not know, which no query uses.
pub state: Option<State>,
}
/// The type of every indexed-key column.
pub fn column_type() -> DataType {
DataType::List(Arc::new(Field::new("item", DataType::Utf8, true)))
}
/// The field of the column holding `key`.
pub fn field(column: &str, key: &str, state: State) -> Field {
Field::new(column, column_type(), true).with_metadata(
[
(KEY_FIELD_META.to_string(), key.to_string()),
(STATE_FIELD_META.to_string(), state.as_str().to_string()),
]
.into_iter()
.collect(),
)
}
/// The indexed-key columns of `schema`, in schema order.
pub fn indexed_keys(schema: &Schema) -> Vec<IndexedKey> {
schema
.fields()
.iter()
.filter_map(|f| indexed_key(f))
.collect()
}
/// The key `field` holds, when it is an indexed-key column.
pub fn indexed_key(field: &Field) -> Option<IndexedKey> {
if !field.name().starts_with(COLUMN_PREFIX) {
return None;
}
let key = field.metadata().get(KEY_FIELD_META)?;
Some(IndexedKey {
column: field.name().clone(),
key: key.clone(),
state: field
.metadata()
.get(STATE_FIELD_META)
.and_then(|s| State::parse(s)),
})
}
/// The strings an S3 `$eq` on a string matches for this stored value.
pub fn json_strings(value: Option<&serde_json::Value>) -> Vec<&str> {
match value {
Some(serde_json::Value::String(s)) => vec![s.as_str()],
Some(serde_json::Value::Array(items)) => items.iter().filter_map(|v| v.as_str()).collect(),
_ => Vec::new(),
}
}
/// [`json_strings`] for `key` of one row's metadata as Lance stores it (jsonb).
pub fn jsonb_strings(metadata: &[u8], key: &str) -> Result<Vec<String>> {
let raw = jsonb::RawJsonb::new(metadata);
let Some(value) = raw.get_by_name(key, false)? else {
return Ok(Vec::new());
};
let value = value.as_raw();
if let Some(s) = value.as_str()? {
return Ok(vec![s.into_owned()]);
}
let Some(items) = value.array_values()? else {
return Ok(Vec::new());
};
let mut out = Vec::new();
for item in items {
if let Some(s) = item.as_raw().as_str()? {
out.push(s.into_owned());
}
}
Ok(out)
}
/// The column for `key` over rows whose metadata is given as JSON objects.
pub fn column_from_json<'a>(
rows: impl IntoIterator<Item = Option<&'a serde_json::Value>>,
key: &str,
) -> ArrayRef {
let mut builder = ListBuilder::new(StringBuilder::new());
for row in rows {
for s in json_strings(row.and_then(|m| m.get(key))) {
builder.values().append_value(s);
}
builder.append(true);
}
Arc::new(builder.finish())
}
/// The column for `key` over a stored `metadata` column: jsonb (LargeBinary)
/// or JSON text (Utf8).
pub fn column_from_metadata(metadata: &dyn Array, key: &str) -> Result<ArrayRef> {
let mut builder = ListBuilder::new(StringBuilder::new());
if let Some(jsonb) = metadata.as_any().downcast_ref::<LargeBinaryArray>() {
for i in 0..jsonb.len() {
if !jsonb.is_null(i) {
for s in jsonb_strings(jsonb.value(i), key)? {
builder.values().append_value(s);
}
}
builder.append(true);
}
} else if let Some(text) = metadata.as_any().downcast_ref::<StringArray>() {
for i in 0..text.len() {
if !text.is_null(i) {
let row: serde_json::Value = serde_json::from_str(text.value(i))?;
for s in json_strings(row.get(key)) {
builder.values().append_value(s);
}
}
builder.append(true);
}
} else {
bail!("metadata column is {}", metadata.data_type());
}
Ok(Arc::new(builder.finish()))
}
#[cfg(test)]
mod tests {
use super::*;
use arrow_array::ListArray;
fn lists(array: &ArrayRef) -> Vec<Vec<String>> {
let list = array.as_any().downcast_ref::<ListArray>().unwrap();
(0..list.len())
.map(|i| {
assert!(!list.is_null(i));
let values = list.value(i);
let values = values.as_any().downcast_ref::<StringArray>().unwrap();
values.iter().map(|v| v.unwrap().to_string()).collect()
})
.collect()
}
const ROWS: [Option<&str>; 12] = [
Some(r#"{"k": "a"}"#),
Some(r#"{"k": 7}"#),
Some(r#"{"k": "7"}"#),
Some(r#"{"k": ["a", "b", 7, true, null, ["c"], {"d": "e"}]}"#),
Some(r#"{"x": "a"}"#),
Some(r#"{"k": null}"#),
Some(r#"{"k": true}"#),
Some(r#"{"k": {"a": "b"}}"#),
Some(r#"{"k": "line\nbreak\u0001"}"#),
Some(r#"{"k": []}"#),
Some(r#"{"K": "upper"}"#),
None,
];
fn want() -> Vec<Vec<String>> {
let s = |v: &[&str]| v.iter().map(|s| s.to_string()).collect::<Vec<_>>();
vec![
s(&["a"]),
s(&[]),
s(&["7"]),
s(&["a", "b"]),
s(&[]),
s(&[]),
s(&[]),
s(&[]),
s(&["line\nbreak\u{1}"]),
s(&[]),
s(&[]),
s(&[]),
]
}
#[test]
fn json_and_jsonb_rows_derive_the_same_strings() {
let values: Vec<Option<serde_json::Value>> = ROWS
.iter()
.map(|r| r.map(|r| serde_json::from_str(r).unwrap()))
.collect();
assert_eq!(
lists(&column_from_json(values.iter().map(|v| v.as_ref()), "k")),
want()
);
let text = StringArray::from(ROWS.to_vec());
assert_eq!(lists(&column_from_metadata(&text, "k").unwrap()), want());
let jsonb = lance_arrow::json::JsonArray::try_from_iter(ROWS.iter().copied())
.unwrap()
.into_inner();
assert_eq!(lists(&column_from_metadata(&jsonb, "k").unwrap()), want());
}
#[test]
fn indexed_keys_come_from_the_field_metadata() {
let mut unknown = field("_mk_2", "c", State::Ready);
unknown.set_metadata(
[
(KEY_FIELD_META.to_string(), "c".to_string()),
(STATE_FIELD_META.to_string(), "later".to_string()),
]
.into_iter()
.collect(),
);
let schema = Schema::new(vec![
Field::new("key", DataType::Utf8, false),
field("_mk_0", "a.b", State::Ready),
Field::new("_mk_1", column_type(), true),
field("_mk_3", "d", State::Building),
unknown,
]);
assert_eq!(
indexed_keys(&schema),
vec![
IndexedKey {
column: "_mk_0".into(),
key: "a.b".into(),
state: Some(State::Ready),
},
IndexedKey {
column: "_mk_3".into(),
key: "d".into(),
state: Some(State::Building),
},
IndexedKey {
column: "_mk_2".into(),
key: "c".into(),
state: None,
},
]
);
}
}
+28
View File
@@ -17,6 +17,9 @@ service VectorService {
rpc DeleteVectors(DeleteVectorsRequest) returns (DeleteVectorsResponse);
rpc ListVectors(ListVectorsRequest) returns (ListVectorsResponse);
rpc QueryVectors(QueryVectorsRequest) returns (QueryVectorsResponse);
// How this engine's queries of an index filtered on each metadata key
// over the last hour.
rpc FilterStats(FilterStatsRequest) returns (FilterStatsResponse);
}
enum DistanceMetric {
@@ -139,3 +142,28 @@ message QueryResult {
message QueryVectorsResponse {
repeated QueryResult vectors = 1;
}
message FilterStatsRequest {
// Only the id is read.
Table table = 1;
}
message FilterKeyStats {
string key = 1;
uint64 queries = 2;
// Their total latency.
double seconds = 3;
// The operand types the key was compared with.
bool string_operand = 4;
bool number_operand = 5;
bool bool_operand = 6;
}
message FilterStatsResponse {
repeated FilterKeyStats keys = 1;
// The stats cover at most this long, less when the engine started later.
uint32 window_seconds = 2;
// Keys of this index not counted because the engine tracks a bounded
// number of them.
bool truncated = 3;
}
+242 -27
View File
@@ -1077,6 +1077,200 @@ func (x *QueryVectorsResponse) GetVectors() []*QueryResult {
return nil
}
type FilterStatsRequest struct {
state protoimpl.MessageState `protogen:"open.v1"`
// Only the id is read.
Table *Table `protobuf:"bytes,1,opt,name=table,proto3" json:"table,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *FilterStatsRequest) Reset() {
*x = FilterStatsRequest{}
mi := &file_vector_proto_msgTypes[17]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *FilterStatsRequest) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*FilterStatsRequest) ProtoMessage() {}
func (x *FilterStatsRequest) ProtoReflect() protoreflect.Message {
mi := &file_vector_proto_msgTypes[17]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use FilterStatsRequest.ProtoReflect.Descriptor instead.
func (*FilterStatsRequest) Descriptor() ([]byte, []int) {
return file_vector_proto_rawDescGZIP(), []int{17}
}
func (x *FilterStatsRequest) GetTable() *Table {
if x != nil {
return x.Table
}
return nil
}
type FilterKeyStats struct {
state protoimpl.MessageState `protogen:"open.v1"`
Key string `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"`
Queries uint64 `protobuf:"varint,2,opt,name=queries,proto3" json:"queries,omitempty"`
// Their total latency.
Seconds float64 `protobuf:"fixed64,3,opt,name=seconds,proto3" json:"seconds,omitempty"`
// The operand types the key was compared with.
StringOperand bool `protobuf:"varint,4,opt,name=string_operand,json=stringOperand,proto3" json:"string_operand,omitempty"`
NumberOperand bool `protobuf:"varint,5,opt,name=number_operand,json=numberOperand,proto3" json:"number_operand,omitempty"`
BoolOperand bool `protobuf:"varint,6,opt,name=bool_operand,json=boolOperand,proto3" json:"bool_operand,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *FilterKeyStats) Reset() {
*x = FilterKeyStats{}
mi := &file_vector_proto_msgTypes[18]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *FilterKeyStats) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*FilterKeyStats) ProtoMessage() {}
func (x *FilterKeyStats) ProtoReflect() protoreflect.Message {
mi := &file_vector_proto_msgTypes[18]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use FilterKeyStats.ProtoReflect.Descriptor instead.
func (*FilterKeyStats) Descriptor() ([]byte, []int) {
return file_vector_proto_rawDescGZIP(), []int{18}
}
func (x *FilterKeyStats) GetKey() string {
if x != nil {
return x.Key
}
return ""
}
func (x *FilterKeyStats) GetQueries() uint64 {
if x != nil {
return x.Queries
}
return 0
}
func (x *FilterKeyStats) GetSeconds() float64 {
if x != nil {
return x.Seconds
}
return 0
}
func (x *FilterKeyStats) GetStringOperand() bool {
if x != nil {
return x.StringOperand
}
return false
}
func (x *FilterKeyStats) GetNumberOperand() bool {
if x != nil {
return x.NumberOperand
}
return false
}
func (x *FilterKeyStats) GetBoolOperand() bool {
if x != nil {
return x.BoolOperand
}
return false
}
type FilterStatsResponse struct {
state protoimpl.MessageState `protogen:"open.v1"`
Keys []*FilterKeyStats `protobuf:"bytes,1,rep,name=keys,proto3" json:"keys,omitempty"`
// The stats cover at most this long, less when the engine started later.
WindowSeconds uint32 `protobuf:"varint,2,opt,name=window_seconds,json=windowSeconds,proto3" json:"window_seconds,omitempty"`
// Keys of this index not counted because the engine tracks a bounded
// number of them.
Truncated bool `protobuf:"varint,3,opt,name=truncated,proto3" json:"truncated,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *FilterStatsResponse) Reset() {
*x = FilterStatsResponse{}
mi := &file_vector_proto_msgTypes[19]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *FilterStatsResponse) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*FilterStatsResponse) ProtoMessage() {}
func (x *FilterStatsResponse) ProtoReflect() protoreflect.Message {
mi := &file_vector_proto_msgTypes[19]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use FilterStatsResponse.ProtoReflect.Descriptor instead.
func (*FilterStatsResponse) Descriptor() ([]byte, []int) {
return file_vector_proto_rawDescGZIP(), []int{19}
}
func (x *FilterStatsResponse) GetKeys() []*FilterKeyStats {
if x != nil {
return x.Keys
}
return nil
}
func (x *FilterStatsResponse) GetWindowSeconds() uint32 {
if x != nil {
return x.WindowSeconds
}
return 0
}
func (x *FilterStatsResponse) GetTruncated() bool {
if x != nil {
return x.Truncated
}
return false
}
var File_vector_proto protoreflect.FileDescriptor
const file_vector_proto_rawDesc = "" +
@@ -1153,13 +1347,26 @@ const file_vector_proto_rawDesc = "" +
"\bmetadata\x18\x03 \x01(\v2\x17.google.protobuf.StructR\bmetadata\x12\x12\n" +
"\x04data\x18\x04 \x03(\x02R\x04data\"H\n" +
"\x14QueryVectorsResponse\x120\n" +
"\avectors\x18\x01 \x03(\v2\x16.vector_pb.QueryResultR\avectors*V\n" +
"\avectors\x18\x01 \x03(\v2\x16.vector_pb.QueryResultR\avectors\"<\n" +
"\x12FilterStatsRequest\x12&\n" +
"\x05table\x18\x01 \x01(\v2\x10.vector_pb.TableR\x05table\"\xc7\x01\n" +
"\x0eFilterKeyStats\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x18\n" +
"\aqueries\x18\x02 \x01(\x04R\aqueries\x12\x18\n" +
"\aseconds\x18\x03 \x01(\x01R\aseconds\x12%\n" +
"\x0estring_operand\x18\x04 \x01(\bR\rstringOperand\x12%\n" +
"\x0enumber_operand\x18\x05 \x01(\bR\rnumberOperand\x12!\n" +
"\fbool_operand\x18\x06 \x01(\bR\vboolOperand\"\x89\x01\n" +
"\x13FilterStatsResponse\x12-\n" +
"\x04keys\x18\x01 \x03(\v2\x19.vector_pb.FilterKeyStatsR\x04keys\x12%\n" +
"\x0ewindow_seconds\x18\x02 \x01(\rR\rwindowSeconds\x12\x1c\n" +
"\ttruncated\x18\x03 \x01(\bR\ttruncated*V\n" +
"\x0eDistanceMetric\x12\x1f\n" +
"\x1bDISTANCE_METRIC_UNSPECIFIED\x10\x00\x12\x06\n" +
"\x02L2\x10\x01\x12\n" +
"\n" +
"\x06COSINE\x10\x02\x12\x0f\n" +
"\vDOT_PRODUCT\x10\x032\xae\x04\n" +
"\vDOT_PRODUCT\x10\x032\xfc\x04\n" +
"\rVectorService\x12L\n" +
"\vCreateIndex\x12\x1d.vector_pb.CreateIndexRequest\x1a\x1e.vector_pb.CreateIndexResponse\x12F\n" +
"\tDropIndex\x12\x1b.vector_pb.DropIndexRequest\x1a\x1c.vector_pb.DropIndexResponse\x12I\n" +
@@ -1169,7 +1376,8 @@ const file_vector_proto_rawDesc = "" +
"GetVectors\x12\x1c.vector_pb.GetVectorsRequest\x1a\x1d.vector_pb.GetVectorsResponse\x12R\n" +
"\rDeleteVectors\x12\x1f.vector_pb.DeleteVectorsRequest\x1a .vector_pb.DeleteVectorsResponse\x12L\n" +
"\vListVectors\x12\x1d.vector_pb.ListVectorsRequest\x1a\x1e.vector_pb.ListVectorsResponse\x12O\n" +
"\fQueryVectors\x12\x1e.vector_pb.QueryVectorsRequest\x1a\x1f.vector_pb.QueryVectorsResponseB2Z0github.com/seaweedfs/seaweedfs/weed/pb/vector_pbb\x06proto3"
"\fQueryVectors\x12\x1e.vector_pb.QueryVectorsRequest\x1a\x1f.vector_pb.QueryVectorsResponse\x12L\n" +
"\vFilterStats\x12\x1d.vector_pb.FilterStatsRequest\x1a\x1e.vector_pb.FilterStatsResponseB2Z0github.com/seaweedfs/seaweedfs/weed/pb/vector_pbb\x06proto3"
var (
file_vector_proto_rawDescOnce sync.Once
@@ -1184,7 +1392,7 @@ func file_vector_proto_rawDescGZIP() []byte {
}
var file_vector_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
var file_vector_proto_msgTypes = make([]protoimpl.MessageInfo, 17)
var file_vector_proto_msgTypes = make([]protoimpl.MessageInfo, 20)
var file_vector_proto_goTypes = []any{
(DistanceMetric)(0), // 0: vector_pb.DistanceMetric
(*Table)(nil), // 1: vector_pb.Table
@@ -1204,13 +1412,16 @@ var file_vector_proto_goTypes = []any{
(*QueryVectorsRequest)(nil), // 15: vector_pb.QueryVectorsRequest
(*QueryResult)(nil), // 16: vector_pb.QueryResult
(*QueryVectorsResponse)(nil), // 17: vector_pb.QueryVectorsResponse
(*structpb.Struct)(nil), // 18: google.protobuf.Struct
(*FilterStatsRequest)(nil), // 18: vector_pb.FilterStatsRequest
(*FilterKeyStats)(nil), // 19: vector_pb.FilterKeyStats
(*FilterStatsResponse)(nil), // 20: vector_pb.FilterStatsResponse
(*structpb.Struct)(nil), // 21: google.protobuf.Struct
}
var file_vector_proto_depIdxs = []int32{
1, // 0: vector_pb.CreateIndexRequest.table:type_name -> vector_pb.Table
0, // 1: vector_pb.CreateIndexRequest.distance_metric:type_name -> vector_pb.DistanceMetric
1, // 2: vector_pb.DropIndexRequest.table:type_name -> vector_pb.Table
18, // 3: vector_pb.Vector.metadata:type_name -> google.protobuf.Struct
21, // 3: vector_pb.Vector.metadata:type_name -> google.protobuf.Struct
1, // 4: vector_pb.PutVectorsRequest.table:type_name -> vector_pb.Table
6, // 5: vector_pb.PutVectorsRequest.vectors:type_name -> vector_pb.Vector
1, // 6: vector_pb.GetVectorsRequest.table:type_name -> vector_pb.Table
@@ -1219,27 +1430,31 @@ var file_vector_proto_depIdxs = []int32{
1, // 9: vector_pb.ListVectorsRequest.table:type_name -> vector_pb.Table
6, // 10: vector_pb.ListVectorsResponse.vectors:type_name -> vector_pb.Vector
1, // 11: vector_pb.QueryVectorsRequest.table:type_name -> vector_pb.Table
18, // 12: vector_pb.QueryResult.metadata:type_name -> google.protobuf.Struct
21, // 12: vector_pb.QueryResult.metadata:type_name -> google.protobuf.Struct
16, // 13: vector_pb.QueryVectorsResponse.vectors:type_name -> vector_pb.QueryResult
2, // 14: vector_pb.VectorService.CreateIndex:input_type -> vector_pb.CreateIndexRequest
4, // 15: vector_pb.VectorService.DropIndex:input_type -> vector_pb.DropIndexRequest
7, // 16: vector_pb.VectorService.PutVectors:input_type -> vector_pb.PutVectorsRequest
9, // 17: vector_pb.VectorService.GetVectors:input_type -> vector_pb.GetVectorsRequest
11, // 18: vector_pb.VectorService.DeleteVectors:input_type -> vector_pb.DeleteVectorsRequest
13, // 19: vector_pb.VectorService.ListVectors:input_type -> vector_pb.ListVectorsRequest
15, // 20: vector_pb.VectorService.QueryVectors:input_type -> vector_pb.QueryVectorsRequest
3, // 21: vector_pb.VectorService.CreateIndex:output_type -> vector_pb.CreateIndexResponse
5, // 22: vector_pb.VectorService.DropIndex:output_type -> vector_pb.DropIndexResponse
8, // 23: vector_pb.VectorService.PutVectors:output_type -> vector_pb.PutVectorsResponse
10, // 24: vector_pb.VectorService.GetVectors:output_type -> vector_pb.GetVectorsResponse
12, // 25: vector_pb.VectorService.DeleteVectors:output_type -> vector_pb.DeleteVectorsResponse
14, // 26: vector_pb.VectorService.ListVectors:output_type -> vector_pb.ListVectorsResponse
17, // 27: vector_pb.VectorService.QueryVectors:output_type -> vector_pb.QueryVectorsResponse
21, // [21:28] is the sub-list for method output_type
14, // [14:21] is the sub-list for method input_type
14, // [14:14] is the sub-list for extension type_name
14, // [14:14] is the sub-list for extension extendee
0, // [0:14] is the sub-list for field type_name
1, // 14: vector_pb.FilterStatsRequest.table:type_name -> vector_pb.Table
19, // 15: vector_pb.FilterStatsResponse.keys:type_name -> vector_pb.FilterKeyStats
2, // 16: vector_pb.VectorService.CreateIndex:input_type -> vector_pb.CreateIndexRequest
4, // 17: vector_pb.VectorService.DropIndex:input_type -> vector_pb.DropIndexRequest
7, // 18: vector_pb.VectorService.PutVectors:input_type -> vector_pb.PutVectorsRequest
9, // 19: vector_pb.VectorService.GetVectors:input_type -> vector_pb.GetVectorsRequest
11, // 20: vector_pb.VectorService.DeleteVectors:input_type -> vector_pb.DeleteVectorsRequest
13, // 21: vector_pb.VectorService.ListVectors:input_type -> vector_pb.ListVectorsRequest
15, // 22: vector_pb.VectorService.QueryVectors:input_type -> vector_pb.QueryVectorsRequest
18, // 23: vector_pb.VectorService.FilterStats:input_type -> vector_pb.FilterStatsRequest
3, // 24: vector_pb.VectorService.CreateIndex:output_type -> vector_pb.CreateIndexResponse
5, // 25: vector_pb.VectorService.DropIndex:output_type -> vector_pb.DropIndexResponse
8, // 26: vector_pb.VectorService.PutVectors:output_type -> vector_pb.PutVectorsResponse
10, // 27: vector_pb.VectorService.GetVectors:output_type -> vector_pb.GetVectorsResponse
12, // 28: vector_pb.VectorService.DeleteVectors:output_type -> vector_pb.DeleteVectorsResponse
14, // 29: vector_pb.VectorService.ListVectors:output_type -> vector_pb.ListVectorsResponse
17, // 30: vector_pb.VectorService.QueryVectors:output_type -> vector_pb.QueryVectorsResponse
20, // 31: vector_pb.VectorService.FilterStats:output_type -> vector_pb.FilterStatsResponse
24, // [24:32] is the sub-list for method output_type
16, // [16:24] is the sub-list for method input_type
16, // [16:16] is the sub-list for extension type_name
16, // [16:16] is the sub-list for extension extendee
0, // [0:16] is the sub-list for field type_name
}
func init() { file_vector_proto_init() }
@@ -1253,7 +1468,7 @@ func file_vector_proto_init() {
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: unsafe.Slice(unsafe.StringData(file_vector_proto_rawDesc), len(file_vector_proto_rawDesc)),
NumEnums: 1,
NumMessages: 17,
NumMessages: 20,
NumExtensions: 0,
NumServices: 1,
},
+42
View File
@@ -26,6 +26,7 @@ const (
VectorService_DeleteVectors_FullMethodName = "/vector_pb.VectorService/DeleteVectors"
VectorService_ListVectors_FullMethodName = "/vector_pb.VectorService/ListVectors"
VectorService_QueryVectors_FullMethodName = "/vector_pb.VectorService/QueryVectors"
VectorService_FilterStats_FullMethodName = "/vector_pb.VectorService/FilterStats"
)
// VectorServiceClient is the client API for VectorService service.
@@ -43,6 +44,9 @@ type VectorServiceClient interface {
DeleteVectors(ctx context.Context, in *DeleteVectorsRequest, opts ...grpc.CallOption) (*DeleteVectorsResponse, error)
ListVectors(ctx context.Context, in *ListVectorsRequest, opts ...grpc.CallOption) (*ListVectorsResponse, error)
QueryVectors(ctx context.Context, in *QueryVectorsRequest, opts ...grpc.CallOption) (*QueryVectorsResponse, error)
// How this engine's queries of an index filtered on each metadata key
// over the last hour.
FilterStats(ctx context.Context, in *FilterStatsRequest, opts ...grpc.CallOption) (*FilterStatsResponse, error)
}
type vectorServiceClient struct {
@@ -123,6 +127,16 @@ func (c *vectorServiceClient) QueryVectors(ctx context.Context, in *QueryVectors
return out, nil
}
func (c *vectorServiceClient) FilterStats(ctx context.Context, in *FilterStatsRequest, opts ...grpc.CallOption) (*FilterStatsResponse, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
out := new(FilterStatsResponse)
err := c.cc.Invoke(ctx, VectorService_FilterStats_FullMethodName, in, out, cOpts...)
if err != nil {
return nil, err
}
return out, nil
}
// VectorServiceServer is the server API for VectorService service.
// All implementations must embed UnimplementedVectorServiceServer
// for forward compatibility.
@@ -138,6 +152,9 @@ type VectorServiceServer interface {
DeleteVectors(context.Context, *DeleteVectorsRequest) (*DeleteVectorsResponse, error)
ListVectors(context.Context, *ListVectorsRequest) (*ListVectorsResponse, error)
QueryVectors(context.Context, *QueryVectorsRequest) (*QueryVectorsResponse, error)
// How this engine's queries of an index filtered on each metadata key
// over the last hour.
FilterStats(context.Context, *FilterStatsRequest) (*FilterStatsResponse, error)
mustEmbedUnimplementedVectorServiceServer()
}
@@ -169,6 +186,9 @@ func (UnimplementedVectorServiceServer) ListVectors(context.Context, *ListVector
func (UnimplementedVectorServiceServer) QueryVectors(context.Context, *QueryVectorsRequest) (*QueryVectorsResponse, error) {
return nil, status.Error(codes.Unimplemented, "method QueryVectors not implemented")
}
func (UnimplementedVectorServiceServer) FilterStats(context.Context, *FilterStatsRequest) (*FilterStatsResponse, error) {
return nil, status.Error(codes.Unimplemented, "method FilterStats not implemented")
}
func (UnimplementedVectorServiceServer) mustEmbedUnimplementedVectorServiceServer() {}
func (UnimplementedVectorServiceServer) testEmbeddedByValue() {}
@@ -316,6 +336,24 @@ func _VectorService_QueryVectors_Handler(srv interface{}, ctx context.Context, d
return interceptor(ctx, in, info, handler)
}
func _VectorService_FilterStats_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(FilterStatsRequest)
if err := dec(in); err != nil {
return nil, err
}
if interceptor == nil {
return srv.(VectorServiceServer).FilterStats(ctx, in)
}
info := &grpc.UnaryServerInfo{
Server: srv,
FullMethod: VectorService_FilterStats_FullMethodName,
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
return srv.(VectorServiceServer).FilterStats(ctx, req.(*FilterStatsRequest))
}
return interceptor(ctx, in, info, handler)
}
// VectorService_ServiceDesc is the grpc.ServiceDesc for VectorService service.
// It's only intended for direct use with grpc.RegisterService,
// and not to be introspected or modified (even as a copy)
@@ -351,6 +389,10 @@ var VectorService_ServiceDesc = grpc.ServiceDesc{
MethodName: "QueryVectors",
Handler: _VectorService_QueryVectors_Handler,
},
{
MethodName: "FilterStats",
Handler: _VectorService_FilterStats_Handler,
},
},
Streams: []grpc.StreamDesc{},
Metadata: "vector.proto",
+21
View File
@@ -326,6 +326,27 @@ func (d *vectorDiscovery) QueryVectors(ctx context.Context, in *vector_pb.QueryV
return nil, err
}
func (d *vectorDiscovery) FilterStats(ctx context.Context, in *vector_pb.FilterStatsRequest, opts ...grpc.CallOption) (resp *vector_pb.FilterStatsResponse, err error) {
for attempt := 0; attempt < 2; attempt++ {
var c vector_pb.VectorServiceClient
if c, err = d.cli(ctx); err != nil {
return nil, err
}
resp, err = c.FilterStats(ctx, in, opts...)
if isTransportError(err) {
d.reset(c)
}
d.release(c)
if err == nil {
return resp, nil
}
if !isTransportError(err) {
return nil, err
}
}
return nil, err
}
func discoverVectorServers(masters []pb.ServerAddress, grpcDialOption grpc.DialOption, filerGroup string) []string {
for _, master := range masters {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)