From afeb82ab8765839c84da3921f6f00d0036b141df Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 25 Apr 2026 01:21:58 -0700 Subject: [PATCH] docs(parquet-design): add filtered-ANN strategy for hybrid pushdown The hybrid scalar+vector flow assumed the vector index could accept arbitrary scalar pre-filters, which most off-the-shelf IVF/HNSW implementations do not. Document the realistic strategies (partitioned IVF for v1, post-filter with overscan as fallback, filtered HNSW deferred) and the planner heuristic for choosing between them. --- PARQUET_PUSHDOWN_DESIGN.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/PARQUET_PUSHDOWN_DESIGN.md b/PARQUET_PUSHDOWN_DESIGN.md index 92999ceb5..0bd64c701 100644 --- a/PARQUET_PUSHDOWN_DESIGN.md +++ b/PARQUET_PUSHDOWN_DESIGN.md @@ -393,6 +393,16 @@ Execution: 6. projected columns are fetched ``` +#### Filtered-ANN strategy + +Step 3 hides a real engineering choice: most off-the-shelf IVF and HNSW indexes do not natively accept arbitrary scalar pre-filters. Three common strategies, in order of v1 preference: + +- **Partitioned IVF (preferred for v1).** Build separate IVF indexes per partition key (e.g. one per `tenant_id`). Scalar filters that align with the partition key reduce search to a single partition; non-aligned filters fall back to post-filter. +- **Post-filter with overscan.** Search the full vector index for `k * overscan_factor` candidates, then apply scalar predicates and trim to `k`. Works for any scalar predicate but accuracy degrades when the scalar predicate is highly selective. +- **Filtered HNSW (e.g. ACORN-style).** Inline scalar predicate evaluation during graph traversal. Most flexible, but requires a custom index and is deferred past v1. + +The pushdown planner picks per query: if the scalar predicate is aligned with a partitioned index, use strategy 1; otherwise fall back to strategy 2 with an overscan factor derived from estimated selectivity. + ## API Sketch ### Pushdown Request