Github-Mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-20 13:30:46 +02:00
Code Issues Packages Projects Releases Wiki Activity

docs(parquet-design): add filtered-ANN strategy for hybrid pushdown

Browse Source 
The hybrid scalar+vector flow assumed the vector index could accept
arbitrary scalar pre-filters, which most off-the-shelf IVF/HNSW
implementations do not. Document the realistic strategies (partitioned
IVF for v1, post-filter with overscan as fallback, filtered HNSW
deferred) and the planner heuristic for choosing between them.
This commit is contained in:
Chris Lu
2026-04-25 01:21:58 -07:00
parent e31cb80812
commit afeb82ab87
1 changed files with 10 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PARQUET_PUSHDOWN_DESIGN.md
+10
View File
@@ -393,6 +393,16 @@ Execution:
6. projected columns are fetched
```
#### Filtered-ANN strategy
Step 3 hides a real engineering choice: most off-the-shelf IVF and HNSW indexes do not natively accept arbitrary scalar pre-filters. Three common strategies, in order of v1 preference:
- **Partitioned IVF (preferred for v1).** Build separate IVF indexes per partition key (e.g. one per `tenant_id`). Scalar filters that align with the partition key reduce search to a single partition; non-aligned filters fall back to post-filter.
- **Post-filter with overscan.** Search the full vector index for `k * overscan_factor` candidates, then apply scalar predicates and trim to `k`. Works for any scalar predicate but accuracy degrades when the scalar predicate is highly selective.
- **Filtered HNSW (e.g. ACORN-style).** Inline scalar predicate evaluation during graph traversal. Most flexible, but requires a custom index and is deferred past v1.
The pushdown planner picks per query: if the scalar predicate is aligned with a partitioned index, use strategy 1; otherwise fall back to strategy 2 with an overscan factor derived from estimated selectivity.
## API Sketch
### Pushdown Request