- Add Index Residency and Failure Model section: filer-managed vs
colocated vs index-serving-tier options, and graceful-degradation
behavior when an index is missing or stale.
- Add Cost Model section spelling out when connectors should skip the
pushdown API (small tables, low selectivity, no predicate).
- Add Security section noting that zone maps, blooms, and inverted
indexes leak information beyond the underlying object's ACL, and
scoping v1 to inherit the file's ACL with row-level security
explicitly out of scope.
- Wording fixes: clarify CONTAINS is engine-side translation of
LIKE/MATCH/array_contains; clarify the embedding column is in
Parquet but the vector index is a side file; flag local filtering
on volume servers as new infrastructure introduced by this design.
- Phase 1 "row group stats cache" was misleading — those stats already
live in the Parquet footer. The Phase 1 win is a parsed-footer cache,
not new index data.
- The previous "/table/_seaweed_index/" path nested side indexes under
the table prefix, which Iceberg orphan-file removal and snapshot
expiration would surface or attempt to clean. Move side indexes to a
separate system bucket / filer mount keyed by file identity, with the
in-table-prefix layout listed only as a fallback.
The hybrid scalar+vector flow assumed the vector index could accept
arbitrary scalar pre-filters, which most off-the-shelf IVF/HNSW
implementations do not. Document the realistic strategies (partitioned
IVF for v1, post-filter with overscan as fallback, filtered HNSW
deferred) and the planner heuristic for choosing between them.
A predicate matching millions of rows would force the server to return
millions of row refs. Default the response to range-oriented data
(file/row-group/page refs), require RequestRowIds opt-in for per-row
output, and cap returned ids via MaxRowIds with a Truncated flag.
Replace opaque Predicate []byte with a tagged (PredicateKind, bytes)
pair, naming Substrait as the canonical v1 format and Iceberg
Expression JSON as a convenience. Replace the free-form Metric string
on VectorQuery with a VectorMetric enum.
Position deletes are file-scoped and precomputable as a per-data-file
bitmap. Equality deletes are predicate-scoped, can apply to many data
files including ones added later, and must be evaluated at query time.
Document both flows separately and remove the implication that
equality-delete state can be flattened into a static bitmap.
Drop modification time from the identity tuple — it is not stable across
replication and metadata-only operations and is unnecessary alongside
ETag or Iceberg manifest fields. Order remaining fields by strength,
with Iceberg manifest identity preferred when available.
The previous wording suggested SeaweedFS would split a Parquet file
into per-column-chunk needles, which contradicts the design's
"do not modify the file" goal. Clarify that the per-column-chunk view
is a planning abstraction over byte ranges in the unchanged object.
Draft design note for keeping Parquet files compatible with existing
engines (Spark, Trino, DuckDB, PyArrow, Iceberg) while building
auxiliary side indexes and a pushdown API for SeaweedFS-aware clients.
Covers physical layout, side index types (footer cache, zone map,
page index, bloom, bitmap, B-tree, inverted, vector), pushdown API
sketch, Iceberg delete handling, and a phased rollout plan.