Commit Graph
9 Commits
Author SHA1 Message Date
Chris Lu fbabfbd353 docs(parquet-design): add residency/cost/security sections and wording fixes
- Add Index Residency and Failure Model section: filer-managed vs
  colocated vs index-serving-tier options, and graceful-degradation
  behavior when an index is missing or stale.
- Add Cost Model section spelling out when connectors should skip the
  pushdown API (small tables, low selectivity, no predicate).
- Add Security section noting that zone maps, blooms, and inverted
  indexes leak information beyond the underlying object's ACL, and
  scoping v1 to inherit the file's ACL with row-level security
  explicitly out of scope.
- Wording fixes: clarify CONTAINS is engine-side translation of
  LIKE/MATCH/array_contains; clarify the embedding column is in
  Parquet but the vector index is a side file; flag local filtering
  on volume servers as new infrastructure introduced by this design.
2026-04-25 01:24:01 -07:00
Chris Lu 605907f522 docs(parquet-design): clarify Phase 1 wording and side-index placement
- Phase 1 "row group stats cache" was misleading — those stats already
  live in the Parquet footer. The Phase 1 win is a parsed-footer cache,
  not new index data.
- The previous "/table/_seaweed_index/" path nested side indexes under
  the table prefix, which Iceberg orphan-file removal and snapshot
  expiration would surface or attempt to clean. Move side indexes to a
  separate system bucket / filer mount keyed by file identity, with the
  in-table-prefix layout listed only as a fallback.
2026-04-25 01:22:31 -07:00
Chris Lu afeb82ab87 docs(parquet-design): add filtered-ANN strategy for hybrid pushdown
The hybrid scalar+vector flow assumed the vector index could accept
arbitrary scalar pre-filters, which most off-the-shelf IVF/HNSW
implementations do not. Document the realistic strategies (partitioned
IVF for v1, post-filter with overscan as fallback, filtered HNSW
deferred) and the planner heuristic for choosing between them.
2026-04-25 01:21:58 -07:00
Chris Lu e31cb80812 docs(parquet-design): make row-id response optional and bounded
A predicate matching millions of rows would force the server to return
millions of row refs. Default the response to range-oriented data
(file/row-group/page refs), require RequestRowIds opt-in for per-row
output, and cap returned ids via MaxRowIds with a Truncated flag.
2026-04-25 01:21:32 -07:00
Chris Lu 28c834c76f docs(parquet-design): specify predicate wire format and enum metrics
Replace opaque Predicate []byte with a tagged (PredicateKind, bytes)
pair, naming Substrait as the canonical v1 format and Iceberg
Expression JSON as a convenience. Replace the free-form Metric string
on VectorQuery with a VectorMetric enum.
2026-04-25 01:21:11 -07:00
Chris Lu 51a209cf60 docs(parquet-design): split position vs equality Iceberg deletes
Position deletes are file-scoped and precomputable as a per-data-file
bitmap. Equality deletes are predicate-scoped, can apply to many data
files including ones added later, and must be evaluated at query time.
Document both flows separately and remove the implication that
equality-delete state can be flattened into a static bitmap.
2026-04-25 01:20:53 -07:00
Chris Lu b114b318ae docs(parquet-design): tighten index identity
Drop modification time from the identity tuple — it is not stable across
replication and metadata-only operations and is unnecessary alongside
ETag or Iceberg manifest fields. Order remaining fields by strength,
with Iceberg manifest identity preferred when available.
2026-04-25 01:20:21 -07:00
Chris Lu 9ecd455310 docs(parquet-design): reframe internal mapping as logical-only
The previous wording suggested SeaweedFS would split a Parquet file
into per-column-chunk needles, which contradicts the design's
"do not modify the file" goal. Clarify that the per-column-chunk view
is a planning abstraction over byte ranges in the unchanged object.
2026-04-25 01:20:02 -07:00
Chris Lu ca4c8d977d docs: add SeaweedFS-aware Parquet pushdown design draft
Draft design note for keeping Parquet files compatible with existing
engines (Spark, Trino, DuckDB, PyArrow, Iceberg) while building
auxiliary side indexes and a pushdown API for SeaweedFS-aware clients.
Covers physical layout, side index types (footer cache, zone map,
page index, bloom, bitmap, B-tree, inverted, vector), pushdown API
sketch, Iceberg delete handling, and a phased rollout plan.
2026-04-25 01:15:42 -07:00