From 929c33980f203a97893f3cc2721c96eb1d9f9e05 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 25 Apr 2026 13:27:18 -0700 Subject: [PATCH] docs(parquet-design): align plan side-index path with .index co-location Mirror the design's switch from a separate-namespace path to a co-located hidden .index/ directory next to the Parquet data, so the plan and the design agree on file-scoped and folder-scoped layouts and on the configurable-hidden-prefix fallback. --- PARQUET_PUSHDOWN_DEV_PLAN.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/PARQUET_PUSHDOWN_DEV_PLAN.md b/PARQUET_PUSHDOWN_DEV_PLAN.md index 633f477e42..62a6676df3 100644 --- a/PARQUET_PUSHDOWN_DEV_PLAN.md +++ b/PARQUET_PUSHDOWN_DEV_PLAN.md @@ -34,7 +34,7 @@ These are commitments unless an open question below changes them. - **Wire protocol: gRPC.** New service definition `parquet_pushdown_pb.ParquetPushdown` in `weed/pb/parquet_pushdown_pb/`. REST/HTTP shim only if a connector requires it. JSON variant deferred. - **Trust mode for v1: catalog-validated, with connector-trusted as a developer-only flag.** The default deployment validates every request's `DataFiles` and `Deletes` against the Iceberg catalog at the requested snapshot before serving (full rules in [Trust Model](./PARQUET_PUSHDOWN_DESIGN.md#trust-model-and-catalog-validation)). Connector-trusted is supported but only enabled by an explicit flag for local development and benchmarks; it is not the default and is rejected in production builds. Manifest-signed is not on the roadmap. - **Predicate engine for v1: built-in subset evaluator.** The wire format accepts Substrait `ExtendedExpression`, but v1 implements only: comparisons (`=, !=, <, <=, >, >=`), `IN`, `BETWEEN`, `IS NULL`, and boolean `AND/OR/NOT` over them. Anything outside this subset returns `unsupported predicate`, and the connector falls back to a standard scan. Full Substrait evaluation lands in Phase 2. -- **Side-index physical location:** under a separate filer namespace `/__pushdown__//...`, governed by the same filer ACLs. Out of any user-visible bucket prefix. Path layout matches the design's `//...//` shape. +- **Side-index physical location:** co-located with the data in a hidden `.index/` directory next to each Parquet file or its parent folder, governed by the same filer ACLs as the data. Path layout matches the design's `/.index///...` (file-scoped) and `/.index//...` (folder-scoped) shapes — see [Physical Layout](./PARQUET_PUSHDOWN_DESIGN.md#physical-layout). The hidden-prefix name is configurable for deployments whose readers do not respect dot-prefix conventions. - **Parquet library: `parquet-go/parquet-go`** (already in use elsewhere in the repo). Apache `arrow-go/v18/parquet` is *not* introduced. - **Bitmap library: `RoaringBitmap/roaring`.** Used for position-delete bitmaps and DV decoding.