diff --git a/PARQUET_PUSHDOWN_DEV_PLAN.md b/PARQUET_PUSHDOWN_DEV_PLAN.md index 633f477e42..62a6676df3 100644 --- a/PARQUET_PUSHDOWN_DEV_PLAN.md +++ b/PARQUET_PUSHDOWN_DEV_PLAN.md @@ -34,7 +34,7 @@ These are commitments unless an open question below changes them. - **Wire protocol: gRPC.** New service definition `parquet_pushdown_pb.ParquetPushdown` in `weed/pb/parquet_pushdown_pb/`. REST/HTTP shim only if a connector requires it. JSON variant deferred. - **Trust mode for v1: catalog-validated, with connector-trusted as a developer-only flag.** The default deployment validates every request's `DataFiles` and `Deletes` against the Iceberg catalog at the requested snapshot before serving (full rules in [Trust Model](./PARQUET_PUSHDOWN_DESIGN.md#trust-model-and-catalog-validation)). Connector-trusted is supported but only enabled by an explicit flag for local development and benchmarks; it is not the default and is rejected in production builds. Manifest-signed is not on the roadmap. - **Predicate engine for v1: built-in subset evaluator.** The wire format accepts Substrait `ExtendedExpression`, but v1 implements only: comparisons (`=, !=, <, <=, >, >=`), `IN`, `BETWEEN`, `IS NULL`, and boolean `AND/OR/NOT` over them. Anything outside this subset returns `unsupported predicate`, and the connector falls back to a standard scan. Full Substrait evaluation lands in Phase 2. -- **Side-index physical location:** under a separate filer namespace `/__pushdown__//...`, governed by the same filer ACLs. Out of any user-visible bucket prefix. Path layout matches the design's `//...//` shape. +- **Side-index physical location:** co-located with the data in a hidden `.index/` directory next to each Parquet file or its parent folder, governed by the same filer ACLs as the data. Path layout matches the design's `/.index///...` (file-scoped) and `/.index//...` (folder-scoped) shapes — see [Physical Layout](./PARQUET_PUSHDOWN_DESIGN.md#physical-layout). The hidden-prefix name is configurable for deployments whose readers do not respect dot-prefix conventions. - **Parquet library: `parquet-go/parquet-go`** (already in use elsewhere in the repo). Apache `arrow-go/v18/parquet` is *not* introduced. - **Bitmap library: `RoaringBitmap/roaring`.** Used for position-delete bitmaps and DV decoding.