docs(parquet-design): align plan side-index path with .index co-location

Mirror the design's switch from a separate-namespace path to a
co-located hidden .index/ directory next to the Parquet data, so the
plan and the design agree on file-scoped and folder-scoped layouts
and on the configurable-hidden-prefix fallback.
This commit is contained in:
Chris Lu committed 2026-04-25 13:27:18 -07:00
1 parent 559439bb9d
commit 929c33980f
1 file changed
+1 -1
+1 -1
View File
@@ -34,7 +34,7 @@ These are commitments unless an open question below changes them.
- **Wire protocol: gRPC.** New service definition `parquet_pushdown_pb.ParquetPushdown` in `weed/pb/parquet_pushdown_pb/`. REST/HTTP shim only if a connector requires it. JSON variant deferred.
- **Trust mode for v1: catalog-validated, with connector-trusted as a developer-only flag.** The default deployment validates every request's `DataFiles` and `Deletes` against the Iceberg catalog at the requested snapshot before serving (full rules in [Trust Model](./PARQUET_PUSHDOWN_DESIGN.md#trust-model-and-catalog-validation)). Connector-trusted is supported but only enabled by an explicit flag for local development and benchmarks; it is not the default and is rejected in production builds. Manifest-signed is not on the roadmap.
- **Predicate engine for v1: built-in subset evaluator.** The wire format accepts Substrait `ExtendedExpression`, but v1 implements only: comparisons (`=, !=, <, <=, >, >=`), `IN`, `BETWEEN`, `IS NULL`, and boolean `AND/OR/NOT` over them. Anything outside this subset returns `unsupported predicate`, and the connector falls back to a standard scan. Full Substrait evaluation lands in Phase 2.
- **Side-index physical location:** under a separate filer namespace `/__pushdown__/<table_uuid>/...`, governed by the same filer ACLs. Out of any user-visible bucket prefix. Path layout matches the design's `<system-prefix>/<table_uuid>/.../<identity>/` shape.
- **Side-index physical location:** co-located with the data in a hidden `.index/` directory next to each Parquet file or its parent folder, governed by the same filer ACLs as the data. Path layout matches the design's `<parquet_parent>/.index/<file_name>/<identity>/...` (file-scoped) and `<parquet_parent>/.index/<identity>/...` (folder-scoped) shapes — see [Physical Layout](./PARQUET_PUSHDOWN_DESIGN.md#physical-layout). The hidden-prefix name is configurable for deployments whose readers do not respect dot-prefix conventions.
- **Parquet library: `parquet-go/parquet-go`** (already in use elsewhere in the repo). Apache `arrow-go/v18/parquet` is *not* introduced.
- **Bitmap library: `RoaringBitmap/roaring`.** Used for position-delete bitmaps and DV decoding.