seaweedfs/PARQUET_PUSHDOWN_DESIGN.md

SeaweedFS-Aware Parquet Layout and Pushdown Design

Status

Draft design note.

Goal

Keep Parquet files fully compatible with existing engines such as Spark, Trino, DuckDB, PyArrow, and Iceberg, while allowing SeaweedFS-aware clients to take advantage of storage-side metadata, indexes, and pushdown execution.

The design avoids replacing Parquet with a custom format. Instead, SeaweedFS stores normal Parquet objects and builds auxiliary metadata/indexes beside them.

Standard Parquet compatibility
+ SeaweedFS side indexes
+ pushdown-aware connectors
= faster scans, filtering, and hybrid vector search

Non-Goals

• Do not require changes to the Parquet file format.
• Do not require existing Spark/Trino/DuckDB readers to change.
• Do not make SeaweedFS the only way to read the data.
• Do not rewrite all Parquet data into Lance or another vector-native format.

Background

Parquet already contains useful metadata:

• file footer
• row group metadata
• column chunk offsets
• column statistics
• optional page indexes
• optional bloom filters

Typical readers use this flow:

1. Read Parquet footer
2. Discover row groups and column chunks
3. Prune row groups using statistics
4. Read selected column chunks/pages
5. Decode/filter/project rows

SeaweedFS can optimize this process by caching metadata, exposing smarter range reads, and building additional side indexes.

High-Level Architecture

Pushdown is split into two tiers, mapping cleanly onto roles SeaweedFS already has:

Spark / Trino / DuckDB / PyArrow / SeaweedFS-aware connector
            |
            | Pushdown API (planning)
            v
+-----------------------------+
|  Catalog Server  (brain)    |
|  - table discovery          |
|  - Iceberg snapshot/version |
|  - Parquet file list        |
|  - index metadata registry  |
|  - index freshness tracking |
|  - planning + pruning       |
+-----------------------------+
            |
            | candidate files / row groups / ranges
            | + side-index pointers
            v
+-----------------------------+        +-----------------+
|  Volume Servers  (hands)    |  <---  |  S3 Gateway     |
|  - range reads              |        |  (compat path)  |
|  - local index lookup       |        |  for unmodified |
|  - vector distance compute  |        |  S3 readers     |
|  - local top-K              |        +-----------------+
+-----------------------------+
            |
            v
        Parquet data needles

• Catalog Server = brain. Existing role: holds table metadata, resolves Iceberg snapshots, maps tables to data files. Pushdown adds: an index registry (which side indexes exist for which files at which identities), a planner that turns a request into candidate files/row groups/ranges/index pointers, and index freshness state. The catalog server does not read Parquet payload bytes; it returns plans, not rows.
• Volume Servers = hands. Existing role: serve byte ranges from needles. Pushdown adds: local side-index lookup, scalar predicate evaluation against indexes, vector distance compute, and local top-K so the volume server returns small result sets instead of raw bytes for hot paths.
• S3 Gateway = compatibility path. Unmodified S3 readers continue to work via plain ranged GETs against the gateway. The gateway never participates in pushdown planning.

What deliberately stays out of the catalog server: large scans, vector ANN computation, side-index building over big files, heavy bitmap merges. Those run on the volume servers next to the data, so the planner stays cheap and one busy table cannot bottleneck planning across the cluster.

There are two access modes:

1. Standard Object Access

Existing engines read Parquet through the normal S3-compatible API. The S3 gateway never sees a pushdown call.

engine -> SeaweedFS S3 -> Parquet object ranges

This preserves compatibility.

2. SeaweedFS-Aware Pushdown Access

Optimized connectors call the catalog server's pushdown API before reading data.

engine -> SeaweedFS-aware connector -> Catalog Server pushdown API
       -> receive candidate ranges + (for hot paths) volume-server endpoints
       -> execute remaining work on volume servers OR read Parquet ranges directly

The catalog server's response always includes byte ranges the connector can fetch through the S3 path, so a connector that does not want to chat with volume servers still gets the file/row-group/page pruning win. Volume-server-side execution is opt-in per request and used for vector ANN, large bitmap unions, and other workloads where the data-locality win outweighs the extra hop.

Physical Layout

The user-visible layout remains normal Iceberg/Parquet layout:

/table/
  metadata/
    v1.metadata.json
    snapshots/...
  data/
    ds=2026-01-01/
      part-00001.parquet
      part-00002.parquet

SeaweedFS stores side indexes in a hidden .index/ directory co-located with the Parquet files they describe. Iceberg, Spark, and Hadoop ecosystems treat dot-prefixed paths as hidden by convention — they are skipped by table listings, snapshot expiration, and orphan-file cleanup — so co-location does not interfere with normal table operations while keeping each index next to the data it indexes (the same backup/replication policy applies; lifecycle is automatic).

Index scope determines whether .index/ lives next to the file or next to the folder:

• File-scoped indexes (footer cache, per-file column indexes, per-file bloom filters, per-file deletion bitmaps): under .index/<file_name>/<identity>/... next to the data file's parent.
• Folder-scoped indexes (table-wide or partition-wide indexes that span multiple files, e.g. a single B-tree covering all files in a partition): under .index/<identity>/... directly inside .index/, with the index payload naming the files it indexes.

Logical layout for a partition:

/table/data/ds=2026-01-01/
  part-00001.parquet
  part-00002.parquet
  .index/
    part-00001.parquet/<identity>/
      footer.cache
      page_index.fid_3      # column "timestamp", field id 3
      bloom.fid_5           # column "user_id", field id 5
      bitmap.fid_7          # column "tenant_id", field id 7
      btree.fid_3           # column "timestamp", field id 3
      inverted.fid_11       # column "message", field id 11
      vector.fid_42.ivf     # column "embedding", field id 42
    part-00002.parquet/<identity>/
      ...
    <identity>/             # folder-scoped, optional
      partition_btree.fid_3

Per-column index files are keyed by Iceberg field ID, not column name. A rename (e.g. tenant_id → org_id) does not invalidate the index, and a column dropped-and-re-added with the same name (a different field ID) does not silently reuse the wrong index. Side-index metadata may carry the human-readable name as a hint for logging. <identity> is derived from the index identity rules in Index Consistency. The original Parquet file is not modified.

For tables that are read by tools that do not respect the dot-prefix convention, deployments can override .index/ to a configurable hidden-prefix name or relocate indexes to a separate filer mount; the design otherwise assumes dot-prefix is hidden.

Logical View for Planning

The Parquet object on disk is unchanged. For planning and pushdown purposes, SeaweedFS exposes a logical view of the file derived from the footer:

Parquet file
 ├── footer metadata
 ├── row group 0
 │    ├── column chunk: id
 │    ├── column chunk: timestamp
 │    ├── column chunk: tenant_id
 │    └── column chunk: payload
 └── row group 1
      ├── column chunk: id
      ├── column chunk: timestamp
      ├── column chunk: tenant_id
      └── column chunk: payload

Each leaf maps to a (file, offset, length) byte range inside the original object:

footer                          -> (file, footer_offset, footer_length)
row_group_0 / column_id         -> (file, offset, length)
row_group_0 / column_timestamp  -> (file, offset, length)
row_group_0 / column_tenant_id  -> (file, offset, length)
row_group_0 / column_payload    -> (file, offset, length)
row_group_1 / column_id         -> (file, offset, length)
...

This is a planning abstraction only. The Parquet file is not split into per-column-chunk needles, and the on-disk object is not modified. Reads are still issued as ranged GETs against the original object.

Side Index Types

1. Footer Cache

Cache the parsed Parquet footer.

Purpose:

• avoid repeated footer reads
• speed up planning
• expose row group and column chunk offsets quickly

Index content:

file path
file size
last modified / etag
schema
row groups
column chunks
statistics
byte offsets

2. Row Group Zone Map

Stores min/max/null count per row group and column.

Useful for predicates such as:

WHERE timestamp BETWEEN t1 AND t2
WHERE id > 1000

Skip condition:

row_group.max < predicate_min OR row_group.min > predicate_max

3. Page-Level Index

Wraps Parquet's native ColumnIndex (per-page min/max/null-count) and OffsetIndex (per-page byte offset, length, and first_row_index within the row group). The side index materializes both for fast access without re-parsing the footer.

Useful when row groups are large and predicates are selective.

Predicate evaluation example:

row_group_3 / column_timestamp:
  page_0 first_row=0       min=2026-01-01 max=2026-01-02
  page_1 first_row=120000  min=2026-01-03 max=2026-01-04
  page_2 first_row=240000  min=2026-01-05 max=2026-01-06

Page pruning is row-range-driven, not byte-range-driven. The flow:

1. Apply the predicate against the predicate column's ColumnIndex to pick surviving pages.
2. Translate surviving pages to row ranges using first_row_index (e.g. pages 1–2 above → row range [120000, 360000) within row group 3).
3. For each projected column, walk that column's OffsetIndex and select the pages whose [first_row_index, next_first_row_index) intersects the surviving row ranges. Page boundaries do not align across columns within a row group, so this translation step is mandatory; reusing the predicate column's page numbers as if they were column-independent is wrong.
4. Return per-column page byte ranges in the response.

Two Parquet-specific pitfalls the index must handle:

• Dictionary pages. A column chunk may begin with a dictionary page that is not described by OffsetIndex (which lists data pages only). The dictionary page must still be fetched whenever any data page from that column chunk is read, so the byte range returned for a surviving page set must include the dictionary page region of the chunk.
• Repeated/nested fields. For columns where one Parquet record spans multiple values (lists, maps), first_row_index refers to row positions in the row group, not value positions; row-range translation across columns still works because rows are aligned per row group, but downstream consumers must remember that data-page row counts and value counts are not the same.

If the file was written without ColumnIndex/OffsetIndex, page-level pruning is unavailable and the planner falls back to row-group pruning.

4. Bloom Filter Index

Useful for equality lookups.

Example:

WHERE user_id = 'u123'

If the bloom filter says the value is absent, SeaweedFS skips the row group or page.

5. Bitmap Index

Useful for low- or medium-cardinality columns.

Examples:

WHERE tenant_id = 'abc'
WHERE status IN ('failed', 'pending')

Bitmap operations:

tenant_id=abc AND status=failed

6. B-Tree / Range Index

Useful for range predicates and point lookups on sortable columns.

Examples:

WHERE timestamp > t
WHERE id BETWEEN a AND b

The index maps values or ranges to:

file
row group
page
optional row ids

7. Inverted Index

Useful for logs, text, tags, and JSON fields.

Examples (engine-side translations, since standard SQL has no CONTAINS):

-- substring search:    LIKE '%timeout%' or full-text MATCH
WHERE message LIKE '%timeout%'
-- array element match: array_contains(tags, 'gpu') / 'gpu' = ANY(tags)
WHERE array_contains(tags, 'gpu')

Index structure:

token -> candidate row groups/pages/row ids

8. Vector Index

The embedding column is stored as a regular column inside the Parquet file. The vector index itself is a side file built from that column and stored alongside the other side indexes.

Example:

ORDER BY embedding <-> query_vector
LIMIT 20

Possible index types:

• IVF
• IVF+PQ
• HNSW
• scalar quantization

Index layout:

<parquet_parent>/.index/<file_name>/<identity>/vector.fid_42.ivf/
  centroids
  list_000001
  list_000002
  pq_codebooks

This allows SeaweedFS to perform approximate nearest neighbor search before reading Parquet columns.

Pushdown Types

File Pruning

Use Iceberg metadata plus SeaweedFS indexes to skip files.

Row Group Pruning

Use row group statistics, bloom filters, and side indexes.

Page Pruning

Use page-level indexes to read only matching pages.

Connector consumability

Returning []PageRef as (file, row_group, column, page, offset, length) is only useful to readers that can issue page-granular I/O on a single Parquet file. That includes parquet-go's ColumnChunkReader.ReadPage, Arrow C++/PyArrow with parquet::PageReader, and DuckDB's internal Parquet reader; it does not include a stock S3-aware Parquet client that only supports row-group-level reads. Two consumption modes are supported:

• Page-aware reader (preferred when available). The connector issues range reads matching the returned page byte ranges plus the column chunk's dictionary-page region (already merged into the response per Page-Level Index). It decodes only the surviving pages.
• Row-group reader with row-selection. Connectors without page-aware I/O ignore the Pages list and consume RowGroups instead. They then apply the same predicate locally, or call back for a row-selection vector (the planner can emit one — see below — keyed on row-group-local row position) to filter post-decode without re-evaluating the predicate.

When the response carries Pages but the connector cannot consume them, it must drop to RowGroups granularity rather than reinterpret page byte ranges as row-group ranges. The page byte ranges do not align to row-group boundaries and are not safe to feed to a row-group-only reader.

Future revision (post-M6): the response may add an optional row-selection bitmap per row group so a row-group reader can apply page-level pruning without a page-aware decoder. v1 returns Pages only; downstream consumers fall back as described above.

Column Pruning

Read only requested columns and predicate columns.

Example:

SELECT id, image_path
WHERE tenant_id = 'abc'
ORDER BY embedding <-> ?
LIMIT 10

Only these columns are needed:

tenant_id
embedding
id
image_path

Scalar Predicate Pushdown

Evaluate simple predicates using indexes:

=, !=, <, <=, >, >=
IN
BETWEEN
IS NULL
CONTAINS

Vector Search Pushdown

Search vector side indexes on volume servers or index-serving nodes.

Flow:

1. Choose candidate vector partitions
2. Route to volume servers holding vector index/data needles
3. Compute local top-K
4. Merge global top-K
5. Fetch projected Parquet columns

Hybrid Scalar + Vector Pushdown

Example:

SELECT image_path
WHERE tenant_id = 'abc'
  AND timestamp >= '2026-01-01'
ORDER BY embedding <-> query_vector
LIMIT 20

Execution:

1. tenant_id bitmap filters candidate rows/pages
2. timestamp zone map or B-tree narrows candidates
3. vector index searches only candidate partitions
4. delete masks are applied if Iceberg delete files exist
5. top-K candidates are returned
6. projected columns are fetched

Filtered-ANN strategy

Step 3 hides a real engineering choice: most off-the-shelf IVF and HNSW indexes do not natively accept arbitrary scalar pre-filters. Three common strategies, in order of v1 preference:

• Partitioned IVF (preferred for v1). Build separate IVF indexes per partition key (e.g. one per tenant_id). Scalar filters that align with the partition key reduce search to a single partition; non-aligned filters fall back to post-filter.
• Post-filter with overscan. Search the full vector index for k * overscan_factor candidates, then apply scalar predicates and trim to k. Works for any scalar predicate but accuracy degrades when the scalar predicate is highly selective.
• Filtered HNSW (e.g. ACORN-style). Inline scalar predicate evaluation during graph traversal. Most flexible, but requires a custom index and is deferred past v1.

The pushdown planner picks per query: if the scalar predicate is aligned with a partitioned index, use strategy 1; otherwise fall back to strategy 2 with an overscan factor derived from estimated selectivity.

API Sketch

Pushdown Request

type ParquetPushdownRequest struct {
    Table          string
    SnapshotId     int64

    // SchemaId is the Iceberg schema id the predicate is bound to —
    // the schema the connector used when resolving column-name
    // references to field IDs. The server confirms it matches the
    // snapshot's current_schema_id (or a known historical schema)
    // before evaluating the predicate. If the predicate refers to
    // field IDs that are not present in this schema, the request is
    // rejected.
    SchemaId int32

    // DataFiles is the authoritative list of files to scan. Each entry
    // carries enough identity for the server to validate that its cached
    // side indexes still apply, and enough delete-file context that the
    // server can compute the correct visible row set without re-running
    // Iceberg planning.
    DataFiles      []DataFileDescriptor

    Columns        []ColumnRef
    PredicateKind  PredicateKind // SUBSTRAIT or ICEBERG_EXPRESSION

    // Predicate is a *bound* expression: every column reference must
    // resolve to an Iceberg field id (Substrait field-reference IDs
    // matching this request's SchemaId, or Iceberg Expression JSON
    // with id-based references). Name-only references — the kind a
    // SQL parser emits — are rejected; the connector is responsible
    // for binding before sending. This avoids name-resolution
    // ambiguity under schema evolution and removes a server-side
    // dependency on the catalog's symbol table.
    Predicate      []byte

    VectorQuery    *VectorQuery
    Limit          int
    RequestRowIds  bool          // include per-row refs in response (default false)
    MaxRowIds      int           // cap on returned row refs; server may truncate
}

type DataFileDescriptor struct {
    Path           string
    SizeBytes      int64  // Iceberg manifest file_size_in_bytes
    RecordCount    int64  // Iceberg manifest record_count
    ETag           string // optional, used when no Iceberg manifest is available

    // DataSequenceNumber is the Iceberg manifest entry's
    // data_sequence_number — the sequence number assigned when the
    // file's data was logically added. This is the field that drives
    // equality-delete and DV scope. It is NOT the same as
    // file_sequence_number (commit-time sequence number for the
    // manifest entry itself).
    DataSequenceNumber int64

    // PartitionSpecId and PartitionValues identify the partition this
    // data file belongs to, in the partition spec used at write time.
    // Required for delete applicability: a delete file with a
    // different (PartitionSpecId, PartitionValues) does not apply,
    // even if its sequence number rule matches. PartitionValues are
    // serialized as a Substrait/Iceberg literal map keyed by partition
    // field id, encoded the same way the planner sees them in the
    // manifest.
    PartitionSpecId int32
    PartitionValues []byte

    // Deletes is the union of position-delete files, equality-delete
    // files, and deletion vectors that apply to this data file.
    // Content type is discriminated by DeleteFileRef.Content.
    Deletes        []DeleteFileRef
}

type DeleteFileRef struct {
    Path           string
    SizeBytes      int64

    // PartitionSpecId and PartitionValues match the partition this
    // delete file was written for. A delete file applies to a data
    // file only when (delete.PartitionSpecId, delete.PartitionValues)
    // == (data.PartitionSpecId, data.PartitionValues). This holds for
    // both position-delete files and equality-delete files; deletion
    // vectors set ReferencedDataFile and so the partition match is
    // implicit but the fields should still be populated for
    // consistency.
    PartitionSpecId int32
    PartitionValues []byte

    // DataSequenceNumber is the delete file's data_sequence_number.
    // The scope rule is: this delete file applies to a data file iff
    //   data_file.DataSequenceNumber  <=  delete_file.DataSequenceNumber
    // for position deletes and deletion vectors (both are
    // position-based and may target data files committed in the same
    // snapshot), and
    //   data_file.DataSequenceNumber  <   delete_file.DataSequenceNumber
    // for equality deletes (predicate-based; only filters older data).
    DataSequenceNumber int64

    // Content matches Iceberg manifest "content" field. DVs are
    // POSITION_DELETES with FileFormat == FileFormatPuffin; there is
    // no separate "deletion vector" content value in Iceberg.
    Content        DeleteContent

    // FileFormat is the on-disk format of this delete file.
    FileFormat     FileFormat

    // EqualityFieldIds carries the Iceberg field IDs the equality
    // predicate is keyed on. Required for Content == EqualityDeletes;
    // empty for position deletes (file or DV).
    EqualityFieldIds []int32

    // Puffin-only fields, used when Content == PositionDeletes and
    // FileFormat == FileFormatPuffin (deletion vectors). The DV blob
    // lives at (Path, BlobOffset, BlobLength). BlobCRC32 is the
    // CRC-32 the Puffin footer records for this blob's
    // (decompressed) bytes; populated when present, used as a
    // tamper / corruption check, NOT as a content hash for cache
    // identity. Cache identity is (puffin file identity per Index
    // Consistency, BlobOffset, BlobLength) — Puffin files are
    // immutable, so that triple is sufficient.
    BlobOffset int64
    BlobLength int64
    BlobCRC32  uint32

    // ReferencedDataFile is set when this delete file targets a single
    // data file (mandatory for v3 deletion vectors; optional for v2
    // position-delete files). Empty when the delete file may target
    // many data files.
    ReferencedDataFile string
}

type DeleteContent int32

const (
    DeleteContentUnspecified DeleteContent = 0
    PositionDeletes          DeleteContent = 1 // Iceberg manifest content=1 (file or DV)
    EqualityDeletes          DeleteContent = 2 // Iceberg manifest content=2
)
// Iceberg v3 deletion vectors are POSITION_DELETES with
// FileFormat == FileFormatPuffin; there is no separate enum value.

type FileFormat int32

const (
    FileFormatUnspecified FileFormat = 0
    FileFormatParquet     FileFormat = 1
    FileFormatAvro        FileFormat = 2
    FileFormatOrc         FileFormat = 3
    FileFormatPuffin      FileFormat = 4
)

type PredicateKind int32

const (
    PredicateUnspecified PredicateKind = 0
    PredicateSubstrait   PredicateKind = 1 // Substrait ExtendedExpression protobuf
    PredicateIceberg     PredicateKind = 2 // Iceberg Expression JSON
)

type VectorMetric int32

const (
    MetricL2     VectorMetric = 0
    MetricCosine VectorMetric = 1
    MetricDot    VectorMetric = 2
)

type VectorQuery struct {
    Column ColumnRef
    Vector []float32
    Metric VectorMetric
    TopK   int
    NProbe int
}

// ColumnRef identifies a column by Iceberg field ID, which is stable
// across rename and reordering. The Path is an optional hint (the
// dotted Iceberg name path, e.g. "user.address.zip") for logging and
// for engines that cannot resolve field IDs; the server must trust
// FieldId when it is set and only fall back to Path when FieldId is
// zero (unspecified) — for example, when querying tables that were
// not written through Iceberg and have no field IDs in the Parquet
// file metadata.
type ColumnRef struct {
    FieldId int32  // Iceberg field ID; 0 means "use Path"
    Path    string // dotted Iceberg name path, optional hint
}

The client is responsible for Iceberg planning (resolving the snapshot to data files and delete files) and passes the resolved set in DataFiles. The server treats this list as authoritative and does not re-read the catalog. See Trust Model below for the security implications and the validation modes a deployment can choose between. Each descriptor carries:

• enough identity (SizeBytes, RecordCount, optional ETag) for the server to verify a cached side index still matches the file,
• the Iceberg DataSequenceNumber (the manifest entry's data_sequence_number, not file_sequence_number) so equality-delete and DV scope can be resolved correctly,
• the Deletes list of position-delete files, equality-delete files, and deletion vectors that the client's planner has attached to this data file. Each entry carries the Iceberg Content discriminator (PositionDeletes or EqualityDeletes) plus the on-disk FileFormat, with the pair (PositionDeletes, FileFormatPuffin) denoting a v3 deletion vector. Content-specific fields cover equality field IDs and Puffin blob offset/length.

v1 implementations should accept Substrait as the canonical wire format. Iceberg Expression JSON is supported as a convenience for connectors that already produce it.

Pushdown Response

The response is range-oriented. Row-id lists are optional and bounded — they are returned only when the request opts in (e.g. for vector top-K) and the planner can verify the result fits within MaxRowIds.

type ParquetPushdownResponse struct {
    FileRanges []FileRange
    RowGroups  []RowGroupRef
    Pages      []PageRef
    RowRefs    []ScoredRowRef // optional; empty unless RequestRowIds set and within MaxRowIds
    Truncated  bool           // true if row-ref list was omitted/truncated due to size cap
    Stats      PushdownStats
}

type FileRange struct {
    File   string
    Offset int64
    Length int64
}

type RowGroupRef struct {
    File     string
    RowGroup int
}

type PageRef struct {
    File     string
    RowGroup int
    Column   ColumnRef
    Page     int
    Offset   int64
    Length   int64
}

// RowRef identifies a single row by its file-absolute row position,
// matching Iceberg's position-delete semantics. RowGroup is included
// for fast locality but is derivable from FilePosition + the file's
// row-group boundaries.
type RowRef struct {
    File        string
    RowGroup    int   // row group containing FilePosition
    FilePosition int64 // 0-based row index within the file (file-absolute)
}

// ScoredRowRef pairs a row reference with an optional similarity
// score. HasScore distinguishes "no score available" from "score is
// 0.0", which is a legal vector-similarity value (a perfect cosine
// match has Score == 1, but L2 distance and dot products commonly
// produce 0.0 for nontrivial inputs). Non-vector queries leave
// HasScore false; vector queries always set it true.
type ScoredRowRef struct {
    Ref      RowRef
    HasScore bool
    Score    float32
}

Row identity uses file-absolute position (matching Iceberg position-delete files), not row-group-local. Row-group-local indexing is exposed via the convenience RowGroup field but is not authoritative — clients converting a RowRef back to a Parquet read should locate the row by FilePosition against the parsed footer's row-group boundaries.

Scores is no longer a parallel array. Pairing each score with its row ref via ScoredRowRef removes the ordering constraint, and the explicit HasScore flag lets a single response mix scored (vector) and unscored (scalar) results without aliasing valid zero-valued scores onto "no score".

Connector Behavior

Existing Connector Path

Spark/Trino/DuckDB
  -> Iceberg planning
  -> Parquet scan
  -> SeaweedFS S3 range reads

Optimized Connector Path

Spark/Trino/DuckDB connector
  -> Iceberg planning
  -> SeaweedFS pushdown API
  -> receive candidate row groups/pages/ranges/row ids
  -> read only needed Parquet ranges

Trust Model and Catalog Validation

The pushdown server treats the request's DataFiles and Deletes as authoritative — it does not re-resolve the snapshot against the Iceberg catalog. That is fine when the caller is a trusted query engine that has just done its own planning, but it is unsafe when the caller is untrusted: a hostile client could omit delete files (returning rows that should be invisible), point at data files outside the table, or claim a snapshot id that does not match the request.

A deployment must pick one of the following modes per endpoint, and the choice should be visible in the pushdown response stats:

• Connector-trusted (default for v1). The pushdown API is reachable only by trusted connectors (Spark/Trino/DuckDB inside the same trust boundary as the catalog). The connector is responsible for honest planning; the server does not validate. Use mTLS, a signed service token, or a network boundary to enforce the trust assumption. Fastest path; appropriate for an internal data plane.
• Catalog-validated. The server validates the request against the Iceberg catalog before serving. For each DataFileDescriptor it confirms (Path, SizeBytes, RecordCount, DataSequenceNumber, PartitionSpecId, PartitionValues) against the manifest at SnapshotId, and confirms the Deletes list matches the manifest's attached delete files. Slower (one catalog read per request, cacheable per snapshot) but safe to expose to untrusted clients.
• Manifest-signed (future). The connector includes a catalog-signed token containing a hash of the resolved planning output; the server verifies the signature without re-reading the catalog. Lowest validation cost on the hot path; requires catalog support that does not exist today, so it is listed as a future option only.

Independent of mode, the server always enforces:

• the underlying object's ACL — a client that cannot read the Parquet file cannot query its indexes (see Security and Access Control);
• request-shape limits (MaxRowIds, predicate complexity caps, request size cap) so a malformed request cannot consume unbounded server resources.

The chosen mode is reported in PushdownStats so the connector and the human operator can verify which trust level actually serviced a request.

Index Consistency

Indexes must be tied to a stable file identity. Preferred identity, in order of strength:

1. Iceberg manifest fields: file_path + file_size_in_bytes + record_count
   (+ snapshot id for cross-snapshot lookups)
2. S3 ETag (canonical for raw S3 access without an Iceberg manifest)
3. Content hash (optional; expensive but fully unambiguous)

Modification time is intentionally not used as identity: replication, re-upload, and metadata-only operations can change mtime without changing content, and content can change without mtime moving on some backends.

If the identity of a Parquet file changes, its indexes are invalidated and rebuilt. Iceberg makes this cheap because data files are immutable; new snapshots add or remove files rather than modifying files in place.

Handling Iceberg Deletes

Iceberg's delete model has evolved across spec versions. Pushdown must support all three forms:

Form	Iceberg spec	Storage	Per-data-file?	Precomputable?
Position delete files	v2	Parquet/Avro/ORC of (file_path, position) rows	No (one delete file may target many)	Yes (merge to bitmap)
Equality delete files	v2	Parquet/Avro/ORC of equality-key rows	No (predicate scope by sequence number)	No (must evaluate per query)
Deletion vectors	v3	Puffin blob (deletion-vector-v1) holding a roaring bitmap	Yes (one DV per data file)	Yes (it is the bitmap)

Position delete files (v2; precomputable)

Position deletes name (data_file, row_position) pairs. The set of position-delete files that apply to a given data file is

{ pdf :
    pdf.DataSequenceNumber  >=  data_file.DataSequenceNumber           // sequence
    AND pdf.PartitionSpecId  ==  data_file.PartitionSpecId             // same spec
    AND pdf.PartitionValues  ==  data_file.PartitionValues             // same partition
    AND pdf.ReferencedDataFile in { empty, data_file.Path }            // file scope
}

For each applicable delete file, the bitmap builder reads its rows and adds row positions to the per-data-file bitmap as follows:

• when pdf.ReferencedDataFile == data_file.Path (single-target), every row contributes its position value;
• when pdf.ReferencedDataFile is empty (multi-target file), each row carries its own (file_path, position) pair, and only rows where file_path == data_file.Path contribute. Skipping the row-level file_path filter would over-delete by attributing every row in the delete file to every applicable data file.

Pushdown merges the filtered rows into a per-data-file roaring bitmap and caches it as a side index:

<parquet_parent>/.index/<file_name>/<identity>/deletes.position.bitmap

The cached bitmap is only valid for the exact set of position-delete files that produced it — see Cache key below.

Pushdown subtracts this bitmap from candidate row sets before returning results.

Deletion vectors (v3; precomputable)

Iceberg v3 replaces position delete files with deletion vectors: a roaring bitmap of file-absolute row positions for one specific data file, stored as a Puffin blob of type deletion-vector-v1. Because each DV already targets a single data file and is already a bitmap, no merge is needed — the cached form is just the decoded bitmap, keyed by the immutable triple (puffin_file_identity, blob_offset, blob_length).

Multiple DV blobs may live in one Puffin file, so the per-data-file pointer is (puffin_file_path, blob_offset, blob_length), not just a file path. The pushdown request must carry that triple per data file (see DeleteFileRef). The Puffin footer also records a CRC-32 per blob, which the server may verify on read; the CRC is not a content hash usable for cross-table cache lookup, only for tamper detection.

Strategy:

1. resolve referenced Puffin file + blob offset/length from the manifest
2. range-GET the Puffin blob
3. decode roaring bitmap (cached)
4. subtract from candidate row sets

For tables that mix v2 position-delete files and v3 DVs (legal during migration), pushdown applies both: union the DV bitmap with the merged-position-delete bitmap before subtracting.

Position-delete bitmap cache key

The merged position-delete bitmap is a function of the set of position-delete files that target the data file at the requested snapshot, not of the data file alone. New snapshots add or compact away position-delete files, which silently changes the merged result. A cache keyed only by data-file identity will hand back stale bitmaps after the first new position-delete file lands.

Cache key per data file:

key = (
    data_file_identity,                  // see Index Consistency
    sorted_set_of_input_delete_file_ids, // each = (path, size, content_hash)
)

Equivalent shorthand for v3 DVs, which are intrinsically per-data-file: (data_file_identity, puffin_file_identity, blob_offset, blob_length). Because Puffin files are immutable, that quadruple is sufficient — no separate content hash is required, and no such hash is exposed by Puffin metadata anyway. The Puffin per-blob CRC-32, when present, is used only to detect corruption on read.

snapshot_id is not a sufficient key — different snapshots can produce the same bitmap, and pinning the cache to snapshot id wastes work after no-op snapshots. Conversely, snapshot_id alone is insufficient across tables because position-delete files are not uniquely identified by snapshot number.

When pushdown receives a request, it computes the key from the position-delete entries of the data file's Deletes list (already supplied by the client's planner) and looks up or builds the bitmap. Cache eviction follows file identity: when a data file or any of its delete files goes out of scope (snapshot expiration, compaction), the corresponding cache entries are reclaimable.

Equality deletes (must evaluate at query time)

An equality-delete file is not a serialized SQL predicate. It is a list of rows, each carrying values for a fixed set of equality-key columns (the file's equality_ids). A data row in scope is deleted if its values for those columns match any row in the delete file.

Execution model:

1. Read the equality-key columns from the data file (using projection / page-level pruning over those columns only).
2. Read the equality-key rows from each applicable delete file. The set of these rows can be loaded once and reused across queries against the same data file (see cache reusability below).
3. For each data row, look up its key tuple in the delete-file key set. Use Iceberg's equality semantics: NULL values match NULL (unlike SQL NULL = NULL), and floating-point comparisons follow Iceberg's documented rules (NaN is a single class for equality matching). Decimal, timestamp, and date types compare by canonical encoded value.
4. Mark matching data rows for deletion.

The applicability rule for which equality-delete files apply is:

{ edf :
    edf.DataSequenceNumber  >  data_file.DataSequenceNumber            // strict; sequence
    AND edf.PartitionSpecId ==  data_file.PartitionSpecId              // same spec
    AND edf.PartitionValues ==  data_file.PartitionValues              // same partition
}

The sequence rule is strict: data files whose data_sequence_number is equal to or greater than the delete file's are not affected — those rows simply never existed when the delete was written. (The Iceberg manifest entry's file_sequence_number is the commit-time sequence number for the manifest entry itself and is not used for delete scoping.) Partition matching is required as well: equality-delete files are written per partition, and a delete in one partition does not delete from another, even if the sequence rule matches. This is why the per-data-file pushdown request must carry both the data file's DataSequenceNumber / PartitionSpecId / PartitionValues and the delete files attached to it: the planner has already resolved this scoping.

Equality-delete bitmap cache reusability

A single union-of-all-applicable-deletes bitmap is snapshot-dependent (the set of applicable delete files changes as snapshots add new ones), but the building blocks are not.

The cache lives at the per-pair level, keyed by (data_file_identity, equality_delete_file_identity). A per-pair bitmap captures "which positions in this data file are deleted by this one equality-delete file" — that fact is determined by the immutable contents of the two files and never changes after both files are written. Per-pair bitmaps are reusable across snapshots, sessions, and queries.

The per-query work is a cheap union over the pre-built per-pair bitmaps for the delete files that apply at this snapshot. New snapshots only require building bitmaps for newly added delete files (against the same data file), not recomputing existing ones.

Eviction follows file identity: a per-pair entry is reclaimable when either side of the pair goes out of scope (snapshot expiration, compaction).

Strategy summary:

• per-pair bitmaps are computed once and cached durably keyed by (data_file_identity, equality_delete_file_identity);
• per-query response unions the applicable per-pair bitmaps (resolved by the snapshot-time applicability rule above) and subtracts from candidate row sets;
• equality-delete evaluation (the hash-join over key tuples that produces a per-pair bitmap) is accelerated by the same scalar indexes (bloom, bitmap, B-tree) the planner already uses.

End-to-end flow

1. Iceberg catalog returns data files plus their attached delete files
   (v2 position-delete files, v2 equality-delete files, v3 deletion
   vectors), each tagged with PartitionSpecId / PartitionValues /
   DataSequenceNumber so applicability is already resolved.
2. For each data file:
     a. apply cached merged position-delete bitmap (v2 position-delete files)
     b. apply DV bitmap (v3 PositionDeletes + FileFormatPuffin entries)
     c. evaluate live equality-delete predicates against side indexes
3. Pushdown execution returns candidate sets minus the union of (a),
   (b), and (c).

Compaction and Maintenance

SeaweedFS can run background index maintenance:

new Parquet file detected
  -> parse footer
  -> build row group stats
  -> optionally build page/bloom/bitmap/vector indexes
  -> publish index metadata

For Iceberg compaction:

old files removed from snapshot
new compacted files added
old indexes become unreachable
new indexes are built lazily or eagerly

Garbage collection can remove indexes for files no longer referenced by active snapshots.

Index Residency and Failure Model

Side indexes have two residencies that match the two-tier architecture:

• Catalog server holds the index registry. For every (table, file_identity, index_kind, field_id) triple, the catalog records: where the index payload lives (.index/... path under the data parent), the identity it was built against, when it was last refreshed, and whether it is healthy. This is metadata only — small per-file rows that the planner consults to decide which indexes to use for a request. The catalog never reads index payload bytes.
• Volume servers hold the index payloads. Index files live in .index/<file_name>/<identity>/... next to the data needles, so the volume server that owns a chunk of a Parquet file also owns the corresponding side-index chunks. Local lookup, scalar predicate evaluation, and vector compute happen on the volume server, returning small results (bitmaps, top-K row refs) instead of moving raw bytes.

Failure modes follow the same split:

• Missing or stale index payload (volume side): the planner requests an index, the volume server reports index_missing or index_stale. The catalog server downgrades the plan: row-group pruning falls back to footer statistics; bloom/bitmap/btree pruning is skipped; vector queries return an explicit "no index" error so the connector can decide whether to read raw vectors or fail. Stats list which indexes were used vs missing.
• Catalog registry stale (catalog side): if the registry says an index exists but the volume server cannot find it, the catalog marks the registry entry stale and returns a downgraded plan; a background reconciler refreshes the entry. This makes the catalog the source of truth for index intent while the volume server is the source of truth for index availability.
• Catalog server unavailable: the connector falls back to standard S3 access; pushdown is unavailable but reads still work.
• Volume server unavailable: standard SeaweedFS replication / re-routing handles the byte-range read; the catalog returns a plan referencing whichever volume servers can serve the file.

In all cases, the response carries a Stats.indexes_used / Stats.indexes_missing list so the connector observes degradations and can decide whether to retry or just proceed with the lighter plan.

Cost Model and When Not to Push Down

Pushdown is not free. Each request adds a planning round-trip and forces the server to load and evaluate side indexes. For small or unselective queries this is a net loss versus a direct ranged GET.

The relevant cost input is the planned scan after Iceberg pruning, not the table's total size. A 100 TiB table where Iceberg planning has already narrowed the query to a single 4 MiB file is a poor candidate for pushdown; a 50 GiB table where Iceberg planning produces 5,000 surviving files is a great candidate even if each individual file is small.

Connectors should skip the pushdown API when:

• the planned scan is small enough that a direct ranged GET is cheaper than the planning round-trip (rule of thumb: total planned scan bytes below ~10 MiB and surviving file count ≤ 1)
• the predicate is non-selective (estimated selectivity > ~0.5) and no projection prunes meaningful columns
• the query has no predicate and no vector clause — file pruning by Iceberg metadata alone already does the job

The pushdown API should expose its own selectivity and cost estimates in the response stats so connectors can learn when to use it; explicit configuration knobs (min_planned_scan_bytes, min_planned_file_count, min_selectivity) can act as a fallback.

Security and Access Control

Side indexes can leak information that the underlying object's ACL would otherwise hide:

• zone maps and B-trees expose min/max values, which can reveal date ranges or identifier ranges to clients with column-level but not row-level visibility
• bloom filters allow probing for the presence of specific values
• inverted indexes expose token vocabularies

For v1, side indexes inherit the same access controls as the underlying Parquet object — a client that cannot read the file cannot query its indexes either. Row- and column-level access policies (e.g. masking, row filters) are out of scope; tables that depend on row-level security should disable pushdown until a finer-grained authorization story exists.

Rollout Plan

The plan follows the architecture: ship the catalog-server brain first, add the index registry, then push heavy execution down to volume servers, and only consider standalone workers if measurements demand it.

Phase 1: Catalog server returns candidate files / row groups / ranges

Catalog server gains a pushdown planner that, given a snapshot and a request, returns the data files, row groups, and column-chunk byte ranges the connector should read. No side indexes yet — pruning is driven by Iceberg metadata plus the parsed Parquet footer.

• parsed-footer cache (row group stats already live in the footer; the win is avoiding repeated Thrift decode and offset lookups, not building new index data)
• column-chunk range optimization
• file / row-group / range-level pushdown over the footer's existing zone-map statistics
• catalog-validated trust mode (server checks the request's DataFiles against the manifest)

Phase 2: Catalog server tracks side indexes

Catalog gains an index registry: a per-file table of which side indexes exist, their on-disk paths under .index/, the data-file identity they were built against, and freshness state. The planner consults the registry to choose which indexes to use; a background reconciler verifies registry entries against volume-server payload availability and rebuilds stale entries.

• index registry schema and CRUD on the catalog
• registration hooks fired when an index is built
• reconciler that downgrades stale entries
• planner consults the registry; missing-index downgrades are reported in stats
• includes Iceberg delete metadata: position-delete files, equality-delete files, v3 deletion vectors are tracked here so the planner attaches them to data-file plans

Phase 3: Volume servers execute local index/vector pushdown

Side-index payloads live next to data needles, and the volume server gains the ability to evaluate predicates and vector queries against them locally. The catalog server's plan now references volume-server endpoints for hot paths, and the connector dispatches to those endpoints to receive pruned bitmaps / row-ref lists / top-K results instead of raw bytes.

• volume-server-side index reader: bloom, bitmap, B-tree, page index
• scalar predicate evaluation against indexes (the v1 predicate subset)
• vector index payload + local distance compute + local top-K
• catalog plan returns (volume_endpoint, file, ranges, index_pointers) tuples for connectors that opt into volume-side execution
• byte-range-only plans remain available for connectors that prefer to read directly via S3

Phase 4: Optional standalone query workers

Only built if measurements show planning, hot-path coordination, or cross-volume merges (e.g. global top-K with very high k) become a bottleneck. A pool of stateless workers takes plans from the catalog, fans out across volume servers, performs cross-volume merges, and returns results to the connector.

• decoupled scaling: workers grow without touching catalog or volume-server capacity
• cross-volume merges (global top-K, large bitmap unions) move out of the catalog
• not on the critical path; remains optional unless profiling demands it

Connector integrations (cross-cutting)

Connector work runs in parallel with phases 1–3 once the catalog API stabilizes:

• SeaweedFS-aware DuckDB extension
• Trino connector pushdown
• Spark DataSource / Iceberg integration

Each connector consumes the catalog's plan and decides per-request whether to read byte ranges directly (compatibility path) or dispatch to volume-server endpoints (hot path).

Benefits

• keeps Parquet compatibility
• accelerates existing Iceberg data
• avoids full file scans
• reduces network and disk I/O
• enables vector search over Parquet data
• lets SeaweedFS become more than object storage

Tradeoffs

• side indexes require storage and maintenance
• consistency must be carefully tied to immutable file identity
• pushdown requires engine connector work
• page-level and row-level filtering are more complex than row-group pruning
• vector search on Parquet is less natural than LanceDB, but easier to adopt for existing lake data

Summary

SeaweedFS should keep Parquet files standard and build auxiliary indexes beside them.

The first useful version can focus on footer caching and row group pruning. Later versions can add scalar, text, delete, and vector indexes.

This creates a practical path:

Iceberg + Parquet compatibility today
+ SeaweedFS-aware pushdown
+ optional vector search acceleration