seaweedfs/PARQUET_PUSHDOWN_DESIGN.md

SeaweedFS-Aware Parquet Layout and Pushdown Design

Status

Draft design note.

Goal

Keep Parquet files fully compatible with existing engines such as Spark, Trino, DuckDB, PyArrow, and Iceberg, while allowing SeaweedFS-aware clients to take advantage of storage-side metadata, indexes, and pushdown execution.

The design avoids replacing Parquet with a custom format. Instead, SeaweedFS stores normal Parquet objects and builds auxiliary metadata/indexes beside them.

Standard Parquet compatibility
+ SeaweedFS side indexes
+ pushdown-aware connectors
= faster scans, filtering, and hybrid vector search

Non-Goals

• Do not require changes to the Parquet file format.
• Do not require existing Spark/Trino/DuckDB readers to change.
• Do not make SeaweedFS the only way to read the data.
• Do not rewrite all Parquet data into Lance or another vector-native format.

Background

Parquet already contains useful metadata:

• file footer
• row group metadata
• column chunk offsets
• column statistics
• optional page indexes
• optional bloom filters

Typical readers use this flow:

1. Read Parquet footer
2. Discover row groups and column chunks
3. Prune row groups using statistics
4. Read selected column chunks/pages
5. Decode/filter/project rows

SeaweedFS can optimize this process by caching metadata, exposing smarter range reads, and building additional side indexes.

High-Level Architecture

Spark / Trino / DuckDB / PyArrow / Custom Query Engine
        |
        | S3 API or SeaweedFS Pushdown API
        v
SeaweedFS S3 Gateway / Filer
        |
        | metadata/index lookup
        v
SeaweedFS Index Metadata
        |
        | range reads / local filtering
        v
SeaweedFS Volume Servers
        |
        v
Parquet data needles

There are two access modes:

1. Standard Object Access

Existing engines read Parquet through the normal S3-compatible API.

engine -> SeaweedFS S3 -> Parquet object ranges

This preserves compatibility.

2. SeaweedFS-Aware Pushdown Access

Optimized connectors call a SeaweedFS pushdown API before reading data.

engine -> Iceberg planning -> SeaweedFS pushdown API -> candidate files/ranges/rows

The engine still reads standard Parquet data, but reads much less of it.

Physical Layout

The user-visible layout remains normal Iceberg/Parquet layout:

/table/
  metadata/
    v1.metadata.json
    snapshots/...
  data/
    ds=2026-01-01/
      part-00001.parquet
      part-00002.parquet

SeaweedFS may store side indexes internally or in a hidden namespace:

/table/_seaweed_index/
  data/ds=2026-01-01/part-00001.parquet/
    footer.cache
    row_group_stats
    page_index.timestamp
    bloom.user_id
    bitmap.tenant_id
    btree.timestamp
    inverted.message
    vector.embedding.ivf

The original Parquet file is not modified.

Logical View for Planning

The Parquet object on disk is unchanged. For planning and pushdown purposes, SeaweedFS exposes a logical view of the file derived from the footer:

Parquet file
 ├── footer metadata
 ├── row group 0
 │    ├── column chunk: id
 │    ├── column chunk: timestamp
 │    ├── column chunk: tenant_id
 │    └── column chunk: payload
 └── row group 1
      ├── column chunk: id
      ├── column chunk: timestamp
      ├── column chunk: tenant_id
      └── column chunk: payload

Each leaf maps to a (file, offset, length) byte range inside the original object:

footer                          -> (file, footer_offset, footer_length)
row_group_0 / column_id         -> (file, offset, length)
row_group_0 / column_timestamp  -> (file, offset, length)
row_group_0 / column_tenant_id  -> (file, offset, length)
row_group_0 / column_payload    -> (file, offset, length)
row_group_1 / column_id         -> (file, offset, length)
...

This is a planning abstraction only. The Parquet file is not split into per-column-chunk needles, and the on-disk object is not modified. Reads are still issued as ranged GETs against the original object.

Side Index Types

1. Footer Cache

Cache the parsed Parquet footer.

Purpose:

• avoid repeated footer reads
• speed up planning
• expose row group and column chunk offsets quickly

Index content:

file path
file size
last modified / etag
schema
row groups
column chunks
statistics
byte offsets

2. Row Group Zone Map

Stores min/max/null count per row group and column.

Useful for predicates such as:

WHERE timestamp BETWEEN t1 AND t2
WHERE id > 1000

Skip condition:

row_group.max < predicate_min OR row_group.min > predicate_max

3. Page-Level Index

Stores min/max and offsets per page.

Useful when row groups are large and predicates are selective.

Example:

row_group_3 / column_timestamp:
  page_0 min=2026-01-01 max=2026-01-02
  page_1 min=2026-01-03 max=2026-01-04
  page_2 min=2026-01-05 max=2026-01-06

4. Bloom Filter Index

Useful for equality lookups.

Example:

WHERE user_id = 'u123'

If the bloom filter says the value is absent, SeaweedFS skips the row group or page.

5. Bitmap Index

Useful for low- or medium-cardinality columns.

Examples:

WHERE tenant_id = 'abc'
WHERE status IN ('failed', 'pending')

Bitmap operations:

tenant_id=abc AND status=failed

6. B-Tree / Range Index

Useful for range predicates and point lookups on sortable columns.

Examples:

WHERE timestamp > t
WHERE id BETWEEN a AND b

The index maps values or ranges to:

file
row group
page
optional row ids

7. Inverted Index

Useful for logs, text, tags, and JSON fields.

Examples:

WHERE message CONTAINS 'timeout'
WHERE tags CONTAINS 'gpu'

Index structure:

token -> candidate row groups/pages/row ids

8. Vector Index

Useful for embedding columns stored inside Parquet.

Example:

ORDER BY embedding <-> query_vector
LIMIT 20

Possible index types:

• IVF
• IVF+PQ
• HNSW
• scalar quantization

Index layout:

/table/_seaweed_index/.../vector.embedding.ivf/
  centroids
  list_000001
  list_000002
  pq_codebooks

This allows SeaweedFS to perform approximate nearest neighbor search before reading Parquet columns.

Pushdown Types

File Pruning

Use Iceberg metadata plus SeaweedFS indexes to skip files.

Row Group Pruning

Use row group statistics, bloom filters, and side indexes.

Page Pruning

Use page-level indexes to read only matching pages.

Column Pruning

Read only requested columns and predicate columns.

Example:

SELECT id, image_path
WHERE tenant_id = 'abc'
ORDER BY embedding <-> ?
LIMIT 10

Only these columns are needed:

tenant_id
embedding
id
image_path

Scalar Predicate Pushdown

Evaluate simple predicates using indexes:

=, !=, <, <=, >, >=
IN
BETWEEN
IS NULL
CONTAINS

Vector Search Pushdown

Search vector side indexes on volume servers or index-serving nodes.

Flow:

1. Choose candidate vector partitions
2. Route to volume servers holding vector index/data needles
3. Compute local top-K
4. Merge global top-K
5. Fetch projected Parquet columns

Hybrid Scalar + Vector Pushdown

Example:

SELECT image_path
WHERE tenant_id = 'abc'
  AND timestamp >= '2026-01-01'
ORDER BY embedding <-> query_vector
LIMIT 20

Execution:

1. tenant_id bitmap filters candidate rows/pages
2. timestamp zone map or B-tree narrows candidates
3. vector index searches only candidate partitions
4. delete masks are applied if Iceberg delete files exist
5. top-K candidates are returned
6. projected columns are fetched

API Sketch

Pushdown Request

type ParquetPushdownRequest struct {
    Table          string
    SnapshotId     int64
    Files          []string
    Columns        []string
    PredicateKind  PredicateKind // SUBSTRAIT or ICEBERG_EXPRESSION
    Predicate      []byte        // serialized per PredicateKind
    VectorQuery    *VectorQuery
    Limit          int
}

type PredicateKind int32

const (
    PredicateUnspecified PredicateKind = 0
    PredicateSubstrait   PredicateKind = 1 // Substrait ExtendedExpression protobuf
    PredicateIceberg     PredicateKind = 2 // Iceberg Expression JSON
)

type VectorMetric int32

const (
    MetricL2     VectorMetric = 0
    MetricCosine VectorMetric = 1
    MetricDot    VectorMetric = 2
)

type VectorQuery struct {
    Column string
    Vector []float32
    Metric VectorMetric
    TopK   int
    NProbe int
}

v1 implementations should accept Substrait as the canonical wire format. Iceberg Expression JSON is supported as a convenience for connectors that already produce it.

Pushdown Response

type ParquetPushdownResponse struct {
    FileRanges []FileRange
    RowGroups  []RowGroupRef
    Pages      []PageRef
    RowIds     []RowRef
    Scores     []float32
    Stats      PushdownStats
}

type FileRange struct {
    File   string
    Offset int64
    Length int64
}

type RowGroupRef struct {
    File     string
    RowGroup int
}

type PageRef struct {
    File     string
    RowGroup int
    Column   string
    Page     int
    Offset   int64
    Length   int64
}

type RowRef struct {
    File     string
    RowGroup int
    RowId    int64
}

Connector Behavior

Existing Connector Path

Spark/Trino/DuckDB
  -> Iceberg planning
  -> Parquet scan
  -> SeaweedFS S3 range reads

Optimized Connector Path

Spark/Trino/DuckDB connector
  -> Iceberg planning
  -> SeaweedFS pushdown API
  -> receive candidate row groups/pages/ranges/row ids
  -> read only needed Parquet ranges

Index Consistency

Indexes must be tied to a stable file identity. Preferred identity, in order of strength:

1. Iceberg manifest fields: file_path + file_size_in_bytes + record_count
   (+ snapshot id for cross-snapshot lookups)
2. S3 ETag (canonical for raw S3 access without an Iceberg manifest)
3. Content hash (optional; expensive but fully unambiguous)

Modification time is intentionally not used as identity: replication, re-upload, and metadata-only operations can change mtime without changing content, and content can change without mtime moving on some backends.

If the identity of a Parquet file changes, its indexes are invalidated and rebuilt. Iceberg makes this cheap because data files are immutable; new snapshots add or remove files rather than modifying files in place.

Handling Iceberg Deletes

Iceberg uses two delete mechanisms, with different implications for indexing:

Position deletes (precomputable)

Position deletes name (data_file, row_position) pairs. They can be merged into a per-data-file roaring bitmap and cached as a side index:

/table/_seaweed_index/.../deletes.position.bitmap

Pushdown subtracts this bitmap from candidate row sets before returning results.

Equality deletes (must evaluate at query time)

Equality deletes carry a predicate (e.g. id = 42) and apply to all data files in scope of the delete file's sequence number. They cannot be precomputed as a static bitmap per data file because:

• one equality delete file can affect many data files
• new data files added later may need to be filtered by the same delete predicate until compaction

Strategy: evaluate equality-delete predicates at query time, accelerated by the same scalar indexes (bloom, bitmap, B-tree). Optionally cache materialized result bitmaps keyed by (data_file_id, equality_delete_file_id) and invalidate on snapshot change.

End-to-end flow

1. Iceberg catalog returns data files plus position+equality delete files
2. For each data file:
     a. apply cached position-delete bitmap
     b. evaluate live equality-delete predicates against side indexes
3. Pushdown execution returns candidate sets minus the union of (a) and (b)

Compaction and Maintenance

SeaweedFS can run background index maintenance:

new Parquet file detected
  -> parse footer
  -> build row group stats
  -> optionally build page/bloom/bitmap/vector indexes
  -> publish index metadata

For Iceberg compaction:

old files removed from snapshot
new compacted files added
old indexes become unreachable
new indexes are built lazily or eagerly

Garbage collection can remove indexes for files no longer referenced by active snapshots.

Rollout Plan

Phase 1: Metadata Acceleration

• Parquet footer cache
• row group stats cache
• column chunk range optimization
• expose file/range-level pushdown

Phase 2: Scalar Indexes

• bloom filter side index
• bitmap index
• B-tree/range index
• basic predicate API

Phase 3: Page-Level Pruning

• page statistics
• page offset map
• return page-level ranges

Phase 4: Iceberg Delete Integration

• position delete masks
• equality delete masks
• snapshot-aware index validity

Phase 5: Vector Indexes

• IVF side index for embedding columns
• local top-K search on volume servers
• global top-K merge
• hybrid scalar + vector search

Phase 6: Query Engine Integrations

• SeaweedFS-aware DuckDB extension
• Trino connector pushdown
• Spark DataSource/Iceberg integration

Benefits

• keeps Parquet compatibility
• accelerates existing Iceberg data
• avoids full file scans
• reduces network and disk I/O
• enables vector search over Parquet data
• lets SeaweedFS become more than object storage

Tradeoffs

• side indexes require storage and maintenance
• consistency must be carefully tied to immutable file identity
• pushdown requires engine connector work
• page-level and row-level filtering are more complex than row-group pruning
• vector search on Parquet is less natural than LanceDB, but easier to adopt for existing lake data

Summary

SeaweedFS should keep Parquet files standard and build auxiliary indexes beside them.

The first useful version can focus on footer caching and row group pruning. Later versions can add scalar, text, delete, and vector indexes.

This creates a practical path:

Iceberg + Parquet compatibility today
+ SeaweedFS-aware pushdown
+ optional vector search acceleration