iceberg maintenance: bound decimal, time, uuid, fixed and rescaled timestamp columns on compacted files (#12190)

* iceberg maintenance: bound decimal, time, uuid, fixed and rescaled timestamp columns on compacted files

The column statistics written for a compacted data file give lower and upper
bounds only for booleans, integers, floats, dates, strings, binaries and
timestamps already stored in the Iceberg type's unit. Every other column is
left unbounded, so a reader can never skip a compacted file on it: a
DECIMAL(18,2) column written by DuckDB (stored as INT64) loses its bounds
after compaction, as does a millisecond timestamp, a TIME or a UUID.

boundLiteral now also converts:
- DECIMAL stored as INT32 or INT64 at the column's scale. A decimal stored
  as a byte array still gets no bound: Parquet compares those statistics as
  bytes, which does not order negative two's-complement values;
- TIME, TIMESTAMP, TIMESTAMPTZ and TIMESTAMP_NS from any Parquet time unit,
  rescaled to the Iceberg unit; a value rescaled to a coarser unit is
  rounded down for a lower bound and up for an upper bound so it stays a
  bound, and a value that would overflow gets none;
- UUID and FIXED[n] from a FIXED_LEN_BYTE_ARRAY of the right length;
- a DOUBLE column read from a FLOAT file.

arrow-go becomes a direct dependency (decimal128, already in the module
graph through iceberg-go).

* iceberg maintenance: bound TIME(MILLIS) columns stored as INT32

* iceberg maintenance: bound timestamptz_ns columns too

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
This commit is contained in:
Javier de la Torreanduser.email authored and GitHub committed 2026-10-29 00:12:46 -07:00
1 parent edf2b0f5ee
commit e9c280e095
3 files changed
+276 -14

No files matched your search

+1 -1
View File
@@ -117,6 +117,7 @@ require (
github.com/Jille/raft-grpc-transport v1.6.1
github.com/ThreeDotsLabs/watermill v1.5.3
github.com/a-h/templ v0.3.1100
github.com/apache/arrow-go/v18 v18.8.0
github.com/apache/cassandra-gocql-driver/v2 v2.1.2
github.com/apache/iceberg-go v0.7.0
github.com/apple/foundationdb/bindings/go v0.0.0-20250911184653-27f7192f47c3
@@ -181,7 +182,6 @@ require (
github.com/adrg/xdg v0.5.3 // indirect
github.com/anchore/go-lzo v0.1.1 // indirect
github.com/antlr4-go/antlr/v4 v4.13.1 // indirect
github.com/apache/arrow-go/v18 v18.8.0 // indirect
github.com/apache/thrift v0.24.0 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.10.1 // indirect
github.com/bahlo/generic-list-go v0.2.0 // indirect