2 Commits

Include renames

Author	SHA1	Message	Date
Javier de la TorreandJavier de la Torre	a882565cd7	iceberg: keep a default-CRS geometry column without a CRS after compaction (#11899) ... * iceberg: keep a default-CRS geometry column without a CRS after compaction Writers such as DuckDB write a GEOMETRY column whose Iceberg type has the default CRS (OGC:CRS84) with no CRS in the Parquet logical type. parquet-go spells the default out when it writes such a column (geometryType's LogicalType), even when the column it read declared none, so every compacted file declared GEOMETRY(crs=OGC:CRS84) where its inputs declared GEOMETRY. Both mean the same to the Parquet spec, but a reader may treat the file column as another type than the table column: DuckDB casts it to the table's GEOMETRY, and a cast column is no longer pruned by its row-group statistics, so a spatial filter read every row group of a compacted file. After a bin is merged, the footer's schema elements of every GEOMETRY column whose Iceberg type has the default CRS (top-level or nested) are rewritten to declare no CRS. The data, the page indexes and every offset stay byte for byte; only the footer is re-encoded. An explicit CRS is left as it is. Retyping the writer's column instead does not work: parquet-go refuses to convert between two GEOMETRY types that both mean the default, and its sorting writer converts. Tests: geometry_crs_test.go, a bin-packed and a sorted compaction of default-CRS inputs (one of them spelling the default out) write no CRS, with a control showing parquet-go spells it out on its own; an explicit CRS is kept; a nested geometry. On master the binpack and sort cases fail with CRS "OGC:CRS84". * iceberg: compress the row lineage columns of compacted v3 files A v3 compaction appends _row_id and _last_updated_sequence_number to the merged file. The two leaves were built without a codec, so they were written uncompressed, 16 bytes a row, whatever codec the data columns use; on a table of points the compacted files grew by about a third. The lineage leaves now take the codec of the reference file's first leaf that declares one, ZSTD when none does. Test: TestCompactCompressesLineageColumns; on master both columns are written UNCOMPRESSED next to ZSTD data columns. * iceberg: match list and map geometry by Iceberg name in files without field ids A Parquet footer path runs through the repeated group of a LIST or MAP (shapes.list.element, attrs.key_value.value), which Iceberg leaves out of its field names (shapes.element, attrs.value). The name lookup for a file without field ids therefore missed every geometry inside a list or a map and left the default CRS spelled out. Name the paths as Iceberg does, including the legacy two-level list forms. * iceberg: rewrite the geometry CRS footer inside the merged file's buffer Removing a CRS shortens the footer, so the new one fits where the old one was. Copying the whole merged file to replace a few hundred bytes of footer doubled the memory of a compaction near the target file size. * iceberg: take the lineage codec from the data columns, not an input's _row_id An input that already carries _row_id may list it before its data columns, and write it uncompressed; the new lineage columns then copied that codec instead of the data columns'. Choose from the columns the layout keeps. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>	2026-10-21 14:50:45 -07:00
Chris LuandDevin	50a9312e1e	iceberg maintenance: v3 row lineage so v3 tables compact and rewrite manifests (#11653) ... * iceberg maintenance: commit v3 row lineage on the three commit paths iceberg-go 0.7.0 refuses every snapshot on a format-version 3 table that lacks first-row-id and added-rows ("invalid row lineage"), so no v3 table was ever compacted or had its manifests rewritten; worse, passing 0 to the manifest list would have handed new files row IDs existing rows already hold. Each commit now plans first-row-id from the table's next-row-id, lets the v3 manifest-list writer assign manifest first_row_id from there, and reports the assigned range as the snapshot's added-rows so next-row-id advances by exactly that. The commit guard also treats a moved next-row-id or a changed format version as a stale plan, like a moved head: a commit to another branch advances next-row-id too, and manifests written for the old version must not be committed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: keep row identity through v3 compaction A rewrite that moves rows into a new data file must not renumber them. On v3 tables the merge now appends _row_id and _last_updated_sequence_number (spec field ids 2147483540/2147483539) to the output and fills each row with the value its input materializes, else the inherited one: the input file's first_row_id plus the row's position before any delete, and the manifest entry's data sequence number. Inputs are matched with their lineage columns set aside, so a file an earlier compaction wrote merges with one an engine wrote; a duplicate lineage column or a reserved name under another field id fails the bin instead of writing a doubled column. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: skip rewrite_position_delete_files on v3 tables The spec forbids adding position-delete files to a format-version 3 table (v3 writes deletion vectors), and this rewrite adds new ones. Detection no longer proposes the operation and the operation returns a skip result; compaction still applies and removes existing position deletes. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: keep Puffin deletion vectors out of position-delete reads A Puffin file in a delete manifest is a deletion vector, not a position-delete file: reading it as Parquet fails, and merging the data file it covers without applying the vector resurrects deleted rows. Skip Puffin entries in the position-delete collection and the delete rewrite group scan, and exclude the files a vector covers from compaction candidates - in compactDataFiles and in eligibility detection alike. A vector whose referenced_data_file is missing or unresolvable fails the run rather than silently compacting whatever it points at. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: name a merged file's own first row ID Rewritten rows keep their IDs, so a merged data file's first_row_id is the _row_id of its first written row - not the next free range the manifest list would otherwise assign to the entry at commit. Read the first materialized value back off the merged output and stamp it on the manifest entry. Add an end-to-end test that populates a v3 table whose entries carry no first_row_id, so the compact path exercises manifest first_row_id inheritance through the stored manifest list, then commits and reads the result back through the real manifest readers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: locate the _row_id column by index, not row position Repeated columns shift parquet.Row positions, so leaf.ColumnIndex is not the value's slot — a first row containing a list or map saved an unrelated value as the merged file's first_row_id. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: read the returned row before treating EOF as empty parquet-go can return a row alongside io.EOF; a one-row merged file then lost its first_row_id and the manifest list would allocate a fresh range. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: avoid nil dereference in first-row-id failure message --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-12 09:29:02 +08:00