2 Commits

Include renames

Author	SHA1	Message	Date
Javier de la Torreanduser.email	55f2234026	iceberg maintenance: apply equality deletes only to older data files in their partition (#12187) ... * iceberg maintenance: apply equality deletes only to older data files in their partition Compaction applied every equality-delete key to every data file of every bin. The Iceberg spec ("Scan Planning") applies an equality delete file only to data files with a strictly lower data sequence number, and only to data files in the delete file's own partition unless its spec is unpartitioned. An upsert writer (Flink, the Iceberg Kafka Connect sink, CDC tools) commits an update as the row's new version plus an equality delete of its key in the same commit, so the compaction deleted both the old and the new version of every updated row. - equalityDeleteGroup is keyed by equality field ids and scope (unpartitioned, or spec id plus partition) and keeps, per key, the highest data sequence number of the delete files carrying it. A delete file without a data sequence number is an error rather than a guess. - Partition values are compared typed through the spec's partition type, so one partition decoded in different Go shapes (a uuid as uuid.UUID or [16]byte, a long as int32 or int64, a null present or missing) is one partition. A spec whose fields are all void is unpartitioned. A delete whose scope cannot be decided (unknown spec, no partition values under a partitioned spec, a value that does not convert) stops the compaction; a bin whose own partition cannot be typed while a delete of its spec exists is not compacted. - Each bin is merged with only the groups that apply to its partition, and a row is deleted only by a key whose sequence number is higher than its data file's. - A where filter no longer drops unpartitioned delete files (the clause's field is not in their spec), which let deleted rows return in a filtered compaction. Tests: the table fixture now commits its delete files after the data (data sequence number 1, deletes 2), as writers do; hand-built entries in the unit tests get sequence numbers. New eqdelete_sequence_test.go covers rows written with and after their delete, partition scope, unpartitioned and all-void specs, typed partition matching, missing sequence numbers and partition values, where filters, and group-key collisions. * iceberg maintenance: split the undecidable-scope branch in deleteInPredicateScope --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com>	2026-10-29 00:09:09 -07:00
Chris LuandDevin	222e96d6ab	iceberg maintenance: merge bin files across parquet schema spellings (#11875) ... * iceberg maintenance: compare parquet schemas by field id in bin-pack merge parquet.EqualNodes ignores Iceberg field ids, so a column dropped and\nre-added under the same name and type compared equal and was merged,\nfiling the new column's values under the dropped field id. Require the\nfield ids to agree wherever both sides carry one; the v3 layout's\nper-field check uses the same test.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: merge bin files whose schemas differ by field id A bin used to take the first file's Parquet schema as the writer's and\nrefuse every file that was not structurally identical, so a table never\ncompacted once files had a different column set or REQUIRED-vs-OPTIONAL\nrepetition. A file whose flat columns the writer's schema covers by\nfield id is converted into it (a column the file lacks reads as null);\na file that covers the writer's schema restarts the bin with it, so no\ncolumn is dropped. Equality-delete columns resolve on each file's own\nschema. Renames, type changes, nested columns and files without field\nids stay refused. The sorted and spatial merges are unchanged.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: test bin-pack merge across parquet schema spellings Regression coverage for the field-id merge: an added column in either\norder, REQUIRED vs OPTIONAL, equality deletes resolved per file, the\nre-added column refused on v1/v2 and v3, and the refusals that stay\n(rename, type change, one-sided columns, an added required column, no\nfield ids).\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: resolve missing delete columns as null in bin-pack merge An equality delete on an optional column added after a data file was\nwritten no longer refuses the bin: the file reads the column as null,\nso a real delete key cannot match its rows and a null key deletes them.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: write bin-pack merges under the files' union schema The writer schema is now the union of the bin's file schemas by field\nid, computed in a schema-only pass before any row is written: a column\nstays REQUIRED only where every file has it and spells it so, and the\nmerge refuses only true name/type/id conflicts. Files that need a union\nneither side covers (an added column plus a REQUIRED/OPTIONAL\ndifference, disjoint added columns) now merge in a single pass instead\nof refusing, and progressively wider bins no longer reread and rewrite\nevery earlier file per restart.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: spell the re-added column's test file for codespell Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: resolve equality-delete columns by field id A column renamed after its data and delete files were written still carries its field id there; resolving it by id instead of the current name keeps a rename from turning every equality key null. The missing-column null-key fallback is now for data files only: an equality-delete file that lacks its equality column is refused, as the spec requires it to carry them. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: read only the parquet footer choosing a bin's writer schema The union-schema pass downloaded every input in full, then the merge downloaded each again. The footer suffices for the schema, and a chunked file's ReadAt pulls only the chunks the footer spans. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-20 23:13:43 -07:00