Commit Graph
4 Commits
Author SHA1 Message Date
ecd778a4a0 iceberg maintenance: a merged file with null row ids names no first_row_id (#11973)
* iceberg maintenance: a merged file with null row ids names no first_row_id

Compaction stamps a merged v3 data file with the _row_id of its first row as
first_row_id (mergedFileFirstRowID). When the bin mixed rows that carry a
materialized _row_id with rows that carry none (a data file with no
first_row_id, as a v2 table's files are after an upgrade to v3, gives null
ids), the null rows then inherit first_row_id + their position in the file:
ids that other rows already hold, so two rows answer to the same row id.

A merged file now names a first_row_id only when its footer shows no null
_row_id in any row group. Otherwise the commit assigns the file a fresh range
from the table's next-row-id: the null rows get new, unused ids and the
materialized ones keep theirs (a materialized value overrides inheritance).

Test: TestMergedFileFirstRowIDOnlyWithoutNullIDs ("materialized first, a null
later" fails without the fix).

* iceberg maintenance: name no first_row_id when the merged file has a null _row_id

mergedFileFirstRowID stamped the merged file with its first row's id
without checking later rows. A null _row_id in a merged file inherits
first_row_id+position, an id another row already holds, so the commit
produced two rows answering to the same row id.

Check the footer's null counts for the _row_id column across row groups
first: any null means the file is committed without a first_row_id, so its
nulls take fresh ids from the table's next-row-id while materialized
values still override inheritance.

* shell: retry filer discovery until master knows a filer

A shell started against a freshly booted cluster (weed mini, the
s3tables/lance test harnesses) can list cluster nodes before the filer's
first heartbeat lands on the master. The empty list left FilerAddress
unset and the next command died with "dial tcp: missing address". Poll
ListClusterNodes until a filer appears or the connect timeout expires.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: discover the filer on first use instead of at startup

The startup retry loop made every weed shell invocation wait out the
connect timeout on clusters that have no filer, even for commands that
only talk to the master. Discovery now runs once, on the first command
that actually needs a filer, keeping the retry-until-registered fix for
the fresh-cluster race without stalling master-only commands.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: stop filer discovery retries on non-transient errors

A master that keeps answering ListClusterNodes with a permanent error
(permission denied, unimplemented) cannot gain a filer by retrying, so
discovery returns instead of polling until the deadline.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-23 22:53:43 -07:00
Javier de la TorreandJavier de la Torre a882565cd7 iceberg: keep a default-CRS geometry column without a CRS after compaction (#11899)
* iceberg: keep a default-CRS geometry column without a CRS after compaction

Writers such as DuckDB write a GEOMETRY column whose Iceberg type has the
default CRS (OGC:CRS84) with no CRS in the Parquet logical type. parquet-go
spells the default out when it writes such a column (geometryType's
LogicalType), even when the column it read declared none, so every
compacted file declared GEOMETRY(crs=OGC:CRS84) where its inputs declared
GEOMETRY. Both mean the same to the Parquet spec, but a reader may treat the
file column as another type than the table column: DuckDB casts it to the
table's GEOMETRY, and a cast column is no longer pruned by its row-group
statistics, so a spatial filter read every row group of a compacted file.

After a bin is merged, the footer's schema elements of every GEOMETRY
column whose Iceberg type has the default CRS (top-level or nested) are
rewritten to declare no CRS. The data, the page indexes and every offset
stay byte for byte; only the footer is re-encoded. An explicit CRS is left
as it is. Retyping the writer's column instead does not work: parquet-go
refuses to convert between two GEOMETRY types that both mean the default,
and its sorting writer converts.

Tests: geometry_crs_test.go, a bin-packed and a sorted compaction of
default-CRS inputs (one of them spelling the default out) write no CRS,
with a control showing parquet-go spells it out on its own; an explicit CRS
is kept; a nested geometry. On master the binpack and sort cases fail with
CRS "OGC:CRS84".

* iceberg: compress the row lineage columns of compacted v3 files

A v3 compaction appends _row_id and _last_updated_sequence_number to the
merged file. The two leaves were built without a codec, so they were
written uncompressed, 16 bytes a row, whatever codec the data columns use;
on a table of points the compacted files grew by about a third.

The lineage leaves now take the codec of the reference file's first leaf
that declares one, ZSTD when none does.

Test: TestCompactCompressesLineageColumns; on master both columns are
written UNCOMPRESSED next to ZSTD data columns.

* iceberg: match list and map geometry by Iceberg name in files without field ids

A Parquet footer path runs through the repeated group of a LIST or MAP
(shapes.list.element, attrs.key_value.value), which Iceberg leaves out of
its field names (shapes.element, attrs.value). The name lookup for a file
without field ids therefore missed every geometry inside a list or a map
and left the default CRS spelled out. Name the paths as Iceberg does,
including the legacy two-level list forms.

* iceberg: rewrite the geometry CRS footer inside the merged file's buffer

Removing a CRS shortens the footer, so the new one fits where the old one
was. Copying the whole merged file to replace a few hundred bytes of
footer doubled the memory of a compaction near the target file size.

* iceberg: take the lineage codec from the data columns, not an input's _row_id

An input that already carries _row_id may list it before its data
columns, and write it uncompressed; the new lineage columns then copied
that codec instead of the data columns'. Choose from the columns the
layout keeps.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 14:50:45 -07:00
Chris LuandDevin 222e96d6ab iceberg maintenance: merge bin files across parquet schema spellings (#11875)
* iceberg maintenance: compare parquet schemas by field id in bin-pack merge

parquet.EqualNodes ignores Iceberg field ids, so a column dropped and\nre-added under the same name and type compared equal and was merged,\nfiling the new column's values under the dropped field id. Require the\nfield ids to agree wherever both sides carry one; the v3 layout's\nper-field check uses the same test.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: merge bin files whose schemas differ by field id

A bin used to take the first file's Parquet schema as the writer's and\nrefuse every file that was not structurally identical, so a table never\ncompacted once files had a different column set or REQUIRED-vs-OPTIONAL\nrepetition. A file whose flat columns the writer's schema covers by\nfield id is converted into it (a column the file lacks reads as null);\na file that covers the writer's schema restarts the bin with it, so no\ncolumn is dropped. Equality-delete columns resolve on each file's own\nschema. Renames, type changes, nested columns and files without field\nids stay refused. The sorted and spatial merges are unchanged.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: test bin-pack merge across parquet schema spellings

Regression coverage for the field-id merge: an added column in either\norder, REQUIRED vs OPTIONAL, equality deletes resolved per file, the\nre-added column refused on v1/v2 and v3, and the refusals that stay\n(rename, type change, one-sided columns, an added required column, no\nfield ids).\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: resolve missing delete columns as null in bin-pack merge

An equality delete on an optional column added after a data file was\nwritten no longer refuses the bin: the file reads the column as null,\nso a real delete key cannot match its rows and a null key deletes them.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: write bin-pack merges under the files' union schema

The writer schema is now the union of the bin's file schemas by field\nid, computed in a schema-only pass before any row is written: a column\nstays REQUIRED only where every file has it and spells it so, and the\nmerge refuses only true name/type/id conflicts. Files that need a union\nneither side covers (an added column plus a REQUIRED/OPTIONAL\ndifference, disjoint added columns) now merge in a single pass instead\nof refusing, and progressively wider bins no longer reread and rewrite\nevery earlier file per restart.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: spell the re-added column's test file for codespell

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: resolve equality-delete columns by field id

A column renamed after its data and delete files were written still carries
its field id there; resolving it by id instead of the current name keeps a
rename from turning every equality key null. The missing-column null-key
fallback is now for data files only: an equality-delete file that lacks its
equality column is refused, as the spec requires it to carry them.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: read only the parquet footer choosing a bin's writer schema

The union-schema pass downloaded every input in full, then the merge
downloaded each again. The footer suffices for the schema, and a chunked
file's ReadAt pulls only the chunks the footer spans.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 23:13:43 -07:00
Chris LuandDevin 50a9312e1e iceberg maintenance: v3 row lineage so v3 tables compact and rewrite manifests (#11653)
* iceberg maintenance: commit v3 row lineage on the three commit paths

iceberg-go 0.7.0 refuses every snapshot on a format-version 3 table
that lacks first-row-id and added-rows ("invalid row lineage"), so no
v3 table was ever compacted or had its manifests rewritten; worse,
passing 0 to the manifest list would have handed new files row IDs
existing rows already hold.

Each commit now plans first-row-id from the table's next-row-id, lets
the v3 manifest-list writer assign manifest first_row_id from there,
and reports the assigned range as the snapshot's added-rows so
next-row-id advances by exactly that. The commit guard also treats a
moved next-row-id or a changed format version as a stale plan, like a
moved head: a commit to another branch advances next-row-id too, and
manifests written for the old version must not be committed.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: keep row identity through v3 compaction

A rewrite that moves rows into a new data file must not renumber them.
On v3 tables the merge now appends _row_id and
_last_updated_sequence_number (spec field ids 2147483540/2147483539)
to the output and fills each row with the value its input
materializes, else the inherited one: the input file's first_row_id
plus the row's position before any delete, and the manifest entry's
data sequence number. Inputs are matched with their lineage columns
set aside, so a file an earlier compaction wrote merges with one an
engine wrote; a duplicate lineage column or a reserved name under
another field id fails the bin instead of writing a doubled column.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: skip rewrite_position_delete_files on v3 tables

The spec forbids adding position-delete files to a format-version 3
table (v3 writes deletion vectors), and this rewrite adds new ones.
Detection no longer proposes the operation and the operation returns a
skip result; compaction still applies and removes existing position
deletes.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: keep Puffin deletion vectors out of position-delete reads

A Puffin file in a delete manifest is a deletion vector, not a
position-delete file: reading it as Parquet fails, and merging the data
file it covers without applying the vector resurrects deleted rows.

Skip Puffin entries in the position-delete collection and the delete
rewrite group scan, and exclude the files a vector covers from
compaction candidates - in compactDataFiles and in eligibility
detection alike. A vector whose referenced_data_file is missing or
unresolvable fails the run rather than silently compacting whatever it
points at.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: name a merged file's own first row ID

Rewritten rows keep their IDs, so a merged data file's first_row_id is
the _row_id of its first written row - not the next free range the
manifest list would otherwise assign to the entry at commit. Read the
first materialized value back off the merged output and stamp it on the
manifest entry.

Add an end-to-end test that populates a v3 table whose entries carry no
first_row_id, so the compact path exercises manifest first_row_id
inheritance through the stored manifest list, then commits and reads
the result back through the real manifest readers.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: locate the _row_id column by index, not row position

Repeated columns shift parquet.Row positions, so leaf.ColumnIndex is not
the value's slot — a first row containing a list or map saved an unrelated
value as the merged file's first_row_id.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: read the returned row before treating EOF as empty

parquet-go can return a row alongside io.EOF; a one-row merged file then
lost its first_row_id and the manifest list would allocate a fresh range.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: avoid nil dereference in first-row-id failure message

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 09:29:02 +08:00