iceberg: let table properties override the worker config (#10772)

* iceberg: carry snapshot retention in milliseconds

Config stored retention as hours, so any sub-hour value would have to be
truncated to 0 and then clamped back up to the 168 hour default. Keep the
plugin config key in hours and convert once at parse time.

* iceberg: let table properties override the worker config

Every other Iceberg implementation lets a table's own properties win over
engine defaults; the worker ignored them entirely. A writer honouring
write.target-file-size-bytes and a compactor rewriting to the plugin
config's size would rewrite each other's output forever.

Resolved once per job rather than per operation, so compaction committing
new metadata mid-job cannot change the settings underneath it.

* iceberg: clamp the orphan cutoff so it cannot overflow

collectOrphanCandidates converts the cutoff to a time.Duration. Past
roughly 2.5 million hours that multiplication wraps negative, putting the
cutoff in the future so every file walked looks like an orphan and gets
deleted, including data a concurrent writer has not yet committed.

Reachable today through orphan_older_than_hours.
This commit is contained in:
Chris Lu authored and GitHub committed 2026-08-16 09:16:05 -07:00
1 parent 4c40ec3a9e
commit a1d3fe236f
9 files changed
+286 -75

No files matched your search

@@ -600,9 +600,9 @@ func testExpireSnapshots(t *testing.T) {
handler := icebergHandler.NewHandler(nil)
config := icebergHandler.Config{
SnapshotRetentionHours: 0, // instant expiry — everything eligible
MaxSnapshotsToKeep: 1, // keep only the current snapshot
MaxCommitRetries: 3,
SnapshotRetentionMs: 0, // instant expiry — everything eligible
MaxSnapshotsToKeep: 1, // keep only the current snapshot
MaxCommitRetries: 3,
}
result, _, err := handler.ExpireSnapshots(context.Background(), client, bucket, path.Join(ns, tbl), config)
@@ -932,9 +932,9 @@ func testFullMaintenanceCycle(t *testing.T) {
// Step 1: Expire snapshots
expireConfig := icebergHandler.Config{
SnapshotRetentionHours: 0, // instant expiry
MaxSnapshotsToKeep: 1,
MaxCommitRetries: 3,
SnapshotRetentionMs: 0, // instant expiry
MaxSnapshotsToKeep: 1,
MaxCommitRetries: 3,
}
result, _, err := handler.ExpireSnapshots(ctx, client, bucket, tablePath, expireConfig)
require.NoError(t, err)