diff --git a/Iceberg-Table-Maintenance.md b/Iceberg-Table-Maintenance.md index ab5fe46..cdec0c5 100644 --- a/Iceberg-Table-Maintenance.md +++ b/Iceberg-Table-Maintenance.md @@ -45,12 +45,14 @@ Compaction merges many small Parquet files within the same partition into fewer, #### How It Works -1. Read the current snapshot's manifest list and all data manifests -2. Group small files (below `target_file_size_bytes`) by partition key +1. Read the current snapshot's manifest list; separate data manifests from delete manifests +2. Group small data files (below `target_file_size_mb`) by **partition spec ID + partition key** so files from different specs are never mixed 3. Filter groups to those with at least `min_input_files` entries -4. For each group (bin), read all source Parquet files and merge rows into a single output file -5. Write a new manifest with ADDED entries (merged files), DELETED entries (originals), and EXISTING entries (untouched files) -6. Commit a new snapshot via optimistic concurrency +4. If `apply_deletes` is enabled and delete manifests exist, collect position deletes and equality deletes from all delete manifest entries +5. For each group (bin), read all source Parquet files, filter out deleted rows (position deletes via binary search, equality deletes via hash set lookup), and merge remaining rows into a single output file +6. Write one manifest per partition spec with ADDED entries (merged files), DELETED entries (originals), and EXISTING entries (untouched files) +7. Carry forward delete manifests if any non-compacted data files remain; drop them if all data files were compacted (deletes fully consumed) +8. Commit a new snapshot via optimistic concurrency #### Before / After: Unpartitioned Table @@ -126,14 +128,24 @@ compact-...-1.parquet (~240 MB, files 9-15) (files 16-20 left alone if fewer than min_input_files) ``` +#### Delete Handling + +When `apply_deletes` is enabled (the default), compaction applies both **position deletes** and **equality deletes** during the merge: + +- **Position deletes**: the delete file contains `file_path` + `pos` columns indicating specific rows to remove. Paths are normalized so absolute S3 URLs and relative paths match correctly. +- **Equality deletes**: the delete file specifies equality field IDs and column values. Rows matching those values are filtered out. Different delete files may use different equality columns — deletes are grouped by field ID set. + +After compaction, delete manifests whose referenced data files were all compacted are dropped (deletes fully consumed). If any data files remain uncompacted, delete manifests are carried forward. + +Set `apply_deletes=false` to revert to the previous behavior of skipping tables with delete manifests. + #### Skip Conditions | Condition | Result | |:---|:---| -| All files >= `target_file_size_bytes` | `"no files eligible for compaction"` | +| All files >= `target_file_size_mb` | `"no files eligible for compaction"` | | No partition has >= `min_input_files` small files | `"no files eligible for compaction"` | -| Delete manifests present (row-level deletes) | `"compaction skipped: delete manifests present"` | -| Multiple partition specs (schema evolution) | `"compaction skipped: multiple partition specs present"` | +| Delete manifests present and `apply_deletes=false` | `"compaction skipped: delete manifests present and apply_deletes is disabled"` | | No current snapshot | `"no current snapshot"` | --- @@ -273,7 +285,7 @@ Result: `"rewrote 12 manifests into 1 (320 entries)"` | Condition | Result | |:---|:---| -| Manifest count < `min_manifests_to_rewrite` | `"only N manifests, below threshold of M"` | +| Data manifest count < `min_manifests_to_rewrite` | `"only N data manifests, below threshold of M"` | | No current snapshot | `"no current snapshot"` | | No data entries | `"no data entries to rewrite"` | @@ -306,8 +318,9 @@ These control **how** maintenance operations behave. Set via the Admin UI or pas | `snapshot_retention_hours` | `168` (7 days) | Expire snapshots older than this | | `max_snapshots_to_keep` | `5` | Always keep at least this many newest snapshots | | `orphan_older_than_hours` | `72` (3 days) | Safety window: only delete orphans older than this | -| `target_file_size_bytes` | `268435456` (256 MB) | Files smaller than this are compaction candidates | +| `target_file_size_mb` | `256` (MB) | Files smaller than this are compaction candidates | | `min_input_files` | `5` | Minimum small files in a partition to trigger compaction | +| `apply_deletes` | `true` | When true, compaction applies position and equality deletes to data files. When false, tables with delete manifests are skipped | | `min_manifests_to_rewrite` | `5` | Minimum manifests before rewriting is triggered | | `max_commit_retries` | `5` | Max optimistic concurrency retries on version conflict | | `operations` | `all` | Comma-separated list of operations, or `all` | @@ -384,8 +397,9 @@ Tables matching either condition are proposed as maintenance jobs. The actual op Maintenance jobs report progress and results through the worker framework: - **Activity events**: scan start, scan complete (with table count), per-operation start/complete -- **Progress updates**: percentage based on completed operations +- **Progress updates**: percentage based on completed operations, with per-bin granularity during compaction - **Job result summary**: per-operation outcomes joined by semicolons +- **Structured metrics**: per-operation metrics in `OutputValues` with dot-prefixed keys Example result summary: ``` @@ -395,11 +409,29 @@ remove_orphans: removed 4 orphan file(s); rewrite_manifests: rewrote 7 manifests into 1 (120 entries) ``` +### Structured Metrics + +Each operation returns structured metrics in the job's `OutputValues` map, keyed with the operation name as prefix: + +| Key | Description | +|:---|:---| +| `compact.files_merged` | Number of input files merged | +| `compact.files_written` | Number of output files produced | +| `compact.bins` | Number of compaction bins processed | +| `compact.duration_ms` | Time spent on compaction | +| `expire_snapshots.snapshots_expired` | Number of snapshots removed | +| `expire_snapshots.files_deleted` | Unreferenced files cleaned up | +| `expire_snapshots.duration_ms` | Time spent on expiration | +| `remove_orphans.orphans_removed` | Number of orphan files deleted | +| `remove_orphans.duration_ms` | Time spent on orphan removal | +| `rewrite_manifests.manifests_rewritten` | Number of data manifests consolidated | +| `rewrite_manifests.entries_total` | Total manifest entries in rewritten output | +| `rewrite_manifests.duration_ms` | Time spent on manifest rewriting | + ## Limitations - **Client-side CAS**: the compare-and-swap on metadata version is enforced client-side. Concurrent maintenance on the same table should be avoided (the deduplication key prevents this under normal scheduling). -- **No delete manifest support in compaction**: tables with row-level deletes (delete manifests) skip compaction. -- **No multi-spec compaction**: tables that have undergone partition spec evolution skip compaction. +- **Cannot write delete manifests**: the compactor cannot produce new delete manifests (iceberg-go limitation). Instead, consumed delete manifests are dropped entirely when all referenced data files have been compacted. - **Single filer per job**: detection tries each filer address in the cluster context until one connects, but each execution job uses the single filer address recorded in its proposal. ## See Also