Update Iceberg maintenance wiki for Phase 3

- Document multi-partition-spec compaction (grouping by spec + partition)
- Document delete-aware compaction (position + equality deletes)
- Add apply_deletes config parameter
- Fix target_file_size_bytes → target_file_size_mb (matches actual config)
- Remove obsolete limitations (delete manifests, multi-spec)
- Add structured metrics reference table
- Update skip conditions and rewrite manifests threshold messaging

Iceberg-Table-Maintenance.md

@@ -45,12 +45,14 @@ Compaction merges many small Parquet files within the same partition into fewer,
 
 #### How It Works
 
-1. Read the current snapshot's manifest list and all data manifests
-2. Group small files (below `target_file_size_bytes`) by partition key
+1. Read the current snapshot's manifest list; separate data manifests from delete manifests
+2. Group small data files (below `target_file_size_mb`) by **partition spec ID + partition key** so files from different specs are never mixed
 3. Filter groups to those with at least `min_input_files` entries
-4. For each group (bin), read all source Parquet files and merge rows into a single output file
-5. Write a new manifest with ADDED entries (merged files), DELETED entries (originals), and EXISTING entries (untouched files)
-6. Commit a new snapshot via optimistic concurrency
+4. If `apply_deletes` is enabled and delete manifests exist, collect position deletes and equality deletes from all delete manifest entries
+5. For each group (bin), read all source Parquet files, filter out deleted rows (position deletes via binary search, equality deletes via hash set lookup), and merge remaining rows into a single output file
+6. Write one manifest per partition spec with ADDED entries (merged files), DELETED entries (originals), and EXISTING entries (untouched files)
+7. Carry forward delete manifests if any non-compacted data files remain; drop them if all data files were compacted (deletes fully consumed)
+8. Commit a new snapshot via optimistic concurrency
 
 #### Before / After: Unpartitioned Table
 
@@ -126,14 +128,24 @@ compact-...-1.parquet  (~240 MB, files 9-15)
 (files 16-20 left alone if fewer than min_input_files)
 ```
 
+#### Delete Handling
+
+When `apply_deletes` is enabled (the default), compaction applies both **position deletes** and **equality deletes** during the merge:
+
+- **Position deletes**: the delete file contains `file_path` + `pos` columns indicating specific rows to remove. Paths are normalized so absolute S3 URLs and relative paths match correctly.
+- **Equality deletes**: the delete file specifies equality field IDs and column values. Rows matching those values are filtered out. Different delete files may use different equality columns — deletes are grouped by field ID set.
+
+After compaction, delete manifests whose referenced data files were all compacted are dropped (deletes fully consumed). If any data files remain uncompacted, delete manifests are carried forward.
+
+Set `apply_deletes=false` to revert to the previous behavior of skipping tables with delete manifests.
+
 #### Skip Conditions
 
 | Condition | Result |
 |:---|:---|
-| All files >= `target_file_size_bytes` | `"no files eligible for compaction"` |
+| All files >= `target_file_size_mb` | `"no files eligible for compaction"` |
 | No partition has >= `min_input_files` small files | `"no files eligible for compaction"` |
-| Delete manifests present (row-level deletes) | `"compaction skipped: delete manifests present"` |
-| Multiple partition specs (schema evolution) | `"compaction skipped: multiple partition specs present"` |
+| Delete manifests present and `apply_deletes=false` | `"compaction skipped: delete manifests present and apply_deletes is disabled"` |
 | No current snapshot | `"no current snapshot"` |
 
 ---
@@ -273,7 +285,7 @@ Result: `"rewrote 12 manifests into 1 (320 entries)"`
 
 | Condition | Result |
 |:---|:---|
-| Manifest count < `min_manifests_to_rewrite` | `"only N manifests, below threshold of M"` |
+| Data manifest count < `min_manifests_to_rewrite` | `"only N data manifests, below threshold of M"` |
 | No current snapshot | `"no current snapshot"` |
 | No data entries | `"no data entries to rewrite"` |
 
@@ -306,8 +318,9 @@ These control **how** maintenance operations behave. Set via the Admin UI or pas
 | `snapshot_retention_hours` | `168` (7 days) | Expire snapshots older than this |
 | `max_snapshots_to_keep` | `5` | Always keep at least this many newest snapshots |
 | `orphan_older_than_hours` | `72` (3 days) | Safety window: only delete orphans older than this |
-| `target_file_size_bytes` | `268435456` (256 MB) | Files smaller than this are compaction candidates |
+| `target_file_size_mb` | `256` (MB) | Files smaller than this are compaction candidates |
 | `min_input_files` | `5` | Minimum small files in a partition to trigger compaction |
+| `apply_deletes` | `true` | When true, compaction applies position and equality deletes to data files. When false, tables with delete manifests are skipped |
 | `min_manifests_to_rewrite` | `5` | Minimum manifests before rewriting is triggered |
 | `max_commit_retries` | `5` | Max optimistic concurrency retries on version conflict |
 | `operations` | `all` | Comma-separated list of operations, or `all` |
@@ -384,8 +397,9 @@ Tables matching either condition are proposed as maintenance jobs. The actual op
 Maintenance jobs report progress and results through the worker framework:
 
 - **Activity events**: scan start, scan complete (with table count), per-operation start/complete
-- **Progress updates**: percentage based on completed operations
+- **Progress updates**: percentage based on completed operations, with per-bin granularity during compaction
 - **Job result summary**: per-operation outcomes joined by semicolons
+- **Structured metrics**: per-operation metrics in `OutputValues` with dot-prefixed keys
 
 Example result summary:
 ```
@@ -395,11 +409,29 @@ remove_orphans: removed 4 orphan file(s);
 rewrite_manifests: rewrote 7 manifests into 1 (120 entries)
 ```
 
+### Structured Metrics
+
+Each operation returns structured metrics in the job's `OutputValues` map, keyed with the operation name as prefix:
+
+| Key | Description |
+|:---|:---|
+| `compact.files_merged` | Number of input files merged |
+| `compact.files_written` | Number of output files produced |
+| `compact.bins` | Number of compaction bins processed |
+| `compact.duration_ms` | Time spent on compaction |
+| `expire_snapshots.snapshots_expired` | Number of snapshots removed |
+| `expire_snapshots.files_deleted` | Unreferenced files cleaned up |
+| `expire_snapshots.duration_ms` | Time spent on expiration |
+| `remove_orphans.orphans_removed` | Number of orphan files deleted |
+| `remove_orphans.duration_ms` | Time spent on orphan removal |
+| `rewrite_manifests.manifests_rewritten` | Number of data manifests consolidated |
+| `rewrite_manifests.entries_total` | Total manifest entries in rewritten output |
+| `rewrite_manifests.duration_ms` | Time spent on manifest rewriting |
+
 ## Limitations
 
 - **Client-side CAS**: the compare-and-swap on metadata version is enforced client-side. Concurrent maintenance on the same table should be avoided (the deduplication key prevents this under normal scheduling).
-- **No delete manifest support in compaction**: tables with row-level deletes (delete manifests) skip compaction.
-- **No multi-spec compaction**: tables that have undergone partition spec evolution skip compaction.
+- **Cannot write delete manifests**: the compactor cannot produce new delete manifests (iceberg-go limitation). Instead, consumed delete manifests are dropped entirely when all referenced data files have been compacted.
 - **Single filer per job**: detection tries each filer address in the cluster context until one connects, but each execution job uses the single filer address recorded in its proposal.
 
 ## See Also