From 2aea32e20118430272f677045d2dd20c1496be90 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 7 Mar 2026 12:11:44 -0800 Subject: [PATCH] add iceberg table maintenance --- Iceberg-Table-Maintenance.md | 411 ++++++++++++++++++ ...ntenance-Scripts-to-Admin-Script-Plugin.md | 57 +-- S3-Table-Bucket.md | 1 + Volume-Management.md | 2 +- _Sidebar.md | 1 + 5 files changed, 428 insertions(+), 44 deletions(-) create mode 100644 Iceberg-Table-Maintenance.md diff --git a/Iceberg-Table-Maintenance.md b/Iceberg-Table-Maintenance.md new file mode 100644 index 0000000..0c72f28 --- /dev/null +++ b/Iceberg-Table-Maintenance.md @@ -0,0 +1,411 @@ +# Iceberg Table Maintenance + +## Introduction + +SeaweedFS includes an automated **Iceberg table maintenance** worker that keeps [S3 Table Buckets](S3-Table-Bucket) healthy. Over time, Iceberg tables accumulate small data files, stale snapshots, orphaned files, and fragmented manifests. Left unchecked, this degrades query performance and wastes storage. + +The maintenance worker runs four operations in the recommended Iceberg order: + +1. **Compact** -- merge small Parquet data files into larger ones +2. **Expire Snapshots** -- remove old snapshots and their unreferenced files +3. **Remove Orphans** -- delete files on disk that no snapshot references +4. **Rewrite Manifests** -- consolidate many small manifest files into fewer large ones + +The worker integrates with the [Worker](Worker) framework: an admin server schedules detection scans, the worker proposes tables that need attention, and the admin assigns execution jobs. + +## Prerequisites + +- A running SeaweedFS cluster with S3 Table Bucket support enabled +- At least one `weed worker` instance with `iceberg_maintenance` in its job types +- Tables created via the [S3 Tables API](S3-Table-Bucket-Commands) or the [Iceberg REST Catalog](SeaweedFS-Iceberg-Catalog) + +## Enabling Maintenance + +Iceberg maintenance is **disabled by default**. Enable it through the admin UI or API: + +1. Navigate to **Admin UI > Job Types > Iceberg Maintenance** +2. Toggle **Enabled** to on +3. Configure detection interval, scope filters, and worker thresholds + +Or start a worker with the job type included (it is in the default set): + +```bash +# Default: includes iceberg_maintenance along with other job types +weed worker -admin=localhost:23646 + +# Explicitly specify job types +weed worker -admin=localhost:23646 -jobType=iceberg_maintenance +``` + +## Operations + +### 1. Compact Data Files + +Compaction merges many small Parquet files within the same partition into fewer, larger files. This reduces the number of file opens during queries and improves I/O throughput. + +#### How It Works + +1. Read the current snapshot's manifest list and all data manifests +2. Group small files (below `target_file_size_bytes`) by partition key +3. Filter groups to those with at least `min_input_files` entries +4. For each group (bin), read all source Parquet files and merge rows into a single output file +5. Write a new manifest with ADDED entries (merged files), DELETED entries (originals), and EXISTING entries (untouched files) +6. Commit a new snapshot via optimistic concurrency + +#### Before / After: Unpartitioned Table + +**Before** -- 6 small files totaling 53 MB: + +``` +data/ + file-001.parquet (10 MB) + file-002.parquet (5 MB) + file-003.parquet (8 MB) + file-004.parquet (12 MB) + file-005.parquet (3 MB) + file-006.parquet (15 MB) +metadata/ + v1.metadata.json + snap-100.avro <- manifest list (1 snapshot) + manifest-100.avro <- manifest (6 entries) +``` + +**After** -- 1 merged file: + +``` +data/ + file-001..006.parquet <- still on disk until orphan removal + compact-100--0.parquet (53 MB, new merged file) +metadata/ + v1.metadata.json + v2-.metadata.json <- new metadata + snap-.avro <- new manifest list (2 snapshots) + compact-.avro <- new manifest: + 1 ADDED (merged file) + 6 DELETED (originals) +``` + +The original data files remain on disk. They become unreferenced after `expire_snapshots` drops the old snapshot, and are cleaned up by `remove_orphans`. + +#### Before / After: Partitioned Table + +**Before** -- `region=us` has 5 small files, `region=eu` has only 2: + +``` +data/ + region=us/part-001.parquet (20 MB) + region=us/part-002.parquet (15 MB) + region=us/part-003.parquet (10 MB) + region=us/part-004.parquet (25 MB) + region=us/part-005.parquet (18 MB) + region=eu/part-006.parquet (5 MB) + region=eu/part-007.parquet (8 MB) +``` + +**After** -- only `region=us` is compacted (5 >= `min_input_files`), `region=eu` is untouched (2 < 5): + +``` +data/ + region=us/part-001..005.parquet <- marked DELETED in manifest + region=eu/part-006.parquet <- EXISTING (carried forward) + region=eu/part-007.parquet <- EXISTING (carried forward) + compact-100--0.parquet <- merged US data (88 MB) +``` + +#### Before / After: Oversized Bin (Split) + +When a partition group exceeds `target_file_size_bytes`, it is split into sub-bins: + +**Before** -- 20 small files in one partition, totaling 600 MB (target = 256 MB): + +**After** -- split into multiple output files: + +``` +compact-...-0.parquet (~250 MB, files 1-8) +compact-...-1.parquet (~240 MB, files 9-15) +(files 16-20 left alone if fewer than min_input_files) +``` + +#### Skip Conditions + +| Condition | Result | +|:---|:---| +| All files >= `target_file_size_bytes` | `"no files eligible for compaction"` | +| No partition has >= `min_input_files` small files | `"no files eligible for compaction"` | +| Delete manifests present (row-level deletes) | `"compaction skipped: delete manifests present"` | +| Multiple partition specs (schema evolution) | `"compaction skipped: multiple partition specs present"` | +| No current snapshot | `"no current snapshot"` | + +--- + +### 2. Expire Snapshots + +Removes old snapshots from table metadata and deletes files that are no longer referenced by any remaining snapshot. + +#### How It Works + +1. Sort snapshots by timestamp (newest first) +2. The current snapshot is **always kept** +3. Keep the newest `max_snapshots_to_keep` snapshots regardless of age +4. Among the rest, expire those older than `snapshot_retention_hours` +5. Snapshots within the retention window are kept even if they exceed the count +6. Commit new metadata with expired snapshots removed +7. Delete files exclusively referenced by expired snapshots (best-effort) + +#### Before / After + +**Before** -- table with 8 snapshots, `max_snapshots_to_keep=5`, `snapshot_retention_hours=168` (7 days): + +``` +Snapshots (newest first): + snap-8 (current, 1 hour ago) <- always kept + snap-7 (2 days ago) <- kept (within budget) + snap-6 (3 days ago) <- kept (within budget) + snap-5 (5 days ago) <- kept (within budget) + snap-4 (6 days ago) <- kept (within budget, count=5) + snap-3 (8 days ago) <- EXPIRED (beyond budget + retention) + snap-2 (10 days ago) <- EXPIRED + snap-1 (14 days ago) <- EXPIRED +``` + +**After**: + +``` +Snapshots: + snap-8 (current) + snap-7 + snap-6 + snap-5 + snap-4 + +Deleted: manifest lists, manifests, and data files + exclusively referenced by snap-1, snap-2, snap-3 +``` + +Result: `"expired 3 snapshot(s), deleted 12 unreferenced file(s)"` + +--- + +### 3. Remove Orphans + +Deletes files in the table's `metadata/` and `data/` directories that are not referenced by any snapshot. This catches files left behind by failed writes, interrupted compactions, or other incomplete operations. + +#### How It Works + +1. Collect all file paths referenced by all snapshots (manifest lists, manifests, data files) plus the active metadata file and previous metadata log entries +2. Walk the `metadata/` and `data/` directories on the filer +3. For each file not in the reference set: + - Skip if the file is newer than `orphan_older_than_hours` (safety window) + - Skip if the file has no attributes (unknown age) + - Delete the file + +#### Before / After + +**Before** -- table directory has leftover files from a failed compaction and an old metadata version: + +``` +metadata/ + v1.metadata.json <- referenced (previous metadata log) + v2-.metadata.json <- referenced (current) + snap-100.avro <- referenced + compact-failed.avro <- NOT referenced, 4 days old + v0.metadata.json <- NOT referenced, 10 days old +data/ + file-001.parquet <- referenced + compact-orphan.parquet <- NOT referenced, 5 days old + temp-upload.parquet <- NOT referenced, 1 hour old +``` + +**After** (with `orphan_older_than_hours=72`): + +``` +Deleted: + metadata/compact-failed.avro (4 days > 72h) + metadata/v0.metadata.json (10 days > 72h) + data/compact-orphan.parquet (5 days > 72h) + +Kept: + data/temp-upload.parquet (1 hour < 72h safety window) +``` + +Result: `"removed 3 orphan file(s)"` + +--- + +### 4. Rewrite Manifests + +Consolidates many small manifest files into fewer large ones. When a table accumulates many manifests (from frequent small commits), query engines must open each one to plan scans. Rewriting reduces this overhead. + +#### How It Works + +1. Read all data manifests from the current snapshot's manifest list +2. Skip if manifest count is below `min_input_files` +3. Group manifest entries by partition spec ID (for spec-evolved tables, one output manifest per spec) +4. Write merged manifest(s) containing all entries +5. Carry forward any delete manifests unchanged +6. Write a new manifest list and commit a new snapshot + +#### Before / After + +**Before** -- 12 small manifests from frequent commits: + +``` +metadata/ + manifest-001.avro (50 entries) + manifest-002.avro (10 entries) + manifest-003.avro (5 entries) + ... + manifest-012.avro (8 entries) + snap-500.avro <- manifest list pointing to 12 manifests +``` + +**After** -- 1 merged manifest: + +``` +metadata/ + merged--spec0-.avro (all entries combined) + snap--.avro <- new manifest list (1 manifest) +``` + +Result: `"rewrote 12 manifests into 1 (320 entries)"` + +#### Skip Conditions + +| Condition | Result | +|:---|:---| +| Manifest count < `min_input_files` | `"only N manifests, below threshold"` | +| No current snapshot | `"no current snapshot"` | +| No data entries | `"no data entries to rewrite"` | + +## Configuration Reference + +### Admin Config (Scope Filters) + +These control **which tables** the detection scan considers. Set via the Admin UI under the Iceberg Maintenance job type. + +| Parameter | Default | Description | +|:---|:---|:---| +| `bucket_filter` | *(blank = all)* | Comma-separated wildcard patterns for table bucket names (`*` and `?` supported) | +| `namespace_filter` | *(blank = all)* | Comma-separated wildcard patterns for namespaces | +| `table_filter` | *(blank = all)* | Comma-separated wildcard patterns for table names | + +**Examples:** + +``` +bucket_filter: prod-*, staging-* +namespace_filter: analytics, events-* +table_filter: clicks, orders-* +``` + +### Worker Config (Thresholds) + +These control **how** maintenance operations behave. Set via the Admin UI or passed as worker config values. + +| Parameter | Default | Description | +|:---|:---|:---| +| `snapshot_retention_hours` | `168` (7 days) | Expire snapshots older than this | +| `max_snapshots_to_keep` | `5` | Always keep at least this many newest snapshots | +| `orphan_older_than_hours` | `72` (3 days) | Safety window: only delete orphans older than this | +| `target_file_size_bytes` | `268435456` (256 MB) | Files smaller than this are compaction candidates | +| `min_input_files` | `5` | Minimum files in a partition/manifest group to trigger action | +| `max_commit_retries` | `5` | Max optimistic concurrency retries on version conflict | +| `operations` | `all` | Comma-separated list of operations, or `all` | + +### Operations Selection + +The `operations` parameter controls which operations run and in what order. Operations always execute in canonical order regardless of the order specified: + +``` +compact -> expire_snapshots -> remove_orphans -> rewrite_manifests +``` + +**Examples:** + +``` +operations: all # run everything (default) +operations: expire_snapshots,remove_orphans # snapshots + cleanup only +operations: compact # compaction only +``` + +### Admin Runtime Defaults + +| Setting | Default | Description | +|:---|:---|:---| +| Enabled | `false` | Must be explicitly enabled | +| Detection interval | `3600s` (1 hour) | How often the worker scans for tables needing maintenance | +| Detection timeout | `300s` | Max time for a detection scan | +| Max jobs per detection | `100` | Cap on proposals per scan | +| Global execution concurrency | `4` | Cluster-wide parallel maintenance jobs | +| Per-worker execution concurrency | `2` | Parallel jobs per worker instance | +| Job max runtime | `3600s` (1 hour) | Timeout for a single maintenance job | +| Retry limit | `1` | Scheduler-level retries for failed jobs | + +## Execution Order + +Operations execute in a fixed order that follows Iceberg best practices: + +``` +compact -> expire_snapshots -> remove_orphans -> rewrite_manifests +``` + +This order matters because: + +1. **Compact** creates a new snapshot with merged files and marks originals as deleted +2. **Expire Snapshots** drops old snapshots, making the original (now-deleted) files unreferenced +3. **Remove Orphans** cleans up unreferenced files from disk (including compaction leftovers) +4. **Rewrite Manifests** consolidates the manifests accumulated from compaction and other commits + +If one operation fails, subsequent operations still attempt to run. The job reports partial success with per-operation results. + +## Commit Protocol + +All metadata-modifying operations (compact, expire_snapshots, rewrite_manifests) use an optimistic concurrency protocol: + +1. **Read** current metadata from the table entry's xattr +2. **Plan** the changes (build new manifests, manifest lists, etc.) +3. **Write** new metadata file with a unique name (`v-.metadata.json`) +4. **Compare-and-swap** the table entry's xattr: verify the stored `metadataVersion` matches what was read, then update atomically +5. On **version conflict**, clean up the staged metadata file and retry with exponential backoff (50ms, 100ms, 200ms, ... capped at 5s) + +Each operation also includes a **stale-plan guard**: if the current snapshot has changed since planning began, the operation aborts rather than committing against outdated state. + +## Detection + +The detection scan runs periodically (default: hourly) and evaluates each table using metadata-only heuristics (no manifest reading): + +- Snapshot count exceeds `max_snapshots_to_keep` +- Any snapshot is older than `snapshot_retention_hours` + +Tables matching either condition are proposed as maintenance jobs. The actual operations then perform precise filtering (e.g., `expire_snapshots` checks both conditions together before expiring). + +## Monitoring + +Maintenance jobs report progress and results through the worker framework: + +- **Activity events**: scan start, scan complete (with table count), per-operation start/complete +- **Progress updates**: percentage based on completed operations +- **Job result summary**: per-operation outcomes joined by semicolons + +Example result summary: +``` +compact: compacted 15 files into 3 (across 3 bins); +expire_snapshots: expired 2 snapshot(s), deleted 8 unreferenced file(s); +remove_orphans: removed 4 orphan file(s); +rewrite_manifests: rewrote 7 manifests into 1 (120 entries) +``` + +## Limitations + +- **Inline files only**: metadata and manifest files are expected to be stored inline in filer entries (suitable for their typical small size). Chunked/large file streaming is not supported. +- **Client-side CAS**: the compare-and-swap on metadata version is enforced client-side. Concurrent maintenance on the same table should be avoided (the deduplication key prevents this under normal scheduling). +- **No delete manifest support in compaction**: tables with row-level deletes (delete manifests) skip compaction. +- **No multi-spec compaction**: tables that have undergone partition spec evolution skip compaction. +- **Single filer**: detection and execution connect to one filer address from the cluster context. + +## See Also + +- [[S3 Table Bucket]] -- Table bucket concepts and file layout +- [[S3 Table Bucket Commands]] -- CLI examples for creating and managing table buckets +- [[SeaweedFS Iceberg Catalog]] -- Using the Iceberg REST Catalog with Spark and Trino +- [[S3 Tables Security]] -- IAM permissions for table buckets +- [[Worker]] -- Worker framework overview diff --git a/Migrate-Maintenance-Scripts-to-Admin-Script-Plugin.md b/Migrate-Maintenance-Scripts-to-Admin-Script-Plugin.md index c9ee7d0..f5c3cd0 100644 --- a/Migrate-Maintenance-Scripts-to-Admin-Script-Plugin.md +++ b/Migrate-Maintenance-Scripts-to-Admin-Script-Plugin.md @@ -20,28 +20,7 @@ The admin script plugin worker addresses all of these: ## Migration steps -### 1. Note your current scripts - -Find your `master.toml` (in `./`, `~/.seaweedfs/`, or `/etc/seaweedfs/`) and note the `[master.maintenance]` section: - -```toml -[master.maintenance] -scripts = """ - lock - ec.encode -fullPercent=95 -quietFor=1h - ec.rebuild -apply - ec.balance -apply - fs.log.purge -daysAgo=7 - volume.deleteEmpty -quietFor=24h -apply - volume.balance -apply - volume.fix.replication -apply - s3.clean.uploads -timeAgo=24h - unlock -""" -sleep_minutes = 17 -``` - -### 2. Start the admin server +### 1. Start the admin server ```bash weed admin -master=localhost:9333 @@ -49,7 +28,7 @@ weed admin -master=localhost:9333 The admin server connects to the master. Once connected, the master automatically skips its built-in maintenance scripts (you will see `Skipping master maintenance scripts because admin server is connected` in the master log). -### 3. Start a worker +### 2. Start a worker ```bash weed worker -admin=localhost:23646 @@ -57,25 +36,17 @@ weed worker -admin=localhost:23646 The worker registers with the admin server and begins handling plugin jobs. By default it handles all available job types including `admin_script` and `erasure_coding`. -### 4. Configure the admin script in the UI +### 3. Automatic migration -Open the admin UI at `http://:23646/plugin` and find the **Admin Script** job type. +When the `admin_script` worker registers for the first time, the admin server automatically fetches your maintenance scripts from `master.toml` and uses them as the default admin script configuration. The `lock`/`unlock` commands are stripped automatically since the admin script worker handles locking on its own. The `sleep_minutes` value is preserved as the run interval. -The default admin script is: +If the admin server cannot reach the master or the `master.toml` has no maintenance scripts configured, the worker's built-in defaults are used instead. -``` -ec.balance -apply -fs.log.purge -daysAgo=7 -volume.deleteEmpty -quietFor=24h -apply -volume.fix.replication -apply -s3.clean.uploads -timeAgo=24h -``` - -The default run interval is **17 minutes**, matching the legacy default. +You can review and adjust the imported script in the admin UI at `http://:23646/plugin` under the **Admin Script** job type. #### Adjustments to make -Compare your old `master.toml` scripts with the defaults above and customize as needed: +Compare the imported script with the table below and customize as needed: | Old command | What to do | |---|---| @@ -91,9 +62,9 @@ Compare your old `master.toml` scripts with the defaults above and customize as | `volume.tier.upload -dest s3 ...` | Add to the admin script if you had tiered storage upload. | | Any custom commands | Add them to the script in the UI. | -If you had a custom `sleep_minutes` value, set the **Run Interval (minutes)** field to match. +Your `sleep_minutes` value is automatically imported as the run interval. Adjust it in the UI if needed. -### 5. Verify +### 4. Verify After the first run interval elapses, check the admin UI at `/plugin` for: @@ -103,7 +74,7 @@ After the first run interval elapses, check the admin UI at `/plugin` for: You can also trigger a manual run from the UI to verify immediately. -### 6. Clean up master.toml (optional) +### 5. Clean up master.toml (optional) Once you have confirmed the admin script plugin worker is running correctly, you can remove or blank out the `[master.maintenance]` section in `master.toml`: @@ -127,15 +98,15 @@ Some shell commands use different flag names between the legacy scripts and the ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ weed master │◄────────│ weed admin │────────►│ weed worker │ │ │ gRPC │ :23646 │ gRPC │ │ -│ master.toml │ │ Admin UI │ │ admin_script │ -│ [maintenance│ │ /plugin │ │ erasure_coding│ -│ scripts │ │ │ │ vacuum │ +│ master.toml │ ─────── │ Admin UI │ │ admin_script │ +│ [maintenance│ scripts │ /plugin │ │ erasure_coding│ +│ scripts │ imported│ │ │ vacuum │ │ SKIPPED] │ │ │ │ balance │ └──────────────┘ └──────────────┘ └──────────────┘ ``` - The **master** still has the legacy maintenance code but skips it when an admin server is connected. -- The **admin server** manages plugin configuration, scheduling, and dispatching. +- The **admin server** manages plugin configuration, scheduling, and dispatching. On first worker registration, it imports maintenance scripts from the master as admin_script defaults. - The **worker** executes admin script commands and dedicated plugin jobs (erasure coding, vacuum, balance). ## See also diff --git a/S3-Table-Bucket.md b/S3-Table-Bucket.md index 9d289c2..e685d41 100644 --- a/S3-Table-Bucket.md +++ b/S3-Table-Bucket.md @@ -126,4 +126,5 @@ sequenceDiagram - [S3 Table Bucket Commands](S3-Table-Bucket-Commands) - Examples of how to create and manage table buckets. - [SeaweedFS Iceberg Catalog](SeaweedFS-Iceberg-Catalog) - Usage with Apache Spark and Trino. +- [Iceberg Table Maintenance](Iceberg-Table-Maintenance) - Automated compaction, snapshot expiration, orphan removal, and manifest rewriting. - [S3 Tables Security](S3-Tables-Security) - IAM permissions and policy reference. diff --git a/Volume-Management.md b/Volume-Management.md index bbae383..871e78f 100644 --- a/Volume-Management.md +++ b/Volume-Management.md @@ -34,7 +34,7 @@ Several commands that were previously part of the maintenance script now have de See the [[Worker]] page for more details on `weed worker` options and capabilities. -> **Legacy note:** Previously, maintenance scripts were configured in `master.toml` under `[master.maintenance]`. That mechanism still exists as a fallback but is automatically skipped when an admin server is connected. New deployments should use the admin script plugin worker instead. See [[Migrate Maintenance Scripts to Admin Script Plugin]] for a step-by-step migration guide. +> **Legacy note:** Previously, maintenance scripts were configured in `master.toml` under `[master.maintenance]`. That mechanism still exists as a fallback but is automatically skipped when an admin server is connected. When migrating, the admin server automatically imports your `master.toml` maintenance scripts as the default admin script configuration. See [[Migrate Maintenance Scripts to Admin Script Plugin]] for details. ## Fix missing volumes When running large clusters, it is common that some volume servers are down. If a volume is replicated and one replica is missing, the volume will be marked as readonly. diff --git a/_Sidebar.md b/_Sidebar.md index 6c4a924..976c37e 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -88,6 +88,7 @@ ### S3 Table Bucket * [[S3 Table Bucket]] * [[SeaweedFS Iceberg Catalog]] +* [[Iceberg Table Maintenance]] * [[S3 Tables Security]] ### S3 Authentication & IAM