Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-10-18 12:08:32 +02:00
Code Issues Packages Projects Releases Wiki Activity

add iceberg table maintenance

Chris Lu committed 2026-03-07 12:11:44 -08:00
1 parent 09d1004740
commit 2aea32e201
5 files changed
+428 -44
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

Iceberg-Table-Maintenance.md
+411
@@ -0,0 +1,411 @@
# Iceberg Table Maintenance
## Introduction
SeaweedFS includes an automated **Iceberg table maintenance** worker that keeps [S3 Table Buckets](S3-Table-Bucket) healthy. Over time, Iceberg tables accumulate small data files, stale snapshots, orphaned files, and fragmented manifests. Left unchecked, this degrades query performance and wastes storage.
The maintenance worker runs four operations in the recommended Iceberg order:
1. **Compact** -- merge small Parquet data files into larger ones
2. **Expire Snapshots** -- remove old snapshots and their unreferenced files
3. **Remove Orphans** -- delete files on disk that no snapshot references
4. **Rewrite Manifests** -- consolidate many small manifest files into fewer large ones
The worker integrates with the [Worker](Worker) framework: an admin server schedules detection scans, the worker proposes tables that need attention, and the admin assigns execution jobs.
## Prerequisites
- A running SeaweedFS cluster with S3 Table Bucket support enabled
- At least one `weed worker` instance with `iceberg_maintenance` in its job types
- Tables created via the [S3 Tables API](S3-Table-Bucket-Commands) or the [Iceberg REST Catalog](SeaweedFS-Iceberg-Catalog)
## Enabling Maintenance
Iceberg maintenance is **disabled by default**. Enable it through the admin UI or API:
1. Navigate to **Admin UI > Job Types > Iceberg Maintenance**
2. Toggle **Enabled** to on
3. Configure detection interval, scope filters, and worker thresholds
Or start a worker with the job type included (it is in the default set):
```bash
# Default: includes iceberg_maintenance along with other job types
weed worker -admin=localhost:23646
# Explicitly specify job types
weed worker -admin=localhost:23646 -jobType=iceberg_maintenance
```
## Operations
### 1. Compact Data Files
Compaction merges many small Parquet files within the same partition into fewer, larger files. This reduces the number of file opens during queries and improves I/O throughput.
#### How It Works
1. Read the current snapshot's manifest list and all data manifests
2. Group small files (below `target_file_size_bytes`) by partition key
3. Filter groups to those with at least `min_input_files` entries
4. For each group (bin), read all source Parquet files and merge rows into a single output file
5. Write a new manifest with ADDED entries (merged files), DELETED entries (originals), and EXISTING entries (untouched files)
6. Commit a new snapshot via optimistic concurrency
#### Before / After: Unpartitioned Table
**Before** -- 6 small files totaling 53 MB:
```
data/
file-001.parquet (10 MB)
file-002.parquet (5 MB)
file-003.parquet (8 MB)
file-004.parquet (12 MB)
file-005.parquet (3 MB)
file-006.parquet (15 MB)
metadata/
v1.metadata.json
snap-100.avro <- manifest list (1 snapshot)
manifest-100.avro <- manifest (6 entries)
```
**After** -- 1 merged file:
```
data/
file-001..006.parquet <- still on disk until orphan removal
compact-100-<ts>-0.parquet (53 MB, new merged file)
metadata/
v1.metadata.json
v2-<nonce>.metadata.json <- new metadata
snap-<ts>.avro <- new manifest list (2 snapshots)
compact-<ts>.avro <- new manifest:
1 ADDED (merged file)
6 DELETED (originals)
```
The original data files remain on disk. They become unreferenced after `expire_snapshots` drops the old snapshot, and are cleaned up by `remove_orphans`.
#### Before / After: Partitioned Table
**Before** -- `region=us` has 5 small files, `region=eu` has only 2:
```
data/
region=us/part-001.parquet (20 MB)
region=us/part-002.parquet (15 MB)
region=us/part-003.parquet (10 MB)
region=us/part-004.parquet (25 MB)
region=us/part-005.parquet (18 MB)
region=eu/part-006.parquet (5 MB)
region=eu/part-007.parquet (8 MB)
```
**After** -- only `region=us` is compacted (5 >= `min_input_files`), `region=eu` is untouched (2 < 5):
```
data/
region=us/part-001..005.parquet <- marked DELETED in manifest
region=eu/part-006.parquet <- EXISTING (carried forward)
region=eu/part-007.parquet <- EXISTING (carried forward)
compact-100-<ts>-0.parquet <- merged US data (88 MB)
```
#### Before / After: Oversized Bin (Split)
When a partition group exceeds `target_file_size_bytes`, it is split into sub-bins:
**Before** -- 20 small files in one partition, totaling 600 MB (target = 256 MB):
**After** -- split into multiple output files:
```
compact-...-0.parquet (~250 MB, files 1-8)
compact-...-1.parquet (~240 MB, files 9-15)
(files 16-20 left alone if fewer than min_input_files)
```
#### Skip Conditions
| Condition | Result |
|:---|:---|
| All files >= `target_file_size_bytes` | `"no files eligible for compaction"` |
| No partition has >= `min_input_files` small files | `"no files eligible for compaction"` |
| Delete manifests present (row-level deletes) | `"compaction skipped: delete manifests present"` |
| Multiple partition specs (schema evolution) | `"compaction skipped: multiple partition specs present"` |
| No current snapshot | `"no current snapshot"` |
---
### 2. Expire Snapshots
Removes old snapshots from table metadata and deletes files that are no longer referenced by any remaining snapshot.
#### How It Works
1. Sort snapshots by timestamp (newest first)
2. The current snapshot is **always kept**
3. Keep the newest `max_snapshots_to_keep` snapshots regardless of age
4. Among the rest, expire those older than `snapshot_retention_hours`
5. Snapshots within the retention window are kept even if they exceed the count
6. Commit new metadata with expired snapshots removed
7. Delete files exclusively referenced by expired snapshots (best-effort)
#### Before / After
**Before** -- table with 8 snapshots, `max_snapshots_to_keep=5`, `snapshot_retention_hours=168` (7 days):
```
Snapshots (newest first):
snap-8 (current, 1 hour ago) <- always kept
snap-7 (2 days ago) <- kept (within budget)
snap-6 (3 days ago) <- kept (within budget)
snap-5 (5 days ago) <- kept (within budget)
snap-4 (6 days ago) <- kept (within budget, count=5)
snap-3 (8 days ago) <- EXPIRED (beyond budget + retention)
snap-2 (10 days ago) <- EXPIRED
snap-1 (14 days ago) <- EXPIRED
```
**After**:
```
Snapshots:
snap-8 (current)
snap-7
snap-6
snap-5
snap-4
Deleted: manifest lists, manifests, and data files
exclusively referenced by snap-1, snap-2, snap-3
```
Result: `"expired 3 snapshot(s), deleted 12 unreferenced file(s)"`
---
### 3. Remove Orphans
Deletes files in the table's `metadata/` and `data/` directories that are not referenced by any snapshot. This catches files left behind by failed writes, interrupted compactions, or other incomplete operations.
#### How It Works
1. Collect all file paths referenced by all snapshots (manifest lists, manifests, data files) plus the active metadata file and previous metadata log entries
2. Walk the `metadata/` and `data/` directories on the filer
3. For each file not in the reference set:
- Skip if the file is newer than `orphan_older_than_hours` (safety window)
- Skip if the file has no attributes (unknown age)
- Delete the file
#### Before / After
**Before** -- table directory has leftover files from a failed compaction and an old metadata version:
```
metadata/
v1.metadata.json <- referenced (previous metadata log)
v2-<nonce>.metadata.json <- referenced (current)
snap-100.avro <- referenced
compact-failed.avro <- NOT referenced, 4 days old
v0.metadata.json <- NOT referenced, 10 days old
data/
file-001.parquet <- referenced
compact-orphan.parquet <- NOT referenced, 5 days old
temp-upload.parquet <- NOT referenced, 1 hour old
```
**After** (with `orphan_older_than_hours=72`):
```
Deleted:
metadata/compact-failed.avro (4 days > 72h)
metadata/v0.metadata.json (10 days > 72h)
data/compact-orphan.parquet (5 days > 72h)
Kept:
data/temp-upload.parquet (1 hour < 72h safety window)
```
Result: `"removed 3 orphan file(s)"`
---
### 4. Rewrite Manifests
Consolidates many small manifest files into fewer large ones. When a table accumulates many manifests (from frequent small commits), query engines must open each one to plan scans. Rewriting reduces this overhead.
#### How It Works
1. Read all data manifests from the current snapshot's manifest list
2. Skip if manifest count is below `min_input_files`
3. Group manifest entries by partition spec ID (for spec-evolved tables, one output manifest per spec)
4. Write merged manifest(s) containing all entries
5. Carry forward any delete manifests unchanged
6. Write a new manifest list and commit a new snapshot
#### Before / After
**Before** -- 12 small manifests from frequent commits:
```
metadata/
manifest-001.avro (50 entries)
manifest-002.avro (10 entries)
manifest-003.avro (5 entries)
...
manifest-012.avro (8 entries)
snap-500.avro <- manifest list pointing to 12 manifests
```
**After** -- 1 merged manifest:
```
metadata/
merged-<ts>-spec0-<ts>.avro (all entries combined)
snap-<ts>-<ts>.avro <- new manifest list (1 manifest)
```
Result: `"rewrote 12 manifests into 1 (320 entries)"`
#### Skip Conditions
| Condition | Result |
|:---|:---|
| Manifest count < `min_input_files` | `"only N manifests, below threshold"` |
| No current snapshot | `"no current snapshot"` |
| No data entries | `"no data entries to rewrite"` |
## Configuration Reference
### Admin Config (Scope Filters)
These control **which tables** the detection scan considers. Set via the Admin UI under the Iceberg Maintenance job type.
| Parameter | Default | Description |
|:---|:---|:---|
| `bucket_filter` | *(blank = all)* | Comma-separated wildcard patterns for table bucket names (`*` and `?` supported) |
| `namespace_filter` | *(blank = all)* | Comma-separated wildcard patterns for namespaces |
| `table_filter` | *(blank = all)* | Comma-separated wildcard patterns for table names |
**Examples:**
```
bucket_filter: prod-*, staging-*
namespace_filter: analytics, events-*
table_filter: clicks, orders-*
```
### Worker Config (Thresholds)
These control **how** maintenance operations behave. Set via the Admin UI or passed as worker config values.
| Parameter | Default | Description |
|:---|:---|:---|
| `snapshot_retention_hours` | `168` (7 days) | Expire snapshots older than this |
| `max_snapshots_to_keep` | `5` | Always keep at least this many newest snapshots |
| `orphan_older_than_hours` | `72` (3 days) | Safety window: only delete orphans older than this |
| `target_file_size_bytes` | `268435456` (256 MB) | Files smaller than this are compaction candidates |
| `min_input_files` | `5` | Minimum files in a partition/manifest group to trigger action |
| `max_commit_retries` | `5` | Max optimistic concurrency retries on version conflict |
| `operations` | `all` | Comma-separated list of operations, or `all` |
### Operations Selection
The `operations` parameter controls which operations run and in what order. Operations always execute in canonical order regardless of the order specified:
```
compact -> expire_snapshots -> remove_orphans -> rewrite_manifests
```
**Examples:**
```
operations: all # run everything (default)
operations: expire_snapshots,remove_orphans # snapshots + cleanup only
operations: compact # compaction only
```
### Admin Runtime Defaults
| Setting | Default | Description |
|:---|:---|:---|
| Enabled | `false` | Must be explicitly enabled |
| Detection interval | `3600s` (1 hour) | How often the worker scans for tables needing maintenance |
| Detection timeout | `300s` | Max time for a detection scan |
| Max jobs per detection | `100` | Cap on proposals per scan |
| Global execution concurrency | `4` | Cluster-wide parallel maintenance jobs |
| Per-worker execution concurrency | `2` | Parallel jobs per worker instance |
| Job max runtime | `3600s` (1 hour) | Timeout for a single maintenance job |
| Retry limit | `1` | Scheduler-level retries for failed jobs |
## Execution Order
Operations execute in a fixed order that follows Iceberg best practices:
```
compact -> expire_snapshots -> remove_orphans -> rewrite_manifests
```
This order matters because:
1. **Compact** creates a new snapshot with merged files and marks originals as deleted
2. **Expire Snapshots** drops old snapshots, making the original (now-deleted) files unreferenced
3. **Remove Orphans** cleans up unreferenced files from disk (including compaction leftovers)
4. **Rewrite Manifests** consolidates the manifests accumulated from compaction and other commits
If one operation fails, subsequent operations still attempt to run. The job reports partial success with per-operation results.
## Commit Protocol
All metadata-modifying operations (compact, expire_snapshots, rewrite_manifests) use an optimistic concurrency protocol:
1. **Read** current metadata from the table entry's xattr
2. **Plan** the changes (build new manifests, manifest lists, etc.)
3. **Write** new metadata file with a unique name (`v<N>-<timestamp>.metadata.json`)
4. **Compare-and-swap** the table entry's xattr: verify the stored `metadataVersion` matches what was read, then update atomically
5. On **version conflict**, clean up the staged metadata file and retry with exponential backoff (50ms, 100ms, 200ms, ... capped at 5s)
Each operation also includes a **stale-plan guard**: if the current snapshot has changed since planning began, the operation aborts rather than committing against outdated state.
## Detection
The detection scan runs periodically (default: hourly) and evaluates each table using metadata-only heuristics (no manifest reading):
- Snapshot count exceeds `max_snapshots_to_keep`
- Any snapshot is older than `snapshot_retention_hours`
Tables matching either condition are proposed as maintenance jobs. The actual operations then perform precise filtering (e.g., `expire_snapshots` checks both conditions together before expiring).
## Monitoring
Maintenance jobs report progress and results through the worker framework:
- **Activity events**: scan start, scan complete (with table count), per-operation start/complete
- **Progress updates**: percentage based on completed operations
- **Job result summary**: per-operation outcomes joined by semicolons
Example result summary:
```
compact: compacted 15 files into 3 (across 3 bins);
expire_snapshots: expired 2 snapshot(s), deleted 8 unreferenced file(s);
remove_orphans: removed 4 orphan file(s);
rewrite_manifests: rewrote 7 manifests into 1 (120 entries)
```
## Limitations
- **Inline files only**: metadata and manifest files are expected to be stored inline in filer entries (suitable for their typical small size). Chunked/large file streaming is not supported.
- **Client-side CAS**: the compare-and-swap on metadata version is enforced client-side. Concurrent maintenance on the same table should be avoided (the deduplication key prevents this under normal scheduling).
- **No delete manifest support in compaction**: tables with row-level deletes (delete manifests) skip compaction.
- **No multi-spec compaction**: tables that have undergone partition spec evolution skip compaction.
- **Single filer**: detection and execution connect to one filer address from the cluster context.
## See Also
- [[S3 Table Bucket]] -- Table bucket concepts and file layout
- [[S3 Table Bucket Commands]] -- CLI examples for creating and managing table buckets
- [[SeaweedFS Iceberg Catalog]] -- Using the Iceberg REST Catalog with Spark and Trino
- [[S3 Tables Security]] -- IAM permissions for table buckets
- [[Worker]] -- Worker framework overview
Migrate-Maintenance-Scripts-to-Admin-Script-Plugin.md
+14 -43
@@ -20,28 +20,7 @@ The admin script plugin worker addresses all of these:
## Migration steps
### 1. Note your current scripts
Find your `master.toml` (in `./`, `~/.seaweedfs/`, or `/etc/seaweedfs/`) and note the `[master.maintenance]` section:
```toml
[master.maintenance]
scripts = """
lock
ec.encode -fullPercent=95 -quietFor=1h
ec.rebuild -apply
ec.balance -apply
fs.log.purge -daysAgo=7
volume.deleteEmpty -quietFor=24h -apply
volume.balance -apply
volume.fix.replication -apply
s3.clean.uploads -timeAgo=24h
unlock
"""
sleep_minutes = 17
```
### 2. Start the admin server
### 1. Start the admin server
```bash
weed admin -master=localhost:9333
@@ -49,7 +28,7 @@ weed admin -master=localhost:9333
The admin server connects to the master. Once connected, the master automatically skips its built-in maintenance scripts (you will see `Skipping master maintenance scripts because admin server is connected` in the master log).
### 3. Start a worker
### 2. Start a worker
```bash
weed worker -admin=localhost:23646
@@ -57,25 +36,17 @@ weed worker -admin=localhost:23646
The worker registers with the admin server and begins handling plugin jobs. By default it handles all available job types including `admin_script` and `erasure_coding`.
### 4. Configure the admin script in the UI
### 3. Automatic migration
Open the admin UI at `http://<admin-host>:23646/plugin` and find the **Admin Script** job type.
When the `admin_script` worker registers for the first time, the admin server automatically fetches your maintenance scripts from `master.toml` and uses them as the default admin script configuration. The `lock`/`unlock` commands are stripped automatically since the admin script worker handles locking on its own. The `sleep_minutes` value is preserved as the run interval.
The default admin script is:
If the admin server cannot reach the master or the `master.toml` has no maintenance scripts configured, the worker's built-in defaults are used instead.
```
ec.balance -apply
fs.log.purge -daysAgo=7
volume.deleteEmpty -quietFor=24h -apply
volume.fix.replication -apply
s3.clean.uploads -timeAgo=24h
```
The default run interval is **17 minutes**, matching the legacy default.
You can review and adjust the imported script in the admin UI at `http://<admin-host>:23646/plugin` under the **Admin Script** job type.
#### Adjustments to make
Compare your old `master.toml` scripts with the defaults above and customize as needed:
Compare the imported script with the table below and customize as needed:
| Old command | What to do |
|---|---|
@@ -91,9 +62,9 @@ Compare your old `master.toml` scripts with the defaults above and customize as
| `volume.tier.upload -dest s3 ...` | Add to the admin script if you had tiered storage upload. |
| Any custom commands | Add them to the script in the UI. |
If you had a custom `sleep_minutes` value, set the **Run Interval (minutes)** field to match.
Your `sleep_minutes` value is automatically imported as the run interval. Adjust it in the UI if needed.
### 5. Verify
### 4. Verify
After the first run interval elapses, check the admin UI at `/plugin` for:
@@ -103,7 +74,7 @@ After the first run interval elapses, check the admin UI at `/plugin` for:
You can also trigger a manual run from the UI to verify immediately.
### 6. Clean up master.toml (optional)
### 5. Clean up master.toml (optional)
Once you have confirmed the admin script plugin worker is running correctly, you can remove or blank out the `[master.maintenance]` section in `master.toml`:
@@ -127,15 +98,15 @@ Some shell commands use different flag names between the legacy scripts and the
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ weed master │◄────────│ weed admin │────────►│ weed worker │
│ │ gRPC │ :23646 │ gRPC │ │
│ master.toml │ │ Admin UI │ │ admin_script │
│ [maintenance│ │ /plugin │ │ erasure_coding│
│ scripts │ │ │ │ vacuum │
│ master.toml │ ─────── │ Admin UI │ │ admin_script │
│ [maintenance│ scripts │ /plugin │ │ erasure_coding│
│ scripts │ imported│ │ │ vacuum │
│ SKIPPED] │ │ │ │ balance │
└──────────────┘ └──────────────┘ └──────────────┘
```
- The **master** still has the legacy maintenance code but skips it when an admin server is connected.
- The **admin server** manages plugin configuration, scheduling, and dispatching.
- The **admin server** manages plugin configuration, scheduling, and dispatching. On first worker registration, it imports maintenance scripts from the master as admin_script defaults.
- The **worker** executes admin script commands and dedicated plugin jobs (erasure coding, vacuum, balance).
## See also
S3-Table-Bucket.md
+1
@@ -126,4 +126,5 @@ sequenceDiagram
- [S3 Table Bucket Commands](S3-Table-Bucket-Commands) - Examples of how to create and manage table buckets.
- [SeaweedFS Iceberg Catalog](SeaweedFS-Iceberg-Catalog) - Usage with Apache Spark and Trino.
- [Iceberg Table Maintenance](Iceberg-Table-Maintenance) - Automated compaction, snapshot expiration, orphan removal, and manifest rewriting.
- [S3 Tables Security](S3-Tables-Security) - IAM permissions and policy reference.
Volume-Management.md
+1 -1
@@ -34,7 +34,7 @@ Several commands that were previously part of the maintenance script now have de
See the [[Worker]] page for more details on `weed worker` options and capabilities.
> **Legacy note:** Previously, maintenance scripts were configured in `master.toml` under `[master.maintenance]`. That mechanism still exists as a fallback but is automatically skipped when an admin server is connected. New deployments should use the admin script plugin worker instead. See [[Migrate Maintenance Scripts to Admin Script Plugin]] for a step-by-step migration guide.
> **Legacy note:** Previously, maintenance scripts were configured in `master.toml` under `[master.maintenance]`. That mechanism still exists as a fallback but is automatically skipped when an admin server is connected. When migrating, the admin server automatically imports your `master.toml` maintenance scripts as the default admin script configuration. See [[Migrate Maintenance Scripts to Admin Script Plugin]] for details.
## Fix missing volumes
When running large clusters, it is common that some volume servers are down. If a volume is replicated and one replica is missing, the volume will be marked as readonly.
_Sidebar.md
+1
@@ -88,6 +88,7 @@
### S3 Table Bucket
* [[S3 Table Bucket]]
* [[SeaweedFS Iceberg Catalog]]
* [[Iceberg Table Maintenance]]
* [[S3 Tables Security]]
### S3 Authentication & IAM