document the S3 Tables maintenance configuration APIs

The maintenance page presented the worker config as the only source of
truth. Table properties now win over it, so the page said the wrong thing
about every threshold it documented.

Adds where settings come from, the five new APIs and their settings, the
new table_properties_override and rewrite_strategy=auto knobs, the IAM
action names, and CLI walkthroughs.
Chris Lu committed 2026-08-16 10:39:38 -07:00
1 parent c950d27415
commit 5febe6078f
4 files changed
+188 -1

No files matched your search

+98 -1
@@ -309,9 +309,32 @@ namespace_filter: analytics, events-*
table_filter: clicks, orders-*
```
### Where Settings Come From
A table's effective settings are resolved from three places, most specific first:
1. **The table's own Iceberg properties.** What every engine already writes, and what wins by default.
2. **The maintenance configuration** set through `PutTableMaintenanceConfiguration` (per table) or `PutTableBucketMaintenanceConfiguration` (per bucket).
3. **The worker config below**, which is the cluster-wide default.
These Iceberg table properties override the worker config:
| Property | Overrides | Iceberg default |
|:---|:---|:---|
| `write.target-file-size-bytes` | `target_file_size_mb` | 512 MiB |
| `write.delete.target-file-size-bytes` | `delete_target_file_size_mb` | 64 MiB |
| `history.expire.max-snapshot-age-ms` | `snapshot_retention_hours` | 5 days |
| `history.expire.min-snapshots-to-keep` | `max_snapshots_to_keep` | 1 |
Properties win because the compactor has to agree with whoever writes the files: a writer honouring `write.target-file-size-bytes` and a compactor rewriting to a different size would rewrite each other's output forever. Set `table_properties_override` to `false` to invert this and make the maintenance configuration authoritative.
A malformed, zero or negative property is logged and ignored, so a misconfigured table falls back to the worker config rather than stopping its own maintenance.
**Enablement is not part of this.** Iceberg has no property meaning "do not compact this table", so a maintenance configuration with `"status": "disabled"` removes that type's operations outright and no property re-enables them. See [Maintenance Configuration API](#maintenance-configuration-api).
### Worker Config (Thresholds)
These control **how** maintenance operations behave. Set via the Admin UI or passed as worker config values.
These control **how** maintenance operations behave, wherever a table does not say otherwise. Set via the Admin UI or passed as worker config values.
| Parameter | Default | Description |
|:---|:---|:---|
@@ -324,6 +347,8 @@ These control **how** maintenance operations behave. Set via the Admin UI or pas
| `min_manifests_to_rewrite` | `5` | Minimum manifests before rewriting is triggered |
| `max_commit_retries` | `5` | Max optimistic concurrency retries on version conflict |
| `operations` | `all` | Comma-separated list of operations, or `all` |
| `rewrite_strategy` | `binpack` | `binpack` keeps the existing row order; `sort` rewrites each bin using the table sort order; `auto` sorts tables that declare one and bin-packs the rest |
| `table_properties_override` | `true` | Let a table's own Iceberg properties override this config and its maintenance configuration |
### Operations Selection
@@ -354,6 +379,77 @@ operations: compact # compaction only
| Job max runtime | `3600s` (1 hour) | Timeout for a single maintenance job |
| Retry limit | `1` | Scheduler-level retries for failed jobs |
## Maintenance Configuration API
Five S3 Tables actions configure maintenance per table or per table bucket, and report what the last run did. They are reachable through the AWS CLI and SDKs at their published REST paths.
| Action | Scope | Types |
|:---|:---|:---|
| `PutTableMaintenanceConfiguration` | Table | `icebergCompaction`, `icebergSnapshotManagement` |
| `GetTableMaintenanceConfiguration` | Table | |
| `GetTableMaintenanceJobStatus` | Table | |
| `PutTableBucketMaintenanceConfiguration` | Table bucket | `icebergUnreferencedFileRemoval` |
| `GetTableBucketMaintenanceConfiguration` | Table bucket | |
### Settings
Every numeric setting is an integer between 1 and 2147483647. Values outside that range are rejected rather than stored and then ignored.
| Type | Setting | Effect |
|:---|:---|:---|
| `icebergCompaction` | `targetFileSizeMB` | Files smaller than this are compaction candidates |
| `icebergCompaction` | `strategy` | `auto`, `binpack` or `sort` |
| `icebergSnapshotManagement` | `minSnapshotsToKeep` | Always keep at least this many newest snapshots |
| `icebergSnapshotManagement` | `maxSnapshotAgeHours` | Expire snapshots older than this |
| `icebergUnreferencedFileRemoval` | `unreferencedDays` | Days a file must be unreferenced before it counts as non-current |
| `icebergUnreferencedFileRemoval` | `nonCurrentDays` | Further days before a non-current file is deleted |
Orphan removal deletes in one step rather than marking files non-current first, so a file is deleted once it is older than `unreferencedDays` **plus** `nonCurrentDays`. A `3`/`10` configuration deletes on day thirteen.
### Disabling maintenance
```bash
aws s3tables put-table-maintenance-configuration \
--table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/analytics \
--namespace sales --name orders \
--type icebergSnapshotManagement \
--value '{"status":"disabled"}' \
--endpoint-url http://localhost:8333
```
A disabled type stops running for that table immediately. This is the one control no table property can override.
### Setting a target file size
```bash
aws s3tables put-table-maintenance-configuration \
--table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/analytics \
--namespace sales --name orders \
--type icebergCompaction \
--value '{"status":"enabled","settings":{"icebergCompaction":{"targetFileSizeMB":512,"strategy":"auto"}}}' \
--endpoint-url http://localhost:8333
```
Remember that `write.target-file-size-bytes` on the table beats this unless `table_properties_override` is `false`.
### Reading job status
```bash
aws s3tables get-table-maintenance-job-status \
--table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/analytics \
--namespace sales --name orders \
--endpoint-url http://localhost:8333
```
Every job type is reported as `Not_Yet_Run`, `Successful`, `Failed` or `Disabled`, with the timestamp of the last run and a failure message where there is one. Manifest rewriting and delete-file rewriting have no AWS equivalent and are reported under `icebergCompaction`.
### Differences from AWS
- `GetTableMaintenanceConfiguration` returns an empty configuration for a table that has never been configured, rather than synthesising defaults. The effective values come from the worker config, which this API cannot express.
- Job status is last-run only. There is no retained history behind it.
- The `z-order` compaction strategy is rejected, because the worker cannot produce that layout and accepting it would mean storing one strategy and running another.
- Disabling `icebergCompaction` also stops manifest and delete-file rewriting, since those ride with it.
## Execution Order
Operations execute in a fixed order that follows Iceberg best practices:
@@ -431,6 +527,7 @@ Each operation returns structured metrics in the job's `OutputValues` map, keyed
## Limitations
- **Client-side CAS**: the compare-and-swap on metadata version is enforced client-side. Concurrent maintenance on the same table should be avoided (the deduplication key prevents this under normal scheduling).
- **Bucket-scoped file removal**: as in AWS, `icebergUnreferencedFileRemoval` is configured on the table bucket rather than per table.
- **Single filer per job**: detection tries each filer address in the cluster context until one connects, but each execution job uses the single filer address recorded in its proposal.
## See Also
+84
@@ -217,6 +217,90 @@ curl -X POST $S3_ENDPOINT/ \
-d '{"resourceArn": "arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket"}'
```
## 5. Maintenance Configuration
Compaction and snapshot management are configured per table; unreferenced file
removal is configured on the table bucket. See
[Iceberg Table Maintenance](Iceberg-Table-Maintenance) for what the worker does
with these settings, and for the table properties that take precedence over them.
### Put Table Maintenance Configuration
**AWS CLI:**
```bash
aws s3tables put-table-maintenance-configuration --endpoint-url $S3_ENDPOINT \
--table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket \
--namespace my_namespace \
--name my_table \
--type icebergCompaction \
--value '{"status":"enabled","settings":{"icebergCompaction":{"targetFileSizeMB":512,"strategy":"auto"}}}'
```
**curl:**
```bash
curl -X PUT "$S3_ENDPOINT/tables/arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket/my_namespace/my_table/maintenance/icebergCompaction" \
-H "Content-Type: application/json" \
-d '{
"value": {
"status": "enabled",
"settings": {"icebergCompaction": {"targetFileSizeMB": 512, "strategy": "auto"}}
}
}'
```
Turn a maintenance type off for a table:
```bash
aws s3tables put-table-maintenance-configuration --endpoint-url $S3_ENDPOINT \
--table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket \
--namespace my_namespace \
--name my_table \
--type icebergSnapshotManagement \
--value '{"status":"disabled"}'
```
### Get Table Maintenance Configuration
**AWS CLI:**
```bash
aws s3tables get-table-maintenance-configuration --endpoint-url $S3_ENDPOINT \
--table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket \
--namespace my_namespace \
--name my_table
```
A table that has never been configured returns an empty configuration; the
effective values then come from the worker config.
### Get Table Maintenance Job Status
**AWS CLI:**
```bash
aws s3tables get-table-maintenance-job-status --endpoint-url $S3_ENDPOINT \
--table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket \
--namespace my_namespace \
--name my_table
```
Each job type reports `Not_Yet_Run`, `Successful`, `Failed` or `Disabled`, with
the time of the last run.
### Put Table Bucket Maintenance Configuration
**AWS CLI:**
```bash
aws s3tables put-table-bucket-maintenance-configuration --endpoint-url $S3_ENDPOINT \
--table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket \
--type icebergUnreferencedFileRemoval \
--value '{"status":"enabled","settings":{"icebergUnreferencedFileRemoval":{"unreferencedDays":3,"nonCurrentDays":10}}}'
```
A file is deleted once it is older than `unreferencedDays` **plus**
`nonCurrentDays` — thirteen days here.
### Get Table Bucket Maintenance Configuration
**AWS CLI:**
```bash
aws s3tables get-table-bucket-maintenance-configuration --endpoint-url $S3_ENDPOINT \
--table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-test-bucket
```
---
## Cleanup
+5
@@ -50,6 +50,8 @@ The following actions are supported in policies. Actions can be specified with t
- `s3tables:PutTableBucketPolicy`
- `s3tables:GetTableBucketPolicy`
- `s3tables:DeleteTableBucketPolicy`
- `s3tables:PutTableBucketMaintenanceConfiguration`
- `s3tables:GetTableBucketMaintenanceConfiguration`
### Namespace Operations
- `s3tables:CreateNamespace`
@@ -65,6 +67,9 @@ The following actions are supported in policies. Actions can be specified with t
- `s3tables:PutTablePolicy`
- `s3tables:GetTablePolicy`
- `s3tables:DeleteTablePolicy`
- `s3tables:PutTableMaintenanceConfiguration`
- `s3tables:GetTableMaintenanceConfiguration`
- `s3tables:GetTableMaintenanceJobStatus`
---
+1
@@ -41,6 +41,7 @@ This page compares the S3 API implementation in SeaweedFS with MinIO.
- **S3 Tables API (Iceberg Support)**: Extensive support for Iceberg tables, namespaces, and buckets.
- `CreateTableBucket`, `ListTableBuckets`, `GetTableBucket`, `DeleteTableBucket`, `CreateNamespace`, `CreateTable`, etc.
- Maintenance configuration: `Put`/`GetTableMaintenanceConfiguration`, `Put`/`GetTableBucketMaintenanceConfiguration`, `GetTableMaintenanceJobStatus`.
- **Bucket Management**: SeaweedFS implements `PublicAccessBlock` and `OwnershipControls` which are rejected or labeled as dummy in some MinIO versions.
- **Object Rename**: `RenameObject` renames an object atomically without copying its data. AWS offers this API only on directory buckets (S3 Express One Zone); SeaweedFS supports it on regular buckets. See [[S3 RenameObject]].