From ec4ec60cfc05f3f9c8af99185f98249b16ddb6e9 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 21 Mar 2026 06:44:31 -0700 Subject: [PATCH] Update Cloud-Tier.md --- Cloud-Tier.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/Cloud-Tier.md b/Cloud-Tier.md index 13b65ae..9163aac 100644 --- a/Cloud-Tier.md +++ b/Cloud-Tier.md @@ -48,6 +48,7 @@ Compared to direct S3 storage, this is both faster and cheaper! 1. Use `weed scaffold -config=master` to generate `master.toml`, tweak it, and start master server with the `master.toml`. 1. Use `volume.tier.upload` in `weed shell` to move volumes to the cloud. 1. Use `volume.tier.download` in `weed shell` to move volumes to the local cluster. +1. Use `volume.tier.compact` in `weed shell` to reclaim space from deleted files on the cloud. ## Configuring Storage Backend @@ -130,6 +131,32 @@ volume.tier.upload -dest=s3.name2 -collection=benchmark -volumeId=37 ``` +## Compacting Cloud Tier Volumes + +When files are deleted from cloud-tiered volumes, the space on the remote storage is not automatically reclaimed. Over time this can cause cloud storage usage to grow indefinitely despite deletions. Use `volume.tier.compact` to reclaim this space. + +The command automates the process of downloading the volume data, compacting it locally to remove deleted entries, and re-uploading the compacted data back to the cloud. + +``` +// compact a specific remote volume +volume.tier.compact -volumeId=37 + +// compact all remote volumes in a collection with >20% garbage +volume.tier.compact -collection=mybucket -garbageThreshold=0.2 + +// compact all remote volumes across all collections +volume.tier.compact -collection=".*" -garbageThreshold=0.3 +``` + +The `-garbageThreshold` flag (default `0.3`) controls the minimum ratio of deleted data before compaction is triggered. Volumes below this threshold are skipped. + +To run this periodically, add `volume.tier.compact -collection=".*" -garbageThreshold=0.3` to the admin script plugin worker, or use a cron job: +``` +echo "lock;volume.tier.compact -collection=\".*\" -garbageThreshold=0.3;unlock" | weed shell +``` + +**Note:** Compaction temporarily requires enough local disk space to hold the volume's `.dat` file during the download and compaction steps. + ## Automated Tiering Add `volume.tier.upload -dest s3 -fullPercent=95 -quietFor=1h` to the admin script plugin worker configuration via the admin UI at `/plugin`. The admin script worker handles locking automatically, so you do not need explicit `lock`/`unlock` commands.