Update Cloud-Tier.md

Chris Lu committed 2026-03-21 06:44:31 -07:00
1 parent fa0781e4fd
commit ec4ec60cfc
1 file changed
+27
+27
@@ -48,6 +48,7 @@ Compared to direct S3 storage, this is both faster and cheaper!
1. Use `weed scaffold -config=master` to generate `master.toml`, tweak it, and start master server with the `master.toml`.
1. Use `volume.tier.upload` in `weed shell` to move volumes to the cloud.
1. Use `volume.tier.download` in `weed shell` to move volumes to the local cluster.
1. Use `volume.tier.compact` in `weed shell` to reclaim space from deleted files on the cloud.
## Configuring Storage Backend
@@ -130,6 +131,32 @@ volume.tier.upload -dest=s3.name2 -collection=benchmark -volumeId=37
```
## Compacting Cloud Tier Volumes
When files are deleted from cloud-tiered volumes, the space on the remote storage is not automatically reclaimed. Over time this can cause cloud storage usage to grow indefinitely despite deletions. Use `volume.tier.compact` to reclaim this space.
The command automates the process of downloading the volume data, compacting it locally to remove deleted entries, and re-uploading the compacted data back to the cloud.
```
// compact a specific remote volume
volume.tier.compact -volumeId=37
// compact all remote volumes in a collection with >20% garbage
volume.tier.compact -collection=mybucket -garbageThreshold=0.2
// compact all remote volumes across all collections
volume.tier.compact -collection=".*" -garbageThreshold=0.3
```
The `-garbageThreshold` flag (default `0.3`) controls the minimum ratio of deleted data before compaction is triggered. Volumes below this threshold are skipped.
To run this periodically, add `volume.tier.compact -collection=".*" -garbageThreshold=0.3` to the admin script plugin worker, or use a cron job:
```
echo "lock;volume.tier.compact -collection=\".*\" -garbageThreshold=0.3;unlock" | weed shell
```
**Note:** Compaction temporarily requires enough local disk space to hold the volume's `.dat` file during the download and compaction steps.
## Automated Tiering
Add `volume.tier.upload -dest s3 -fullPercent=95 -quietFor=1h` to the admin script plugin worker configuration via the admin UI at `/plugin`. The admin script worker handles locking automatically, so you do not need explicit `lock`/`unlock` commands.