wiki updates for maintenance scripts

Chris Lu
2026-03-04 20:06:22 -08:00
parent 74646f5a54
commit e8a32509dd
3 changed files with 55 additions and 36 deletions
+9 -10
@@ -131,19 +131,18 @@ volume.tier.upload -dest=s3.name2 -collection=benchmark -volumeId=37
```
## Automated Tiering
In `master.toml` generated by `weed scaffold -config=master`, you can adjust the `master.maintenance` script section:
```
[master.maintenance]
# periodically run these scripts are the same as running them from 'weed shell'
scripts = """
lock
volume.tier.upload -dest s3 -fullPercent=95 -quietFor=1h
...
unlock
"""
Add `volume.tier.upload -dest s3 -fullPercent=95 -quietFor=1h` to the admin script plugin worker configuration via the admin UI at `/plugin`. The admin script worker handles locking automatically, so you do not need explicit `lock`/`unlock` commands.
```bash
# Start admin server and worker
weed admin -master=localhost:9333
weed worker -admin=localhost:23646
```
Or you can run the `weed shell` periodically in some cron job:
```
echo "lock;volume.tier.upload -dest s3 -fullPercent=95 -quietFor=1h;unlock" | weed shell
```
> **Legacy note:** Previously, this was configured in `master.toml` under `[master.maintenance]`. That mechanism is skipped when an admin server is connected. New deployments should use the admin script plugin worker instead.
+22 -13
@@ -35,21 +35,32 @@ For smaller volumes less than 10GB, and for edge cases, the volume is split into
The 14 EC shards should be spread into disks, volume servers and racks as evenly as possible, to protect against the hardware failure caused data loss.
## How to enable it?
Run `weed scaffold -config=master` to generate a `master.toml` file, put it in current directory, `~/.seaweedfs/`, or `/etc/seaweedfs/`.
It will add a list of commands executed periodically. Actually the commands can also be executed via `weed shell` with exactly the same effect. The scripts stored in the `master.toml` file is to make the deployment convenient.
Erasure coding is handled by the **`erasure_coding` plugin worker**. Start the admin server and a worker:
The script in the `master.toml` is executed on the master. If you have a large number of EC volumes, processing all of them on master may cost some CPU resources. It's better to run them with `weed shell` via some cron job in a separate machine.
```bash
weed admin -master=localhost:9333
weed worker -admin=localhost:23646
```
## How the scripts works?
The scripts have 3 steps related to erasure coding.
The `erasure_coding` plugin worker automatically detects volumes that are candidates for EC encoding and executes the encoding. It is enabled by default and configurable from the admin UI at `/plugin`.
### Erasure Encode Sealed Data
`ec.encode` command will find volumes that are almost full and has been stale for a period of time.
### Detection Thresholds
The default command is `ec.encode -fullPercent=95 -quietFor=1h -rebalance`. It will find volumes at least 95% of the maximum volume size, which is usually 30GB, and have no updates for 1 hour. Once the encoding is completed, EC shards are re-balanced; see [EC data balancing](#ec-data-balancing) below.
The worker detects volumes eligible for EC encoding based on configurable thresholds:
Note that if you have any collections, i.e. because you're using s3 where every bucket is a collection you explicitly need to specify the collection for the ec.encode command for them to be erasure coded `ec.encode -collection="collection" -fullPercent=95 -quietFor=1h`.
| Setting | Default | Description |
|---------|---------|-------------|
| Fullness Ratio | 0.8 (80%) | Minimum volume fullness ratio to trigger encoding |
| Quiet Period | 300s (5 min) | Volume must be unmodified for at least this long |
| Minimum Size | 30 MB | Only volumes larger than this are considered |
| Collection Filter | (all) | Optionally restrict to a specific collection |
Detection runs every 5 minutes by default. These settings can be adjusted in the admin UI.
### How it works
The worker scans the cluster topology, identifies volumes meeting the thresholds above, and proposes EC encoding jobs. The admin server deduplicates proposals and dispatches them to available workers for execution.
If the volume is replicated, only one copy will be erasure encoded. All the original copies will be purged after a successful erasure encoding.
@@ -58,14 +69,12 @@ Note: One collection can contain both normal volumes and erasure coded volumes,
### Data Repair
If disks fail or servers fail, some data shards are lost. With erasure coding, we can recover the lost data shards from the remaining data shards.
The default command is `ec.rebuild -force`.
The data repair happens for the whole volume, instead of one small file at a time. It is much more efficient and fast to reconstruct the missing data shards than processing each file individually.
### EC data balancing
With servers added or removed, some data shards may not be laid out optimally. For example, one volume's 5 data shards could be on the same server. If that server goes down, the volume would be unrepairable or part of the data is lost permanently.
With servers added or removed, some data shards may not be laid out optimally. For example, one volume's 5 data shards could be on the same server. If that server goes down, the volume would be unrepairable or part of the data is lost permanently.
The default command is `ec.balance -force`. It will try to spread the data shards evenly to minimize the data shard loss risk.
The default admin script includes `ec.balance -apply` which spreads EC shards evenly to minimize data loss risk.
EC shard re-balancing happens in three steps:
+24 -13
@@ -4,23 +4,34 @@ When managing large clusters, it's common to add more volume servers, have some
See [[Optimization]] page on how to optimize for concurrent writes and concurrent reads.
## Configure volume management scripts
Run `weed scaffold -config=master` will generate `master.toml` which has sections as these.
```
[master.maintenance]
# periodically run these scripts are the same as running them from 'weed shell'
scripts = """
ec.encode -fullPercent=95 -quietFor=1h
ec.rebuild -force
ec.balance -force
volume.balance -force
volume.fix.replication -force
"""
sleep_minutes = 17 # sleep minutes between each script execution
Maintenance scripts are managed by the **admin script plugin worker**. Start the admin server and a worker:
```bash
# Start admin server (connects to master)
weed admin -master=localhost:9333
# Start worker (connects to admin server)
weed worker -admin=localhost:23646
```
If the `master.toml` has the above configuration, the scripts will run every 17 minutes. It has the same effect as running from `weed shell` directly.
The admin script plugin has a built-in default script:
```
ec.balance -apply
fs.log.purge -daysAgo=7
volume.deleteEmpty -quietFor=24h -apply
volume.fix.replication -apply
s3.clean.uploads -timeAgo=24h
```
The script and run interval (default: 17 minutes) are configurable from the admin UI at `/plugin`. You can customize commands, add `volume.balance`, etc. as needed.
Erasure coding (previously `ec.encode`) is handled by a dedicated **`erasure_coding` plugin worker** that automatically detects and encodes eligible volumes. See [[Erasure Coding for warm storage]] for details.
See the [[Worker]] page for more details on `weed worker` options and capabilities.
> **Legacy note:** Previously, maintenance scripts were configured in `master.toml` under `[master.maintenance]`. That mechanism still exists as a fallback but is automatically skipped when an admin server is connected. New deployments should use the admin script plugin worker instead.
## Fix missing volumes
When running large clusters, it is common that some volume servers are down. If a volume is replicated and one replica is missing, the volume will be marked as readonly.