mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-20 13:30:46 +02:00
wiki updates for maintenance scripts
+9
-10
@@ -131,19 +131,18 @@ volume.tier.upload -dest=s3.name2 -collection=benchmark -volumeId=37
|
||||
```
|
||||
|
||||
## Automated Tiering
|
||||
In `master.toml` generated by `weed scaffold -config=master`, you can adjust the `master.maintenance` script section:
|
||||
```
|
||||
[master.maintenance]
|
||||
# periodically run these scripts are the same as running them from 'weed shell'
|
||||
scripts = """
|
||||
lock
|
||||
volume.tier.upload -dest s3 -fullPercent=95 -quietFor=1h
|
||||
...
|
||||
unlock
|
||||
"""
|
||||
|
||||
Add `volume.tier.upload -dest s3 -fullPercent=95 -quietFor=1h` to the admin script plugin worker configuration via the admin UI at `/plugin`. The admin script worker handles locking automatically, so you do not need explicit `lock`/`unlock` commands.
|
||||
|
||||
```bash
|
||||
# Start admin server and worker
|
||||
weed admin -master=localhost:9333
|
||||
weed worker -admin=localhost:23646
|
||||
```
|
||||
|
||||
Or you can run the `weed shell` periodically in some cron job:
|
||||
```
|
||||
echo "lock;volume.tier.upload -dest s3 -fullPercent=95 -quietFor=1h;unlock" | weed shell
|
||||
```
|
||||
|
||||
> **Legacy note:** Previously, this was configured in `master.toml` under `[master.maintenance]`. That mechanism is skipped when an admin server is connected. New deployments should use the admin script plugin worker instead.
|
||||
|
||||
@@ -35,21 +35,32 @@ For smaller volumes less than 10GB, and for edge cases, the volume is split into
|
||||
The 14 EC shards should be spread into disks, volume servers and racks as evenly as possible, to protect against the hardware failure caused data loss.
|
||||
|
||||
## How to enable it?
|
||||
Run `weed scaffold -config=master` to generate a `master.toml` file, put it in current directory, `~/.seaweedfs/`, or `/etc/seaweedfs/`.
|
||||
|
||||
It will add a list of commands executed periodically. Actually the commands can also be executed via `weed shell` with exactly the same effect. The scripts stored in the `master.toml` file is to make the deployment convenient.
|
||||
Erasure coding is handled by the **`erasure_coding` plugin worker**. Start the admin server and a worker:
|
||||
|
||||
The script in the `master.toml` is executed on the master. If you have a large number of EC volumes, processing all of them on master may cost some CPU resources. It's better to run them with `weed shell` via some cron job in a separate machine.
|
||||
```bash
|
||||
weed admin -master=localhost:9333
|
||||
weed worker -admin=localhost:23646
|
||||
```
|
||||
|
||||
## How the scripts works?
|
||||
The scripts have 3 steps related to erasure coding.
|
||||
The `erasure_coding` plugin worker automatically detects volumes that are candidates for EC encoding and executes the encoding. It is enabled by default and configurable from the admin UI at `/plugin`.
|
||||
|
||||
### Erasure Encode Sealed Data
|
||||
`ec.encode` command will find volumes that are almost full and has been stale for a period of time.
|
||||
### Detection Thresholds
|
||||
|
||||
The default command is `ec.encode -fullPercent=95 -quietFor=1h -rebalance`. It will find volumes at least 95% of the maximum volume size, which is usually 30GB, and have no updates for 1 hour. Once the encoding is completed, EC shards are re-balanced; see [EC data balancing](#ec-data-balancing) below.
|
||||
The worker detects volumes eligible for EC encoding based on configurable thresholds:
|
||||
|
||||
Note that if you have any collections, i.e. because you're using s3 where every bucket is a collection you explicitly need to specify the collection for the ec.encode command for them to be erasure coded `ec.encode -collection="collection" -fullPercent=95 -quietFor=1h`.
|
||||
| Setting | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| Fullness Ratio | 0.8 (80%) | Minimum volume fullness ratio to trigger encoding |
|
||||
| Quiet Period | 300s (5 min) | Volume must be unmodified for at least this long |
|
||||
| Minimum Size | 30 MB | Only volumes larger than this are considered |
|
||||
| Collection Filter | (all) | Optionally restrict to a specific collection |
|
||||
|
||||
Detection runs every 5 minutes by default. These settings can be adjusted in the admin UI.
|
||||
|
||||
### How it works
|
||||
|
||||
The worker scans the cluster topology, identifies volumes meeting the thresholds above, and proposes EC encoding jobs. The admin server deduplicates proposals and dispatches them to available workers for execution.
|
||||
|
||||
If the volume is replicated, only one copy will be erasure encoded. All the original copies will be purged after a successful erasure encoding.
|
||||
|
||||
@@ -58,14 +69,12 @@ Note: One collection can contain both normal volumes and erasure coded volumes,
|
||||
### Data Repair
|
||||
If disks fail or servers fail, some data shards are lost. With erasure coding, we can recover the lost data shards from the remaining data shards.
|
||||
|
||||
The default command is `ec.rebuild -force`.
|
||||
|
||||
The data repair happens for the whole volume, instead of one small file at a time. It is much more efficient and fast to reconstruct the missing data shards than processing each file individually.
|
||||
|
||||
### EC data balancing
|
||||
With servers added or removed, some data shards may not be laid out optimally. For example, one volume's 5 data shards could be on the same server. If that server goes down, the volume would be unrepairable or part of the data is lost permanently.
|
||||
With servers added or removed, some data shards may not be laid out optimally. For example, one volume's 5 data shards could be on the same server. If that server goes down, the volume would be unrepairable or part of the data is lost permanently.
|
||||
|
||||
The default command is `ec.balance -force`. It will try to spread the data shards evenly to minimize the data shard loss risk.
|
||||
The default admin script includes `ec.balance -apply` which spreads EC shards evenly to minimize data loss risk.
|
||||
|
||||
EC shard re-balancing happens in three steps:
|
||||
|
||||
|
||||
+24
-13
@@ -4,23 +4,34 @@ When managing large clusters, it's common to add more volume servers, have some
|
||||
See [[Optimization]] page on how to optimize for concurrent writes and concurrent reads.
|
||||
|
||||
## Configure volume management scripts
|
||||
Run `weed scaffold -config=master` will generate `master.toml` which has sections as these.
|
||||
|
||||
```
|
||||
[master.maintenance]
|
||||
# periodically run these scripts are the same as running them from 'weed shell'
|
||||
scripts = """
|
||||
ec.encode -fullPercent=95 -quietFor=1h
|
||||
ec.rebuild -force
|
||||
ec.balance -force
|
||||
volume.balance -force
|
||||
volume.fix.replication -force
|
||||
"""
|
||||
sleep_minutes = 17 # sleep minutes between each script execution
|
||||
Maintenance scripts are managed by the **admin script plugin worker**. Start the admin server and a worker:
|
||||
|
||||
```bash
|
||||
# Start admin server (connects to master)
|
||||
weed admin -master=localhost:9333
|
||||
|
||||
# Start worker (connects to admin server)
|
||||
weed worker -admin=localhost:23646
|
||||
```
|
||||
|
||||
If the `master.toml` has the above configuration, the scripts will run every 17 minutes. It has the same effect as running from `weed shell` directly.
|
||||
The admin script plugin has a built-in default script:
|
||||
|
||||
```
|
||||
ec.balance -apply
|
||||
fs.log.purge -daysAgo=7
|
||||
volume.deleteEmpty -quietFor=24h -apply
|
||||
volume.fix.replication -apply
|
||||
s3.clean.uploads -timeAgo=24h
|
||||
```
|
||||
|
||||
The script and run interval (default: 17 minutes) are configurable from the admin UI at `/plugin`. You can customize commands, add `volume.balance`, etc. as needed.
|
||||
|
||||
Erasure coding (previously `ec.encode`) is handled by a dedicated **`erasure_coding` plugin worker** that automatically detects and encodes eligible volumes. See [[Erasure Coding for warm storage]] for details.
|
||||
|
||||
See the [[Worker]] page for more details on `weed worker` options and capabilities.
|
||||
|
||||
> **Legacy note:** Previously, maintenance scripts were configured in `master.toml` under `[master.maintenance]`. That mechanism still exists as a fallback but is automatically skipped when an admin server is connected. New deployments should use the admin script plugin worker instead.
|
||||
|
||||
## Fix missing volumes
|
||||
When running large clusters, it is common that some volume servers are down. If a volume is replicated and one replica is missing, the volume will be marked as readonly.
|
||||
|
||||
Reference in New Issue
Block a user