diff --git a/Erasure-coding-for-warm-storage.md b/Erasure-coding-for-warm-storage.md index 831a439..dd5d1a8 100644 --- a/Erasure-coding-for-warm-storage.md +++ b/Erasure-coding-for-warm-storage.md @@ -5,4 +5,30 @@ Warm data are accessed less frequently. To store them more efficiently, you can * SeaweedFS uses continuous 1GB block layout with 1MB block sizes for edge cases, optimized for both small file reads and storage efficiency. * Efficient memory usage and faster startup time. The volume server does not load index data into memory. -## How to enable it? \ No newline at end of file +## How to enable it? +Run `weed scaffold -conf=master` to generate a `master.toml` file, put it in current directory, `~/.seaweedfs/`, or `/etc/seaweedfs/`. + +It will add a list of commands executed periodically. Actually the commands can also be executed via `weed shell` with exactly the same effect. The scripts stored in the `master.toml` file is to make the deployment convenient. + +## How the scripts works? +The scripts has 3 steps. + +### Erasure Encode Sealed Data +`ec.encode` command will find volumes that are almost full and has been stale for a period of time. + +The default command is `ec.encode -fullPercent=95 -quietFor=1h`. It will find volumes at least 95% of the maximum volume size, which is usually 30GB, and has no updates for 1 hour. + +### Data Repair +The default command is `ec.rebuild -force`. If disk fails or server fails, some data shards are lost. With erasure coding, we can recover the lost data shards from remaining data shards. + +### EC data balancing +With servers added or removed, some data shards may not be laid out optimally. For example, one volume's 5 data shards could be on the same server. If the server goes down, the volume would be unrepairable or part of the data is lost permanently. + +The default command is `ec.balance -force`. It will try to spread the data shards to minimize the data loss risk. + +## How the read works? +When all data shards are online, the read are randomly assigned to one volume server (A) that has at least one data shard. Server A will read its copy of index file, and locate the volume server (B), and read from server B for the file. + + + +## Read Performance