Updated Erasure coding for warm storage (markdown)

Chris Lu committed 2019-06-05 11:59:47 -07:00
1 parent e8559a755b
commit 16c5e22fa2
1 file changed
+27 -1
+27 -1
@@ -5,4 +5,30 @@ Warm data are accessed less frequently. To store them more efficiently, you can
* SeaweedFS uses continuous 1GB block layout with 1MB block sizes for edge cases, optimized for both small file reads and storage efficiency.
* Efficient memory usage and faster startup time. The volume server does not load index data into memory.
## How to enable it?
## How to enable it?
Run `weed scaffold -conf=master` to generate a `master.toml` file, put it in current directory, `~/.seaweedfs/`, or `/etc/seaweedfs/`.
It will add a list of commands executed periodically. Actually the commands can also be executed via `weed shell` with exactly the same effect. The scripts stored in the `master.toml` file is to make the deployment convenient.
## How the scripts works?
The scripts has 3 steps.
### Erasure Encode Sealed Data
`ec.encode` command will find volumes that are almost full and has been stale for a period of time.
The default command is `ec.encode -fullPercent=95 -quietFor=1h`. It will find volumes at least 95% of the maximum volume size, which is usually 30GB, and has no updates for 1 hour.
### Data Repair
The default command is `ec.rebuild -force`. If disk fails or server fails, some data shards are lost. With erasure coding, we can recover the lost data shards from remaining data shards.
### EC data balancing
With servers added or removed, some data shards may not be laid out optimally. For example, one volume's 5 data shards could be on the same server. If the server goes down, the volume would be unrepairable or part of the data is lost permanently.
The default command is `ec.balance -force`. It will try to spread the data shards to minimize the data loss risk.
## How the read works?
When all data shards are online, the read are randomly assigned to one volume server (A) that has at least one data shard. Server A will read its copy of index file, and locate the volume server (B), and read from server B for the file.
## Read Performance