mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-13 07:19:18 +01:00
Updated Erasure coding for warm storage (markdown)
1 parent
e8559a755b
commit
16c5e22fa2
1 file changed
+27
-1
@@ -5,4 +5,30 @@ Warm data are accessed less frequently. To store them more efficiently, you can
|
||||
* SeaweedFS uses continuous 1GB block layout with 1MB block sizes for edge cases, optimized for both small file reads and storage efficiency.
|
||||
* Efficient memory usage and faster startup time. The volume server does not load index data into memory.
|
||||
|
||||
## How to enable it?
|
||||
## How to enable it?
|
||||
Run `weed scaffold -conf=master` to generate a `master.toml` file, put it in current directory, `~/.seaweedfs/`, or `/etc/seaweedfs/`.
|
||||
|
||||
It will add a list of commands executed periodically. Actually the commands can also be executed via `weed shell` with exactly the same effect. The scripts stored in the `master.toml` file is to make the deployment convenient.
|
||||
|
||||
## How the scripts works?
|
||||
The scripts has 3 steps.
|
||||
|
||||
### Erasure Encode Sealed Data
|
||||
`ec.encode` command will find volumes that are almost full and has been stale for a period of time.
|
||||
|
||||
The default command is `ec.encode -fullPercent=95 -quietFor=1h`. It will find volumes at least 95% of the maximum volume size, which is usually 30GB, and has no updates for 1 hour.
|
||||
|
||||
### Data Repair
|
||||
The default command is `ec.rebuild -force`. If disk fails or server fails, some data shards are lost. With erasure coding, we can recover the lost data shards from remaining data shards.
|
||||
|
||||
### EC data balancing
|
||||
With servers added or removed, some data shards may not be laid out optimally. For example, one volume's 5 data shards could be on the same server. If the server goes down, the volume would be unrepairable or part of the data is lost permanently.
|
||||
|
||||
The default command is `ec.balance -force`. It will try to spread the data shards to minimize the data loss risk.
|
||||
|
||||
## How the read works?
|
||||
When all data shards are online, the read are randomly assigned to one volume server (A) that has at least one data shard. Server A will read its copy of index file, and locate the volume server (B), and read from server B for the file.
|
||||
|
||||
|
||||
|
||||
## Read Performance
|
||||
Reference in new issue
Block a user