EC: document fault-tolerance guarantees and -shardReplicaPlacement

Explains that the guarantee follows from per-domain shard caps, not just
the parity count, and that ec.encode now fails (keeping originals) when
the requested per-node/per-rack limits cannot be met.
Chris Lu committed 2026-10-24 05:17:42 -07:00
1 parent 6914ee5685
commit 21ce6ce1f2
1 file changed
+10
+10
@@ -96,6 +96,16 @@ This bounds correlated loss: losing `k` racks loses at most `k × cap` shards, a
If the cap had to be raised for a volume, `ec.balance` prints it and the worker logs a warning — treat it as a capacity / spread signal, not a clean bill of health.
## Fault-tolerance guarantees
The parity count alone does not state a guarantee. A 10+4 volume rebuilds while at most 4 of its 14 shards are missing, so the real guarantee follows from *where the shards land*: the volume survives losing any combination of nodes, racks or data centers that hold at most 4 shards between them.
`ec.encode -shardReplicaPlacement=XYZ` turns that into an encode-time requirement. The digits cap shards per failure domain — rack digit `Y` = maximum shards per rack, node digit `Z` = maximum shards per node (the data-center digit `X` is not used for EC). For example `-shardReplicaPlacement=021` holds every node to at most 1 shard and every rack to at most 2, so losing any single rack or node costs at most 2 shards — comfortably inside 10+4's 4-shard budget. Before the source volumes are deleted, the final placement is measured against the requested caps; if the cluster cannot honor them, the encode fails and the original volumes are kept. An empty value falls back to the master default replica placement.
The same caps drive `ec.balance -apply` and the worker `ec_balance` task, which move shards back inside the limits after topology changes. Racks are counted qualified by data center, so same-named racks in different DCs are separate domains.
A quick self-check for a target resiliency: losing `k` domains of a level must cost at most `parityShards` shards. With the rack cap at 2, two rack failures cost at most 4 shards — a 10+4 volume survives; a cap of 3 would only survive one.
## How the read works?
When all data shards are online, the read for one file key are assigned to one volume server (A) that has at least one data shard for the volume. Server A will read its copy of index file, and locate the volume server (B), and read from server B for the file key.