mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-20 13:30:46 +02:00
ec: uniform shard block layout (#10932)
* ec: uniform shard block layout An EC volume is striped as 1GiB blocks until less than one row remains, then 1MiB blocks, and consecutive blocks land on different shards. With ec.encode's -fullPercent 95 against the 30GiB default limit, ~30% of every volume sits in that 1MiB tail, so a 4MB filer chunk there is five stripes on five servers. New encodes now use one block per shard, sized ceil(datSize/dataShards) rounded up to 1MiB and recorded in the .vif (EcShardConfig.block_size, also carried by the .ecsum manifest). A needle now maps to one shard unless it is larger than the block or straddles a boundary. The chosen size equals the legacy layout's padded shard length for every input, so shard sizes, capacity math, and the shard-size credibility checks are unchanged; only the byte placement moved. Reads, decode, and scrub resolve the block sizes from the volume's .vif; absence keeps the legacy interpretation, so existing EC volumes read exactly as before. Rebuild is layout-agnostic. weed fix -ecx recovers the layout from the .vif, else the .ecsum sidecar, and with neither de-stripes under both candidate layouts and keeps the one that indexes more valid needles. Same change in the Rust volume server, which now also streams the encode in 256KB sub-batches like Go instead of allocating whole blocks, and computes the large-row count as shardSize/largeBlock to match Go on exact multiples. On a 26MB fixture both encoders produce byte-identical shards, and a Go-written .vif parses in Rust with the block size intact. * ec: resolve the rust ecx rebuild through the recorded layout The Rust rebuild path regenerated a lost .ecx by scanning the logical .dat through a hand-rolled pure-1MiB striping, which was already wrong for legacy volumes with large-block rows and is wrong for any uniform volume with a block past 1MiB. Route the scan through locate_data with the .vif-recorded block size, the same mapping the read path uses. Also seed the new tests' random data instead of the deprecated global math/rand.Read. * ec: fail the Rust ecx rebuild on any shard read error A read error mid-scan published the entries collected so far as a successful .ecx, and read_at's byte count was ignored so a legal short read passed as complete — a truncated or failing shard could produce a silently incomplete recovery index. Exact-read semantics in read_from_data_shards, error propagation in the needle walk, and a truncated-shard regression test. * ec: fail the mount on an unreadable or malformed vif Both servers silently fell back to the legacy layout when an existing .vif could not be read or parsed. Every new encode records a positive uniform block size there, so the fallback mounted the same shards with legacy offset math and could return wrong data. Absent stays legal (legacy volumes predate the sidecar), and a zero-byte stub still reads as absent (Go's MaybeLoadVolumeInfo convention, now mirrored in Rust); a present-but-unreadable or malformed .vif fails the mount instead. * ec: bound the reconstruct fan-out of one needle's intervals A degraded interval fans out a read to every reachable shard location, each with a buffer the size of the interval. Reading a needle's intervals in parallel multiplied that by the interval concurrency: a needle spanning 8 blocks could hold 8 x MaxShardCount remote reads and buffers at once, where the sequential version peaked at MaxShardCount. Give each needle a single reconstruct budget its intervals share, held for the buffer's lifetime, so separate reads stay independent but one read cannot multiply its own fan-out. * ec: drop the duplicated shard-size formula calculateExpectedShardSize reimplemented the padding rule that UniformBlockSize already owns — TestUniformBlockSizeMatchesLegacyShardSize asserts the two agree for every input — so a change to the rule would have had to be made in both. Defer to the helper, keeping the historic answer for an empty .dat. * ec: resolve the shard block layout from whatever records it Four places still answered the layout question by inference when a record of it was available, or accepted an answer that was not one: - A mount with no .vif defaulted to the legacy layout; the bitrot sidecar records the same config at encode time, so take it when present, as weed fix -ecx already does. The vif itself is now parsed once per mount rather than twice. - The Rust ecx rebuild derived its row count from the padded shard extent, which under the legacy layout reads a shard that is an exact large-block multiple as one row too many. Pass the encode-time .dat size from the .vif and keep the extent as the fallback. - weed fix -ecx read the block size outside the EC-config guard (collapsing the unknown sentinel into a definitive legacy), only wrote the recovered layout back when the .vif was absent rather than unusable, and broke a scan tie by candidate order instead of the documented reach. - The uniform layout tripped writeDatFile's large-block ambiguity guard, which cannot apply when the large and small blocks are the same size. * ec: give the index-recovery tests a parseable vif The fixtures wrote the literal bytes "volinfo" as the source .vif and the recovery copies it verbatim, so the receiving server then mounted the volume from a .vif it could not parse. That used to pass by silently defaulting to the legacy layout; a mount now refuses a vif it cannot read, which is what the tests were exercising all along without meaning to. * ec: validate the layout a vif records, not just its syntax Review follow-ups on the mount-strictness change: - A .vif can parse and still record a block size no encoder could have produced (negative, or not a whole number of small blocks). Both servers took it and mapped every read through it. ValidateBlockSize / the Rust mirror now refuse the mount, the same way an unparseable vif does; 0 stays valid as the legacy two-tier layout. - The bitrot-sidecar fallback accepted parity_shards == 0 and summed the counts in their own width, so values near the ceiling wrapped past the MaxShardCount bound. Require both counts and sum in a wider type. - weed fix -ecx treated a config with only DataShards > 0 as usable, so a half-written .vif suppressed the recovery paths AND survived the rewrite. Require a complete, in-range config before trusting it. - Returning the vif-load error left the .ecx and .ecj descriptors open; repeated mount attempts on malformed metadata could exhaust them. * ec: refuse to act on a layout the metadata does not establish - The worker encode only logged a failed .vif write and skipped it in the distribution set, and treated the .ecsum write as best-effort. A worker whose disk filled after the much larger shards landed could still distribute, mount, verify shard inventory, and delete the source replicas — leaving holders with shards whose geometry nothing records. Both writes and both inclusions are encode success conditions now. - A generation-matching .ecsum that disagreed with the .vif geometry only disabled checksums in Go, and in Rust was not compared at all, so protection stayed On while reads used the other layout. Both files record the layout their generation was encoded with, so a disagreement now fails the mount. * ec: reject an invalid recorded block size in weed fix -ecx A .vif with valid shard counts but a negative or unaligned block size was marked usable: a positive invalid value pinned the scan to a geometry that de-stripes to garbage, and a negative one ran the dual scan but left the invalid .vif in place afterwards. Validate it with the same rule the mount applies, and when it fails leave the layout unknown so the scan recovers it and the file is rewritten. * ec: validate the sidecar layout weed fix -ecx recovers from The .ecsum fallback was taken on DataShards > 0 alone, so a CRC-valid sidecar carrying the wrong generation, an incomplete ratio, or an unaligned block size would pin the reconstruction to one incorrect uniform-layout candidate instead of letting the dual scan decide. Require generation 0, a complete in-range ratio, and a valid block size; anything less leaves the layout unknown, which is the answer that still recovers by scanning. * ec: let only a genuinely absent sidecar choose the legacy layout With no .vif the bitrot sidecar is the only record of a volume's layout, and the mount fallback read a failed load, an unusable config, or a sidecar stamped for another generation as "assume legacy". A uniform generation-0 volume could therefore mount with legacy or another generation's geometry and answer reads with the wrong bytes. Present-but-unusable now fails the mount; only actual absence keeps the legacy defaults. Shared as EcShardConfigFromSidecar so every caller reads the sidecar the same way. * ec: treat a recorded-but-impossible layout as corruption, not as legacy - A .vif whose ecShardConfig is PRESENT but records an impossible ratio was answered with the default 10+4 and the legacy block layout, in both languages. That reads a uniform volume's shards at the wrong offsets and returns the wrong bytes. Only an entirely absent config still means "this predates the record"; a present one that cannot be true fails the mount. - The shard-count bound summed two uint32 counts as int, which wraps on a 32-bit build: 0x7fffffff + 0x7fffffff lands at -2 and slips under MaxShardCount. ValidEcShardCounts sums in uint64, and every EC call site that checked a recorded ratio now goes through it. * ec: rebuild on the geometry the sidecar records, and flag it when it disagrees The rebuild RPC passes BackgroundECContext, so RebuildEcFiles resolves the layout itself — and it resolved a missing or invalid .vif to the default 10+4 with the legacy block size. Two consequences: a 12+4 volume was reconstructed through a 10+4 matrix, which produces wrong bytes and never regenerates shards 14-15; and the chosen geometry then contradicted a valid uniform sidecar, which loadRebuildSidecar reported as BitrotOff — silently skipping the input and regenerated-shard checksum checks precisely when the volume had already lost its metadata. The layout now resolves from the bitrot sidecar (found across the server's disks, not just beside the base name) before falling back to the defaults, and a present-but-impossible ratio fails instead of being replaced. A sidecar that contradicts the chosen geometry is BitrotInvalid, which the existing unsafeIgnoreSidecar override still lets an operator push past. * ec: let the Rust rebuild read metadata off a sibling disk read_ec_shard_config searches only the location the rebuild writes into, so a volume whose .vif or generation-0 .ecsum sits on another of the server's disks resolved to the default 10+4 with the legacy block layout — the Rust half of the geometry-guessing the Go rebuild just stopped doing. It then reconstructs a custom-ratio or uniform volume through the wrong Reed-Solomon matrix and de-striping geometry. The rebuild now looks for the .vif in its own location and then each sibling, falls back to the generation-0 sidecar wherever that lives, and only defaults when neither exists anywhere. The encode-time .dat size the ecx rebuild needs is resolved the same way. * ec: resolve a rebuild's vif from every directory that may hold it RebuildEcFiles probed only <data-base>.vif. The caller knows the selected location's index directory and the sibling locations, but passed neither for metadata: additionalDirs carried shard directories only, and were searched for shards and the checksum sidecar. A split -dir/-dir.idx layout, or a disk holding only shards, therefore resolved a pre-sidecar custom-ratio volume to 10+4 and reconstructed through the wrong matrix — never regenerating shards 14-15. The caller now hands over the index and sibling directories, and the resolver probes the vif across all of them, matching what the Rust resolver already does for both the vif and the sidecar. * ec: make every rebuild consumer agree on the layout it resolved - The post-rebuild bitrot backfill re-derived the geometry from this directory's .vif alone and dropped the block size entirely, so a rebuild that resolved its layout from a sibling, the sidecar, or a uniform vif wrote a manifest describing a DIFFERENT layout — one later mounts reject, or that covers only the default shard count. The layout is resolved once now, through an exported ResolveRebuildECContext, and the rebuild and the backfill share that answer. - The Rust rebuild collected only each location's data directory, so a sibling's INDEX directory — where a split -dir/-dir.idx layout keeps .ecx/.ecj/.vif — was never probed, and a custom-ratio volume still resolved to 10+4 with the legacy layout. Both directories of every location are carried now, deduped against the rebuild's own. - A shard delivery can bring the checksum manifest with it, but the receive path only writes the file: a server that already had the volume mounted kept its resolved protection state (off) until a remount. The mount RPC re-resolves it once the shards it describes have been added. * ec: cover the rebuild's directory search with tests Reviewers flagged the sibling index directory twice, and the fix that closed it had no test of its own: the assembly sat inline in the rebuild handler, reachable only through a gRPC call against a populated store. Lifting it into rebuildSearchDirs / select_rebuild_location makes the rule assertable — a sibling contributes BOTH its data and its index directory, a shared index directory is listed once, and the rebuild's own data directory never repeats. Writing the Rust cases surfaced that the two implementations do not agree on where the rebuild's own index directory belongs, and both are right: Go's resolver takes a single directory list, so that directory has to be inside it, while Rust's takes the rebuild's data and index directories as their own arguments and would search them twice. The tests now state which contract each side is holding to, so neither drifts into the other's shape. Pure refactor otherwise; no behaviour change. * ec: search the index directory for the layout sidecar The Rust resolver looked for the generation-0 .ecsum in the rebuild's data directory and the sibling list, but not in the rebuild's own index directory — while the .vif lookup directly above it did, and Go's findBitrotSidecar has always checked both bases. On a split -dir/-dir.idx location that directory is where the metadata lives, and callers leave it out of the sibling list precisely because it is passed here separately, so nothing searched it. With no .vif anywhere the sidecar is the only surviving record of the layout. Missing it resolved a 12+4 uniform volume to 10+4 with the legacy striping — the test added here fails with (10, 4, 0) against the old code — and the rebuild then reconstructs through the wrong matrix and writes .ecx offsets that no reader can follow. * ec: let the rebuild see its own index directory The Rust rebuild takes a single flat directory list — the shape Go's RebuildEcFiles uses — so it cannot be handed the rebuild location's index directory separately the way the layout resolvers are, and the handler was passing the sibling list, which deliberately omits exactly that directory. On a split -dir/-dir.idx location that is where .ecx and .vif live, so the shard and index lookups could not see them. Go has always carried that directory in additionalDirs; this lines the two call sites up. * ec: let a config-free vif fall through to the layout sidecar A .vif that carries no ecShardConfig answers nothing about the layout, so it is no more informative than an absent one — but both trees treated its mere existence as the end of the search. Go went straight to the 10+4 legacy defaults without consulting the sidecar at all; Rust returned whatever ec_shard_config_from could make of a single directory. A 12+4 uniform volume with a legacy config-free vif therefore resolved as 10+4 legacy, and every read landed at the wrong shard offset. The sidecar lookup was also single-directory on both sides, while a split -dir/-dir.idx layout keeps .vif and .ecsum with the INDEX. Go's findBitrotSidecar has always taken both bases; the callers here passed only the data base, and the Rust bitrot resolver derived its path from the data base alone. Rust's layout resolver now takes a candidate directory list — data, index, then any siblings — and searches all of it, which also removes the early return that made the vif's presence decisive. load_vif_info_across_dirs reported `dir` even when load_vif_info had found the vif in `dir_idx`. Nothing reads that field today, so this changes no behaviour; it stops the next caller that resolves the rest of the volume's metadata against the answer from being sent to a disk holding none of it. Absence stays legal throughout: a volume with neither record is genuinely legacy. Present-but-unusable still fails the mount, now in the config-free-vif branch too. * ec: activate a delivered sidecar on every per-disk runtime A vid mounts as one EcVolume per disk, each with its own resolved protection state, but the post-delivery reload used the first-match lookup and so touched exactly one of them. The siblings kept reporting no protection until a remount — and since shard distribution deduplicates the metadata files onto the first target disk for a node, the runtime that got the .ecsum is not necessarily the one the lookup returns. Iterate every runtime instead, via a new FindAllEcVolumes and its Rust mut equivalent. Combined with each runtime now resolving its sidecar against its index directory as well as its data directory, a server sharing one -dir.idx across its disks activates all of them from the single delivered copy. The Rust volume server had no post-mount reload at all; it gets one here, matching Go. * ec: resolve the delivered sidecar across every EC metadata directory Reloading every per-disk runtime, added last round, did not by itself make the delivered manifest reachable. Startup mirroring copies .ecx/.ecj/.vif to every shard-bearing disk so each mounts self-contained, but deliberately not .ecsum, and a repair delivers exactly one copy. Each runtime was resolving against its own two directories, so every sibling of the disk that received the file kept reporting no protection however often it reloaded. Resolve one authoritative copy across every EC metadata directory instead of duplicating the file. Mirroring .ecsum would have to keep pace with a file that is rewritten as shards are repaired, and would not help the reported case at all: the delivery happens at runtime, and mirroring only runs at startup. The regression test pins both halves — a reload restricted to the volume's own directories still finds nothing, and the same reload given the server's metadata directories turns protection on. * ec: ask every directory before writing a TOFU baseline After a rebuild the opportunistic backfill asks whether this volume already has a checksum manifest, and answered from the data base alone. A split -dir/-dir.idx layout keeps the sidecar with the index, and a multi-disk server may keep it on a sibling, so an existing manifest read as absent. The consequence is worse than a missed read. On a false "no" the backfill writes a fresh sidecar at the data base from whatever the shards say right now — and the data base is the first candidate every resolver checks, so that TOFU baseline shadows the real manifest rather than sitting beside it. A shard that was silently corrupt gets blessed, and the record that would have caught it stops being consulted. FindBitrotSidecar exports the search the package already used internally, so the question is asked of the data base, the index base and the sibling disks — the same candidates the rebuild resolves its layout from. * ec: refuse a shard block size no encoder could have produced weed fix -ecx derived one from the raw shard extent, so a truncated or partially copied shard wrote a .vif that NewEcVolume then permanently refuses — the volume the tool was run to rescue could never mount again. An extent that is not a whole number of small blocks cannot have come from a uniform encode, so it is no longer offered as a candidate, and nothing unvalidated reaches the .vif. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: derive the .vif's dat size and block size from one measurement VolumeEcShardsGenerate stat'ed the .dat before the encode while WriteEcFiles stat'ed it again to size the blocks. A write landing between the two produced a .vif whose own two fields describe different files. WriteEcFiles now leaves both on the context, and fills a placeholder context in place so the caller can read them back. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: keep the source volume until every holder serves its shard layout The uniform layout rides in a .vif field older volume servers never knew: they discard it, mount the shards as legacy and return wrong bytes with nothing erroring, and the shard files are the same length either way so no other check notices. The upgrade order lived only in the release note. VolumeEcShardsInfo now reports the block size the holder actually serves, in both the Go and Rust servers, and the pre-delete verification refuses to drop the source unless every reachable holder echoes the one the shards were encoded with — while a rollback still exists. A server that predates the field answers 0, which is the negative answer. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: drop the rebuild's dead block-size parameters generateMissingEcFiles never reads largeBlockSize/smallBlockSize — Reed-Solomon reconstruction is layout-agnostic — so passing the legacy constants only advertised a layout the rebuild does not use. Also move UniformBlockSize's doc off ValidateBlockSize. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: warn about EC defaults only when the mount used them The "vif file not found, using defaults" warning fired even after the bitrot sidecar supplied a non-default layout, sending anyone triaging wrong bytes after the legacy layout the volume never mounted on. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: stat the distributed bitrot sidecar once The strict check re-stat'ed the file immediately before the stat that already gates inclusion, and a failed sidecar write now fails the encode outright, so the first could only fire on a deletion between the two lines. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: say what the reconstruct budget actually bounds A shard's buffer stays in bufs until its interval reconstructs, which is after the read that filled it released its permit, so the semaphore bounds round trips in flight and not retained bytes. Peak memory is the intervals reconstructing at once times the shards each reaches times the interval size. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * test: let the fake volume server report its delivered EC layout The pre-delete verification now asks each holder which shard block layout it serves, and a fake that always answered "unset" looked exactly like a volume server too old to know the field. Distribution ships the .vif to every holder alongside its shards, so read the layout back out of it as a real holder does. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
This commit is contained in:
Chris Lu
@@ -195,6 +195,7 @@ func buildProtectionFromBuilders(ctx *ECContext, builders []*shardChecksumBuilde
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: uint32(ctx.DataShards),
|
||||
ParityShards: uint32(ctx.ParityShards),
|
||||
BlockSize: ctx.BlockSize,
|
||||
},
|
||||
Shards: shards,
|
||||
EncodeUuid: NewEncodeUUID(),
|
||||
@@ -431,6 +432,7 @@ func ComputeProtectionFromShards(baseFileName string, ctx *ECContext, generation
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: uint32(ctx.DataShards),
|
||||
ParityShards: uint32(ctx.ParityShards),
|
||||
BlockSize: ctx.BlockSize,
|
||||
},
|
||||
Shards: shards,
|
||||
EncodeUuid: NewEncodeUUID(),
|
||||
@@ -474,53 +476,141 @@ func (ev *EcVolume) BitrotProtection() (*volume_server_pb.EcBitrotProtection, Bi
|
||||
return ev.bitrot, ev.bitrotStatus
|
||||
}
|
||||
|
||||
// ReloadBitrotSidecar re-resolves the checksum sidecar for a volume that is
|
||||
// already mounted — a shard delivery can bring the manifest with it, and the
|
||||
// receive path only writes the file, so without this the in-memory volume
|
||||
// keeps the protection state it resolved at mount (off) until a remount.
|
||||
func (ev *EcVolume) ReloadBitrotSidecar(additionalDirs ...string) {
|
||||
if err := ev.loadActiveBitrotSidecar(additionalDirs...); err != nil {
|
||||
glog.Warningf("ec volume %d: reload bitrot sidecar: %v", ev.VolumeId, err)
|
||||
}
|
||||
}
|
||||
|
||||
// loadActiveBitrotSidecar loads the generation-0 checksum sidecar into the
|
||||
// volume. Best-effort: any failure leaves protection off/invalid without
|
||||
// failing the mount. OSS only produces generation-0 (fresh-encode) sidecars.
|
||||
func (ev *EcVolume) loadActiveBitrotSidecar() {
|
||||
ev.loadBitrotForGeneration(0)
|
||||
func (ev *EcVolume) loadActiveBitrotSidecar(additionalDirs ...string) error {
|
||||
return ev.loadBitrotForGeneration(0, additionalDirs...)
|
||||
}
|
||||
|
||||
// loadBitrotForGeneration loads and validates the sidecar describing generation
|
||||
// `generation`, setting ev.bitrot/ev.bitrotStatus. Called at mount. Absent or
|
||||
// generation/config-mismatched => BitrotOff; self-integrity/manifest failure =>
|
||||
// generation-mismatched => BitrotOff; self-integrity/manifest failure =>
|
||||
// BitrotInvalid; usable => BitrotOn.
|
||||
func (ev *EcVolume) loadBitrotForGeneration(generation uint32) {
|
||||
//
|
||||
// A sidecar written FOR THIS generation that disagrees with the volume's shard
|
||||
// geometry is different in kind from those: both files record the layout the
|
||||
// generation was encoded with, so a disagreement means one of them is wrong and
|
||||
// reads through the other would land at the wrong shard offsets. That returns
|
||||
// an error and fails the mount instead of quietly dropping to unprotected
|
||||
// reads.
|
||||
func (ev *EcVolume) loadBitrotForGeneration(generation uint32, additionalDirs ...string) error {
|
||||
ev.bitrotLock.Lock()
|
||||
defer ev.bitrotLock.Unlock()
|
||||
ev.bitrot = nil
|
||||
ev.bitrotStatus = BitrotOff
|
||||
|
||||
if ev.ECContext == nil {
|
||||
return
|
||||
return nil
|
||||
}
|
||||
path := findBitrotSidecar(generation, ev.DataBaseFileName(), ev.IndexBaseFileName())
|
||||
path := findBitrotSidecar(generation, ev.DataBaseFileName(), ev.IndexBaseFileName(), additionalDirs...)
|
||||
if path == "" {
|
||||
return
|
||||
return nil
|
||||
}
|
||||
prot, err := LoadBitrotSidecar(path)
|
||||
if err != nil {
|
||||
glog.Warningf("ec volume %d: bitrot sidecar %s self-integrity failed: %v", ev.VolumeId, path, err)
|
||||
ev.bitrotStatus = BitrotInvalid
|
||||
return
|
||||
return nil
|
||||
}
|
||||
if prot.Generation != generation {
|
||||
return // not for this generation -> off, not corruption
|
||||
return nil // not for this generation -> off, not corruption
|
||||
}
|
||||
if prot.EcShardConfig == nil ||
|
||||
int(prot.EcShardConfig.DataShards) != ev.ECContext.DataShards ||
|
||||
int(prot.EcShardConfig.ParityShards) != ev.ECContext.ParityShards {
|
||||
return
|
||||
if prot.EcShardConfig == nil {
|
||||
return nil // records no geometry -> nothing to contradict
|
||||
}
|
||||
if int(prot.EcShardConfig.DataShards) != ev.ECContext.DataShards ||
|
||||
int(prot.EcShardConfig.ParityShards) != ev.ECContext.ParityShards ||
|
||||
prot.EcShardConfig.BlockSize != ev.ECContext.BlockSize {
|
||||
return fmt.Errorf("ec volume %d generation %d: %s records layout %d+%d block %d but the volume is mounted as %d+%d block %d; refusing to serve one of the two layouts",
|
||||
ev.VolumeId, generation, path,
|
||||
prot.EcShardConfig.DataShards, prot.EcShardConfig.ParityShards, prot.EcShardConfig.BlockSize,
|
||||
ev.ECContext.DataShards, ev.ECContext.ParityShards, ev.ECContext.BlockSize)
|
||||
}
|
||||
if err := ValidateBitrotManifest(prot, ev.ECContext.DataShards, ev.ECContext.ParityShards); err != nil {
|
||||
glog.Warningf("ec volume %d: bitrot sidecar %s manifest invalid: %v", ev.VolumeId, path, err)
|
||||
ev.bitrotStatus = BitrotInvalid
|
||||
return
|
||||
return nil
|
||||
}
|
||||
ev.bitrot = prot
|
||||
ev.bitrotStatus = BitrotOn
|
||||
glog.V(1).Infof("ec volume %d: loaded bitrot protection generation %d (%d shards, block_size %d)",
|
||||
ev.VolumeId, generation, len(prot.Shards), prot.BlockSize)
|
||||
return nil
|
||||
}
|
||||
|
||||
// layoutFromSidecar resolves a volume's EC layout from its generation-0
|
||||
// checksum sidecar, searching the data base and then the index base. A split
|
||||
// -dir/-dir.idx layout keeps the sidecar with the index, so probing the data
|
||||
// base alone reports "absent" for a volume that has the record right there —
|
||||
// and absent is the one answer that selects the legacy 10+4 layout.
|
||||
//
|
||||
// The three answers stay distinct: (nil, false, nil) means genuinely absent
|
||||
// and the caller may fall back to the defaults; a non-nil error means the
|
||||
// record exists but cannot establish the layout, which must fail rather than
|
||||
// default; otherwise the config is usable.
|
||||
func layoutFromSidecar(dataBaseFileName, indexBaseFileName string) (cfg *volume_server_pb.EcShardConfig, found bool, err error) {
|
||||
path := findBitrotSidecar(0, dataBaseFileName, indexBaseFileName)
|
||||
if path == "" {
|
||||
return nil, false, nil
|
||||
}
|
||||
cfg, err = EcShardConfigFromSidecarPath(path)
|
||||
return cfg, true, err
|
||||
}
|
||||
|
||||
// EcShardConfigFromSidecar reads the generation-0 bitrot sidecar's record of a
|
||||
// volume's EC config. The three answers are distinct on purpose: absent means
|
||||
// the volume may genuinely predate the sidecar, which is the only case a caller
|
||||
// may answer with the legacy layout; present-but-unusable means the one
|
||||
// surviving record of the geometry is corrupt, and guessing from there returns
|
||||
// wrong bytes rather than no bytes.
|
||||
func EcShardConfigFromSidecar(baseFileName string) (cfg *volume_server_pb.EcShardConfig, found bool, err error) {
|
||||
path := BitrotSidecarPath(baseFileName, 0)
|
||||
if _, statErr := os.Stat(path); statErr != nil {
|
||||
if os.IsNotExist(statErr) {
|
||||
return nil, false, nil
|
||||
}
|
||||
return nil, false, fmt.Errorf("stat %s: %w", path, statErr)
|
||||
}
|
||||
cfg, err = EcShardConfigFromSidecarPath(path)
|
||||
return cfg, true, err
|
||||
}
|
||||
|
||||
// EcShardConfigFromSidecarPath is EcShardConfigFromSidecar for a sidecar whose
|
||||
// path the caller already resolved — the rebuild finds it across a multi-disk
|
||||
// server's directories, not only next to the base name.
|
||||
func EcShardConfigFromSidecarPath(path string) (*volume_server_pb.EcShardConfig, error) {
|
||||
prot, loadErr := LoadBitrotSidecar(path)
|
||||
if loadErr != nil {
|
||||
return nil, fmt.Errorf("read %s: %w", path, loadErr)
|
||||
}
|
||||
// The un-suffixed sidecar describes generation 0 and nothing else; one
|
||||
// stamped for another generation is not a record of these shards.
|
||||
if prot.GetGeneration() != 0 {
|
||||
return nil, fmt.Errorf("%s records generation %d, not generation 0", path, prot.GetGeneration())
|
||||
}
|
||||
cfg := prot.GetEcShardConfig()
|
||||
if cfg == nil {
|
||||
return nil, fmt.Errorf("%s records no EC config", path)
|
||||
}
|
||||
if !ValidEcShardCounts(cfg.GetDataShards(), cfg.GetParityShards()) {
|
||||
return nil, fmt.Errorf("%s records invalid shard counts %d+%d",
|
||||
path, cfg.GetDataShards(), cfg.GetParityShards())
|
||||
}
|
||||
if bsErr := ValidateBlockSize(cfg.GetBlockSize()); bsErr != nil {
|
||||
return nil, fmt.Errorf("%s: %w", path, bsErr)
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// RemoveBitrotSidecars removes the legacy <base>.ecsum and any versioned
|
||||
@@ -534,6 +624,15 @@ func RemoveBitrotSidecars(base string) {
|
||||
}
|
||||
}
|
||||
|
||||
// FindBitrotSidecar is findBitrotSidecar for callers outside this package. It
|
||||
// answers "does this volume already have a manifest, and where" — a question
|
||||
// that cannot be settled from one base name, because a split -dir/-dir.idx
|
||||
// layout keeps the sidecar with the index and a multi-disk server may keep it
|
||||
// on a sibling. Returns "" when no candidate exists.
|
||||
func FindBitrotSidecar(generation uint32, dataBase, indexBase string, additionalDirs ...string) string {
|
||||
return findBitrotSidecar(generation, dataBase, indexBase, additionalDirs...)
|
||||
}
|
||||
|
||||
// findBitrotSidecar resolves the sidecar path for a generation, searching the
|
||||
// data base, the index base, and any additional directories — mirroring how
|
||||
// shard/.vif lookups handle split data/idx layouts and per-disk mirrors.
|
||||
|
||||
@@ -13,6 +13,16 @@ type ECContext struct {
|
||||
ParityShards int
|
||||
Collection string
|
||||
VolumeId needle.VolumeId
|
||||
// BlockSize > 0 selects the uniform block layout: every shard is a single
|
||||
// contiguous block of this many bytes, so consecutive .dat ranges stay on
|
||||
// one shard instead of striping across all of them at 1MiB granularity.
|
||||
// 0 is the legacy two-tier 1GiB/1MiB layout.
|
||||
BlockSize int64
|
||||
// DatFileSize is the length of the .dat the encode actually read, set by
|
||||
// WriteEcFiles from the same measurement it derives BlockSize from. The
|
||||
// .vif must record the two together: they describe one file, and a live
|
||||
// volume can grow between two separate stats.
|
||||
DatFileSize int64
|
||||
}
|
||||
|
||||
// Total returns the total number of shards (data + parity)
|
||||
@@ -20,6 +30,33 @@ func (ctx *ECContext) Total() int {
|
||||
return ctx.DataShards + ctx.ParityShards
|
||||
}
|
||||
|
||||
// ValidEcShardCounts reports whether a recorded (data, parity) pair could
|
||||
// describe a real EC volume. The sum is taken in uint64 deliberately: on a
|
||||
// 32-bit build `int` is 32 bits, so converting each count first and adding
|
||||
// them wraps for values near the uint32 ceiling — 0x7fffffff + 0x7fffffff
|
||||
// lands at -2, which slips under the MaxShardCount bound.
|
||||
func ValidEcShardCounts(dataShards, parityShards uint32) bool {
|
||||
return dataShards > 0 && parityShards > 0 &&
|
||||
uint64(dataShards)+uint64(parityShards) <= uint64(MaxShardCount)
|
||||
}
|
||||
|
||||
// LargeBlockSize returns the large-block length of this context's shard
|
||||
// layout; nil-safe so callers can pass an unset context for the legacy layout.
|
||||
func (ctx *ECContext) LargeBlockSize() int64 {
|
||||
if ctx != nil && ctx.BlockSize > 0 {
|
||||
return ctx.BlockSize
|
||||
}
|
||||
return ErasureCodingLargeBlockSize
|
||||
}
|
||||
|
||||
// SmallBlockSize returns the small-block length of this context's shard layout.
|
||||
func (ctx *ECContext) SmallBlockSize() int64 {
|
||||
if ctx != nil && ctx.BlockSize > 0 {
|
||||
return ctx.BlockSize
|
||||
}
|
||||
return ErasureCodingSmallBlockSize
|
||||
}
|
||||
|
||||
// NewDefaultECContext creates a context with default 10+4 shard configuration
|
||||
func NewDefaultECContext(collection string, volumeId needle.VolumeId) *ECContext {
|
||||
return &ECContext{
|
||||
|
||||
@@ -233,8 +233,10 @@ func iterateEcjFile(baseFileName string, processNeedleFn func(key types.NeedleId
|
||||
// large-block row boundary, and deriving the layout from the shrunk extent
|
||||
// would read the shards in the wrong block order. Pass zero when the .vif does
|
||||
// not record the encode-time size to infer the layout from the shard size.
|
||||
func WriteDatFile(baseFileName string, datFileSize int64, encodedDatFileSize int64, shardFileNames []string) error {
|
||||
return writeDatFile(baseFileName, datFileSize, encodedDatFileSize, shardFileNames, ErasureCodingLargeBlockSize, ErasureCodingSmallBlockSize)
|
||||
// largeBlockSize/smallBlockSize are the volume's shard block layout, e.g.
|
||||
// ctx.LargeBlockSize()/ctx.SmallBlockSize() from its .vif EC config.
|
||||
func WriteDatFile(baseFileName string, datFileSize int64, encodedDatFileSize int64, shardFileNames []string, largeBlockSize int64, smallBlockSize int64) error {
|
||||
return writeDatFile(baseFileName, datFileSize, encodedDatFileSize, shardFileNames, largeBlockSize, smallBlockSize)
|
||||
}
|
||||
|
||||
func writeDatFile(baseFileName string, datFileSize int64, encodedDatFileSize int64, shardFileNames []string, largeBlockSize int64, smallBlockSize int64) error {
|
||||
@@ -288,7 +290,11 @@ func writeDatFile(baseFileName string, datFileSize int64, encodedDatFileSize int
|
||||
// A shard size that is an exact multiple of the large block size is
|
||||
// ambiguous: N large rows, or N-1 large rows plus a full small-block
|
||||
// region. The two layouts only agree below the last large row.
|
||||
if shardSize%largeBlockSize == 0 && datFileSize > (shardSize/largeBlockSize-1)*largeBlockSize*int64(dataShards) {
|
||||
// A uniform layout has no such ambiguity — its large and small blocks
|
||||
// are the same size, so every reading of the shard is the same one, and
|
||||
// without this precondition the check fires on every volume.
|
||||
if largeBlockSize != smallBlockSize &&
|
||||
shardSize%largeBlockSize == 0 && datFileSize > (shardSize/largeBlockSize-1)*largeBlockSize*int64(dataShards) {
|
||||
return fmt.Errorf("shard size %d of %s does not identify the block layout; re-encode to record the dat size in .vif", shardSize, baseFileName)
|
||||
}
|
||||
encodedDatFileSize = int64(dataShards) * shardSize
|
||||
|
||||
@@ -217,7 +217,7 @@ func TestDecodeAtomicPublish(t *testing.T) {
|
||||
// final .dat nor a partial .dat.tmp behind.
|
||||
datBase := filepath.Join(dir, "bar_2")
|
||||
missingShards := []string{filepath.Join(dir, "does_not_exist.ec00")}
|
||||
if err := erasure_coding.WriteDatFile(datBase, 100, 100, missingShards); err == nil {
|
||||
if err := erasure_coding.WriteDatFile(datBase, 100, 100, missingShards, erasure_coding.ErasureCodingLargeBlockSize, erasure_coding.ErasureCodingSmallBlockSize); err == nil {
|
||||
t.Fatalf("expected WriteDatFile to fail on missing shard")
|
||||
}
|
||||
if _, err := os.Stat(datBase + ".dat"); !os.IsNotExist(err) {
|
||||
|
||||
@@ -62,12 +62,126 @@ func WriteSortedFileFromIdx(baseFileName string, ext string) (e error) {
|
||||
// BackgroundECContext for the default ratio, or an explicit ctx for a configured
|
||||
// (e.g. custom-ratio) layout. It returns the bitrot protection (per-shard block
|
||||
// CRC32C) computed during the single encode pass; the caller persists it as a
|
||||
// <base>.ecsum sidecar.
|
||||
// <base>.ecsum sidecar, and persists ctx.BlockSize and ctx.DatFileSize (both
|
||||
// set here, from one measurement of the .dat) to the .vif so readers resolve
|
||||
// the shard block layout.
|
||||
func WriteEcFiles(baseFileName string, ctx *ECContext) (*volume_server_pb.EcBitrotProtection, error) {
|
||||
if ctx == nil || ctx.Total() == 0 {
|
||||
if ctx == nil {
|
||||
ctx = NewDefaultECContext("", 0)
|
||||
} else if ctx.Total() == 0 {
|
||||
// Fill the placeholder in place rather than swapping the pointer: the
|
||||
// caller reads BlockSize and DatFileSize back off the context it
|
||||
// passed, and a replacement leaves it holding the zero values.
|
||||
ctx.DataShards, ctx.ParityShards = DataShardsCount, ParityShardsCount
|
||||
}
|
||||
return generateEcFiles(baseFileName, 256*1024, ErasureCodingLargeBlockSize, ErasureCodingSmallBlockSize, ctx)
|
||||
// Always encode with the uniform block layout, sized for this .dat. Both
|
||||
// the block size and the .dat length it was derived from are left on ctx,
|
||||
// so the caller persists a .vif whose two fields describe one measurement
|
||||
// — a second stat could see a different size on a volume still taking
|
||||
// writes.
|
||||
fi, err := os.Stat(baseFileName + ".dat")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to stat dat file: %w", err)
|
||||
}
|
||||
ctx.DatFileSize = fi.Size()
|
||||
ctx.BlockSize = UniformBlockSize(fi.Size(), ctx.DataShards)
|
||||
return generateEcFiles(baseFileName, 256*1024, ctx.BlockSize, ctx.BlockSize, ctx)
|
||||
}
|
||||
|
||||
// ValidateBlockSize reports whether a `.vif`-recorded shard block size is one
|
||||
// an encoder could have produced. 0 means the legacy two-tier layout, which is
|
||||
// always valid; anything positive must be a whole number of small blocks,
|
||||
// because that is what UniformBlockSize rounds to. A negative or unaligned
|
||||
// value is corruption, and using it would map every read to the wrong shard
|
||||
// offset.
|
||||
func ValidateBlockSize(blockSize int64) error {
|
||||
if blockSize == 0 {
|
||||
return nil
|
||||
}
|
||||
if blockSize < 0 || blockSize%ErasureCodingSmallBlockSize != 0 {
|
||||
return fmt.Errorf("invalid shard block size %d: expected 0 (legacy) or a multiple of %d",
|
||||
blockSize, ErasureCodingSmallBlockSize)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// UniformBlockSize returns the per-shard block size of the uniform layout for
|
||||
// a .dat of the given size: ceil(datFileSize/dataShards) rounded up to a whole
|
||||
// small block. For every input this equals the legacy layout's padded shard
|
||||
// size, so only the byte placement differs between the two layouts, never the
|
||||
// shard length.
|
||||
func UniformBlockSize(datFileSize int64, dataShards int) int64 {
|
||||
perShard := (datFileSize + int64(dataShards) - 1) / int64(dataShards)
|
||||
blocks := (perShard + ErasureCodingSmallBlockSize - 1) / ErasureCodingSmallBlockSize
|
||||
if blocks < 1 {
|
||||
blocks = 1
|
||||
}
|
||||
return blocks * ErasureCodingSmallBlockSize
|
||||
}
|
||||
|
||||
// ResolveRebuildECContext answers which shard layout a rebuild of baseFileName
|
||||
// will use: the caller's context when it already states one, else the volume's
|
||||
// own metadata — its `.vif` in any of the directories the rebuild can read,
|
||||
// then the generation-0 bitrot sidecar, and only then the build defaults.
|
||||
// Exported so a caller that must agree with the rebuild (the post-rebuild
|
||||
// bitrot backfill writes a manifest describing these very shards) resolves
|
||||
// once and uses the same answer, instead of re-deriving it from a narrower
|
||||
// search and recording a layout the rebuild did not use.
|
||||
func ResolveRebuildECContext(baseFileName string, ctx *ECContext, additionalDirs []string) (*ECContext, error) {
|
||||
if ctx == nil || ctx.Total() == 0 {
|
||||
// Resolve the layout from the .vif to preserve the original configuration.
|
||||
vifPath := findVifPath(baseFileName, additionalDirs)
|
||||
volumeInfo, _, foundVif, vifErr := volume_info.MaybeLoadVolumeInfo(vifPath)
|
||||
if vifErr != nil {
|
||||
// The .vif exists but cannot be read or parsed. Fail closed rather
|
||||
// than silently falling back to the default ratio, which would
|
||||
// rebuild a custom-ratio volume with the wrong layout. Pass an
|
||||
// explicit ctx to override.
|
||||
return nil, fmt.Errorf("RebuildEcFiles %s: cannot load .vif: %w", baseFileName, vifErr)
|
||||
}
|
||||
switch {
|
||||
case foundVif && volumeInfo.EcShardConfig != nil &&
|
||||
ValidEcShardCounts(volumeInfo.EcShardConfig.DataShards, volumeInfo.EcShardConfig.ParityShards):
|
||||
if bsErr := ValidateBlockSize(volumeInfo.EcShardConfig.GetBlockSize()); bsErr != nil {
|
||||
return nil, fmt.Errorf("RebuildEcFiles %s: %s: %w", baseFileName, vifPath, bsErr)
|
||||
}
|
||||
ctx = &ECContext{
|
||||
DataShards: int(volumeInfo.EcShardConfig.DataShards),
|
||||
ParityShards: int(volumeInfo.EcShardConfig.ParityShards),
|
||||
BlockSize: volumeInfo.EcShardConfig.GetBlockSize(),
|
||||
}
|
||||
glog.V(0).Infof("Rebuilding EC files for %s with config from .vif: %s", baseFileName, ctx.String())
|
||||
case foundVif && volumeInfo.EcShardConfig != nil:
|
||||
// A recorded-but-impossible ratio is corruption, not a reason to
|
||||
// substitute the default one: a 12+4 volume rebuilt as 10+4
|
||||
// reconstructs from the wrong matrix and never regenerates shards
|
||||
// 14-15. Pass an explicit ctx to override.
|
||||
return nil, fmt.Errorf("RebuildEcFiles %s: %s records invalid shard counts %d+%d",
|
||||
baseFileName, vifPath, volumeInfo.EcShardConfig.DataShards, volumeInfo.EcShardConfig.ParityShards)
|
||||
default:
|
||||
// No usable .vif: the bitrot sidecar records the same config at
|
||||
// encode time and is then the surviving authority. Reading the
|
||||
// default ratio and the legacy block size instead would rebuild
|
||||
// from the wrong geometry AND make the sidecar look like it
|
||||
// disagrees, which silently skips every checksum check below.
|
||||
if sidecarPath := findBitrotSidecar(0, baseFileName, baseFileName, additionalDirs...); sidecarPath != "" {
|
||||
cfg, cfgErr := EcShardConfigFromSidecarPath(sidecarPath)
|
||||
if cfgErr != nil {
|
||||
return nil, fmt.Errorf("RebuildEcFiles %s: no usable .vif and %w", baseFileName, cfgErr)
|
||||
}
|
||||
ctx = &ECContext{
|
||||
DataShards: int(cfg.GetDataShards()),
|
||||
ParityShards: int(cfg.GetParityShards()),
|
||||
BlockSize: cfg.GetBlockSize(),
|
||||
}
|
||||
glog.V(0).Infof("Rebuilding EC files for %s with config from the bitrot sidecar: %s", baseFileName, ctx.String())
|
||||
break
|
||||
}
|
||||
glog.V(0).Infof("Rebuilding EC files for %s with default config", baseFileName)
|
||||
ctx = NewDefaultECContext("", 0)
|
||||
}
|
||||
}
|
||||
return ctx, nil
|
||||
}
|
||||
|
||||
// RebuildEcFiles rebuilds missing EC shard files. Pass BackgroundECContext to
|
||||
@@ -79,38 +193,11 @@ func WriteEcFiles(baseFileName string, ctx *ECContext) (*volume_server_pb.EcBitr
|
||||
// present input shards are verified against it and corrupt ones are excluded
|
||||
// from Reed-Solomon and regenerated; unsafeIgnoreSidecar bypasses that guard.
|
||||
func RebuildEcFiles(baseFileName string, ctx *ECContext, unsafeIgnoreSidecar bool, additionalDirs ...string) ([]uint32, error) {
|
||||
if ctx == nil || ctx.Total() == 0 {
|
||||
// Resolve the layout from the .vif to preserve the original configuration.
|
||||
volumeInfo, _, foundVif, vifErr := volume_info.MaybeLoadVolumeInfo(baseFileName + ".vif")
|
||||
if vifErr != nil {
|
||||
// The .vif exists but cannot be read or parsed. Fail closed rather
|
||||
// than silently falling back to the default ratio, which would
|
||||
// rebuild a custom-ratio volume with the wrong layout. Pass an
|
||||
// explicit ctx to override.
|
||||
return nil, fmt.Errorf("RebuildEcFiles %s: cannot load .vif: %w", baseFileName, vifErr)
|
||||
}
|
||||
if foundVif && volumeInfo.EcShardConfig != nil {
|
||||
ds := int(volumeInfo.EcShardConfig.DataShards)
|
||||
ps := int(volumeInfo.EcShardConfig.ParityShards)
|
||||
|
||||
// Validate EC config before using it
|
||||
if ds > 0 && ps > 0 && ds+ps <= MaxShardCount {
|
||||
ctx = &ECContext{
|
||||
DataShards: ds,
|
||||
ParityShards: ps,
|
||||
}
|
||||
glog.V(0).Infof("Rebuilding EC files for %s with config from .vif: %s", baseFileName, ctx.String())
|
||||
} else {
|
||||
glog.Warningf("Invalid EC config in .vif for %s (data=%d, parity=%d), using default", baseFileName, ds, ps)
|
||||
ctx = NewDefaultECContext("", 0)
|
||||
}
|
||||
} else {
|
||||
glog.V(0).Infof("Rebuilding EC files for %s with default config", baseFileName)
|
||||
ctx = NewDefaultECContext("", 0)
|
||||
}
|
||||
ctx, err := ResolveRebuildECContext(baseFileName, ctx, additionalDirs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return generateMissingEcFiles(baseFileName, 256*1024, ErasureCodingLargeBlockSize, ErasureCodingSmallBlockSize, ctx, unsafeIgnoreSidecar, additionalDirs)
|
||||
return generateMissingEcFiles(baseFileName, 256*1024, ctx, unsafeIgnoreSidecar, additionalDirs)
|
||||
}
|
||||
|
||||
func ToExt(ecIndex int) string {
|
||||
@@ -159,7 +246,10 @@ func findShardFile(baseFileName string, ext string, additionalDirs []string) str
|
||||
return ""
|
||||
}
|
||||
|
||||
func generateMissingEcFiles(baseFileName string, bufferSize int, largeBlockSize int64, smallBlockSize int64, ctx *ECContext, unsafeIgnoreSidecar bool, additionalDirs []string) (generatedShardIds []uint32, err error) {
|
||||
// generateMissingEcFiles takes no block sizes: Reed-Solomon reconstruction is
|
||||
// layout-agnostic — it rebuilds a missing shard from the same offsets of the
|
||||
// survivors — so the shard layout only ever reaches it through ctx.
|
||||
func generateMissingEcFiles(baseFileName string, bufferSize int, ctx *ECContext, unsafeIgnoreSidecar bool, additionalDirs []string) (generatedShardIds []uint32, err error) {
|
||||
|
||||
// Pass 1: discover which shards exist and which are missing,
|
||||
// opening input files but NOT creating output files yet.
|
||||
@@ -363,6 +453,30 @@ func cleanupRebuildOutputs(outputFiles []*os.File, writePaths []string) {
|
||||
}
|
||||
}
|
||||
|
||||
// findVifPath locates the volume's `.vif` for a rebuild: next to the shards
|
||||
// first, then in every directory the caller also handed us — the index
|
||||
// directory of a split `-dir`/`-dir.idx` layout, and the sibling disks of a
|
||||
// multi-disk server, where `.ecx`/`.ecj`/`.vif` may live while this disk holds
|
||||
// only shards. Returns the data-base path when nothing exists, so the caller's
|
||||
// "not found" handling stays on the canonical name.
|
||||
func findVifPath(baseFileName string, additionalDirs []string) string {
|
||||
dataPath := baseFileName + ".vif"
|
||||
candidates := []string{dataPath}
|
||||
base := filepath.Base(baseFileName)
|
||||
for _, dir := range additionalDirs {
|
||||
if dir == "" {
|
||||
continue
|
||||
}
|
||||
candidates = append(candidates, filepath.Join(dir, base)+".vif")
|
||||
}
|
||||
for _, c := range candidates {
|
||||
if _, err := os.Stat(c); err == nil {
|
||||
return c
|
||||
}
|
||||
}
|
||||
return dataPath
|
||||
}
|
||||
|
||||
// loadRebuildSidecar loads and validates the generation-0 checksum sidecar for a
|
||||
// rebuild. RebuildEcFiles operates on the un-suffixed (generation 0) shard
|
||||
// names, so only the legacy sidecar is relevant here. Returns BitrotOff when
|
||||
@@ -381,10 +495,20 @@ func loadRebuildSidecar(baseFileName string, ctx *ECContext, additionalDirs []st
|
||||
if prot.Generation != 0 {
|
||||
return nil, BitrotOff
|
||||
}
|
||||
if prot.EcShardConfig == nil ||
|
||||
int(prot.EcShardConfig.DataShards) != ctx.DataShards ||
|
||||
int(prot.EcShardConfig.ParityShards) != ctx.ParityShards {
|
||||
return nil, BitrotOff
|
||||
if prot.EcShardConfig == nil {
|
||||
return nil, BitrotOff // records no geometry -> nothing to contradict
|
||||
}
|
||||
if int(prot.EcShardConfig.DataShards) != ctx.DataShards ||
|
||||
int(prot.EcShardConfig.ParityShards) != ctx.ParityShards ||
|
||||
prot.EcShardConfig.BlockSize != ctx.BlockSize {
|
||||
// Both records describe the same encode, so a disagreement means the
|
||||
// rebuild is about to reconstruct under a geometry the checksums do not
|
||||
// cover. Treating that as "no protection" skipped every input and
|
||||
// output check exactly when they matter most.
|
||||
glog.Warningf("bitrot: sidecar %s records layout %d+%d block %d but the rebuild uses %d+%d block %d",
|
||||
path, prot.EcShardConfig.DataShards, prot.EcShardConfig.ParityShards, prot.EcShardConfig.BlockSize,
|
||||
ctx.DataShards, ctx.ParityShards, ctx.BlockSize)
|
||||
return nil, BitrotInvalid
|
||||
}
|
||||
if err := ValidateBitrotManifest(prot, ctx.DataShards, ctx.ParityShards); err != nil {
|
||||
glog.Warningf("bitrot: sidecar %s manifest invalid: %v", path, err)
|
||||
|
||||
@@ -5,7 +5,10 @@ import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/types"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/volume_info"
|
||||
)
|
||||
|
||||
// readEcxSizeField returns the size field of the .ecx entry for needleId, or
|
||||
@@ -295,3 +298,101 @@ func TestRebuildEcFiles_CustomRatioRebuildsByteIdentical(t *testing.T) {
|
||||
t.Errorf("unexpected shard .ec12 for a 9+3 volume")
|
||||
}
|
||||
}
|
||||
|
||||
// With no .vif, the bitrot sidecar is the surviving record of the volume's
|
||||
// geometry. Defaulting to 10+4 and the legacy block layout instead would
|
||||
// reconstruct a custom-ratio volume from the wrong matrix — and would make the
|
||||
// sidecar look like it disagrees, silently skipping every checksum check.
|
||||
func TestRebuildEcFiles_ResolvesGeometryFromSidecarWithoutVif(t *testing.T) {
|
||||
if !BitrotProtectionEnabled {
|
||||
t.Skip("bitrot protection is compiled out")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "vol")
|
||||
ctx := &ECContext{DataShards: 12, ParityShards: 4}
|
||||
writeRandomDat(t, base, 7000)
|
||||
|
||||
prot, err := WriteEcFiles(base, ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("WriteEcFiles: %v", err)
|
||||
}
|
||||
if err := SaveBitrotSidecar(BitrotSidecarPath(base, 0), prot); err != nil {
|
||||
t.Fatalf("save sidecar: %v", err)
|
||||
}
|
||||
// No .vif at all: the sidecar has to answer.
|
||||
const dropped = 11
|
||||
want, err := os.ReadFile(base + ToExt(dropped))
|
||||
if err != nil {
|
||||
t.Fatalf("read shard: %v", err)
|
||||
}
|
||||
if err := os.Remove(base + ToExt(dropped)); err != nil {
|
||||
t.Fatalf("remove shard: %v", err)
|
||||
}
|
||||
|
||||
// nil ctx: the production RPC's BackgroundECContext path.
|
||||
if _, err := RebuildEcFiles(base, nil, false); err != nil {
|
||||
t.Fatalf("RebuildEcFiles: %v", err)
|
||||
}
|
||||
got, err := os.ReadFile(base + ToExt(dropped))
|
||||
if err != nil {
|
||||
t.Fatalf("read rebuilt shard: %v", err)
|
||||
}
|
||||
if string(got) != string(want) {
|
||||
t.Errorf("rebuilt shard %d differs; the rebuild used the wrong geometry", dropped)
|
||||
}
|
||||
// The 12+4 shards past the default ratio must still be there.
|
||||
for _, id := range []int{14, 15} {
|
||||
if _, err := os.Stat(base + ToExt(id)); err != nil {
|
||||
t.Errorf("shard %d missing after rebuild: %v", id, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A split -dir/-dir.idx layout keeps the .vif with the index, and a multi-disk
|
||||
// server may leave this disk holding only shards. Probing just the data base
|
||||
// resolves such a volume to the default ratio and the legacy block layout.
|
||||
func TestRebuildEcFiles_FindsVifInAnAdditionalDir(t *testing.T) {
|
||||
dataDir := t.TempDir()
|
||||
idxDir := t.TempDir()
|
||||
base := filepath.Join(dataDir, "vol")
|
||||
ctx := &ECContext{DataShards: 12, ParityShards: 4}
|
||||
writeRandomDat(t, base, 7000)
|
||||
|
||||
if _, err := WriteEcFiles(base, ctx); err != nil {
|
||||
t.Fatalf("WriteEcFiles: %v", err)
|
||||
}
|
||||
// The volume's .vif lives with the index, not the shards.
|
||||
if err := volume_info.SaveVolumeInfo(filepath.Join(idxDir, "vol")+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.GetCurrentVersion()),
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: 12, ParityShards: 4, BlockSize: ctx.BlockSize,
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("save .vif: %v", err)
|
||||
}
|
||||
|
||||
const dropped = 11
|
||||
want, err := os.ReadFile(base + ToExt(dropped))
|
||||
if err != nil {
|
||||
t.Fatalf("read shard: %v", err)
|
||||
}
|
||||
if err := os.Remove(base + ToExt(dropped)); err != nil {
|
||||
t.Fatalf("remove shard: %v", err)
|
||||
}
|
||||
|
||||
if _, err := RebuildEcFiles(base, nil, true, idxDir); err != nil {
|
||||
t.Fatalf("RebuildEcFiles: %v", err)
|
||||
}
|
||||
got, err := os.ReadFile(base + ToExt(dropped))
|
||||
if err != nil {
|
||||
t.Fatalf("read rebuilt shard: %v", err)
|
||||
}
|
||||
if string(got) != string(want) {
|
||||
t.Errorf("rebuilt shard %d differs; the rebuild used the wrong geometry", dropped)
|
||||
}
|
||||
for _, id := range []int{14, 15} {
|
||||
if _, err := os.Stat(base + ToExt(id)); err != nil {
|
||||
t.Errorf("shard %d missing after rebuild: %v", id, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -326,7 +326,7 @@ func testDecodeDat(t *testing.T, datSize int64) {
|
||||
shardFileNames[i] = fmt.Sprintf("%s%s", baseFileName, ctx.ToExt(i))
|
||||
}
|
||||
|
||||
err = WriteDatFile(decodedBase, datSize, datSize, shardFileNames)
|
||||
err = WriteDatFile(decodedBase, datSize, datSize, shardFileNames, ErasureCodingLargeBlockSize, ErasureCodingSmallBlockSize)
|
||||
require.NoError(t, err, "WriteDatFile")
|
||||
|
||||
// The atomic publish must rename the temp file away, never leaving it behind.
|
||||
|
||||
@@ -0,0 +1,250 @@
|
||||
package erasure_coding
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/types"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/volume_info"
|
||||
)
|
||||
|
||||
// legacyShardSize is the padded shard length the two-tier layout produces:
|
||||
// whole 1GiB rows, then the remainder in 1MiB rows.
|
||||
func legacyShardSize(datFileSize int64, dataShards int) int64 {
|
||||
largeRow := int64(ErasureCodingLargeBlockSize) * int64(dataShards)
|
||||
nLarge := datFileSize / largeRow
|
||||
size := nLarge * ErasureCodingLargeBlockSize
|
||||
if rem := datFileSize - nLarge*largeRow; rem > 0 {
|
||||
smallRow := int64(ErasureCodingSmallBlockSize) * int64(dataShards)
|
||||
size += (rem + smallRow - 1) / smallRow * ErasureCodingSmallBlockSize
|
||||
}
|
||||
return size
|
||||
}
|
||||
|
||||
// The uniform layout must never change a shard's length, only the byte
|
||||
// placement — capacity math and shard-size credibility checks depend on it.
|
||||
func TestUniformBlockSizeMatchesLegacyShardSize(t *testing.T) {
|
||||
sizes := []int64{
|
||||
8, 1000, ErasureCodingSmallBlockSize,
|
||||
10 * ErasureCodingSmallBlockSize,
|
||||
10*ErasureCodingSmallBlockSize + 1,
|
||||
25 * 1024 * 1024,
|
||||
10 * ErasureCodingLargeBlockSize,
|
||||
10*ErasureCodingLargeBlockSize + 1,
|
||||
10*ErasureCodingLargeBlockSize - ErasureCodingSmallBlockSize,
|
||||
30 * 1024 * 1024 * 1024,
|
||||
30*1024*1024*1024 + 12345,
|
||||
}
|
||||
for _, datSize := range sizes {
|
||||
for _, ds := range []int{10, 9, 5, 1} {
|
||||
if got, want := UniformBlockSize(datSize, ds), legacyShardSize(datSize, ds); got != want {
|
||||
t.Errorf("UniformBlockSize(%d, %d) = %d, legacy shard size %d", datSize, ds, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
if got := UniformBlockSize(0, 10); got != ErasureCodingSmallBlockSize {
|
||||
t.Errorf("UniformBlockSize(0, 10) = %d, want one small block", got)
|
||||
}
|
||||
}
|
||||
|
||||
// encodeLayoutFixture writes a random .dat and encodes it with the given block
|
||||
// sizes, returning the .dat content.
|
||||
func encodeLayoutFixture(t *testing.T, baseFileName string, datSize int64, large, small int64, ctx *ECContext) []byte {
|
||||
t.Helper()
|
||||
data := make([]byte, datSize)
|
||||
rand.New(rand.NewSource(datSize)).Read(data)
|
||||
if err := os.WriteFile(baseFileName+".dat", data, 0o644); err != nil {
|
||||
t.Fatalf("write .dat: %v", err)
|
||||
}
|
||||
if _, err := generateEcFiles(baseFileName, 256*1024, large, small, ctx); err != nil {
|
||||
t.Fatalf("generateEcFiles: %v", err)
|
||||
}
|
||||
return data
|
||||
}
|
||||
|
||||
// readIntervals assembles a byte range from the shard files the way the read
|
||||
// path does.
|
||||
func readIntervals(t *testing.T, baseFileName string, ctx *ECContext, intervals []Interval) []byte {
|
||||
t.Helper()
|
||||
var assembled []byte
|
||||
for _, iv := range intervals {
|
||||
shardId, shardOffset := iv.ToShardIdAndOffset(ctx.LargeBlockSize(), ctx.SmallBlockSize())
|
||||
f, err := os.Open(baseFileName + ctx.ToExt(int(shardId)))
|
||||
if err != nil {
|
||||
t.Fatalf("open shard %d: %v", shardId, err)
|
||||
}
|
||||
buf := make([]byte, iv.Size)
|
||||
if _, err := f.ReadAt(buf, shardOffset); err != nil {
|
||||
t.Fatalf("read shard %d at %d: %v", shardId, shardOffset, err)
|
||||
}
|
||||
f.Close()
|
||||
assembled = append(assembled, buf...)
|
||||
}
|
||||
return assembled
|
||||
}
|
||||
|
||||
// Encode 25MB — large enough that the uniform and legacy layouts place bytes
|
||||
// differently — under both layouts and verify LocateData maps every probed
|
||||
// range back to the original bytes.
|
||||
func TestLocateDataMatchesEncoderForBothLayouts(t *testing.T) {
|
||||
const datSize = 25*1024*1024 + 12345
|
||||
dir := t.TempDir()
|
||||
|
||||
uniformCtx := NewDefaultECContext("", 1)
|
||||
uniformCtx.BlockSize = UniformBlockSize(datSize, uniformCtx.DataShards)
|
||||
legacyCtx := NewDefaultECContext("", 2)
|
||||
|
||||
for name, ctx := range map[string]*ECContext{"uniform": uniformCtx, "legacy": legacyCtx} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
base := fmt.Sprintf("%s/%s", dir, name)
|
||||
data := encodeLayoutFixture(t, base, datSize, ctx.LargeBlockSize(), ctx.SmallBlockSize(), ctx)
|
||||
|
||||
shardSize := datSize / int64(ctx.DataShards)
|
||||
for _, probe := range []struct{ offset, size int64 }{
|
||||
{0, 1000},
|
||||
{ctx.SmallBlockSize() - 100, 200}, // straddles a block boundary
|
||||
{5 * 1024 * 1024, 4 * 1024 * 1024},
|
||||
{datSize - 2000, 2000},
|
||||
} {
|
||||
intervals := LocateData(ctx.LargeBlockSize(), ctx.SmallBlockSize(), shardSize, probe.offset, types.Size(probe.size))
|
||||
got := readIntervals(t, base, ctx, intervals)
|
||||
if !bytes.Equal(got, data[probe.offset:probe.offset+probe.size]) {
|
||||
t.Fatalf("%s layout: bytes at [%d,+%d) do not round-trip", name, probe.offset, probe.size)
|
||||
}
|
||||
}
|
||||
|
||||
// The point of the uniform layout: a 2MB range inside one 3MB
|
||||
// block is a single interval instead of three 1MB stripes.
|
||||
intervals := LocateData(ctx.LargeBlockSize(), ctx.SmallBlockSize(), shardSize, 6*1024*1024+512*1024, types.Size(2*1024*1024))
|
||||
if name == "uniform" && len(intervals) != 1 {
|
||||
t.Errorf("uniform layout reads 2MB in %d intervals, want 1", len(intervals))
|
||||
}
|
||||
if name == "legacy" && len(intervals) != 3 {
|
||||
t.Errorf("legacy layout reads 2MB in %d intervals, want 3", len(intervals))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Encode under both layouts and verify WriteDatFile reconstructs the original
|
||||
// .dat byte for byte — and that decoding a uniform volume with the legacy
|
||||
// geometry does NOT, i.e. the recorded block size is load-bearing.
|
||||
func TestUniformDecodeRoundTrip(t *testing.T) {
|
||||
const datSize = 25 * 1024 * 1024
|
||||
dir := t.TempDir()
|
||||
base := dir + "/1"
|
||||
|
||||
ctx := NewDefaultECContext("", 1)
|
||||
ctx.BlockSize = UniformBlockSize(datSize, ctx.DataShards)
|
||||
data := encodeLayoutFixture(t, base, datSize, ctx.BlockSize, ctx.BlockSize, ctx)
|
||||
|
||||
shardFileNames := make([]string, ctx.DataShards)
|
||||
for i := range shardFileNames {
|
||||
shardFileNames[i] = base + ctx.ToExt(i)
|
||||
}
|
||||
|
||||
if err := WriteDatFile(dir+"/decoded", datSize, datSize, shardFileNames, ctx.LargeBlockSize(), ctx.SmallBlockSize()); err != nil {
|
||||
t.Fatalf("WriteDatFile: %v", err)
|
||||
}
|
||||
decoded, err := os.ReadFile(dir + "/decoded.dat")
|
||||
if err != nil {
|
||||
t.Fatalf("read decoded: %v", err)
|
||||
}
|
||||
if !bytes.Equal(decoded, data) {
|
||||
t.Fatal("uniform decode with recorded geometry does not round-trip")
|
||||
}
|
||||
|
||||
if err := WriteDatFile(dir+"/wrong", datSize, datSize, shardFileNames, ErasureCodingLargeBlockSize, ErasureCodingSmallBlockSize); err != nil {
|
||||
t.Fatalf("WriteDatFile legacy geometry: %v", err)
|
||||
}
|
||||
wrong, err := os.ReadFile(dir + "/wrong.dat")
|
||||
if err != nil {
|
||||
t.Fatalf("read wrong: %v", err)
|
||||
}
|
||||
if bytes.Equal(wrong, data) {
|
||||
t.Fatal("decoding a uniform volume with legacy geometry should scramble it; the layouts did not diverge")
|
||||
}
|
||||
}
|
||||
|
||||
// Mount fixtures through NewEcVolume and read a large extent back through
|
||||
// LocateEcShardNeedleInterval, proving the .vif block size steers the read
|
||||
// path: a legacy .vif (no block size) keeps the legacy interpretation, a
|
||||
// uniform .vif reads uniform shards.
|
||||
func TestEcVolumeGeometryFromVif(t *testing.T) {
|
||||
const datSize = 25 * 1024 * 1024
|
||||
|
||||
for _, layout := range []string{"legacy", "uniform"} {
|
||||
t.Run(layout, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
vid := needle.VolumeId(7)
|
||||
base := EcShardFileName("", dir, int(vid))
|
||||
|
||||
ctx := NewDefaultECContext("", vid)
|
||||
if layout == "uniform" {
|
||||
ctx.BlockSize = UniformBlockSize(datSize, ctx.DataShards)
|
||||
}
|
||||
data := encodeLayoutFixture(t, base, datSize, ctx.LargeBlockSize(), ctx.SmallBlockSize(), ctx)
|
||||
|
||||
if err := os.WriteFile(base+".ecx", nil, 0o644); err != nil {
|
||||
t.Fatalf("write .ecx: %v", err)
|
||||
}
|
||||
if err := volume_info.SaveVolumeInfo(base+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.GetCurrentVersion()),
|
||||
DatFileSize: datSize,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: uint32(ctx.DataShards),
|
||||
ParityShards: uint32(ctx.ParityShards),
|
||||
BlockSize: ctx.BlockSize,
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("save .vif: %v", err)
|
||||
}
|
||||
|
||||
ev, err := NewEcVolume(types.HardDriveType, dir, dir, "", vid)
|
||||
if err != nil {
|
||||
t.Fatalf("NewEcVolume: %v", err)
|
||||
}
|
||||
defer ev.Close()
|
||||
if ev.ECContext.BlockSize != ctx.BlockSize {
|
||||
t.Fatalf("loaded BlockSize = %d, want %d", ev.ECContext.BlockSize, ctx.BlockSize)
|
||||
}
|
||||
for i := 0; i < ctx.DataShards; i++ {
|
||||
shard, err := NewEcVolumeShard(types.HardDriveType, dir, "", vid, ShardId(i))
|
||||
if err != nil {
|
||||
t.Fatalf("NewEcVolumeShard %d: %v", i, err)
|
||||
}
|
||||
ev.AddEcVolumeShard(shard)
|
||||
}
|
||||
|
||||
// Sweep a large extent through the volume's own interval mapping.
|
||||
// LocateEcShardNeedleInterval expands the size by the needle
|
||||
// overhead, so compare only the probed prefix.
|
||||
const probeSize = datSize - 64*1024
|
||||
intervals := ev.LocateEcShardNeedleInterval(needle.GetCurrentVersion(), 0, types.Size(probeSize))
|
||||
var assembled []byte
|
||||
for _, iv := range intervals {
|
||||
shardId, shardOffset := ev.IntervalToShardIdAndOffset(iv)
|
||||
shard, found := ev.FindEcVolumeShard(shardId)
|
||||
if !found {
|
||||
t.Fatalf("shard %d not mounted", shardId)
|
||||
}
|
||||
buf := make([]byte, iv.Size)
|
||||
if _, err := shard.ReadAt(buf, shardOffset); err != nil {
|
||||
t.Fatalf("read shard %d at %d: %v", shardId, shardOffset, err)
|
||||
}
|
||||
assembled = append(assembled, buf...)
|
||||
}
|
||||
if len(assembled) < probeSize {
|
||||
t.Fatalf("assembled %d bytes, want at least %d", len(assembled), probeSize)
|
||||
}
|
||||
if !bytes.Equal(assembled[:probeSize], data[:probeSize]) {
|
||||
t.Fatalf("%s layout: full-extent read through the %s .vif does not match the .dat", layout, layout)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -173,7 +173,16 @@ func NewEcVolume(diskType types.DiskType, dir string, dirIdx string, collection
|
||||
}
|
||||
}
|
||||
ev.Version = needle.Version3
|
||||
if volumeInfo, _, found, _ := volume_info.MaybeLoadVolumeInfo(vifFileName); found {
|
||||
// A present-but-unreadable or malformed .vif FAILS the mount: every new
|
||||
// encode records a positive uniform block size there, and defaulting to
|
||||
// the legacy layout would serve those shards with the wrong offset math.
|
||||
// Absent stays legal — legacy volumes predate the sidecar.
|
||||
volumeInfo, _, found, vifErr := volume_info.MaybeLoadVolumeInfo(vifFileName)
|
||||
if vifErr != nil {
|
||||
ev.Close()
|
||||
return nil, fmt.Errorf("ec volume %d: load %s: %w", vid, vifFileName, vifErr)
|
||||
}
|
||||
if found {
|
||||
ev.Version = needle.Version(volumeInfo.Version)
|
||||
ev.datFileSize = volumeInfo.DatFileSize
|
||||
ev.ExpireAtSec = volumeInfo.ExpireAtSec
|
||||
@@ -184,21 +193,56 @@ func NewEcVolume(diskType types.DiskType, dir string, dirIdx string, collection
|
||||
ps := int(volumeInfo.EcShardConfig.ParityShards)
|
||||
ev.EncodeTsNs = volumeInfo.EcShardConfig.GetEncodeTsNs()
|
||||
|
||||
// Validate shard counts to prevent zero or invalid values
|
||||
if ds <= 0 || ps <= 0 || ds+ps > MaxShardCount {
|
||||
glog.Warningf("Invalid EC config in VolumeInfo for volume %d (data=%d, parity=%d), using defaults", vid, ds, ps)
|
||||
ev.ECContext = NewDefaultECContext(collection, vid)
|
||||
// A config that is PRESENT but records an impossible ratio is not
|
||||
// a volume to fall back on: substituting the default 10+4 with the
|
||||
// legacy layout would read uniform shards with the wrong offset
|
||||
// math and answer with the wrong bytes. Only an ENTIRELY absent
|
||||
// config means "this predates the record", which the else-branch
|
||||
// below serves with the legacy defaults.
|
||||
if !ValidEcShardCounts(volumeInfo.EcShardConfig.DataShards, volumeInfo.EcShardConfig.ParityShards) {
|
||||
ev.Close()
|
||||
return nil, fmt.Errorf("ec volume %d: %s records invalid shard counts %d+%d",
|
||||
vid, vifFileName, volumeInfo.EcShardConfig.DataShards, volumeInfo.EcShardConfig.ParityShards)
|
||||
} else if blockErr := ValidateBlockSize(volumeInfo.EcShardConfig.GetBlockSize()); blockErr != nil {
|
||||
// A recorded block size that no encoder could have produced maps
|
||||
// every read to the wrong shard offset. Refuse the mount rather
|
||||
// than serve those bytes or silently pick a layout.
|
||||
ev.Close()
|
||||
return nil, fmt.Errorf("ec volume %d: %s: %w", vid, vifFileName, blockErr)
|
||||
} else {
|
||||
ev.ECContext = &ECContext{
|
||||
Collection: collection,
|
||||
VolumeId: vid,
|
||||
DataShards: ds,
|
||||
ParityShards: ps,
|
||||
BlockSize: volumeInfo.EcShardConfig.GetBlockSize(),
|
||||
}
|
||||
glog.V(1).Infof("Loaded EC config from VolumeInfo for volume %d: %s", vid, ev.ECContext.String())
|
||||
}
|
||||
} else {
|
||||
ev.ECContext = NewDefaultECContext(collection, vid)
|
||||
// A vif that carries no ecShardConfig answers nothing about the
|
||||
// layout — it is no more informative than an absent one, so it
|
||||
// must not skip the sidecar. Going straight to the defaults here
|
||||
// read a uniform volume with the legacy offset math.
|
||||
cfg, sidecarFound, sidecarErr := layoutFromSidecar(dataBaseFileName, indexBaseFileName)
|
||||
if sidecarErr != nil {
|
||||
ev.Close()
|
||||
return nil, fmt.Errorf("ec volume %d: %s records no EC config and the bitrot sidecar cannot establish the layout: %w", vid, vifFileName, sidecarErr)
|
||||
}
|
||||
if sidecarFound {
|
||||
ev.ECContext = &ECContext{
|
||||
Collection: collection,
|
||||
VolumeId: vid,
|
||||
DataShards: int(cfg.GetDataShards()),
|
||||
ParityShards: int(cfg.GetParityShards()),
|
||||
BlockSize: cfg.GetBlockSize(),
|
||||
}
|
||||
ev.EncodeTsNs = cfg.GetEncodeTsNs()
|
||||
glog.V(0).Infof("ec volume %d: .vif records no EC config; took it from the bitrot sidecar: %s",
|
||||
vid, ev.ECContext.String())
|
||||
} else {
|
||||
ev.ECContext = NewDefaultECContext(collection, vid)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Don't fabricate a stub .vif here: a version-only stub implies the
|
||||
@@ -207,14 +251,48 @@ func NewEcVolume(diskType types.DiskType, dir string, dirIdx string, collection
|
||||
// mistake for an authoritative config. Mount with in-memory defaults and
|
||||
// leave the real .vif to the encoder or a recovery tool (the Rust volume
|
||||
// server already behaves this way).
|
||||
glog.Warningf("vif file not found, using defaults, volumeId:%d, filename:%s", vid, vifFileName)
|
||||
//
|
||||
// The bitrot sidecar records the same EC config at encode time, so when
|
||||
// it is present it answers the layout question the missing .vif cannot:
|
||||
// defaulting a uniform-layout volume to the legacy block sizes maps
|
||||
// every read to the wrong shard offset. `weed fix -ecx` reads the
|
||||
// sidecar for the same reason.
|
||||
ev.ECContext = NewDefaultECContext(collection, vid)
|
||||
cfg, sidecarFound, sidecarErr := layoutFromSidecar(dataBaseFileName, indexBaseFileName)
|
||||
if sidecarErr != nil {
|
||||
// With no .vif the sidecar is the ONLY record of this volume's
|
||||
// layout. Present but unusable is not "assume legacy" — that
|
||||
// answers reads with the wrong shard offsets, which is worse than
|
||||
// not answering at all.
|
||||
ev.Close()
|
||||
return nil, fmt.Errorf("ec volume %d: no .vif and the bitrot sidecar cannot establish the layout: %w", vid, sidecarErr)
|
||||
}
|
||||
if sidecarFound {
|
||||
ev.ECContext = &ECContext{
|
||||
Collection: collection,
|
||||
VolumeId: vid,
|
||||
DataShards: int(cfg.GetDataShards()),
|
||||
ParityShards: int(cfg.GetParityShards()),
|
||||
BlockSize: cfg.GetBlockSize(),
|
||||
}
|
||||
ev.EncodeTsNs = cfg.GetEncodeTsNs()
|
||||
glog.V(0).Infof("ec volume %d: .vif missing; took EC config from the bitrot sidecar: %s",
|
||||
vid, ev.ECContext.String())
|
||||
} else {
|
||||
// Only now are the defaults what the volume actually mounted on;
|
||||
// logging this after the sidecar answered would send an operator
|
||||
// triaging wrong bytes after the legacy layout instead.
|
||||
glog.Warningf("vif file not found, using defaults, volumeId:%d, filename:%s", vid, vifFileName)
|
||||
}
|
||||
}
|
||||
|
||||
ev.ShardLocations = make(map[ShardId][]pb.ServerAddress)
|
||||
|
||||
// Load the active-generation bitrot checksum sidecar (optional).
|
||||
ev.loadActiveBitrotSidecar()
|
||||
if err := ev.loadActiveBitrotSidecar(); err != nil {
|
||||
ev.Close()
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
@@ -528,11 +606,17 @@ func (ev *EcVolume) LocateEcShardNeedleInterval(version needle.Version, offset i
|
||||
shardSize = shard.ecdFileSize - 1
|
||||
}
|
||||
// calculate the locations in the ec shards
|
||||
intervals = LocateData(ErasureCodingLargeBlockSize, ErasureCodingSmallBlockSize, shardSize, offset, types.Size(needle.GetActualSize(size, version)))
|
||||
intervals = LocateData(ev.ECContext.LargeBlockSize(), ev.ECContext.SmallBlockSize(), shardSize, offset, types.Size(needle.GetActualSize(size, version)))
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// IntervalToShardIdAndOffset resolves an interval against this volume's shard
|
||||
// block layout.
|
||||
func (ev *EcVolume) IntervalToShardIdAndOffset(interval Interval) (ShardId, int64) {
|
||||
return interval.ToShardIdAndOffset(ev.ECContext.LargeBlockSize(), ev.ECContext.SmallBlockSize())
|
||||
}
|
||||
|
||||
func (ev *EcVolume) FindNeedleFromEcx(needleId types.NeedleId) (offset types.Offset, size types.Size, err error) {
|
||||
offset, size, err = SearchNeedleFromSortedIndex(ev.ecxFile, ev.ecxFileSize, needleId, nil)
|
||||
if err != nil {
|
||||
|
||||
@@ -238,7 +238,7 @@ func (ecv *EcVolume) ScrubLocal() (int64, []*volume_server_pb.EcShardInfo, []err
|
||||
localShardIds := []ShardId{}
|
||||
|
||||
for i, iv := range locations {
|
||||
sid, soffset := iv.ToShardIdAndOffset(ErasureCodingLargeBlockSize, ErasureCodingSmallBlockSize)
|
||||
sid, soffset := ecv.IntervalToShardIdAndOffset(iv)
|
||||
ssize := int64(iv.Size.Raw())
|
||||
shard, found := ecv.FindEcVolumeShard(sid)
|
||||
|
||||
|
||||
@@ -0,0 +1,246 @@
|
||||
package erasure_coding
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/types"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/volume_info"
|
||||
)
|
||||
|
||||
func uniformSidecar(t *testing.T, base string, ds, ps uint32, blockSize int64) {
|
||||
t.Helper()
|
||||
prot := &volume_server_pb.EcBitrotProtection{
|
||||
Algorithm: volume_server_pb.ChecksumAlgorithm_CHECKSUM_CRC32C,
|
||||
BlockSize: uint32(BitrotBlockSize),
|
||||
Generation: 0,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: ds, ParityShards: ps, BlockSize: blockSize,
|
||||
},
|
||||
}
|
||||
if err := SaveBitrotSidecar(BitrotSidecarPath(base, 0), prot); err != nil {
|
||||
t.Fatalf("seed .ecsum: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The sidecar records the layout at encode time, so it answers what a .vif
|
||||
// cannot. A .vif that merely omits ecShardConfig is no more informative than an
|
||||
// absent one — going straight to the defaults reads a uniform volume with the
|
||||
// legacy offset math and returns the wrong bytes.
|
||||
func TestNewEcVolumeTakesLayoutFromSidecarWhenVifHasNoConfig(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "1")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
// A config-free .vif: version and dat size only, as a legacy encode left it.
|
||||
if err := volume_info.SaveVolumeInfo(base+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
uniformSidecar(t, base, 12, 4, 3*1024*1024)
|
||||
|
||||
ev, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(1))
|
||||
if err != nil {
|
||||
t.Fatalf("mount: %v", err)
|
||||
}
|
||||
defer ev.Close()
|
||||
if got := ev.ECContext; got.DataShards != 12 || got.ParityShards != 4 || got.BlockSize != 3*1024*1024 {
|
||||
t.Errorf("layout = %d+%d block %d, want 12+4 block 3MiB", got.DataShards, got.ParityShards, got.BlockSize)
|
||||
}
|
||||
}
|
||||
|
||||
// A split -dir/-dir.idx layout keeps the sidecar with the INDEX. Probing only
|
||||
// the data base reports "absent", and absent is the one answer that selects the
|
||||
// legacy layout.
|
||||
func TestNewEcVolumeFindsSidecarInTheIndexDirectory(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
writeVif bool
|
||||
}{
|
||||
{name: "no vif at all"},
|
||||
{name: "vif without ec config", writeVif: true},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
dataDir, idxDir := t.TempDir(), t.TempDir()
|
||||
idxBase := filepath.Join(idxDir, "2")
|
||||
if err := os.WriteFile(idxBase+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
if tc.writeVif {
|
||||
if err := volume_info.SaveVolumeInfo(idxBase+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
}
|
||||
uniformSidecar(t, idxBase, 12, 4, 3*1024*1024)
|
||||
|
||||
ev, err := NewEcVolume(types.HardDriveType, dataDir, idxDir, "", needle.VolumeId(2))
|
||||
if err != nil {
|
||||
t.Fatalf("mount: %v", err)
|
||||
}
|
||||
defer ev.Close()
|
||||
if got := ev.ECContext; got.DataShards != 12 || got.ParityShards != 4 || got.BlockSize != 3*1024*1024 {
|
||||
t.Errorf("layout = %d+%d block %d, want 12+4 block 3MiB", got.DataShards, got.ParityShards, got.BlockSize)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Absence stays legal — a volume encoded before either record exists is a
|
||||
// genuine legacy volume, and only that case may select the legacy layout.
|
||||
func TestNewEcVolumeConfigFreeVifWithNoSidecarUsesDefaults(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "3")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
if err := volume_info.SaveVolumeInfo(base+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
ev, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(3))
|
||||
if err != nil {
|
||||
t.Fatalf("mount: %v", err)
|
||||
}
|
||||
defer ev.Close()
|
||||
if got := ev.ECContext; got.DataShards != DataShardsCount || got.ParityShards != ParityShardsCount || got.BlockSize != 0 {
|
||||
t.Errorf("layout = %d+%d block %d, want the legacy defaults", got.DataShards, got.ParityShards, got.BlockSize)
|
||||
}
|
||||
}
|
||||
|
||||
// Present but unusable must fail the mount rather than default, in the
|
||||
// config-free-vif branch exactly as in the absent-vif one.
|
||||
func TestNewEcVolumeConfigFreeVifWithUnusableSidecarFails(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "4")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
if err := volume_info.SaveVolumeInfo(base+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
// Stamped for another generation: not a record of these shards.
|
||||
if err := SaveBitrotSidecar(BitrotSidecarPath(base, 0), &volume_server_pb.EcBitrotProtection{
|
||||
Algorithm: volume_server_pb.ChecksumAlgorithm_CHECKSUM_CRC32C,
|
||||
BlockSize: uint32(BitrotBlockSize),
|
||||
Generation: 5,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{DataShards: 12, ParityShards: 4},
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .ecsum: %v", err)
|
||||
}
|
||||
if _, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(4)); err == nil {
|
||||
t.Fatal("an unusable sole layout record must fail the mount")
|
||||
}
|
||||
}
|
||||
|
||||
// Startup mirroring gives each shard-bearing disk its own .ecx/.ecj/.vif but
|
||||
// deliberately not the .ecsum, and a repair delivers exactly one copy. A
|
||||
// runtime restricted to its own two directories therefore reports no
|
||||
// protection no matter how often it reloads — so the resolution has to reach
|
||||
// the sibling disk that actually received the manifest.
|
||||
func TestReloadBitrotSidecarFindsItOnASiblingDisk(t *testing.T) {
|
||||
mine, sibling := t.TempDir(), t.TempDir()
|
||||
myBase := filepath.Join(mine, "7")
|
||||
if err := os.WriteFile(myBase+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
if err := volume_info.SaveVolumeInfo(myBase+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: 10, ParityShards: 4, BlockSize: 0,
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
|
||||
ev, err := NewEcVolume(types.HardDriveType, mine, mine, "", needle.VolumeId(7))
|
||||
if err != nil {
|
||||
t.Fatalf("mount: %v", err)
|
||||
}
|
||||
defer ev.Close()
|
||||
if got := ev.bitrotStatus; got != BitrotOff {
|
||||
t.Fatalf("precondition: status = %v, want BitrotOff before the delivery", got)
|
||||
}
|
||||
|
||||
// The delivery lands the manifest on the sibling disk only.
|
||||
prot := &volume_server_pb.EcBitrotProtection{
|
||||
Algorithm: volume_server_pb.ChecksumAlgorithm_CHECKSUM_CRC32C,
|
||||
BlockSize: uint32(BitrotBlockSize),
|
||||
Generation: 0,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: 10, ParityShards: 4, BlockSize: 0,
|
||||
},
|
||||
}
|
||||
for i := 0; i < 14; i++ {
|
||||
prot.Shards = append(prot.Shards, &volume_server_pb.EcShardChecksums{
|
||||
ShardId: uint32(i), CoveredSize: 1, BlockCrc32C: make([]byte, 4),
|
||||
})
|
||||
}
|
||||
if err := SaveBitrotSidecar(BitrotSidecarPath(filepath.Join(sibling, "7"), 0), prot); err != nil {
|
||||
t.Fatalf("seed sibling .ecsum: %v", err)
|
||||
}
|
||||
|
||||
// Reloading against only its own directories still finds nothing.
|
||||
ev.ReloadBitrotSidecar()
|
||||
if got := ev.bitrotStatus; got != BitrotOff {
|
||||
t.Errorf("own-directories reload: status = %v, want BitrotOff", got)
|
||||
}
|
||||
|
||||
ev.ReloadBitrotSidecar(mine, sibling)
|
||||
if got := ev.bitrotStatus; got != BitrotOn {
|
||||
t.Errorf("cross-disk reload: status = %v, want BitrotOn", got)
|
||||
}
|
||||
}
|
||||
|
||||
// "Does this volume already have a manifest" cannot be answered from one base
|
||||
// name. The rebuild's opportunistic backfill asks it before writing a TOFU
|
||||
// baseline, and a false "no" there writes a second sidecar at the data base
|
||||
// that then shadows the real one — blessing whatever the shards currently say.
|
||||
func TestFindBitrotSidecarSearchesEveryCandidate(t *testing.T) {
|
||||
dataDir, idxDir, sibling := t.TempDir(), t.TempDir(), t.TempDir()
|
||||
dataBase := filepath.Join(dataDir, "8")
|
||||
idxBase := filepath.Join(idxDir, "8")
|
||||
siblingBase := filepath.Join(sibling, "8")
|
||||
|
||||
if got := FindBitrotSidecar(0, dataBase, idxBase, sibling); got != "" {
|
||||
t.Errorf("no sidecar anywhere should answer empty, got %q", got)
|
||||
}
|
||||
|
||||
// Each location on its own: any one of them is enough to answer "yes".
|
||||
for _, tc := range []struct{ name, base string }{
|
||||
{"data directory", dataBase},
|
||||
{"index directory", idxBase},
|
||||
{"sibling disk", siblingBase},
|
||||
} {
|
||||
want := BitrotSidecarPath(tc.base, 0)
|
||||
if err := os.WriteFile(want, []byte("x"), 0644); err != nil {
|
||||
t.Fatalf("%s: seed: %v", tc.name, err)
|
||||
}
|
||||
if got := FindBitrotSidecar(0, dataBase, idxBase, sibling); got != want {
|
||||
t.Errorf("%s: got %q, want %q", tc.name, got, want)
|
||||
}
|
||||
if err := os.Remove(want); err != nil {
|
||||
t.Fatalf("%s: cleanup: %v", tc.name, err)
|
||||
}
|
||||
}
|
||||
|
||||
// With copies everywhere the data base wins, which is the order every
|
||||
// resolver uses — so a stray copy there shadows the others.
|
||||
for _, base := range []string{siblingBase, idxBase, dataBase} {
|
||||
if err := os.WriteFile(BitrotSidecarPath(base, 0), []byte("x"), 0644); err != nil {
|
||||
t.Fatalf("seed %s: %v", base, err)
|
||||
}
|
||||
}
|
||||
if got, want := FindBitrotSidecar(0, dataBase, idxBase, sibling), BitrotSidecarPath(dataBase, 0); got != want {
|
||||
t.Errorf("precedence: got %q, want the data base %q", got, want)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,187 @@
|
||||
package erasure_coding
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/types"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/volume_info"
|
||||
)
|
||||
|
||||
// A present-but-malformed .vif must FAIL the mount: every new encode
|
||||
// records a positive uniform block size there, and silently defaulting
|
||||
// to the legacy layout would serve those shards with the wrong offset
|
||||
// math. Absence stays legal — legacy volumes predate the sidecar.
|
||||
func TestNewEcVolumeFailsOnMalformedVif(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "1")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(base+".vif", []byte("not json"), 0644); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
_, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(1))
|
||||
if err == nil {
|
||||
t.Fatal("mount over a malformed .vif must fail")
|
||||
}
|
||||
if !strings.Contains(err.Error(), ".vif") {
|
||||
t.Errorf("error should name the vif: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewEcVolumeAbsentVifMountsWithDefaults(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "1")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
ev, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(1))
|
||||
if err != nil {
|
||||
t.Fatalf("legacy mount without .vif must succeed: %v", err)
|
||||
}
|
||||
defer ev.Close()
|
||||
if ev.ECContext.BlockSize != 0 {
|
||||
t.Errorf("legacy mount must use the legacy layout: BlockSize=%d", ev.ECContext.BlockSize)
|
||||
}
|
||||
}
|
||||
|
||||
// A block size no encoder could have produced (negative, or not a whole
|
||||
// number of small blocks) maps every read to the wrong shard offset, so the
|
||||
// mount refuses it rather than serving those bytes.
|
||||
func TestNewEcVolumeRejectsImplausibleBlockSize(t *testing.T) {
|
||||
for _, blockSize := range []int64{1, -1, 3*1024*1024 + 1} {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "1")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
if err := volume_info.SaveVolumeInfo(base+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: 10, ParityShards: 4, BlockSize: blockSize,
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
if _, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(1)); err == nil {
|
||||
t.Errorf("block size %d must fail the mount", blockSize)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewEcVolumeAcceptsAlignedBlockSize(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "1")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
if err := volume_info.SaveVolumeInfo(base+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: 10, ParityShards: 4, BlockSize: 3 * 1024 * 1024,
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
ev, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(1))
|
||||
if err != nil {
|
||||
t.Fatalf("aligned block size must mount: %v", err)
|
||||
}
|
||||
defer ev.Close()
|
||||
if ev.ECContext.BlockSize != 3*1024*1024 {
|
||||
t.Errorf("BlockSize = %d, want 3MiB", ev.ECContext.BlockSize)
|
||||
}
|
||||
}
|
||||
|
||||
// The .vif and the .ecsum both record the layout their generation was encoded
|
||||
// with. When a generation-matching sidecar disagrees, one of them is wrong and
|
||||
// reads through the other land at the wrong shard offsets — so the mount must
|
||||
// refuse rather than quietly drop to unprotected reads.
|
||||
func TestNewEcVolumeRejectsSidecarGeometryDisagreement(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "1")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
if err := volume_info.SaveVolumeInfo(base+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: 10, ParityShards: 4, BlockSize: 0, // says legacy
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
// The sidecar for the same generation says uniform.
|
||||
prot := &volume_server_pb.EcBitrotProtection{
|
||||
Algorithm: volume_server_pb.ChecksumAlgorithm_CHECKSUM_CRC32C,
|
||||
BlockSize: uint32(BitrotBlockSize),
|
||||
Generation: 0,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: 10, ParityShards: 4, BlockSize: 3 * 1024 * 1024,
|
||||
},
|
||||
}
|
||||
if err := SaveBitrotSidecar(BitrotSidecarPath(base, 0), prot); err != nil {
|
||||
t.Fatalf("seed .ecsum: %v", err)
|
||||
}
|
||||
|
||||
_, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(1))
|
||||
if err == nil {
|
||||
t.Fatal("a generation-matching layout disagreement must fail the mount")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "block") {
|
||||
t.Errorf("the error should name the disagreement: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// With no .vif the bitrot sidecar is the ONLY record of the volume's layout.
|
||||
// Present but unusable — unreadable, stamped for another generation, or
|
||||
// carrying an incomplete ratio — must fail the mount: answering reads from a
|
||||
// guessed layout returns wrong bytes rather than none. Genuine absence stays
|
||||
// legal, and is covered by TestNewEcVolumeAbsentVifMountsWithDefaults.
|
||||
func TestNewEcVolumeRejectsUnusableSoleSidecar(t *testing.T) {
|
||||
seed := map[string]*volume_server_pb.EcBitrotProtection{
|
||||
"another generation": {
|
||||
Algorithm: volume_server_pb.ChecksumAlgorithm_CHECKSUM_CRC32C,
|
||||
BlockSize: uint32(BitrotBlockSize),
|
||||
Generation: 5,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{DataShards: 10, ParityShards: 4},
|
||||
},
|
||||
"no ec config": {
|
||||
Algorithm: volume_server_pb.ChecksumAlgorithm_CHECKSUM_CRC32C,
|
||||
BlockSize: uint32(BitrotBlockSize),
|
||||
Generation: 0,
|
||||
},
|
||||
"incomplete ratio": {
|
||||
Algorithm: volume_server_pb.ChecksumAlgorithm_CHECKSUM_CRC32C,
|
||||
BlockSize: uint32(BitrotBlockSize),
|
||||
Generation: 0,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{DataShards: 10},
|
||||
},
|
||||
"invalid block size": {
|
||||
Algorithm: volume_server_pb.ChecksumAlgorithm_CHECKSUM_CRC32C,
|
||||
BlockSize: uint32(BitrotBlockSize),
|
||||
Generation: 0,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: 10, ParityShards: 4, BlockSize: 3*1024*1024 + 1,
|
||||
},
|
||||
},
|
||||
}
|
||||
for name, prot := range seed {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "1")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("%s: seed .ecx: %v", name, err)
|
||||
}
|
||||
if err := SaveBitrotSidecar(BitrotSidecarPath(base, 0), prot); err != nil {
|
||||
t.Fatalf("%s: seed .ecsum: %v", name, err)
|
||||
}
|
||||
if _, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(1)); err == nil {
|
||||
t.Errorf("%s: an unusable sole layout record must fail the mount", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -14,6 +14,11 @@ import (
|
||||
type ServerShardInventory struct {
|
||||
Bits ShardBits
|
||||
QueryError error
|
||||
// BlockSize is the shard block layout this holder reports for the volume —
|
||||
// the one it will serve reads through. A binary predating the uniform
|
||||
// layout drops the field off the wire and reports 0, which is how
|
||||
// RequireAgreedBlockLayout tells such a holder from one that agrees.
|
||||
BlockSize int64
|
||||
}
|
||||
|
||||
// Query errors are recorded per-server and treated as zero shards rather
|
||||
@@ -50,6 +55,7 @@ func VerifyShardsAcrossServers(ctx context.Context, volumeID uint32,
|
||||
}
|
||||
inv.Bits = inv.Bits.Set(ShardId(s.ShardId))
|
||||
}
|
||||
inv.BlockSize = resp.GetEcShardConfig().GetBlockSize()
|
||||
return nil
|
||||
})
|
||||
if callErr != nil {
|
||||
@@ -99,6 +105,39 @@ func RequireRecoverableShardSet(volumeID uint32, shardsPresent ShardBits, dataSh
|
||||
volumeID, totalShards-len(missing), totalShards, dataShards, missing)
|
||||
}
|
||||
|
||||
// RequireAgreedBlockLayout gates source-volume deletion on every holder that
|
||||
// answered agreeing with the layout the shards were encoded under.
|
||||
//
|
||||
// The uniform block layout lives in a `.vif` field that older volume servers
|
||||
// have never heard of: such a server parses the file, silently discards the
|
||||
// unknown field, and mounts the volume on the legacy 1GiB/1MiB striping. Its
|
||||
// reads then land at the wrong shard offsets and return wrong bytes, and
|
||||
// nothing else in the encode path notices — the shard files are the same
|
||||
// length under either layout. Asking each holder which layout it is serving is
|
||||
// the one question that separates the two, and asking it here means the answer
|
||||
// arrives while the source .dat is still on disk.
|
||||
//
|
||||
// Holders that could not be reached, or that report no shard of this volume,
|
||||
// are skipped: RequireRecoverableShardSet already covers a missing holder, and
|
||||
// one that answered nothing has promised nothing.
|
||||
func RequireAgreedBlockLayout(volumeID uint32, encodedBlockSize int64, perServer map[string]ServerShardInventory) error {
|
||||
disagreeing := make([]string, 0, len(perServer))
|
||||
for server, inv := range perServer {
|
||||
if inv.QueryError != nil || inv.Bits.Count() == 0 {
|
||||
continue
|
||||
}
|
||||
if inv.BlockSize != encodedBlockSize {
|
||||
disagreeing = append(disagreeing, fmt.Sprintf("%s serves block size %d", server, inv.BlockSize))
|
||||
}
|
||||
}
|
||||
if len(disagreeing) == 0 {
|
||||
return nil
|
||||
}
|
||||
sort.Strings(disagreeing)
|
||||
return fmt.Errorf("volume %d was encoded with shard block size %d but %v; upgrade every volume server to a build that understands the uniform shard block layout before encoding",
|
||||
volumeID, encodedBlockSize, disagreeing)
|
||||
}
|
||||
|
||||
func SummarizeShardInventory(perServer map[string]ServerShardInventory) string {
|
||||
servers := make([]string, 0, len(perServer))
|
||||
for s := range perServer {
|
||||
|
||||
Reference in New Issue
Block a user