ec.rebuild: repair shards that fail a checksum scrub in complete volumes (#12410)

* ec.rebuild: rebuild shards that fail a checksum scrub, also in complete volumes

ec.rebuild skipped every volume with all shards present, so a shard whose
bytes no longer match the .ecsum sidecar stayed in place until some other
shard went missing. With -checksum, ec.rebuild first scrubs each node's
shards in checksum mode and takes the failing copies out of the plan; a
shard left without a good copy counts as missing and is regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: remove the copies that failed the checksum once a good copy serves

A shard regenerated on the rebuilder left the failing copy mounted on its
old holder, so the volume kept a corrupt duplicate that a later dedup could
keep instead of the good one. Remove each failing copy once the rebuilt
shard is mounted, and right away when another holder already serves a copy
that passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: test the checksum scrub against fake volume servers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: check the lock before removing a corrupt copy

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: print the checksum scrub details

A malformed sidecar or a wholesale mismatch comes back only in the scrub
details, with no broken shard, so the volume would look verified.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: fail the run when a corrupt duplicate cannot be removed

For a volume whose only problem is a corrupt duplicate, the removal is the
whole repair, so its failure has to reach the caller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: test duplicate removal failures and a lost lock

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: fail the run when a node's checksum scrub fails

The node's copies stay in the plan unverified, so a corrupt shard there is
left in place; the rest of the cluster is still repaired.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: preview the corrupt-copy removals in dry-run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: test a failed checksum scrub

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

weed/shell/command_ec_rebuild.go

@@ -27,7 +27,7 @@ func (c *commandEcRebuild) Name() string {
 func (c *commandEcRebuild) Help() string {
 	return `find and rebuild missing ec shards among volume servers
 
-	ec.rebuild [-c EACH_COLLECTION|<collection_name>] [-apply] [-maxParallelization N] [-diskType=<disk_type>]
+	ec.rebuild [-c EACH_COLLECTION|<collection_name>] [-apply] [-maxParallelization N] [-diskType=<disk_type>] [-checksum]
 
 	Before rebuilding, asks volume servers to recover any shards left unmounted by
 	a missing .ecx index (the index resides only on a peer server). Such shards are
@@ -41,6 +41,11 @@ func (c *commandEcRebuild) Help() string {
 	                       Increase for faster rebuilds with more system resources.
 	                       Decrease if experiencing resource contention or instability.
 	  -diskType: disk type for EC shards (hdd, ssd, or empty for default hdd)
+	  -checksum: first verify every shard against its checksum sidecar (as
+	             ec.scrub -mode=checksum does), and rebuild the shards that fail
+	             it, also in volumes that have every shard. A failing copy is
+	             removed once a good copy of its shard is mounted elsewhere.
+	             This reads every shard in full.
 
 	Algorithm:
 
@@ -79,6 +84,7 @@ func (c *commandEcRebuild) Do(args []string, commandEnv *CommandEnv, writer io.W
 	maxParallelization := fixCommand.Int("maxParallelization", DefaultMaxParallelization, "run up to X tasks in parallel, whenever possible")
 	applyChanges := fixCommand.Bool("apply", false, "apply the changes")
 	diskTypeStr := fixCommand.String("diskType", "", "disk type for EC shards (hdd, ssd, or empty for default hdd)")
+	verifyChecksums := fixCommand.Bool("checksum", false, "verify every shard against its checksum sidecar first, and rebuild the shards that fail it, also in volumes that have every shard")
 	// TODO: remove this alias
 	applyChangesAlias := fixCommand.Bool("force", false, "apply the changes (alias for -apply)")
 	if err = fixCommand.Parse(args); err != nil {
@@ -119,5 +125,5 @@ func (c *commandEcRebuild) Do(args []string, commandEnv *CommandEnv, writer io.W
 		}
 	}
 
-	return ec.RebuildEcVolumes(commandEnv.ecEnv(), allEcNodes, writer, collections, volumeIds, diskType, *maxParallelization, *applyChanges)
+	return ec.RebuildEcVolumes(commandEnv.ecEnv(), allEcNodes, writer, collections, volumeIds, diskType, *maxParallelization, *applyChanges, *verifyChecksums)
 }