weed/server: verify data shards against bitrot sidecar before ec decode (#12422) (#12427)

* weed/server: verify data shards against bitrot sidecar before ec decode (#12422)

* ec.decode: verify data shards against the mounted volume's bitrot protection

The decode looked up only the generation-0 .ecsum beside the data base and
each location's data directory, so it missed a sidecar kept with the index
and ignored the generation the volume actually serves.

Take the protection the volume's per-disk runtimes resolved instead (On, else
Invalid, else Off), and test a corrupt block past the first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: check for live entries before verifying the shards

A volume whose needles are all deleted gets its shards purged on the
no-live-entries status; a failing checksum must not keep it around.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: verify data shards against the bitrot protection before a decode

Mirrors the Go volume server: the decode takes the protection the volume's
per-disk runtimes resolved and refuses a data shard that fails its .ecsum
checksum, after the no-live-entries check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: a decode refuses a data shard that fails its .ecsum checksum

Runs against both volume server implementations: the refused decode keeps
every shard and the sidecar, and the needle stays readable through EC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: reload the bitrot sidecar before verifying the shards

A rebuild's backfill writes the sidecar without reloading the mounted
runtimes, which then still report no protection; resolve it again across
the EC metadata directories, as a shard mount does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: reload the bitrot sidecar before verifying the shards

Mirrors the Go volume server: the decode job resolves the sidecar again on
every runtime of the volume before choosing its protection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: a decode finds a sidecar written after the shards were mounted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read through the EC shards after the refused decode

The source volume stayed mounted, so the final read never touched the
shards it is meant to check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: refuse when a sidecar reload reports another layout

ReloadBitrotSidecar logged and dropped the layout-mismatch error, so the
runtime stayed Off and the decode skipped the data-shard check. It now
returns that error and the decode refuses on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a decode when a sidecar reload reports another layout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: a decode refuses a late sidecar that records another layout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: refuse when an unmount left no runtime to resolve the sidecar

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a decode when an unmount left no runtime to resolve the sidecar

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

weed/server/volume_grpc_erasure_coding.go

@@ -1302,6 +1302,25 @@ func (vs *VolumeServer) VolumeEcShardsToVolume(ctx context.Context, req *volume_
 		return nil, status.Errorf(codes.FailedPrecondition, "ec volume %d %s", req.VolumeId, erasure_coding.EcNoLiveEntriesSubstring)
 	}
 
+	// A rebuild's backfill writes the sidecar without reloading the mounted
+	// runtimes, so resolve it again the way a shard mount does. Any runtime
+	// left finds it on every disk; with none left (an unmount raced the
+	// decode) the protection would read as Off.
+	ecVolumes := vs.store.FindAllEcVolumes(needle.VolumeId(req.VolumeId))
+	if len(ecVolumes) == 0 {
+		return nil, fmt.Errorf("ec volume %d was unmounted during the decode", req.VolumeId)
+	}
+	ecMetadataDirs := vs.store.EcMetadataDirs()
+	for _, ecVolume := range ecVolumes {
+		if err := ecVolume.ReloadBitrotSidecar(ecMetadataDirs...); err != nil {
+			return nil, fmt.Errorf("ec volume %d: %w", req.VolumeId, err)
+		}
+	}
+	prot, bitrotStatus := erasure_coding.SelectBitrotProtection(ecVolumes)
+	if err := erasure_coding.VerifyEcVolumeDataShards(prot, bitrotStatus, shardFileNames); err != nil {
+		return nil, fmt.Errorf("ec volume %d: %w", req.VolumeId, err)
+	}
+
 	// calculate .dat file size. Pass shard 0's resolved path: on a multi-disk
 	// server the volume's shards can sit on several disks, so the .ec00 is not
 	// necessarily beside the EcVolume's own base path.