filer: self-heal chunk manifest reads when volume locations go stale (#11107)

* filer: self-heal fetchWholeChunk on stale volume locations

Upstream #10156/#10800 wired cache invalidation into the buffer-based
read paths, but manifest resolution still goes through fetchWholeChunk,
which returns the raw error on failure. When cached volume locations
are stale (volume tiered to remote storage, server rolled), resolving
a large multipart file fails permanently even though other locations
are healthy.

Thread the ChunkGroup's cacheInvalidator through ResolveChunkManifest /
ResolveOneChunkManifest / fetchWholeChunk, and on failure invalidate,
re-lookup and retry once via the existing retryFetchWithFreshLocations
helper. The streaming bytesBuffer is reset before the retry so partial
bytes from the failed attempt cannot corrupt the manifest
proto.Unmarshal. Non-mount callers pass nil and keep their semantics.

* filer: move the manifest self-heal tests in with the other manifest tests

Also make the stale server stream a prefix and then abort mid-body, which is
what actually leaves partial bytes in the buffer: an HTTP error status returns
before ReadUrlAsStream ever calls the writer, so a 500 never exercised the
Reset the tests claimed to cover.

Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD

* filer: keep the cached volume locations when a manifest read is cancelled

A cancelled or timed-out read says nothing about where the volume lives, so
dropping the location and going back to the master only costs the next reader
a round trip. PrepareStreamContentWithThrottler already guards its self-heal
this way. The guard also goes inside retryFetchWithFreshLocations, since the
caller can be cancelled between its own check and the invalidation, and that
covers the reader cache and prefetch paths too.

fetchWholeChunk returns the context error rather than the stream failure it
provoked, and ResolveOneChunkManifest wraps with %w so errors.Is still sees it.
That matters even where no invalidator is passed: volume.fsck resolves
manifests with nil and tells its own abort from a corrupt manifest that way,
so the cancellation check sits ahead of the nil-invalidator return.

Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD

* filer: self-heal manifest reads on the filer and s3 paths too

Every caller that already holds the location cache backing its lookup function
can hand it over: the filer's read, copy and deletion paths and the log cache
have the MasterClient right there, and s3api has the FilerClient. MinusChunks
takes one for the same reason, since the deletion path resolves manifests
through it. Only the shell tools and the replication sinks, whose lookup
functions cache privately with nothing to invalidate, keep passing nil.

Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD

---------

Co-authored-by: bruce-zzz <bruce.zou@hhy-data.com>

weed/filer/stream.go

@@ -107,6 +107,12 @@ type VolumeServerJwtFunction func(fileId string) string
 // resolved locations actually changed (so we never retry against the same servers). originalErr
 // is returned unchanged when no retry is attempted, so callers surface the real fetch failure.
 func retryFetchWithFreshLocations(ctx context.Context, invalidator CacheInvalidator, lookupFn wdclient.LookupFileIdFunctionType, fileId string, oldUrls []string, originalErr error, refetch func(newUrls []string) error) error {
+	// the caller may have gone away between its own check and this one; a
+	// cancelled read is no evidence the locations are wrong, and callers such
+	// as volume.fsck tell an abort from real corruption with errors.Is
+	if ctxErr := ctx.Err(); ctxErr != nil {
+		return ctxErr
+	}
 	if invalidator == nil {
 		return originalErr
 	}