s3: invalidate stale reader cache locations on chunk read failure (#10156)

* s3: invalidate stale reader cache locations on chunk read failure

* filer: share the chunk-read self-heal across reader cache and streaming paths

The reader cache retry added a third copy of the invalidate-relookup-compare-retry
dance already inlined in PrepareStreamContentWithThrottler and duplicated in
retryWithCacheInvalidation. Extract retryFetchWithFreshLocations and route all
three through it, parameterized by the refetch primitive.

* filer: drop redundant completedTimeNew store in reader cache success path

startCaching already stamps completedTimeNew unconditionally before the
fetchErr branch; the second store inside the success branch is dead.

* filer: make NewReaderCache cache invalidator an explicit parameter

The variadic ...CacheInvalidator only ever read the first element, so a caller
could pass two and silently get one. Take a single explicit argument and have
the non-S3 callers pass nil.

* filer: inject reader cache chunk fetch as a struct field

Replace the process-global readerCacheFetchChunkData test seam with a
per-instance fetchChunkDataFn field defaulted in NewReaderCache, matching how
lookupFileIdFn is already wired. Tests set the field on the cache instead of
swapping a shared global.

* filer: log the location count, not full URLs, on self-heal retry

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>

weed/s3api/s3api_server.go

@@ -277,7 +277,7 @@ func NewS3ApiServerWithStore(router *mux.Router, option *S3ApiServerOption, expl
 	} else {
 		chunkCache = (*chunk_cache.TieredChunkCache)(nil)
 	}
-	readerCache := filer.NewReaderCache(s3ReaderCacheDownloaderLimit, chunkCache, filerClient.GetLookupFileIdFunction())
+	readerCache := filer.NewReaderCache(s3ReaderCacheDownloaderLimit, chunkCache, filerClient.GetLookupFileIdFunction(), filerClient)
 
 	s3ApiServer = &S3ApiServer{
 		option:                option,