3 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	fe98520358	read: try a replica that stopped answering last, and relearn its volume's locations (#11130) ... * http: try a volume server that failed to answer last A cached location list is shuffled on every read, so once a replica dies half the reads keep dialing it first and pay a connect failure or timeout before the healthy replica answers. Remember, per host, when a request got no answer at all and order such hosts last for the next half minute. Once that passes, one read probes the host in its usual place while the others keep it last until the probe settles, so a black-holed server costs one stalled read per interval instead of one per read. Nothing is ever skipped: a host that failed is still tried when the others fail too. Any response, including an error status, counts as reachable. Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs * filer: refresh a chunk's locations after one of them fails A mount's location cache is only relearned when every cached location fails. When one replica dies and the other still answers, every read succeeds and the dead replica stays in the cache, and in the shuffled order it keeps being dialed first long after the master has dropped it. When a read fails on one location and a later one answers, call the refresh hook so the cached entry is dropped and looked up again. The read that already paid for the failure returns its data; the reads after it start from the locations the master knows now. Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs * http: claim the probe for every expired host, and try it first The claim was only checked for the first url, so with two replicas whose marks expired together the second was probed by every read at once. Claim each expired host on its own and put the reads that won a claim ahead of the reachable hosts, so a probe is always a real attempt and a lost claim always means the host is tried last. Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs * filer: refresh a chunk's locations in the streaming read path too The streaming loop had no refresh hook, so a manifest or streamed chunk that failed on one cached location and was served by another kept the stale entry until every location failed. Give it the same hook as the buffered loop, built by one refreshUrls function shared by the reader cache and the stream callers. Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs * http: probe at most one expired host per read Claiming every expired host in one ordering left all but the first claim without an attempt, since a read stops at its first answer, and a host that had come back waited another interval for nothing. Claim only the first expired host a read sees and leave the rest last and unclaimed, so each following read probes one of them. Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs * test: start the live server before releasing the dead server's port Closing the dead server first let the live server come up on the same port, in which case the dead location answers and the partial failure under test never happens. Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs	2026-09-03 11:51:48 -07:00
AlekseyandChris Lu	424cd164e9	s3: invalidate stale reader cache locations on chunk read failure (#10156) ... * s3: invalidate stale reader cache locations on chunk read failure * filer: share the chunk-read self-heal across reader cache and streaming paths The reader cache retry added a third copy of the invalidate-relookup-compare-retry dance already inlined in PrepareStreamContentWithThrottler and duplicated in retryWithCacheInvalidation. Extract retryFetchWithFreshLocations and route all three through it, parameterized by the refetch primitive. * filer: drop redundant completedTimeNew store in reader cache success path startCaching already stamps completedTimeNew unconditionally before the fetchErr branch; the second store inside the success branch is dead. * filer: make NewReaderCache cache invalidator an explicit parameter The variadic ...CacheInvalidator only ever read the first element, so a caller could pass two and silently get one. Take a single explicit argument and have the non-S3 callers pass nil. * filer: inject reader cache chunk fetch as a struct field Replace the process-global readerCacheFetchChunkData test seam with a per-instance fetchChunkDataFn field defaulted in NewReaderCache, matching how lookupFileIdFn is already wired. Tests set the field on the cache instead of swapping a shared global. * filer: log the location count, not full URLs, on self-heal retry --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-06-30 13:27:49 -07:00
Chris Lu	0798b274dd	feat(s3): add concurrent chunk prefetch for large file downloads (#8917) ... * feat(s3): add concurrent chunk prefetch for large file downloads Add a pipe-based prefetch pipeline that overlaps chunk fetching with response writing during S3 GetObject, SSE downloads, and filer proxy. While chunk N streams to the HTTP response, fetch goroutines for the next K chunks establish HTTP connections to volume servers ahead of time, eliminating the RTT gap between sequential chunk fetches. Uses io.Pipe for minimal memory overhead (~1MB per download regardless of chunk size, vs buffering entire chunks). Also increases the streaming read buffer from 64KB to 256KB to reduce syscall overhead. Benchmark results (64KB chunks, prefetch=4): - 0ms latency: 1058 → 2362 MB/s (2.2× faster) - 5ms latency: 11.0 → 41.7 MB/s (3.8× faster) - 10ms latency: 5.9 → 23.3 MB/s (4.0× faster) - 20ms latency: 3.1 → 12.1 MB/s (3.9× faster) * fix: address review feedback for prefetch pipeline - Fix data race: use *chunkPipeResult (pointer) on channel to avoid copying struct while fetch goroutines write to it. Confirmed clean with -race detector. - Remove concurrent map write: retryWithCacheInvalidation no longer updates fileId2Url map. Producer only reads it; consumer never writes. - Use mem.Allocate/mem.Free for copy buffer to reduce GC pressure. - Add local cancellable context so consumer errors (client disconnect) immediately stop the producer and all in-flight fetch goroutines. * fix(test): remove dead code and add Range header support in test server - Remove unused allData variable in makeChunksAndServer - Add Range header handling to createTestServer for partial chunk read coverage (206 Partial Content, 416 Range Not Satisfiable) * fix: correct retry condition and goroutine leak in prefetch pipeline - Fix retry condition: use result.fetchErr/result.written instead of copied to decide cache-invalidation retry. The old condition wrongly triggered retry when the fetch succeeded but the response writer failed on the first write (copied==0 despite fetcher having data). Now matches the sequential path (stream.go:197) which checks whether the fetcher itself wrote zero bytes. - Fix goroutine leak: when the producer's send to the results channel is interrupted by context cancellation, the fetch goroutine was already launched but the result was never sent to the channel. The drain loop couldn't handle it. Now waits on result.done before returning so every fetch goroutine is properly awaited.	2026-04-03 19:57:30 -07:00