filer: keep sequential prefetch within prefetchCount chunks (#11674)

ChunkReadAt.ReadAt asks the ReaderCache to prefetch after every read,
and MaybeCache counted only the downloads it newly started toward the
window, so each call reached prefetchCount chunks further than the last.
A reader consuming a chunk in several reads, as the S3 GET path does
with a 256 KiB copy buffer over 4 MiB chunks, ran the read-ahead up to
the ReaderCache downloader limit and kept most of a large object in
memory, enough to get an S3 gateway OOM-killed mid-response.

Count every chunk in the window, downloading or not, so the read-ahead
stays at prefetchCount chunks past the reader.


Assisted-by: LLM

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
This commit is contained in:
scooby87 authored and GitHub committed 2026-10-12 17:41:01 +08:00
1 parent 603a1221a7
commit 73949daa98
2 files changed
+78 -3

No files matched your search

+4 -3
View File
@@ -92,8 +92,10 @@ func (rc *ReaderCache) MaybeCache(chunkViews *Interval[*ChunkView], count int) {
return
}
cached := 0
for x := chunkViews; x != nil && cached < count; x = x.Next {
// The window is the next count chunks, whether or not a download for them
// is already running: counting only newly started downloads would push the
// window count chunks further on every call.
for x, i := chunkViews, 0; x != nil && i < count; x, i = x.Next, i+1 {
chunkView := x.Value
if _, found := rc.downloaders[chunkView.FileId]; found {
continue
@@ -124,7 +126,6 @@ func (rc *ReaderCache) MaybeCache(chunkViews *Interval[*ChunkView], count int) {
rc.downloaders[chunkView.FileId] = cacher
go cacher.startCaching()
started = append(started, cacher)
cached++
}
return