* fix(filer): leave reader cache unbounded without an explicit budget
NewReaderCache silently installed a 256MiB ReaderCacheBudget when the
caller passed none. Only weed mount opts into a budget; every other
caller (S3 gateway, WebDAV, query engine, mq logstore) inherited the
cap. Under ~90 concurrent S3 GETs of medium objects, prefetch wants
far more than 64 chunk buffers, so reserve() serialized chunk fetches,
clients timed out and retried, and the retry re-downloaded chunks the
cancelled request had already fetched.
A nil budget now means unbounded, restoring the pre-4.47 behavior for
callers that never asked for a memory cap; reserve/complete/release
are nil-safe. The mount path is unchanged and still enforces
-readerCacheSizeMB.
Fixes#11380
* feat(s3): expose -s3.readerCacheSizeMB reader buffer budget
Operators who want the S3 gateway read path memory-bounded can now
opt in: -s3.readerCacheSizeMB on weed filer/server/mini and
-readerCacheSizeMB on standalone weed s3, matching the mount flag.
The default 0 keeps the unbounded pre-4.47 behavior; a positive value
installs a shared ReaderCacheBudget across in-flight and retained
chunk buffers for all S3 GETs.
* fix(filer): validate chunk size before consulting the reader budget
A nil budget returned early and skipped the negative chunkSize check,
letting a corrupted size reach mem.Allocate and panic. Also drop the
command-specific flag prefix from the S3 validation error since
standalone weed s3 exposes the option as -readerCacheSizeMB.
* filer: drop chunk buffers once fully consumed
ReaderCache retained every completed chunk buffer in the downloaders
map until the slot limit evicted it, so buffers lingered after all
readers finished with them.
Track attached readers on each SingleChunkCacher and remove the cacher
when the last reader consumes the buffer to its end. In-flight download
deduplication and the prefetch handoff are unchanged: a buffer always
survives until fully read, partial reads keep it available, and an
attached reader pins a consumed buffer until it detaches. Repeat reads
now go through the chunk cache where enabled, or refetch.
* filer: drop consumed buffers on last detach, rechecked under cache lock
Two review findings on the drop-on-consume change:
- Removal only fired when the detaching reader itself reached the chunk
end. If the end-reaching reader finished first and the last remaining
reader did a partial read or cancelled, the consumed buffer and its
budget reservation lingered until eviction. Track a persistent
consumed flag instead, so any end-reaching read marks the buffer and
the last detach drops it.
- remove() checked only map identity, so a reader attaching between the
reader count hitting zero and removal could attach to a cacher that
was then deleted underneath it. removeConsumed() re-checks identity,
readers == 0, and consumed under the ReaderCache lock; a raced attach
keeps the cacher and its own detach retries the removal.
* cache resolved chunk manifests for Mount
* Address PR review: per-mount cache, singleflight, reuse ResolveOneChunkManifest
- Own the manifest cache per WFS mount instead of a process-global
variable, so manifests from one filer backend are never served to
another (Devin/CodeRabbit major bug).
- Coalesce concurrent cold misses via singleflight so only one fetch
runs during a cold burst (Greptile P2).
- Copy cached data after releasing the mutex so a large copy does not
block concurrent hits, inserts, and evictions (CodeRabbit nitpick).
- Reuse the existing ResolveOneChunkManifest function name instead of
introducing a new resolveOneChunkManifest wrapper.
- Validate (unmarshal) manifest bytes before caching so malformed
manifests do not poison the cache.
- Add TestChunkGroupManifestResolutionCoalescesColdMisses covering
the singleflight cold-miss path.
* Address round 2 review: coalesced-miss cancellation, test overlap
- Use singleflight.DoChan in fetchOrLoad and select on ctx.Done() so a
caller whose context is canceled while waiting for an in-flight fetch
returns ctx.Err() promptly instead of blocking for the leader's
result (Devin BUG).
- Add TestResolveOneChunkManifestCanceledWaiterReturnsDuringCoalescedMiss
covering the canceled-waiter path.
- Delay the cold-miss fixture response so the leader's fetch is still
in flight when concurrent opens join the singleflight, making the
one-fetch assertions reliable (CodeRabbit Minor).
* Address review: keep ResolveOneChunkManifest four-argument
Restore the exported ResolveOneChunkManifest to its original
four-argument signature so external callers keep compiling. Move the
cache-aware resolution into an unexported resolveOneChunkManifest
helper that accepts the per-mount ChunkManifestCache. The exported
function delegates to the helper with a nil cache, preserving the
historical uncached behavior for every non-Mount caller. The Mount
path (ChunkGroup.SetChunks) now calls the unexported helper with the
mount-owned cache. Tests and benchmarks that exercise the cache path
call the unexported helper directly.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* fix(filer): bound retained reader cache buffers by bytes
* test(filer): keep in-flight downloads during cache trimming
* feat(mem): expose pooled allocation capacity for byte reservations
* fix(mount): share a configurable reader buffer budget across files
* fix(filer): release failed prefetch slots and memory reservations
* feat(mount): expose a soft Go runtime memory limit
* docs(filer): restore shared-download rationale in startCaching
The one-line comment replacing the original context.Background() explanation was too thin for readChunkAt to cross-reference shared resource semantics. Restore a concise note on why request cancellation must not abort a download shared by concurrent readers.
* test(filer): loosen reader cache test deadlines to 5s
Three tests used 1-second deadlines that can flake on CI under load:
TestReaderCacheBudgetInFlight, TestReaderCacheEvictionDoesNotHoldCacheLock,
and TestReaderCacheFailedPrefetchReleasesBudget. Increase to 5 seconds.
* test(filer): cover re-read after reader cache eviction
Add TestReaderCacheReReadAfterEviction: reads chunk 'a', reads chunk 'b'
(evicting 'a' via budget pressure), then re-reads 'a' and asserts a
fresh download returns correct data. Verifies the core correctness
property that eviction never exposes missing or stale data to readers.
Chris Lu