* fix(mount): skip pressure-eviction of gappy page chunks (#9330)
A page chunk whose written-interval list has an internal hole was being
sealed under buffer-pressure eviction, then SaveContent would emit one
volume chunk per maximal adjacent run with no chunk covering the hole;
reads then silently zero-fill the gap (filer/stream.go:177-186). On a
sequential cp through FUSE, that bakes in-flight 4 KiB writes into split
volume chunks and leaves chunk-sized blocks of zeros on the destination.
Filter the pressure-driven sealers (SaveDataAt's over-limit path,
EvictOneWritableChunk, ProactiveFlush) to only seal chunks whose written
intervals form one unbroken run. The flush-on-close path (FlushAll) is
unchanged: at close every gap is by definition a sparse-file write that
the app legitimately never made.
* fix(mount): also gate IsContiguouslyWritten on leading zero-offset
Tighten IsContiguouslyWritten to also reject empty lists and lists whose
first interval does not start at offset 0. The internal-gap and
leading-gap cases are symmetric for pressure-driven sealing: both put
in-flight FUSE writeback for the missing range at risk of being baked
into split volume chunks. The flush-on-close path is still unfiltered
(sparse writes are sealed legitimately at FlushAll).
Also align EvictOneWritableChunk's bestBytes initialization with
SaveDataAt (start at 0) so an empty chunk is never picked, matching
the new semantic.
Addresses gemini-code-assist review on PR #9334.
* fix(mount): preserve cap-pressure liveness in EvictOneWritableChunk
The previous version of this fix had EvictOneWritableChunk return false
whenever every dirty chunk was gappy. That broke the accountant's
Reserve loop: cond.Wait only wakes on Release, Release only fires on
upload completion, and refusing to seal anything means no upload starts
— the writer hangs at the -writeBufferSizeMB cap forever.
Two-pass selection: prefer the fullest gap-free chunk (issue #9330: this
is what protects sequential cp from racing FUSE writeback), fall back to
the oldest non-empty writer when nothing is gap-free. Oldest-first
maximizes the chance that FUSE writeback for the gap range has already
settled. The actual sealing path is unchanged — SaveContent still emits
one volume chunk per maximal adjacent run; pages that arrive after the
seal land in a fresh MemChunk for the same logicChunkIndex and are
sealed in turn, so coverage is reconstructed at read time by
readResolvedChunks.
Sequential cp at default settings always hits the strict pass (writes
arrive contiguous-from-0 within their logicChunkIndex), so the bug-fix
behavior is preserved; the fallback only runs under genuinely sparse
workloads or under FUSE writeback so backed up that no chunk has
settled, where forced progress is preferable to a hung mount.
* test(mount): pin ProactiveFlush gap-skip behavior (#9330)
Sibling regression test for the ProactiveFlush guard added in this
series: same 3-chunk setup as TestEvictOneWritableChunk_SkipsGappyChunks
(internal gap, leading gap, contiguous). Verifies ProactiveFlush picks
the contiguous chunk when staleness criteria are otherwise satisfied,
returns false when only gappy chunks remain (no liveness fallback like
EvictOneWritableChunk has — failing here is just a missed optimization),
and that filling the holes lets the chunks auto-seal via maybeMoveToSealed.
* style(mount): trim verbose comments on #9330 fix
* feat(mount): proactive flush of idle writable chunks
Add a background goroutine that periodically scans writable chunks
across all open file handles and seals those that are idle and
unlikely to receive further writes, submitting them for async upload.
A writable chunk is proactively flushed when it has been idle for
500ms AND meets one of: nearly full (>=90%), behind the sequential
write frontier by 2+ chunks, or stale for 5+ seconds. Flushing only
happens when the upload pipeline has spare capacity (< half of
concurrent writer slots in use).
This prevents partial chunks from accumulating until fsync/close,
which is particularly beneficial for bursty or small-file workloads
where chunks may never reach IsComplete().
Also fixes a latent bug in ActivityScore where MarkRead/MarkWrite
used value receivers, silently discarding all mutations.
* refactor(mount): reuse WriterPattern instead of duplicating sequential detection
Remove the isSequential atomic from UploadPipeline. The proactive
flusher now reads IsSequentialMode() from the existing WriterPattern
on PageWriter and passes it as a parameter to ProactiveFlush. This
avoids duplicating the sequential/random detection that WriterPattern
already maintains.
* fix(mount): address PR review feedback
- Make ActivityScore thread-safe using atomics (CAS loop for score
updates, atomic load/swap for timestamp). Previously MarkRead was
called under RLock while MarkWrite held a write lock, creating a
data race on the shared fields.
- Fix ProactiveFlush half-capacity guard: use multiplication
(uploaderCount*2 >= max) instead of floor division (max/2) which
misbehaves for odd or small concurrentWriterMax values.
* fix(mount): review fixes for proactive flush
- Fix TOCTOU race in lastWriteChunkIndex update: use CAS loop so
concurrent writers cannot regress the frontier.
- Remove unused UploaderCount() getter.
- Reuse the caller-provided tsNs instead of calling time.Now() again
in WriteDataAt for lastWriteTsNs, eliminating a redundant syscall
per write.
* compare chunks by timestamp
* fix slab clearing error
* fix test compilation
* move oldest chunk to sealed, instead of by fullness
* lock on fh.entryViewCache
* remove verbose logs
* revert slat clearing
* less logs
* less logs
* track write and read by timestamp
* remove useless logic
* add entry lock on file handle release
* use mem chunk only, swap file chunk has problems
* comment out code that maybe used later
* add debug mode to compare data read and write
* more efficient readResolvedChunks with linked list
* small optimization
* fix test compilation
* minor fix on writer
* add SeparateGarbageChunks
* group chunks into sections
* turn off debug mode
* fix tests
* fix tests
* tmp enable swap file chunk
* Revert "tmp enable swap file chunk"
This reverts commit 985137ec47.
* simple refactoring
* simple refactoring
* do not re-use swap file chunk. Sealed chunks should not be re-used.
* comment out debugging facilities
* either mem chunk or swap file chunk is fine now
* remove orderedMutex as *semaphore.Weighted
not found impactful
* optimize size calculation for changing large files
* optimize performance to avoid going through the long list of chunks
* still problems with swap file chunk
* rename
* tiny optimization
* swap file chunk save only successfully read data
* fix
* enable both mem and swap file chunk
* resolve chunks with range
* rename
* fix chunk interval list
* also change file handle chunk group when adding chunks
* pick in-active chunk with time-decayed counter
* fix compilation
* avoid nil with empty fh.entry
* refactoring
* rename
* rename
* refactor visible intervals to *list.List
* refactor chunkViews to *list.List
* add IntervalList for generic interval list
* change visible interval to use IntervalList in generics
* cahnge chunkViews to *IntervalList[*ChunkView]
* use NewFileChunkSection to create
* rename variables
* refactor
* fix renaming leftover
* renaming
* renaming
* add insert interval
* interval list adds lock
* incrementally add chunks to readers
Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order
* fix tests compilation
* fix tests compilation
Chris Lu