Commit Graph
26 Commits
Author SHA1 Message Date
11af0e0c12 mount: move an open file's read view to the chunks a merge wrote (#12722)
* mount: move an open file's read view to the chunks a merge wrote

When a flush finds that a file's chunks take more than twice its size,
maybeMergeChunks reads the file back and writes it again as new chunks.
The commit that publishes them makes the filer delete the chunks they
replaced. The flush put the new chunks into the handle's entry but left
its read view on the old ones, so every descriptor that kept the file
open went on reading the replaced chunks. Those read back until their
volume is vacuumed. After that a read first retries the fetch of the
missing needle for about 13 s; then refreshEntryForRead installs the
filer's entry and the read is retried. The refresh skips a handle with
writes it has not flushed, and such a read returns EIO instead.

Files that are kept open and rewritten in place, such as PostgreSQL
data files and SQLite databases, are the ones whose chunks bloat, so
they are the ones that hit this.

installRewrittenChunks puts the merge's chunks into the entry and the
read view together, under the entry lock. Chunks attached while the
merge ran were not part of it and stay. Flushes without a merge are
unchanged.

TestMergeMovesReadViewOffReplacedChunks covers the handle.
TestReadOpenHandleAfterChunkMerge runs it on a real mount, with and
without an unflushed write; for it, the fuse_integration framework
starts mini on a free master port and exposes the address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drain uploads before a merge replaces the chunks

Uploads attach their chunks without the handle lock, first to the entry
and then to the read view, and writes that landed after doFlush drained
the pages may still be uploading when the flush takes its lock. An
upload that attached during the merge could keep a chunk older than
data the merge already holds, or put a chunk the merge replaced back
into the read view. Either outranks the merge's zero-timestamp chunk.

When the chunks are bloated enough to merge, the flush now drains the
pages again while its lock holds writes off, and takes the chunks again.
FlushAll returns only after every upload has attached to both the entry
and the read view, so nothing attaches during the merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave a pre-merge upload failure to the next flush

The drain before a merge returned an upload failure through the
metadata flush. doFlush then retried it as a filer error and answered
EIO, where the page flush answers a full disk with ENOSPC. The drain is
there only to wait for the uploads: a failure stays with the pages, and
the next flush reports it as a write error, as it would without a merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep chunks attached while a flush installs its compacted list

A write that lands after doFlush drained the pages can still be uploading
when flushMetadataToFiler takes its chunk snapshot. Its upload attaches the
chunk without the handle lock, and the plain flush then replaced the
entry's chunks with the list compacted from the snapshot, dropping it from
the handle and from every later commit. Keep chunks attached after the
snapshot, as the merge path already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fail a merge's flush on a failed drain

A failed upload found by the drain before a merge was only logged, and
the flush went on to publish a size covering bytes that no chunk stores.
It now fails the flush as a pageUploadError before anything is
published; doFlush maps it with writeErrorToFuseStatus, as it does its
own FlushData failure, and does not retry it as a filer error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: clone a flush's request entry under the entry lock

The request snapshot cloned the live entry after GetEntry had released
its read lock. Uploads attach chunks without the handle lock, so an
AppendChunks that landed during the clone raced it on the Chunks field.
Clone inside UpdateEntry, as content flush recovery does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave a merge to a later flush while uploads are pending

Before a merge the flush drained the pages while it held the handle and
publication locks. A writeback upload retries a transient storage error
until it succeeds, so that drain could hold both locks for as long as
storage stayed down, stalling reads, writes and stat on the handle.
The flush now checks for pending uploads and, if any, publishes the
compacted chunks and leaves the merge to a later flush. With none
pending the lock still holds writes off, so nothing attaches during the
merge, and a failed upload still fails the flush as a write error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep room for pending uploads when a flush merges onto a moved entry

A flush that leaves the chunk merge to a later flush publishes while
uploads are still pending, and writes that landed after the drain can be
uploading on any flush. Merging a peer's inline content or a shrink into
such a publication lets the upload attach its chunk next to the inline
bytes, where reads never reach it. Tell createContentEntry when uploads
are pending so the merge keeps room for them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test chunk merges through the filer's chunk proxy, also under load

The tests serve the filer's chunk proxy, and like the filer the fake
deletes the chunks each commit replaces, so a read view still naming
them fails:
- TestFlushMergeThroughChunkProxy/Quiet: the flush merges, and reads
  through the open handle return the file from the merged chunk alone.
- TestFlushMergeThroughChunkProxy/UploadInFlight: an older page is still
  uploading while a newer page of the same bytes has attached. The flush
  leaves the merge, reads return the newer page, and the next flush
  merges.
- TestFlushMergeRacesWithWritesAndReads: four writers, three flushers
  and a reader on one handle, with uploads delayed at random. The
  writers pause now and then, so 14 to 18 flushes a run merge and the
  others leave the merge to a later one. Every read must succeed within
  2 s, and at the end the handle and the filer must both hold exactly
  what was written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: let a held upload go when UploadInFlight fails

The subtest's proxy holds the older page's upload until the test
releases it, and only the passing path did. A failure in between left
the handler blocked, so the proxy's cleanup waited on it and the failure
surfaced as a suite timeout. Release it from a cleanup as well, once,
and bound the waits for the upload and the drains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 10:51:31 -08:00
4f70ad015e mount: fail the write when the swap file chunk cannot be created (#12655)
* mount: fail the write when the swap file chunk cannot be created

#7957 added a nil check after NewSwapFileChunk so a failed swap file
creation returns an error instead of panicking. The result was assigned
to pageChunk, a PageChunk interface, before the check, and an interface
holding a nil *SwapFileChunk is not nil, so the check never fired:
SaveDataAt stored the nil chunk and panicked in WriteDataAt.

Check the *SwapFileChunk before converting it. The new test points the
swap directory below a regular file; on master it panics in
SwapFileChunk.WriteDataAt.

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>

* mount: grow the file only after its write is buffered

Write set the new size, times, pending append offset and uncommitted
bytes before AddPage. When AddPage failed (now that a swap file chunk
that cannot be created returns an error), the write was reported as
failed but the larger size stayed on the handle's entry, so the next
flush published it with a hole where the data should be.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report a partly buffered write as a short write

PageWriter.AddPage fills one chunk at a time, so when a later chunk
cannot be created the earlier chunks already hold their bytes. Return
how much was buffered, and let Write grow the size and times over that
prefix and report it, instead of failing the whole write while the
prefix still reaches the next flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report the bytes a failed copy_file_range already buffered

A streaming copy whose page fails partway returned an error even though
earlier pages, or the start of the failing one, were already buffered and
would reach the next flush with the destination size left short. Count
the buffered bytes, grow the destination over them, and return them as a
short copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: free the copy test's destination pages

The buffered memory chunk counts toward the process-wide memory chunk
ceiling, so repeated runs of the test pushed later ones onto the
unusable swap directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: count only the bytes a swap chunk stored

A swap chunk whose file write fails stores a prefix or nothing, but
SaveDataAt returned no error and AddPage counted the whole span as
buffered. SaveDataAt now reports the short write, and AddPage passes the
stored count up so a write is never acknowledged past it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fail a partly buffered writeback write

Kernel writeback ignores the count in a WRITE reply, so a short write
there drops the unbuffered rest without an error. A write flagged as
page-cache writeback now keeps its buffered prefix but returns the
error, which the kernel reports on fsync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 19:34:43 -08:00
Chris Lu ae4839e005 mount: keep a sealed chunk alive until its own upload finishes (#10504)
Sealing a logic chunk index that already held a sealed chunk dropped the
old chunk's only reference and freed its page chunk. That chunk's upload
may not have started reading it yet — Execute() returns as soon as the
job is handed to a goroutine — so mem.Free could hand a live 2 MiB mem
chunk back to the slot pool, the next NewMemChunk would overwrite it,
and the in-flight upload shipped whatever bytes were there. Under fio
randwrite the volume server rejected those needles with "Content-MD5 did
not match md5 of file data" and the FUSE write failed with EIO.

Give the sealed chunk a second reference for its upload, dropped only by
the upload itself, and let the upload unindex itself only while it still
owns the index — the unconditional delete could evict a newer sealed
chunk and hide its dirty pages from readers.
2026-07-31 01:16:02 -07:00
Chris Lu e96190d128 fix(mount): skip pressure-eviction of gappy page chunks (#9330) (#9334)
* fix(mount): skip pressure-eviction of gappy page chunks (#9330)

A page chunk whose written-interval list has an internal hole was being
sealed under buffer-pressure eviction, then SaveContent would emit one
volume chunk per maximal adjacent run with no chunk covering the hole;
reads then silently zero-fill the gap (filer/stream.go:177-186). On a
sequential cp through FUSE, that bakes in-flight 4 KiB writes into split
volume chunks and leaves chunk-sized blocks of zeros on the destination.

Filter the pressure-driven sealers (SaveDataAt's over-limit path,
EvictOneWritableChunk, ProactiveFlush) to only seal chunks whose written
intervals form one unbroken run. The flush-on-close path (FlushAll) is
unchanged: at close every gap is by definition a sparse-file write that
the app legitimately never made.

* fix(mount): also gate IsContiguouslyWritten on leading zero-offset

Tighten IsContiguouslyWritten to also reject empty lists and lists whose
first interval does not start at offset 0. The internal-gap and
leading-gap cases are symmetric for pressure-driven sealing: both put
in-flight FUSE writeback for the missing range at risk of being baked
into split volume chunks. The flush-on-close path is still unfiltered
(sparse writes are sealed legitimately at FlushAll).

Also align EvictOneWritableChunk's bestBytes initialization with
SaveDataAt (start at 0) so an empty chunk is never picked, matching
the new semantic.

Addresses gemini-code-assist review on PR #9334.

* fix(mount): preserve cap-pressure liveness in EvictOneWritableChunk

The previous version of this fix had EvictOneWritableChunk return false
whenever every dirty chunk was gappy. That broke the accountant's
Reserve loop: cond.Wait only wakes on Release, Release only fires on
upload completion, and refusing to seal anything means no upload starts
— the writer hangs at the -writeBufferSizeMB cap forever.

Two-pass selection: prefer the fullest gap-free chunk (issue #9330: this
is what protects sequential cp from racing FUSE writeback), fall back to
the oldest non-empty writer when nothing is gap-free. Oldest-first
maximizes the chance that FUSE writeback for the gap range has already
settled. The actual sealing path is unchanged — SaveContent still emits
one volume chunk per maximal adjacent run; pages that arrive after the
seal land in a fresh MemChunk for the same logicChunkIndex and are
sealed in turn, so coverage is reconstructed at read time by
readResolvedChunks.

Sequential cp at default settings always hits the strict pass (writes
arrive contiguous-from-0 within their logicChunkIndex), so the bug-fix
behavior is preserved; the fallback only runs under genuinely sparse
workloads or under FUSE writeback so backed up that no chunk has
settled, where forced progress is preferable to a hung mount.

* test(mount): pin ProactiveFlush gap-skip behavior (#9330)

Sibling regression test for the ProactiveFlush guard added in this
series: same 3-chunk setup as TestEvictOneWritableChunk_SkipsGappyChunks
(internal gap, leading gap, contiguous). Verifies ProactiveFlush picks
the contiguous chunk when staleness criteria are otherwise satisfied,
returns false when only gappy chunks remain (no liveness fallback like
EvictOneWritableChunk has — failing here is just a missed optimization),
and that filling the holes lets the chunks auto-seal via maybeMoveToSealed.

* style(mount): trim verbose comments on #9330 fix
2026-05-06 15:26:56 -07:00
Chris Lu d7865909ba feat(mount): proactive flush of idle writable chunks (#9094)
* feat(mount): proactive flush of idle writable chunks

Add a background goroutine that periodically scans writable chunks
across all open file handles and seals those that are idle and
unlikely to receive further writes, submitting them for async upload.

A writable chunk is proactively flushed when it has been idle for
500ms AND meets one of: nearly full (>=90%), behind the sequential
write frontier by 2+ chunks, or stale for 5+ seconds. Flushing only
happens when the upload pipeline has spare capacity (< half of
concurrent writer slots in use).

This prevents partial chunks from accumulating until fsync/close,
which is particularly beneficial for bursty or small-file workloads
where chunks may never reach IsComplete().

Also fixes a latent bug in ActivityScore where MarkRead/MarkWrite
used value receivers, silently discarding all mutations.

* refactor(mount): reuse WriterPattern instead of duplicating sequential detection

Remove the isSequential atomic from UploadPipeline. The proactive
flusher now reads IsSequentialMode() from the existing WriterPattern
on PageWriter and passes it as a parameter to ProactiveFlush. This
avoids duplicating the sequential/random detection that WriterPattern
already maintains.

* fix(mount): address PR review feedback

- Make ActivityScore thread-safe using atomics (CAS loop for score
  updates, atomic load/swap for timestamp). Previously MarkRead was
  called under RLock while MarkWrite held a write lock, creating a
  data race on the shared fields.

- Fix ProactiveFlush half-capacity guard: use multiplication
  (uploaderCount*2 >= max) instead of floor division (max/2) which
  misbehaves for odd or small concurrentWriterMax values.

* fix(mount): review fixes for proactive flush

- Fix TOCTOU race in lastWriteChunkIndex update: use CAS loop so
  concurrent writers cannot regress the frontier.
- Remove unused UploaderCount() getter.
- Reuse the caller-provided tsNs instead of calling time.Now() again
  in WriteDataAt for lastWriteTsNs, eliminating a redundant syscall
  per write.
2026-04-16 00:44:24 -07:00
Chris Lu bdebd63f7c fix(mount): evict writable chunks when writeBufferSizeMB cap is reached (#9091)
* fix(mount): evict writable chunks when writeBufferSizeMB cap is reached

Reported on #8777: fio 4k randwrite with --nrfiles=1000 on a weed mount
hangs indefinitely after ~1 second of activity, volume server QPS
drops to zero, and the test never makes progress beyond a few hundred
writes. A scaled-down local repro (4 jobs x 250 nrfiles x 40 MiB with
-chunkSizeLimitMB=32 -writeBufferSizeMB=10240) reproduced the hang
exactly: fio stuck in fio_state=SETTING_UP, mount idle, all writer
goroutines parked in page_writer.(*WriteBufferAccountant).Reserve
with n=0x2000000 (32 MiB).

Root cause: UploadPipeline.SaveDataAt reserves a full chunkSize slot
against the global WriteBufferAccountant the first time it allocates
a writable chunk for a given file. The reservation is released by
SealedChunk.FreeReference only after the chunk's async upload
completes, and chunks only move from writable to sealed when they
fill or when the file is flushed/closed. For 4k random writes with
small per-file data and iodepth holding files open, writable chunks
never fill and never close, so their slots are pinned forever. Once
openFiles * chunkSize exceeds writeBufferSizeMB, every new writer
blocks in Reserve's cond.Wait with no possible waker, a hard deadlock.
The user-visible symptoms (volume QPS=0, waited 30min, still running)
are exactly that state. In the reported config, 8 jobs * 1000 nrfiles
* 32 MiB = 256 GB of reservations against a 10 GB cap.

Fix: add a SetEvictor hook on WriteBufferAccountant. When Reserve
would otherwise block on the cond, it single-flights an evictor
callback that walks wfs.fhMap, picks the first open file handle with
a writable chunk, and force-seals its fullest writable chunk via a
new UploadPipeline.EvictOneWritableChunk. Force-sealing submits the
chunk for async upload on the existing uploader pool; the upload's
SealedChunk.FreeReference releases the global slot, broadcasts on
the accountant cond, and unblocks the waiter. The fullest-first
heuristic matches the existing over-limit path in SaveDataAt and
keeps us from thrashing on repeatedly re-sealing half-empty chunks.

The original #9066 motivation (cap global write-pipeline growth so
swap can't be filled while uploads stall) is preserved: Reserve
still blocks when the cap is actually exhausted by in-flight
uploads, and a failed evictor (no writable chunks to seal) falls
through to cond.Wait exactly as before.

Verified end-to-end in a privileged debian container running a
full master + volume + filer + weed mount stack. Against the
reporter's config shrunk to the same memory footprint (4 jobs x
250 nrfiles x 40 MiB, -chunkSizeLimitMB=32 -writeBufferSizeMB=10240):
  - baseline binary (master): fio stalls after ~320 writes, killed
    by timeout at 400s with io=1280 KiB and bw=2383 B/s.
  - patched binary: fio completes in 1.9 s at 82.8 MiB/s, all
    40,000 writes issued, 156 MiB written, rc=0.
page_writer unit tests still pass.

Touches: WriteBufferAccountant (SetEvictor + single-flight in
Reserve), UploadPipeline.EvictOneWritableChunk, DirtyPages
interface + the two existing implementers (ChunkedDirtyPages,
PageWriter), and a new weedfs_write_buffer_evict.go holding the
WFS.evictOneWritableChunk callback that walks fhMap.

* fix(mount): re-check write-budget cap after evict and make eviction panic-safe

Addresses two review concerns on #9091:

1. CodeRabbit flagged that the original Reserve loop only considered the
   break condition when the evictor reported success:

     if evicted {
         if a.used+n <= a.cap || a.used == 0 {
             break
         }
     }
     a.cond.Wait()

   A concurrent Release firing during the evictor's execution window (we
   drop a.mu around the callback so uploader goroutines can drain) would
   land its Broadcast on an empty waiter set and then be missed, because
   we unconditionally call cond.Wait() on the same iteration. In practice
   an in-flight async upload eventually fires another broadcast so this
   would delay rather than permanently hang — but the logic is wrong
   either way and should re-check the cap after every evictor round.

2. Gemini flagged that setting `a.evicting = true` and dropping `a.mu`
   without a deferred unwind leaves the accountant in a broken state if
   the evictor panics: the flag stays set forever and no Reserve caller
   can ever run another eviction, even though the mutex gets unlocked by
   the normal stack unwind.

Pull the evictor call into a tiny helper runEvictorLocked whose defer
re-acquires a.mu and clears a.evicting regardless of panic status, then
broadcasts so other blocked Reservers re-evaluate. Reserve checks the
cap condition immediately after the helper returns; only falls through
to cond.Wait() when the budget is still exhausted. No behavior change on
the happy path; the fix is in the error and race corners.

* doc(mount): clarify eviction-scan cost on the blocked-Reserve path

Gemini review on #9091 flagged that the fhMap scan inside
evictOneWritableChunk is O(open files). Document why the cost is
bounded to the Reserve-blocked path and paid at most once per
chunkSize drained, rather than on the write hot path, and record the
reason we do not preempt with an LRU.
2026-04-15 14:06:06 -07:00
Chris Lu 7a7f220224 feat(mount): cap write buffer with -writeBufferSizeMB (#9066)
* feat(mount): cap write buffer with -writeBufferSizeMB

Without a bound on the per-mount write pipeline, sustained upload
failures (e.g. volume server returning "Volume Size Exceeded" while
the master hasn't yet rotated assignments) let sealed chunks pile up
across open file handles until the swap directory — by default
os.TempDir() — fills the disk. Reported on 4.19 filling /tmp to 1.8 TB
during a large rclone sync.

Add a global WriteBufferAccountant shared across every UploadPipeline
in a mount. Creating a new page chunk (memory or swap) first reserves
ChunkSize bytes; when the cap is reached the writer blocks until an
uploader finishes and releases, turning swap overflow into natural
FUSE-level backpressure instead of unbounded disk growth.

The new -writeBufferSizeMB flag (also accepted via fuse.conf) defaults
to 0 = unlimited, preserving current behavior. Reserve drops
chunksLock while blocking so uploader goroutines — which take
chunksLock on completion before calling Release — cannot deadlock,
and an oversized reservation on an empty accountant succeeds to avoid
single-handle starvation.

* fix(mount): plug write-budget leaks in pipeline Shutdown

Review on #9066 caught two accounting bugs on the Destroy() path:

1. Writable-chunk leak (high). SaveDataAt() reserves ChunkSize before
   inserting into writableChunks, but Shutdown() only iterated
   sealedChunks. Truncate / metadata-invalidation flows call Destroy()
   (via ResetDirtyPages) without flushing first, so any dirty but
   unsealed chunks would permanently shrink the global write budget.
   Shutdown now frees and releases writable chunks too.

2. Double release with racing uploader (medium). Shutdown called
   accountant.Release directly after FreeReference, while the async
   uploader goroutine did the same on normal completion — under a
   Destroy-before-flush race this could underflow the accountant and
   let later writes exceed the configured cap. Move accounting into
   SealedChunk.FreeReference itself: the refcount-zero transition is
   exactly-once by construction, so any number of FreeReference calls
   release the slot precisely once.

Add regression tests for the writable-leak and the FreeReference
idempotency guarantee.

* test(mount): remove sleep-based race in accountant blocking test

Address review nits on #9066:
- Replace time.Sleep(50ms) proxy for "goroutine entered Reserve" with
  a started channel the goroutine closes immediately before calling
  Reserve. Reserve cannot make progress until Release is called, so
  landed is guaranteed false after the handshake — no arbitrary wait.
- Short-circuit WriteBufferAccountant.Used() in unlimited mode for
  consistency with Reserve/Release, avoiding a mutex round-trip.

* test(mount): add end-to-end write-buffer cap integration test

Exercises the full write-budget plumbing with a small cap (4 chunks of
64 KiB = 256 KiB) shared across three UploadPipelines fed by six
concurrent writers. A gated saveFn models the "volume server rejecting
uploads" condition from the original report: no sealed chunk can drain
until the test opens the gate. A background sampler records the peak
value of accountant.Used() throughout the run.

The test asserts:
  - writers fill the budget and then block on Reserve (Used() stays at
    the cap while stalled)
  - Used() never exceeds the configured cap even under concurrent
    pressure from multiple pipelines
  - after the gate opens, writers drain to zero
  - peak observed Used() matches the cap (262144 bytes in this run)

While wiring this up, the race detector surfaced a pre-existing data
race on UploadPipeline.uploaderCount: the two glog.V(4) lines around
the atomic Add sites read the field non-atomically. Capture the new
value from AddInt32 and log that instead — one-liner each, no
behavioral change.

* test(fuse): end-to-end integration test for -writeBufferSizeMB

Exercise the new write-buffer cap against a real weed mount so CI
(fuse-integration.yml) covers the FUSE→upload-pipeline→filer path, not
just the in-package unit tests. Uses a 4 MiB cap with 2 MiB chunks so
every subtest's total write demand is multiples of the budget and
Reserve/Release must drive forward progress for writes to complete.

Subtests:
- ConcurrentLargeWrites: six parallel 6 MiB files (36 MiB total, ~18
  chunk allocations) through the same mount, verifies every byte
  round-trips.
- SingleFileExceedingCap: one 20 MiB file (10 chunks) through a single
  handle, catching any self-deadlock when the pipeline's own earlier
  chunks already fill the global budget.
- DoesNotDeadlockAfterPressure: final small write with a 30s timeout,
  catching budget-slot leaks that would otherwise hang subsequent
  writes on a still-full accountant.

Ran locally on Darwin with macfuse against a real weed mini + mount:
  === RUN   TestWriteBufferCap
  --- PASS: TestWriteBufferCap (1.82s)

* test(fuse): loosen write-buffer cap e2e test + fail-fast on hang

On Linux CI the previous configuration (-writeBufferSizeMB=4,
-concurrentWriters=4 against a 20 MiB single-handle write)
deterministically hung the "Run FUSE Integration Tests" step to the
45-minute workflow timeout, while on macOS / macfuse the same test
completes in ~2 seconds (see run 24386197483). The Linux hang shows
up after TestWriteBufferCap/ConcurrentLargeWrites completes cleanly,
then TestWriteBufferCap/SingleFileExceedingCap starts and never
emits its PASS line.

Change:
- Loosen the cap to 16 MiB (8 × 2 MiB chunk slots) and drop the
  custom -concurrentWriters override. The subtests still drive demand
  well above the cap (32 MiB concurrent, 12 MiB single-handle), so
  Reserve/Release is still on every chunk-allocation path; the cap
  just gives the pipeline enough headroom that interactions with the
  per-file writableChunkLimit and the go-fuse MaxWrite batching don't
  wedge a single-handle writer on a slow runner.
- Wrap every os.WriteFile in a writeWithTimeout helper that dumps every
  live goroutine on timeout. If this ever re-regresses, CI surfaces
  the actual stuck goroutines instead of a 45-minute walltime.
- Also guard the concurrent-writer goroutines with the same timeout +
  stack dump.

The in-package unit test TestWriteBufferCap_SharedAcrossPipelines
remains the deterministic, controlled verification of the blocking
Reserve/Release path — this e2e test is now a smoke test for
correctness and absence of deadlocks through a real FUSE mount, which
is all it should be.

* fix: address PR #9066 review — idempotent FreeReference, subtest watchdog, larger single-handle test

FreeReference on SealedChunk now early-returns when referenceCounter is
already <= 0. The existing == 0 body guard already made side effects
idempotent, but the counter itself would still decrement into the
negatives on a double-call — ugly and a latent landmine for any future
caller that does math on the counter. Make double-call a strict no-op.

test(fuse): per-subtest watchdog + larger single-handle test

- Add runSubtestWithWatchdog and wrap every TestWriteBufferCap subtest
  with a 3-minute deadline. Individual writes were already
  timeout-wrapped but the readback loops and surrounding bookkeeping
  were not, leaving a gap where a subtest body could still hang. On
  watchdog fire, every live goroutine is dumped so CI surfaces the
  wedge instead of a 45-minute walltime.

- Bump testLargeFileUnderCap from 12 MiB → 20 MiB (10 chunks) to
  exceed the 16 MiB cap (8 slots) again and actually exercise
  Reserve/Release backpressure on a single file handle. The earlier
  e2e hang was under much tighter params (-writeBufferSizeMB=4,
  -concurrentWriters=4, writable limit 4); with the current loosened
  config the pressure is gentle and the goroutine-dump-on-timeout
  safety net is in place if it ever regresses.

Declined: adding an observable peak-Used() assertion to the e2e test.
The mount runs as a subprocess so its in-process WriteBufferAccountant
state isn't reachable from the test without adding a metrics/RPC
surface. The deterministic peak-vs-cap verification already lives in
the in-package unit test TestWriteBufferCap_SharedAcrossPipelines.
Recorded this rationale inline in TestWriteBufferCap's doc comment.

* test(fuse): capture mount pprof goroutine dump on write-timeout

The previous run (24388549058) hung on LargeFileUnderCap and the
test-side dumpAllGoroutines only showed the test process — the test's
syscall.Write is blocked in the kernel waiting for FUSE to respond,
which tells us nothing about where the MOUNT is stuck. The mount runs
as a subprocess so its in-process stacks aren't reachable from the
test.

Enable the mount's pprof endpoint via -debug=true -debug.port=<free>,
allocate the port from the test, and on write-timeout fetch
/debug/pprof/goroutine?debug=2 from the mount process and log it. This
gives CI the only view that can actually diagnose a write-buffer
backpressure deadlock (writer goroutines blocked on Reserve, uploader
goroutines stalled on something, etc).

Kept fileSize at 20 MiB so the Linux CI run will still hit the hang
(if it's genuinely there) and produce an actionable mount-side dump;
the alternative — silently shrinking the test below the cap — would
lose the regression signal entirely.

* review: constructor-inject accountant + subtest watchdog body on main

Two PR-#9066 review fixes:

1. NewUploadPipeline now takes the WriteBufferAccountant as a
   constructor parameter; SetWriteBufferAccountant is removed. In
   practice the previous setter was only called once during
   newMemoryChunkPages, before any goroutine could touch the
   pipeline, so there was no actual race — but constructor injection
   makes the "accountant is fixed at construction time" invariant
   explicit and eliminates the possibility of a future caller
   mutating it mid-flight. All three call sites (real + two tests)
   updated; the legacy TestUploadPipeline passes a nil accountant,
   preserving backward-compatible unlimited-mode behavior.

2. runSubtestWithWatchdog now runs body on the subtest main goroutine
   and starts a watcher goroutine that only calls goroutine-safe t
   methods (t.Log, t.Logf, t.Errorf). The previous version ran body
   on a spawned goroutine, which meant any require.* or writeWithTimeout
   t.Fatalf inside body was being called from a non-test goroutine —
   explicitly disallowed by Go's testing docs. The watcher no longer
   interrupts body (it can't), so body must return on its own —
   which it does via writeWithTimeout's internal 90s timeout firing
   t.Fatalf on (now) the main goroutine. The watchdog still provides
   the critical diagnostic: on timeout it dumps both test-side and
   mount-side (via pprof) goroutine stacks and marks the test failed
   via t.Errorf.

* fix(mount): IsComplete must detect coverage across adjacent intervals

Linux FUSE caps per-op writes at FUSE_MAX_PAGES_PER_REQ (typically
1 MiB on x86_64) regardless of go-fuse's requested MaxWrite, so a
2 MiB chunk filled by a sequential writer arrives as two adjacent
1 MiB write ops. addInterval in ChunkWrittenIntervalList does not
merge adjacent intervals, so the resulting list has two elements
{[0,1M], [1M,2M]} — fully covered, but list.size()==2.

IsComplete previously returned `list.size() == 1 &&
list.head.next.isComplete(chunkSize)`, which required a single
interval covering [0, chunkSize). Under that rule, chunks filled by
adjacent writes never reach IsComplete==true, so maybeMoveToSealed
never fires, and the chunks sit in writableChunks until
FlushAll/close. SaveContent handles the adjacency correctly via its
inline merge loop, so uploads work once they're triggered — but
IsComplete is the gate that triggers them.

This was a latent bug: without the write-buffer cap, the overflow
path kicks in at writableChunkLimit (default 128) and force-seals
chunks, hiding the leak. #9066's -writeBufferSizeMB adds a tighter
global cap, and with 8 slots / 20 MiB test, the budget trips long
before overflow. The writer blocks in Reserve, waiting for a slot
that never frees because no uploader ever ran — observed in the CI
run 24390596623 mount pprof dump: goroutine 1 stuck in
WriteBufferAccountant.Reserve → cond.Wait, zero uploader goroutines
anywhere in the 89-goroutine dump.

Walk the (sorted) interval list tracking the furthest covered
offset; return true if coverage reaches chunkSize with no gaps. This
correctly handles adjacent intervals, overlapping intervals, and
out-of-order inserts. Added TestIsComplete_AdjacentIntervals
covering single-write, two adjacent halves (both orderings), eight
adjacent eighths, gaps, missing edges, and overlaps.

* test(fuse): route mount glog to stderr + dump mount on any write error

Run 24392087737 (with the IsComplete fix) no longer hangs on Linux —
huge progress. Now TestWriteBufferCap/LargeFileUnderCap fails with
'close(...write_buffer_cap_large.bin): input/output error', meaning
a chunk upload failed and pages.lastErr propagated via FlushData to
close(). But the mount log in the CI artifact is empty because weed
mount's glog defaults to /tmp/weed.* files, which the CI upload step
never sees, so we can't tell WHICH upload failed or WHY.

Add -logtostderr=true -v=2 to MountOptions so glog output goes to
the mount process's stderr, which the framework's startProcess
redirects into f.logDir/mount.log, which the framework's DumpLogs
then prints to the test output on failure. The -v=2 floor enables
saveDataAsChunk upload errors (currently logged at V(0)) plus the
medium-level write_pipeline/upload traces without drowning the log
in V(4) noise.

Also dump MOUNT goroutines on any writeWithTimeout error (not just
timeout). The IsComplete fix means we now get explicit errors
instead of silent hangs, and the goroutine dump at the error moment
shows in-flight upload state (pending sealed chunks, retry loops,
etc) that a post-failure log alone can't capture.
2026-04-14 07:47:35 -07:00
c909724bf1 Fix: prevent panic when swap file creation fails (#7957)
* Fix: prevent panic when swap file creation fails

* weed mount: fix race condition in swap file initialization

Ensure thread-safe access to sf.file and other state in NewSwapFileChunk
and FreeResource by using sf.chunkTrackingLock consistently. Also
set sf.file to nil after closing to prevent reuse.

* weed mount: improve swap directory creation logic

- Check error for os.MkdirAll and log it if it fails.
- Use 0700 permissions for the swap directory for better security.
- Improve error logging context.

* weed mount: add unit tests for swap file creation

Add tests to verify:
- Concurrent initialization of the swap file.
- Correct directory permissions (0700).
- Automatic directory recreation if deleted.

* weed mount: fix thread-safety in swap file unit tests

Use atomic.Uint32 to track failures within goroutines in
TestSwapFile_NewSwapFileChunk_Concurrent to avoid unsafe calls to
t.Errorf from multiple goroutines.

* weed mount: simplify swap file creation logic

Refactor the directory check and retry logic for better readability and
to avoid re-using the main error variable for directory creation errors.
Remove redundant error logging.

* weed mount: improve error checking in swap file tests

Explicitly check if NewSwapFileChunk returns nil to provide more
informative failures.

* weed mount: update DirtyPages interface to return error

Propagate errors from SaveDataAt when swap file creation fails. This
prevents potential panics in the write path.

* weed mount: handle AddPage errors in write paths

Update ChunkedDirtyPages and PageWriter to propagate errors and update
WFS.Write and WFS.CopyFileRange to return fuse.EIO on failure.

* weed mount: update swap directory creation error message

Change "recreate" to "create/recreate" to better reflect that this path
is also taken during the initial creation of the swap directory.

---------

Co-authored-by: lixiang58 <lixiang58@lenovo.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-01-04 01:12:37 -08:00
Viktor Kuzmin 06e723f629 Waiting for writes to complete with acquired locks may lead to deadlock, fixes #4952 2023-11-17 07:38:14 -08:00
chrislu bfe5d910c6 use one readerCache for the whole file 2023-01-16 22:43:02 -08:00
chrislu c24678153c use memory chunks only for sequential writes 2023-01-15 21:10:08 -08:00
chrislu 313fd17ae4 allow random writes for mem chunk 2023-01-13 20:32:18 -08:00
chrislu 5423790b2c add back previous chunk upload selection algo 2023-01-03 22:50:39 -08:00
Chris Lu d4566d4aaa more solid weed mount (#4089)
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
2023-01-02 23:20:45 -08:00
chrislu d1797deccb remove duplicated logic of removing from writableChunks 2022-12-21 14:15:11 -08:00
chrislu f7beba8515 unlock before submitting the uploading jobs 2022-12-20 12:52:30 -08:00
chrislu 28d479e5c0 mount: adjust locking for upload pipeline 2022-12-19 15:07:22 -08:00
chrislu ae93c966d9 ensure memory is aligned
fix https://github.com/seaweedfs/seaweedfs/issues/3427
2022-08-10 22:27:13 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu 8b3d76b24d prevent concurrent access during shutdown 2022-04-05 10:53:36 -07:00
chrislu 5cba8e51c5 refactor 2022-03-13 18:34:57 -07:00
chrislu f2f68f675e write to disk during random writes, limiting total disk spaces used 2022-03-13 18:17:35 -07:00
chrislu 011a41b561 add back writes to swap file when too many in memory chunks are used. 2022-03-09 22:26:51 -08:00
chrislu 3aeee3d748 ensure releasing file handle 2022-03-07 14:01:24 -08:00
chrislu 8136384473 remove debug message 2022-03-07 11:22:26 -08:00
chrislu 2b955c1713 support read 2022-02-13 22:50:44 -08:00