mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-19 02:15:02 +01:00
94de65b3b5d79b85e1112321199e81ccc1273cc1
26
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
11af0e0c12 |
mount: move an open file's read view to the chunks a merge wrote (#12722)
* mount: move an open file's read view to the chunks a merge wrote When a flush finds that a file's chunks take more than twice its size, maybeMergeChunks reads the file back and writes it again as new chunks. The commit that publishes them makes the filer delete the chunks they replaced. The flush put the new chunks into the handle's entry but left its read view on the old ones, so every descriptor that kept the file open went on reading the replaced chunks. Those read back until their volume is vacuumed. After that a read first retries the fetch of the missing needle for about 13 s; then refreshEntryForRead installs the filer's entry and the read is retried. The refresh skips a handle with writes it has not flushed, and such a read returns EIO instead. Files that are kept open and rewritten in place, such as PostgreSQL data files and SQLite databases, are the ones whose chunks bloat, so they are the ones that hit this. installRewrittenChunks puts the merge's chunks into the entry and the read view together, under the entry lock. Chunks attached while the merge ran were not part of it and stay. Flushes without a merge are unchanged. TestMergeMovesReadViewOffReplacedChunks covers the handle. TestReadOpenHandleAfterChunkMerge runs it on a real mount, with and without an unflushed write; for it, the fuse_integration framework starts mini on a free master port and exposes the address. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: drain uploads before a merge replaces the chunks Uploads attach their chunks without the handle lock, first to the entry and then to the read view, and writes that landed after doFlush drained the pages may still be uploading when the flush takes its lock. An upload that attached during the merge could keep a chunk older than data the merge already holds, or put a chunk the merge replaced back into the read view. Either outranks the merge's zero-timestamp chunk. When the chunks are bloated enough to merge, the flush now drains the pages again while its lock holds writes off, and takes the chunks again. FlushAll returns only after every upload has attached to both the entry and the read view, so nothing attaches during the merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: leave a pre-merge upload failure to the next flush The drain before a merge returned an upload failure through the metadata flush. doFlush then retried it as a filer error and answered EIO, where the page flush answers a full disk with ENOSPC. The drain is there only to wait for the uploads: a failure stays with the pages, and the next flush reports it as a write error, as it would without a merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep chunks attached while a flush installs its compacted list A write that lands after doFlush drained the pages can still be uploading when flushMetadataToFiler takes its chunk snapshot. Its upload attaches the chunk without the handle lock, and the plain flush then replaced the entry's chunks with the list compacted from the snapshot, dropping it from the handle and from every later commit. Keep chunks attached after the snapshot, as the merge path already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: fail a merge's flush on a failed drain A failed upload found by the drain before a merge was only logged, and the flush went on to publish a size covering bytes that no chunk stores. It now fails the flush as a pageUploadError before anything is published; doFlush maps it with writeErrorToFuseStatus, as it does its own FlushData failure, and does not retry it as a filer error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: clone a flush's request entry under the entry lock The request snapshot cloned the live entry after GetEntry had released its read lock. Uploads attach chunks without the handle lock, so an AppendChunks that landed during the clone raced it on the Chunks field. Clone inside UpdateEntry, as content flush recovery does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: leave a merge to a later flush while uploads are pending Before a merge the flush drained the pages while it held the handle and publication locks. A writeback upload retries a transient storage error until it succeeds, so that drain could hold both locks for as long as storage stayed down, stalling reads, writes and stat on the handle. The flush now checks for pending uploads and, if any, publishes the compacted chunks and leaves the merge to a later flush. With none pending the lock still holds writes off, so nothing attaches during the merge, and a failed upload still fails the flush as a write error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep room for pending uploads when a flush merges onto a moved entry A flush that leaves the chunk merge to a later flush publishes while uploads are still pending, and writes that landed after the drain can be uploading on any flush. Merging a peer's inline content or a shrink into such a publication lets the upload attach its chunk next to the inline bytes, where reads never reach it. Tell createContentEntry when uploads are pending so the merge keeps room for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test chunk merges through the filer's chunk proxy, also under load The tests serve the filer's chunk proxy, and like the filer the fake deletes the chunks each commit replaces, so a read view still naming them fails: - TestFlushMergeThroughChunkProxy/Quiet: the flush merges, and reads through the open handle return the file from the merged chunk alone. - TestFlushMergeThroughChunkProxy/UploadInFlight: an older page is still uploading while a newer page of the same bytes has attached. The flush leaves the merge, reads return the newer page, and the next flush merges. - TestFlushMergeRacesWithWritesAndReads: four writers, three flushers and a reader on one handle, with uploads delayed at random. The writers pause now and then, so 14 to 18 flushes a run merge and the others leave the merge to a later one. Every read must succeed within 2 s, and at the end the handle and the filer must both hold exactly what was written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: let a held upload go when UploadInFlight fails The subtest's proxy holds the older page's upload until the test releases it, and only the passing path did. A failure in between left the handler blocked, so the proxy's cleanup waited on it and the failure surfaced as a suite timeout. Release it from a cleanup as well, once, and bound the waits for the upload and the drains. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
4f70ad015e |
mount: fail the write when the swap file chunk cannot be created (#12655)
* mount: fail the write when the swap file chunk cannot be created #7957 added a nil check after NewSwapFileChunk so a failed swap file creation returns an error instead of panicking. The result was assigned to pageChunk, a PageChunk interface, before the check, and an interface holding a nil *SwapFileChunk is not nil, so the check never fired: SaveDataAt stored the nil chunk and panicked in WriteDataAt. Check the *SwapFileChunk before converting it. The new test points the swap directory below a regular file; on master it panics in SwapFileChunk.WriteDataAt. Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com> * mount: grow the file only after its write is buffered Write set the new size, times, pending append offset and uncommitted bytes before AddPage. When AddPage failed (now that a swap file chunk that cannot be created returns an error), the write was reported as failed but the larger size stayed on the handle's entry, so the next flush published it with a hole where the data should be. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: report a partly buffered write as a short write PageWriter.AddPage fills one chunk at a time, so when a later chunk cannot be created the earlier chunks already hold their bytes. Return how much was buffered, and let Write grow the size and times over that prefix and report it, instead of failing the whole write while the prefix still reaches the next flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: report the bytes a failed copy_file_range already buffered A streaming copy whose page fails partway returned an error even though earlier pages, or the start of the failing one, were already buffered and would reach the next flush with the destination size left short. Count the buffered bytes, grow the destination over them, and return them as a short copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: free the copy test's destination pages The buffered memory chunk counts toward the process-wide memory chunk ceiling, so repeated runs of the test pushed later ones onto the unusable swap directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: count only the bytes a swap chunk stored A swap chunk whose file write fails stores a prefix or nothing, but SaveDataAt returned no error and AddPage counted the whole span as buffered. SaveDataAt now reports the short write, and AddPage passes the stored count up so a write is never acknowledged past it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: fail a partly buffered writeback write Kernel writeback ignores the count in a WRITE reply, so a short write there drops the unbuffered rest without an error. A write flagged as page-cache writeback now keeps its buffered prefix but returns the error, which the kernel reports on fsync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com> Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
fd81a50c57 |
mount: keep retrying writeback chunk uploads through a volume outage (#12465)
* mount: keep retrying writeback chunk uploads through a volume outage With -writebackCache, close() returns before the data is uploaded. When the volume servers stay unreachable past the upload's own retry (~20 s), the chunk was dropped and the file never appeared, with no error to the application. Retry transient upload errors with the same bytes until they succeed, unless the dirty pages were discarded. Permanent errors and the synchronous mode keep failing as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test writeback upload retries and the cases that still fail Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * operation: keep the filer's read-only verdict on assign failures UploadWithRetry flattened AssignVolumeResponse errors into a string, so callers could not tell a read-only rule from an outage. Wrap the filer's coded error so errors.Is(err, filer_pb.ErrReadOnly) works; the message is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: retry writeback uploads the upload layer reassigns, never read-only A filer proxy that cannot reach the volume answers 500 with an empty body, which util.IsTransientError does not match, so that outage still dropped the chunk. Also retry what operation.ShouldReassignUpload takes. A read-only rule stays final even when its path or message contains a transient word such as "unavailable". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: stop a writeback upload once its dirty pages are discarded Destroy() was only seen after a failed attempt, so a truncate during the retry wait still started another upload, and a chunk that finished after its pages were replaced was attached to the new content. Check before each attempt, and drop the chunk instead of attaching it; volume.fsck reclaims it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep Destroy from slipping between the discard check and the append truncateEntry destroys the pages and then installs the new entry, so a Destroy landing after the check but before AddChunks put the old chunk into the replacing entry. Hold one lock across the check and the append, and take it in Destroy while setting the flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: leave the Windows interrupt drain to the capped filesystem hook Interrupt hooks run in reverse registration order, so the Windows hook ran before the filesystem hook and waited on pending writeback flushes without a bound. With writeback uploads now retrying through an outage, Ctrl-C could wait until the volume servers came back. Unmount only, as on Linux, and let the filesystem hook drain for at most 30 s and keep the write cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test a writeback upload whose assign keeps timing out With every volume server gone the assign RPC times out until the filer client gives up; that error must be retried like a refused upload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
214d3599d3 |
windows mount: cache file data, resolved paths and attributes (#10703)
* benchmark tool for mounted filesystems * ci: on-demand mount benchmark, native WinFsp vs rclone plus a Linux reference * windows mount: let the Windows cache manager cache file data WinFsp only turns the cache manager on for a file when FileInfoTimeout is infinite; at any finite value every application read and write is a synchronous trip into the mount process at whatever size the application issued. Metadata events already reach FspFileSystemNotify, which purges a changed file's cached pages and attributes, so an infinite timeout stays coherent. The dir listing, volume info and EA timeouts are pinned to one second so they do not silently inherit the infinity. * windows mount: cache resolved paths and attributes in the adapter WinFsp addresses every operation by path and has no FORGET, so the adapter walked the whole path through Lookup on each one, and in a directory the filer has not listed yet every walk was a filer round trip; nothing played the part of the kernel's dentry and attribute caches. The path cache owns one lookup reference per entry the way the kernel holds one until FORGET, serves attribute reads for files without an open handle, and is purged by the mount's own mutations and by metadata events, with the timeout as backstop. * windows mount: keep a closed file's attributes cached Open steals the path's cache entry for its handle and Release returned the reference with a purge, so the stat that follows every copied file walked to the filer again. Reading the handle's final attributes before it goes away and moving the reference back into the cache serves that stat locally, the way the kernel's attribute cache does after a close. Only if the path still names that inode, though: WinFsp reports the path the handle opened with, and after a delete-on-close or a rename caching it would resurrect an entry that is gone. * windows mount: persist entries at create, and let the flush stay at close WinFsp posts the cleanup and close that carry the flush after CloseHandle has returned, so deferring the filer entry to the flush let everything that reads through the filer race an unflushed close: a listing missed just-written files, and a directory rename moved a directory on the filer before its newest child existed there, leaving the straggler flush to recreate the child under the dead path. Flush-at-cleanup is not the answer either: it makes every handle's cleanup flush, and those flushes race the unlinks of delete-on-close, re-inserting the entry the unlink just removed. Persisting the entry at create takes the ordering question away. * mount: flush written pages before a truncate shrinks past them The shrink trims chunks, but written pages that have not become chunks yet are invisible to it, so the next flush wrote them back and the file grew again, resurrecting the truncated bytes. Windows hits this on every write-then-shrink because its flush runs after CloseHandle, but the gap is platform-neutral. * mount: order a file's unlink against its in-flight flush Unlink set the handle's deleted flag bare, so a flush already past its own check of that flag wrote the entry back right after the delete removed it, and a delete-on-close file outlived its last handle. The flag is now set under the handle's flush lock and re-checked under it, so a flush either completes before the delete or sees the flag and skips. An eagerly created handle also starts clean: the dirty mark existed to make the deferred filer create happen at flush, and eager creates have nothing to flush. |
||
|
|
5553e7b876 |
mount: surface ENOSPC instead of endless waiting when the cluster is full (#10341)
When every volume is full, a chunk upload fails only after the assign retry budget, the failed chunk's data is dropped, and the mount kept accepting writes anyway. cp would crawl for hours pushing the rest of the file through a pipeline that could not persist it, and only close() reported an error - a generic EIO. Poison the file handle on the first failed chunk upload so subsequent writes fail immediately, and map "no writable volumes" / "no free volumes" upload errors to ENOSPC so the writing process aborts with "No space left on device". Also guard lastErr with a mutex: it was written concurrently by uploader goroutines, and keep the first error so later failures do not mask the root cause. |
||
|
|
00a2e22478 |
fix(mount): remove fid pool to stop master over-allocating volumes (#9111)
* fix(mount): remove fid pool to stop master over-allocating volumes
The writeback-cache fid pool pre-allocated file IDs with
ExpectedDataSize = ChunkSizeLimit (typically 8+ MB). The master's
PickForWrite charges count * expectedDataSize against the volume's
effectiveSize, so a full pool refill could charge hundreds of MB
against a single volume before any bytes were actually written.
That tripped RecordAssign's hard-limit path and eagerly removed
volumes from writable, causing the master to grow new volumes
even when the real data being written was tiny.
Drop the pool entirely. Every chunk upload goes through
UploadWithRetry -> AssignVolume with no ExpectedDataSize hint,
letting the master fall back to the 1 MB default estimate. The
mount->filer grpc connection is already cached in pb.WithGrpcClient
(non-streaming mode), so per-chunk AssignVolume is a unary RPC
over an existing HTTP/2 stream, not a full dial. Path-based
filer.conf storage rules now apply to mount chunk assigns again,
which the pool had to skip.
Also remove the now-unused operation.UploadWithAssignFunc and its
AssignFunc type.
* fix(upload): populate ExpectedDataSize from actual chunk bytes
UploadWithRetry already buffers the full chunk into `data` before
calling AssignVolume, so the real size is known. Previously the
assign request went out with ExpectedDataSize=0, making the master
fall back to the 1 MB DefaultNeedleSizeEstimate per fid — same
over-reservation symptom the pool had, just smaller per call.
Stamp ExpectedDataSize = len(data) before the assign RPC when the
caller hasn't already set it. This covers mount chunk uploads,
filer_copy, filersink, mq/logstore, broker_write, gateway_upload,
and nfs — all the UploadWithRetry paths.
* fix(assign): pass real ExpectedDataSize at every assign call site
After removing the mount fid pool, per-chunk AssignVolume calls went
out with ExpectedDataSize=0, making the master fall back to its 1 MB
DefaultNeedleSizeEstimate. That's still an over-estimate for small
writes. Thread the real payload size through every remaining assign
site so RecordAssign charges effectiveSize accurately and stops
prematurely marking volumes full.
- filer: assignNewFileInfo now takes expectedDataSize and stamps it
on both primary and alternate VolumeAssignRequests. Callers pass:
- SSE data-to-chunk: len(data)
- copy manifest save: len(data)
- streamCopyChunk: srcChunk.Size
- TUS sub-chunk: bytes read
- saveAsChunk (autochunk/manifestize): 0 (small, size unknown
until the reader is drained; master uses 1 MB default)
- filer gRPC remote fetch-and-write: ExpectedDataSize = chunkSize
after the adaptive chunkSize is computed.
- ChunkedUploadOption.AssignFunc gains an expectedDataSize parameter;
upload_chunked.go passes the buffered dataSize at the call site.
S3 PUT assignFunc stamps it on the AssignVolumeRequest.
- S3 copy: assignNewVolume / prepareChunkCopy take expectedDataSize;
all seven call sites pass the source chunk's Size.
- operation.SubmitFiles / FilePart.Upload: derive per-fid size from
FileSize (average for batched requests, real per-chunk size for
sequential chunk assigns).
- benchmark: pass fileSize.
- filer append-to-file: pass len(data).
* fix(assign): thread size through SaveDataAsChunkFunctionType
The saveAsChunk path (autochunk, filer_copy, webdav, mount) ran
AssignVolume before the reader was drained, so it had to pass
ExpectedDataSize=0 and fall back to the master's 1 MB default.
Add an expectedDataSize parameter to SaveDataAsChunkFunctionType.
- mergeIntoManifest already has the serialized manifest bytes, so
it passes uint64(len(data)) directly.
- Mount's saveDataAsChunk ignores the parameter because it uses
UploadWithRetry, which already stamps len(data) on the assign
after reading the payload.
- webdav and filer_copy saveDataAsChunk follow the same UploadWithRetry
path and also ignore the hint.
- Filer's saveAsChunk (used for manifestize) plumbs the value to
assignNewFileInfo so manifest-chunk assigns get a real size.
Callers of saveFunc-as-value (weedfs_file_sync, dirty_pages_chunked)
pass the chunk size they're about to upload.
|
||
|
|
d7865909ba |
feat(mount): proactive flush of idle writable chunks (#9094)
* feat(mount): proactive flush of idle writable chunks Add a background goroutine that periodically scans writable chunks across all open file handles and seals those that are idle and unlikely to receive further writes, submitting them for async upload. A writable chunk is proactively flushed when it has been idle for 500ms AND meets one of: nearly full (>=90%), behind the sequential write frontier by 2+ chunks, or stale for 5+ seconds. Flushing only happens when the upload pipeline has spare capacity (< half of concurrent writer slots in use). This prevents partial chunks from accumulating until fsync/close, which is particularly beneficial for bursty or small-file workloads where chunks may never reach IsComplete(). Also fixes a latent bug in ActivityScore where MarkRead/MarkWrite used value receivers, silently discarding all mutations. * refactor(mount): reuse WriterPattern instead of duplicating sequential detection Remove the isSequential atomic from UploadPipeline. The proactive flusher now reads IsSequentialMode() from the existing WriterPattern on PageWriter and passes it as a parameter to ProactiveFlush. This avoids duplicating the sequential/random detection that WriterPattern already maintains. * fix(mount): address PR review feedback - Make ActivityScore thread-safe using atomics (CAS loop for score updates, atomic load/swap for timestamp). Previously MarkRead was called under RLock while MarkWrite held a write lock, creating a data race on the shared fields. - Fix ProactiveFlush half-capacity guard: use multiplication (uploaderCount*2 >= max) instead of floor division (max/2) which misbehaves for odd or small concurrentWriterMax values. * fix(mount): review fixes for proactive flush - Fix TOCTOU race in lastWriteChunkIndex update: use CAS loop so concurrent writers cannot regress the frontier. - Remove unused UploaderCount() getter. - Reuse the caller-provided tsNs instead of calling time.Now() again in WriteDataAt for lastWriteTsNs, eliminating a redundant syscall per write. |
||
|
|
bdebd63f7c |
fix(mount): evict writable chunks when writeBufferSizeMB cap is reached (#9091)
* fix(mount): evict writable chunks when writeBufferSizeMB cap is reached Reported on #8777: fio 4k randwrite with --nrfiles=1000 on a weed mount hangs indefinitely after ~1 second of activity, volume server QPS drops to zero, and the test never makes progress beyond a few hundred writes. A scaled-down local repro (4 jobs x 250 nrfiles x 40 MiB with -chunkSizeLimitMB=32 -writeBufferSizeMB=10240) reproduced the hang exactly: fio stuck in fio_state=SETTING_UP, mount idle, all writer goroutines parked in page_writer.(*WriteBufferAccountant).Reserve with n=0x2000000 (32 MiB). Root cause: UploadPipeline.SaveDataAt reserves a full chunkSize slot against the global WriteBufferAccountant the first time it allocates a writable chunk for a given file. The reservation is released by SealedChunk.FreeReference only after the chunk's async upload completes, and chunks only move from writable to sealed when they fill or when the file is flushed/closed. For 4k random writes with small per-file data and iodepth holding files open, writable chunks never fill and never close, so their slots are pinned forever. Once openFiles * chunkSize exceeds writeBufferSizeMB, every new writer blocks in Reserve's cond.Wait with no possible waker, a hard deadlock. The user-visible symptoms (volume QPS=0, waited 30min, still running) are exactly that state. In the reported config, 8 jobs * 1000 nrfiles * 32 MiB = 256 GB of reservations against a 10 GB cap. Fix: add a SetEvictor hook on WriteBufferAccountant. When Reserve would otherwise block on the cond, it single-flights an evictor callback that walks wfs.fhMap, picks the first open file handle with a writable chunk, and force-seals its fullest writable chunk via a new UploadPipeline.EvictOneWritableChunk. Force-sealing submits the chunk for async upload on the existing uploader pool; the upload's SealedChunk.FreeReference releases the global slot, broadcasts on the accountant cond, and unblocks the waiter. The fullest-first heuristic matches the existing over-limit path in SaveDataAt and keeps us from thrashing on repeatedly re-sealing half-empty chunks. The original #9066 motivation (cap global write-pipeline growth so swap can't be filled while uploads stall) is preserved: Reserve still blocks when the cap is actually exhausted by in-flight uploads, and a failed evictor (no writable chunks to seal) falls through to cond.Wait exactly as before. Verified end-to-end in a privileged debian container running a full master + volume + filer + weed mount stack. Against the reporter's config shrunk to the same memory footprint (4 jobs x 250 nrfiles x 40 MiB, -chunkSizeLimitMB=32 -writeBufferSizeMB=10240): - baseline binary (master): fio stalls after ~320 writes, killed by timeout at 400s with io=1280 KiB and bw=2383 B/s. - patched binary: fio completes in 1.9 s at 82.8 MiB/s, all 40,000 writes issued, 156 MiB written, rc=0. page_writer unit tests still pass. Touches: WriteBufferAccountant (SetEvictor + single-flight in Reserve), UploadPipeline.EvictOneWritableChunk, DirtyPages interface + the two existing implementers (ChunkedDirtyPages, PageWriter), and a new weedfs_write_buffer_evict.go holding the WFS.evictOneWritableChunk callback that walks fhMap. * fix(mount): re-check write-budget cap after evict and make eviction panic-safe Addresses two review concerns on #9091: 1. CodeRabbit flagged that the original Reserve loop only considered the break condition when the evictor reported success: if evicted { if a.used+n <= a.cap || a.used == 0 { break } } a.cond.Wait() A concurrent Release firing during the evictor's execution window (we drop a.mu around the callback so uploader goroutines can drain) would land its Broadcast on an empty waiter set and then be missed, because we unconditionally call cond.Wait() on the same iteration. In practice an in-flight async upload eventually fires another broadcast so this would delay rather than permanently hang — but the logic is wrong either way and should re-check the cap after every evictor round. 2. Gemini flagged that setting `a.evicting = true` and dropping `a.mu` without a deferred unwind leaves the accountant in a broken state if the evictor panics: the flag stays set forever and no Reserve caller can ever run another eviction, even though the mutex gets unlocked by the normal stack unwind. Pull the evictor call into a tiny helper runEvictorLocked whose defer re-acquires a.mu and clears a.evicting regardless of panic status, then broadcasts so other blocked Reservers re-evaluate. Reserve checks the cap condition immediately after the helper returns; only falls through to cond.Wait() when the budget is still exhausted. No behavior change on the happy path; the fix is in the error and race corners. * doc(mount): clarify eviction-scan cost on the blocked-Reserve path Gemini review on #9091 flagged that the fhMap scan inside evictOneWritableChunk is O(open files). Document why the cost is bounded to the Reserve-blocked path and paid at most once per chunkSize drained, rather than on the write hot path, and record the reason we do not preempt with an LRU. |
||
|
|
7a7f220224 |
feat(mount): cap write buffer with -writeBufferSizeMB (#9066)
* feat(mount): cap write buffer with -writeBufferSizeMB Without a bound on the per-mount write pipeline, sustained upload failures (e.g. volume server returning "Volume Size Exceeded" while the master hasn't yet rotated assignments) let sealed chunks pile up across open file handles until the swap directory — by default os.TempDir() — fills the disk. Reported on 4.19 filling /tmp to 1.8 TB during a large rclone sync. Add a global WriteBufferAccountant shared across every UploadPipeline in a mount. Creating a new page chunk (memory or swap) first reserves ChunkSize bytes; when the cap is reached the writer blocks until an uploader finishes and releases, turning swap overflow into natural FUSE-level backpressure instead of unbounded disk growth. The new -writeBufferSizeMB flag (also accepted via fuse.conf) defaults to 0 = unlimited, preserving current behavior. Reserve drops chunksLock while blocking so uploader goroutines — which take chunksLock on completion before calling Release — cannot deadlock, and an oversized reservation on an empty accountant succeeds to avoid single-handle starvation. * fix(mount): plug write-budget leaks in pipeline Shutdown Review on #9066 caught two accounting bugs on the Destroy() path: 1. Writable-chunk leak (high). SaveDataAt() reserves ChunkSize before inserting into writableChunks, but Shutdown() only iterated sealedChunks. Truncate / metadata-invalidation flows call Destroy() (via ResetDirtyPages) without flushing first, so any dirty but unsealed chunks would permanently shrink the global write budget. Shutdown now frees and releases writable chunks too. 2. Double release with racing uploader (medium). Shutdown called accountant.Release directly after FreeReference, while the async uploader goroutine did the same on normal completion — under a Destroy-before-flush race this could underflow the accountant and let later writes exceed the configured cap. Move accounting into SealedChunk.FreeReference itself: the refcount-zero transition is exactly-once by construction, so any number of FreeReference calls release the slot precisely once. Add regression tests for the writable-leak and the FreeReference idempotency guarantee. * test(mount): remove sleep-based race in accountant blocking test Address review nits on #9066: - Replace time.Sleep(50ms) proxy for "goroutine entered Reserve" with a started channel the goroutine closes immediately before calling Reserve. Reserve cannot make progress until Release is called, so landed is guaranteed false after the handshake — no arbitrary wait. - Short-circuit WriteBufferAccountant.Used() in unlimited mode for consistency with Reserve/Release, avoiding a mutex round-trip. * test(mount): add end-to-end write-buffer cap integration test Exercises the full write-budget plumbing with a small cap (4 chunks of 64 KiB = 256 KiB) shared across three UploadPipelines fed by six concurrent writers. A gated saveFn models the "volume server rejecting uploads" condition from the original report: no sealed chunk can drain until the test opens the gate. A background sampler records the peak value of accountant.Used() throughout the run. The test asserts: - writers fill the budget and then block on Reserve (Used() stays at the cap while stalled) - Used() never exceeds the configured cap even under concurrent pressure from multiple pipelines - after the gate opens, writers drain to zero - peak observed Used() matches the cap (262144 bytes in this run) While wiring this up, the race detector surfaced a pre-existing data race on UploadPipeline.uploaderCount: the two glog.V(4) lines around the atomic Add sites read the field non-atomically. Capture the new value from AddInt32 and log that instead — one-liner each, no behavioral change. * test(fuse): end-to-end integration test for -writeBufferSizeMB Exercise the new write-buffer cap against a real weed mount so CI (fuse-integration.yml) covers the FUSE→upload-pipeline→filer path, not just the in-package unit tests. Uses a 4 MiB cap with 2 MiB chunks so every subtest's total write demand is multiples of the budget and Reserve/Release must drive forward progress for writes to complete. Subtests: - ConcurrentLargeWrites: six parallel 6 MiB files (36 MiB total, ~18 chunk allocations) through the same mount, verifies every byte round-trips. - SingleFileExceedingCap: one 20 MiB file (10 chunks) through a single handle, catching any self-deadlock when the pipeline's own earlier chunks already fill the global budget. - DoesNotDeadlockAfterPressure: final small write with a 30s timeout, catching budget-slot leaks that would otherwise hang subsequent writes on a still-full accountant. Ran locally on Darwin with macfuse against a real weed mini + mount: === RUN TestWriteBufferCap --- PASS: TestWriteBufferCap (1.82s) * test(fuse): loosen write-buffer cap e2e test + fail-fast on hang On Linux CI the previous configuration (-writeBufferSizeMB=4, -concurrentWriters=4 against a 20 MiB single-handle write) deterministically hung the "Run FUSE Integration Tests" step to the 45-minute workflow timeout, while on macOS / macfuse the same test completes in ~2 seconds (see run 24386197483). The Linux hang shows up after TestWriteBufferCap/ConcurrentLargeWrites completes cleanly, then TestWriteBufferCap/SingleFileExceedingCap starts and never emits its PASS line. Change: - Loosen the cap to 16 MiB (8 × 2 MiB chunk slots) and drop the custom -concurrentWriters override. The subtests still drive demand well above the cap (32 MiB concurrent, 12 MiB single-handle), so Reserve/Release is still on every chunk-allocation path; the cap just gives the pipeline enough headroom that interactions with the per-file writableChunkLimit and the go-fuse MaxWrite batching don't wedge a single-handle writer on a slow runner. - Wrap every os.WriteFile in a writeWithTimeout helper that dumps every live goroutine on timeout. If this ever re-regresses, CI surfaces the actual stuck goroutines instead of a 45-minute walltime. - Also guard the concurrent-writer goroutines with the same timeout + stack dump. The in-package unit test TestWriteBufferCap_SharedAcrossPipelines remains the deterministic, controlled verification of the blocking Reserve/Release path — this e2e test is now a smoke test for correctness and absence of deadlocks through a real FUSE mount, which is all it should be. * fix: address PR #9066 review — idempotent FreeReference, subtest watchdog, larger single-handle test FreeReference on SealedChunk now early-returns when referenceCounter is already <= 0. The existing == 0 body guard already made side effects idempotent, but the counter itself would still decrement into the negatives on a double-call — ugly and a latent landmine for any future caller that does math on the counter. Make double-call a strict no-op. test(fuse): per-subtest watchdog + larger single-handle test - Add runSubtestWithWatchdog and wrap every TestWriteBufferCap subtest with a 3-minute deadline. Individual writes were already timeout-wrapped but the readback loops and surrounding bookkeeping were not, leaving a gap where a subtest body could still hang. On watchdog fire, every live goroutine is dumped so CI surfaces the wedge instead of a 45-minute walltime. - Bump testLargeFileUnderCap from 12 MiB → 20 MiB (10 chunks) to exceed the 16 MiB cap (8 slots) again and actually exercise Reserve/Release backpressure on a single file handle. The earlier e2e hang was under much tighter params (-writeBufferSizeMB=4, -concurrentWriters=4, writable limit 4); with the current loosened config the pressure is gentle and the goroutine-dump-on-timeout safety net is in place if it ever regresses. Declined: adding an observable peak-Used() assertion to the e2e test. The mount runs as a subprocess so its in-process WriteBufferAccountant state isn't reachable from the test without adding a metrics/RPC surface. The deterministic peak-vs-cap verification already lives in the in-package unit test TestWriteBufferCap_SharedAcrossPipelines. Recorded this rationale inline in TestWriteBufferCap's doc comment. * test(fuse): capture mount pprof goroutine dump on write-timeout The previous run (24388549058) hung on LargeFileUnderCap and the test-side dumpAllGoroutines only showed the test process — the test's syscall.Write is blocked in the kernel waiting for FUSE to respond, which tells us nothing about where the MOUNT is stuck. The mount runs as a subprocess so its in-process stacks aren't reachable from the test. Enable the mount's pprof endpoint via -debug=true -debug.port=<free>, allocate the port from the test, and on write-timeout fetch /debug/pprof/goroutine?debug=2 from the mount process and log it. This gives CI the only view that can actually diagnose a write-buffer backpressure deadlock (writer goroutines blocked on Reserve, uploader goroutines stalled on something, etc). Kept fileSize at 20 MiB so the Linux CI run will still hit the hang (if it's genuinely there) and produce an actionable mount-side dump; the alternative — silently shrinking the test below the cap — would lose the regression signal entirely. * review: constructor-inject accountant + subtest watchdog body on main Two PR-#9066 review fixes: 1. NewUploadPipeline now takes the WriteBufferAccountant as a constructor parameter; SetWriteBufferAccountant is removed. In practice the previous setter was only called once during newMemoryChunkPages, before any goroutine could touch the pipeline, so there was no actual race — but constructor injection makes the "accountant is fixed at construction time" invariant explicit and eliminates the possibility of a future caller mutating it mid-flight. All three call sites (real + two tests) updated; the legacy TestUploadPipeline passes a nil accountant, preserving backward-compatible unlimited-mode behavior. 2. runSubtestWithWatchdog now runs body on the subtest main goroutine and starts a watcher goroutine that only calls goroutine-safe t methods (t.Log, t.Logf, t.Errorf). The previous version ran body on a spawned goroutine, which meant any require.* or writeWithTimeout t.Fatalf inside body was being called from a non-test goroutine — explicitly disallowed by Go's testing docs. The watcher no longer interrupts body (it can't), so body must return on its own — which it does via writeWithTimeout's internal 90s timeout firing t.Fatalf on (now) the main goroutine. The watchdog still provides the critical diagnostic: on timeout it dumps both test-side and mount-side (via pprof) goroutine stacks and marks the test failed via t.Errorf. * fix(mount): IsComplete must detect coverage across adjacent intervals Linux FUSE caps per-op writes at FUSE_MAX_PAGES_PER_REQ (typically 1 MiB on x86_64) regardless of go-fuse's requested MaxWrite, so a 2 MiB chunk filled by a sequential writer arrives as two adjacent 1 MiB write ops. addInterval in ChunkWrittenIntervalList does not merge adjacent intervals, so the resulting list has two elements {[0,1M], [1M,2M]} — fully covered, but list.size()==2. IsComplete previously returned `list.size() == 1 && list.head.next.isComplete(chunkSize)`, which required a single interval covering [0, chunkSize). Under that rule, chunks filled by adjacent writes never reach IsComplete==true, so maybeMoveToSealed never fires, and the chunks sit in writableChunks until FlushAll/close. SaveContent handles the adjacency correctly via its inline merge loop, so uploads work once they're triggered — but IsComplete is the gate that triggers them. This was a latent bug: without the write-buffer cap, the overflow path kicks in at writableChunkLimit (default 128) and force-seals chunks, hiding the leak. #9066's -writeBufferSizeMB adds a tighter global cap, and with 8 slots / 20 MiB test, the budget trips long before overflow. The writer blocks in Reserve, waiting for a slot that never frees because no uploader ever ran — observed in the CI run 24390596623 mount pprof dump: goroutine 1 stuck in WriteBufferAccountant.Reserve → cond.Wait, zero uploader goroutines anywhere in the 89-goroutine dump. Walk the (sorted) interval list tracking the furthest covered offset; return true if coverage reaches chunkSize with no gaps. This correctly handles adjacent intervals, overlapping intervals, and out-of-order inserts. Added TestIsComplete_AdjacentIntervals covering single-write, two adjacent halves (both orderings), eight adjacent eighths, gaps, missing edges, and overlaps. * test(fuse): route mount glog to stderr + dump mount on any write error Run 24392087737 (with the IsComplete fix) no longer hangs on Linux — huge progress. Now TestWriteBufferCap/LargeFileUnderCap fails with 'close(...write_buffer_cap_large.bin): input/output error', meaning a chunk upload failed and pages.lastErr propagated via FlushData to close(). But the mount log in the CI artifact is empty because weed mount's glog defaults to /tmp/weed.* files, which the CI upload step never sees, so we can't tell WHICH upload failed or WHY. Add -logtostderr=true -v=2 to MountOptions so glog output goes to the mount process's stderr, which the framework's startProcess redirects into f.logDir/mount.log, which the framework's DumpLogs then prints to the test output on failure. The -v=2 floor enables saveDataAsChunk upload errors (currently logged at V(0)) plus the medium-level write_pipeline/upload traces without drowning the log in V(4) noise. Also dump MOUNT goroutines on any writeWithTimeout error (not just timeout). The IsComplete fix means we now get explicit errors instead of silent hangs, and the goroutine dump at the error moment shows in-flight upload state (pending sealed chunks, retry loops, etc) that a post-failure log alone can't capture. |
||
|
|
c909724bf1 |
Fix: prevent panic when swap file creation fails (#7957)
* Fix: prevent panic when swap file creation fails * weed mount: fix race condition in swap file initialization Ensure thread-safe access to sf.file and other state in NewSwapFileChunk and FreeResource by using sf.chunkTrackingLock consistently. Also set sf.file to nil after closing to prevent reuse. * weed mount: improve swap directory creation logic - Check error for os.MkdirAll and log it if it fails. - Use 0700 permissions for the swap directory for better security. - Improve error logging context. * weed mount: add unit tests for swap file creation Add tests to verify: - Concurrent initialization of the swap file. - Correct directory permissions (0700). - Automatic directory recreation if deleted. * weed mount: fix thread-safety in swap file unit tests Use atomic.Uint32 to track failures within goroutines in TestSwapFile_NewSwapFileChunk_Concurrent to avoid unsafe calls to t.Errorf from multiple goroutines. * weed mount: simplify swap file creation logic Refactor the directory check and retry logic for better readability and to avoid re-using the main error variable for directory creation errors. Remove redundant error logging. * weed mount: improve error checking in swap file tests Explicitly check if NewSwapFileChunk returns nil to provide more informative failures. * weed mount: update DirtyPages interface to return error Propagate errors from SaveDataAt when swap file creation fails. This prevents potential panics in the write path. * weed mount: handle AddPage errors in write paths Update ChunkedDirtyPages and PageWriter to propagate errors and update WFS.Write and WFS.CopyFileRange to return fuse.EIO on failure. * weed mount: update swap directory creation error message Change "recreate" to "create/recreate" to better reflect that this path is also taken during the initial creation of the swap directory. --------- Co-authored-by: lixiang58 <lixiang58@lenovo.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
27af11f1e8 |
Revert "Revert "Merge branch 'master' into sub""
This reverts commit
|
||
|
|
0bb97709d4 |
Revert "Merge branch 'master' into sub"
This reverts commit |
||
|
|
9e5b1eef9b | avoid passe lock by value | ||
|
|
3852307e94 | renaming | ||
|
|
d4566d4aaa |
more solid weed mount (#4089)
* compare chunks by timestamp
* fix slab clearing error
* fix test compilation
* move oldest chunk to sealed, instead of by fullness
* lock on fh.entryViewCache
* remove verbose logs
* revert slat clearing
* less logs
* less logs
* track write and read by timestamp
* remove useless logic
* add entry lock on file handle release
* use mem chunk only, swap file chunk has problems
* comment out code that maybe used later
* add debug mode to compare data read and write
* more efficient readResolvedChunks with linked list
* small optimization
* fix test compilation
* minor fix on writer
* add SeparateGarbageChunks
* group chunks into sections
* turn off debug mode
* fix tests
* fix tests
* tmp enable swap file chunk
* Revert "tmp enable swap file chunk"
This reverts commit
|
||
|
|
ca49ef8ed8 | duplicated logic | ||
|
|
ea2637734a | refactor filer proto chunk variable from mtime to modified_ts_ns | ||
|
|
72d8a9f9a8 |
refactor(exclusive_locker): Interval readability batch of updates (#3668)
* refactor(filechunk_manifest): `localProcesed` -> `localProcessed` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor: `saveChunkedFileIntevalToStorage` -> `saveChunkedFileIntervalToStorage` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor: `SafeRenewInteval` -> `SafeRenewInterval` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor: `InitLockInteval` -> `InitLockInterval` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor: `RenewInteval` -> `RenewInterval` Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org> |
||
|
|
689b4ecdcc | remove unused collection and replication from upload result | ||
|
|
26dbc6c905 | move to https://github.com/seaweedfs/seaweedfs | ||
|
|
d65bb2c6df |
mount: file handle locks entry better
related to https://github.com/chrislusf/seaweedfs/issues/2952 |
||
|
|
f2f68f675e | write to disk during random writes, limiting total disk spaces used | ||
|
|
011a41b561 | add back writes to swap file when too many in memory chunks are used. | ||
|
|
c3792c8352 | remove dead code | ||
|
|
b9c2bff931 | clean up | ||
|
|
2b955c1713 | support read |