* mount: drain OFD locks when a file description is released
The kernel scopes OFD fcntl locks to the open file description but
reports no lock_owner for them at RELEASE — it only sends one for flock
holders — so an OFD write lock taken on a file handle outlived the
holder and blocked every later open of that file (e.g. etcd failing to
reopen its WAL after a restart). The mount now records the owner of each
granted non-flock lock under its file handle and releases those owners
on RELEASE, which also covers any POSIX locks left over when FLUSH could
not run.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: give every open its own fh value over the shared handle
AcquireFileHandle reuses one FileHandle per inode, so every open reported
the same fh to the kernel. A RELEASE for one description could then not
be told apart from a sibling still holding the inode open — draining
lock owners recorded under the shared fh dropped locks belonging to
descriptions that stay open.
Each open now registers its own fh value in fh2inode, all resolving to
the shared FileHandle, and RELEASE removes exactly the id that is
closing. This also stops a duplicated RELEASE on one id from
double-decrementing the open count.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: forget flushed lock owners so a later release cannot re-drop locks
POSIX lock owners are process-scoped: Flush releases every lock the
process holds on the inode, but the owner stayed recorded under each
open's fh for the Release-time drain. If the process took a fresh lock
through another open before an old description finished closing, the
stale record made that RELEASE drop the new lock too.
Flush now drops the released owner from every recorded open of the
inode, so the Release drain only ever fires for owners whose locks were
never flushed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: keep close-time lock owner records when the release fails
A routed POSIX owner release that errors keeps the flush hint for a
retry, but Flush also dropped the owner from every open's release
records unconditionally — so the final Release skipped the retry and
the owner filer's lock was stranded until lease reaping. Only forget
the owner once its locks were actually released.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: release flock locks before the close-time flush
Release ran the dirty-handle flush before dropping the file
description's flock locks. A flush parked on a dead filer kept a dead
holder's lock in the table, so a restarted process taking flock on the
same file saw "file already locked" until the mount restarted.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: finish the blocked-flush release test deterministically
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: let the release test's parked flush talk to a live filer
Marking the handle deleted raced with the flush's unlocked early read
of isDeleted. Pointing the fixture at a live fake filer lets the flush
complete on its own once the handle lock frees, no flag write needed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: propagate request context through metadata lookups
The FUSE interrupt channel never reached the filer RPC: lookupEntry ran
its LookupEntry under context.Background(), so a stalled filer parked
ops like ListXAttr forever even after the kernel gave up on them.
Thread a context from each op's cancel channel through
maybeReadEntry, maybeLoadEntry, lookupEntry, ensureDirectoryVisited and
waitForPendingAsyncFlush; the existing withFilerClient attempt bound
turns it into both an interrupt and a deadline. A caller cancellation
now surfaces as EINTR instead of waiting out the RPC.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: reuse ctxFromCancel for the read path's interrupt bridge
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: regression tests for canceled and stalled filer lookups
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: thread ctx through AcquireHandle
Open, Create and Mknod all carry the request's cancel channel; let it
bound the pending-flush wait and metadata load inside AcquireHandle too.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: keep flush waits and write sequences uninterruptible
The cancel-derived context must only cover metadata reads: an
interrupted Unlink or Rename that skips the pending-flush wait can
race the async write back into existence, and Link/Rename mutations
that run under a canceled context abort mid-sequence, strand the
rollback, and leave wrong hard-link counts. Restore the blocking flush
wait and run each op's write phase on context.Background(). Rmdir and
Unlink also propagate EINTR from the parent lookup instead of
proceeding past a permission check the interrupt prevented.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: interrupt shared directory builds and directory reads
Waiters behind another request's singleflight directory build blocked
until it finished even when interrupted; switch the wait to DoChan so
the request context can abandon it. ReadDir paths now carry the
request context through EnsureListingFresh, cache listings and direct
reads, and map its cancellation to EINTR rather than EIO.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: assert exact statuses in filer-stall regression tests
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: let read-side flush waits release on interrupt
waitForPendingAsyncFlush reports whether the flush actually finished:
lookupEntry and AcquireHandle return EINTR when the request is
abandoned mid-wait, while Unlink and Rename keep waiting on
context.Background() so the delete/rename still sees the flushed
state.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: propagate interrupt through remaining lookup checks
Rename's source and destination permission lookups now return EINTR
on interrupt instead of skipping checkStickyBit or masking an
existing target, and the readdir preload visit maps cancellation to
EINTR like the other branches.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: fail closed on errored rename prerequisite lookups
* mount: stop rmdir on interrupted target lookup, fail closed on dir lookup errors
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* operation: bound AssignVolume with a deadline
AssignVolume ran on context.Background(), so when the filer is overwhelmed
the RPC could block indefinitely and wedge every caller holding the
connection. Give it a 30s deadline so a stuck assign fails and the caller's
retry/error path runs instead of hanging forever.
* mount: abort flush when the FUSE request is interrupted
On close(), a killed process blocks in fuse_flush waiting for the mount to
answer. doFlush ran its metadata CreateEntry on context.Background() and
ignored the kernel interrupt channel, so against an overwhelmed filer the
flush never completed and the process stayed in uninterruptible sleep --
making the pod un-killable.
Derive a context from the FUSE cancel channel in Flush/Fsync and thread it
through doFlush -> flushMetadataToFiler -> streamCreateEntry; the retry loop
stops as soon as the context is cancelled. Release and the pre-rename flush
keep a non-cancellable context since they must finish regardless.
* operation: harden the AssignVolume timeout test
Make the test double's signal send non-blocking and bound the receive with a
timeout so a regression can't wedge the test instead of failing it.
With -dlm, GetLk/SetLk/SetLkw and the flush/release cleanup paths go to
the inode's owner filer via the PosixLock RPC instead of the local table,
so flock/fcntl are honored across mounts. Advisory locking rides the same
switch as whole-file write coordination — and is therefore off under
writeback cache, which implies single-writer. Keys are the inode identity
(HardLinkId else path); SetLkw is client-side polling with the FUSE cancel
channel (no server wait queue); a per-mount session id namespaces owners;
a local hint avoids a release RPC on every close. Background unlock/release
RPCs are bounded so a stuck filer can't hang close().
* fix(mount): flush dirty handles on Release when kernel skipped Flush
The FUSE protocol allows the kernel to send Release without a preceding
Flush; file handles that reach Release with dirtyMetadata=true (notably
deferred creates that never saw any write) would then have their pending
filer CreateEntry dropped on the floor, leaving the mount and filer out
of sync.
Detect dirty handles in Release and call doFlush before tearing the
handle down. Skip the fallback when an async flush is already pending so
we don't double-submit. Flock-unlock Releases stay on the synchronous
path so close()-time serialization is preserved.
Adds TestReleaseFlushesDirtyCreateIfFlushWasSkipped covering the
create-without-flush path.
* address review: drop racy dirty-flag peek, let doFlush self-gate
fh.dirtyMetadata / fh.asyncFlushPending are written from the periodic
metadata flusher and async flush worker under fhLockTable, so the
unsynchronized read in Release was a data race per the reviewer.
Just call doFlush unconditionally on every Release; it already fast-
paths the clean case (dirtyPages.FlushData early-returns when hasWrites
is false, and the dirty-metadata branch short-circuits), so the extra
call after a normal Flush is cheap while the no-Flush-before-Release
path still recovers a deferred create.
* feat(mount): set FOPEN_KEEP_CACHE when file mtime is unchanged
On re-open of an unmodified file, signal the kernel to preserve its
existing page cache. This eliminates redundant volume server reads for
workloads that repeatedly open-read-close the same files (build systems,
config readers, etc.).
* fix(mount): use guarded type assertion for openMtimeCache load
Use the two-value form of type assertion when loading from sync.Map
to prevent potential panics if a non-int64 value is ever stored.
* fix(mount): skip redundant mtime store and invalidate on truncation
- Avoid redundant sync.Map Store when cached mtime already matches
the current mtime, reducing contention on the hot open path.
- Invalidate openMtimeCache in SetAttr when file size changes
(truncation), preventing stale kernel page cache after ftruncate.
* fix(mount): use nanosecond mtime precision and bounded cache for FOPEN_KEEP_CACHE
- Compare both Mtime (seconds) and MtimeNs (nanoseconds) to detect
sub-second modifications common in automated workloads.
- Replace unbounded sync.Map with a bounded map + mutex (8192 entries,
random eviction when full), following the existing atimeMap pattern.
- Extract applyKeepCacheFlag and invalidateOpenMtimeCache methods for
clarity and testability.
- Add tests for nanosecond precision and cache eviction.
* fix(mount): invalidate mtime cache in truncateEntry for O_TRUNC consistency
Add invalidateOpenMtimeCache call to truncateEntry so the Create path
with O_TRUNC follows the same explicit invalidation pattern as SetAttr
and Write.
* mount: implement create for rsync temp files
* mount: move access implementation out of unsupported
* mount: tighten access checks
* mount: log access group lookup failures
* mount: reset dirty pages on truncate
* mount: tighten create and root access handling
* mount: handle existing creates before quota checks
* mount: restrict access fallback when group lookup fails
When lookupSupplementaryGroupIDs returns an error, the previous code
fell through to checking only the "other" permission bits, which could
overgrant access. Require both group and other permission classes to
satisfy the mask so access is never broader than intended.
* mount: guard against nil entry in Create existing-file path
maybeLoadEntry can return OK with a nil entry or nil Attributes in
edge cases. Check before dereferencing to prevent a panic.
* mount: reopen existing file on create race without O_EXCL
When createRegularFile returns EEXIST because another process won the
race, and O_EXCL is not set, reload the winner's entry and open it
instead of propagating the error to the caller.
* mount: check parent directory permission in createRegularFile
Verify the caller has write+search (W_OK|X_OK) permission on the
parent directory before creating a file. This applies to both
Create and Mknod. Update test fixture mount mode to 0o777 so the
existing tests pass with the new check.
* mount: enforce file permission bits in AcquireHandle
Map the open flags (O_RDONLY/O_WRONLY/O_RDWR) to an access mask and
call hasAccess before handing out a file handle. This makes
AcquireHandle the single source of truth for mode-based access
control across Open, Create-existing, and Create-new paths.
---------
Co-authored-by: Copilot <copilot@github.com>
* Add FUSE integration tests for POSIX file locking
Test flock() and fcntl() advisory locks over the FUSE mount:
- Exclusive and shared flock with conflict detection
- flock upgrade (shared to exclusive) and release on close
- fcntl F_SETLK write lock conflicts and shared read locks
- fcntl F_GETLK conflict reporting on overlapping byte ranges
- Non-overlapping byte-range locks held independently
- F_SETLKW blocking until conflicting lock is released
- Lock release on file descriptor close
- Concurrent lock contention with multiple workers
* Fix review feedback in POSIX lock integration tests
- Assert specific EAGAIN error on fcntl lock conflicts instead of generic Error
- Use O_APPEND in concurrent contention test so workers append rather than overwrite
- Verify exact line count (numWorkers * writesPerWorker) after concurrent test
- Check unlock error in F_SETLKW blocking test goroutine
* Refactor fcntl tests to use subprocesses for inter-process semantics
POSIX fcntl locks use the process's files_struct as lock owner, so all
fds in the same process share the same owner and never conflict. This
caused the fcntl tests to silently pass without exercising lock conflicts.
Changes:
- Add TestFcntlLockHelper subprocess entry point with hold/try/getlk actions
- Add lockHolder with channel-based coordination (no scanner race)
- Rewrite all fcntl tests to run contenders in separate subprocesses
- Fix F_UNLCK int16 cast in GetLk assertion for type-safe comparison
- Fix concurrent test: use non-blocking flock with retry to avoid
exhausting go-fuse server reader goroutines (blocking FUSE SETLKW
can starve unlock request processing, causing deadlock)
flock tests remain same-process since flock uses per-struct-file owners.
* Fix misleading comment and error handling in lock test subprocess
- Fix comment: tryLockInSubprocess tests a subprocess, not the test process
- Distinguish EAGAIN/EACCES from unexpected errors in subprocess try mode
so real failures aren't silently masked as lock conflicts
* Fix CI race in FcntlReleaseOnClose and increase flock retry budget
- FcntlReleaseOnClose: retry lock acquisition after subprocess exits
since the FUSE server may not process Release immediately
- ConcurrentLockContention: increase retry limit from 500 to 3000
(5s → 30s budget) to handle CI load
* separating flock and fcntl in the in-memory lock table and cleaning them up through the right release path: PID for POSIX locks, lock owner for flock
* ReleasePosixOwner
* weed/mount: flush before releasing posix close owner
* weed/mount: keep woken lock waiters from losing inode state
* test/fuse: make blocking fcntl helper state explicit
* test/fuse: assert flock contention never overlaps
* test/fuse: stabilize concurrent lock contention check
* test/fuse: make concurrent contention writes deterministic
* weed/mount: retry synchronous metadata flushes
* Add POSIX byte-range lock table for FUSE mount
Implement PosixLockTable with per-inode range lock tracking supporting:
- Shared (F_RDLCK) and exclusive (F_WRLCK) byte-range locks
- Conflict detection across different lock owners
- Lock coalescing for adjacent/overlapping same-owner same-type locks
- Lock splitting on partial-range unlock
- Blocking waiter support for SetLkw with cancellation
- Owner-based cleanup for Release
* Wire POSIX lock handlers into FUSE mount
Implement GetLk, SetLk, SetLkw on WFS delegating to PosixLockTable.
Add posixLocks field to WFS and initialize in constructor.
Clean up locks on Release via ReleaseOwner using ReleaseIn.LockOwner.
Remove ENOSYS stubs from weedfs_unsupported.go.
* Enable POSIX and flock lock capabilities in FUSE mount
Set EnableLocks: true in mount options to advertise
CAP_POSIX_LOCKS and CAP_FLOCK_LOCKS during FUSE INIT.
* Avoid thundering herd in lock waiter wake-up
Replace broadcast-all wakeWaiters with selective wakeEligibleWaiters
that checks each waiter's requested lock against remaining held locks.
Only waiters whose request no longer conflicts are woken; others stay
queued. Store the requested lockRange in each lockWaiter to enable this.
* Fix uint64 overflow in adjacency check for lock coalescing
Guard h.End+1 and lk.End+1 with < ^uint64(0) checks so that
End == math.MaxUint64 (EOF) does not wrap to 0 and falsely merge
non-adjacent locks.
* Add test for non-adjacent ranges with gap not being coalesced
* Enable writeback_cache and async_dio FUSE options
Fixes#7978
- Update mount_std.go to use EnableWriteback and EnableAsyncDio from go-fuse
- Add go.mod replace directive to use local go-fuse with capability support
- Remove temporary workaround that disabled these options
This enables proper FUSE kernel capability negotiation for writeback cache
and async direct I/O, improving performance for small writes and concurrent
direct I/O operations.
* Address PR review comments
- Remove redundant nil checks for writebackCache and asyncDio flags
- Update go.mod replace directive to use seaweedfs/go-fuse fork instead of local path
* Add TODO comment for go.mod replace directive
The replace directive must use a local path until seaweedfs/go-fuse#1 is merged.
After merge, this should be updated to use the proper version.
* Use seaweedfs/go-fuse v2.9.0 instead of local repository
Replace local path with seaweedfs/go-fuse v2.9.0 fork which includes
the writeback_cache and async_dio capability support.
* Use github.com/seaweedfs/go-fuse/v2 directly without replace directive
- Updated all imports to use github.com/seaweedfs/go-fuse/v2
- Removed replace directive from go.mod
- Using seaweedfs/go-fuse v2.0.0-20260106181308-87f90219ce09 which includes:
* writeback_cache and async_dio support
* Corrected module path
* Update to seaweedfs/go-fuse v2.9.1
Use v2.9.1 tag which includes the corrected module path
(github.com/seaweedfs/go-fuse/v2) along with writeback_cache
and async_dio support.
