Commit Graph
4 Commits
Author SHA1 Message Date
Chris Lu bc06505b40 mount: keep metadata operations working on an unlinked open file (#10989)
* mount: serve metadata ops from the open handle of an unlinked file

ftruncate on a descriptor whose file was unlinked failed with ENOENT:
maybeReadEntry resolved the inode to a path first, and unlink had already
dropped it. GetAttr worked around that with its own handle fallback;
SetAttr and the xattr handlers had none.

Look the handle up first and let it answer whether or not a name still
points at the inode. GetAttr keeps reporting nlink 0 there, now off the
empty path.

Claude-Session: https://claude.ai/code/session_01U1R8BM4bVT46KwPDEj2Ega

* mount: read an open handle's attributes under the handle lock too

GetAttr held only the LockedEntry lock, which covers the async uploader's
chunk appends but not Write or the metadata flush: those rewrite size,
times and the whole chunk slice under the handle lock, so FileSize could
walk a slice mid-reassignment. The branch this replaced took both locks;
take both here, outer handle lock first, as Read and Lseek do.

Claude-Session: https://claude.ai/code/session_01U1R8BM4bVT46KwPDEj2Ega

* mount: report nlink 0 from SetAttr for an unlinked open file

The kernel caches the attributes a SETATTR reply carries, so an ftruncate
on an unlinked file left fstat reporting nlink 1 until the cache expired,
even though GetAttr had it right. Both replies go through the same rule.

Claude-Session: https://claude.ai/code/session_01U1R8BM4bVT46KwPDEj2Ega
2026-08-27 16:22:25 -07:00
Chris Lu 1ddec72707 Recover from a dead volume server on the mount read path (#10798)
* mount: re-resolve volume locations after a failed chunk read

NewChunkGroup passed nil as the ReaderCache's CacheInvalidator, so
retryFetchAfterCacheInvalidation was dead code on the FUSE read path. A
mount that cached a volume's locations while one server was down kept
retrying that server after it died, then returned EIO, even though the
master and filer both resolved the live replica. The S3 gateway already
passes its filerClient; do the same for the mount.

* test: FUSE integration tests for volume server failover

One mount appends while a second tails, and a volume server is killed,
started or restarted mid-stream against a 001-replicated cluster of three
volume servers. Automates the scenario matrix reported for Docker Swarm
mounts, including the large-file variant and a no-chaos control.

* test: report the filer's own view when append content mismatches

A mismatch between what the writer wrote and what the reader sees can come
from either side's cache. Read the file back through the filer's HTTP
handler as well, and let the mount verbosity be raised from the
environment, so a failing run says which layer lost the data.

* test: wait for the reader mount to converge before comparing

A mount caches metadata for about a second, so reading the file the instant
the writer's last close returned can legitimately come back short. Poll the
reader until it matches or the timeout expires; content that is wrong rather
than merely late never converges and still fails, now with the writer's
mount and the filer's own view alongside it.

* test: detect a failover cluster child that exited at startup

Signal(0) succeeds for a zombie and nothing reaped these children until
shutdown, so a process that died on startup looked alive until the readiness
timeout expired. Reap each child as it is started and consult the result.

* test: read a file the killed volume server actually holds

Placement decides which two of three servers back each volume, so killing
volume N and reading readfile-N could pass without the victim ever holding a
replica of it. Resolve each file's volumes through the filer and the master,
and pick one the victim backs, preferring a file the reader has not cached.

* ci: stop persisting checkout credentials in the failover workflow

The job does not use the token after cloning. Also tag the README's command
block as bash and match the timeout the workflow actually uses.

* test: discard the ignored errors errcheck flags in the failover harness

* test: resolve manifests when mapping a file to its volumes

A manifest chunk's own fid names the volume holding the manifest, not the
volumes holding the data, so a large enough file would point the failover
victim at the wrong server.

* test: pin the stale-location recovery path with a primed reader

Reading a file for the first time after a server dies proves nothing: the
lookup is fresh and returns the survivor. Kill one holder and wait for the
master to drop it, read a file on that volume so the reader caches the lone
survivor, restart the first server, then kill the survivor. The reader's only
cached location is now dead while the data is live elsewhere, which is the
case the invalidator exists for: EIO without it, recovery with it.
2026-08-17 17:30:33 -07:00
Chris Lu 214d3599d3 windows mount: cache file data, resolved paths and attributes (#10703)
* benchmark tool for mounted filesystems

* ci: on-demand mount benchmark, native WinFsp vs rclone plus a Linux reference

* windows mount: let the Windows cache manager cache file data

WinFsp only turns the cache manager on for a file when FileInfoTimeout
is infinite; at any finite value every application read and write is a
synchronous trip into the mount process at whatever size the application
issued. Metadata events already reach FspFileSystemNotify, which purges
a changed file's cached pages and attributes, so an infinite timeout
stays coherent. The dir listing, volume info and EA timeouts are pinned
to one second so they do not silently inherit the infinity.

* windows mount: cache resolved paths and attributes in the adapter

WinFsp addresses every operation by path and has no FORGET, so the
adapter walked the whole path through Lookup on each one, and in a
directory the filer has not listed yet every walk was a filer round
trip; nothing played the part of the kernel's dentry and attribute
caches. The path cache owns one lookup reference per entry the way the
kernel holds one until FORGET, serves attribute reads for files without
an open handle, and is purged by the mount's own mutations and by
metadata events, with the timeout as backstop.

* windows mount: keep a closed file's attributes cached

Open steals the path's cache entry for its handle and Release returned
the reference with a purge, so the stat that follows every copied file
walked to the filer again. Reading the handle's final attributes before
it goes away and moving the reference back into the cache serves that
stat locally, the way the kernel's attribute cache does after a close.

Only if the path still names that inode, though: WinFsp reports the
path the handle opened with, and after a delete-on-close or a rename
caching it would resurrect an entry that is gone.

* windows mount: persist entries at create, and let the flush stay at close

WinFsp posts the cleanup and close that carry the flush after
CloseHandle has returned, so deferring the filer entry to the flush let
everything that reads through the filer race an unflushed close: a
listing missed just-written files, and a directory rename moved a
directory on the filer before its newest child existed there, leaving
the straggler flush to recreate the child under the dead path.

Flush-at-cleanup is not the answer either: it makes every handle's
cleanup flush, and those flushes race the unlinks of delete-on-close,
re-inserting the entry the unlink just removed. Persisting the entry at
create takes the ordering question away.

* mount: flush written pages before a truncate shrinks past them

The shrink trims chunks, but written pages that have not become chunks
yet are invisible to it, so the next flush wrote them back and the file
grew again, resurrecting the truncated bytes. Windows hits this on
every write-then-shrink because its flush runs after CloseHandle, but
the gap is platform-neutral.

* mount: order a file's unlink against its in-flight flush

Unlink set the handle's deleted flag bare, so a flush already past its
own check of that flag wrote the entry back right after the delete
removed it, and a delete-on-close file outlived its last handle. The
flag is now set under the handle's flush lock and re-checked under it,
so a flush either completes before the delete or sees the flag and
skips. An eagerly created handle also starts clean: the dirty mark
existed to make the deferred filer create happen at flush, and eager
creates have nothing to flush.
2026-08-10 18:46:18 -07:00
Chris Lu 68cae26c0b mount: fix SetAttr/GetAttr crash from concurrent chunk append under writebackCache (#9667)
* mount: hold the entry lock while reading chunk size in GetAttr/SetAttr

Async upload workers append chunks to an open handle's shared entry under
the LockedEntry lock (FileHandle.AddChunks), but GetAttr and SetAttr
computed FileSize by iterating entry.Chunks without taking it. A concurrent
append that reallocated the backing array tore the slice read and crashed in
filer.TotalSize. Surfaces with -writebackCache, where handles stay open and
flush asynchronously while metadata ops keep arriving.

Take the LockedEntry lock for those reads (and SetAttr's truncate rewrite).

* mount: re-read entry under the lock in GetAttr/SetAttr

If SetEntry swapped the handle's entry pointer between maybeReadEntry and the
lock acquisition, the old pointer is orphaned. Re-read fh.entry.Entry under
the lock so SetAttr mutates the live entry instead of losing the update, and
GetAttr reports the current one.

* mount: cover the truncate path in TestAttrChunkRace

Alternate SetAttr between mtime-only and a shrinking size so the test also
exercises the entry.Chunks rewrite under fh.entry.Lock, not just the read-side
size walk.

* mount: snapshot chunks under the entry lock on the read path

readFromChunks holds fh.entryLock (excludes SetAttr) but not the LockedEntry
lock the async uploader appends under, so IsInRemoteOnly, the FileSize
fallback, and the RDMA/peer chunk walks read entry.Chunks while AddChunks
reallocated it — the same torn-slice crash as GetAttr/SetAttr.

Snapshot size, inline content, and the chunk list under a brief LockedEntry
RLock, then hand the snapshot to the RDMA/peer helpers instead of holding the
lock across network I/O. The captured slice stays valid: append never mutates
the old backing array, and truncate is excluded by the fh.entryLock.
2026-05-24 23:49:41 -07:00