Commit Graph
3 Commits
Author SHA1 Message Date
b50d4a9cab mount: don't spend the close()/fsync commit deadline on uploads (#12641)
* mount: start the close()/fsync commit deadline after the data upload

Flush, Fsync and the truncate path of SetAttr create the 30 s
metadataFlushTimeout context on entry and pass it to doFlush, so the
clock also runs through FlushData, the upload of the dirty pages. That
upload has its own timeouts and retries. When an assign through a stalled
filer or a slow volume server takes most of the 30 s, CreateEntry then
fails at once with "context deadline exceeded" and close() returns EIO
for data that was already stored. The deadline is meant for the commit
alone (#10102: "a healthy CreateEntry completes in well under a second").

doFlush now takes the commit timeout and starts the deadline once the
data is uploaded. close() still cannot hang on a stuck filer: the commit
keeps its 30 s bound. Release and the flush before a rename pass 0, as
before (no overall bound, only the per-call filer timeouts).

Seen in production on a training cluster: a 30 s stall of one replica's
disk, and a filer that stalled for about 30 s, each failed close() with
EIO although the upload had succeeded.

Tests: close(), fsync() and a truncate below dirty data, each after an
assign slower than the deadline (all three fail before this change), and
a filer that never answers CreateEntry still fails close() at the
deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: charge the flush commit deadline only for its filer calls

Starting the 30 s deadline after the data upload is not enough.
flushMetadataToFiler does more than call the filer before it commits: it
waits for the handle's locks, reads the file back and uploads it again
when its chunks are bloated (maybeMergeChunks), and uploads a manifest
once enough chunks have built up (MaybeManifestize). None of that takes
the context. When it outlasts the deadline, CreateEntry again fails at
once with "context deadline exceeded", and close() or fsync() returns an
error for data that was stored.

The deadline becomes a budget, commitBudget, charged only while a filer
call of the commit runs: CreateEntry, and the read of a changed hard
link before it. The attempts of one flush share it, and a flush is not
retried once it is spent, so a stuck filer still fails close() after
30 s. Release, the flush before a rename and the async flush pass no
budget, as before.

Seen in production: the merge of a 1.8 MiB sqlite file with 183 chunks
took 32 s because one chunk's read first tried a volume server that was
down, and the commit failed 32 microseconds after the merge finished.

Tests: a flush whose manifest upload is slower than the deadline commits
(it fails without this change), and the filer calls of one flush share
the budget.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: give the stalled-lookup refusal test the new doFlush argument

TestContentFlushRefusalWithStalledLookup came in from master after this
branch changed doFlush's signature, so the merged package did not build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-13 21:56:44 -08:00
Chris LuandClaude Opus 5.5 652648d9c4 mount: merge concurrent content changes from another mount instead of failing or dropping them (#12596)
* mount: merge a refused content flush onto the stored entry

A content flush conditioned on the handle's base failed with EIO whenever a
peer had changed the file since, so two mounts appending to one file could
not both close it cleanly. On a refused condition, reread the entry and
replay this handle's changes since its base onto it: the chunks it added or
dropped, and the attributes, extended keys and inline content it changed.
A local truncation keeps its size and clips the peer's chunks to it;
otherwise the larger size wins. The merged entry is published against the
stored one, up to three times, and installed in the handle so its next
flush keeps the peer's chunks. A replaced or deleted file is still refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: merge a peer's content change into a dirty handle

A subscription event carrying a peer's content change replaced a dirty
handle's entry and discarded its unflushed pages, so the handle's writes
were lost and its close still returned OK. Fold the peer's entry into the
handle with the same merge a refused flush uses, keep the dirty pages, and
take the peer's entry as the new base. A replaced file still invalidates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: stop a flush merge once a filer accepts the write

withFilerClient can call the callback on several filers. A refusal from
one left its stored entry behind, so a later filer's success went on to
another merge round.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: refuse a merge that puts inline content next to chunks

A read inside the inline bytes returns them without looking at chunks,
so one side's inline content merged with the other side's chunks hid
data after a successful flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: refuse to merge a whole-file rewrite over a peer's chunk change

maybeMergeChunks re-uploads the handle's whole view as unstamped chunks.
Merged after a peer's change, they tie with a peer's own rewrite or bring
back a range the peer truncated, so that flush fails as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: clip peer chunks at the lowest truncate, not the final size

A handle that truncated and then wrote past the old end ends up larger
than its base, so the merge did not see the truncate and kept the peer's
chunks inside the range the truncate cleared. SetAttr now records the
lowest size truncated to over the current base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a peer's truncate when the handle left the size alone

The merge took the larger of the handle's and the stored size, so a peer
truncate came back as a zero-filled tail when the handle only overwrote
bytes inside it. The stored size now stands unless the handle changed
its size or wrote past it, and rebasing keeps only the growth writes made
after the flush was taken.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: retry a read view whose merged manifests did not resolve

A failed SetChunks leaves the chunk group answering every read with the
resolution error, and nothing re-ran it while the entry stayed the same.
The handle now marks such a view (on rebase and on SetEntry) and the
next read or lseek resolves the entry's chunks again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a dirty handle's writes when a peer change cannot merge

The new merge refusals sent a peer event on the same file down the old
path, which discarded the dirty pages and adopted the peer's entry, so
close returned OK without the writes. The handle now keeps its writes and
base, and the flush meets the conflict and fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep an unpublished truncate across a peer's base change

The truncate mark was tied to the base pointer, so a peer event that moved
a dirty handle's base (a merged content change or a metadata-only change)
dropped it, and a later merge on a regrown file kept peer chunks inside
the truncated range. Those base moves now carry the mark; a flush ack or a
discarded handle state still clears it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: wait for the flush before merging a shrink over buffered writes

A write inside the file leaves its bytes in dirty pages without moving
the size. Merging a peer's truncate then adopted the smaller size, and the
chunk uploaded later sat past it, unreadable through the handle. The event
path and the periodic metadata flush now refuse a merge that shrinks the
handle; the close flush uploads the pages first and merges then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: tell a shortened base chunk from a whole-file rewrite

A truncate shortens a base chunk in place, so it gets a new reference
key, and an old chunk can carry no timestamp. The rewrite refusal now
applies only to unstamped chunks whose file id is new to the base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check the creation time when the base has no inode

An entry without an inode or hard link id matched any replacement at the
same path, so a stale handle could merge into a different file. Without
an inode the merge now needs the same creation time. The filer keeps it
across updates; only a synthesized base has neither and still merges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a dirty handle's writes when a peer replaces the file

A peer entry with another inode at the handle's path went down the
fallback that destroys the dirty pages and installs the replacement, so
close returned OK without the writes. The handle now keeps its writes
and base like it does for an unmergeable change, and its flush refuses to
update the replacement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: build the content flush tests on freebsd

The invalidation tests, which every platform compiles, now use the
content flush helpers, so the freebsd cross-compile vet failed with
undefined: flushConditionEntry. Nothing in the file is freebsd-specific.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a dirty handle's base when a same-content file replaces it

A replacement with the base's chunks but another inode counted as a
metadata-only change, so the dirty handle adopted it as its base and the
next conditional flush overwrote the new file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: bring a peer's metadata change into a dirty handle

A metadata-only event moved a dirty handle's base but left its live
entry, so the next flush read the peer's chmod as a local edit and
published the old mode back. Replay the handle's own changes onto the
peer's entry instead, as for a content change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: wait for the flush before merging peer inline content over buffered writes

A buffered write is not a chunk yet, so a merge could install a peer's
inline content that later reads return ahead of the write's chunk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: track a whole-file chunk rewrite on the handle

The merge told a rewrite by its unstamped chunks, which a manifest fold
hides: manifests are unstamped either way. Record the rewrite against the
base it was made from, carried across peer base changes like a pending
truncate, and refuse the merge from that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: refuse a merge that shortens a manifest at the truncate

Clipping a manifest reference leaves its members whole, so peer bytes past
the handle's truncate could reappear once the file grew again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: flush a dirty handle after a peer replaces its file

The replacement test checked only the retained handle state. It now
flushes against a filer holding the replacement, including one with the
base's content, and checks the flush fails and leaves it unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test an append fsync after a peer's attribute-only change

A second mount's acknowledged chmod, chown, utimes or setxattr leaves
the content alone, so the writer's conditional content flush must still
publish its append and keep that change. On master the refused flush
surfaces as EIO from Fsync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:02:42 -08:00
c2b2ed769b fix(mount): preserve appended chunks during stale attribute updates (#12593)
* fix(mount): preserve acknowledged chunks during attribute updates

* mount: keep the WFS field block aligned

Adding the publication-lock fields inside the aligned block made gofmt
re-indent 25 unrelated lines. Put them in their own group after
dirIdleEvict, matching the enterprise layout, so syncs do not conflict there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: finish attribute saves through a FUSE interrupt

saveEntryAttributes derived its RPC context from the syscall's cancel
channel, so an interrupt while the conditional UpdateEntry was at the
owner returned EIO for an update the owner then applied, and skipped the
local install and retention. Keep the metadata deadline but drop the
cancellation, like the other metadata mutations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: map attribute CAS ids in place instead of cloning the entry

saveEntryAttributes cloned the whole entry three times per attempt only
to rewrite uid/gid for the filer, and each clone is as large as the
chunk list. Switch the ids in place and restore the exact local values
after the call, keeping one clone. The test now checks the reply carries
the local uid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: compare attribute updates against the filer's own uid/gid

With -map.uid/-map.gid, a filer id that is also a mapped local id reads
back unchanged and maps forward to another filer id. The IF_ENTRY_EQUAL
expected entry then never matched, so chmod, utimes and xattr changes on
such files failed with EAGAIN after three attempts, and an update would
have rewritten the owner. Keep the filer ids read with the entry for the
ids the syscall leaves alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mount): align attribute retention with read lock order

* mount: take the handle lock before entryLock in the attribute snapshot

attributeEntrySnapshot held entryLock while it queued for the shared
handle lock. A Read holding that lock waits on entryLock, the snapshot
waits behind a queued exclusive (an invalidation or a Write), and the
exclusive waits on the Read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mount): complete handle lock order and preserve explicit ID intent

* mount: re-read a refused attribute update from the file's owner filer

The filer checks the IF_ENTRY_EQUAL condition on the owner of the path's
write route key. The retry read any filer, so behind one that had not
applied the owner's latest update, every attempt sent the same stale base
and the syscall failed with EAGAIN. Read from the owner once the lock ring
is known, like lookupFilerEntry, but keep the filer's raw ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:28:47 -08:00