mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-15 08:18:42 +01:00
c819e3fbea269f8f33b8d219baa71f91052ced4e
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
b50d4a9cab |
mount: don't spend the close()/fsync commit deadline on uploads (#12641)
* mount: start the close()/fsync commit deadline after the data upload Flush, Fsync and the truncate path of SetAttr create the 30 s metadataFlushTimeout context on entry and pass it to doFlush, so the clock also runs through FlushData, the upload of the dirty pages. That upload has its own timeouts and retries. When an assign through a stalled filer or a slow volume server takes most of the 30 s, CreateEntry then fails at once with "context deadline exceeded" and close() returns EIO for data that was already stored. The deadline is meant for the commit alone (#10102: "a healthy CreateEntry completes in well under a second"). doFlush now takes the commit timeout and starts the deadline once the data is uploaded. close() still cannot hang on a stuck filer: the commit keeps its 30 s bound. Release and the flush before a rename pass 0, as before (no overall bound, only the per-call filer timeouts). Seen in production on a training cluster: a 30 s stall of one replica's disk, and a filer that stalled for about 30 s, each failed close() with EIO although the upload had succeeded. Tests: close(), fsync() and a truncate below dirty data, each after an assign slower than the deadline (all three fail before this change), and a filer that never answers CreateEntry still fails close() at the deadline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: charge the flush commit deadline only for its filer calls Starting the 30 s deadline after the data upload is not enough. flushMetadataToFiler does more than call the filer before it commits: it waits for the handle's locks, reads the file back and uploads it again when its chunks are bloated (maybeMergeChunks), and uploads a manifest once enough chunks have built up (MaybeManifestize). None of that takes the context. When it outlasts the deadline, CreateEntry again fails at once with "context deadline exceeded", and close() or fsync() returns an error for data that was stored. The deadline becomes a budget, commitBudget, charged only while a filer call of the commit runs: CreateEntry, and the read of a changed hard link before it. The attempts of one flush share it, and a flush is not retried once it is spent, so a stuck filer still fails close() after 30 s. Release, the flush before a rename and the async flush pass no budget, as before. Seen in production: the merge of a 1.8 MiB sqlite file with 183 chunks took 32 s because one chunk's read first tried a volume server that was down, and the commit failed 32 microseconds after the merge finished. Tests: a flush whose manifest upload is slower than the deadline commits (it fails without this change), and the filer calls of one flush share the budget. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * mount: give the stalled-lookup refusal test the new doFlush argument TestContentFlushRefusalWithStalledLookup came in from master after this branch changed doFlush's signature, so the merged package did not build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandClaude Opus 5.5
|
737ef68a11 |
mount: keep publishing an open file through its surviving hard link (#12609)
* mount: keep flushing an open file whose other hard link survives an unlink Unlink marks the inode's open handle deleted so no flush recreates the removed name. When the entry was one of several hard links the inode lives on under another name, yet the mark stayed and every later fsync and close returned OK without publishing metadata. Lift the unlink's own mark once the name is gone and the inode still has a path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: carry a link's stored entry into the file's open handle Link rewrites the source entry with the new hard link id, count and times, but an open handle kept its old base and entry. Its next conditional content flush was refused (EIO), and an unconditional one would have dropped the hard link id. Move the handle onto the stored entry when only the link fields and times changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: carry an alias unlink's stored entry into the open handle Removing one of several hard links lowers the stored link count and stamps a new ctime, and the handle may still hold the removed name. Look up the entry at the name the handle flushes through and adopt it, so the next content flush matches the stored entry, writes the right count, and attribute saves address the surviving name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test an open handle across link and alias unlink Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: read an alias unlink's lost entry again at the next flush When the read of the surviving entry right after an alias unlink fails, the handle kept a base with the old link count and ctime, and every later flush was refused even after the filer recovered. Leave the hard link id on the handle and have the next flush read the stored entry and adopt it before publishing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: publish a queued close before unlinking a name another link keeps With writeback caching, close only queues the flush. Unlink marked the handle deleted before waiting for that flush, so the flush skipped its metadata and the surviving link kept the content from before the close. When another link keeps the file, wait for the queued flush first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: let a queued close reach the filer before linking the file With writeback caching a new file is created on the filer by its queued close, and a listed directory serves the source from the cache, so a link right after close updated a name the filer did not have yet and failed with EIO. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: skip a link change no newer than the handle's entry A link change carried into the handle could replace a base already moved past it by a later acknowledgment. Use the same clock-domain check as installAckedEntry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep a link change's ctime in the dirty entry The flush does not restamp times, so a dirty entry carrying an earlier write's ctime moved the filer's ctime back from the link or unlink. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: flush through the file's current name when the resolved one is gone doFlush resolves the name before uploading the data. An unlink of that alias, or a rename, in the meantime sent the publish to a name the file no longer has. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep the pending link read when its name is already gone A flush could read the removed alias name, find nothing, and drop the pending read of the surviving link, leaving the stale base in place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test link and alias unlink across queued closes, flush races, versions and ctime Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: pick a flush's name again once its locks are held An unlink or rename that lands between the name pick and the locks would otherwise send the flush to the removed name, where the create guard fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test a flush waiting on its lock through an alias unlink Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: check the flush target after every lock acquisition A flush whose name kept changing published to the name chosen before its last lock wait. Check after each acquisition and return a retryable error once the attempts run out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test a flush whose name changes during every lock wait Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |