3 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandDevin	9e41500a3a	mount: rebuild dropped directory listings and stale sections on lookup (#12195) ... * mount: rebuild dropped directory listings on lookup Invalidation, idle eviction, and a kernel Forget all clear a directory's cached-listing flag, but only an expired listing was rebuilt on lookup — a dropped one read through per entry until a ReadDir happened to run. Track directories that once held a listing in wasCachedDirs, which survives those drops, and let ShouldRebuildExpiredDir rebuild them on the next lookup. Never-cached and read-through directories keep reading through. * mount: re-validate a stale directory section on lookup A burst of remote changes invalidates only the directory section it lands in, but nothing re-validates that section until a ReadDir runs — so a lookup-heavy workload reads through per entry indefinitely. EnsureNameFresh re-lists just the section covering the looked-up name, after which lookups in it are served from the cache again. * mount: wait for pending flush before a stale-section refresh on lookup A section refresh lists the range from the filer; without the wait a pending async flush of the looked-up entry could land after the listing and leave the refreshed section holding pre-flush metadata — the same hazard the rebuild path already waits out. * mount: bound the dropped-listing rebuild history wasCachedDirs grew one full path per directory ever cached, removed only by the read-through mark, so deleted and forgotten directories accumulated forever. Remove the record when its name is deleted, move it on rename so rebuild eligibility travels with the directory, and cap the history with FIFO eviction — past the bound an evicted directory reads through per entry, the same end state as a never-cached one. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: drop oversized stale sections and back off failed refreshes A section that outgrew one refresh failed EnsureNameFresh with ErrRefreshRangeTooLarge on every lookup, repeating the multi-page listing before each per-entry read, and other refresh failures retried with no cooldown. Drop the cached listing on ErrRefreshRangeTooLarge so a later lookup rebuilds it re-tiled, as ReadDir does, and stamp the rebuild attempt on other failures so retries back off under the same cooldown the rebuild gate uses. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: version hard-link sibling syncs at the committed log position * mount: tag dropped-listing queue slots so stale ones cannot evict RemovePath, MovePath, and MarkDirectoryReadThrough delete the rebuild record but leave the FIFO slot behind, so a path removed and re-cached holds two queue entries. When the stale slot reaches the front it evicts the newer record, and the directory reads through per lookup until the next listing. Tag each record with an insertion generation and only evict on a match, keeping the newer record for its own slot's turn. * mount: refresh hard-link siblings from the filer, not the cached copy Stamping a cached sibling with the mutation's version claimed that position for content that may predate it — an in-flight event with newer chunks or attributes then hits the version gate and the sibling serves stale data until a refresh. Fetch each cached sibling's entry from the filer after the commit and record the higher of the fetched and committed positions, falling back to the in-place counter patch only when the fetch itself fails. * mount: back off failed section refreshes per section, not per directory A failed refresh of one stale section marked a directory-wide attempt, so every other stale section in that directory waited out the cooldown with per-entry filer reads. Key the attempt timestamp by the section's lower bound so unrelated ranges still refresh on their own schedule. * mount: fence hard-linked entries by the group's newest write position Every path of a hard-link group reads one shared content blob, but the version gate only tracks per-path records — a sibling absent from inodeToPath got no version from the sibling sync, so a stale section refresh or event could roll the shared blob back for all of them. Record a monotonic version floor per hard-link id wherever versioned content lands, and check it in the section-refresh and event gates. * mount: keep hard-link names landing when the group floor fences content The per-group version floor rejected a delayed event's newEntry while its oldPath still applied — a rename could drop the source without adding the destination, and a section refresh could skip then mark a live link fresh. Rebase the group's current cached content onto the new path so the name lands without rolling the shared blob back, only skip refresh entries already cached, and gate the sibling refresh write under the same lock so a racing event or delete wins. * mount: rebase hard-link entries from the shared blob when no sibling is cached Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: park group content on uncached hard-link names a stale refresh installs Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: stale the destination section when a fenced hard-link name cannot land Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: bound the sibling hard-link refresh under one request deadline Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: replace a reused-name occupant the hard-link floor would skip * mount: claim no version for a sibling counter patched without a fetch * mount: vacate a reused-name occupant without decrementing its group A sibling write can fold the occupant's filer-side unlink into the shared blob counter while the stale entry is still cached, so replacing it through the plain insert may decrement a group that no longer counts the name and drop a blob live siblings still read. Vacate the path like a rename first; the counter then only ever drifts high, which leaks a blob rather than losing shared content. * mount: keep a fetch-failed sibling's version when patching its counter The fallback marked the entry unversioned, which fences nothing — a replayed delete older than the cached content could remove a live link. Claim the version the entry already holds instead: newer in-flight events still land and correct it, older replays stay fenced. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 00:55:47 -07:00
Chris Lu	8dcdb70594	mount: let a rename remove its source at the source's own version (#10973) ... * mount: let a rename remove its source at the source's own version A rename stamps the source and takes the name away at the same log position, so the removal reaches the meta cache carrying exactly the version the source already records. The version gate read that as a write already reflected and dropped it, while the destination half of the same event still applied -- the source stayed cached beside the destination, and readdir and stat went on serving a name the filer no longer had: gate dropped removal of /winfsp-test-TestRenameOverExisting/src eventTs=1787761708173717200 record=1787761708173717200 floor=1787761708173717200 tombstone=false A removal asks a different question from a write. An entry still present at exactly that version has the write reflected but not its removal, so only a strictly newer record fences one out; a tombstone is the removal already reflected and goes on fencing as before. * mount: sweep a section's vanished name recorded at the snapshot The refresh deletes the names its listing did not return, but asked the gate whether a write at the snapshot was reflected. A name recorded at exactly that version has the write reflected and not its removal, so it survived the sweep and stayed cached until some later event happened to touch it. Same reading as the rename source a commit earlier: the call site removes, so it asks about a removal.	2026-08-26 10:20:38 -07:00
Chris Lu	65114575eb	mount: invalidate hot directory listings by section (#10712) ... * mount: invalidate hot directory listings by section A cached directory used to be dropped whole when it saw 64 changes in 2s: with a continuous writer the listing cycled through wipe, direct listing and full rebuild for as long as the writer kept going, and every sibling lookup fell through to the filer in between. Split each cached listing into name-range sections of 1024 entries. A burst of foreign changes invalidates just the section it lands in; entries stay served and events keep applying, and the next readdir re-lists only that range from the filer, reconciled through the version gate so it cannot roll back newer applied events. Lookups in an invalidated section read through until then. The mount's own writes no longer invalidate anything: they are ground truth for its cache. * meta_cache: drop the version floor with a deleted or moved directory The other teardown paths already clear both maps; a floor left behind here would fence the listing of a directory re-created at the same path. * mount: harden section refresh An unversioned listing (pre-upgrade filer) now only fills gaps instead of reconciling: without a snapshot to order against, an overwrite or the deletion sweep could roll back an event applied after the listing. The section table can be rebuilt or re-split between the listing and its apply, so the refresh only marks fresh or splits when the section still covers the range it read. Splicing bounds from a stale range into a rebuilt table could leave them unsorted. Bound the wait: a readdir gives a refresh five seconds before serving the maintained-but-unverified cache. Bound the size: a range grown past four sections aborts the refresh and drops the directory cache, re-tiling it with a full rebuild, with that request served direct. Cover the filer-facing path with a listing server: paging with the snapshot pinned across pages, the section cutoff, no calls for a fresh section, and the overgrown-range abort. * meta_cache: make the section table a self-contained state machine Churn counting, freshness, stale-range scanning and the refresh completion with its guard and split now live on dirSections itself, free of the lock, the store and the apply loop, so they test directly with synthetic clocks and tables. MetaCache keeps thin wrappers that hold its mutex and find the directory's table. * meta_cache: keep section internals out of the apply request The request now carries the completed build's table and one refresh as opaque values built by section code, and the boundary-derivation rule moves out of the build loop into a collector next to the rest of the section logic. * mount: fence refreshed sections with a snapshot floor A refresh versioned the entries it fetched and tombstoned the ones it swept, but a name absent from both cache and listing kept the old directory floor, so a delayed event between the two snapshots could resurrect it into a section already marked fresh. The section now carries its own floor, consulted next to the directory floor, covering every name in the range, present or absent — which also retires the refresh's per-entry version stamps and sweep tombstones. An unversioned listing sets no floor and vouches for nothing: it may still fill gaps, but the section stays stale and reads through until a filer that stamps snapshots re-validates it. A listing's reach is unknowable up front — a resumed handle can skip far ahead, and shrunken sections let one batch span many — so a readdir now re-validates every stale section from its start name to the end of the directory instead of the next two. * mount: fence tombstoned names with floors and gate the reconcile A tombstone answered for its name before the floors were consulted, so one at an old position let through events the newer listing floor should have fenced; a build never hit this because it prunes superseded tombstones, which a section refresh does not. The version gate now raises a tombstone to the floors like any other record. With no per-entry versions, only the section floor fences a reconcile's work, so a range the rebuilt or re-split table no longer has must not touch the store either: the range check moves ahead of the mutations, under the same lock the floor install holds. An unversioned refresh no longer retries: the section is remembered as unverifiable and skipped by the stale scan, or every batch of every readdir would re-list the same ranges against a filer that cannot vouch for them. * mount: clear beaten unversioned markers and skip refresh mid-build An unversioned marker outliving the snapshot write that replaced its content bypassed the section floor the same way an old tombstone did, letting a delayed pre-snapshot event roll the entry back. The refresh now clears the marker when its write wins; pinned local-only entries are not replaced at all, keeping their content and marker. A rebuild wipes and repopulates the store off the apply loop, so a refresh reconciling meanwhile could sweep children the build had already inserted and let it publish the directory incomplete. The refresh now skips a building directory, as events (buffered) and purges (skipped) already do; its staleness dies with the build's fresh table. * mount: clear the unversioned marker only after its replacement lands Clearing before the insert meant a failed write left the old local content claiming the listing floors, fencing the very events that were still entitled to correct it. * meta_cache: rename the section state machine to sectionList dirSections named both the type and the map of them. * mount: raise the default cacheDirMaxEntries to 100000 The low ceiling guarded against whole-listing rebuild churn: a big cached directory under writes kept re-streaming everything. Sectioned invalidation ended that — a burst now costs one range listing — so the remaining cost of caching a large directory is its one-time build, comparable to the single direct listing that read-through mode pays on every enumeration instead. * meta_cache: cover section border and edge cases A bound-named entry belongs to the section starting at the bound: the neighboring refresh's sweep stops before it, its own section's covers it. Churn past everything the build saw lands in the tail section, a rename spanning two sections invalidates both, and a listed entry at the section's end name is cut off with the ones beyond it.	2026-08-11 21:11:18 -07:00