mount: keep lookups from stalling behind directory cache rebuilds (#12754)

* mount: release the meta cache lock between batches of a rebuild wipe

deleteFolderChildrenForRebuild held mc's write lock while it listed every
cached child of the directory and deleted them. Every metadata read in the
mount (FindEntry, IsNameFresh, isOversized) takes the read lock, and a
waiting writer blocks new readers, so while a large directory was wiped,
lookups and getattrs in unrelated directories queued behind the scan. The
scan gets slower as leveldb accumulates the deletion markers of earlier
wipes of the same directory, which happens quickly once lookups rebuild
expired directories.

Wipe rebuildWipeBatchSize (1024) children per lock hold and drop the lock
between batches. A child is judged pinned under the same lock that deletes
it, so a local-only create is kept whether it was pinned before the wipe
started or between batches. The listing that refills the directory already
ran outside the lock, so batching adds no interleaving the build did not
allow.

Add LevelDBStore.BatchDeleteEntries, the counterpart of BatchInsertEntries.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: bound how long a lookup waits on the directory rebuild it starts

Since #11268 a lookup whose parent's cache expired rebuilds the parent and
every expired ancestor up to the mount root before it answers, and every
lookup beneath them joins the same builds. With a slow listing (a large
directory, a busy filer, or the meta cache lock held by another rebuild) a
plain stat, an ls -l, or the parent touch after mkdir waited for all of it.

Wait at most lookupRebuildTimeout (2s), then fall back to one LookupEntry
RPC. Since #12206 the shared build does not run on the requester's context,
so it keeps going after the waiter gives up and later lookups find the
directory cached. A timed-out wait records a rebuild attempt, so lookups
during the cooldown do not queue on a build that is already known to be
slow.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: wait for the rebuild's listing in TestLookupDoesNotWaitOutSlowRebuild

The build runs on its own goroutine, so the lookup that stopped waiting
for it can return before the build's ListEntries reaches the filer. The
test read `started` without waiting right after the lookup returned, which
relied on an ordering nothing guarantees, and the ListEntries count taken
afterwards assumed the same. Wait for the listing to arrive, bounded by
the test's deadline.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: keep children inserted while a rebuild wipe runs

A deferred create inserts its placeholder before Create registers the
handle that pins it. The single-lock wipe made such an insert wait until
it finished, so the placeholder survived; a batch that runs in that gap
judged it unpinned and dropped it, and the rebuilt listing then omitted
the file until its flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drop a cached path that an event vacates during a rebuild wipe

A build replays only the events later than its listing's snapshot and
leaves the earlier ones to the listing, which starts after the wipe. With
the wipe in batches a file's whole life fits inside it: created, flushed,
then unlinked or renamed away. Both events are earlier than the snapshot
and are skipped, the listing has no such file, and the placeholder the
wipe kept stays in the published directory until the next rebuild.

While a directory's wipe runs, an event buffered for it now takes the
path it vacates out of the store at once, fenced by versions like any
other event. After the wipe the store holds what the listing put there,
and a removal earlier than the snapshot may still arrive, so from then on
the event is left to the replay as before.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: check which file a removal took away during a rebuild wipe

Two things the removal applied during a rebuild wipe got wrong.

A placeholder is unversioned, so the version gate lets every removal
through. One that arrived late, for an earlier file of the same name,
took out the placeholder of a file created since. The entry the event
took away is now compared with the cached one, by inode or, with no
inode to compare, by creation time. An event that only names the file
cannot tell and is taken at its word.

The entry was deleted without OP=MV, so taking one name of a hard-linked
file out of the cache also took one off the count its other names share.
A replayed event that vacates a path already deletes with OP=MV.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: keep a removal the rebuild wipe declined out of the replay

The wipe declines a removal when what is cached there is newer than the
event or is another file than the one it took away. The event stayed
buffered anyway. An unversioned listing clears the children's version
records and replays every buffered event, so the declined removal then
deleted the entry the wipe kept and the directory was published without
it.

A declined removal now buffers only the name it puts in place, so a
rename still lands on replay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check a declined removal again when the build replays it

The previous commit dropped the removal half of an event the wipe
declined, judged by the file cached when the event arrived. Earlier
buffered events can replace that file during the replay: a listing
attempt that failed leaves its events buffered, so the retry's wipe may
see the first attempt's file while the replay puts a later one in place
first. The dropped delete then left that later file published.

The wipe now records which removals it declined, and the replay checks
each against the entry it finds at that point, replaying only the name
the event puts in place when that entry is still another file. A
removal declined by version is replayed as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a declined removal's invalidation off the placeholder's handle

After the replay the build invalidates every buffered event again,
raised to the snapshot, and it still sent the original of a removal the
replay rejected. A late delete of an earlier file then reached the
handle of the placeholder at that name: the placeholder is unversioned,
so the raised timestamp got past it and the handle was marked deleted.

The replay's decision on a declined removal now replaces the event in
the buffer, and the invalidations use what was replayed. It is made
before the snapshot check, so a removal the listing covered is judged
the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep an event's signatures on the halves a rename is split into

A declined removal replays as its rename's create half, and a rename across
a building directory is split in two. Both dropped the event's filer
signatures, so a handle stamped by that filer no longer fenced the replayed
invalidation by timestamp and could take older listed contents over newer
ones it had read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drain invalidations around the declined-removal build check

Invalidations run on a background worker, so the baseline could miss the
removal's own and the final read could run before the build's arrived.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
This commit is contained in:
authored and GitHub committed 2026-11-19 00:04:03 -08:00
1 parent 3159b8408c
commit e012fca9a2
7 files changed
+1019 -20

No files matched your search

@@ -473,6 +473,34 @@ func TestCollectEntryInvalidationsCarryAuthoritativeEntries(t *testing.T) {
}
}
// TestRenameFragmentsKeepSignatures: a handle fences an invalidation by its
// timestamp only within the filer clock the signatures name, so the halves a
// rename is split into keep the event's.
func TestRenameFragmentsKeepSignatures(t *testing.T) {
rename := &filer_pb.SubscribeMetadataResponse{
Directory: "/src",
TsNs: 78,
EventNotification: &filer_pb.EventNotification{
OldEntry: &filer_pb.Entry{Name: "file.tmp"},
NewEntry: &filer_pb.Entry{Name: "file.txt"},
NewParentPath: "/dst",
Signatures: []int32{7},
},
}
var got []EntryInvalidation
for _, fragment := range []*filer_pb.SubscribeMetadataResponse{metadataDeleteFragment(rename), metadataCreateFragment(rename)} {
got = append(got, collectEntryInvalidations(fragment)...)
}
if len(got) != 2 {
t.Fatalf("fragment invalidations = %+v, want one per half", got)
}
for _, invalidation := range got {
if len(invalidation.Signatures) != 1 || invalidation.Signatures[0] != 7 {
t.Errorf("invalidation of %s carries signatures %v, want [7]", invalidation.Path, invalidation.Signatures)
}
}
}
func newTestMetaCache(t *testing.T, cached map[util.FullPath]bool) (*MetaCache, map[util.FullPath]bool, *recordedPaths, *recordedPaths) {
t.Helper()