mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-20 03:13:53 +01:00
mount: keep lookups from stalling behind directory cache rebuilds (#12754)
* mount: release the meta cache lock between batches of a rebuild wipe deleteFolderChildrenForRebuild held mc's write lock while it listed every cached child of the directory and deleted them. Every metadata read in the mount (FindEntry, IsNameFresh, isOversized) takes the read lock, and a waiting writer blocks new readers, so while a large directory was wiped, lookups and getattrs in unrelated directories queued behind the scan. The scan gets slower as leveldb accumulates the deletion markers of earlier wipes of the same directory, which happens quickly once lookups rebuild expired directories. Wipe rebuildWipeBatchSize (1024) children per lock hold and drop the lock between batches. A child is judged pinned under the same lock that deletes it, so a local-only create is kept whether it was pinned before the wipe started or between batches. The listing that refills the directory already ran outside the lock, so batching adds no interleaving the build did not allow. Add LevelDBStore.BatchDeleteEntries, the counterpart of BatchInsertEntries. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * mount: bound how long a lookup waits on the directory rebuild it starts Since #11268 a lookup whose parent's cache expired rebuilds the parent and every expired ancestor up to the mount root before it answers, and every lookup beneath them joins the same builds. With a slow listing (a large directory, a busy filer, or the meta cache lock held by another rebuild) a plain stat, an ls -l, or the parent touch after mkdir waited for all of it. Wait at most lookupRebuildTimeout (2s), then fall back to one LookupEntry RPC. Since #12206 the shared build does not run on the requester's context, so it keeps going after the waiter gives up and later lookups find the directory cached. A timed-out wait records a rebuild attempt, so lookups during the cooldown do not queue on a build that is already known to be slow. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * mount: wait for the rebuild's listing in TestLookupDoesNotWaitOutSlowRebuild The build runs on its own goroutine, so the lookup that stopped waiting for it can return before the build's ListEntries reaches the filer. The test read `started` without waiting right after the lookup returned, which relied on an ordering nothing guarantees, and the ListEntries count taken afterwards assumed the same. Wait for the listing to arrive, bounded by the test's deadline. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * mount: keep children inserted while a rebuild wipe runs A deferred create inserts its placeholder before Create registers the handle that pins it. The single-lock wipe made such an insert wait until it finished, so the placeholder survived; a batch that runs in that gap judged it unpinned and dropped it, and the rebuilt listing then omitted the file until its flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: drop a cached path that an event vacates during a rebuild wipe A build replays only the events later than its listing's snapshot and leaves the earlier ones to the listing, which starts after the wipe. With the wipe in batches a file's whole life fits inside it: created, flushed, then unlinked or renamed away. Both events are earlier than the snapshot and are skipped, the listing has no such file, and the placeholder the wipe kept stays in the published directory until the next rebuild. While a directory's wipe runs, an event buffered for it now takes the path it vacates out of the store at once, fenced by versions like any other event. After the wipe the store holds what the listing put there, and a removal earlier than the snapshot may still arrive, so from then on the event is left to the replay as before. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * mount: check which file a removal took away during a rebuild wipe Two things the removal applied during a rebuild wipe got wrong. A placeholder is unversioned, so the version gate lets every removal through. One that arrived late, for an earlier file of the same name, took out the placeholder of a file created since. The entry the event took away is now compared with the cached one, by inode or, with no inode to compare, by creation time. An event that only names the file cannot tell and is taken at its word. The entry was deleted without OP=MV, so taking one name of a hard-linked file out of the cache also took one off the count its other names share. A replayed event that vacates a path already deletes with OP=MV. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * mount: keep a removal the rebuild wipe declined out of the replay The wipe declines a removal when what is cached there is newer than the event or is another file than the one it took away. The event stayed buffered anyway. An unversioned listing clears the children's version records and replays every buffered event, so the declined removal then deleted the entry the wipe kept and the directory was published without it. A declined removal now buffers only the name it puts in place, so a rename still lands on replay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: check a declined removal again when the build replays it The previous commit dropped the removal half of an event the wipe declined, judged by the file cached when the event arrived. Earlier buffered events can replace that file during the replay: a listing attempt that failed leaves its events buffered, so the retry's wipe may see the first attempt's file while the replay puts a later one in place first. The dropped delete then left that later file published. The wipe now records which removals it declined, and the replay checks each against the entry it finds at that point, replaying only the name the event puts in place when that entry is still another file. A removal declined by version is replayed as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep a declined removal's invalidation off the placeholder's handle After the replay the build invalidates every buffered event again, raised to the snapshot, and it still sent the original of a removal the replay rejected. A late delete of an earlier file then reached the handle of the placeholder at that name: the placeholder is unversioned, so the raised timestamp got past it and the handle was marked deleted. The replay's decision on a declined removal now replaces the event in the buffer, and the invalidations use what was replayed. It is made before the snapshot check, so a removal the listing covered is judged the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep an event's signatures on the halves a rename is split into A declined removal replays as its rename's create half, and a rename across a building directory is split in two. Both dropped the event's filer signatures, so a handle stamped by that filer no longer fenced the replayed invalidation by timestamp and could take older listed contents over newer ones it had read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: drain invalidations around the declined-removal build check Invalidations run on a background worker, so the baseline could miss the removal's own and the final read could run before the build's arrived. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com> Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
This commit is contained in:
7 files changed
+1019
-20
No files matched your search
@@ -4,7 +4,6 @@ import (
|
||||
"context"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"math"
|
||||
"os"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -49,6 +48,10 @@ type MetaCache struct {
|
||||
// instead of streaming to the limit again to rediscover them.
|
||||
oversizedDirs map[util.FullPath]struct{}
|
||||
|
||||
// rebuildWipeInserts holds the children inserted into each directory
|
||||
// while its rebuild wipe runs; the wipe keeps them.
|
||||
rebuildWipeInserts map[util.FullPath]map[string]struct{}
|
||||
|
||||
// dirVersionFloors is each cached directory's listing snapshot: the
|
||||
// version of every child the listing covered, present or absent, unless
|
||||
// a later event gave that child its own record. One map write per build
|
||||
@@ -86,6 +89,9 @@ var (
|
||||
|
||||
type directoryBuildState struct {
|
||||
bufferedEvents []*filer_pb.SubscribeMetadataResponse
|
||||
// removals the rebuild wipe declined as being of another file than the
|
||||
// one cached; the replay checks them again against what it finds
|
||||
declinedRemovals map[*filer_pb.SubscribeMetadataResponse]struct{}
|
||||
}
|
||||
|
||||
const recentEventDedupWindow = 4096
|
||||
@@ -135,6 +141,7 @@ func NewMetaCache(dbFolder string, uidGidMapper *UidGidMapper, root util.FullPat
|
||||
dirVersionFloors: make(map[util.FullPath]int64),
|
||||
dirSections: make(map[util.FullPath]*sectionList),
|
||||
oversizedDirs: make(map[util.FullPath]struct{}),
|
||||
rebuildWipeInserts: make(map[util.FullPath]map[string]struct{}),
|
||||
}
|
||||
mc.invalidateWorker = util.NewAsyncBatchWorker(func(batch []EntryInvalidation) {
|
||||
for _, invalidation := range batch {
|
||||
@@ -176,6 +183,10 @@ func (mc *MetaCache) doInsertEntry(ctx context.Context, entry *filer.Entry, vers
|
||||
if err := mc.localStore.InsertEntry(ctx, entry); err != nil {
|
||||
return err
|
||||
}
|
||||
dir, name := entry.FullPath.DirAndName()
|
||||
if inserted, wiping := mc.rebuildWipeInserts[util.FullPath(dir)]; wiping {
|
||||
inserted[name] = struct{}{}
|
||||
}
|
||||
mc.setEntryVersionLocked(ctx, entry.FullPath, versionTsNs)
|
||||
mc.noteHardLinkVersionLocked(ctx, entry.HardLinkId, versionTsNs)
|
||||
return nil
|
||||
@@ -718,34 +729,87 @@ func (mc *MetaCache) SetPinnedChildFn(fn func(*filer.Entry) bool) {
|
||||
mc.pinnedChildFn = fn
|
||||
}
|
||||
|
||||
// rebuildWipeBatchSize caps how many cached children one locked step of a
|
||||
// rebuild's wipe visits. Every metadata read in the mount takes mc's read lock,
|
||||
// and a pending writer blocks new readers, so a wipe that held the write lock
|
||||
// across a whole directory stalled lookups in unrelated directories for as long
|
||||
// as the scan took — longer still while leveldb carries the deletion markers of
|
||||
// earlier wipes of the same directory.
|
||||
const rebuildWipeBatchSize = 1024
|
||||
|
||||
// deleteFolderChildrenForRebuild clears a directory's cached children ahead of a
|
||||
// rebuild, but keeps any child flagged pinned by pinnedChildFn — a local-only
|
||||
// create not yet flushed to the filer. A rebuild refills from a filer listing
|
||||
// that does not include such a create; a blind wipe would drop it and then
|
||||
// markCachedFn publishes the directory authoritatively cached without a file the
|
||||
// client created, so it vanishes from the mount.
|
||||
//
|
||||
// The wipe runs in batches and lets go of the lock between them, so what the
|
||||
// single-lock wipe made wait now lands while it runs: a create's placeholder,
|
||||
// which keepInsertsDuringWipe keeps, and the unlink or rename that ends such a
|
||||
// file before the listing starts, which vacateDuringWipe applies.
|
||||
func (mc *MetaCache) deleteFolderChildrenForRebuild(ctx context.Context, dirPath util.FullPath) error {
|
||||
defer mc.keepInsertsDuringWipe(dirPath)()
|
||||
startFrom := ""
|
||||
for {
|
||||
last, err := mc.deleteFolderChildrenBatch(ctx, dirPath, startFrom)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if last == "" {
|
||||
return nil
|
||||
}
|
||||
startFrom = last
|
||||
}
|
||||
}
|
||||
|
||||
// keepInsertsDuringWipe makes a wipe of dirPath keep the children inserted
|
||||
// until the returned func runs. A deferred create inserts its placeholder
|
||||
// before its handle pins it; the single-lock wipe kept such an insert by
|
||||
// making it wait, and a batch running in that gap must not drop it either.
|
||||
func (mc *MetaCache) keepInsertsDuringWipe(dirPath util.FullPath) func() {
|
||||
mc.Lock()
|
||||
mc.rebuildWipeInserts[dirPath] = make(map[string]struct{})
|
||||
mc.Unlock()
|
||||
return func() {
|
||||
mc.Lock()
|
||||
delete(mc.rebuildWipeInserts, dirPath)
|
||||
mc.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
// deleteFolderChildrenBatch deletes the unpinned ones among the next
|
||||
// rebuildWipeBatchSize children of dirPath that sort after startFrom. It
|
||||
// returns the name to resume from, or "" once the directory is exhausted. A
|
||||
// child is judged pinned under the same lock that deletes it, so a create
|
||||
// pinned between batches survives as surely as one pinned before the wipe
|
||||
// began.
|
||||
func (mc *MetaCache) deleteFolderChildrenBatch(ctx context.Context, dirPath util.FullPath, startFrom string) (string, error) {
|
||||
mc.Lock()
|
||||
defer mc.Unlock()
|
||||
if mc.pinnedChildFn == nil {
|
||||
return mc.localStore.DeleteFolderChildren(ctx, dirPath)
|
||||
}
|
||||
var pinned []*filer.Entry
|
||||
if _, err := mc.localStore.ListDirectoryEntries(ctx, dirPath, "", true, math.MaxInt64, func(entry *filer.Entry) (bool, error) {
|
||||
if mc.pinnedChildFn(entry) {
|
||||
pinned = append(pinned, entry)
|
||||
visited := 0
|
||||
var doomed []util.FullPath
|
||||
inserted := mc.rebuildWipeInserts[dirPath]
|
||||
last, err := mc.localStore.ListDirectoryEntries(ctx, dirPath, startFrom, false, rebuildWipeBatchSize, func(entry *filer.Entry) (bool, error) {
|
||||
visited++
|
||||
if _, kept := inserted[entry.Name()]; kept {
|
||||
return true, nil
|
||||
}
|
||||
if mc.pinnedChildFn == nil || !mc.pinnedChildFn(entry) {
|
||||
doomed = append(doomed, entry.FullPath)
|
||||
}
|
||||
return true, nil
|
||||
}); err != nil {
|
||||
return err
|
||||
})
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := mc.localStore.DeleteFolderChildren(ctx, dirPath); err != nil {
|
||||
return err
|
||||
if err := mc.leveldbStore.BatchDeleteEntries(ctx, doomed); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(pinned) > 0 {
|
||||
return mc.doBatchInsertEntries(ctx, pinned)
|
||||
if visited < rebuildWipeBatchSize {
|
||||
return "", nil
|
||||
}
|
||||
return nil
|
||||
return last, nil
|
||||
}
|
||||
|
||||
// ListDirectoryEntries reports the last name the store reached, which is not
|
||||
@@ -973,10 +1037,102 @@ func (mc *MetaCache) applyMetadataResponseNow(ctx context.Context, resp *filer_p
|
||||
continue
|
||||
}
|
||||
state.bufferedEvents = append(state.bufferedEvents, events...)
|
||||
for _, declined := range mc.vacateDuringWipe(ctx, events) {
|
||||
if state.declinedRemovals == nil {
|
||||
state.declinedRemovals = make(map[*filer_pb.SubscribeMetadataResponse]struct{})
|
||||
}
|
||||
state.declinedRemovals[declined] = struct{}{}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// vacateDuringWipe takes out of the store, at once, the paths that events
|
||||
// buffered for a directory vacate while its rebuild wipe runs. The build
|
||||
// replays only the events later than its listing's snapshot and leaves the
|
||||
// earlier ones to the listing, which starts after the wipe. A listing shows
|
||||
// that a file is gone only by not having it, and that removes nothing: an
|
||||
// entry the wipe kept, a deferred create's placeholder, would stay cached
|
||||
// after its file was flushed and then unlinked or renamed away before the
|
||||
// wipe ended. Once the wipe is over the store holds what the listing put
|
||||
// there, and a removal earlier than the snapshot may still arrive, so the
|
||||
// event is left to the replay again.
|
||||
//
|
||||
// It returns the removals it declined as being of another file, which an
|
||||
// unversioned listing would otherwise replay over the entry the wipe kept.
|
||||
func (mc *MetaCache) vacateDuringWipe(ctx context.Context, events []*filer_pb.SubscribeMetadataResponse) (declined []*filer_pb.SubscribeMetadataResponse) {
|
||||
mc.Lock()
|
||||
defer mc.Unlock()
|
||||
for _, event := range events {
|
||||
oldDir, hasOld := metadataOldParentDir(event)
|
||||
if !hasOld {
|
||||
continue
|
||||
}
|
||||
if _, wiping := mc.rebuildWipeInserts[oldDir]; !wiping {
|
||||
continue
|
||||
}
|
||||
old := event.EventNotification.OldEntry
|
||||
oldPath := oldDir.Child(old.Name)
|
||||
if newDir, hasNew := metadataNewParentDir(event); hasNew && newDir.Child(event.EventNotification.NewEntry.Name) == oldPath {
|
||||
continue // rewritten in place
|
||||
}
|
||||
if event.TsNs != 0 && mc.entryVersionBlocksLocked(ctx, oldPath, event.TsNs, true) {
|
||||
continue // older than what is cached there
|
||||
}
|
||||
// A placeholder is unversioned and fenced against nothing, so a late
|
||||
// removal of an earlier file of the same name is told apart by the
|
||||
// file it took away.
|
||||
cached, err := mc.localStore.FindEntry(ctx, oldPath)
|
||||
if err != nil || cached == nil {
|
||||
continue
|
||||
}
|
||||
if !isCachedFile(old, cached) {
|
||||
declined = append(declined, event)
|
||||
continue
|
||||
}
|
||||
// OP=MV, as where a replayed event vacates a path: taking a name out
|
||||
// of the cache is not an unlink of a hard link's shared record.
|
||||
if err := mc.localStore.DeleteEntry(context.WithValue(ctx, "OP", "MV"), oldPath); err != nil {
|
||||
glog.V(1).Infof("vacate %s while its directory is wiped: %v", oldPath, err)
|
||||
continue
|
||||
}
|
||||
// The placeholder's record goes with it, or the name would stay
|
||||
// unversioned.
|
||||
mc.clearEntryVersionLocked(ctx, oldPath)
|
||||
}
|
||||
return declined
|
||||
}
|
||||
|
||||
// declinedRemovalForReplay returns the removal the wipe declined, or only the
|
||||
// name it puts in place if the entry the replay finds there is still another
|
||||
// file. Earlier buffered events may have replaced the one the wipe saw.
|
||||
func (mc *MetaCache) declinedRemovalForReplay(ctx context.Context, event *filer_pb.SubscribeMetadataResponse) *filer_pb.SubscribeMetadataResponse {
|
||||
mc.Lock()
|
||||
defer mc.Unlock()
|
||||
oldDir, _ := metadataOldParentDir(event)
|
||||
old := event.EventNotification.OldEntry
|
||||
cached, err := mc.localStore.FindEntry(ctx, oldDir.Child(old.Name))
|
||||
if err != nil || cached == nil || isCachedFile(old, cached) {
|
||||
return event
|
||||
}
|
||||
return metadataCreateFragment(event)
|
||||
}
|
||||
|
||||
// isCachedFile reports whether old, the entry an event took away, is the file
|
||||
// cached at its path: the same inode, or with no inode to compare, the same
|
||||
// creation time. An event that only names the file cannot tell, and is taken
|
||||
// at its word.
|
||||
func isCachedFile(old *filer_pb.Entry, cached *filer.Entry) bool {
|
||||
attributes := old.GetAttributes()
|
||||
if attributes == nil {
|
||||
return true
|
||||
}
|
||||
if attributes.Inode != 0 && cached.Attr.Inode != 0 {
|
||||
return attributes.Inode == cached.Attr.Inode
|
||||
}
|
||||
return attributes.Crtime == 0 || attributes.Crtime == cached.Attr.Crtime.Unix()
|
||||
}
|
||||
|
||||
func (mc *MetaCache) applyMetadataResponseDirect(ctx context.Context, resp *filer_pb.SubscribeMetadataResponse, options MetadataResponseApplyOptions, allowUncachedInsert bool) error {
|
||||
if _, err := mc.applyMetadataResponseLocked(ctx, resp, options, allowUncachedInsert); err != nil {
|
||||
return err
|
||||
@@ -1171,7 +1327,15 @@ func (mc *MetaCache) completeDirectoryBuildNow(ctx context.Context, dirPath util
|
||||
}
|
||||
mc.Unlock()
|
||||
|
||||
for _, event := range state.bufferedEvents {
|
||||
for i, event := range state.bufferedEvents {
|
||||
// Decided here for the invalidations below too.
|
||||
if _, declined := state.declinedRemovals[event]; declined {
|
||||
event = mc.declinedRemovalForReplay(ctx, event)
|
||||
state.bufferedEvents[i] = event
|
||||
if event == nil {
|
||||
continue
|
||||
}
|
||||
}
|
||||
// When the server provided a snapshot timestamp, skip events that
|
||||
// the listing already included. When snapshotTsNs == 0 (empty
|
||||
// directory — server returned no entries and no snapshot), replay
|
||||
@@ -1191,6 +1355,9 @@ func (mc *MetaCache) completeDirectoryBuildNow(ctx context.Context, dirPath util
|
||||
// so a handle can hold older state than the completed directory. After
|
||||
// markCachedFn, versioned at the snapshot to outrank the mid-build install.
|
||||
for _, event := range state.bufferedEvents {
|
||||
if event == nil {
|
||||
continue
|
||||
}
|
||||
if event.TsNs < snapshotTsNs {
|
||||
event.TsNs = snapshotTsNs
|
||||
}
|
||||
@@ -1273,7 +1440,8 @@ func metadataDeleteFragment(resp *filer_pb.SubscribeMetadataResponse) *filer_pb.
|
||||
return &filer_pb.SubscribeMetadataResponse{
|
||||
Directory: resp.Directory,
|
||||
EventNotification: &filer_pb.EventNotification{
|
||||
OldEntry: proto.Clone(resp.EventNotification.OldEntry).(*filer_pb.Entry),
|
||||
OldEntry: proto.Clone(resp.EventNotification.OldEntry).(*filer_pb.Entry),
|
||||
Signatures: resp.EventNotification.Signatures,
|
||||
},
|
||||
TsNs: resp.TsNs,
|
||||
}
|
||||
@@ -1292,6 +1460,7 @@ func metadataCreateFragment(resp *filer_pb.SubscribeMetadataResponse) *filer_pb.
|
||||
EventNotification: &filer_pb.EventNotification{
|
||||
NewEntry: proto.Clone(resp.EventNotification.NewEntry).(*filer_pb.Entry),
|
||||
NewParentPath: newDir,
|
||||
Signatures: resp.EventNotification.Signatures,
|
||||
},
|
||||
TsNs: resp.TsNs,
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user