Files
seaweedfs/weed/mount/weedfs_dir_lookup_cache_test.go
T
e012fca9a2 mount: keep lookups from stalling behind directory cache rebuilds (#12754)
* mount: release the meta cache lock between batches of a rebuild wipe

deleteFolderChildrenForRebuild held mc's write lock while it listed every
cached child of the directory and deleted them. Every metadata read in the
mount (FindEntry, IsNameFresh, isOversized) takes the read lock, and a
waiting writer blocks new readers, so while a large directory was wiped,
lookups and getattrs in unrelated directories queued behind the scan. The
scan gets slower as leveldb accumulates the deletion markers of earlier
wipes of the same directory, which happens quickly once lookups rebuild
expired directories.

Wipe rebuildWipeBatchSize (1024) children per lock hold and drop the lock
between batches. A child is judged pinned under the same lock that deletes
it, so a local-only create is kept whether it was pinned before the wipe
started or between batches. The listing that refills the directory already
ran outside the lock, so batching adds no interleaving the build did not
allow.

Add LevelDBStore.BatchDeleteEntries, the counterpart of BatchInsertEntries.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: bound how long a lookup waits on the directory rebuild it starts

Since #11268 a lookup whose parent's cache expired rebuilds the parent and
every expired ancestor up to the mount root before it answers, and every
lookup beneath them joins the same builds. With a slow listing (a large
directory, a busy filer, or the meta cache lock held by another rebuild) a
plain stat, an ls -l, or the parent touch after mkdir waited for all of it.

Wait at most lookupRebuildTimeout (2s), then fall back to one LookupEntry
RPC. Since #12206 the shared build does not run on the requester's context,
so it keeps going after the waiter gives up and later lookups find the
directory cached. A timed-out wait records a rebuild attempt, so lookups
during the cooldown do not queue on a build that is already known to be
slow.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: wait for the rebuild's listing in TestLookupDoesNotWaitOutSlowRebuild

The build runs on its own goroutine, so the lookup that stopped waiting
for it can return before the build's ListEntries reaches the filer. The
test read `started` without waiting right after the lookup returned, which
relied on an ordering nothing guarantees, and the ListEntries count taken
afterwards assumed the same. Wait for the listing to arrive, bounded by
the test's deadline.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: keep children inserted while a rebuild wipe runs

A deferred create inserts its placeholder before Create registers the
handle that pins it. The single-lock wipe made such an insert wait until
it finished, so the placeholder survived; a batch that runs in that gap
judged it unpinned and dropped it, and the rebuilt listing then omitted
the file until its flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drop a cached path that an event vacates during a rebuild wipe

A build replays only the events later than its listing's snapshot and
leaves the earlier ones to the listing, which starts after the wipe. With
the wipe in batches a file's whole life fits inside it: created, flushed,
then unlinked or renamed away. Both events are earlier than the snapshot
and are skipped, the listing has no such file, and the placeholder the
wipe kept stays in the published directory until the next rebuild.

While a directory's wipe runs, an event buffered for it now takes the
path it vacates out of the store at once, fenced by versions like any
other event. After the wipe the store holds what the listing put there,
and a removal earlier than the snapshot may still arrive, so from then on
the event is left to the replay as before.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: check which file a removal took away during a rebuild wipe

Two things the removal applied during a rebuild wipe got wrong.

A placeholder is unversioned, so the version gate lets every removal
through. One that arrived late, for an earlier file of the same name,
took out the placeholder of a file created since. The entry the event
took away is now compared with the cached one, by inode or, with no
inode to compare, by creation time. An event that only names the file
cannot tell and is taken at its word.

The entry was deleted without OP=MV, so taking one name of a hard-linked
file out of the cache also took one off the count its other names share.
A replayed event that vacates a path already deletes with OP=MV.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: keep a removal the rebuild wipe declined out of the replay

The wipe declines a removal when what is cached there is newer than the
event or is another file than the one it took away. The event stayed
buffered anyway. An unversioned listing clears the children's version
records and replays every buffered event, so the declined removal then
deleted the entry the wipe kept and the directory was published without
it.

A declined removal now buffers only the name it puts in place, so a
rename still lands on replay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check a declined removal again when the build replays it

The previous commit dropped the removal half of an event the wipe
declined, judged by the file cached when the event arrived. Earlier
buffered events can replace that file during the replay: a listing
attempt that failed leaves its events buffered, so the retry's wipe may
see the first attempt's file while the replay puts a later one in place
first. The dropped delete then left that later file published.

The wipe now records which removals it declined, and the replay checks
each against the entry it finds at that point, replaying only the name
the event puts in place when that entry is still another file. A
removal declined by version is replayed as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a declined removal's invalidation off the placeholder's handle

After the replay the build invalidates every buffered event again,
raised to the snapshot, and it still sent the original of a removal the
replay rejected. A late delete of an earlier file then reached the
handle of the placeholder at that name: the placeholder is unversioned,
so the raised timestamp got past it and the handle was marked deleted.

The replay's decision on a declined removal now replaces the event in
the buffer, and the invalidations use what was replayed. It is made
before the snapshot check, so a removal the listing covered is judged
the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep an event's signatures on the halves a rename is split into

A declined removal replays as its rename's create half, and a rename across
a building directory is split in two. Both dropped the event's filer
signatures, so a handle stamped by that filer no longer fenced the replayed
invalidation by timestamp and could take older listed contents over newer
ones it had read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drain invalidations around the declined-removal build check

Invalidations run on a background worker, so the baseline could miss the
removal's own and the final read could run before the build's arrived.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-19 00:04:03 -08:00

629 lines
22 KiB
Go

package mount
import (
"context"
"errors"
"fmt"
"path/filepath"
"runtime"
"strconv"
"sync"
"sync/atomic"
"testing"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/metadata"
"github.com/seaweedfs/go-fuse/v2/fuse"
"github.com/seaweedfs/seaweedfs/weed/mount/meta_cache"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/util"
)
func newLookupCacheTestWFS(t *testing.T, ttlSec int) *WFS {
t.Helper()
uidGidMapper, err := meta_cache.NewUidGidMapper("", "")
if err != nil {
t.Fatalf("create uid/gid mapper: %v", err)
}
root := util.FullPath("/")
wfs := &WFS{
signature: 1,
inodeToPath: NewInodeToPath(root, ttlSec),
fhMap: NewFileHandleToInode(),
fhLockTable: util.NewLockTable[FileHandleId](),
hardLinkLockTable: util.NewLockTable[string](),
option: &Option{
ChunkSizeLimit: 1024,
ConcurrentReaders: 1,
VolumeServerAccess: "filerProxy",
FilerAddresses: []pb.ServerAddress{
pb.NewServerAddressWithGrpcPort("127.0.0.1:1", 1),
},
GrpcDialOption: grpc.WithTransportCredentials(insecure.NewCredentials()),
UidGidMapper: uidGidMapper,
},
}
wfs.metaCache = meta_cache.NewMetaCache(
filepath.Join(t.TempDir(), "meta"),
uidGidMapper, root, false,
func(path util.FullPath) { wfs.inodeToPath.MarkChildrenCached(path) },
func(path util.FullPath) bool { return wfs.inodeToPath.IsChildrenCached(path) },
wfs.onEntryInvalidation, nil,
)
t.Cleanup(wfs.metaCache.Shutdown)
return wfs
}
type lookupCacheTestFiler struct {
filer_pb.UnimplementedSeaweedFilerServer
entries []*filer_pb.Entry
snapshotTsNs int64
listErr error
listCalls atomic.Int32
lookupCalls atomic.Int32
mu sync.Mutex
listGate chan struct{}
startedCh chan struct{}
startedOnce sync.Once
}
func (s *lookupCacheTestFiler) setListGate(gate, started chan struct{}) {
s.mu.Lock()
defer s.mu.Unlock()
s.listGate = gate
s.startedCh = started
s.startedOnce = sync.Once{}
}
func (s *lookupCacheTestFiler) signalStarted() {
s.startedOnce.Do(func() {
if s.startedCh != nil {
close(s.startedCh)
}
})
}
func (s *lookupCacheTestFiler) LookupDirectoryEntry(ctx context.Context, req *filer_pb.LookupDirectoryEntryRequest) (*filer_pb.LookupDirectoryEntryResponse, error) {
s.lookupCalls.Add(1)
return &filer_pb.LookupDirectoryEntryResponse{
Entry: &filer_pb.Entry{
Name: req.Name,
Attributes: &filer_pb.FuseAttributes{FileSize: 123, FileMode: 0100644},
},
}, nil
}
func (s *lookupCacheTestFiler) ListEntries(req *filer_pb.ListEntriesRequest, stream filer_pb.SeaweedFiler_ListEntriesServer) error {
s.listCalls.Add(1)
s.mu.Lock()
gate := s.listGate
s.mu.Unlock()
if gate != nil {
s.signalStarted()
<-gate
}
if s.listErr != nil {
return s.listErr
}
for _, e := range s.entries {
if err := stream.Send(&filer_pb.ListEntriesResponse{Entry: e}); err != nil {
return err
}
}
if s.snapshotTsNs != 0 {
stream.SetTrailer(metadata.Pairs(filer_pb.ListSnapshotTsNsTrailerKey, strconv.FormatInt(s.snapshotTsNs, 10)))
}
return nil
}
// expireDirCache marks a cached directory's TTL as elapsed without invalidating
// it, reproducing the state the kernel leaves the mount in when it still holds
// the directory listing (FOPEN_CACHE_DIR | FOPEN_KEEP_CACHE) past cacheMetaTtlSec.
func expireDirCache(t *testing.T, itp *InodeToPath, dir util.FullPath) {
t.Helper()
itp.Lock()
defer itp.Unlock()
d := itp.dirStateOf(dir)
if d == nil {
t.Fatalf("expireDirCache: no dirState for %s", dir)
}
if !d.isChildrenCached {
t.Fatalf("expireDirCache: %s not cached", dir)
}
d.cachedExpiresTime = time.Now().Add(-time.Second)
}
// TestLookupRebuildsExpiredDirectoryCache reproduces issue #11262: after the
// metadata cache TTL elapses the kernel can still serve directory listings
// from its page cache, so ReadDir never runs and EnsureVisited is not called.
// Metadata-heavy listings (e.g. `ls --color`) then reach Lookup, which must
// rebuild the directory cache once instead of issuing one LookupEntry RPC
// per entry.
func TestLookupRebuildsExpiredDirectoryCache(t *testing.T) {
wfs := newLookupCacheTestWFS(t, 60)
dir := util.FullPath("/dir")
wfs.inodeToPath.Lookup(dir, time.Now().Unix(), true, false, 0, true)
fake := &lookupCacheTestFiler{
entries: []*filer_pb.Entry{
{Name: "a", Attributes: &filer_pb.FuseAttributes{FileSize: 1, FileMode: 0100644}},
{Name: "b", Attributes: &filer_pb.FuseAttributes{FileSize: 2, FileMode: 0100644}},
{Name: "c", Attributes: &filer_pb.FuseAttributes{FileSize: 3, FileMode: 0100644}},
},
snapshotTsNs: 5000,
}
startFakeFiler(t, wfs, fake)
if err := meta_cache.EnsureVisited(context.Background(), wfs.metaCache, wfs, dir, 0); err != nil {
t.Fatalf("EnsureVisited: %v", err)
}
if !wfs.inodeToPath.IsChildrenCached(dir) {
t.Fatal("directory should be cached after EnsureVisited")
}
expireDirCache(t, wfs.inodeToPath, dir)
if wfs.inodeToPath.IsChildrenCached(dir) {
t.Fatal("directory cache should be expired")
}
listCallsBefore := fake.listCalls.Load()
lookupCallsBefore := fake.lookupCalls.Load()
for _, name := range []string{"a", "b", "c"} {
entry, _, status := wfs.lookupEntry(context.Background(), dir.Child(name))
if status != fuse.OK {
t.Fatalf("lookupEntry %s: %v", name, status)
}
if entry == nil || entry.Name() != name {
t.Fatalf("lookupEntry %s: got %v", name, entry)
}
}
listCalls := fake.listCalls.Load() - listCallsBefore
lookupCalls := fake.lookupCalls.Load() - lookupCallsBefore
if lookupCalls != 0 {
t.Errorf("LookupDirectoryEntry RPCs = %d, want 0 (expired dir cache should be rebuilt, not read per-entry)", lookupCalls)
}
if listCalls != 1 {
t.Errorf("ListEntries RPCs = %d, want 1 (expired dir cache should be rebuilt once)", listCalls)
}
}
// TestLookupRebuildDeduplicatesConcurrentRebuilds exercises the singleflight in
// EnsureVisited: concurrent lookups into the same expired directory share one
// rebuild rather than each issuing its own ListEntries.
func TestLookupRebuildDeduplicatesConcurrentRebuilds(t *testing.T) {
wfs := newLookupCacheTestWFS(t, 60)
dir := util.FullPath("/dir")
wfs.inodeToPath.Lookup(dir, time.Now().Unix(), true, false, 0, true)
fake := &lookupCacheTestFiler{
entries: []*filer_pb.Entry{
{Name: "a", Attributes: &filer_pb.FuseAttributes{FileSize: 1, FileMode: 0100644}},
{Name: "b", Attributes: &filer_pb.FuseAttributes{FileSize: 2, FileMode: 0100644}},
{Name: "c", Attributes: &filer_pb.FuseAttributes{FileSize: 3, FileMode: 0100644}},
},
snapshotTsNs: 5000,
}
startFakeFiler(t, wfs, fake)
if err := meta_cache.EnsureVisited(context.Background(), wfs.metaCache, wfs, dir, 0); err != nil {
t.Fatalf("EnsureVisited: %v", err)
}
expireDirCache(t, wfs.inodeToPath, dir)
started := make(chan struct{})
gate := make(chan struct{})
fake.setListGate(gate, started)
listCallsBefore := fake.listCalls.Load()
names := []string{"a", "b", "c"}
var wg sync.WaitGroup
errs := make([]error, len(names))
for i, name := range names {
wg.Add(1)
go func(i int, name string) {
defer wg.Done()
if _, _, status := wfs.lookupEntry(context.Background(), dir.Child(name)); status != fuse.OK {
errs[i] = fmt.Errorf("lookupEntry %s: %v", name, status)
}
}(i, name)
}
<-started
// Hold the rebuild so the other lookups reach the singleflight while it
// is in flight, exercising the deduplication rather than serializing.
time.Sleep(50 * time.Millisecond)
close(gate)
wg.Wait()
for _, err := range errs {
if err != nil {
t.Fatal(err)
}
}
if got := fake.listCalls.Load() - listCallsBefore; got != 1 {
t.Errorf("rebuild ListEntries RPCs = %d, want 1 (concurrent lookups should share one rebuild)", got)
}
if got := fake.lookupCalls.Load(); got != 0 {
t.Errorf("LookupDirectoryEntry RPCs = %d, want 0", got)
}
}
// TestLookupRebuildCooldownAfterFailure guards the failure path: a failed
// rebuild records the attempt so an immediate retry is suppressed (per-entry
// RPC fallback), and once the cooldown elapses and the filer recovers a later
// lookup rebuilds the cache again.
func TestLookupRebuildCooldownAfterFailure(t *testing.T) {
wfs := newLookupCacheTestWFS(t, 60)
dir := util.FullPath("/dir")
wfs.inodeToPath.Lookup(dir, time.Now().Unix(), true, false, 0, true)
fake := &lookupCacheTestFiler{
entries: []*filer_pb.Entry{
{Name: "a", Attributes: &filer_pb.FuseAttributes{FileSize: 1, FileMode: 0100644}},
},
snapshotTsNs: 5000,
}
startFakeFiler(t, wfs, fake)
if err := meta_cache.EnsureVisited(context.Background(), wfs.metaCache, wfs, dir, 0); err != nil {
t.Fatalf("EnsureVisited: %v", err)
}
expireDirCache(t, wfs.inodeToPath, dir)
// First lookup: rebuild fails, falls through to the per-entry RPC.
fake.listErr = errors.New("persistent listing failure")
listCallsBefore := fake.listCalls.Load()
lookupCallsBefore := fake.lookupCalls.Load()
if entry, _, status := wfs.lookupEntry(context.Background(), dir.Child("a")); status != fuse.OK || entry == nil || entry.Name() != "a" {
t.Fatalf("lookupEntry a after failed rebuild: status=%v entry=%v", status, entry)
}
if got := fake.lookupCalls.Load() - lookupCallsBefore; got != 1 {
t.Errorf("first lookup LookupDirectoryEntry RPCs = %d, want 1 (fall through to per-entry RPC)", got)
}
if got := fake.listCalls.Load() - listCallsBefore; got == 0 {
t.Error("first lookup should have attempted a rebuild")
}
// Immediate retry: cooldown suppresses the rebuild, falls through again.
listCallsBefore = fake.listCalls.Load()
lookupCallsBefore = fake.lookupCalls.Load()
if _, _, status := wfs.lookupEntry(context.Background(), dir.Child("a")); status != fuse.OK {
t.Fatalf("second lookupEntry a: %v", status)
}
if got := fake.listCalls.Load() - listCallsBefore; got != 0 {
t.Errorf("second lookup ListEntries RPCs = %d, want 0 (cooldown suppresses rebuild)", got)
}
if got := fake.lookupCalls.Load() - lookupCallsBefore; got != 1 {
t.Errorf("second lookup LookupDirectoryEntry RPCs = %d, want 1", got)
}
// Filer recovers and the cooldown elapses: the next lookup rebuilds.
fake.listErr = nil
advanceRebuildAttempt(t, wfs.inodeToPath, dir, expiredDirRebuildCooldown+time.Second)
listCallsBefore = fake.listCalls.Load()
lookupCallsBefore = fake.lookupCalls.Load()
if _, _, status := wfs.lookupEntry(context.Background(), dir.Child("a")); status != fuse.OK {
t.Fatalf("third lookupEntry a: %v", status)
}
if got := fake.listCalls.Load() - listCallsBefore; got != 1 {
t.Errorf("third lookup ListEntries RPCs = %d, want 1 (rebuild after cooldown)", got)
}
if got := fake.lookupCalls.Load() - lookupCallsBefore; got != 0 {
t.Errorf("third lookup LookupDirectoryEntry RPCs = %d, want 0 (served from rebuilt cache)", got)
}
}
// advanceRebuildAttempt moves a directory's last rebuild attempt back by d,
// simulating the cooldown having elapsed without sleeping.
func advanceRebuildAttempt(t *testing.T, itp *InodeToPath, dir util.FullPath, d time.Duration) {
t.Helper()
itp.Lock()
defer itp.Unlock()
state := itp.dirStateOf(dir)
if state == nil {
t.Fatalf("advanceRebuildAttempt: no dirState for %s", dir)
}
state.lastRebuildAttempt = time.Now().Add(-d)
}
// TestLookupRebuildsInvalidatedDirectoryCache covers a dropped listing rather
// than an expired one: invalidation, idle eviction, or the purge on kernel
// Forget all clear the directory's cached flag, and without a rebuild path a
// lookup-only workload (e.g. git object reads) reads through per entry until
// a ReadDir happened to run.
func TestLookupRebuildsInvalidatedDirectoryCache(t *testing.T) {
wfs := newLookupCacheTestWFS(t, 60)
dir := util.FullPath("/dir")
wfs.inodeToPath.Lookup(dir, time.Now().Unix(), true, false, 0, true)
fake := &lookupCacheTestFiler{
entries: []*filer_pb.Entry{
{Name: "a", Attributes: &filer_pb.FuseAttributes{FileSize: 1, FileMode: 0100644}},
{Name: "b", Attributes: &filer_pb.FuseAttributes{FileSize: 2, FileMode: 0100644}},
{Name: "c", Attributes: &filer_pb.FuseAttributes{FileSize: 3, FileMode: 0100644}},
},
snapshotTsNs: 5000,
}
startFakeFiler(t, wfs, fake)
if err := meta_cache.EnsureVisited(context.Background(), wfs.metaCache, wfs, dir, 0); err != nil {
t.Fatalf("EnsureVisited: %v", err)
}
wfs.inodeToPath.InvalidateChildrenCache(dir)
if wfs.inodeToPath.IsChildrenCached(dir) {
t.Fatal("directory cache should be dropped")
}
listCallsBefore := fake.listCalls.Load()
lookupCallsBefore := fake.lookupCalls.Load()
for _, name := range []string{"a", "b", "c"} {
entry, _, status := wfs.lookupEntry(context.Background(), dir.Child(name))
if status != fuse.OK {
t.Fatalf("lookupEntry %s: %v", name, status)
}
if entry == nil || entry.Name() != name {
t.Fatalf("lookupEntry %s: got %v", name, entry)
}
}
if got := fake.listCalls.Load() - listCallsBefore; got != 1 {
t.Errorf("ListEntries RPCs = %d, want 1 (dropped dir cache should be rebuilt once)", got)
}
if got := fake.lookupCalls.Load() - lookupCallsBefore; got != 0 {
t.Errorf("LookupDirectoryEntry RPCs = %d, want 0", got)
}
}
// TestLookupNeverCachedDirReadsThrough guards the other side: a directory
// that never held a cached listing must not be rebuilt on a single lookup,
// or one stat could pin an arbitrarily large listing.
func TestLookupNeverCachedDirReadsThrough(t *testing.T) {
wfs := newLookupCacheTestWFS(t, 60)
dir := util.FullPath("/dir")
wfs.inodeToPath.Lookup(dir, time.Now().Unix(), true, false, 0, true)
fake := &lookupCacheTestFiler{snapshotTsNs: 5000}
startFakeFiler(t, wfs, fake)
if _, _, status := wfs.lookupEntry(context.Background(), dir.Child("a")); status != fuse.OK {
t.Fatalf("lookupEntry a: %v", status)
}
if got := fake.listCalls.Load(); got != 0 {
t.Errorf("ListEntries RPCs = %d, want 0 (never-cached dir stays read-through)", got)
}
if got := fake.lookupCalls.Load(); got != 1 {
t.Errorf("LookupDirectoryEntry RPCs = %d, want 1", got)
}
}
// TestForgetKeepsDirRebuildable exercises the kernel Forget path: the dropped
// dirState must not lose the record that the directory once held a listing,
// or a re-looked-up directory would read through forever.
func TestForgetKeepsDirRebuildable(t *testing.T) {
itp := NewInodeToPath("/", 60)
dir := util.FullPath("/dir")
inode := itp.Lookup(dir, time.Now().Unix(), true, false, 0, true)
itp.MarkChildrenCached(dir)
itp.Forget(inode, 1, nil, func(util.FullPath) {})
if itp.dirStateOf(dir) != nil {
t.Fatal("Forget should drop the dirState")
}
if itp.ShouldRebuildExpiredDir(dir, expiredDirRebuildCooldown) {
t.Fatal("a forgotten directory has no state to mark cached; lookup must re-track it first")
}
itp.Lookup(dir, time.Now().Unix(), true, false, 0, true)
if !itp.ShouldRebuildExpiredDir(dir, expiredDirRebuildCooldown) {
t.Fatal("re-looked-up directory that once held a listing should rebuild")
}
}
// TestReusedPathKeepsRebuildRecord covers a directory deleted and re-cached
// while its first queue slot is still pending: eviction reaching the stale
// slot must not delete the newer record, or the directory reads through per
// lookup until the next listing.
func TestReusedPathKeepsRebuildRecord(t *testing.T) {
itp := NewInodeToPath("/", 60)
dir := util.FullPath("/hot")
now := time.Now().Unix()
itp.Lookup(dir, now, true, false, 0, true)
itp.MarkChildrenCached(dir)
itp.RemovePath(dir, nil)
inode := itp.Lookup(dir, now+1, true, false, 0, true)
itp.MarkChildrenCached(dir)
for n := 0; n < maxWasCachedDirs-1; n++ {
itp.markWasCached(util.FullPath(fmt.Sprintf("/filler%d", n)))
}
itp.Forget(inode, 1, nil, nil)
itp.Lookup(dir, now+2, true, false, 0, true)
if !itp.ShouldRebuildExpiredDir(dir, expiredDirRebuildCooldown) {
t.Fatal("re-cached directory lost its rebuild record to a stale queue entry")
}
}
// TestLookupRefreshesStaleSection covers a directory still cached but with an
// invalidated section: without a refresh on lookup, every lookup in that
// range reads through until a listing re-validates it.
func TestLookupRefreshesStaleSection(t *testing.T) {
wfs := newLookupCacheTestWFS(t, 60)
dir := util.FullPath("/dir")
wfs.inodeToPath.Lookup(dir, time.Now().Unix(), true, false, 0, true)
fake := &lookupCacheTestFiler{
entries: []*filer_pb.Entry{
{Name: "a", Attributes: &filer_pb.FuseAttributes{FileSize: 1, FileMode: 0100644}},
{Name: "b", Attributes: &filer_pb.FuseAttributes{FileSize: 2, FileMode: 0100644}},
{Name: "c", Attributes: &filer_pb.FuseAttributes{FileSize: 3, FileMode: 0100644}},
},
snapshotTsNs: 5000,
}
startFakeFiler(t, wfs, fake)
if err := meta_cache.EnsureVisited(context.Background(), wfs.metaCache, wfs, dir, 0); err != nil {
t.Fatalf("EnsureVisited: %v", err)
}
// A burst of remote changes in the only section invalidates it.
for i := 0; i < 64; i++ {
resp := &filer_pb.SubscribeMetadataResponse{
Directory: string(dir),
TsNs: 6000 + int64(i),
EventNotification: &filer_pb.EventNotification{
NewEntry: &filer_pb.Entry{
Name: fmt.Sprintf("x-%03d", i),
Attributes: &filer_pb.FuseAttributes{Crtime: 1, Mtime: 1, FileMode: 0100644, FileSize: 1},
},
},
}
if err := wfs.metaCache.ApplyMetadataResponse(context.Background(), resp, meta_cache.SubscriberMetadataResponseApplyOptions); err != nil {
t.Fatalf("apply churn %d: %v", i, err)
}
}
if wfs.metaCache.IsNameFresh(dir.Child("a")) {
t.Fatal("section should be stale after the change burst")
}
listCallsBefore := fake.listCalls.Load()
lookupCallsBefore := fake.lookupCalls.Load()
for _, name := range []string{"a", "b", "c"} {
if entry, _, status := wfs.lookupEntry(context.Background(), dir.Child(name)); status != fuse.OK || entry == nil {
t.Fatalf("lookupEntry %s: status=%v entry=%v", name, status, entry)
}
}
if got := fake.lookupCalls.Load() - lookupCallsBefore; got != 0 {
t.Errorf("LookupDirectoryEntry RPCs = %d, want 0 (stale section should be refreshed, not read per-entry)", got)
}
if got := fake.listCalls.Load() - listCallsBefore; got != 1 {
t.Errorf("ListEntries RPCs = %d, want 1 (one section refresh)", got)
}
}
// lookupEntryWithin runs lookupEntry on its own goroutine and reports whether
// it returned before the deadline. The deadline is not a timing assertion: a
// lookup that returns at all returns well inside it, and one that queued on a
// stalled build does not return until the test releases the build.
func lookupEntryWithin(wfs *WFS, fullpath util.FullPath, deadline time.Duration) (name string, status fuse.Status, returned bool) {
type result struct {
name string
status fuse.Status
}
done := make(chan result, 1)
go func() {
entry, _, status := wfs.lookupEntry(context.Background(), fullpath)
r := result{status: status}
if entry != nil {
r.name = entry.Name()
}
done <- r
}()
select {
case r := <-done:
return r.name, r.status, true
case <-time.After(deadline):
return "", fuse.EIO, false
}
}
// TestLookupDoesNotWaitOutSlowRebuild: a lookup that starts the rebuild of an
// expired directory stops waiting for it after lookupRebuildWaitTime and
// answers with one LookupEntry RPC. The build carries on without it, so once
// the listing completes later lookups are served from the cache again.
func TestLookupDoesNotWaitOutSlowRebuild(t *testing.T) {
wfs := newLookupCacheTestWFS(t, 60)
wfs.lookupRebuildWait = 50 * time.Millisecond
dir := util.FullPath("/dir")
wfs.inodeToPath.Lookup(dir, time.Now().Unix(), true, false, 0, true)
fake := &lookupCacheTestFiler{
entries: []*filer_pb.Entry{
{Name: "a", Attributes: &filer_pb.FuseAttributes{FileSize: 1, FileMode: 0100644}},
{Name: "b", Attributes: &filer_pb.FuseAttributes{FileSize: 2, FileMode: 0100644}},
},
snapshotTsNs: 5000,
}
startFakeFiler(t, wfs, fake)
if err := meta_cache.EnsureVisited(context.Background(), wfs.metaCache, wfs, dir, 0); err != nil {
t.Fatalf("EnsureVisited: %v", err)
}
expireDirCache(t, wfs.inodeToPath, dir)
// Stall the rebuild's listing until released.
started := make(chan struct{})
gate := make(chan struct{})
var releaseOnce sync.Once
release := func() { releaseOnce.Do(func() { close(gate) }) }
t.Cleanup(release)
fake.setListGate(gate, started)
const deadline = 10 * time.Second
lookupCallsBefore := fake.lookupCalls.Load()
name, status, returned := lookupEntryWithin(wfs, dir.Child("a"), deadline)
if !returned {
t.Fatal("lookup is still waiting on a rebuild whose listing is stalled")
}
if status != fuse.OK || name != "a" {
t.Fatalf("lookupEntry a: status=%v name=%q", status, name)
}
if got := fake.lookupCalls.Load() - lookupCallsBefore; got != 1 {
t.Errorf("LookupDirectoryEntry RPCs = %d, want 1 (fall back to the per-entry RPC)", got)
}
// The build runs on its own goroutine and may reach the filer only after
// the lookup stopped waiting for it, so wait for its listing to arrive
// rather than reading how far it got when the lookup returned. From here
// on that listing is in flight, parked at the gate, and already counted.
select {
case <-started:
case <-time.After(deadline):
t.Fatal("lookup should have started a rebuild")
}
// The wait that timed out counts as a rebuild attempt, so until the
// cooldown passes a lookup goes straight to the per-entry RPC. Were it to
// queue on the build again it could not return: the build is still
// stalled and the wait is now far longer than the deadline.
wfs.lookupRebuildWait = time.Hour
listCallsBefore := fake.listCalls.Load()
if _, status, returned := lookupEntryWithin(wfs, dir.Child("b"), deadline); !returned {
t.Fatal("second lookup queued on the build that is still in flight")
} else if status != fuse.OK {
t.Fatalf("lookupEntry b: %v", status)
}
if got := fake.listCalls.Load() - listCallsBefore; got != 0 {
t.Errorf("second lookup ListEntries RPCs = %d, want 0 (no second rebuild during the cooldown)", got)
}
// Let the listing finish: the build the first lookup gave up on completes
// and publishes the directory.
release()
for end := time.Now().Add(deadline); !wfs.inodeToPath.IsChildrenCached(dir); {
if time.Now().After(end) {
t.Fatal("the rebuild the lookup stopped waiting for never completed")
}
runtime.Gosched()
}
lookupCallsBefore = fake.lookupCalls.Load()
if name, status, returned := lookupEntryWithin(wfs, dir.Child("a"), deadline); !returned || status != fuse.OK || name != "a" {
t.Fatalf("lookupEntry a after rebuild: returned=%v status=%v name=%q", returned, status, name)
}
if got := fake.lookupCalls.Load() - lookupCallsBefore; got != 0 {
t.Errorf("LookupDirectoryEntry RPCs = %d, want 0 (served from the rebuilt cache)", got)
}
}