mount: read oversized directories through instead of caching them (#10631)

* mount: read oversized directories through instead of caching them

Visiting a directory pulls every child from the filer into the local
LevelDB before the first listing returns. For a directory of a few
million entries that is minutes of streaming, gigabytes of local store,
and gigabytes of decoded entries in flight -- paid by a mount that may
only walk the directory once.

A build that crosses -cacheDirMaxEntries (default ten thousand) now
stops, cleans up, and marks the directory read-through: listings stream
from the filer with pagination, the way update-hot directories already
do, and lookups in it consult the filer per entry as any uncached
directory does. The refusal is remembered, so the next visit fails fast
instead of streaming to the limit again, and an oversized ancestor is
stepped over when caching its subdirectories rather than wedging every
listing beneath it.

The direct path keeps the same pagination state on the handle, so a walk
that crosses the limit mid-flight carries on from where the cached walk
reached.

* mount: an ancestor found oversized must not fail its descendants

Visiting a directory builds its whole uncached ancestor chain in one
group, so the first discovery that an ancestor is oversized cancelled the
group and surfaced as the listed directory's own refusal: the descendant
build was aborted and the caller marked the descendant read-through,
leaving a perfectly cacheable directory streaming from the filer until
its inode was forgotten. The earlier test missed this by pre-marking the
ancestor, which exercises only the fast path.

The refusal of any directory other than the one being listed is now kept
out of the group's result; it is already remembered for the next visit.

weed/mount/meta_cache/meta_cache.go

@@ -43,6 +43,11 @@ type MetaCache struct {
 	dedupRing            dedupRingBuffer
 	includeSystemEntries bool
 
+	// oversizedDirs are directories the mount refused to cache for their size.
+	// Their listings read through to the filer, and a later visit fails fast
+	// instead of streaming to the limit again to rediscover them.
+	oversizedDirs map[util.FullPath]struct{}
+
 	// dirVersionFloors is each cached directory's listing snapshot: the
 	// version of every child the listing covered, present or absent, unless
 	// a later event gave that child its own record. One map write per build
@@ -119,6 +124,7 @@ func NewMetaCache(dbFolder string, uidGidMapper *UidGidMapper, root util.FullPat
 		buildingDirs:         make(map[util.FullPath]*directoryBuildState),
 		dedupRing:            newDedupRingBuffer(),
 		dirVersionFloors:     make(map[util.FullPath]int64),
+		oversizedDirs:        make(map[util.FullPath]struct{}),
 	}
 	mc.invalidateWorker = util.NewAsyncBatchWorker(func(batch []EntryInvalidation) {
 		for _, invalidation := range batch {
@@ -649,6 +655,19 @@ func (mc *MetaCache) ListDirectoryEntries(ctx context.Context, dirPath util.Full
 	})
 }
 
+func (mc *MetaCache) markOversized(dirPath util.FullPath) {
+	mc.Lock()
+	defer mc.Unlock()
+	mc.oversizedDirs[dirPath] = struct{}{}
+}
+
+func (mc *MetaCache) isOversized(dirPath util.FullPath) bool {
+	mc.RLock()
+	defer mc.RUnlock()
+	_, found := mc.oversizedDirs[dirPath]
+	return found
+}
+
 func (mc *MetaCache) Shutdown() {
 	done := make(chan error, 1)