mount: rebuild dropped directory listings and stale sections on lookup (#12195)

* mount: rebuild dropped directory listings on lookup

Invalidation, idle eviction, and a kernel Forget all clear a directory's
cached-listing flag, but only an expired listing was rebuilt on lookup — a
dropped one read through per entry until a ReadDir happened to run. Track
directories that once held a listing in wasCachedDirs, which survives those
drops, and let ShouldRebuildExpiredDir rebuild them on the next lookup.
Never-cached and read-through directories keep reading through.

* mount: re-validate a stale directory section on lookup

A burst of remote changes invalidates only the directory section it lands
in, but nothing re-validates that section until a ReadDir runs — so a
lookup-heavy workload reads through per entry indefinitely. EnsureNameFresh
re-lists just the section covering the looked-up name, after which lookups
in it are served from the cache again.

* mount: wait for pending flush before a stale-section refresh on lookup

A section refresh lists the range from the filer; without the wait a
pending async flush of the looked-up entry could land after the listing
and leave the refreshed section holding pre-flush metadata — the same
hazard the rebuild path already waits out.

* mount: bound the dropped-listing rebuild history

wasCachedDirs grew one full path per directory ever cached, removed only
by the read-through mark, so deleted and forgotten directories
accumulated forever. Remove the record when its name is deleted, move it
on rename so rebuild eligibility travels with the directory, and cap the
history with FIFO eviction — past the bound an evicted directory reads
through per entry, the same end state as a never-cached one.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: drop oversized stale sections and back off failed refreshes

A section that outgrew one refresh failed EnsureNameFresh with
ErrRefreshRangeTooLarge on every lookup, repeating the multi-page
listing before each per-entry read, and other refresh failures retried
with no cooldown. Drop the cached listing on ErrRefreshRangeTooLarge so
a later lookup rebuilds it re-tiled, as ReadDir does, and stamp the
rebuild attempt on other failures so retries back off under the same
cooldown the rebuild gate uses.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: version hard-link sibling syncs at the committed log position

* mount: tag dropped-listing queue slots so stale ones cannot evict

RemovePath, MovePath, and MarkDirectoryReadThrough delete the rebuild
record but leave the FIFO slot behind, so a path removed and re-cached
holds two queue entries. When the stale slot reaches the front it evicts
the newer record, and the directory reads through per lookup until the
next listing. Tag each record with an insertion generation and only
evict on a match, keeping the newer record for its own slot's turn.

* mount: refresh hard-link siblings from the filer, not the cached copy

Stamping a cached sibling with the mutation's version claimed that
position for content that may predate it — an in-flight event with
newer chunks or attributes then hits the version gate and the sibling
serves stale data until a refresh. Fetch each cached sibling's entry
from the filer after the commit and record the higher of the fetched
and committed positions, falling back to the in-place counter patch
only when the fetch itself fails.

* mount: back off failed section refreshes per section, not per directory

A failed refresh of one stale section marked a directory-wide attempt,
so every other stale section in that directory waited out the cooldown
with per-entry filer reads. Key the attempt timestamp by the section's
lower bound so unrelated ranges still refresh on their own schedule.

* mount: fence hard-linked entries by the group's newest write position

Every path of a hard-link group reads one shared content blob, but the
version gate only tracks per-path records — a sibling absent from
inodeToPath got no version from the sibling sync, so a stale section
refresh or event could roll the shared blob back for all of them.
Record a monotonic version floor per hard-link id wherever versioned
content lands, and check it in the section-refresh and event gates.

* mount: keep hard-link names landing when the group floor fences content

The per-group version floor rejected a delayed event's newEntry while
its oldPath still applied — a rename could drop the source without
adding the destination, and a section refresh could skip then mark a
live link fresh. Rebase the group's current cached content onto the
new path so the name lands without rolling the shared blob back, only
skip refresh entries already cached, and gate the sibling refresh
write under the same lock so a racing event or delete wins.

* mount: rebase hard-link entries from the shared blob when no sibling is cached

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: park group content on uncached hard-link names a stale refresh installs

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: stale the destination section when a fenced hard-link name cannot land

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: bound the sibling hard-link refresh under one request deadline

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: replace a reused-name occupant the hard-link floor would skip

* mount: claim no version for a sibling counter patched without a fetch

* mount: vacate a reused-name occupant without decrementing its group

A sibling write can fold the occupant's filer-side unlink into the shared
blob counter while the stale entry is still cached, so replacing it through
the plain insert may decrement a group that no longer counts the name and
drop a blob live siblings still read. Vacate the path like a rename first;
the counter then only ever drifts high, which leaks a blob rather than
losing shared content.

* mount: keep a fetch-failed sibling's version when patching its counter

The fallback marked the entry unversioned, which fences nothing — a
replayed delete older than the cached content could remove a live link.
Claim the version the entry already holds instead: newer in-flight events
still land and correct it, older replays stay fenced.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

weed/mount/meta_cache/meta_cache_sections.go

@@ -248,6 +248,42 @@ func EnsureListingFresh(ctx context.Context, mc *MetaCache, client filer_pb.File
 	return nil
 }
 
+// EnsureNameFresh re-validates only the section covering name. A lookup into
+// an invalidated range re-establishes it here rather than every later lookup
+// in that range reading through until the next listing.
+func EnsureNameFresh(ctx context.Context, mc *MetaCache, client filer_pb.FilerClient, dirPath util.FullPath, name string) error {
+	r, stale := mc.staleRangeOf(dirPath, name)
+	if !stale {
+		return nil
+	}
+	ctx, cancel := context.WithTimeout(ctx, sectionRefreshTimeout)
+	defer cancel()
+	return mc.refreshSection(ctx, client, dirPath, r.lo, r.hi)
+}
+
+// StaleRangeOf returns the lower bound of the section covering name when it
+// needs re-validation — the key a caller may use to scope refresh backoff
+// per section.
+func StaleRangeOf(mc *MetaCache, dirPath util.FullPath, name string) (string, bool) {
+	r, stale := mc.staleRangeOf(dirPath, name)
+	return r.lo, stale
+}
+
+func (mc *MetaCache) staleRangeOf(dirPath util.FullPath, name string) (nameRange, bool) {
+	mc.RLock()
+	defer mc.RUnlock()
+	sl := mc.dirSections[dirPath]
+	if sl == nil {
+		return nameRange{}, false
+	}
+	idx := sl.sectionOf(name)
+	if s := sl.sections[idx]; !s.stale || s.unverifiable {
+		return nameRange{}, false
+	}
+	lo, hi := sl.sectionRange(idx)
+	return nameRange{lo: lo, hi: hi}, true
+}
+
 func (mc *MetaCache) refreshSection(ctx context.Context, client filer_pb.FilerClient, dirPath util.FullPath, lo, hi string) error {
 	_, err, _ := mc.visitGroup.Do(string(dirPath)+"\x00section\x00"+lo, func() (interface{}, error) {
 		if !mc.rangeStale(dirPath, lo) {
@@ -365,6 +401,25 @@ func (mc *MetaCache) applySectionRefreshNow(ctx context.Context, dirPath util.Fu
 		if mc.entryVersionBlocksLocked(ctx, entry.FullPath, snapshotTsNs, false) {
 			continue
 		}
+		// A hard-linked entry shares one content blob with its siblings: a
+		// snapshot older than the group's newest write would roll that
+		// content back for every path reading through this entry. A cached
+		// member of the same group is skipped outright; any other live entry
+		// at the path is a stale occupant of a reused name, so the listed
+		// name still lands — carrying the group's parked content rather
+		// than the snapshot's.
+		staleOccupant := false
+		if mc.hardLinkVersionFloorLocked(ctx, entry.HardLinkId) > snapshotTsNs {
+			if existing, findErr := mc.localStore.FindEntry(ctx, entry.FullPath); findErr == nil && existing != nil {
+				if !isTtlExpired(existing) && string(existing.HardLinkId) == string(entry.HardLinkId) {
+					continue
+				}
+				staleOccupant = len(existing.HardLinkId) != 0 && string(existing.HardLinkId) != string(entry.HardLinkId)
+			}
+			if rebased := mc.rebaseHardLinkEntryLocked(ctx, entry, ""); rebased != nil {
+				entry = rebased
+			}
+		}
 		// An unversioned marker would bypass the section floor, so it cannot
 		// outlive the snapshot write that replaces its content — but pinned
 		// local-only state stays authoritative and is not replaced at all.
@@ -374,10 +429,20 @@ func (mc *MetaCache) applySectionRefreshNow(ctx context.Context, dirPath util.Fu
 				continue
 			}
 		}
+		if staleOccupant {
+			// The name already left the occupant's group on the filer, and a
+			// later write to a sibling can have folded that unlink into the
+			// shared blob's counter — vacate the path like a rename so the
+			// insert does not decrement a group that may no longer count it.
+			if err := mc.localStore.DeleteEntry(context.WithValue(ctx, "OP", "MV"), entry.FullPath); err != nil {
+				return err
+			}
+		}
 		// no per-entry version: the section floor set below covers the range
 		if err := mc.localStore.InsertEntry(ctx, entry); err != nil {
 			return err
 		}
+		mc.noteHardLinkVersionLocked(ctx, entry.HardLinkId, snapshotTsNs)
 		if unversioned {
 			// only once the write landed: if it fails, the old content keeps
 			// the marker, and with it the right to be corrected by any event