readdir built dirPath.Child(name) for every child while entry.FullPath was
already that exact string, from NewFullPath in the meta cache store or from
FromPbEntry on the read-through path. One allocation per entry, and on a wide
tree with long paths that is most of what a listing allocates.
BenchmarkReadDirectory/kernel_readdirplus over 200k entries: 2,039,656 ->
1,839,318 allocs/op, 174.5 -> 167.8 MB/op, 152.6 -> 135.1 ms/op.
The cached readdir trims the head of the handle's entry stream as the
client walks past it; the read-through path never did, so a directory
too large to cache -- the only kind that takes that path -- was held
whole in the handle for the length of the walk. Hoist the trim to cover
both paths.
* mount: invalidate hot directory listings by section
A cached directory used to be dropped whole when it saw 64 changes in
2s: with a continuous writer the listing cycled through wipe, direct
listing and full rebuild for as long as the writer kept going, and
every sibling lookup fell through to the filer in between.
Split each cached listing into name-range sections of 1024 entries. A
burst of foreign changes invalidates just the section it lands in;
entries stay served and events keep applying, and the next readdir
re-lists only that range from the filer, reconciled through the version
gate so it cannot roll back newer applied events. Lookups in an
invalidated section read through until then. The mount's own writes no
longer invalidate anything: they are ground truth for its cache.
* meta_cache: drop the version floor with a deleted or moved directory
The other teardown paths already clear both maps; a floor left behind
here would fence the listing of a directory re-created at the same
path.
* mount: harden section refresh
An unversioned listing (pre-upgrade filer) now only fills gaps instead
of reconciling: without a snapshot to order against, an overwrite or
the deletion sweep could roll back an event applied after the listing.
The section table can be rebuilt or re-split between the listing and
its apply, so the refresh only marks fresh or splits when the section
still covers the range it read. Splicing bounds from a stale range
into a rebuilt table could leave them unsorted.
Bound the wait: a readdir gives a refresh five seconds before serving
the maintained-but-unverified cache. Bound the size: a range grown
past four sections aborts the refresh and drops the directory cache,
re-tiling it with a full rebuild, with that request served direct.
Cover the filer-facing path with a listing server: paging with the
snapshot pinned across pages, the section cutoff, no calls for a
fresh section, and the overgrown-range abort.
* meta_cache: make the section table a self-contained state machine
Churn counting, freshness, stale-range scanning and the refresh
completion with its guard and split now live on dirSections itself,
free of the lock, the store and the apply loop, so they test directly
with synthetic clocks and tables. MetaCache keeps thin wrappers that
hold its mutex and find the directory's table.
* meta_cache: keep section internals out of the apply request
The request now carries the completed build's table and one refresh as
opaque values built by section code, and the boundary-derivation rule
moves out of the build loop into a collector next to the rest of the
section logic.
* mount: fence refreshed sections with a snapshot floor
A refresh versioned the entries it fetched and tombstoned the ones it
swept, but a name absent from both cache and listing kept the old
directory floor, so a delayed event between the two snapshots could
resurrect it into a section already marked fresh. The section now
carries its own floor, consulted next to the directory floor, covering
every name in the range, present or absent — which also retires the
refresh's per-entry version stamps and sweep tombstones.
An unversioned listing sets no floor and vouches for nothing: it may
still fill gaps, but the section stays stale and reads through until a
filer that stamps snapshots re-validates it.
A listing's reach is unknowable up front — a resumed handle can skip
far ahead, and shrunken sections let one batch span many — so a
readdir now re-validates every stale section from its start name to
the end of the directory instead of the next two.
* mount: fence tombstoned names with floors and gate the reconcile
A tombstone answered for its name before the floors were consulted, so
one at an old position let through events the newer listing floor
should have fenced; a build never hit this because it prunes
superseded tombstones, which a section refresh does not. The version
gate now raises a tombstone to the floors like any other record.
With no per-entry versions, only the section floor fences a
reconcile's work, so a range the rebuilt or re-split table no longer
has must not touch the store either: the range check moves ahead of
the mutations, under the same lock the floor install holds.
An unversioned refresh no longer retries: the section is remembered as
unverifiable and skipped by the stale scan, or every batch of every
readdir would re-list the same ranges against a filer that cannot
vouch for them.
* mount: clear beaten unversioned markers and skip refresh mid-build
An unversioned marker outliving the snapshot write that replaced its
content bypassed the section floor the same way an old tombstone did,
letting a delayed pre-snapshot event roll the entry back. The refresh
now clears the marker when its write wins; pinned local-only entries
are not replaced at all, keeping their content and marker.
A rebuild wipes and repopulates the store off the apply loop, so a
refresh reconciling meanwhile could sweep children the build had
already inserted and let it publish the directory incomplete. The
refresh now skips a building directory, as events (buffered) and
purges (skipped) already do; its staleness dies with the build's
fresh table.
* mount: clear the unversioned marker only after its replacement lands
Clearing before the insert meant a failed write left the old local
content claiming the listing floors, fencing the very events that were
still entitled to correct it.
* meta_cache: rename the section state machine to sectionList
dirSections named both the type and the map of them.
* mount: raise the default cacheDirMaxEntries to 100000
The low ceiling guarded against whole-listing rebuild churn: a big
cached directory under writes kept re-streaming everything. Sectioned
invalidation ended that — a burst now costs one range listing — so the
remaining cost of caching a large directory is its one-time build,
comparable to the single direct listing that read-through mode pays on
every enumeration instead.
* meta_cache: cover section border and edge cases
A bound-named entry belongs to the section starting at the bound: the
neighboring refresh's sweep stops before it, its own section's covers
it. Churn past everything the build saw lands in the tail section, a
rename spanning two sections invalidates both, and a listed entry at
the section's end name is cut off with the ones beyond it.
Every enumeration of a directory walked the whole FUSE machinery, so
reopening a folder cost what opening it did. The kernel has a cache for
exactly this: with FOPEN_CACHE_DIR the listing lives in the directory's
page cache and a repeat enumeration never reaches the mount at all.
Local mutations already drop that cache in the kernel. Remote ones
arrive through the metadata subscription, so the entry invalidation
worker now also tells the kernel which directory changed. The worker is
the one place this is safe from: notifying from a thread serving a
kernel request can deadlock against the page it holds, which is why the
file paths deliberately avoid InodeNotify.
Measured in a Linux container, 20k-entry directory, ls repeated:
warm listing before 199-355ms after 6-9ms
A file written from outside the mount appeared in the next listing
within a second, through the subscription notify, and the listing
re-cached after.
The memory is the kernel's page cache: reclaimed under pressure, owned
per-directory, and covering read-through directories the mount-side
caches never see.
* mount: read oversized directories through instead of caching them
Visiting a directory pulls every child from the filer into the local
LevelDB before the first listing returns. For a directory of a few
million entries that is minutes of streaming, gigabytes of local store,
and gigabytes of decoded entries in flight -- paid by a mount that may
only walk the directory once.
A build that crosses -cacheDirMaxEntries (default ten thousand) now
stops, cleans up, and marks the directory read-through: listings stream
from the filer with pagination, the way update-hot directories already
do, and lookups in it consult the filer per entry as any uncached
directory does. The refusal is remembered, so the next visit fails fast
instead of streaming to the limit again, and an oversized ancestor is
stepped over when caching its subdirectories rather than wedging every
listing beneath it.
The direct path keeps the same pagination state on the handle, so a walk
that crosses the limit mid-flight carries on from where the cached walk
reached.
* mount: an ancestor found oversized must not fail its descendants
Visiting a directory builds its whole uncached ancestor chain in one
group, so the first discovery that an ancestor is oversized cancelled the
group and surfaced as the listed directory's own refusal: the descendant
build was aborted and the caller marked the descendant read-through,
leaving a perfectly cacheable directory streaming from the filer until
its inode was forgotten. The earlier test missed this by pre-marking the
ancestor, which exercises only the fast path.
The refusal of any directory other than the one being listed is now kept
out of the group's result; it is already remembered for the next visit.
* mount: size the direct listing slice from the batch, not the offset
The limit passed here is skipCount+batchSize when a client resumes a
fresh handle partway through a directory, so preallocating for it turns
the client's cookie into an allocation: a readdir at offset 3,000,000
reserves 24MB before the first entry arrives, and an offset near the
uint32 ceiling asks makeslice for ~4.29e9 elements.
* mount: stop replaying a directory that shrank past the resume offset
A client that opens a fresh handle and resumes at a cookie from an
earlier, larger listing gets a preload that cannot reach the entry before
that offset. The resume name was then left empty and the follow-up batch
listed from the directory's first child again, so the client was handed
every name a second time.
The stream always runs from the first child, so failing to reach that
entry means the directory is simply shorter than the offset. That is the
end of it.
* mount: page a directory from where the store reached
The batch loader treated a short batch as the end of the directory, but
the meta cache drops an expired child after the store has already spent
it against the limit, so a batch that filled up could still deliver fewer
entries than asked for. A directory with a handful of expired children
would stop listing early and hide every child behind them; a whole batch
of expired ones truncated the listing to nothing.
ListDirectoryEntries now reports the name the store itself reached. That
is both the sound end-of-directory signal -- the store returning nothing
-- and the right cursor, since resuming from the last visible name would
re-read the dropped children every round and never get past a batch that
was entirely expired.
* mount: drop the entries a directory walk has already passed
entryStreamOffset was only ever written by reset, so the stream a handle
holds grew for the life of the walk and a directory was retained whole
even though nothing could read the entries behind the client's position
again. A 10M-entry walk parked millions of entries per handle, and
NFS-Ganesha opens several on the same directory.
Offsets index into the stream from entryStreamOffset, so advancing the
two together keeps them lined up. One entry is held back because the next
batch resumes from the name immediately before the offset. Seeking back
behind what is still held now restarts the directory, which is what the
offset scheme can honestly support -- it previously returned nothing.
* filer: decode a listed entry without building its chunk list
A readdir reads attributes and never looks at chunks, but decoding an
entry builds the whole chunk list first: four allocations per chunk, all
of it thrown away. On a directory of ordinary 4MB-chunked files that is
most of what listing costs.
DecodeAttributesOnly walks the wire format and hands everything except
the chunks to the generated unmarshaller, so new fields in filer.proto
need no attention here. The chunks are still measured, because the S3
copy and multipart paths deliberately store a zero FileSize and let the
chunk extents define the size, but nothing is allocated to do it.
The blob is only re-encoded once a chunk is actually seen, so an entry
without any -- every directory, for one -- is unmarshalled where it lies
and pays nothing for the walk.
Listings opt in through the context, the way the lazy remote paths
already do; a store that ignores it stays correct.
chunks full attrs-only allocs
0 312.8n 310.1n ~ 1 -> 1
1 686.1n 411.1n -40.07% 7 -> 1
4 1.742u 667.4n -61.69% 24 -> 1
16 5.770u 1.544u -73.25% 86 -> 1
64 25.23u 6.004u -76.20% 328 -> 1
* mount: list directories with chunk lists omitted
The two meta cache listings behind a readdir are the only callers, and
neither reads a chunk. On 200k single-chunk files one enumeration goes
from 364ms to 277ms and drops a million allocations.
The read-through listing still fetches whole entries from the filer,
which would need the request to say it wants attributes only.
* mount: give the readdir benchmark's entries a chunk
Chunkless entries made the decode look far cheaper than it is, which is
the part of a listing worth measuring.
* filer: let a listing ask for entries without their chunk lists
The read-through readdir fetches whole entries over gRPC, and for a wide
directory the chunk lists are most of what crosses the wire and most of
what the client then unmarshals. A 4MB-chunked file is 113 bytes of
entry against 46 without its chunk.
ListEntriesRequest gains omit_chunks. The size a client needs is already
in the attributes, where the store decode folded the chunk extents in,
so dropping the list costs the client nothing.
The filer still reads the entries whole. A listing is where a TTL-expired
entry gets collected and deleted, and deleting one needs its chunks to
find the data, so omitting them there would leak. Only the response is
trimmed.
The hint moves to filer_pb so one context flag serves both transports:
the gRPC request sets omit_chunks, and a listing served from the local
store skips building the chunks. Cache population is unaffected either
way, since EnsureVisited starts from its own context.
* filer: reject a chunk the full decoder would reject
The walk skipped a chunk's bytes without looking inside them, so a
FileChunk carrying a corrupt nested fid, or a string that is not valid
UTF-8, sailed past the listing decoder while every other read of the same
entry still failed. The file listed with a plausible size and then gave
EIO on open, and corruption that used to fail the listing loudly was
hidden instead.
The chunk bytes are the one part of the blob the generated unmarshaller
never sees, so the two checks it would have made are made here: a
submessage has to parse, and a proto3 string has to be valid UTF-8.
FileChunk's only submessages are FileIds of scalars, so walking them is a
complete check. A descriptor-driven test fails if FileChunk ever gains a
field of either kind that the walk does not know to check, which is the
part that keeps this honest as filer.proto grows.
Taking the scratch buffer lazily, only once a chunk is actually dropped,
also takes the pool out of the path for entries that have none. Those
were measurably slower than the full decoder before; they are now level
with it. Each chunk's length prefix is parsed once rather than twice.
chunks full attrs-only vs base
0 171.4n 176.9n ~ (p=0.670)
1 366.6n 259.4n -29.24%
4 1.034u 500.2n -51.60%
16 3.905u 1.464u -62.52%
64 13.48u 5.195u -61.46%
* filer: carry the size before dropping chunks over the wire
Dropping the chunk list assumed every store folds the chunk extents into
FileSize when it decodes. A store that keeps entries as JSON rather than
as an encoded Entry never re-derives it, so an object written with a zero
FileSize kept its real size only in the chunks, and stripping them left
the client reading the file as empty. Stamp the size into the attributes
first, which costs nothing and does not depend on how the store loaded
the entry.
* mount: test that the readdir context reaches the store decode
Everything else exercises the decoder directly, so a refactor that
stopped threading the context would have reverted the whole thing with
every test still passing.
The benchmark's chunks also carried a constant legacy FileId, which
BeforeEntrySerialization reparses over Fid on the way in, so all 200k
entries stored one byte-identical chunk rather than the varying fixture
it looked like.
* mount: readdir enters a child in the inode table only when it takes a reference
Only readdirplus into the kernel takes a reference on the children it
reports, and only that reference brings a FORGET later to take the entry
back out. Every other listing was inserting all its children anyway.
On WinFsp that meant a listing looked each child up, took a reference,
and immediately gave it back, so a walk of a wide directory paid three
write-lock acquisitions per entry to leave the table exactly as it found
it. On a plain kernel readdir nothing gives the entry back at all, so
listing a directory of 200k files grew both maps by 200k entries that
were never reclaimed.
A dirent's inode number is informational either way: the kernel must
LOOKUP before it can use a nodeid, and the WinFsp adapter re-resolves
every operation by path. So report the number and let the mapping be
built when something actually looks the entry up.
* mount: take the readdirplus reference without a second full lookup
The entry has just been resolved a few lines above, so redoing the whole
lookup only rebuilds the child path and walks both maps again to reach a
counter. Bump it directly, falling back to the full lookup if a Forget
removed the entry in between.
* mount: benchmark a readdir over a 200k directory
Drives doReadDirectory against a meta cache holding 200k entries, one
round of 4096 at a time, for the three front ends that behave
differently: a plain kernel readdir, kernel readdirplus, and a WinFsp
listing that gets attributes but never returns a reference. Reports what
each leaves behind in the inode table alongside the usual metrics.
The sink declares TakesLookupRef as an ordinary method rather than
through the interface, so the same file runs unchanged against an older
tree for comparison.
* mount: stamp an inode on the benchmark's entries
The filer stores one on every entry it writes, so a real listing arrives
with an inode and never derives its own. Leaving it zero made every child
in the benchmark fall through to the MD5 in AsInode, work no filer-backed
mount does, and charged it to both sides of the comparison.
* mount: drop the unused go-fuse fs package dependency
WFS embedded fs.Inode but never used any of its methods, and the only
other reference was RENAME_EXCHANGE, a constant sitting next to three
literals. Removing both drops fs and five internal packages from the
mount build graph.
* mount: build the package on windows
Windows has no fcntl lock types, no O_ACCMODE and no x/sys/unix, so a
handful of constants kept weed/mount pinned to unix even though the code
using them is portable in-memory logic. Route them through per-OS shims
and give setBlksize a windows no-op.
The POSIX lock table now compiles on windows but stays unreachable:
WinFsp resolves byte-range locks in its own kernel driver, so nothing
will feed it there.
go.mod points at a go-fuse branch commit and needs repinning to a release
tag once that lands.
* ci: cross-compile for windows
Nothing caught the unix-only constants creeping into weed/mount until a
release build failed.
* mount: let readdir feed a sink instead of the kernel buffer
doReadDirectory wrote directly into fuse.DirEntryList, which is the
kernel's wire format. A front end that is not the kernel would have to
pack entries only to parse them straight back out.
Route it through DirEntrySink instead. ReadDir and ReadDirPlus pass the
reply buffer, so nothing changes for the FUSE server.
* mount: pin go-fuse v2.9.4 for the windows build
* mount: add option to show system entries
* address gemini code review's suggested changes
* rename flag from -showSystemEntries to -includeSystemEntries
* meta_cache: purge hidden system entries on filer events
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: expose metadata events and list snapshots
* mount: invalidate hot directory caches
* mount: read hot directories directly from filer
* mount: add sequenced metadata cache applier
* mount: apply metadata responses through cache applier
* mount: replay snapshot-consistent directory builds
* mount: dedupe self metadata events
* mount: factor directory build cleanup
* mount: replace proto marshal dedup with composite key and ring buffer
The dedup logic was doing a full deterministic proto.Marshal on every
metadata event just to produce a dedup key. Replace with a cheap
composite string key (TsNs|Directory|OldName|NewName).
Also replace the sliding-window slice (which leaked the backing array
unboundedly) with a fixed-size ring buffer that reuses the same array.
* filer: remove mutex and proto.Clone from request-scoped MetadataEventSink
MetadataEventSink is created per-request and only accessed by the
goroutine handling the gRPC call. The mutex and double proto.Clone
(once in Record, once in Last) were unnecessary overhead on every
filer write operation. Store the pointer directly instead.
* mount: skip proto.Clone for caller-owned metadata events
Add ApplyMetadataResponseOwned that takes ownership of the response
without cloning. Local metadata events (mkdir, create, flush, etc.)
are freshly constructed and never shared, so the clone is unnecessary.
* filer: only populate MetadataEvent on successful DeleteEntry
Avoid calling eventSink.Last() on error paths where the sink may
contain a partial event from an intermediate child deletion during
recursive deletes.
* mount: avoid map allocation in collectDirectoryNotifications
Replace the map with a fixed-size array and linear dedup. There are
at most 3 directories to notify (old parent, new parent, new child
if directory), so a 3-element array avoids the heap allocation on
every metadata event.
* mount: fix potential deadlock in enqueueApplyRequest
Release applyStateMu before the blocking channel send. Previously,
if the channel was full (cap 128), the send would block while holding
the mutex, preventing Shutdown from acquiring it to set applyClosed.
* mount: restore signature-based self-event filtering as fast path
Re-add the signature check that was removed when content-based dedup
was introduced. Checking signatures is O(1) on a small slice and
avoids enqueuing and processing events that originated from this
mount instance. The content-based dedup remains as a fallback.
* filer: send snapshotTsNs only in first ListEntries response
The snapshot timestamp is identical for every entry in a single
ListEntries stream. Sending it in every response message wastes
wire bandwidth for large directories. The client already reads
it only from the first response.
* mount: exit read-through mode after successful full directory listing
MarkDirectoryRefreshed was defined but never called, so directories
that entered read-through mode (hot invalidation threshold) stayed
there permanently, hitting the filer on every readdir even when cold.
Call it after a complete read-through listing finishes.
* mount: include event shape and full paths in dedup key
The previous dedup key only used Names, which could collapse distinct
rename targets. Include the event shape (C/D/U/R), source directory,
new parent path, and both entry names so structurally different events
are never treated as duplicates.
* mount: drain pending requests on shutdown in runApplyLoop
After receiving the shutdown sentinel, drain any remaining requests
from applyCh non-blockingly and signal each with errMetaCacheClosed
so callers waiting on req.done are released.
* mount: include IsDirectory in synthetic delete events
metadataDeleteEvent now accepts an isDirectory parameter so the
applier can distinguish directory deletes from file deletes. Rmdir
passes true, Unlink passes false.
* mount: fall back to synthetic event when MetadataEvent is nil
In mknod and mkdir, if the filer response omits MetadataEvent (e.g.
older filer without the field), synthesize an equivalent local
metadata event so the cache is always updated.
* mount: make Flush metadata apply best-effort after successful commit
After filer_pb.CreateEntryWithResponse succeeds, the entry is
persisted. Don't fail the Flush syscall if the local metadata cache
apply fails — log and invalidate the directory cache instead.
Also fall back to a synthetic event when MetadataEvent is nil.
* mount: make Rename metadata apply best-effort
The rename has already succeeded on the filer by the time we apply
the local metadata event. Log failures instead of returning errors
that would be dropped by the caller anyway.
* mount: make saveEntry metadata apply best-effort with fallback
After UpdateEntryWithResponse succeeds, treat local metadata apply
as non-fatal. Log and invalidate the directory cache on failure.
Also fall back to a synthetic event when MetadataEvent is nil.
* filer_pb: preserve snapshotTsNs on error in ReadDirAllEntriesWithSnapshot
Return the snapshot timestamp even when the first page fails, so
callers receive the snapshot boundary when partial data was received.
* filer: send snapshot token for empty directory listings
When no entries are streamed, send a final ListEntriesResponse with
only SnapshotTsNs so clients always receive the snapshot boundary.
* mount: distinguish not-found vs transient errors in lookupEntry
Return fuse.EIO for non-not-found filer errors instead of
unconditionally returning ENOENT, so transient failures don't
masquerade as missing entries.
* mount: make CacheRemoteObject metadata apply best-effort
The file content has already been cached successfully. Don't fail
the read if the local metadata cache update fails.
* mount: use consistent snapshot for readdir in direct mode
Capture the SnapshotTsNs from the first loadDirectoryEntriesDirect
call and store it on the DirectoryHandle. Subsequent batch loads
pass this stored timestamp so all batches use the same snapshot.
Also export DoSeaweedListWithSnapshot so mount can use it directly
with snapshot passthrough.
* filer_pb: fix test fake to send SnapshotTsNs only on first response
Match the server behavior: only the first ListEntriesResponse in a
page carries the snapshot timestamp, subsequent entries leave it zero.
* Fix nil pointer dereference in ListEntries stream consumers
Remove the empty-directory snapshot-only response from ListEntries
that sent a ListEntriesResponse with Entry==nil, which crashed every
raw stream consumer that assumed resp.Entry is always non-nil.
Also add defensive nil checks for resp.Entry in all raw ListEntries
stream consumers across: S3 listing, broker topic lookup, broker
topic config, admin dashboard, topic retention, hybrid message
scanner, Kafka integration, and consumer offset storage.
* Add nil guards for resp.Entry in remaining ListEntries stream consumers
Covers: S3 object lock check, MQ management dashboard (version/
partition/offset loops), and topic retention version loop.
* Make applyLocalMetadataEvent best-effort in Link and Symlink
The filer operations already succeeded; failing the syscall because
the local cache apply failed is wrong. Log a warning and invalidate
the parent directory cache instead.
* Make applyLocalMetadataEvent best-effort in Mkdir/Rmdir/Mknod/Unlink
The filer RPC already committed; don't fail the syscall when the
local metadata cache apply fails. Log a warning and invalidate the
parent directory cache to force a re-fetch on next access.
* flushFileMetadata: add nil-fallback for metadata event and best-effort apply
Synthesize a metadata event when resp.GetMetadataEvent() is nil
(matching doFlush), and make the apply best-effort with cache
invalidation on failure.
* Prevent double-invocation of cleanupBuild in doEnsureVisited
Add a cleanupDone guard so the deferred cleanup and inline error-path
cleanup don't both call DeleteFolderChildren/AbortDirectoryBuild.
* Fix comment: signature check is O(n) not O(1)
* Prevent deferred cleanup after successful CompleteDirectoryBuild
Set cleanupDone before returning from the success path so the
deferred context-cancellation check cannot undo a published build.
* Invalidate parent directory caches on rename metadata apply failure
When applyLocalMetadataEvent fails during rename, invalidate the
source and destination parent directory caches so subsequent accesses
trigger a re-fetch from the filer.
* Add event nil-fallback and cache invalidation to Link and Symlink
Synthesize metadata events when the server doesn't return one, and
invalidate parent directory caches on apply failure.
* Match requested partition when scanning partition directories
Parse the partition range format (NNNN-NNNN) and match against the
requested partition parameter instead of using the first directory.
* Preserve snapshot timestamp across empty directory listings
Initialize actualSnapshotTsNs from the caller-requested value so it
isn't lost when the server returns no entries. Re-add the server-side
snapshot-only response for empty directories (all raw stream consumers
now have nil guards for Entry).
* Fix CreateEntry error wrapping to support errors.Is/errors.As
Use errors.New + %w instead of %v for resp.Error so callers can
unwrap the underlying error.
* Fix object lock pagination: only advance on non-nil entries
Move entriesReceived inside the nil check so nil entries don't
cause repeated ListEntries calls with the same lastFileName.
* Guard Attributes nil check before accessing Mtime in MQ management
* Do not send nil-Entry response for empty directory listings
The snapshot-only ListEntriesResponse (with Entry == nil) for empty
directories breaks consumers that treat any received response as an
entry (Java FilerClient, S3 listing). The Go client-side
DoSeaweedListWithSnapshot already preserves the caller-requested
snapshot via actualSnapshotTsNs initialization, so the server-side
send is unnecessary.
* Fix review findings: subscriber dedup, invalidation normalization, nil guards, shutdown race
- Remove self-signature early-return in processEventFn so all events
flow through the applier (directory-build buffering sees self-originated
events that arrive after a snapshot)
- Normalize NewParentPath in collectEntryInvalidations to avoid duplicate
invalidations when NewParentPath is empty (same-directory update)
- Guard resp.Entry.Attributes for nil in admin_server.go and
topic_retention.go to prevent panics on entries without attributes
- Fix enqueueApplyRequest race with shutdown by using select on both
applyCh and applyDone, preventing sends after the apply loop exits
- Add cleanupDone check to deferred cleanup in meta_cache_init.go for
clarity alongside the existing guard in cleanupBuild
- Add empty directory test case for snapshot consistency
* Propagate authoritative metadata event from CacheRemoteObjectToLocalCluster and generate client-side snapshot for empty directories
- Add metadata_event field to CacheRemoteObjectToLocalClusterResponse
proto so the filer-emitted event is available to callers
- Use WithMetadataEventSink in the server handler to capture the event
from NotifyUpdateEvent and return it on the response
- Update filehandle_read.go to prefer the RPC's metadata event over
a locally fabricated one, falling back to metadataUpdateEvent when
the server doesn't provide one (e.g., older filers)
- Generate a client-side snapshot cutoff in DoSeaweedListWithSnapshot
when the server sends no snapshot (empty directory), so callers like
CompleteDirectoryBuild get a meaningful boundary for filtering
buffered events
* Skip directory notifications for dirs being built to prevent mid-build cache wipe
When a metadata event is buffered during a directory build,
applyMetadataSideEffects was still firing noteDirectoryUpdate for the
building directory. If the directory accumulated enough updates to
become "hot", markDirectoryReadThrough would call DeleteFolderChildren,
wiping entries that EnsureVisited had already inserted. The build would
then complete and mark the directory cached with incomplete data.
Fix by using applyMetadataSideEffectsSkippingBuildingDirs for buffered
events, which suppresses directory notifications for dirs currently in
buildingDirs while still applying entry invalidations.
* Add test for directory notification suppression during active build
TestDirectoryNotificationsSuppressedDuringBuild verifies that metadata
events targeting a directory under active EnsureVisited build do NOT
fire onDirectoryUpdate for that directory. In production, this prevents
markDirectoryReadThrough from calling DeleteFolderChildren mid-build,
which would wipe entries already inserted by the listing.
The test inserts an entry during a build, sends multiple metadata events
for the building directory, asserts no notifications fired for it,
verifies the entry survives, and confirms buffered events are replayed
after CompleteDirectoryBuild.
* Fix create invalidations, build guard, event shape, context, and snapshot error path
- collectEntryInvalidations: invalidate FUSE kernel cache on pure
create events (OldEntry==nil && NewEntry!=nil), not just updates
and deletes
- completeDirectoryBuildNow: only call markCachedFn when an active
build existed (state != nil), preventing an unpopulated directory
from being marked as cached
- Add metadataCreateEvent helper that produces a create-shaped event
(NewEntry only, no OldEntry) and use it in mkdir, mknod, symlink,
and hardlink create fallback paths instead of metadataUpdateEvent
which incorrectly set both OldEntry and NewEntry
- applyMetadataResponseEnqueue: use context.Background() for the
queued mutation so a cancelled caller context cannot abort the
apply loop mid-write
- DoSeaweedListWithSnapshot: move snapshot initialization before
ListEntries call so the error path returns the preserved snapshot
instead of 0
* Fix review findings: test loop, cache race, context safety, snapshot consistency
- Fix build test loop starting at i=1 instead of i=0, missing new-0.txt verification
- Re-check IsDirectoryCached after cache miss to avoid ENOENT race with markDirectoryReadThrough
- Use context.Background() in enqueueAndWait so caller cancellation can't abort build/complete mid-way
- Pass dh.snapshotTsNs in skip-batch loadDirectoryEntriesDirect for snapshot consistency
- Prefer resp.MetadataEvent over fallback in Unlink event derivation
- Add comment on MetadataEventSink.Record single-event assumption
* Fix empty-directory snapshot clock skew and build cancellation race
Empty-directory snapshot: Remove client-side time.Now() synthesis when
the server returns no entries. Instead return snapshotTsNs=0, and in
completeDirectoryBuildNow replay ALL buffered events when snapshot is 0.
This eliminates the clock-skew bug where a client ahead of the filer
would filter out legitimate post-list events.
Build cancellation: Use context.Background() for BeginDirectoryBuild
and CompleteDirectoryBuild calls in doEnsureVisited, so errgroup
cancellation doesn't cause enqueueAndWait to return early and trigger
cleanupBuild while the operation is still queued.
* Add tests for empty-directory build replay and cancellation resilience
TestEmptyDirectoryBuildReplaysAllBufferedEvents: verifies that when
CompleteDirectoryBuild receives snapshotTsNs=0 (empty directory, no
server snapshot), ALL buffered events are replayed regardless of their
TsNs values — no clock-skew-sensitive filtering occurs.
TestBuildCompletionSurvivesCallerCancellation: verifies that once
CompleteDirectoryBuild is enqueued, a cancelled caller context does not
prevent the build from completing. The apply loop runs with
context.Background(), so the directory becomes cached and buffered
events are replayed even when the caller gives up waiting.
* Fix directory subtree cleanup, Link rollback, test robustness
- applyMetadataResponseLocked: when a directory entry is deleted or
moved, call DeleteFolderChildren on the old path so cached descendants
don't leak as stale entries.
- Link: save original HardLinkId/Counter before mutation. If
CreateEntryWithResponse fails after the source was already updated,
rollback the source entry to its original state via UpdateEntry.
- TestBuildCompletionSurvivesCallerCancellation: replace fixed
time.Sleep(50ms) with a deadline-based poll that checks
IsDirectoryCached in a loop, failing only after 2s timeout.
- TestReadDirAllEntriesWithSnapshotEmptyDirectory: assert that
ListEntries was actually invoked on the mock client so the test
exercises the RPC path.
- newMetadataEvent: add early return when both oldEntry and newEntry are
nil to avoid emitting events with empty Directory.
---------
Co-authored-by: Copilot <copilot@github.com>
* mount: refresh and evict hot dir cache
* mount: guard dir update window and extend TTL
* mount: reuse timestamp for cache mark
* Apply suggestion from @gemini-code-assist[bot]
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* mount: make dir cache tuning configurable
* mount: dedupe dir update notices
* mount: restore invalidate-all cache helper
* mount: keep hot dir tuning constants
* mount: centralize cache state reset
* mount: mark refresh completion time
* mount: allow disabling idle eviction
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Enable writeback_cache and async_dio FUSE options
Fixes#7978
- Update mount_std.go to use EnableWriteback and EnableAsyncDio from go-fuse
- Add go.mod replace directive to use local go-fuse with capability support
- Remove temporary workaround that disabled these options
This enables proper FUSE kernel capability negotiation for writeback cache
and async direct I/O, improving performance for small writes and concurrent
direct I/O operations.
* Address PR review comments
- Remove redundant nil checks for writebackCache and asyncDio flags
- Update go.mod replace directive to use seaweedfs/go-fuse fork instead of local path
* Add TODO comment for go.mod replace directive
The replace directive must use a local path until seaweedfs/go-fuse#1 is merged.
After merge, this should be updated to use the proper version.
* Use seaweedfs/go-fuse v2.9.0 instead of local repository
Replace local path with seaweedfs/go-fuse v2.9.0 fork which includes
the writeback_cache and async_dio capability support.
* Use github.com/seaweedfs/go-fuse/v2 directly without replace directive
- Updated all imports to use github.com/seaweedfs/go-fuse/v2
- Removed replace directive from go.mod
- Using seaweedfs/go-fuse v2.0.0-20260106181308-87f90219ce09 which includes:
* writeback_cache and async_dio support
* Corrected module path
* Update to seaweedfs/go-fuse v2.9.1
Use v2.9.1 tag which includes the corrected module path
(github.com/seaweedfs/go-fuse/v2) along with writeback_cache
and async_dio support.
mount: remove unused isEarlyTerminated variable
The variable was redundant because when processEachEntryFn returns false,
we immediately return fuse.OK, so the check was always false.
* mount: fix weed inode nlookup do not equel kernel inode nlookup
* mount: add underflow protection for nlookup decrement in Forget
* mount: use consistent == 0 check for uint64 nlookup
* Update weed/mount/inode_to_path.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* mount: snapshot data before unlock in Forget to avoid using deleted InodeEntry
---------
Co-authored-by: chrislu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* mount: add mutex to DirectoryHandle to fix race condition
When using Ganesha NFS on top of FUSE mount, ls operations would hang
forever on directories with hundreds of files. This was caused by a
race condition in DirectoryHandle where multiple concurrent readdir
operations could modify shared state (entryStream, entryStreamOffset,
isFinished) without synchronization.
The fix adds a mutex to DirectoryHandle and holds it for the entire
duration of doReadDirectory. This serializes concurrent readdir calls
on the same handle, which is the correct behavior for a directory
handle and fixes the race condition.
Key changes:
- Added sync.Mutex to DirectoryHandle struct
- Lock the mutex at the start of doReadDirectory
- This ensures thread-safe access to entryStream and other state
The lock is per-handle (not global), so different directories can
still be listed concurrently. Only concurrent operations on the
same directory handle are serialized.
Fixes: https://github.com/seaweedfs/seaweedfs/issues/7672
* mount: add mutex to DirectoryHandle to fix race condition
When using Ganesha NFS on top of FUSE mount, ls operations would hang
forever on directories with hundreds of files. This was caused by a
race condition in DirectoryHandle where multiple concurrent readdir
operations could modify shared state (entryStream, entryStreamOffset,
isFinished) without synchronization.
The fix adds a mutex to DirectoryHandle and holds it for the entire
duration of doReadDirectory. This serializes concurrent readdir calls
on the same handle, which is the correct behavior for a directory
handle and fixes the race condition.
Key changes:
- Added sync.Mutex to DirectoryHandle struct
- Lock the mutex at the start of doReadDirectory
- Optimized reset() to reuse slice capacity and allow GC of old entries
The lock is per-handle (not global), so different directories can
still be listed concurrently. Only concurrent operations on the
same directory handle are serialized.
Fixes: https://github.com/seaweedfs/seaweedfs/issues/7672
Chris Lu