* refactor: extract remote mount resolution into shared helpers
* refactor: share the adaptive remote cache wait policy
* filer: stream cold remote reads from the origin while caching
* s3: stream cold remote reads from the origin instead of 503 retries
* test: cover the S3 origin stream-through path
* remote mounts: match on path components and prefer the longest mount
* fail short origin streams instead of silently truncating
* s3: try the origin before failing a cold read on a local cache error
* s3: gate origin streaming on the entry's resolved version
* return the cache RPC's NotFound as a canonical status and classify it everywhere
* filer: keep multipart-range cold reads on the retry path
* filer: stamp a log position on lookup and remote-cache responses
Metadata events are logged after their store write and stamped with the
filer clock. Reading that clock before serving an entry therefore gives
a timestamp with a causal guarantee: every event at or below it is
reflected in the returned entry. Clients caching filer state can use it
as the entry's version to order the response against subscription
events, including events committed before the call but delivered after
it.
* mount: version open file handles by filer log position
A subscription event refreshing an open handle did a second lookup; a
transient failure left the handle pinned to its old entry with no
retry, since the subscription cursor had already advanced. The deeper
problem is ordering: the handle is a cache written by three unordered
channels — the async invalidation worker, local mutation acks, and
open-time lookups — and overwriting cached state safely requires
knowing which write is newer.
The filer log timestamp is that order, and it now travels with every
value instead of being derived out of band. Events carry it natively;
lookup and remote-cache responses carry the log position stamped before
the serving read; mutation acks carry it in their returned event; and
the local store pairs each read with a version cursor advanced under
the same lock as the store write. Each handle records the version its
entry reflects, and one rule replaces the per-site reasoning: state at
or below the handle's version is old news and must not be installed.
The invalidation itself applies the event's own entry — no lookup, so
no transient-failure window — except under a cached parent, where the
store entry is the ordered merge of the event and anything applied
since, and its version outranks the event's. An uncached parent
receives no store writes, so a hit there would be a stale leftover
masking the event. A vacated path (delete, rename away) keeps the last
entry so unlinked-but-open reads still work. Directory builds version
the completed directory at the listing snapshot and re-invalidate
buffered events at that version, since their mid-build refresh ran
against an incomplete store.
The tests replay every race this replaces machinery for: rollback of a
newer local flush (queued, cached, and read-through), stale leftovers
under uncached parents, the build window including abort, handles
opened after an event was queued, events landing mid-lookup, and
undelivered events at remote-cache time across a filer failover.
* filer: serialize the log position fence with mutations, stamp mutation acks
The fence stamped before an unlocked entry read could precede state the
read returned: a mutation writes storage first and assigns its event
timestamp only at notify time, so a lookup racing that window handed
the mount an entry newer than its fence, and the event's later delivery
looked like fresh news — destroying dirty pages for a change the handle
already had. The mutation handlers already hold an exclusive per-path
lock across read, write, and notify; the lookup and remote-cache reads
now take it shared around the stamp and the read, making the fence
exact: everything at or below it is in the entry, nothing above it is.
A no-change update returns success without an event, leaving the mount
nothing to fence with even though the response confirms current state.
Create and update acks now carry a log position stamped under the same
lock, and the mount falls back to it whenever the ack has no event.
Also regenerate the VT marshalers, which the earlier generation missed:
without them a VT round-trip silently zeroed every log position.
* java: sync filer.proto
* mount: scope store versions to what they vouch for; atomic handle install
The store's version cursor claimed too much. Advanced by local mutation
acks and directory listing snapshots, it inflated the version of store
reads for unrelated paths whose events the subscription still owed, and
those events were then fenced out permanently. The cursor now tracks
subscription progress only — events arrive in log order, so everything
at or below it has been delivered for every path — and a completed
listing records its snapshot as a per-directory floor instead of a
global claim. Local acks never touch it: they version their own handle
directly. Buffered build events advance the cursor at delivery, since
their store write may never happen (abort) while their invalidation is
already queued; their read-through directory pairs no store read with
it, and rename fragments are applied first.
Concurrent first opens raced: a slower opener's older lookup could
overwrite the newer entry a faster opener had installed, while the
monotonic version kept the newer timestamp — an old entry fenced at a
new version, immune to every correcting event. Entry and version are
now installed as one decision under the handle map lock, and an install
that does not outrank the handle's version is dropped.
The remote-cache commit also escaped the fence: it wrote storage and
notified without the path lock, so a lookup's shared-locked fence and
read could land between the two and hand out the cached state
under-versioned. The commit now re-reads and writes under the exclusive
path lock, and backs off entirely when the entry changed during the
download — the concurrent writer supersedes the cached content.
* mount: floors gate store applies; installs respect handle users; renames join the fence
A directory floor certifies the listing state as of its snapshot, but a
delayed event at or below the floor was still applied to the store —
rolling the content back to pre-snapshot state while the floor kept
claiming the snapshot version, so the correcting events were fenced out
of every future read. Events are now gated against the affected
directory's floor, each half of a rename independently.
Fences are lower bounds: a listing or lookup can include a mutation
whose event has not been delivered yet, and that event later passes
every gate carrying state the handle already holds. Such a re-delivery
now advances the version without destroying dirty pages or reinstalling
the entry — invalidating local writes over a no-op was the real damage
in every remaining under-fence window, including the unlocked listing
snapshot, which no per-path lock can serialize.
The concurrent-open install moved from the map lock to the handle lock
every reader, writer, and invalidation synchronizes on, and rejects
what cannot improve the handle: dirty state (local writes would be
lost), unversioned lookup responses (they cannot outrank anything, and
two zero-version opens must not overwrite each other), and anything not
strictly newer. New handles are still fully initialized before the map
exposes them.
Renames committed metadata and emitted events with no path lock, so a
lookup could read the renamed state under a fence preceding its events.
Both rename handlers now hold the source and destination locks, ordered
by path, across commit and notification; descendants of a renamed
directory are not individually locked and rely on the no-op re-delivery
handling above.
* mount: per-entry store versions replace the cursor and directory floors
The store's aggregate versions — a global subscription cursor and
per-directory listing floors — were versions at coarser granularity
than the values they described, and every over-claiming bug in this
series traced to that gap: an aggregate vouching for state its source
never saw. Each store entry now carries the filer log position of the
write that produced it — the event that applied it, or the listing
snapshot that inserted it, recorded in the store's key-value space
under the same lock as the entry write. The store becomes what the
handle already is: a last-writer-wins register with one rule, install
only what outranks the current claim.
The cursor, the floors, their advancement rules, the pairing ordering
constraint, and the floor gating all collapse into that rule. Applies
are gated per entry, each half of a rename independently; an
unversioned local write clears the claim its content no longer proves;
version records lingering after a bulk folder wipe cannot fence a
recreate, since a claim only blocks while its entry exists. Listing
inserts are stamped at build completion, before the buffered replay so
newer replayed events override the stamp.
Filer side, the fence dance every versioned read must perform is now a
single choke point, fencedFindEntry, so a future read RPC gets the
lock-serialized stamp by construction rather than by convention.
* mount: judge no-op re-deliveries against an immutable base, not the live entry
The equal-state skip compared the incoming event to the live handle
entry, but local writes mutate the live entry — size, timestamps,
chunks — so a delayed event re-delivering the base the handle was
opened with no longer matched, and the installer destroyed the dirty
pages and rolled the entry back over nothing new. The handle now keeps
an immutable snapshot of the filer state it last installed or
acknowledged, refreshed at every install and mutation ack (flush acks
snapshot the request entry before the id mapping mutates it), and the
no-op judgment runs against that base: an event carrying the base
brings nothing, whatever the live entry has diverged to since.
* mount: tombstones for versioned deletes, absence floors, copy enrollment
Four gaps in the per-entry version protocol, all the same shape: a
versioned fact with nothing carrying its version.
A deletion is a fact about a path with no entry left to hold it —
clearing the record let a delayed older event resurrect the deleted
path, permanently, since the deletion's own redelivery is
dedup-suppressed. Versioned deletes now leave a tombstone record that
fences without an entry; renames tombstone their source the same way.
Plain records still only block while their entry exists, so records
lingering after a bulk folder wipe cannot fence a recreate.
A completed listing proves absences as well as presences: a name it
omitted was deleted as of the snapshot, and a delayed create below the
snapshot re-creates it. The snapshot is kept per directory strictly as
an absence fence, consulted only when a path has neither an entry nor
a version record — present entries carry their own versions and never
touch it, which is what separates this from the over-claiming floor it
replaces.
A rebuild against a pre-upgrade filer returns no snapshot; stamping
now clears the children's records in that case, so a reinserted entry
cannot reactivate the stale claim its previous incarnation left
behind and reject valid events below it.
Server-side copies installed the copied entry without enrolling in the
base protocol, so the copy's own event differed from the stale
pre-copy base and destroyed writes made to the destination after the
copy. The install now refreshes the base and takes its version from
the fenced readback.
* mount: deletion facts outlive the cache's knowledge of the entry
A versioned delete of a path the store held no entry for recorded
nothing, so a delayed older event recreated the path — permanently,
with the deletion's redelivery dedup-suppressed. The tombstone is now
written whenever a versioned event vacates a path: the deletion is a
fact about the path, not about what this cache happened to hold.
For an absent entry, the listing's absence floor now speaks whatever
older record remains: a tombstone at one position does not exhaust
what is known about the path when a newer snapshot has confirmed the
name still absent, and an event between the two was slipping past
both.
A committed copy whose readback failed installed a synthesized base
with local timestamps; the copy's real event legitimately differs from
it, and was read as foreign state — destroying writes made to the
destination after the copy. The handle now marks that its own event is
en route and adopts that event's state as the base without touching
the live entry or the dirty pages; the adoption is one-shot, so a
genuinely foreign event still invalidates.
* mount: authoritative acks cancel pending event adoption; tombstones scoped and pruned
The copy-event adoption flag could outlive its purpose: a flush after
the failed readback installs a newer base and advances the version, the
copy's own event is then version gated without consuming the flag, and
the next genuinely foreign event was silently adopted — base advanced,
live entry and dirty pages untouched — leaving the mount to later
overwrite that remote change. Every local acknowledgment now installs
its base through one helper that also cancels any pending adoption: the
ack supersedes the mutation the adoption was waiting for.
Tombstones were written for every versioned delete under the mount and
survived directory eviction by design, growing LevelDB with historical
deletions on delete-heavy mounts. They are now scoped to directories
whose cached state the fence actually protects — an uncached parent
never serves from the store nor applies the resurrecting insert — and a
completed listing prunes the direct-child tombstones its absence floor
supersedes, leaving only those above the snapshot. The store gains a
key-prefix visitor for the sweep.
* mount: acked saves install their value; trailer snapshots; direct-child prune range
A version must never advance without its value. saveEntry stamped any
open handle with the acknowledgment's version, but a handle opened
while the save was in flight holds the pre-mutation entry — stamping it
fenced out the events carrying the state it lacked, permanently, with
the local apply performing no invalidation and the redelivery
deduplicated. The acknowledged entry is now installed together with its
version, through the same guarded install the racing-open path uses:
under the handle lock, only when it outranks the handle, never over
dirty local writes.
Empty listings return no in-band snapshot — a snapshot-only response
would be read as an entry by older consumers — so directories that end
empty gained no absence floor and their tombstones were never pruned.
The filer now sends the snapshot in the stream trailer, which older
clients ignore, and the client reads it when no in-band snapshot
arrived. Empty directories get real floors, their tombstones prune,
and their buffered replays gain the snapshot filter instead of the
replay-all fallback.
Version records now encode the parent directory and name separated by
a NUL, making a directory's direct children one contiguous key range:
the tombstone prune scans exactly them under the cache lock, instead
of walking every descendant record — the whole store, for root.
* mount: fix dirty-page loss, uid/gid base, download race, copy adopt, leak; dedup
Correctness fixes from the versioned-invalidation review:
- A foreign delete/rename-away of a file held open with unflushed local
writes destroyed the dirty pages unconditionally. A process may keep
writing to an unlinked-but-open file and those writes were already
acknowledged; preserve the pages when the handle is dirty.
- downloadRemoteEntry stored the handle's base with filer-side uid/gid
while every candidate it is later compared against is in local form,
so under a non-identity UidGidMapper an unchanged re-delivery looked
foreign and force-destroyed dirty pages. Map the base to local.
- downloadRemoteEntry wrote the entry/base/version triple under only the
handle's shared lock, so two concurrent reads of the same remote-only
file could tear it. Serialize the install with a dedicated mutex
(invalidation is already excluded by the exclusive handle lock).
- A committed server-side copy whose readback failed adopted the FIRST
event past the version gate as its base; a foreign write delivered
first was silently swallowed. Adopt only an event whose content
matches the synthesized base — the copy's own event — and install any
other normally.
- The deferred-create path relied on AcquireFileHandle installing the
passed entry on a pre-existing handle, which the version rework
dropped. Restore that install in the compat wrapper; the versioned
open path keeps its gated install.
Growth and hot-path cost:
- Per-entry version records and tombstones leaked when a directory was
evicted or read-through without a rebuild. An uncached directory
gates its own inserts, so its records fence nothing; clear a
directory's child version records when it is wiped for eviction.
- FindEntry paid for the version KvGet on every lookup/getattr cache hit
and threw it away. FindEntry now reads only the entry; the hot
lookupEntry cache-hit path skips the version entirely.
Cleanups:
- Extract ackVersionTsNs over the shared response interface, replacing
the metadata-event-else-log-ts snippet copy-pasted at four ack sites.
- Extract acquireRenamePathLocks, replacing the verbatim sorted
two-path lock fence in both rename handlers.
* mount: no resurrection on foreign delete, version no-event acks, gate downloads, tighten copy adopt
Follow-ups to the review patches:
- Preserving dirty pages on a foreign delete let the next flush pass the
isDeleted guard and CreateEntry, resurrecting the remotely-unlinked
name. Mark the handle deleted in the vacate branch: the open fd can
still read its buffered writes, but a flush no longer recreates the
file.
- A no-event acknowledgment (log fence only) synthesized a metadata
event with TsNs 0, so the cache stored the entry unversioned and an
older subscriber event rolled it back. Stamp the synthesized event
with the ack's log position at all four ack sites.
- downloadRemoteEntry serialized its install but did not check the
version, so an older response arriving last overwrote the entry/base
while the monotonic version kept the newer value, fencing corrections
out. Install only when the response is at least as new as the handle.
- sameEntryContent compared only size and chunks, so a foreign chmod
with unchanged content was adopted as the copy's own event. Compare
everything except server-assigned timestamps, so a metadata-only
foreign change installs instead.
* mount: trim comments to the non-obvious why
The versioning work accumulated multi-line comment blocks restating what
the code says. Keep the constraint a reader cannot derive — why a fence
is exact, why a version must not advance without its value, why an
uncached parent's records fence nothing — and drop the rest.
* mount: distinguish rename from delete, tighten the download and adopt gates
- A rename emits a nil old-path invalidation just like an unlink, so the
vacate branch marked the handle deleted and later writes through the
already-open descriptor were skipped instead of persisted. Carry the
delete/rename distinction on the invalidation and mark only an actual
delete.
- The remote-download install accepted an unversioned response
regardless of the handle's version, so during a rolling upgrade a
delayed response could install stale content under a newer version.
Require the response to be at least as new, with one exception: a
handle still lacking local chunks takes the content anyway — it cannot
read without it — but does not claim the response's log position.
- Copy-event adoption returned without installing, so a foreign touch
arriving before the copy's own event lost its timestamps. Content is
unchanged either way, so the dirty pages stay valid; a clean handle now
takes the entry, while a dirty one keeps its diverged version.
* mount: one directory floor instead of a record per child; agree on TTL
Review feedback:
- Build completion wrote one KV record per direct child inside the cache
write lock, so a large directory stalled every other cache operation
for O(children) store writes. The directory's listing snapshot already
covers every child it saw; make that floor the version for any child
without a record of its own, and a child earns a record only when a
later event touches it. One map write per build replaces the per-child
writes, with the same fencing.
- The presence probe read the store directly and so counted a
TTL-expired entry as present, judging the path by a record describing
content that has logically vanished. It now applies the same expiry
the read path does, and an expired path falls back to its directory
floor.
- Preserve ErrNotFound identity when the commit-time re-read finds the
object deleted, so callers still surface a 404.
- Assert the rename-away source fence timestamp in the invalidation test.
Also record the tombstone ceiling: distinct deleted names in a cached
directory accumulate until it is rebuilt or evicted, which prunes
everything at or below the new snapshot.
* mount: pin the fence's clock domain instead of letting skew decide
A log-position fence is stamped by one filer's clock under that filer's
in-process lock, so comparing it to an event another filer logged is
comparing two unrelated clocks. The two error directions are not equally
costly: applying an event the fence already covered is a re-apply the
base-equality check absorbs, while skipping one it does not cover leaves
the handle holding exactly the state the event was meant to correct,
with the subscription cursor already past it — the unhealable staleness
this whole PR exists to remove.
So refuse to guess. Fences now carry the signature of the filer that
stamped them, and a handle records it alongside the position. An event
is only fenced out when the filer that logged it is the one that stamped
the fence — the logging filer appends its own signature, so its presence
identifies the clock domain. Events from any other filer are applied.
Positions taken from events keep comparing as before; the subscription
already delivers those in order.
The invalidation callback takes a struct now: it carries the path,
entry, position, delete/rename distinction, and signatures, and was
about to need a fifth positional parameter.
* mount: follow a foreign rename; key page invalidation on content, not equality
- A rename's old-path invalidation now carries the destination, and the
handle follows the file there: an open fd tracks the inode, and leaving
it on the old path made its next flush recreate that name instead of
updating the renamed file.
- Dirty pages overlay content, so only a content change invalidates them.
Keying that on exact equality meant any timestamp-only event destroyed
them, which the copy-adoption marker existed to paper over — a foreign
touch could consume the marker and leave the copy's own event to drop
the post-copy writes. Comparing content instead makes the marker
unnecessary, so it is gone: a metadata-only event keeps the overlay,
and a dirty handle keeps its diverged entry unless foreign content
supersedes it.
- A remote download response that is merely older is now refused even
when the handle still lacks chunks; only an unversioned one is taken
(and claims no position), since an older response's content predates
what the handle reflects.
- A refused or unversioned download no longer publishes to the metadata
cache, where a zero-position event would clear the entry's version and
let an older subscriber event roll the cache back.
* mount: page invalidation keys on content alone; unversioned writes claim no position
- sameEntryContent compared everything but timestamps, so a foreign
chmod, chown, or xattr change counted as a content change and
destroyed the dirty-page overlay. It was strict only to serve the
copy-adoption marker, which is gone; its one caller now asks the
question it actually needs — did the bytes change — so metadata-only
events leave the overlay alone.
- A rename over an existing file destroys that file, but its open handle
was left live and still pointed at the name the renamed source now
occupies, so its flush could overwrite it. MovePath already reports the
displaced inode; mark that handle deleted.
- An acknowledgment was refused whenever its position was numerically
lower, even when a different filer stamped the fence it lost to. Two
known, differing signatures mean unrelated clocks, so the comparison no
longer applies there; unknown signatures still compare as before.
- A local write with no log position behind it now records that
explicitly instead of deleting its version record. Absence means the
directory listing covers the path, which is why the snapshot floor
applies; local content the listing never saw must not inherit it, or
the events that would correct it are fenced out.
* mount: widen the existing lookup functions instead of forking WithVersion twins
The versioning work grew a parallel function for every accessor that
needed to return a log position — lookupEntryWithVersion beside
lookupEntry, maybeLoadEntryWithVersion beside maybeLoadEntry,
FindEntryWithVersion beside FindEntry, AcquireFileHandleWithVersion
beside AcquireFileHandle, advanceEntryVersion beside
advanceEntryVersionTsNs, plus a getPbEntryWithVersion wrapper and an
InsertListedEntriesForTest hook. Two names for one operation is two
places to keep in step, and the split let callers pick the one that
happened to compile.
Each pair is now the single original name carrying the position, with
callers that do not want it discarding it. filer_pb.GetEntry returns the
fence its response already carried rather than a mount-side wrapper
re-issuing the lookup, and InsertEntry takes the position its content
reflects rather than a test-only twin that inserted without one.
The one behavioural knot the merge exposed: AcquireFileHandle had been
installing the entry on a pre-existing handle only in its unversioned
form, which conflated 'the caller is authoritative' with 'the lookup had
no version'. Deferred create is the only caller that means the former,
so it now installs explicitly and the map function just acquires.
* fix(filer/remote): keep re-cache work alive past caller cancellation (#9174)
For multi-GB remote blobs, doCacheRemoteObjectToLocalCluster cannot
finish before the S3 gateway's initial cache wait elapses. When it
does, the gRPC ctx cancellation cascades into the filer's chunk
downloads, the error path calls DeleteUncommittedChunks on every chunk
already written, and the next retry starts over. boto3 splitting the
GET into concurrent ranges (or any client tear-down on first failure)
shortens the window between retries, so the loop never converges.
Detach the caller's ctx with context.WithoutCancel before invoking
the singleflight work so the download runs to completion regardless
of client cancellations. Subsequent waiters — via the in-flight
singleflight, or a fresh retry landing after completion — observe the
cached entry and stream normally.
Same detach pattern is used in filer_server_handlers_write.go:53 and
volume_server_handlers_write.go:51.
* simplify rationale comment
* switch to DoChan so handler can return on caller cancel
Do keeps the handler goroutine blocked for the full detached download
even after the client is gone. DoChan lets the handler select on
ctx.Done() and exit immediately; the singleflight goroutine continues
on bgCtx and the next request either joins it or finds the entry
cached.
* fix(mount): remove fid pool to stop master over-allocating volumes
The writeback-cache fid pool pre-allocated file IDs with
ExpectedDataSize = ChunkSizeLimit (typically 8+ MB). The master's
PickForWrite charges count * expectedDataSize against the volume's
effectiveSize, so a full pool refill could charge hundreds of MB
against a single volume before any bytes were actually written.
That tripped RecordAssign's hard-limit path and eagerly removed
volumes from writable, causing the master to grow new volumes
even when the real data being written was tiny.
Drop the pool entirely. Every chunk upload goes through
UploadWithRetry -> AssignVolume with no ExpectedDataSize hint,
letting the master fall back to the 1 MB default estimate. The
mount->filer grpc connection is already cached in pb.WithGrpcClient
(non-streaming mode), so per-chunk AssignVolume is a unary RPC
over an existing HTTP/2 stream, not a full dial. Path-based
filer.conf storage rules now apply to mount chunk assigns again,
which the pool had to skip.
Also remove the now-unused operation.UploadWithAssignFunc and its
AssignFunc type.
* fix(upload): populate ExpectedDataSize from actual chunk bytes
UploadWithRetry already buffers the full chunk into `data` before
calling AssignVolume, so the real size is known. Previously the
assign request went out with ExpectedDataSize=0, making the master
fall back to the 1 MB DefaultNeedleSizeEstimate per fid — same
over-reservation symptom the pool had, just smaller per call.
Stamp ExpectedDataSize = len(data) before the assign RPC when the
caller hasn't already set it. This covers mount chunk uploads,
filer_copy, filersink, mq/logstore, broker_write, gateway_upload,
and nfs — all the UploadWithRetry paths.
* fix(assign): pass real ExpectedDataSize at every assign call site
After removing the mount fid pool, per-chunk AssignVolume calls went
out with ExpectedDataSize=0, making the master fall back to its 1 MB
DefaultNeedleSizeEstimate. That's still an over-estimate for small
writes. Thread the real payload size through every remaining assign
site so RecordAssign charges effectiveSize accurately and stops
prematurely marking volumes full.
- filer: assignNewFileInfo now takes expectedDataSize and stamps it
on both primary and alternate VolumeAssignRequests. Callers pass:
- SSE data-to-chunk: len(data)
- copy manifest save: len(data)
- streamCopyChunk: srcChunk.Size
- TUS sub-chunk: bytes read
- saveAsChunk (autochunk/manifestize): 0 (small, size unknown
until the reader is drained; master uses 1 MB default)
- filer gRPC remote fetch-and-write: ExpectedDataSize = chunkSize
after the adaptive chunkSize is computed.
- ChunkedUploadOption.AssignFunc gains an expectedDataSize parameter;
upload_chunked.go passes the buffered dataSize at the call site.
S3 PUT assignFunc stamps it on the AssignVolumeRequest.
- S3 copy: assignNewVolume / prepareChunkCopy take expectedDataSize;
all seven call sites pass the source chunk's Size.
- operation.SubmitFiles / FilePart.Upload: derive per-fid size from
FileSize (average for batched requests, real per-chunk size for
sequential chunk assigns).
- benchmark: pass fileSize.
- filer append-to-file: pass len(data).
* fix(assign): thread size through SaveDataAsChunkFunctionType
The saveAsChunk path (autochunk, filer_copy, webdav, mount) ran
AssignVolume before the reader was drained, so it had to pass
ExpectedDataSize=0 and fall back to the master's 1 MB default.
Add an expectedDataSize parameter to SaveDataAsChunkFunctionType.
- mergeIntoManifest already has the serialized manifest bytes, so
it passes uint64(len(data)) directly.
- Mount's saveDataAsChunk ignores the parameter because it uses
UploadWithRetry, which already stamps len(data) on the assign
after reading the payload.
- webdav and filer_copy saveDataAsChunk follow the same UploadWithRetry
path and also ignore the hint.
- Filer's saveAsChunk (used for manifestize) plumbs the value to
assignNewFileInfo so manifest-chunk assigns get a real size.
Callers of saveFunc-as-value (weedfs_file_sync, dirty_pages_chunked)
pass the chunk size they're about to upload.
* improve large file sync throughput for remote.cache and filer.sync
Three main throughput improvements:
1. Adaptive chunk sizing for remote.cache: targets ~32 chunks per file
instead of always starting at 5MB. A 500MB file now uses ~16MB chunks
(32 chunks) instead of 5MB chunks (100 chunks), reducing per-chunk
overhead (volume assign, gRPC call, needle write) by 3x.
2. Configurable concurrency at every layer:
- remote.cache chunk concurrency: -chunkConcurrency flag (default 8)
- remote.cache S3 download concurrency: -downloadConcurrency flag
(default raised from 1 to 5 per chunk)
- filer.sync chunk concurrency: -chunkConcurrency flag (default 32)
3. S3 multipart download concurrency raised from 1 to 5: the S3 manager
downloader was using Concurrency=1, serializing all part downloads
within each chunk. This alone can 5x per-chunk download speed.
The concurrency values flow through the gRPC request chain:
shell command → CacheRemoteObjectToLocalClusterRequest →
FetchAndWriteNeedleRequest → S3 downloader
Zero values in the request mean "use server defaults", maintaining
full backward compatibility with existing callers.
Ref #8481
* fix: use full maxMB for chunk size cap and remove loop guard
Address review feedback:
- Use full maxMB instead of maxMB/2 for maxChunkSize to avoid
unnecessarily limiting chunk size for very large files.
- Remove chunkSize < maxChunkSize guard from the safety loop so it
can always grow past maxChunkSize when needed to stay under 1000
chunks (e.g., extremely large files with small maxMB).
* address review feedback: help text, validation, naming, docs
- Fix help text for -chunkConcurrency and -downloadConcurrency flags
to say "0 = server default" instead of advertising specific numeric
defaults that could drift from the server implementation.
- Validate chunkConcurrency and downloadConcurrency are within int32
range before narrowing, returning a user-facing error if out of range.
- Rename ReadRemoteErr to readRemoteErr to follow Go naming conventions.
- Add doc comment to SetChunkConcurrency noting it must be called
during initialization before replication goroutines start.
- Replace doubling loop in chunk size safety check with direct
ceil(remoteSize/1000) computation to guarantee the 1000-chunk cap.
* address Copilot review: clamp concurrency, fix chunk count, clarify proto docs
- Use ceiling division for chunk count check to avoid overcounting
when file size is an exact multiple of chunk size.
- Clamp chunkConcurrency (max 1024) and downloadConcurrency (max 1024
at filer, max 64 at volume server) to prevent excessive goroutines.
- Always use ReadFileWithConcurrency when the client supports it,
falling back to the implementation's default when value is 0.
- Clarify proto comments that download_concurrency only applies when
the remote storage client supports it (currently S3).
- Include specific server defaults in help text (e.g., "0 = server
default 8") so users see the actual values in -h output.
* fix data race on executionErr and use %w for error wrapping
- Protect concurrent writes to executionErr in remote.cache worker
goroutines with a sync.Mutex to eliminate the data race.
- Use %w instead of %v in volume_grpc_remote.go error formatting
to preserve the error chain for errors.Is/errors.As callers.
When remote.cache downloads a file in parallel chunks and a gRPC
connection drops mid-transfer, chunks already written to volume servers
were not cleaned up. Since the filer metadata was never updated, these
needles became orphaned — invisible to volume.vacuum and never
referenced by the filer. On subsequent cache cycles the file was still
treated as uncached, creating more orphans each attempt.
Call DeleteUncommittedChunks on the download-error path, matching the
cleanup already present for the metadata-update-failure path.
Fixes#8481
* filer: expose metadata events and list snapshots
* mount: invalidate hot directory caches
* mount: read hot directories directly from filer
* mount: add sequenced metadata cache applier
* mount: apply metadata responses through cache applier
* mount: replay snapshot-consistent directory builds
* mount: dedupe self metadata events
* mount: factor directory build cleanup
* mount: replace proto marshal dedup with composite key and ring buffer
The dedup logic was doing a full deterministic proto.Marshal on every
metadata event just to produce a dedup key. Replace with a cheap
composite string key (TsNs|Directory|OldName|NewName).
Also replace the sliding-window slice (which leaked the backing array
unboundedly) with a fixed-size ring buffer that reuses the same array.
* filer: remove mutex and proto.Clone from request-scoped MetadataEventSink
MetadataEventSink is created per-request and only accessed by the
goroutine handling the gRPC call. The mutex and double proto.Clone
(once in Record, once in Last) were unnecessary overhead on every
filer write operation. Store the pointer directly instead.
* mount: skip proto.Clone for caller-owned metadata events
Add ApplyMetadataResponseOwned that takes ownership of the response
without cloning. Local metadata events (mkdir, create, flush, etc.)
are freshly constructed and never shared, so the clone is unnecessary.
* filer: only populate MetadataEvent on successful DeleteEntry
Avoid calling eventSink.Last() on error paths where the sink may
contain a partial event from an intermediate child deletion during
recursive deletes.
* mount: avoid map allocation in collectDirectoryNotifications
Replace the map with a fixed-size array and linear dedup. There are
at most 3 directories to notify (old parent, new parent, new child
if directory), so a 3-element array avoids the heap allocation on
every metadata event.
* mount: fix potential deadlock in enqueueApplyRequest
Release applyStateMu before the blocking channel send. Previously,
if the channel was full (cap 128), the send would block while holding
the mutex, preventing Shutdown from acquiring it to set applyClosed.
* mount: restore signature-based self-event filtering as fast path
Re-add the signature check that was removed when content-based dedup
was introduced. Checking signatures is O(1) on a small slice and
avoids enqueuing and processing events that originated from this
mount instance. The content-based dedup remains as a fallback.
* filer: send snapshotTsNs only in first ListEntries response
The snapshot timestamp is identical for every entry in a single
ListEntries stream. Sending it in every response message wastes
wire bandwidth for large directories. The client already reads
it only from the first response.
* mount: exit read-through mode after successful full directory listing
MarkDirectoryRefreshed was defined but never called, so directories
that entered read-through mode (hot invalidation threshold) stayed
there permanently, hitting the filer on every readdir even when cold.
Call it after a complete read-through listing finishes.
* mount: include event shape and full paths in dedup key
The previous dedup key only used Names, which could collapse distinct
rename targets. Include the event shape (C/D/U/R), source directory,
new parent path, and both entry names so structurally different events
are never treated as duplicates.
* mount: drain pending requests on shutdown in runApplyLoop
After receiving the shutdown sentinel, drain any remaining requests
from applyCh non-blockingly and signal each with errMetaCacheClosed
so callers waiting on req.done are released.
* mount: include IsDirectory in synthetic delete events
metadataDeleteEvent now accepts an isDirectory parameter so the
applier can distinguish directory deletes from file deletes. Rmdir
passes true, Unlink passes false.
* mount: fall back to synthetic event when MetadataEvent is nil
In mknod and mkdir, if the filer response omits MetadataEvent (e.g.
older filer without the field), synthesize an equivalent local
metadata event so the cache is always updated.
* mount: make Flush metadata apply best-effort after successful commit
After filer_pb.CreateEntryWithResponse succeeds, the entry is
persisted. Don't fail the Flush syscall if the local metadata cache
apply fails — log and invalidate the directory cache instead.
Also fall back to a synthetic event when MetadataEvent is nil.
* mount: make Rename metadata apply best-effort
The rename has already succeeded on the filer by the time we apply
the local metadata event. Log failures instead of returning errors
that would be dropped by the caller anyway.
* mount: make saveEntry metadata apply best-effort with fallback
After UpdateEntryWithResponse succeeds, treat local metadata apply
as non-fatal. Log and invalidate the directory cache on failure.
Also fall back to a synthetic event when MetadataEvent is nil.
* filer_pb: preserve snapshotTsNs on error in ReadDirAllEntriesWithSnapshot
Return the snapshot timestamp even when the first page fails, so
callers receive the snapshot boundary when partial data was received.
* filer: send snapshot token for empty directory listings
When no entries are streamed, send a final ListEntriesResponse with
only SnapshotTsNs so clients always receive the snapshot boundary.
* mount: distinguish not-found vs transient errors in lookupEntry
Return fuse.EIO for non-not-found filer errors instead of
unconditionally returning ENOENT, so transient failures don't
masquerade as missing entries.
* mount: make CacheRemoteObject metadata apply best-effort
The file content has already been cached successfully. Don't fail
the read if the local metadata cache update fails.
* mount: use consistent snapshot for readdir in direct mode
Capture the SnapshotTsNs from the first loadDirectoryEntriesDirect
call and store it on the DirectoryHandle. Subsequent batch loads
pass this stored timestamp so all batches use the same snapshot.
Also export DoSeaweedListWithSnapshot so mount can use it directly
with snapshot passthrough.
* filer_pb: fix test fake to send SnapshotTsNs only on first response
Match the server behavior: only the first ListEntriesResponse in a
page carries the snapshot timestamp, subsequent entries leave it zero.
* Fix nil pointer dereference in ListEntries stream consumers
Remove the empty-directory snapshot-only response from ListEntries
that sent a ListEntriesResponse with Entry==nil, which crashed every
raw stream consumer that assumed resp.Entry is always non-nil.
Also add defensive nil checks for resp.Entry in all raw ListEntries
stream consumers across: S3 listing, broker topic lookup, broker
topic config, admin dashboard, topic retention, hybrid message
scanner, Kafka integration, and consumer offset storage.
* Add nil guards for resp.Entry in remaining ListEntries stream consumers
Covers: S3 object lock check, MQ management dashboard (version/
partition/offset loops), and topic retention version loop.
* Make applyLocalMetadataEvent best-effort in Link and Symlink
The filer operations already succeeded; failing the syscall because
the local cache apply failed is wrong. Log a warning and invalidate
the parent directory cache instead.
* Make applyLocalMetadataEvent best-effort in Mkdir/Rmdir/Mknod/Unlink
The filer RPC already committed; don't fail the syscall when the
local metadata cache apply fails. Log a warning and invalidate the
parent directory cache to force a re-fetch on next access.
* flushFileMetadata: add nil-fallback for metadata event and best-effort apply
Synthesize a metadata event when resp.GetMetadataEvent() is nil
(matching doFlush), and make the apply best-effort with cache
invalidation on failure.
* Prevent double-invocation of cleanupBuild in doEnsureVisited
Add a cleanupDone guard so the deferred cleanup and inline error-path
cleanup don't both call DeleteFolderChildren/AbortDirectoryBuild.
* Fix comment: signature check is O(n) not O(1)
* Prevent deferred cleanup after successful CompleteDirectoryBuild
Set cleanupDone before returning from the success path so the
deferred context-cancellation check cannot undo a published build.
* Invalidate parent directory caches on rename metadata apply failure
When applyLocalMetadataEvent fails during rename, invalidate the
source and destination parent directory caches so subsequent accesses
trigger a re-fetch from the filer.
* Add event nil-fallback and cache invalidation to Link and Symlink
Synthesize metadata events when the server doesn't return one, and
invalidate parent directory caches on apply failure.
* Match requested partition when scanning partition directories
Parse the partition range format (NNNN-NNNN) and match against the
requested partition parameter instead of using the first directory.
* Preserve snapshot timestamp across empty directory listings
Initialize actualSnapshotTsNs from the caller-requested value so it
isn't lost when the server returns no entries. Re-add the server-side
snapshot-only response for empty directories (all raw stream consumers
now have nil guards for Entry).
* Fix CreateEntry error wrapping to support errors.Is/errors.As
Use errors.New + %w instead of %v for resp.Error so callers can
unwrap the underlying error.
* Fix object lock pagination: only advance on non-nil entries
Move entriesReceived inside the nil check so nil entries don't
cause repeated ListEntries calls with the same lastFileName.
* Guard Attributes nil check before accessing Mtime in MQ management
* Do not send nil-Entry response for empty directory listings
The snapshot-only ListEntriesResponse (with Entry == nil) for empty
directories breaks consumers that treat any received response as an
entry (Java FilerClient, S3 listing). The Go client-side
DoSeaweedListWithSnapshot already preserves the caller-requested
snapshot via actualSnapshotTsNs initialization, so the server-side
send is unnecessary.
* Fix review findings: subscriber dedup, invalidation normalization, nil guards, shutdown race
- Remove self-signature early-return in processEventFn so all events
flow through the applier (directory-build buffering sees self-originated
events that arrive after a snapshot)
- Normalize NewParentPath in collectEntryInvalidations to avoid duplicate
invalidations when NewParentPath is empty (same-directory update)
- Guard resp.Entry.Attributes for nil in admin_server.go and
topic_retention.go to prevent panics on entries without attributes
- Fix enqueueApplyRequest race with shutdown by using select on both
applyCh and applyDone, preventing sends after the apply loop exits
- Add cleanupDone check to deferred cleanup in meta_cache_init.go for
clarity alongside the existing guard in cleanupBuild
- Add empty directory test case for snapshot consistency
* Propagate authoritative metadata event from CacheRemoteObjectToLocalCluster and generate client-side snapshot for empty directories
- Add metadata_event field to CacheRemoteObjectToLocalClusterResponse
proto so the filer-emitted event is available to callers
- Use WithMetadataEventSink in the server handler to capture the event
from NotifyUpdateEvent and return it on the response
- Update filehandle_read.go to prefer the RPC's metadata event over
a locally fabricated one, falling back to metadataUpdateEvent when
the server doesn't provide one (e.g., older filers)
- Generate a client-side snapshot cutoff in DoSeaweedListWithSnapshot
when the server sends no snapshot (empty directory), so callers like
CompleteDirectoryBuild get a meaningful boundary for filtering
buffered events
* Skip directory notifications for dirs being built to prevent mid-build cache wipe
When a metadata event is buffered during a directory build,
applyMetadataSideEffects was still firing noteDirectoryUpdate for the
building directory. If the directory accumulated enough updates to
become "hot", markDirectoryReadThrough would call DeleteFolderChildren,
wiping entries that EnsureVisited had already inserted. The build would
then complete and mark the directory cached with incomplete data.
Fix by using applyMetadataSideEffectsSkippingBuildingDirs for buffered
events, which suppresses directory notifications for dirs currently in
buildingDirs while still applying entry invalidations.
* Add test for directory notification suppression during active build
TestDirectoryNotificationsSuppressedDuringBuild verifies that metadata
events targeting a directory under active EnsureVisited build do NOT
fire onDirectoryUpdate for that directory. In production, this prevents
markDirectoryReadThrough from calling DeleteFolderChildren mid-build,
which would wipe entries already inserted by the listing.
The test inserts an entry during a build, sends multiple metadata events
for the building directory, asserts no notifications fired for it,
verifies the entry survives, and confirms buffered events are replayed
after CompleteDirectoryBuild.
* Fix create invalidations, build guard, event shape, context, and snapshot error path
- collectEntryInvalidations: invalidate FUSE kernel cache on pure
create events (OldEntry==nil && NewEntry!=nil), not just updates
and deletes
- completeDirectoryBuildNow: only call markCachedFn when an active
build existed (state != nil), preventing an unpopulated directory
from being marked as cached
- Add metadataCreateEvent helper that produces a create-shaped event
(NewEntry only, no OldEntry) and use it in mkdir, mknod, symlink,
and hardlink create fallback paths instead of metadataUpdateEvent
which incorrectly set both OldEntry and NewEntry
- applyMetadataResponseEnqueue: use context.Background() for the
queued mutation so a cancelled caller context cannot abort the
apply loop mid-write
- DoSeaweedListWithSnapshot: move snapshot initialization before
ListEntries call so the error path returns the preserved snapshot
instead of 0
* Fix review findings: test loop, cache race, context safety, snapshot consistency
- Fix build test loop starting at i=1 instead of i=0, missing new-0.txt verification
- Re-check IsDirectoryCached after cache miss to avoid ENOENT race with markDirectoryReadThrough
- Use context.Background() in enqueueAndWait so caller cancellation can't abort build/complete mid-way
- Pass dh.snapshotTsNs in skip-batch loadDirectoryEntriesDirect for snapshot consistency
- Prefer resp.MetadataEvent over fallback in Unlink event derivation
- Add comment on MetadataEventSink.Record single-event assumption
* Fix empty-directory snapshot clock skew and build cancellation race
Empty-directory snapshot: Remove client-side time.Now() synthesis when
the server returns no entries. Instead return snapshotTsNs=0, and in
completeDirectoryBuildNow replay ALL buffered events when snapshot is 0.
This eliminates the clock-skew bug where a client ahead of the filer
would filter out legitimate post-list events.
Build cancellation: Use context.Background() for BeginDirectoryBuild
and CompleteDirectoryBuild calls in doEnsureVisited, so errgroup
cancellation doesn't cause enqueueAndWait to return early and trigger
cleanupBuild while the operation is still queued.
* Add tests for empty-directory build replay and cancellation resilience
TestEmptyDirectoryBuildReplaysAllBufferedEvents: verifies that when
CompleteDirectoryBuild receives snapshotTsNs=0 (empty directory, no
server snapshot), ALL buffered events are replayed regardless of their
TsNs values — no clock-skew-sensitive filtering occurs.
TestBuildCompletionSurvivesCallerCancellation: verifies that once
CompleteDirectoryBuild is enqueued, a cancelled caller context does not
prevent the build from completing. The apply loop runs with
context.Background(), so the directory becomes cached and buffered
events are replayed even when the caller gives up waiting.
* Fix directory subtree cleanup, Link rollback, test robustness
- applyMetadataResponseLocked: when a directory entry is deleted or
moved, call DeleteFolderChildren on the old path so cached descendants
don't leak as stale entries.
- Link: save original HardLinkId/Counter before mutation. If
CreateEntryWithResponse fails after the source was already updated,
rollback the source entry to its original state via UpdateEntry.
- TestBuildCompletionSurvivesCallerCancellation: replace fixed
time.Sleep(50ms) with a deadline-based poll that checks
IsDirectoryCached in a loop, failing only after 2s timeout.
- TestReadDirAllEntriesWithSnapshotEmptyDirectory: assert that
ListEntries was actually invoked on the mock client so the test
exercises the RPC path.
- newMetadataEvent: add early return when both oldEntry and newEntry are
nil to avoid emitting events with empty Directory.
---------
Co-authored-by: Copilot <copilot@github.com>
* s3: fix remote object not caching
* s3: address review comments for remote object caching
- Fix leading slash in object name by using strings.TrimPrefix
- Return cached entry from CacheRemoteObjectToLocalCluster to get updated local chunk locations
- Reuse existing helper function instead of inline gRPC call
* s3/filer: add singleflight deduplication for remote object caching
- Add singleflight.Group to FilerServer to deduplicate concurrent cache operations
- Wrap CacheRemoteObjectToLocalCluster with singleflight to ensure only one
caching operation runs per object when multiple clients request the same file
- Add early-return check for already-cached objects
- S3 API calls filer gRPC with timeout and graceful fallback on error
- Clear negative bucket cache when bucket is created via weed shell
- Add integration tests for remote cache with singleflight deduplication
This benefits all clients (S3, HTTP, Hadoop) accessing remote-mounted objects
by preventing redundant cache operations and improving concurrent access performance.
Fixes: https://github.com/seaweedfs/seaweedfs/discussions/7599
* fix: data race in concurrent remote object caching
- Add mutex to protect chunks slice from concurrent append
- Add mutex to protect fetchAndWriteErr from concurrent read/write
- Fix incorrect error check (was checking assignResult.Error instead of parseErr)
- Rename inner variable to avoid shadowing fetchAndWriteErr
* fix: address code review comments
- Remove duplicate remote caching block in GetObjectHandler, keep only singleflight version
- Add mutex protection for concurrent chunk slice and error access (data race fix)
- Use lazy initialization for S3 client in tests to avoid panic during package load
- Fix markdown linting: add language specifier to code fence, blank lines around tables
- Add 'all' target to Makefile as alias for test-with-server
- Remove unused 'util' import
* style: remove emojis from test files
* fix: add defensive checks and sort chunks by offset
- Add nil check and type assertion check for singleflight result
- Sort chunks by offset after concurrent fetching to maintain file order
* fix: improve test diagnostics and path normalization
- runWeedShell now returns error for better test diagnostics
- Add all targets to .PHONY in Makefile (logs-primary, logs-remote, health)
- Strip leading slash from normalizedObject to avoid double slashes in path
---------
Co-authored-by: chrislu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Chris Lu