* filer.remote.sync: bounded filer lookups, one chunk cache per process
The filer lookups on the sync path (currentEntry, isSuperseded,
uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no
deadline. A lookup that never answers keeps its job alive, the job keeps
its path in the MetadataProcessor conflict index, AddSyncJob waits on the
condition and the daemon stops consuming events with no log line. Each
lookup now carries a 30 s deadline, so the job fails and the processor's
stop-and-resubscribe path runs.
remoteUploadReader built a chunk cache per upload; the cache's worker
goroutine is never stopped, so each upload leaked one goroutine and its
cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory
limit). One chunk cache per process.
(cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78)
* filer.remote.sync: a failed mount lookup at startup never resolves to now
collectLastSyncOffset returned time.Now() when the mounted directory lookup
failed, which skips every event since the saved offset. The saved offset is
used instead; without one the daemon exits so a restart retries.
(cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae)
* filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read
retriedWriteFile returns the supersession lookup's error instead of
treating an unanswered lookup as "not superseded": in a replay an older
snapshot could otherwise overwrite a newer remote object. The processor
retries the event. isSuperseded keeps its boolean form for the other call
sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls
back to the saved offset cannot hang on it.
TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed
lookup; it now asserts the withheld upload and the lookup error.
(cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa)
* filer.remote.sync test: keep the slices import the other gateway tests use
* filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo
After a failed upload attempt the supersession lookup's error ended in
isSuperseded's boolean and the loop could upload the snapshot again
unverified. The lookup error now ends the loop (errLookupUnanswered, not
retried) and the processor retries the event. With -timeAgo set, a failed
read of the saved offset resolved to now minus timeAgo, skipping everything
older; the daemon exits instead so a restart retries.
(cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566)
* filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting
resolveLastSyncOffset returns the lookup or offset-read error; both callers
(directory and buckets) hand it to util.RetryUntil, which retries with
backoff instead of the process exiting.
* filer.remote.sync: keep doc comments on the functions they describe
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 952d80c4d3)
* filer.remote.sync: isSuperseded returns the lookup error itself
Its only production callers needed the error, so the boolean wrapper was
left to the tests. The tests now assert the error on a failed lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 51f603b5ec)
* filer.remote.sync: bound the stamp written after each upload
updateLocalEntry's UpdateEntry was the last filer call on the sync path
without a deadline. A filer that never answers the stamp keeps the job and
its path in the conflict index the same way an unanswered lookup did. The
stamp carries IF_ENTRY_EQUAL, so a retried event after a timeout either
stamps again or skips a stamp that already landed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c7338fb68f)
* filer.remote.sync: an unread offset never resolves to the mount's creation time
With the offset read now bounded, a timed-out read fell through to the
directory's creation time and replayed every event already synced. The
error goes back to the startup retry loop, as with -timeAgo; a missing
offset still reads as zero without an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 0a1b6c2944)
* filer.remote.sync: stamp a copy of the event entry
updateLocalEntry set the RemoteEntry and dropped the rename marker on the
caller's entry before the RPC. A transient stamp failure retries the same
event in place, which then read as already synced and skipped both the
upload and the stamp. The stub filer now applies a successful update the
way the filer does, since tests had relied on that shared object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 1f332e5866)
* filer.remote.sync: a directory not created yet starts now, as before
A gateway started before the first bucket looked up a missing buckets
directory, had no saved offset, and returned an error to the retry loop
on every attempt until a bucket appeared. A missing directory with no
saved offset has nothing to replay, so it starts now as it did before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c31497a9a3)
* filer.remote.sync: take the start time before the mount lookup
A bucket created between a not-found lookup and the end of the offset
read had its creation before a start taken afterwards, so the gateway
never created its remote bucket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit f535fca8f6)
* filer.remote.sync: save the resume position when a subscription ends
A failed event ends the subscription with ErrResubscribe, and the retry
loop runs startup again, recomputing -timeAgo from a later now. The
periodic checkpoint only runs as events arrive, so a failure before the
first one could start the retry after the event it failed on. The resume
position is now saved whenever the subscription returns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 7be919ded1)
* remote_storage: bound the sync offset write like the read
The offset write runs in the subscription's checkpoint callback and, now,
when a subscription ends; a filer that never answers it stalled either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit f888ac6b8f)
* filer.remote.sync: say that any saved offset wins over the creation time
The creation time in seconds was compared with the offset in
milliseconds, which made any saved offset win. That is the behaviour to
keep: remote.unmount recreates the directory and then saves the offset,
usually in the same second, so a seconds comparison would start at the
creation second and replay the purge. The comparison now states it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit a0d60ee00f)
* filer.remote.sync: keep the resume position for the retry when its write fails
The retry in the same process now resumes from the position the ended
subscription returned, as filer.backup does, so a failed offset write
followed by an empty read cannot recompute -timeAgo past a failed event.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 39ef551bcf)
* filer.remote.sync test: give the same-second saved offset a fractional part
With whole seconds, returning the creation second resolved to the same
time and passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 4bb74382a6)
---------
Co-authored-by: james <james@medable.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
