filer.sync: skip superseded replays before fetching any chunks (#12212)

* filer.sync: skip superseded replays before fetching any chunks

On a directory under heavy overwrite, most entries the sync replays are
already superseded on the source. The sink only learned that after chunk
reads failed — or worse, after fetching and uploading bytes a later event
replaces — so the backlog drained slowly and the log filled with 404s.

CreateEntry and UpdateEntry now check sourceMovedOn before touching a
chunk: one entry lookup skips a stale replay entirely. A mid-replication
supersession is still caught by the existing retry gates.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: bound the source liveness probes with a timeout

The supersession checks added a source entry lookup to the replication
path, and every lookup ran on context.Background() with no deadline. A
source that accepts the call but whose metadata store stalls would hold
the sync forever — before a single chunk fetch, even when every chunk is
still readable. Bound the probe at 30s; an unanswered lookup proves
nothing, so replication proceeds and the existing retry gates decide.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: sync the live version a superseded replay points at

Skipping a stale replay assumed a later event always re-carries the
current content. It does not when the superseding write was itself
replicated in from another cluster: those events are dropped before they
reach a filer sink, so the destination could be left missing the file.
When the source reports a live entry newer than the replay, commit that
version through the normal path instead — equivalent to applying the
later event, with no dependence on its delivery. A gone source entry
still skips, since the destination already matches the source.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: reconcile moved entry content after a native rename

AtomicRenameEntry relocates whatever version the destination held. If the
preceding update replay was skipped because the source had already moved
to the new name, the moved entry keeps the old bytes and — with no
further writes — stays stale forever. MoveEntry now runs the same
reconciliation as an in-place update against the rename's snapshot, so
the moved entry picks up pending content changes.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: repair from the live snapshot a superseded replay resolves to

An incremental diff against the replayed oldEntry can resolve manifests the
overwrite already vacuumed, which fails every retry of the event.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
Chris LuandDevin authored and GitHub committed 2026-10-29 13:37:27 -07:00
1 parent 98fe70a247
commit 8a2fec8610
8 files changed
+464 -27

No files matched your search

+1 -1
View File
@@ -768,7 +768,7 @@ func genProcessFunction(sourcePath string, targetPath string, excludePaths []str
oldKey := buildKey(dataSink, message, targetPath, sourceOldKey, sourcePath)
if mover, ok := dataSink.(sink.EntryMover); ok {
// native atomic move: no re-copy, no descendant gap, no chunk leak.
return mover.MoveEntry(oldKey, newKey, message.NewEntry, message.Signatures)
return mover.MoveEntry(oldKey, newKey, message.OldEntry, message.NewEntry, message.Signatures)
}
// no native move: create the new entry first, then delete the
// old, so a crash between the two leaves the entry visible.