filer: resume a rejoining peer's meta subscription at its last position (#12279)

* filer: resume a rejoining peer's meta subscription at its last position

OnPeerUpdate starts every peer add at the update's arrival time. For
filers sharing one store there is no persisted offset to correct it, so a
peer that drops out and rejoins (a graceful restart) resubscribes from
"now" and the events it logged while its stream was down never reach the
aggregated buffer - filer.backup and other meta subscribers silently
skip them.

Track each peer's delivered-through position (the same timestamp domain
as the stream's events and idle-heartbeat claims) and rewind the start
of a rejoining subscription to it. First-seen peers keep the given
start, and non-shared peers still resume from their persisted offset.

* filer: keep the re-add resume point behind a replay-floored event

The session's replay floor already pins the persisted offset and the
reconnect resume point so an event that outlived its retry budget is
redelivered on resubscribe; cap peerResumeTsNs to the same position on
session exit so the new re-add resume cannot skip past it either.

* filer: stop test aggregators' log-buffer workers

* filer: cover peer rejoin resubscribing at the resume position

* filer: drop the unreachable replay-floor cap on the resume point

Only a peer with its own store can floor an event, and that peer's
subscribe start always comes from its persisted offset, never from the
re-add resume point. The cap also ran in the old session's defer, after
the re-add had already read the value.

* filer: say why a re-add may resume from an idle heartbeat

The heartbeat comments promised a reconnect never resumes from one,
while the re-add resume point does. Spell out the difference and the
fence that makes the local-stream claim safe to resume from.

* filer: forget a removed peer's resume point after an hour

The map held an entry for every address ever seen; with pod IPs that
is one per restart. The removal mark now outlives the watermark grace
and takes the resume point with it once the retention passes.

* filer: advance the resume point only for tracked peers

Same rule as the watermarks: a straggler from a dropped session must
not recreate the entry.

* filer: set the test aggregator's subscribe loop instead of a nil fallback
This commit is contained in:
Chris Lu authored and GitHub committed 2026-10-31 11:49:50 -07:00
1 parent f5989a21fb
commit 8d3cb2ed2d
4 files changed
+157 -14

No files matched your search

+3 -1
View File
@@ -1385,7 +1385,9 @@ func (fs *FilerServer) maybeSendFlushReport(req *filer_pb.SubscribeMetadataReque
// maybeSendIdleHeartbeat emits an empty response carrying the current time when
// the subscriber has consumed everything up to the buffer head. The client uses
// it to advance freshness signals (e.g. filer.sync's sync_offset) without moving
// its resume checkpoint, so a restart still re-reads from the last real event.
// its persisted resume checkpoint, so a restart still re-reads from the last
// real event. The fenced local-stream claim is the one exception: a peer
// aggregator resumes a re-added peer from it (MetaAggregator.peerResumeStartFrom).
//
// The catch-up floor is the max of two read-progress markers:
// - readPositionTsNs: how far the read cursor has advanced. It starts at