mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-18 09:46:24 +01:00
* filer: serve caught-up aggregated subscribers from memory, not a held disk pass An aggregated SubscribeMetadata stream that does not take metadata chunks (filer.sync, filer.backup, filer.meta.backup, filer.meta.tail, the filer.remote.* syncs, the S3 gateway's subscriptions, and mounts that predate chunk support) re-ran the persisted-log pass whenever its cursor moved: after every memory hold at the delivery watermark and on every 2 s re-probe. The pass holds at the peers' flush low-watermark. Peers flush on their own one-minute schedules, so on a cluster that keeps writing some peer has always flushed past a caught-up cursor while the slowest has not: the pass held at its first entry, the loop came back to it, and memory was never read. The stream got each change only once the slowest peer's flush covered it, in bursts up to a flush interval late, and every pass listed and re-read the newest log files. When the aggregated ring was already filling at the cursor and evicted nothing after it, memory holds everything the persisted logs could add, since every peer's log reaches the ring through the aggregator. Skip the pass and read memory, held at the delivery watermark as before. The skipped pass proves nothing past the cursor, and the next-day scan for an empty pass does not apply. An empty ring, a ring that starts past the cursor, and a cursor memory just refused still take the pass. Chunk subscribers already skip it for a caught-up cursor: their ref listing stops below the flush hold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: keep a new peer's pre-start history on the disk pass The ring only receives a peer's events from the start of its subscription. A peer followed for the first time after this filer started (a filer added to the cluster, a restarted pod with a new address, or the late-arriving peers bootstrap) starts at its notice time, so what it logged before then is only in the persisted logs. A cursor below that start passed the ring check and skipped the disk pass, losing those events for subscribers that do not take metadata chunks. Record the newest start of a peer subscription that had no resume point and send cursors below it through the disk pass as before. Peers that rejoin with a resume point (a restart under the same address) do not move it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
36 lines
1.2 KiB
Go
36 lines
1.2 KiB
Go
package filer
|
|
|
|
import (
|
|
"time"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/pb"
|
|
)
|
|
|
|
// TrackPeerForTesting registers a peer as if the master had announced it, but
|
|
// without starting its subscription goroutine, so loop tests can drive the
|
|
// low-watermarks by hand. Test support only.
|
|
func (ma *MetaAggregator) TrackPeerForTesting(peer pb.ServerAddress) {
|
|
ma.peerChansLock.Lock()
|
|
ma.peerChans[peer] = make(chan struct{})
|
|
if peer != ma.self {
|
|
ma.noteRemotePeerArrivalLocked()
|
|
}
|
|
ma.peerChansLock.Unlock()
|
|
ma.initPeerWatermark(peer)
|
|
}
|
|
|
|
// StartPeerForTesting is TrackPeerForTesting for a peer first followed from
|
|
// startFrom, as OnPeerUpdate records it. Test support only.
|
|
func (ma *MetaAggregator) StartPeerForTesting(peer pb.ServerAddress, startFrom time.Time) {
|
|
ma.TrackPeerForTesting(peer)
|
|
ma.peerResumeStartFrom(peer, startFrom)
|
|
}
|
|
|
|
// ReportPeerWatermarksForTesting stands in for what a peer's stream reports:
|
|
// its delivery watermark (events and idle heartbeats) and its flush
|
|
// watermark. Test support only.
|
|
func (ma *MetaAggregator) ReportPeerWatermarksForTesting(peer pb.ServerAddress, deliveredTsNs, flushedTsNs int64) {
|
|
ma.advancePeerWatermark(peer, deliveredTsNs)
|
|
ma.advancePeerFlushWatermark(peer, flushedTsNs)
|
|
}
|