filer.backup: process metadata events through the concurrent processor (#11951)

* filer.backup: process metadata events through the concurrent processor

Each event was handled inline on the subscription loop, so one
fetch+write at a time capped the stream at a few entries per second and
a busy source kept building lag. Route events through the same
MetadataProcessor filer.sync uses: it fans jobs out behind -concurrency
while serializing same-path and subtree-barrier conflicts, and the
checkpoint now advances on the processor's contiguous watermark via
GetResumeTsNs, with Resubscribe replaying anything a failure pins.
-disableErrorRetry keeps its log-and-skip meaning by returning nil after
logging, since the processor would otherwise pin the checkpoint forever.

* filer.backup: drain in-flight jobs before the stream retry loop reconnects

FollowMetadata returns as soon as the stream ends while admitted jobs keep writing; the next doFilerBackup builds a fresh processor that cannot order its replayed writes against them, so an older upload could finish last and clobber a newer file.

* filer.backup: checkpoint the completed watermark on a ticker and at stream end

The offset callback only fires while events arrive, so work draining on a quiet stream stayed uncheckpointed until the next event and a restart recopied it. Persist the processed watermark every 3s when it advances, and once more after the post-stream drain.

* filer.backup: bound the checkpoint wait before reconnecting

setOffset has no context, so a stalled KvPut would hold persistDone and block the retry loop forever. The persist goroutine now performs the final write itself on stopPersist, and the caller waits at most 30s.

* filer.backup: carry the processed position across retry attempts

A follow attempt that ended with a pinned failure recomputed startFrom on retry, so a -timeAgo window could move past the failed event and skip it. doFilerBackup now returns the failure pin (or processed watermark) and the retry loop clamps startFrom to it.

* filer.backup: resume retries at the previous attempt's pinned position

The retry loop threaded resumeTsNs through but never consulted it, so a
-timeAgo run recomputed startFrom as now-minus-the-window on every attempt
— a failed event older than the window was skipped on reconnect. Use the
pinned position when one exists; -timeAgo and the persisted checkpoint
only choose the first starting point.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: resume below the failure pin so the failed event replays

The metadata reader treats the resume position as exclusive — events at
or below it are skipped — so resuming at the failed event's own timestamp
would skip it. Back one tick off the pin.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: label the progress line as the received-event rate

The counter increments when a job is admitted, not when its copy
finishes, so the old wording showed backup throughput during a burst of
slow copies while the processed watermark had not moved.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: bound the offset checkpoint RPC

setOffset called KvPut on the background context, so a stalled write
could pin a caller's drain or persistence loop indefinitely.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: keep the resume pin when filer-source setup fails

A retried attempt carries resumeTsNs so the failed event replays from the
same position, but filer-source and SSE initialization failures returned 0,
so the next attempt recomputed -timeAgo and could move the window past the
pinned event.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: serialize checkpoint writes per key

A checkpoint write can outlive its caller's wait and land beside the next
subscription attempt's writes, moving saved progress backward. Serialize
writes per key inside setOffset and drop any value older than one already
written from this process.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: gofmt

* replication: serialize checkpoint writes per filer and key

A single process-wide lock made a stalled checkpoint RPC block every
stream's checkpoint, including the opposite direction of a bidirectional
filer.sync. Keeping the last-written watermark per persisted key also let
one stream's offset shadow another stream checkpointing the same key to
a different target filer. Lock and track progress per (filer, key).

* replication: keep the resume position behind a cleared failure pin

A redelivery that clears failedTs lets the processed watermark advance
past the earliest failure, but events dropped while the processor was
stopped — possibly sharing the failure's timestamp — have to replay.
Latched earliestFailedTsNs floors the resume position one tick below
that failure so reconnects still cover them.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: persist floored checkpoints through the write ordering

setOffset drops a value older than the last written, which also swallows
a deliberate regression: the resume floor, an operator-seeded offset, or
a snapshot high-water mark. allowRegress marks those callers so the
checkpoint can move backward when a failure forces replay.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: order checkpoint writes by issuance, not value

The value-based stale guard could never fire — every caller passed
allowRegress because floors and seeds are deliberate regressions — and
ordering by value cannot tell a stale high watermark apart from real
progress: a write that outlives its attempt could still land after a
floored resume and skip events owed a replay. Order by issuance instead:
a write issued before the last one to land under the key is dropped, so
late writes from a superseded attempt can no longer move the checkpoint
in either direction while deliberate regressions still apply.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: keep the resume position one tick below the frontier

* remote.sync: resume and persist through the floored watermark

* test: concurrent metadata jobs fan out up to the limit

Distinct-path events must all be in flight before any finishes — a
serial loop could only ever have one running — while the limit still
bounds the pool. Pins the property behind the backup-throughput fix.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
Chris LuandDevin authored and GitHub committed 2026-10-23 20:02:55 -07:00
1 parent cd562b8514
commit 6e8a2cdb14
7 files changed
+443 -26

No files matched your search

+70 -9
View File
@@ -7,6 +7,7 @@ import (
"os"
"regexp"
"strings"
"sync"
"sync/atomic"
"time"
@@ -222,7 +223,7 @@ func runFilerSynchronize(cmd *Command, args []string) bool {
if state == nil || state.processor == nil {
return
}
offsetTsNs := state.processor.processedTsWatermark.Load()
offsetTsNs := resumeTsNsFor(state.processor)
if offsetTsNs == 0 {
return
}
@@ -475,7 +476,7 @@ func doSubscribeFilerMetaChanges(clientId int32, clientEpoch int32, sourceGrpcDi
lastProgressedTsNs = offsetTsNs
// collect synchronous offset
statsCollect.FilerSyncOffsetGauge.WithLabelValues(sourceFiler.String(), targetFiler.String(), clientName, sourcePath).Set(float64(offsetTsNs))
return setOffset(targetGrpcDialOption, targetFiler, signaturePrefix, sourceFilerSignature, offsetTsNs)
return setOffset(targetGrpcDialOption, targetFiler, signaturePrefix, sourceFilerSignature, resumeTsNsFor(processor))
})
prefix := sourcePath
@@ -495,7 +496,7 @@ func doSubscribeFilerMetaChanges(clientId int32, clientEpoch int32, sourceGrpcDi
StopTsNs: 0,
EventErrorType: pb.RetryForeverOnError,
GetResumeTsNs: func() int64 {
return processor.processedTsWatermark.Load()
return resumeTsNsFor(processor)
},
Resubscribe: processor.ResubscribeCh(),
// While the source has only read activity it emits no metadata events, so
@@ -576,16 +577,77 @@ func getOffsetWithFallback(grpcDialOption grpc.DialOption, filer pb.ServerAddres
return lastOffsetTsNs, err
}
func setOffset(grpcDialOption grpc.DialOption, filer pb.ServerAddress, signaturePrefix string, signature int32, offsetTsNs int64) error {
return pb.WithFilerClient(false, signature, filer, grpcDialOption, func(client filer_pb.SeaweedFilerClient) error {
// A checkpoint write can outlive its caller's wait and land beside the next
// subscription attempt's writes; issued concurrently they may be applied out
// of order, moving saved progress backward or forward past events still owed
// a replay. Writes to the same persisted key are serialized and each is
// ordered by issuance: a write issued before the last one to land is stale
// and dropped. Deliberate regressions — resume floors, operator seeds — are
// issued after the writes they supersede, so they still apply. State is keyed
// by target filer so unrelated streams — or the two directions of a
// bidirectional sync — never block or shadow each other.
var offsetCheckpointWrites = struct {
sync.Mutex
byKey map[string]*offsetCheckpoint
}{byKey: map[string]*offsetCheckpoint{}}
syncKey := []byte(signaturePrefix + "____")
util.Uint32toBytes(syncKey[len(signaturePrefix):len(signaturePrefix)+4], uint32(signature))
type offsetCheckpoint struct {
sync.Mutex
lastSeq int64
}
// resumeTsNsFor floors the processed watermark by the earliest failure the
// processor saw: a redelivery can clear the pin, but events dropped while it
// was stopped only replay if the resume position stays behind it. It also
// stays one tick below the newest timestamp seen — a watermark on the
// frontier can have finished every job it admitted while more events at the
// same timestamp still sit unread in the log, and an exclusive resume there
// would skip them.
func resumeTsNsFor(p *MetadataProcessor) int64 {
ts := p.processedTsWatermark.Load()
if newestTsNs := p.NewestSeenTsNs(); newestTsNs > 0 && ts >= newestTsNs {
ts = newestTsNs - 1
}
if failedTsNs := p.EarliestFailedTsNs(); failedTsNs > 0 && failedTsNs-1 < ts {
ts = failedTsNs - 1
}
return ts
}
// offsetWriteSeq numbers checkpoint writes in issuance order so a write that
// lands late can be told apart from the fresher write it would shadow.
var offsetWriteSeq atomic.Int64
func setOffset(grpcDialOption grpc.DialOption, filer pb.ServerAddress, signaturePrefix string, signature int32, offsetTsNs int64) error {
syncKey := []byte(signaturePrefix + "____")
util.Uint32toBytes(syncKey[len(signaturePrefix):len(signaturePrefix)+4], uint32(signature))
key := string(filer) + "\x00" + string(syncKey)
offsetCheckpointWrites.Lock()
checkpoint := offsetCheckpointWrites.byKey[key]
if checkpoint == nil {
checkpoint = &offsetCheckpoint{}
offsetCheckpointWrites.byKey[key] = checkpoint
}
offsetCheckpointWrites.Unlock()
seq := offsetWriteSeq.Add(1)
checkpoint.Lock()
defer checkpoint.Unlock()
if seq < checkpoint.lastSeq {
return nil
}
checkpoint.lastSeq = seq
return pb.WithFilerClient(false, signature, filer, grpcDialOption, func(client filer_pb.SeaweedFilerClient) error {
valueBuf := make([]byte, 8)
util.Uint64toBytes(valueBuf, uint64(offsetTsNs))
resp, err := client.KvPut(context.Background(), &filer_pb.KvPutRequest{
// a stalled KvPut must not hold a caller's drain or persistence loop
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
resp, err := client.KvPut(ctx, &filer_pb.KvPutRequest{
Key: syncKey,
Value: valueBuf,
})
@@ -600,7 +662,6 @@ func setOffset(grpcDialOption grpc.DialOption, filer pb.ServerAddress, signature
return nil
})
}
func genProcessFunction(sourcePath string, targetPath string, excludePaths []string, reExcludeFileName *regexp.Regexp, excludeFileNames []*wildcard.WildcardMatcher, excludePathPatterns []*wildcard.WildcardMatcher, dataSink sink.ReplicationSink, doDeleteFiles bool, debug bool) func(resp *filer_pb.SubscribeMetadataResponse) error {