mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-18 17:55:26 +01:00
filer.backup: process metadata events through the concurrent processor (#11951)
* filer.backup: process metadata events through the concurrent processor Each event was handled inline on the subscription loop, so one fetch+write at a time capped the stream at a few entries per second and a busy source kept building lag. Route events through the same MetadataProcessor filer.sync uses: it fans jobs out behind -concurrency while serializing same-path and subtree-barrier conflicts, and the checkpoint now advances on the processor's contiguous watermark via GetResumeTsNs, with Resubscribe replaying anything a failure pins. -disableErrorRetry keeps its log-and-skip meaning by returning nil after logging, since the processor would otherwise pin the checkpoint forever. * filer.backup: drain in-flight jobs before the stream retry loop reconnects FollowMetadata returns as soon as the stream ends while admitted jobs keep writing; the next doFilerBackup builds a fresh processor that cannot order its replayed writes against them, so an older upload could finish last and clobber a newer file. * filer.backup: checkpoint the completed watermark on a ticker and at stream end The offset callback only fires while events arrive, so work draining on a quiet stream stayed uncheckpointed until the next event and a restart recopied it. Persist the processed watermark every 3s when it advances, and once more after the post-stream drain. * filer.backup: bound the checkpoint wait before reconnecting setOffset has no context, so a stalled KvPut would hold persistDone and block the retry loop forever. The persist goroutine now performs the final write itself on stopPersist, and the caller waits at most 30s. * filer.backup: carry the processed position across retry attempts A follow attempt that ended with a pinned failure recomputed startFrom on retry, so a -timeAgo window could move past the failed event and skip it. doFilerBackup now returns the failure pin (or processed watermark) and the retry loop clamps startFrom to it. * filer.backup: resume retries at the previous attempt's pinned position The retry loop threaded resumeTsNs through but never consulted it, so a -timeAgo run recomputed startFrom as now-minus-the-window on every attempt — a failed event older than the window was skipped on reconnect. Use the pinned position when one exists; -timeAgo and the persisted checkpoint only choose the first starting point. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: resume below the failure pin so the failed event replays The metadata reader treats the resume position as exclusive — events at or below it are skipped — so resuming at the failed event's own timestamp would skip it. Back one tick off the pin. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: label the progress line as the received-event rate The counter increments when a job is admitted, not when its copy finishes, so the old wording showed backup throughput during a burst of slow copies while the processed watermark had not moved. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: bound the offset checkpoint RPC setOffset called KvPut on the background context, so a stalled write could pin a caller's drain or persistence loop indefinitely. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: keep the resume pin when filer-source setup fails A retried attempt carries resumeTsNs so the failed event replays from the same position, but filer-source and SSE initialization failures returned 0, so the next attempt recomputed -timeAgo and could move the window past the pinned event. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: serialize checkpoint writes per key A checkpoint write can outlive its caller's wait and land beside the next subscription attempt's writes, moving saved progress backward. Serialize writes per key inside setOffset and drop any value older than one already written from this process. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: gofmt * replication: serialize checkpoint writes per filer and key A single process-wide lock made a stalled checkpoint RPC block every stream's checkpoint, including the opposite direction of a bidirectional filer.sync. Keeping the last-written watermark per persisted key also let one stream's offset shadow another stream checkpointing the same key to a different target filer. Lock and track progress per (filer, key). * replication: keep the resume position behind a cleared failure pin A redelivery that clears failedTs lets the processed watermark advance past the earliest failure, but events dropped while the processor was stopped — possibly sharing the failure's timestamp — have to replay. Latched earliestFailedTsNs floors the resume position one tick below that failure so reconnects still cover them. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: persist floored checkpoints through the write ordering setOffset drops a value older than the last written, which also swallows a deliberate regression: the resume floor, an operator-seeded offset, or a snapshot high-water mark. allowRegress marks those callers so the checkpoint can move backward when a failure forces replay. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: order checkpoint writes by issuance, not value The value-based stale guard could never fire — every caller passed allowRegress because floors and seeds are deliberate regressions — and ordering by value cannot tell a stale high watermark apart from real progress: a write that outlives its attempt could still land after a floored resume and skip events owed a replay. Order by issuance instead: a write issued before the last one to land under the key is dropped, so late writes from a superseded attempt can no longer move the checkpoint in either direction while deliberate regressions still apply. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: keep the resume position one tick below the frontier * remote.sync: resume and persist through the floored watermark * test: concurrent metadata jobs fan out up to the limit Distinct-path events must all be in flight before any finishes — a serial loop could only ever have one running — while the limit still bounds the pool. Pins the property behind the backup-throughput fix. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
cd562b8514
commit
6e8a2cdb14
7 files changed
+443
-26
No files matched your search
@@ -7,6 +7,7 @@ import (
|
||||
"os"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
@@ -222,7 +223,7 @@ func runFilerSynchronize(cmd *Command, args []string) bool {
|
||||
if state == nil || state.processor == nil {
|
||||
return
|
||||
}
|
||||
offsetTsNs := state.processor.processedTsWatermark.Load()
|
||||
offsetTsNs := resumeTsNsFor(state.processor)
|
||||
if offsetTsNs == 0 {
|
||||
return
|
||||
}
|
||||
@@ -475,7 +476,7 @@ func doSubscribeFilerMetaChanges(clientId int32, clientEpoch int32, sourceGrpcDi
|
||||
lastProgressedTsNs = offsetTsNs
|
||||
// collect synchronous offset
|
||||
statsCollect.FilerSyncOffsetGauge.WithLabelValues(sourceFiler.String(), targetFiler.String(), clientName, sourcePath).Set(float64(offsetTsNs))
|
||||
return setOffset(targetGrpcDialOption, targetFiler, signaturePrefix, sourceFilerSignature, offsetTsNs)
|
||||
return setOffset(targetGrpcDialOption, targetFiler, signaturePrefix, sourceFilerSignature, resumeTsNsFor(processor))
|
||||
})
|
||||
|
||||
prefix := sourcePath
|
||||
@@ -495,7 +496,7 @@ func doSubscribeFilerMetaChanges(clientId int32, clientEpoch int32, sourceGrpcDi
|
||||
StopTsNs: 0,
|
||||
EventErrorType: pb.RetryForeverOnError,
|
||||
GetResumeTsNs: func() int64 {
|
||||
return processor.processedTsWatermark.Load()
|
||||
return resumeTsNsFor(processor)
|
||||
},
|
||||
Resubscribe: processor.ResubscribeCh(),
|
||||
// While the source has only read activity it emits no metadata events, so
|
||||
@@ -576,16 +577,77 @@ func getOffsetWithFallback(grpcDialOption grpc.DialOption, filer pb.ServerAddres
|
||||
return lastOffsetTsNs, err
|
||||
}
|
||||
|
||||
func setOffset(grpcDialOption grpc.DialOption, filer pb.ServerAddress, signaturePrefix string, signature int32, offsetTsNs int64) error {
|
||||
return pb.WithFilerClient(false, signature, filer, grpcDialOption, func(client filer_pb.SeaweedFilerClient) error {
|
||||
// A checkpoint write can outlive its caller's wait and land beside the next
|
||||
// subscription attempt's writes; issued concurrently they may be applied out
|
||||
// of order, moving saved progress backward or forward past events still owed
|
||||
// a replay. Writes to the same persisted key are serialized and each is
|
||||
// ordered by issuance: a write issued before the last one to land is stale
|
||||
// and dropped. Deliberate regressions — resume floors, operator seeds — are
|
||||
// issued after the writes they supersede, so they still apply. State is keyed
|
||||
// by target filer so unrelated streams — or the two directions of a
|
||||
// bidirectional sync — never block or shadow each other.
|
||||
var offsetCheckpointWrites = struct {
|
||||
sync.Mutex
|
||||
byKey map[string]*offsetCheckpoint
|
||||
}{byKey: map[string]*offsetCheckpoint{}}
|
||||
|
||||
syncKey := []byte(signaturePrefix + "____")
|
||||
util.Uint32toBytes(syncKey[len(signaturePrefix):len(signaturePrefix)+4], uint32(signature))
|
||||
type offsetCheckpoint struct {
|
||||
sync.Mutex
|
||||
lastSeq int64
|
||||
}
|
||||
|
||||
// resumeTsNsFor floors the processed watermark by the earliest failure the
|
||||
// processor saw: a redelivery can clear the pin, but events dropped while it
|
||||
// was stopped only replay if the resume position stays behind it. It also
|
||||
// stays one tick below the newest timestamp seen — a watermark on the
|
||||
// frontier can have finished every job it admitted while more events at the
|
||||
// same timestamp still sit unread in the log, and an exclusive resume there
|
||||
// would skip them.
|
||||
func resumeTsNsFor(p *MetadataProcessor) int64 {
|
||||
ts := p.processedTsWatermark.Load()
|
||||
if newestTsNs := p.NewestSeenTsNs(); newestTsNs > 0 && ts >= newestTsNs {
|
||||
ts = newestTsNs - 1
|
||||
}
|
||||
if failedTsNs := p.EarliestFailedTsNs(); failedTsNs > 0 && failedTsNs-1 < ts {
|
||||
ts = failedTsNs - 1
|
||||
}
|
||||
return ts
|
||||
}
|
||||
|
||||
// offsetWriteSeq numbers checkpoint writes in issuance order so a write that
|
||||
// lands late can be told apart from the fresher write it would shadow.
|
||||
var offsetWriteSeq atomic.Int64
|
||||
|
||||
func setOffset(grpcDialOption grpc.DialOption, filer pb.ServerAddress, signaturePrefix string, signature int32, offsetTsNs int64) error {
|
||||
syncKey := []byte(signaturePrefix + "____")
|
||||
util.Uint32toBytes(syncKey[len(signaturePrefix):len(signaturePrefix)+4], uint32(signature))
|
||||
key := string(filer) + "\x00" + string(syncKey)
|
||||
|
||||
offsetCheckpointWrites.Lock()
|
||||
checkpoint := offsetCheckpointWrites.byKey[key]
|
||||
if checkpoint == nil {
|
||||
checkpoint = &offsetCheckpoint{}
|
||||
offsetCheckpointWrites.byKey[key] = checkpoint
|
||||
}
|
||||
offsetCheckpointWrites.Unlock()
|
||||
|
||||
seq := offsetWriteSeq.Add(1)
|
||||
checkpoint.Lock()
|
||||
defer checkpoint.Unlock()
|
||||
if seq < checkpoint.lastSeq {
|
||||
return nil
|
||||
}
|
||||
checkpoint.lastSeq = seq
|
||||
|
||||
return pb.WithFilerClient(false, signature, filer, grpcDialOption, func(client filer_pb.SeaweedFilerClient) error {
|
||||
|
||||
valueBuf := make([]byte, 8)
|
||||
util.Uint64toBytes(valueBuf, uint64(offsetTsNs))
|
||||
|
||||
resp, err := client.KvPut(context.Background(), &filer_pb.KvPutRequest{
|
||||
// a stalled KvPut must not hold a caller's drain or persistence loop
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
resp, err := client.KvPut(ctx, &filer_pb.KvPutRequest{
|
||||
Key: syncKey,
|
||||
Value: valueBuf,
|
||||
})
|
||||
@@ -600,7 +662,6 @@ func setOffset(grpcDialOption grpc.DialOption, filer pb.ServerAddress, signature
|
||||
return nil
|
||||
|
||||
})
|
||||
|
||||
}
|
||||
|
||||
func genProcessFunction(sourcePath string, targetPath string, excludePaths []string, reExcludeFileName *regexp.Regexp, excludeFileNames []*wildcard.WildcardMatcher, excludePathPatterns []*wildcard.WildcardMatcher, dataSink sink.ReplicationSink, doDeleteFiles bool, debug bool) func(resp *filer_pb.SubscribeMetadataResponse) error {
|
||||
|
||||
Reference in new issue
Block a user