2 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandDevin	6e8a2cdb14	filer.backup: process metadata events through the concurrent processor (#11951) ... * filer.backup: process metadata events through the concurrent processor Each event was handled inline on the subscription loop, so one fetch+write at a time capped the stream at a few entries per second and a busy source kept building lag. Route events through the same MetadataProcessor filer.sync uses: it fans jobs out behind -concurrency while serializing same-path and subtree-barrier conflicts, and the checkpoint now advances on the processor's contiguous watermark via GetResumeTsNs, with Resubscribe replaying anything a failure pins. -disableErrorRetry keeps its log-and-skip meaning by returning nil after logging, since the processor would otherwise pin the checkpoint forever. * filer.backup: drain in-flight jobs before the stream retry loop reconnects FollowMetadata returns as soon as the stream ends while admitted jobs keep writing; the next doFilerBackup builds a fresh processor that cannot order its replayed writes against them, so an older upload could finish last and clobber a newer file. * filer.backup: checkpoint the completed watermark on a ticker and at stream end The offset callback only fires while events arrive, so work draining on a quiet stream stayed uncheckpointed until the next event and a restart recopied it. Persist the processed watermark every 3s when it advances, and once more after the post-stream drain. * filer.backup: bound the checkpoint wait before reconnecting setOffset has no context, so a stalled KvPut would hold persistDone and block the retry loop forever. The persist goroutine now performs the final write itself on stopPersist, and the caller waits at most 30s. * filer.backup: carry the processed position across retry attempts A follow attempt that ended with a pinned failure recomputed startFrom on retry, so a -timeAgo window could move past the failed event and skip it. doFilerBackup now returns the failure pin (or processed watermark) and the retry loop clamps startFrom to it. * filer.backup: resume retries at the previous attempt's pinned position The retry loop threaded resumeTsNs through but never consulted it, so a -timeAgo run recomputed startFrom as now-minus-the-window on every attempt — a failed event older than the window was skipped on reconnect. Use the pinned position when one exists; -timeAgo and the persisted checkpoint only choose the first starting point. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: resume below the failure pin so the failed event replays The metadata reader treats the resume position as exclusive — events at or below it are skipped — so resuming at the failed event's own timestamp would skip it. Back one tick off the pin. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: label the progress line as the received-event rate The counter increments when a job is admitted, not when its copy finishes, so the old wording showed backup throughput during a burst of slow copies while the processed watermark had not moved. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: bound the offset checkpoint RPC setOffset called KvPut on the background context, so a stalled write could pin a caller's drain or persistence loop indefinitely. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: keep the resume pin when filer-source setup fails A retried attempt carries resumeTsNs so the failed event replays from the same position, but filer-source and SSE initialization failures returned 0, so the next attempt recomputed -timeAgo and could move the window past the pinned event. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: serialize checkpoint writes per key A checkpoint write can outlive its caller's wait and land beside the next subscription attempt's writes, moving saved progress backward. Serialize writes per key inside setOffset and drop any value older than one already written from this process. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: gofmt * replication: serialize checkpoint writes per filer and key A single process-wide lock made a stalled checkpoint RPC block every stream's checkpoint, including the opposite direction of a bidirectional filer.sync. Keeping the last-written watermark per persisted key also let one stream's offset shadow another stream checkpointing the same key to a different target filer. Lock and track progress per (filer, key). * replication: keep the resume position behind a cleared failure pin A redelivery that clears failedTs lets the processed watermark advance past the earliest failure, but events dropped while the processor was stopped — possibly sharing the failure's timestamp — have to replay. Latched earliestFailedTsNs floors the resume position one tick below that failure so reconnects still cover them. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: persist floored checkpoints through the write ordering setOffset drops a value older than the last written, which also swallows a deliberate regression: the resume floor, an operator-seeded offset, or a snapshot high-water mark. allowRegress marks those callers so the checkpoint can move backward when a failure forces replay. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: order checkpoint writes by issuance, not value The value-based stale guard could never fire — every caller passed allowRegress because floors and seeds are deliberate regressions — and ordering by value cannot tell a stale high watermark apart from real progress: a write that outlives its attempt could still land after a floored resume and skip events owed a replay. Order by issuance instead: a write issued before the last one to land under the key is dropped, so late writes from a superseded attempt can no longer move the checkpoint in either direction while deliberate regressions still apply. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: keep the resume position one tick below the frontier * remote.sync: resume and persist through the floored watermark * test: concurrent metadata jobs fan out up to the limit Distinct-path events must all be in flight before any finishes — a serial loop could only ever have one running — while the limit still bounds the pool. Pins the property behind the backup-throughput fix. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-23 20:02:55 -07:00
Chris Lu	b3be2f5449	filer.backup, filer.sync: stop sharing resume checkpoints across destinations (#10934) ... * filer.backup: key the checkpoint by source path and sink destination The checkpoint id hashed only sink name + directory, so two backups to different buckets or endpoints sharing a directory layout advanced one checkpoint: whichever job was running pushed the shared offset forward, and a stopped or failing job later resumed from the other's position, silently skipping changes. Backups of different source paths to the same destination shared a checkpoint the same way. Each sink now reports a destination identity (endpoint or account, bucket or container, directory) and the checkpoint is keyed by the source path plus that identity. Reads fall back to the historical name+directory key when the new key has no value, so existing backups resume where they left off; writes go only to the new key. * filer.sync: include the target path in the offset key The offset stored on the target filer was keyed by source path and source filer signature only, so two syncs from the same source cluster and path to different directories on the same target cluster advanced one shared checkpoint, and the slower one could resume past events it never applied. The target path now participates in the key; "/" keeps the historical form, and a sync with a non-root target path falls back to the historical key once when its own key has no value yet. * join checkpoint key fields with NUL so they cannot alias A path or configuration value spelling out the separator could concatenate two different field tuples to the same checkpoint key. NUL cannot appear in a CLI path argument or any sane configuration value, making the encoding injective.	2026-08-24 19:30:20 -07:00