3 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandDevin	ad6257ebbc	filer.backup: add -s3.flattenVersions to mirror the current S3 view of versioned buckets (#12255) ... * filer.backup: add -s3FlattenVersions to mirror the current S3 view Versioned buckets are stored as <key>.versions/v_<versionId> files plus a latest-version pointer on the .versions directory, so a backup mirror held internal version files and could never be restored through the S3 API. With -s3FlattenVersions the backup maps events on .versions directories onto the flattened object key: a new latest version uploads to <key>, a latest delete marker or a cleared pointer deletes <key> when -doDeleteFiles is set, and the suspended-versioning null object resolves to the regular path. Version files themselves are never mirrored. The initial snapshot seeds only the version the pointer names. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: serialize flattened writes per object key Same-path non-barrier dir updates run concurrently in the metadata processor, so two pointer updates for one object -- or a pointer update and a regular-path write to it -- could overlap and let the older upload finish last. Lock on the flattened key: versions events lock the object key, every other event its own path, via a striped mutex. * filer.backup: apply exclusions to the flattened object key File-name and path excludes ran against the .versions entry, so an excluded object was still copied flat and its deletion still removed the destination copy. Match the rules against the flattened key on both the apply and the versions-dir-removal paths. * filer.backup: re-resolve the pointer on version-file events A tag or attribute update on the current version touches the v_* file, not the .versions pointer, so dropping every version-file event left the mirror with stale tags until the next pointer flip. Re-resolve the pointer on each version-file event; noncurrent versions still stay out. * filer.backup: tolerate vanished versions dirs in flattened snapshot The flattened seed path returned source errors before the existing missing-source policy ran, so one version deleted mid-walk could force a full re-walk. Apply the same 404/superseded skip as the generic seed branch, and cover the review fixes with tests. * filer.backup: lock both flattened keys an event touches A rename or delete can retire one flattened key while writing another, so serialize on the old- and new-side keys together. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: refresh the flattened key only for the still-current child A v_* event on a non-pointed version cannot move the flattened key, so skip the re-upload; tag and attribute updates on the current version still flow through. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: let resolved-latest writes bypass the source staleness check A filer sink skips a CreateEntry whose source path no longer holds the entry, but a flattened write lands on a key whose source path is not the one it was read from, so the check dropped every flattened upload. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: reclaim destination chunks on flattened deletes Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: route regular-path events for versioned keys through the pointer A null-object write or delete at the regular path can coexist with a .versions pointer that names a live version; the pointer decides the flattened key, so deleting the null object must not remove it. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: make the checkpoint identity cover the flattening mode Resuming a checkpoint taken in the other mode would skip the flattened re-seed and leave mirrored version files behind; a mode change now starts a fresh snapshot. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover flattened regular-path routing and per-mode checkpoints Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: resolve both sides of a flattened rename * test: cover flattened renames with one versioned side * replication: let resolved-latest writes overwrite a newer destination The stale-destination skip trusted a newer destination mtime, but a resolved-latest write can legitimately install an older version over the copy of a since-deleted latest. Skip only on provably identical content (matching chunk provenance, or equal ETag with full byte coverage). * filer.backup: rescan .versions when the latest null object is gone Deleting ?versionId=null removes the regular-path object without touching the null-is-latest pointer, so the flattened key kept its stale copy. Fall back to the newest remaining version (or a flat delete), the same rescan the S3 read path applies. * replication: compare inline and remote content in the destination-holds checks A chunk-less entry keeps its bytes in Content and a remote-only entry in the object RemoteEntry points at, but the identical-content shortcuts only compared chunk lists and chunk-derived ETags. Two different inline files looked identical, so a flattened re-copy or a same-mtime rewrite kept the previous version's bytes. Compare Content bytes and remote object identity wherever the checks claim the destination already holds the reference's content. * filer.backup: paginate the null-latest .versions rescan A limit=0 listing is capped at DirListingLimit server-side, and old-format version names sort oldest-first, so a history longer than one page left the flattened copy on an older version after the null object was removed. Page through the directory like the S3 read path does. * replication: require matching writable metadata for resolved-latest skips A skip keeps the destination's stored metadata, so identical bytes were not enough: stale attributes or extended entries (e.g. a superseded version id) survived under equal content. Compare the fields a rewrite controls — attributes minus filer-assigned inode/crtime, with the store's atime/ctime zero-defaults applied — plus the extended map, in both the provenance and ETag fast paths. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: require the extended key to exist when comparing metadata * replication: ignore a stored rule TTL when the resolved write carries none * filer.backup: rename -s3FlattenVersions to -s3.flattenVersions Match the -s3.* flag naming convention used by the other s3 options. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-30 09:09:05 -07:00
3 people	81c90cb446	filer.backup: retry a failed checkpoint read instead of replaying from the epoch (#12036) ... * filer.backup: retry a failed checkpoint read instead of replaying from the epoch Fixes #12035 * test: reset sink.local.directory in backup offset test cleanup The new test sets sink.local.directory on the shared viper but only reset the enabled flag, leaking a key that points at a deleted t.TempDir() to later tests in the package. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover the retried attempt re-reading the checkpoint The fake filer now fails only its first KvGet, and the test runs a second doFilerBackup attempt asserting the checkpoint is re-read and the attempt proceeds past the offset stage. LookupDirectoryEntry is stubbed so the SSE-S3 KEK probe returns a miss instead of retrying 10x2s. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 08:22:29 -07:00
Chris Lu	b3be2f5449	filer.backup, filer.sync: stop sharing resume checkpoints across destinations (#10934) ... * filer.backup: key the checkpoint by source path and sink destination The checkpoint id hashed only sink name + directory, so two backups to different buckets or endpoints sharing a directory layout advanced one checkpoint: whichever job was running pushed the shared offset forward, and a stopped or failing job later resumed from the other's position, silently skipping changes. Backups of different source paths to the same destination shared a checkpoint the same way. Each sink now reports a destination identity (endpoint or account, bucket or container, directory) and the checkpoint is keyed by the source path plus that identity. Reads fall back to the historical name+directory key when the new key has no value, so existing backups resume where they left off; writes go only to the new key. * filer.sync: include the target path in the offset key The offset stored on the target filer was keyed by source path and source filer signature only, so two syncs from the same source cluster and path to different directories on the same target cluster advanced one shared checkpoint, and the slower one could resume past events it never applied. The target path now participates in the key; "/" keeps the historical form, and a sync with a non-root target path falls back to the historical key once when its own key has no value yet. * join checkpoint key fields with NUL so they cannot alias A path or configuration value spelling out the separator could concatenate two different field tuples to the same checkpoint key. NUL cannot appear in a CLI path argument or any sane configuration value, making the encoding injective.	2026-08-24 19:30:20 -07:00