* filer.backup: add -s3FlattenVersions to mirror the current S3 view
Versioned buckets are stored as <key>.versions/v_<versionId> files plus a
latest-version pointer on the .versions directory, so a backup mirror held
internal version files and could never be restored through the S3 API.
With -s3FlattenVersions the backup maps events on .versions directories
onto the flattened object key: a new latest version uploads to <key>, a
latest delete marker or a cleared pointer deletes <key> when
-doDeleteFiles is set, and the suspended-versioning null object resolves
to the regular path. Version files themselves are never mirrored. The
initial snapshot seeds only the version the pointer names.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: serialize flattened writes per object key
Same-path non-barrier dir updates run concurrently in the metadata
processor, so two pointer updates for one object -- or a pointer update
and a regular-path write to it -- could overlap and let the older upload
finish last. Lock on the flattened key: versions events lock the object
key, every other event its own path, via a striped mutex.
* filer.backup: apply exclusions to the flattened object key
File-name and path excludes ran against the .versions entry, so an
excluded object was still copied flat and its deletion still removed the
destination copy. Match the rules against the flattened key on both the
apply and the versions-dir-removal paths.
* filer.backup: re-resolve the pointer on version-file events
A tag or attribute update on the current version touches the v_* file,
not the .versions pointer, so dropping every version-file event left the
mirror with stale tags until the next pointer flip. Re-resolve the
pointer on each version-file event; noncurrent versions still stay out.
* filer.backup: tolerate vanished versions dirs in flattened snapshot
The flattened seed path returned source errors before the existing
missing-source policy ran, so one version deleted mid-walk could force a
full re-walk. Apply the same 404/superseded skip as the generic seed
branch, and cover the review fixes with tests.
* filer.backup: lock both flattened keys an event touches
A rename or delete can retire one flattened key while writing another,
so serialize on the old- and new-side keys together.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: refresh the flattened key only for the still-current child
A v_* event on a non-pointed version cannot move the flattened key, so
skip the re-upload; tag and attribute updates on the current version
still flow through.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: let resolved-latest writes bypass the source staleness check
A filer sink skips a CreateEntry whose source path no longer holds the
entry, but a flattened write lands on a key whose source path is not the
one it was read from, so the check dropped every flattened upload.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: reclaim destination chunks on flattened deletes
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: route regular-path events for versioned keys through the pointer
A null-object write or delete at the regular path can coexist with a
.versions pointer that names a live version; the pointer decides the
flattened key, so deleting the null object must not remove it.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: make the checkpoint identity cover the flattening mode
Resuming a checkpoint taken in the other mode would skip the flattened
re-seed and leave mirrored version files behind; a mode change now
starts a fresh snapshot.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover flattened regular-path routing and per-mode checkpoints
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: resolve both sides of a flattened rename
* test: cover flattened renames with one versioned side
* replication: let resolved-latest writes overwrite a newer destination
The stale-destination skip trusted a newer destination mtime, but a
resolved-latest write can legitimately install an older version over the
copy of a since-deleted latest. Skip only on provably identical content
(matching chunk provenance, or equal ETag with full byte coverage).
* filer.backup: rescan .versions when the latest null object is gone
Deleting ?versionId=null removes the regular-path object without touching
the null-is-latest pointer, so the flattened key kept its stale copy.
Fall back to the newest remaining version (or a flat delete), the same
rescan the S3 read path applies.
* replication: compare inline and remote content in the destination-holds checks
A chunk-less entry keeps its bytes in Content and a remote-only entry in
the object RemoteEntry points at, but the identical-content shortcuts
only compared chunk lists and chunk-derived ETags. Two different inline
files looked identical, so a flattened re-copy or a same-mtime rewrite
kept the previous version's bytes. Compare Content bytes and remote
object identity wherever the checks claim the destination already holds
the reference's content.
* filer.backup: paginate the null-latest .versions rescan
A limit=0 listing is capped at DirListingLimit server-side, and
old-format version names sort oldest-first, so a history longer than one
page left the flattened copy on an older version after the null object
was removed. Page through the directory like the S3 read path does.
* replication: require matching writable metadata for resolved-latest skips
A skip keeps the destination's stored metadata, so identical bytes were not
enough: stale attributes or extended entries (e.g. a superseded version id)
survived under equal content. Compare the fields a rewrite controls —
attributes minus filer-assigned inode/crtime, with the store's atime/ctime
zero-defaults applied — plus the extended map, in both the provenance and
ETag fast paths.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: require the extended key to exist when comparing metadata
* replication: ignore a stored rule TTL when the resolved write carries none
* filer.backup: rename -s3FlattenVersions to -s3.flattenVersions
Match the -s3.* flag naming convention used by the other s3 options.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.sync: skip superseded replays before fetching any chunks
On a directory under heavy overwrite, most entries the sync replays are
already superseded on the source. The sink only learned that after chunk
reads failed — or worse, after fetching and uploading bytes a later event
replaces — so the backlog drained slowly and the log filled with 404s.
CreateEntry and UpdateEntry now check sourceMovedOn before touching a
chunk: one entry lookup skips a stale replay entirely. A mid-replication
supersession is still caught by the existing retry gates.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: bound the source liveness probes with a timeout
The supersession checks added a source entry lookup to the replication
path, and every lookup ran on context.Background() with no deadline. A
source that accepts the call but whose metadata store stalls would hold
the sync forever — before a single chunk fetch, even when every chunk is
still readable. Bound the probe at 30s; an unanswered lookup proves
nothing, so replication proceeds and the existing retry gates decide.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: sync the live version a superseded replay points at
Skipping a stale replay assumed a later event always re-carries the
current content. It does not when the superseding write was itself
replicated in from another cluster: those events are dropped before they
reach a filer sink, so the destination could be left missing the file.
When the source reports a live entry newer than the replay, commit that
version through the normal path instead — equivalent to applying the
later event, with no dependence on its delivery. A gone source entry
still skips, since the destination already matches the source.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: reconcile moved entry content after a native rename
AtomicRenameEntry relocates whatever version the destination held. If the
preceding update replay was skipped because the source had already moved
to the new name, the moved entry keeps the old bytes and — with no
further writes — stays stale forever. MoveEntry now runs the same
reconciliation as an in-place update against the rename's snapshot, so
the moved entry picks up pending content changes.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: repair from the live snapshot a superseded replay resolves to
An incremental diff against the replayed oldEntry can resolve manifests the
overwrite already vacuumed, which fails every retry of the event.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A remote-only entry whose object was deleted from the remote storage
outside the filer answered GET with 500 and stayed in the filer. The
remote's not-found was lost on the way: the backends' ReadFile returned
it as an untyped error, so FetchAndWriteNeedle failed with codes.Unknown
and nothing downstream could tell it from any other failure.
- remote_storage: GCS, S3 (NoSuchKey) and Azure (BlobNotFound) reads
return ErrRemoteObjectNotFound. GCS reports a missing bucket the same
way as a missing object, so it confirms the bucket with a listing.
- volume server and filer: the not-found crosses gRPC as codes.NotFound
carrying the sentinel's text, and the filer's cache RPC returns
codes.NotFound, which the S3 gateway already maps to NoSuchKey.
- s3api: the origin fallback answers NoSuchKey on a confirmed not-found.
- filer: a confirmed not-found removes the stale entry, so the lazy
remote-metadata cache converges on the remote. Only remote-only files
outside .versions and without an active object lock are removed, only
if unchanged since the fetch (checked on the object's write owner,
under the lock S3 object writes take), and with a metadata-only
delete: the filer skips its inline remote delete, and the delete
events' entries carry a marker that makes filer.remote.sync and
filer.remote.gateway skip their remote delete, while filer.sync still
replicates it. The replicated DeleteEntryRequest carries
keep_remote_object, so the destination's delete events are marked too.
The store drops the marker from every write, so clients cannot plant
it.
* filer.backup: key the checkpoint by source path and sink destination
The checkpoint id hashed only sink name + directory, so two backups to
different buckets or endpoints sharing a directory layout advanced one
checkpoint: whichever job was running pushed the shared offset forward,
and a stopped or failing job later resumed from the other's position,
silently skipping changes. Backups of different source paths to the same
destination shared a checkpoint the same way.
Each sink now reports a destination identity (endpoint or account,
bucket or container, directory) and the checkpoint is keyed by the
source path plus that identity. Reads fall back to the historical
name+directory key when the new key has no value, so existing backups
resume where they left off; writes go only to the new key.
* filer.sync: include the target path in the offset key
The offset stored on the target filer was keyed by source path and
source filer signature only, so two syncs from the same source cluster
and path to different directories on the same target cluster advanced
one shared checkpoint, and the slower one could resume past events it
never applied. The target path now participates in the key; "/" keeps
the historical form, and a sync with a non-root target path falls back
to the historical key once when its own key has no value yet.
* join checkpoint key fields with NUL so they cannot alias
A path or configuration value spelling out the separator could
concatenate two different field tuples to the same checkpoint key.
NUL cannot appear in a CLI path argument or any sane configuration
value, making the encoding injective.
* fix(filer.sync): replicate a rename as create-then-delete, not an in-place update
A rename arrives as a single metadata event carrying both the old and new
entry. The filer sink was routed to UpdateEntry, which looks up the old
path but issues the update against the new parent without changing the
name — and the filer UpdateEntry RPC cannot move an entry. So the rename
was dropped: the old path lingered and the new path never appeared
(same-dir renames rewrote the old name in place).
Route a real move (the sink path changed) through CreateEntry(new) then
DeleteEntry(old) in both the replicator and the filer.sync/backup driver,
the way the other sinks already handle it; reach UpdateEntry only for true
in-place updates. Create before delete so a crash between the two leaves
the entry visible rather than lost.
* fix(filer.sync): derive the rename delete key like the create key, guard the watched root
The rename delete leg rebuilt the old key with a raw util.Join, bypassing the
sink-side key normalization the create leg gets from buildKey — so a rename
could create the new entry and then fail to delete the old one under a
transformed key. Build the old key through buildKey too, and skip the delete
when the moved entry is the watched root itself (where the old key would
resolve to the target root and recursively delete the whole sink tree).
* test(filer.sync): cover the in-place update delete-then-create fallback order
The recording sinks always reported foundExisting, so the fallback that an
in-place update takes when the entry is missing on the sink was never run.
Make it configurable and assert the fallback deletes before it recreates the
same key, in both the replicator and the filer.sync drivers.
* feat(filer.sync): move filer-sink renames natively via AtomicRenameEntry
create-then-delete is unsafe for the filer sink: CreateEntry returns nil
without creating on a transient chunk-copy error, so the paired delete could
remove the only valid destination copy; a directory rename also deleted the
old subtree before descendants were recreated, and left old chunks behind.
Add an optional EntryMover sink capability and implement it on the filer sink
via AtomicRenameEntry — one atomic, metadata-only move that relocates a whole
subtree in a single transaction. Renames prefer it; sinks without a native
move keep create-then-delete. When the old path is already gone (a descendant
the parent rename moved, or one never replicated) MoveEntry creates the new
path instead, re-checking existence with a lookup so a rolled-back move that
left the old entry intact is retried rather than mistaken for gone.
* docs(filer.sync): note entryMissing's gRPC not-found string fallback is deliberate
