5 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandClaude Opus 5.5	728b476ba2	filer.remote.gateway: keep the remote object while a version under a delete marker is locked (#12459) ... * filer.remote.gateway: keep the remote object while a version under a delete marker is locked Every version of a versioned bucket uploads to the object's key, so on a remote without versioning that object is the only remote copy of the latest version. A delete marker deleted it even when a version still held a legal hold or unexpired retention. Keep it while any version of the key is locked, the same rule the filer uses before pruning a remote entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway, filer.remote.sync: say that earlier versions need remote versioning Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: delete a kept delete marker's object once the version lock is released A delete marker kept for a locked version stays unstamped. When an update or delete leaves a version that carried a legal hold or retention without an active lock, look up the key's latest version and, if it is still that unstamped marker, run the delete again. Only changes to versions that carried a lock trigger it, so the noncurrent mark and tag updates of unlocked versions do not send repeat remote deletes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: test the kept delete marker retry through the bucket mount The retry reads the key's versions through a filer client parameter, as the rename helpers do, so a test can check the mapped object key and the marker stamp, and that lock-releasing updates and deletes reach it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: check the lock release lookup goes to the gateway's filer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 13:37:08 -08:00
4 people	206d7b1418	filer.remote.sync: bounded filer lookups, one chunk cache per process (#12441) ... * filer.remote.sync: bounded filer lookups, one chunk cache per process The filer lookups on the sync path (currentEntry, isSuperseded, uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no deadline. A lookup that never answers keeps its job alive, the job keeps its path in the MetadataProcessor conflict index, AddSyncJob waits on the condition and the daemon stops consuming events with no log line. Each lookup now carries a 30 s deadline, so the job fails and the processor's stop-and-resubscribe path runs. remoteUploadReader built a chunk cache per upload; the cache's worker goroutine is never stopped, so each upload leaked one goroutine and its cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory limit). One chunk cache per process. (cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78) * filer.remote.sync: a failed mount lookup at startup never resolves to now collectLastSyncOffset returned time.Now() when the mounted directory lookup failed, which skips every event since the saved offset. The saved offset is used instead; without one the daemon exits so a restart retries. (cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae) * filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read retriedWriteFile returns the supersession lookup's error instead of treating an unanswered lookup as "not superseded": in a replay an older snapshot could otherwise overwrite a newer remote object. The processor retries the event. isSuperseded keeps its boolean form for the other call sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls back to the saved offset cannot hang on it. TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed lookup; it now asserts the withheld upload and the lookup error. (cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa) * filer.remote.sync test: keep the slices import the other gateway tests use * filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo After a failed upload attempt the supersession lookup's error ended in isSuperseded's boolean and the loop could upload the snapshot again unverified. The lookup error now ends the loop (errLookupUnanswered, not retried) and the processor retries the event. With -timeAgo set, a failed read of the saved offset resolved to now minus timeAgo, skipping everything older; the daemon exits instead so a restart retries. (cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566) * filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting resolveLastSyncOffset returns the lookup or offset-read error; both callers (directory and buckets) hand it to util.RetryUntil, which retries with backoff instead of the process exiting. * filer.remote.sync: keep doc comments on the functions they describe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 952d80c4d3) * filer.remote.sync: isSuperseded returns the lookup error itself Its only production callers needed the error, so the boolean wrapper was left to the tests. The tests now assert the error on a failed lookup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 51f603b5ec) * filer.remote.sync: bound the stamp written after each upload updateLocalEntry's UpdateEntry was the last filer call on the sync path without a deadline. A filer that never answers the stamp keeps the job and its path in the conflict index the same way an unanswered lookup did. The stamp carries IF_ENTRY_EQUAL, so a retried event after a timeout either stamps again or skips a stamp that already landed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit c7338fb68f) * filer.remote.sync: an unread offset never resolves to the mount's creation time With the offset read now bounded, a timed-out read fell through to the directory's creation time and replayed every event already synced. The error goes back to the startup retry loop, as with -timeAgo; a missing offset still reads as zero without an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 0a1b6c2944) * filer.remote.sync: stamp a copy of the event entry updateLocalEntry set the RemoteEntry and dropped the rename marker on the caller's entry before the RPC. A transient stamp failure retries the same event in place, which then read as already synced and skipped both the upload and the stamp. The stub filer now applies a successful update the way the filer does, since tests had relied on that shared object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 1f332e5866) * filer.remote.sync: a directory not created yet starts now, as before A gateway started before the first bucket looked up a missing buckets directory, had no saved offset, and returned an error to the retry loop on every attempt until a bucket appeared. A missing directory with no saved offset has nothing to replay, so it starts now as it did before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit c31497a9a3) * filer.remote.sync: take the start time before the mount lookup A bucket created between a not-found lookup and the end of the offset read had its creation before a start taken afterwards, so the gateway never created its remote bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit f535fca8f6) * filer.remote.sync: save the resume position when a subscription ends A failed event ends the subscription with ErrResubscribe, and the retry loop runs startup again, recomputing -timeAgo from a later now. The periodic checkpoint only runs as events arrive, so a failure before the first one could start the retry after the event it failed on. The resume position is now saved whenever the subscription returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 7be919ded1) * remote_storage: bound the sync offset write like the read The offset write runs in the subscription's checkpoint callback and, now, when a subscription ends; a filer that never answers it stalled either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit f888ac6b8f) * filer.remote.sync: say that any saved offset wins over the creation time The creation time in seconds was compared with the offset in milliseconds, which made any saved offset win. That is the behaviour to keep: remote.unmount recreates the directory and then saves the offset, usually in the same second, so a seconds comparison would start at the creation second and replay the purge. The comparison now states it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a0d60ee00f) * filer.remote.sync: keep the resume position for the retry when its write fails The retry in the same process now resumes from the position the ended subscription returned, as filer.backup does, so a failed offset write followed by an empty read cannot recompute -timeAgo past a failed event. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 39ef551bcf) * filer.remote.sync test: give the same-second saved offset a fractional part With whole seconds, returning the creation second resolved to the same time and passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 4bb74382a6) --------- Co-authored-by: james <james@medable.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 09:35:16 -08:00
3 people	80f32bc074	weed/command: skip remote bucket delete when bucket is unmounted (#12418) (#12425) ... * weed/command: skip remote bucket delete when bucket is unmounted * remote_storage: count deleting an already-missing bucket as done filer.remote.gateway can see the same bucket delete more than once, for example once per filer when several filers peer their metadata. A copy that still finds the mount mapping calls DeleteBucket again, and S3 (NoSuchBucket), Azure (ContainerNotFound, ContainerBeingDeleted) and GCS (404) all failed it, which held the sync offset on the delete. A missing bucket now counts as deleted, the way deleting a missing object already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: keep the mapping table writable after mount.mapping is deleted Deleting mount.mapping replaced the gateway's mappings with a table whose map was nil, so the next bucket create panicked writing its mount into it. Reset to an empty map instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: count a delete of an absent object as done filer.remote.sync already treats ErrRemoteObjectNotFound from DeleteFile as deleted, since GCS reports a missing object that way where S3 and Azure succeed. The gateway returned it, so a delete event seen twice, as it is when several filers peer their metadata, held the sync offset on GCS. Use the same deleteRemoteFile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: drop cached remote clients after the new gateway tests GetRemoteStorage caches clients by configuration, so a repeated run (-count=2) got the first run's recorder back and saw no calls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.gateway: keep the default storage across mount.mapping changes The default storage for new buckets is derived at startup and kept only in the gateway's copy of the mappings; mount.mapping never stores it. Replacing or deleting that file dropped it, so a later bucket create with no -createBucketAt skipped the remote bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 05:06:41 -08:00
Chris LuandDevin	67240f538d	filer.remote.gateway: stop zero-byte uploads of remote-only entries (#12110) ... * filer.remote.gateway: skip uploading a dropped local copy on update The bucketed update path uploaded the chunkless entry whenever the update was not metadata-only. remote.uncache, eviction, and remote.meta.sync drop a synced entry's chunks and clear its sync stamp, which reads as dirty when the remote mtime shares the entry's second; the upload then replaced the object that is the entry's only content with zero bytes. Skip the upload the same way the mounted-directory sync does. * filer.remote.gateway: copy a remote-only entry's object on rename A rename deleted the source object and uploaded the chunkless entry, replacing the entry's only content with zero bytes — the same loss remote.uncache caused on plain updates. Complete the rename on the remote the way the mounted-directory sync does: copy the source object over (or take the destination when it already describes the entry), restamp so reads stop resolving through the pending-source marker, and remove the source only while it is still the object the stamp describes. A remote-only entry renamed out of the gateway's coverage keeps its object, and the plain rename's old-key delete only removes the object the event's stamp still describes. * filer.remote.gateway: scope the sync-stamp skip to same-path updates shouldSendToRemote reads the entry's sync stamp, which still covers a rename's unchanged mtime, so a synced or pending-stamped rename event was skipped before its copy or upload could run. Only updates in place may skip on the stamp. * filer.remote.gateway: keep the source a pending rename moved on from A rename event whose pending destination has since been renamed again still names the source a later event reads through its own marker; deleting the old key here loses the only copy. Mirror the remote.sync check: leave the source when the event's entry carries the pending-source marker and no entry holds the destination any more. * filer.remote.gateway: release a rename's pending source once complete A rename whose old entry read through a pending-source marker leaves that earlier source behind: the destination is released by its own key, never the marker's. Mirror releaseRenamedSource — once the renamed entry no longer reads through a marker, delete the pending source guarded by the old entry's stamp, keeping it while a remote-only entry holds its path. * filer.remote.gateway: finalize a complete rename before failing its source A rename source outside the gateway's coverage errored before the destination was checked, and the recorded failure held the event offset. Stat the destination first: when it already holds the content the entry describes, refresh the stamp and leave the unmanaged source untouched; fail only when nothing holds the content. * filer.remote.sync: document the source-of-truth protocol both processors share --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-26 18:03:12 -07:00
Peter Dodd	5f1a742726	fix: return NoSuchKey and drop stale entries when a remote-mounted object is gone from the remote (#11680) ... A remote-only entry whose object was deleted from the remote storage outside the filer answered GET with 500 and stayed in the filer. The remote's not-found was lost on the way: the backends' ReadFile returned it as an untyped error, so FetchAndWriteNeedle failed with codes.Unknown and nothing downstream could tell it from any other failure. - remote_storage: GCS, S3 (NoSuchKey) and Azure (BlobNotFound) reads return ErrRemoteObjectNotFound. GCS reports a missing bucket the same way as a missing object, so it confirms the bucket with a listing. - volume server and filer: the not-found crosses gRPC as codes.NotFound carrying the sentinel's text, and the filer's cache RPC returns codes.NotFound, which the S3 gateway already maps to NoSuchKey. - s3api: the origin fallback answers NoSuchKey on a confirmed not-found. - filer: a confirmed not-found removes the stale entry, so the lazy remote-metadata cache converges on the remote. Only remote-only files outside .versions and without an active object lock are removed, only if unchanged since the fetch (checked on the object's write owner, under the lock S3 object writes take), and with a metadata-only delete: the filer skips its inline remote delete, and the delete events' entries carry a marker that makes filer.remote.sync and filer.remote.gateway skip their remote delete, while filer.sync still replicates it. The replicated DeleteEntryRequest carries keep_remote_object, so the destination's delete events are marked too. The store drops the marker from every write, so clients cannot plant it.	2026-10-10 11:00:04 +08:00