mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-28 07:50:44 +01:00
filer.remote.gateway: stop zero-byte uploads of remote-only entries (#12110)
* filer.remote.gateway: skip uploading a dropped local copy on update The bucketed update path uploaded the chunkless entry whenever the update was not metadata-only. remote.uncache, eviction, and remote.meta.sync drop a synced entry's chunks and clear its sync stamp, which reads as dirty when the remote mtime shares the entry's second; the upload then replaced the object that is the entry's only content with zero bytes. Skip the upload the same way the mounted-directory sync does. * filer.remote.gateway: copy a remote-only entry's object on rename A rename deleted the source object and uploaded the chunkless entry, replacing the entry's only content with zero bytes — the same loss remote.uncache caused on plain updates. Complete the rename on the remote the way the mounted-directory sync does: copy the source object over (or take the destination when it already describes the entry), restamp so reads stop resolving through the pending-source marker, and remove the source only while it is still the object the stamp describes. A remote-only entry renamed out of the gateway's coverage keeps its object, and the plain rename's old-key delete only removes the object the event's stamp still describes. * filer.remote.gateway: scope the sync-stamp skip to same-path updates shouldSendToRemote reads the entry's sync stamp, which still covers a rename's unchanged mtime, so a synced or pending-stamped rename event was skipped before its copy or upload could run. Only updates in place may skip on the stamp. * filer.remote.gateway: keep the source a pending rename moved on from A rename event whose pending destination has since been renamed again still names the source a later event reads through its own marker; deleting the old key here loses the only copy. Mirror the remote.sync check: leave the source when the event's entry carries the pending-source marker and no entry holds the destination any more. * filer.remote.gateway: release a rename's pending source once complete A rename whose old entry read through a pending-source marker leaves that earlier source behind: the destination is released by its own key, never the marker's. Mirror releaseRenamedSource — once the renamed entry no longer reads through a marker, delete the pending source guarded by the old entry's stamp, keeping it while a remote-only entry holds its path. * filer.remote.gateway: finalize a complete rename before failing its source A rename source outside the gateway's coverage errored before the destination was checked, and the recorded failure held the event offset. Stat the destination first: when it already holds the content the entry describes, refresh the stamp and leave the unmanaged source untouched; fail only when nothing holds the content. * filer.remote.sync: document the source-of-truth protocol both processors share --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
0f1e73cc3a
commit
67240f538d
3 files changed
+500
-24
No files matched your search
@@ -26,6 +26,27 @@ import (
|
||||
"google.golang.org/protobuf/proto"
|
||||
)
|
||||
|
||||
// Which side holds an entry's content, and the two processors that keep the
|
||||
// remote in step with it.
|
||||
//
|
||||
// An entry with local data (chunks or inline content) is authoritative: the
|
||||
// update uploads it. A chunkless entry stamped with a RemoteEntry is
|
||||
// remote-only: its content is the remote object its stamp describes, so the
|
||||
// chunkless entry is never uploaded, since it would write zero bytes over
|
||||
// the only copy, and the object goes only while the stamp still describes it
|
||||
// (remote_storage.StillCurrent). While the entry carries a
|
||||
// filer.ExtRemoteRenameSourceKey marker the content is the object at the
|
||||
// renamed-from path, until the rename's copy or upload restamps the entry
|
||||
// and clears it. A dropped local copy (remote.uncache, eviction,
|
||||
// remote.meta.sync) keeps the remote object's size; a size that moved off
|
||||
// the stamp (a truncate) is new content and uploads.
|
||||
//
|
||||
// processUpdateEvent serves filer remote.sync over one mounted directory;
|
||||
// makeBucketedEventProcessor in filer_remote_gateway_buckets.go serves filer
|
||||
// remote.gateway over the bucketed space. They implement the same protocol
|
||||
// and must change together: a guard added to one and not the other is how
|
||||
// the zero-byte re-upload on remote.uncache happened.
|
||||
|
||||
func followUpdatesAndUploadToRemote(option *RemoteSyncOptions, filerSource *source.FilerSource, mountedDir string) error {
|
||||
|
||||
// read filer remote storage mount mappings
|
||||
|
||||
Reference in new issue
Block a user