mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-22 22:11:39 +02:00
filer.remote.sync: do not upload an entry whose only content is the remote object (#11890)
* filer.remote.sync: do not upload an entry whose local copy was dropped remote.uncache and the filer's eviction under storage pressure drop a synced entry's chunks and clear LastLocalSyncTsNs; remote.meta.sync does the same when it pulls a newer remote version. When the remote mtime falls in the entry's mtime second, which is the common case for a small file synced promptly, shouldSendToRemote falls back to the cleared stamp and reads the entry as dirty. The dropped chunks make the update a content change, so the chunkless entry was uploaded and a zero-byte object replaced the remote one, the entry's only content. Skip an update that leaves the entry at the same key with no local data and a sized RemoteEntry: the object at that key already is the content. The stamp stays cleared, so remote.cache and eviction still see the entry as uncached. Renames keep going through completeRemoteOnlyRename, metadata-only edits of a remote-only entry still reach the remote, and a rewrite in the remote mtime's second holds local data and is uploaded. * filer.remote.sync: upload a truncate that leaves the entry remote-only A FUSE truncate of a cached, remote-backed file to zero (truncateEntry, or SetAttr with size 0) drops the chunks and keeps the sized RemoteEntry, so the entry reads as remote-only and the dropped-copy skip left the remote object with its old content. Skip only while the entry's size is still the remote object's. Uncache and eviction keep the size and remote.meta.sync sets it from the object, so those stay skipped; a truncate changes the size and is uploaded, in the remote mtime's second too.
This commit is contained in:
1 parent
f8cfea0dcf
commit
03065155db
2 files changed
+219
No files matched your search
@@ -384,6 +384,16 @@ func processUpdateEvent(
|
||||
}
|
||||
glog.V(0).Infof("never replicated, uploading %s", remote_storage.FormatLocation(dest))
|
||||
}
|
||||
if proto.Equal(oldDest, dest) && isDroppedLocalCopy(message.NewEntry) {
|
||||
// The update dropped the local copy: remote.uncache, eviction under
|
||||
// storage pressure, or remote.meta.sync pulling a newer remote
|
||||
// version. Each clears the sync stamp, so shouldSendToRemote reads
|
||||
// the entry as dirty when the remote mtime shares its second. The
|
||||
// content is the object at this very key, and uploading the chunkless
|
||||
// entry would replace it with zero bytes.
|
||||
glog.V(2).Infof("skipping update of remote-only entry: %+v", resp)
|
||||
return nil
|
||||
}
|
||||
glog.V(2).Infof("update: %+v", resp)
|
||||
if !proto.Equal(oldDest, dest) {
|
||||
// A renamed entry that holds no local data now (the snapshot was
|
||||
@@ -488,6 +498,15 @@ func isRemoteOnly(entry *filer_pb.Entry) bool {
|
||||
return entry != nil && !filer.HasData(entry) && entry.IsInRemoteOnly()
|
||||
}
|
||||
|
||||
// isDroppedLocalCopy reports a remote-only entry whose size is still the
|
||||
// remote object's: its local copy was dropped and the object describes it.
|
||||
// Uncache and eviction keep the size, and remote.meta.sync sets it from the
|
||||
// object. A truncate drops the chunks too but changes the size, and the
|
||||
// truncated content has to reach the remote.
|
||||
func isDroppedLocalCopy(entry *filer_pb.Entry) bool {
|
||||
return isRemoteOnly(entry) && entry.GetAttributes().GetFileSize() == uint64(entry.RemoteEntry.RemoteSize)
|
||||
}
|
||||
|
||||
// completeRemoteOnlyRename finishes a rename whose content exists only on the
|
||||
// remote. When the destination object already holds what the entry's stamp
|
||||
// describes (the rename ran before, its offset was not persisted, and
|
||||
|
||||
Reference in new issue
Block a user