filer.remote.sync: skip an upload whose source entry was deleted or rewritten (#11149)

* filer.remote.sync: skip an upload whose source entry was deleted or rewritten

A replay from an earlier offset (-timeAgo) re-emits create and update
events for entries the filer has since deleted or rewritten. Their chunks
are gone from the volume servers, so the upload can never succeed, and
failing the event holds the sync offset before it: every restart of the
subscription replays it into the same dead chunks, and progress on
everything after it in the log is never persisted. One such entry stops
replication for the whole mount.

When the upload fails, look the entry up on the filer. Gone, or holding
other content than the event described, the event is superseded and is
skipped with an error log; the event that superseded it follows in the
log and brings the remote to the current state. Otherwise the failure
stands and the event is retried as before.

Fixes #11148

* filer.remote.sync: compare chunks by file id when deciding an event is superseded

filer.IsSameData compares chunk ETags, so a delete-and-recreate of
identical bytes, which stores the same content under new file ids and
drops the old ones, looked still as described and kept failing the event
on its dead chunks. Compare by file id with DoMinusChunks, the way the
filer itself decides which chunks an update leaves for deletion: the
event is superseded when the current entry no longer references every
chunk it named, and still as described when it does, including when more
chunks were appended after it.

* filer.remote.sync: ask the filer on the first failed upload attempt, not after the backoff

The superseded check ran after util.Retry had given up, so every dead
entry still cost the full retry cycle, about 13s, before it was skipped:
the SDK reports a missing chunk as "RequestError", which
IsTransientError takes as worth retrying. Move the check into the retry
loop with util.RetryOnError. Any failed attempt asks the filer, and the
loop stops at once when the entry is gone, surfacing errSuperseded for
the caller to skip. An entry the filer still holds keeps the retry policy
it had.

filer.remote.gateway shares retriedWriteFile and the same offset-pinning
processor, so its three call sites skip a superseded event the same way.

weed/command/filer_remote_sync_dir.go

@@ -1,6 +1,7 @@
 package command
 
 import (
+	"bytes"
 	"context"
 	"errors"
 	"fmt"
@@ -177,7 +178,11 @@ func (option *RemoteSyncOptions) makeEventProcessor(remoteStorage *remote_pb.Rem
 				return client.WriteDirectory(dest, message.NewEntry)
 			}
 			glog.V(0).Infof("create %s", remote_storage.FormatLocation(dest))
-			remoteEntry, writeErr := retriedWriteFile(client, filerSource, message.NewEntry, dest)
+			remoteEntry, writeErr := retriedWriteFile(client, filerSource, message.NewParentPath, message.NewEntry, dest)
+			if errors.Is(writeErr, errSuperseded) {
+				glog.Errorf("skipping %s: %v", remote_storage.FormatLocation(dest), writeErr)
+				return nil
+			}
 			if writeErr != nil {
 				return writeErr
 			}
@@ -247,7 +252,11 @@ func (option *RemoteSyncOptions) makeEventProcessor(remoteStorage *remote_pb.Rem
 					return nil
 				}
 			}
-			remoteEntry, writeErr := retriedWriteFile(client, filerSource, message.NewEntry, dest)
+			remoteEntry, writeErr := retriedWriteFile(client, filerSource, message.NewParentPath, message.NewEntry, dest)
+			if errors.Is(writeErr, errSuperseded) {
+				glog.Errorf("skipping %s: %v", remote_storage.FormatLocation(dest), writeErr)
+				return nil
+			}
 			if writeErr != nil {
 				return writeErr
 			}
@@ -259,18 +268,55 @@ func (option *RemoteSyncOptions) makeEventProcessor(remoteStorage *remote_pb.Rem
 	return eachEntryFunc, nil
 }
 
-func retriedWriteFile(client remote_storage.RemoteStorageClient, filerSource *source.FilerSource, newEntry *filer_pb.Entry, dest *remote_pb.RemoteStorageLocation) (remoteEntry *filer_pb.RemoteEntry, err error) {
-	var writeErr error
-	err = util.Retry("writeFile", func() error {
+// isSuperseded reports whether the filer has moved past the entry an event
+// described: it is deleted, or it no longer references every chunk the event
+// named. Those are the chunks the filer deletes when an entry is updated, so
+// they are gone from the volume servers, or about to be, and no retry of the
+// upload can succeed. Chunks are compared by file id, not content: a rewrite
+// stores even identical bytes under new ids and drops the old ones. A replay
+// from an earlier offset (-timeAgo) re-emits such events. Failing one holds the
+// sync offset before it, so every restart of the subscription replays it into
+// the same dead chunks, and progress on everything after it in the log is never
+// persisted. Skipping is safe: the event that superseded this one follows in
+// the log, and the delete removes the remote object or the rewrite uploads the
+// current content. A lookup that fails for any other reason keeps the write
+// failure, so the event is retried.
+func isSuperseded(filerClient filer_pb.FilerClient, dir string, entry *filer_pb.Entry) bool {
+	current, _, _, err := filer_pb.GetEntry(context.Background(), filerClient, util.NewFullPath(dir, entry.Name))
+	if errors.Is(err, filer_pb.ErrNotFound) {
+		return true
+	}
+	if err != nil {
+		return false
+	}
+	if len(entry.Content) > 0 || len(current.Content) > 0 {
+		return !bytes.Equal(entry.Content, current.Content)
+	}
+	return len(filer.DoMinusChunks(entry.GetChunks(), current.GetChunks())) > 0
+}
+
+// errSuperseded marks an upload that failed for an entry the filer has since
+// moved past (isSuperseded). The caller skips the event instead of failing it.
+var errSuperseded = errors.New("deleted or rewritten since the event was logged")
+
+// retriedWriteFile uploads the entry, retrying transient failures. Every failed
+// attempt first asks the filer whether the entry is superseded, and stops at
+// once when it is: a dead chunk reads as a transient "RequestError" from the
+// SDK, and waiting out the backoff on it buys nothing.
+func retriedWriteFile(client remote_storage.RemoteStorageClient, filerSource filer_pb.FilerClient, dir string, newEntry *filer_pb.Entry, dest *remote_pb.RemoteStorageLocation) (remoteEntry *filer_pb.RemoteEntry, err error) {
+	err = util.RetryOnError("writeFile", func(err error) bool {
+		return !errors.Is(err, errSuperseded) && util.IsTransientError(err)
+	}, func() error {
 		reader := filer.NewFileReader(filerSource, newEntry)
 		glog.V(0).Infof("create %s", remote_storage.FormatLocation(dest))
+		var writeErr error
 		remoteEntry, writeErr = client.WriteFile(dest, newEntry, reader)
-		if writeErr != nil {
-			return writeErr
+		if writeErr != nil && isSuperseded(filerSource, dir, newEntry) {
+			return fmt.Errorf("%s %w: %w", util.NewFullPath(dir, newEntry.Name), errSuperseded, writeErr)
 		}
-		return nil
+		return writeErr
 	})
-	if err != nil {
+	if err != nil && !errors.Is(err, errSuperseded) {
 		glog.Errorf("write to %s: %v", dest, err)
 	}
 	return