remote.uncache, remote.meta.sync, remote.cache: keep local writes that land while they run (#12382)

* remote.uncache: skip a file overwritten since the walk read it

The uncache cleared the chunks of the entry its directory walk returned and
wrote it back unconditionally. A file overwritten after the walk read it
was reset to the remote pointer, and its new chunks were dropped, so an
acknowledged write was lost. Write back only while the filer still holds
the entry as read, and report the file as skipped otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: keep files written locally after the directory listing

meta.sync lists each local directory once and works from that listing.
A changed remote file was written back over the listed entry
unconditionally, and a name missing from the listing was created without
O_EXCL, so a file written locally in between was replaced by the remote
pointer either way. Update only while the filer still holds the listed
entry, create exclusively, and report such files as skipped. The create
now also surfaces errors the filer returns in its response.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.cache: keep files written locally after the lookup

The metadata step looks each file up and then either rewrites it from the
remote unconditionally or creates it without O_EXCL, so a local write
landing between the lookup and the write was replaced by the remote
pointer. Reuse doSaveRemoteEntry, which updates only while the filer still
holds the looked-up entry, create exclusively, and report such files as
skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: do not descend into a name a local file took

When the exclusive create of a remote directory finds the name taken, the
sync still descended into it, and creating the remote children under a
local file would turn that file into a directory. Descend only when the
name now holds a directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: delete only the file version it listed

The type-change and -deleteStale paths removed a remote-backed file
unconditionally, so a local write landing after the directory listing was
deleted with it. Condition the delete on the listed mtime, as fs.verify
and volume.fsck do; the filer then keeps a file written since.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: condition directory and mtime-less deletes too

The listed-mtime cutoff was only sent for files, so a remote-backed
directory replaced locally by a file between the emptiness check and the
delete was removed unconditionally, and so was a file listed without an
mtime. Send the cutoff for directories as well, and for an entry listed
without an mtime keep it once a write has set one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test that DeleteEntry applies the request's mtime cutoff

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: keep the listed nanoseconds as the cutoff of an epoch mtime

An entry listed at the epoch with a sub-second mtime was sent a 1 ns
cutoff, so the filer kept it unchanged although the old code deleted it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: report an entry the filer kept instead of deleting it

The filer answers a delete it skipped under the mtime cutoff with
success, so the sync printed (delete) for an entry a local write kept.
deleteLocalEntry now looks the name up after the delete and prints the
outcome, and a replaced file that was kept is not recreated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

weed/shell/command_remote_meta_sync.go

@@ -2,6 +2,7 @@ package shell
 
 import (
 	"context"
+	"errors"
 	"flag"
 	"fmt"
 	"io"
@@ -214,10 +215,13 @@ func pullMetadataDirectory(ctx context.Context, client filer_pb.SeaweedFilerClie
 					continue
 				}
 			} else {
-				fmt.Fprintf(writer, "%s (delete)\n", localPath)
-				if err := deleteLocalEntry(ctx, client, localDir, name); err != nil {
+				kept, err := deleteLocalEntry(ctx, client, writer, localDir, existingEntry)
+				if err != nil {
 					return err
 				}
+				if kept {
+					continue
+				}
 			}
 			existingEntry = nil
 		}
@@ -240,9 +244,20 @@ func pullMetadataDirectory(ctx context.Context, client filer_pb.SeaweedFilerClie
 		}
 
 		if existingEntry == nil {
-			if err := createRemoteEntry(ctx, client, writer, localDir, name, child, remoteLoc.Name); err != nil {
+			taken, err := createRemoteEntry(ctx, client, writer, localDir, name, child, remoteLoc.Name)
+			if err != nil {
 				return err
 			}
+			if taken && child.isDirectory {
+				// descending into a name a local file took would turn it into a directory
+				resp, err := filer_pb.LookupEntry(ctx, client, &filer_pb.LookupDirectoryEntryRequest{Directory: string(localDir), Name: name})
+				if err != nil && !errors.Is(err, filer_pb.ErrNotFound) {
+					return err
+				}
+				if resp == nil || !resp.Entry.IsDirectory {
+					continue
+				}
+			}
 		} else if !child.isDirectory {
 			if existingEntry.RemoteEntry == nil {
 				// a local change that should not be overwritten
@@ -255,9 +270,13 @@ func pullMetadataDirectory(ctx context.Context, client filer_pb.SeaweedFilerClie
 				// persist the stat-derived encoding so the next run skips the stat
 				(statFiles && existingEntry.RemoteEntry.RemoteContentEncoding == nil &&
 					child.remoteEntry.RemoteContentEncoding != nil) {
-				fmt.Fprintf(writer, "%s (update)\n", localPath)
-				if err := doSaveRemoteEntry(client, string(localDir), existingEntry, child.remoteEntry); err != nil {
+				stale, err := doSaveRemoteEntry(client, string(localDir), existingEntry, child.remoteEntry)
+				if stale {
+					fmt.Fprintf(writer, "%s (skip, changed since listed)\n", localPath)
+				} else if err != nil {
 					return err
+				} else {
+					fmt.Fprintf(writer, "%s (update)\n", localPath)
 				}
 			} else {
 				fmt.Fprintf(writer, "%s (skip)\n", localPath)
@@ -297,8 +316,8 @@ func deleteRemoteBackedEntry(ctx context.Context, client filer_pb.SeaweedFilerCl
 		if existingEntry.RemoteEntry == nil {
 			return nil
 		}
-		fmt.Fprintf(writer, "%s (delete)\n", localPath)
-		return deleteLocalEntry(ctx, client, localDir, name)
+		_, err := deleteLocalEntry(ctx, client, writer, localDir, existingEntry)
+		return err
 	}
 
 	children, err := listLocalDirectory(ctx, client, localPath)
@@ -319,14 +338,35 @@ func deleteRemoteBackedEntry(ctx context.Context, client filer_pb.SeaweedFilerCl
 		return err
 	}
 	if empty {
-		fmt.Fprintf(writer, "%s (delete)\n", localPath)
-		return deleteLocalEntry(ctx, client, localDir, name)
+		_, err = deleteLocalEntry(ctx, client, writer, localDir, existingEntry)
 	}
-	return nil
+	return err
 }
 
-func deleteLocalEntry(ctx context.Context, client filer_pb.SeaweedFilerClient, localDir util.FullPath, name string) error {
-	return filer_pb.DoRemove(ctx, client, string(localDir), name, true, false, false, false, nil)
+// deleteLocalEntry removes the entry as it was listed; the filer keeps an entry
+// whose mtime a local write has moved past the listed one, reported as kept.
+// An entry listed without an mtime is kept once a write sets one.
+func deleteLocalEntry(ctx context.Context, client filer_pb.SeaweedFilerClient, writer io.Writer, localDir util.FullPath, entry *filer_pb.Entry) (kept bool, err error) {
+	req := &filer_pb.DeleteEntryRequest{Directory: string(localDir), Name: entry.Name, IsDeleteData: true}
+	var ok bool
+	req.IfNotModifiedAfter, req.IfNotModifiedAfterNs, ok = deleteEntryCutoff(entry.Attributes.GetMtime(), entry.Attributes.GetMtimeNs())
+	if !ok {
+		req.IfNotModifiedAfterNs = max(1, int64(entry.Attributes.GetMtimeNs()))
+	}
+	if _, err = filer_pb.DoRemoveRequest(ctx, client, req); err != nil {
+		return false, err
+	}
+	localPath := localDir.Child(entry.Name)
+	_, err = filer_pb.LookupEntry(ctx, client, &filer_pb.LookupDirectoryEntryRequest{Directory: string(localDir), Name: entry.Name})
+	if err == nil {
+		fmt.Fprintf(writer, "%s (skip, changed since listed)\n", localPath)
+		return true, nil
+	}
+	if !errors.Is(err, filer_pb.ErrNotFound) {
+		return false, err
+	}
+	fmt.Fprintf(writer, "%s (delete)\n", localPath)
+	return false, nil
 }
 
 func isLocalDirectoryEmpty(ctx context.Context, client filer_pb.SeaweedFilerClient, dir util.FullPath) (bool, error) {
@@ -338,7 +378,9 @@ func isLocalDirectoryEmpty(ctx context.Context, client filer_pb.SeaweedFilerClie
 	return empty, err
 }
 
-func createRemoteEntry(ctx context.Context, client filer_pb.SeaweedFilerClient, writer io.Writer, localDir util.FullPath, name string, child *remoteChild, storageName string) error {
+// createRemoteEntry creates the entry only if the name is still free; taken
+// reports that something was created there after the directory was listed.
+func createRemoteEntry(ctx context.Context, client filer_pb.SeaweedFilerClient, writer io.Writer, localDir util.FullPath, name string, child *remoteChild, storageName string) (taken bool, err error) {
 	attributes := &filer_pb.FuseAttributes{
 		FileMode: remoteEntryFileMode(child.isDirectory),
 		TtlSec:   0, // Remote entries should not have TTL
@@ -356,7 +398,7 @@ func createRemoteEntry(ctx context.Context, client filer_pb.SeaweedFilerClient,
 		attributes.FileSize = uint64(child.remoteEntry.RemoteSize)
 		attributes.Mtime = child.remoteEntry.RemoteMtime
 	}
-	_, err := client.CreateEntry(ctx, &filer_pb.CreateEntryRequest{
+	err = filer_pb.CreateEntry(ctx, client, &filer_pb.CreateEntryRequest{
 		Directory: string(localDir),
 		Entry: &filer_pb.Entry{
 			Name:        name,
@@ -365,9 +407,17 @@ func createRemoteEntry(ctx context.Context, client filer_pb.SeaweedFilerClient,
 			Extended:    filer.MergeRemoteContentEncoding(remoteEntry, nil),
 			RemoteEntry: remoteEntry,
 		},
+		OExcl: true,
 	})
+	if errors.Is(err, filer_pb.ErrEntryAlreadyExists) {
+		fmt.Fprintf(writer, "%s (skip, created since listed)\n", localDir.Child(name))
+		return true, nil
+	}
+	if err != nil {
+		return false, err
+	}
 	fmt.Fprintf(writer, "%s (create)\n", localDir.Child(name))
-	return err
+	return false, nil
 }
 
 func childRemoteLocation(remoteLoc *remote_pb.RemoteStorageLocation, name string) *remote_pb.RemoteStorageLocation {