fix(remote): correct content and permissions when syncing/caching remote objects (#9879)

* fix(remote): reject short reads when caching remote objects

A short read from the remote (stale listing size, truncated or flaky
response) was silently zero-padded: the S3 and Azure clients pre-size
the buffer and discard the downloaded byte count, and the chunk is
recorded with the requested size. The cached file then matched the
expected size but its tail was NULL, and the entry was marked cached
so it never re-fetched.

Check the byte count against the requested size in both clients, and
add a backend-agnostic guard in FetchAndWriteNeedle. The cache now
fails loudly and the entry stays remote-only for a later retry.

* fix(remote): match S3 default modes when syncing remote metadata

Remote object listings carry no POSIX mode, so synced entries were
created with a hardcoded 0644. Against a SeaweedFS remote, whose S3
layer writes objects as 0660 and auto-creates directories as 0771
(0660|0111), the mounted copy ended up 0644/0755 and the permissions
visibly diverged from the source.

Default to the S3 modes instead (files 0660, directories 0771). The
filer derives parent-dir modes from the child as fileMode|0111, so
fixing the file default also brings the directories into line.

Directory mtimes still reflect sync time: S3 listings don't enumerate
directories, so the remote's directory timestamps aren't available.
This commit is contained in:
Chris Lu authored and GitHub committed 2026-06-08 13:55:53 -07:00
1 parent 6475d22774
commit 8cc10460b4
5 files changed
+36 -4

No files matched your search

+16 -1
View File
@@ -5,6 +5,7 @@ import (
"flag"
"fmt"
"io"
"os"
"github.com/seaweedfs/seaweedfs/weed/filer"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
@@ -121,6 +122,20 @@ If entry.RemoteEntry == nil, this is a new local change and should not be overwr
the remote version is updated, need to pull meta
}
*/
// remoteEntryFileMode returns the POSIX mode for an entry synced from a remote
// whose listing carries no mode. It matches what SeaweedFS S3 assigns native
// objects (files 0660) so a mounted bucket matches the source, deriving the
// directory mode with the same 0111 traversal mask the filer uses for
// auto-created parents (0660 -> 0771).
func remoteEntryFileMode(isDirectory bool) uint32 {
mode := uint32(0660)
if isDirectory {
mode = uint32(os.ModeDir) | mode | 0111
}
return mode
}
func pullMetadata(commandEnv *CommandEnv, writer io.Writer, localMountedDir util.FullPath, remoteMountedLocation *remote_pb.RemoteStorageLocation, dirToCache util.FullPath, remoteConf *remote_pb.RemoteConf) error {
// visit remote storage
@@ -159,7 +174,7 @@ func pullMetadata(commandEnv *CommandEnv, writer io.Writer, localMountedDir util
Attributes: &filer_pb.FuseAttributes{
FileSize: uint64(remoteEntry.RemoteSize),
Mtime: remoteEntry.RemoteMtime,
FileMode: uint32(0644),
FileMode: remoteEntryFileMode(isDirectory),
TtlSec: 0, // Remote entries should not have TTL
},
RemoteEntry: remoteEntry,