Files
seaweedfs/weed/mount/weedfs_dir_mkrm.go
T
Chris Lu da2e90aefd fix(mount): sanitize non-UTF-8 filenames; keep marshal errors per-request (#9207)
* fix(mount): sanitize non-UTF-8 filenames; keep marshal errors per-request (#9139)

A single file with invalid-UTF-8 bytes in its name (e.g. a GNOME Trash
"partial" like \x10\x98=\\\x8a\x7f.trashinfo.9a51454f.partial) made every
FUSE-initiated filer RPC fail with:

  rpc error: code = Internal desc = grpc: error while marshaling:
  string field contains invalid UTF-8

and then produced an avalanche of "connection is closing" errors on
unrelated LookupEntry / ReadDirAll / UpdateEntry calls, causing the
volume-server QPS dips reported in #9139.

Root cause is twofold:

1. Proto3 `string` fields require valid UTF-8, but the FUSE kernel passes
   raw name bytes. Create/Mknod/Mkdir/Unlink/Rmdir/Rename/Lookup/Link/
   Symlink all forwarded those bytes directly into CreateEntryRequest.Name,
   DeleteEntryRequest.Name, StreamRenameEntryRequest.{Old,New}Name and
   Entry.Name. saveDataAsChunk also copied the FullPath into
   AssignVolumeRequest.Path unchecked.

2. When the marshal failed, shouldInvalidateConnection treated the
   resulting codes.Internal as a connection problem and dropped the
   shared cached ClientConn — canceling every other in-flight RPC on it.

Fix:

- Add sanitizeFuseName (strings.ToValidUTF8 with '?' replacement, matching
  util.FullPath.DirAndName) and make checkName return the sanitized name.
  Apply at every FUSE entry point that passes a name to the filer RPC,
  including Unlink/Rmdir (which did not previously call checkName) and
  both oldName/newName in Rename. Add a backstop scrub for
  AssignVolumeRequest.Path so async flush paths cannot reintroduce
  invalid bytes from a pre-sanitization cached FullPath.

- In weed/pb.shouldInvalidateConnection, detect client-side marshal
  errors via the gRPC library's "error while marshaling" prefix and
  return false: the connection is healthy, only the request is bad.

Refs: https://github.com/seaweedfs/seaweedfs/issues/9139#issuecomment-4301184231

* fix(mount,util): use '_' for invalid-UTF-8 replacement (URL-safe)

Sanitized filenames flow downstream into HTTP URLs (volume-server uploads,
filer HTTP API, S3/WebDAV gateways). '?' is the URL query-string
delimiter and would split the path the first time the name lands in one,
so swap every invalid-UTF-8 replacement to '_'. This covers the two
pre-existing sites in weed/util/fullpath.go as well, keeping all paths
sanitized the same way.

* refactor(pb): detect client-side marshal errors via errors.As, not substring

Replace the raw `strings.Contains(err.Error(), ...)` check with a
type-based carve-out: use errors.As against the `GRPCStatus() *Status`
interface to pull the original Status out of any fmt.Errorf("...: %w")
wrapping, then match the library-owned "grpc:" prefix on that Status's
Message.

Why not errors.Is against a proto-level sentinel: gRPC's encode()
collapses the inner proto error with "%v" (stringification) before
wrapping it in a Status, so the original error type does not survive
into the caller. The Status itself is the structural signal that does
survive.

Why not status.FromError: when the caller wraps the Status error with
fmt.Errorf("...: %w", ...), status.FromError rewrites Status.Message
with the full err.Error() of the outermost wrapper, which defeats a
prefix check on the library-owned message. errors.As gives us the
original Status whose Message is still verbatim from the gRPC library.

A new test asserts that a plain errors.New("grpc: error while marshaling: …")
— i.e. the same text attached to something that is NOT a gRPC status —
does not short-circuit invalidation, so we never silently keep a cached
connection alive based on a coincidental substring match.

* refactor(util): centralize UTF-8 sanitization; add FullPath.Sanitized

Addresses review feedback on PR #9207.

Nitpick: every invalid-UTF-8 replacement across the codebase (DirAndName,
Name, mount.sanitizeFuseName, the weedfs_write.go backstop) now goes
through a single util.SanitizeUTF8Name helper, so the replacement char
('_' — URL-safe) is chosen in one place.

Outside-diff: three proto fields took raw FullPath strings that could
break marshaling if an entry ever carried invalid UTF-8
(CreateEntryRequest.Directory in Mkdir, DeleteEntryRequest.Directory in
Unlink, AssignVolumeRequest.Path in command_fs_merge_volumes). The
reviewer's suggested fix — using DirAndName() — would have silently
changed Directory from parent to grandparent, because DirAndName
sanitizes only the trailing component. Added FullPath.Sanitized(), which
scrubs every component, and applied it at the three sites. Exposure is
narrow in practice (FUSE-boundary sanitization and the gRPC-side
isClientSideMarshalError carve-out already cover the #9139 cascade),
but the defense-in-depth is cheap and consistent with the existing
AssignVolume backstop.

New tests in weed/util/fullpath_test.go document:
- SanitizeUTF8Name: valid UTF-8 passes through unchanged; invalid bytes
  become '_' (not '?', which is URL-special).
- FullPath.Sanitized: scrubs bytes in any component, not just the last.
- FullPath.DirAndName: dir remains raw on purpose — callers needing a
  clean full path must use Sanitized(). The test pins this behavior so
  it is not accidentally "fixed" in a way that changes the (dir, name)
  semantics callers depend on.
2026-04-23 19:17:35 -07:00

186 lines
6.0 KiB
Go

package mount
import (
"context"
"os"
"strings"
"syscall"
"time"
"github.com/seaweedfs/go-fuse/v2/fuse"
"github.com/seaweedfs/seaweedfs/weed/filer"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
)
/** Create a directory
*
* Note that the mode argument may not have the type specification
* bits set, i.e. S_ISDIR(mode) can be false. To obtain the
* correct directory type bits use mode|S_IFDIR
* */
func (wfs *WFS) Mkdir(cancel <-chan struct{}, in *fuse.MkdirIn, name string, out *fuse.EntryOut) (code fuse.Status) {
if wfs.IsOverQuotaWithUncommitted() {
return fuse.Status(syscall.ENOSPC)
}
var s fuse.Status
if name, s = checkName(name); s != fuse.OK {
return s
}
now := time.Now().Unix()
dirFullPath, code := wfs.inodeToPath.GetPath(in.NodeId)
if code != fuse.OK {
return
}
entryFullPath := dirFullPath.Child(name)
// Pre-allocate the mount's local inode and stamp it into the create
// request so both the mount and the filer agree on object identity from
// the start. Without this, the filer assigns its own inode in CreateEntry
// and the cached entry then reports a different value than the one we
// return to the kernel here.
inode := wfs.inodeToPath.AllocateInode(entryFullPath, now)
newEntry := &filer_pb.Entry{
Name: name,
IsDirectory: true,
Attributes: &filer_pb.FuseAttributes{
Mtime: now,
Crtime: now,
Ctime: now,
FileMode: uint32(os.ModeDir) | in.Mode,
Uid: in.Uid,
Gid: in.Gid,
Inode: inode,
},
}
wfs.mapPbIdFromLocalToFiler(newEntry)
// Defer restoring to local uid/gid AFTER the entry is sent to the filer
// but BEFORE outputPbEntry writes attributes to the kernel. We restore
// explicitly below instead of using defer so the kernel gets local values.
request := &filer_pb.CreateEntryRequest{
// Defensive: dirFullPath is clean by construction for mount-originated
// mutations, but could carry invalid-UTF-8 bytes if metaCache was
// populated from a non-gRPC source (direct store write, legacy import).
// Sanitizing here keeps the marshal strictly per-request on the off
// chance invalid bytes do reach us.
Directory: dirFullPath.Sanitized(),
Entry: newEntry,
Signatures: []int32{wfs.signature},
SkipCheckParentDirectory: true,
}
glog.V(1).Infof("mkdir: %v", request)
resp, err := wfs.streamCreateEntry(context.Background(), request)
if err != nil {
glog.V(0).Infof("mkdir %s: %v", entryFullPath, err)
} else {
event := resp.GetMetadataEvent()
if event == nil {
event = metadataCreateEvent(string(dirFullPath), newEntry)
}
if applyErr := wfs.applyLocalMetadataEvent(context.Background(), event); applyErr != nil {
glog.Warningf("mkdir %s: best-effort metadata apply failed: %v", entryFullPath, applyErr)
wfs.inodeToPath.InvalidateChildrenCache(dirFullPath)
}
wfs.inodeToPath.TouchDirectory(dirFullPath)
wfs.touchDirMtimeCtimeBest(dirFullPath)
wfs.inodeToPath.AdjustSubdirCount(dirFullPath, 1)
}
glog.V(3).Infof("mkdir %s: %v", entryFullPath, err)
if err != nil {
wfs.mapPbIdFromFilerToLocal(newEntry)
return fuse.EIO
}
// Map uid/gid back to local-space before writing attributes to the
// kernel. The kernel (especially macFUSE) caches these and uses them
// for subsequent permission checks on children.
wfs.mapPbIdFromFilerToLocal(newEntry)
inode = wfs.inodeToPath.Lookup(entryFullPath, newEntry.Attributes.Crtime, true, false, inode, true)
// The newly created directory is guaranteed to be empty, so mark it as
// cached immediately to avoid a needless filer round-trip on the first
// Lookup or ReadDir inside this directory.
wfs.inodeToPath.MarkChildrenCached(entryFullPath)
wfs.outputPbEntry(out, inode, newEntry)
return fuse.OK
}
/** Remove a directory */
func (wfs *WFS) Rmdir(cancel <-chan struct{}, header *fuse.InHeader, name string) (code fuse.Status) {
if name == "." {
return fuse.Status(syscall.EINVAL)
}
if name == ".." {
return fuse.Status(syscall.ENOTEMPTY)
}
// Sanitize before it reaches DeleteEntryRequest.Name; see sanitizeFuseName.
name = sanitizeFuseName(name)
dirFullPath, code := wfs.inodeToPath.GetPath(header.NodeId)
if code != fuse.OK {
return
}
entryFullPath := dirFullPath.Child(name)
// POSIX: enforce sticky bit on the parent directory.
if dirEntry, dirCode := wfs.maybeLoadEntry(dirFullPath); dirCode == fuse.OK && dirEntry != nil && dirEntry.Attributes != nil {
targetUid := uint32(0)
if targetEntry, targetCode := wfs.maybeLoadEntry(entryFullPath); targetCode == fuse.OK && targetEntry != nil && targetEntry.Attributes != nil {
targetUid = targetEntry.Attributes.Uid
}
if code := checkStickyBit(dirEntry.Attributes.FileMode, dirEntry.Attributes.Uid, targetUid, header.Uid); code != fuse.OK {
return code
}
}
glog.V(3).Infof("remove directory: %v", entryFullPath)
deleteReq := &filer_pb.DeleteEntryRequest{
Directory: string(dirFullPath),
Name: name,
IsDeleteData: true,
IgnoreRecursiveError: true, // ignore recursion error since the OS should manage it
Signatures: []int32{wfs.signature},
}
resp, err := wfs.streamDeleteEntry(context.Background(), deleteReq)
if err != nil {
glog.V(1).Infof("remove %s: %v", entryFullPath, err)
if strings.Contains(err.Error(), filer.MsgFailDelNonEmptyFolder) {
return fuse.Status(syscall.ENOTEMPTY)
}
return fuse.ENOENT
}
event := metadataDeleteEvent(string(dirFullPath), name, true)
if resp != nil && resp.MetadataEvent != nil {
event = resp.MetadataEvent
}
if applyErr := wfs.applyLocalMetadataEvent(context.Background(), event); applyErr != nil {
glog.Warningf("rmdir %s: best-effort metadata apply failed: %v", entryFullPath, applyErr)
wfs.inodeToPath.InvalidateChildrenCache(dirFullPath)
}
wfs.inodeToPath.RemovePath(entryFullPath)
wfs.inodeToPath.TouchDirectory(dirFullPath)
wfs.touchDirMtimeCtimeBest(dirFullPath)
wfs.inodeToPath.AdjustSubdirCount(dirFullPath, -1)
return fuse.OK
}