mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-19 02:15:02 +01:00
* mount: report opens for reading with TouchAccessTime (-accessReport) An open answered from the metadata cache never reaches the filer, so a filer that tracks access cannot see that a cached file is still being read. With -accessReport, the mount sends one TouchAccessTime per open for reading, with the open time, whether the open was answered from the cache or not. Write-only opens are not reported. Reports go out in the background and never hold up the open: a report that does not fit in the queue is dropped, and drops and failures are logged once a minute. A filer that answers Unimplemented pauses reporting for 10 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: address review of the access report - Drop reports still queued when reporting pauses; the filer that answered Unimplemented would answer them the same way. - Bound each filer attempt to 2s, so a filer that accepts the connection but never answers leaves time for the next one, and keep using the filer that answered. - Fail over only when a filer cannot be reached. Any answer, Unimplemented included, ends the walk. - Report an O_CREAT open of an existing file with read access, which comes through Create instead of Open. A new file has nothing to read and is not reported. - Stop the reporter on unmount: stop ends the workers and cancels the report in flight. - Say in the flag help that it has no effect unless the filer implements TouchAccessTime. - Tests no longer wait a fixed time to see that nothing else was sent; they look at the queue of a reporter that is not started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: fail an access report over only when the filer did not answer A status the filer sent ends the walk even when its message contains a word the transient classifier matches, so an Unimplemented answer still pauses reporting and a rejected request is not resent elsewhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep a filer selection made while an access report failed over A report that reached a later filer moves the mount there only if the selection it started from is still current, so a slower report cannot undo another call's failover. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: share an access report's time with the filers not yet tried Each attempt is capped by an equal share of the report time left, so a row of stalled filers cannot use up the report before a healthy one later in the list is tried. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: report the path an open for reading came through A create that opens an existing hard link reports the name it was given rather than the inode's first known name. Open reads the inode's path from the inode table rather than the handle's saved path, which other handle operations may rewrite concurrently. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: check the access report stop and failover tests without races The post-stop report is checked on a reporter without workers, so no worker can take an accepted report before the check, and the failover test asserts that the stalled filer was tried first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * pb: keep the shared channel when the peer answers with its own status gRPC never raises InvalidArgument, NotFound, AlreadyExists, FailedPrecondition, OutOfRange or DataLoss itself, so such a status is the peer's answer. It fell through to the string matching, and a message naming a path like /transport closed the cached channel and cancelled every other call on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: say which filer selection an access report keeps Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: bound the in-flight call in the rejected-report channel test The call held in flight went through touchAccessTime, whose 2 s attempt cap could expire on a slow runner and fail the test on a healthy channel. It now uses callFiler with the test deadline. The wait for it to reach the filer also ends if it returns early or the deadline passes, and the blocked handler is released on every exit path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: report each file at most once per -accessReport.interval Every read open used to send a TouchAccessTime, adding a filer metadata call per open. A path reported within the interval (default 1h) is now skipped; the last report times live in a bounded LRU. 0 keeps reporting every open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: report an open again once the file was modified since its last report Within -accessReport.interval a read open was skipped even after a write, so the filer kept an atime older than the mtime until the interval ran out. relatime records that first read after a write; do the same by comparing the entry's mtime with the last report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep the report time apart from the mtime it saw Recording max(now, mtime) as the last report let a file dated a day ahead suppress reports for a day plus the interval. Record when the report was sent and the mtime the entry had, and treat any change of that mtime as a modification, so neither clock can stretch or shorten the interval. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
209 lines
13 KiB
Go
209 lines
13 KiB
Go
package command
|
|
|
|
import (
|
|
"os"
|
|
"time"
|
|
)
|
|
|
|
type MountOptions struct {
|
|
filer *string
|
|
filerMountRootPath *string
|
|
dir *string
|
|
volumeName *string
|
|
dirAutoCreate *bool
|
|
collection *string
|
|
collectionQuota *int
|
|
logicalDiskUsage *bool
|
|
replication *string
|
|
diskType *string
|
|
ttlSec *int
|
|
chunkSizeLimitMB *int
|
|
concurrentWriters *int
|
|
concurrentReaders *int
|
|
readerCacheSizeMB *int64
|
|
memoryLimitMB *int64
|
|
cacheMetaTtlSec *int
|
|
cacheDirMaxEntries *int
|
|
cacheDirForRead *string
|
|
cacheDirForWrite *string
|
|
cacheSizeMBForRead *int64
|
|
writeBufferSizeMB *int64
|
|
dataCenter *string
|
|
allowOthers *bool
|
|
defaultPermissions *bool
|
|
umaskString *string
|
|
nonempty *bool
|
|
volumeServerAccess *string
|
|
uidMap *string
|
|
gidMap *string
|
|
readOnly *bool
|
|
includeSystemEntries *bool
|
|
debug *bool
|
|
debugPort *int
|
|
debugFuse *bool
|
|
localSocket *string
|
|
disableXAttr *bool
|
|
windowsUid *int
|
|
windowsGid *int
|
|
extraOptions []string
|
|
fuseCommandPid int
|
|
|
|
// Periodic metadata flush to protect against orphan chunk cleanup
|
|
metadataFlushSeconds *int
|
|
|
|
// Peer chunk sharing options (design-weed-mount-peer-chunk-sharing.md).
|
|
peerEnabled *bool
|
|
peerListen *string
|
|
peerAdvertise *string
|
|
peerDataCenter *string
|
|
peerRack *string
|
|
|
|
dirIdleEvictSec *int
|
|
|
|
// Distributed locking for cross-mount write coordination and POSIX
|
|
// advisory locks (flock/fcntl)
|
|
distributedLock *bool
|
|
|
|
// POSIX compliance options
|
|
posixDirNlink *bool
|
|
|
|
// report opens for reading to the filer
|
|
accessReport *bool
|
|
accessReportInterval *time.Duration
|
|
|
|
// FUSE performance options
|
|
writebackCache *bool
|
|
asyncDio *bool
|
|
cacheSymlink *bool
|
|
fuseMaxBackground *int
|
|
fuseCongestionThreshold *int
|
|
|
|
// macOS-specific FUSE options
|
|
novncache *bool
|
|
|
|
// if true, we assume autofs exists over current mount point. Autofs (the kernel one, used by systemd automount)
|
|
// is expected to be mounted as a shim between auto-mounted fs and original mount point to provide auto mount.
|
|
// with this option, we ignore autofs mounted on the same point.
|
|
hasAutofs *bool
|
|
}
|
|
|
|
var (
|
|
mountOptions MountOptions
|
|
mountCpuProfile *string
|
|
mountMemProfile *string
|
|
mountReadRetryTime *time.Duration
|
|
)
|
|
|
|
func init() {
|
|
cmdMount.Run = runMount // break init cycle
|
|
mountOptions.filer = cmdMount.Flag.String("filer", "localhost:8888", "comma-separated weed filer location")
|
|
mountOptions.filerMountRootPath = cmdMount.Flag.String("filer.path", "/", "mount this remote path from filer server")
|
|
mountOptions.dir = cmdMount.Flag.String("dir", ".", "mount weed filer to this directory")
|
|
mountOptions.volumeName = cmdMount.Flag.String("volumeName", "", "name the mount shown by the platform (Finder/Explorer, or \"mount\"/\"df\" on Linux and FreeBSD), overriding the name taken from -filer.path or -dir")
|
|
mountOptions.dirAutoCreate = cmdMount.Flag.Bool("dirAutoCreate", false, "auto create the directory to mount to")
|
|
mountOptions.collection = cmdMount.Flag.String("collection", "", "collection to create the files")
|
|
mountOptions.collectionQuota = cmdMount.Flag.Int("collectionQuotaMB", 0, "quota for the collection")
|
|
mountOptions.logicalDiskUsage = cmdMount.Flag.Bool("df.logical", false, "report data sizes to df and the quota, instead of the space they occupy with replicas and ec parity")
|
|
mountOptions.replication = cmdMount.Flag.String("replication", "", "replication(e.g. 000, 001) to create to files. If empty, let filer decide.")
|
|
mountOptions.diskType = cmdMount.Flag.String("disk", "", "[hdd|ssd|<tag>] hard drive or solid state drive or any tag")
|
|
mountOptions.ttlSec = cmdMount.Flag.Int("ttl", 0, "file ttl in seconds")
|
|
mountOptions.chunkSizeLimitMB = cmdMount.Flag.Int("chunkSizeLimitMB", 2, "local write buffer size, also chunk large files")
|
|
mountOptions.concurrentWriters = cmdMount.Flag.Int("concurrentWriters", 128, "limit concurrent goroutine writers")
|
|
mountOptions.concurrentReaders = cmdMount.Flag.Int("concurrentReaders", 128, "limit concurrent chunk fetches for read operations")
|
|
mountOptions.memoryLimitMB = cmdMount.Flag.Int64("memoryLimitMB", 0, "soft Go runtime memory limit in MiB; 0 preserves GOMEMLIMIT; leave headroom below the container limit")
|
|
mountOptions.readerCacheSizeMB = cmdMount.Flag.Int64("readerCacheSizeMB", 256, "memory budget in MiB for downloaded and in-flight reader buffers across all files; must fit the largest pooled chunk buffer")
|
|
mountOptions.cacheDirForRead = cmdMount.Flag.String("cacheDir", os.TempDir(), "local cache directory for file chunks and meta data")
|
|
mountOptions.cacheSizeMBForRead = cmdMount.Flag.Int64("cacheCapacityMB", 128, "file chunk read cache capacity in MB")
|
|
mountOptions.cacheDirForWrite = cmdMount.Flag.String("cacheDirWrite", "", "buffer writes mostly for large files")
|
|
mountOptions.writeBufferSizeMB = cmdMount.Flag.Int64("writeBufferSizeMB", 0, "global cap on the per-mount write buffer (memory + swap) in MB, 0 means unlimited. Bounds /tmp growth when volume uploads stall")
|
|
mountOptions.cacheMetaTtlSec = cmdMount.Flag.Int("cacheMetaTtlSec", 60, "metadata cache validity seconds")
|
|
mountOptions.cacheDirMaxEntries = cmdMount.Flag.Int("cacheDirMaxEntries", 100000, "a directory with more children than this is not cached locally but read directly from the filer; 0 caches everything")
|
|
mountOptions.dataCenter = cmdMount.Flag.String("dataCenter", "", "prefer to write to the data center")
|
|
mountOptions.allowOthers = cmdMount.Flag.Bool("allowOthers", true, "allows other users to access the file system; on Linux, when mounting as a non-root user, this requires 'user_allow_other' in /etc/fuse.conf, so set -allowOthers=false otherwise")
|
|
mountOptions.defaultPermissions = cmdMount.Flag.Bool("defaultPermissions", true, "enforce permissions by the operating system")
|
|
mountOptions.umaskString = cmdMount.Flag.String("umask", "022", "octal umask, e.g., 022, 0111")
|
|
mountOptions.nonempty = cmdMount.Flag.Bool("nonempty", false, "allows the mounting over a non-empty directory")
|
|
mountOptions.volumeServerAccess = cmdMount.Flag.String("volumeServerAccess", "direct", "access volume servers by [direct|publicUrl|filerProxy]")
|
|
mountOptions.uidMap = cmdMount.Flag.String("map.uid", "", "map local uid to uid on filer, comma-separated <local_uid>:<filer_uid>")
|
|
mountOptions.gidMap = cmdMount.Flag.String("map.gid", "", "map local gid to gid on filer, comma-separated <local_gid>:<filer_gid>")
|
|
mountOptions.readOnly = cmdMount.Flag.Bool("readOnly", false, "read only")
|
|
mountOptions.includeSystemEntries = cmdMount.Flag.Bool("includeSystemEntries", false, "show filer system entries (e.g. /topics, /etc) in directory listings")
|
|
mountOptions.debug = cmdMount.Flag.Bool("debug", false, "serves runtime profiling data, e.g., http://localhost:<debug.port>/debug/pprof/goroutine?debug=2")
|
|
mountOptions.debugPort = cmdMount.Flag.Int("debug.port", 6061, "http port for debugging")
|
|
mountOptions.debugFuse = cmdMount.Flag.Bool("debug.fuse", false, "log raw FUSE protocol requests and responses")
|
|
mountOptions.localSocket = cmdMount.Flag.String("localSocket", "", "default to /tmp/seaweedfs-mount-<mount_dir_hash>.sock")
|
|
mountOptions.disableXAttr = cmdMount.Flag.Bool("disableXAttr", false, "disable xattr")
|
|
mountOptions.windowsUid = cmdMount.Flag.Int("windows.uid", 0, "windows only: uid recorded on entries this mount creates, which other clients read")
|
|
mountOptions.windowsGid = cmdMount.Flag.Int("windows.gid", 0, "windows only: gid recorded on entries this mount creates, which other clients read")
|
|
mountOptions.hasAutofs = cmdMount.Flag.Bool("autofs", false, "ignore autofs mounted on the same mountpoint (useful when systemd.automount and autofs is used)")
|
|
mountOptions.fuseCommandPid = 0
|
|
|
|
// Periodic metadata flush to protect against orphan chunk cleanup
|
|
mountOptions.metadataFlushSeconds = cmdMount.Flag.Int("metadataFlushSeconds", 120, "periodically flush file metadata to filer in seconds (0 to disable). This protects chunks from being purged by volume.fsck for long-running writes")
|
|
|
|
// Peer chunk sharing flags.
|
|
mountOptions.peerEnabled = cmdMount.Flag.Bool("peer.enable", false, "opt in to peer chunk sharing — mount serves its chunk cache to other mounts and fetches from peers instead of volume servers when available")
|
|
mountOptions.peerListen = cmdMount.Flag.String("peer.listen", ":18080", "bind address for peer gRPC (directory RPCs + FetchChunk streaming)")
|
|
mountOptions.peerAdvertise = cmdMount.Flag.String("peer.advertise", "", "externally-reachable host:port other mounts use to reach this one (defaults to autodetected host + -peer.listen port)")
|
|
mountOptions.peerDataCenter = cmdMount.Flag.String("peer.dataCenter", "", "optional data-center label advertised to peers; used with -peer.rack for two-level locality ranking")
|
|
mountOptions.peerRack = cmdMount.Flag.String("peer.rack", "", "optional rack label advertised to peers")
|
|
|
|
mountOptions.dirIdleEvictSec = cmdMount.Flag.Int("dirIdleEvictSec", 600, "seconds to evict idle cached directories (0 to disable)")
|
|
|
|
mountCpuProfile = cmdMount.Flag.String("cpuprofile", "", "cpu profile output file")
|
|
mountMemProfile = cmdMount.Flag.String("memprofile", "", "memory profile output file")
|
|
mountReadRetryTime = cmdMount.Flag.Duration("readRetryTime", 6*time.Second, "maximum read retry wait time")
|
|
|
|
// Distributed lock for cross-mount write coordination
|
|
mountOptions.distributedLock = cmdMount.Flag.Bool("dlm", false, "coordinate writes across mounts (only one mount writes a file at a time) and honor POSIX advisory locks (flock/fcntl) across mounts by routing them to the owner filer")
|
|
|
|
// POSIX compliance options
|
|
mountOptions.posixDirNlink = cmdMount.Flag.Bool("posix.dirNLink", false, "report POSIX-compliant directory nlink (2 + subdirectory count); costs one directory listing per stat")
|
|
|
|
mountOptions.accessReport = cmdMount.Flag.Bool("accessReport", false, "report opens of a file for reading to the filer with TouchAccessTime, including opens answered from the metadata cache, at most once per -accessReport.interval for each file unless it was modified since its last report. Has no effect unless the filer implements TouchAccessTime")
|
|
mountOptions.accessReportInterval = cmdMount.Flag.Duration("accessReport.interval", time.Hour, "with -accessReport, the least time between two reports of the same unmodified file; 0 reports every open")
|
|
|
|
// FUSE performance options
|
|
mountOptions.writebackCache = cmdMount.Flag.Bool("writebackCache", false, "enable FUSE writeback cache for improved write performance (at risk of data loss on crash)")
|
|
mountOptions.asyncDio = cmdMount.Flag.Bool("asyncDio", false, "enable async direct I/O for better concurrency")
|
|
mountOptions.cacheSymlink = cmdMount.Flag.Bool("cacheSymlink", false, "enable symlink caching to reduce metadata lookups")
|
|
mountOptions.fuseMaxBackground = cmdMount.Flag.Int("fuse.maxBackground", 128, "FUSE max_background: maximum in-flight asynchronous requests the kernel will queue. Heavy upload workloads may benefit from higher values (e.g. 2048). Equivalent to writing /sys/fs/fuse/connections/<id>/max_background. If -fuse.congestionThreshold is 0, the kernel derives it as 3/4 of this value.")
|
|
mountOptions.fuseCongestionThreshold = cmdMount.Flag.Int("fuse.congestionThreshold", 0, "FUSE congestion_threshold: in-flight async request count at which the kernel marks the FUSE bdi as congested and throttles new submissions. 0 means use the default (3/4 of -fuse.maxBackground). Equivalent to writing /sys/fs/fuse/connections/<id>/congestion_threshold. The kernel silently clamps this to -fuse.maxBackground when set higher.")
|
|
|
|
// macOS-specific FUSE options
|
|
mountOptions.novncache = cmdMount.Flag.Bool("sys.novncache", false, "(macOS only) disable vnode name caching to avoid stale data")
|
|
}
|
|
|
|
var cmdMount = &Command{
|
|
UsageLine: "mount -filer=localhost:8888 -dir=/some/dir",
|
|
Short: "mount weed filer to a directory as file system in userspace(FUSE)",
|
|
Long: `mount weed filer to userspace.
|
|
|
|
Pre-requisites:
|
|
1) have SeaweedFS master and volume servers running
|
|
2) have a "weed filer" running
|
|
These 2 requirements can be achieved with one command "weed server -filer=true"
|
|
|
|
This uses github.com/seaweedfs/fuse, which enables writing FUSE file systems on
|
|
Linux, and OS X.
|
|
|
|
On OS X, it requires OSXFUSE (https://osxfuse.github.io/).
|
|
|
|
On Windows, it requires WinFsp (https://winfsp.dev/). The mount point can be
|
|
a drive letter (-dir=S:), a directory that does not exist yet, or a network
|
|
path (-dir=\\seaweedfs\share). A drive letter belongs to the logon session
|
|
that created it, so one mounted by a service is usually invisible to users
|
|
at the desktop; the network path form is reachable from every session, and
|
|
each user can map their own drive letter to it.
|
|
|
|
Where the platform labels the disk, in Finder and in Explorer, the mounted
|
|
path names it: -filer.path="/Image Disk" shows up as "Image Disk". Mounting
|
|
the whole tree takes the name from the mount point instead, so
|
|
-dir=\\seaweedfs\Images labels the disk "Images" while still mounting
|
|
everything, and only a bare drive letter falls back to the filer address.
|
|
-volumeName overrides whatever name either of those would otherwise give it,
|
|
including what Linux and FreeBSD show for the mount in "mount" and "df".
|
|
|
|
`,
|
|
}
|