mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-20 13:30:46 +02:00
mount
Chris Lu
+96
@@ -16,6 +16,102 @@ With "weed mount", the files can be operated as a local file. The following oper
|
||||
* advisory file locking (`flock(2)` and POSIX `fcntl(2)` byte-range locks)
|
||||
* extended attributes (xattr)
|
||||
|
||||
### Distributed Lock (cross-mount write coordination)
|
||||
|
||||
`weed mount` can coordinate writers across different mount instances using
|
||||
the filer's distributed lock manager (DLM). When enabled, opening a file
|
||||
for writing acquires a cluster-wide lock on the file's filer path, so only
|
||||
one mount can write a given file at a time. Other mounts that open the same
|
||||
file for writing block until the holder releases it.
|
||||
|
||||
#### Enabling
|
||||
|
||||
Pass `-dlm` when starting `weed mount`:
|
||||
|
||||
```bash
|
||||
weed mount -filer=localhost:8888 -dir=/mnt/seaweedfs -dlm
|
||||
```
|
||||
|
||||
DLM is off by default. It is automatically disabled if `-writebackCache` is
|
||||
also set, since writeback implies single-writer semantics and the extra
|
||||
coordination would only add latency. When DLM is disabled, writers across
|
||||
mounts are unordered — last writer wins on flush.
|
||||
|
||||
#### Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant A as App on Mount A
|
||||
participant MA as weed mount A<br/>(owner mount-1)
|
||||
participant F as Filer DLM
|
||||
participant MB as weed mount B<br/>(owner mount-2)
|
||||
participant B as App on Mount B
|
||||
|
||||
A->>MA: open("/data/file", O_WRONLY)
|
||||
MA->>F: Lock("/data/file", owner=mount-1, ttl=7s)
|
||||
F-->>MA: granted
|
||||
Note over MA,F: Heartbeat renews lock every ~7s
|
||||
MA-->>A: fd
|
||||
|
||||
B->>MB: open("/data/file", O_WRONLY)
|
||||
MB->>F: Lock("/data/file", owner=mount-2, ttl=7s)
|
||||
F-->>MB: blocked (held by mount-1)
|
||||
|
||||
A->>MA: write / close
|
||||
MA->>F: flush to filer, then Unlock
|
||||
F-->>MB: granted (mount-1 released)
|
||||
MB-->>B: fd
|
||||
B->>MB: write / close
|
||||
MB->>F: flush to filer, then Unlock
|
||||
```
|
||||
|
||||
If mount A crashes without closing, the filer drops the lock after the
|
||||
7-second TTL and mount B's pending acquire unblocks automatically.
|
||||
|
||||
#### Semantics
|
||||
|
||||
* **Scope.** The lock key is the file's filer path (not the local inode,
|
||||
which is per-mount). Two mounts opening the same path for writing contend
|
||||
on the same key.
|
||||
* **Acquisition.** The lock is taken when a handle is opened with any write
|
||||
flag (`O_WRONLY`, `O_RDWR`, `O_APPEND`, `O_CREAT`, `O_TRUNC`). Read-only
|
||||
opens are not locked. Acquisition is blocking: a second mount's
|
||||
`open(2)`-for-write blocks until the first mount closes the file or the
|
||||
lock TTL expires.
|
||||
* **Release.** The lock is held for the full lifetime of the file handle
|
||||
and released on close. For handles running the writeback async flush
|
||||
path, the lock is released after the background flush completes — so a
|
||||
reader on another mount after a successful close always sees the flushed
|
||||
data.
|
||||
* **Auto-renewal.** The lock is renewed on a 7-second TTL heartbeat while
|
||||
the handle is open. If a mount crashes, the lock frees on TTL expiry.
|
||||
* **Rename and unlink.** `rename(2)` acquires DLM locks on both the source
|
||||
and destination paths (sorted to avoid A→B / B→A deadlocks) so no other
|
||||
mount can open either path for writing during the rename. `unlink(2)` of
|
||||
a file that is currently open on this mount coordinates with the held
|
||||
handle's lock.
|
||||
* **Owner identifier.** Each mount is tagged with `mount-<signature>`,
|
||||
which appears in filer lock logs and in `weed shell`'s lock commands for
|
||||
debugging.
|
||||
|
||||
#### Requirements and constraints
|
||||
|
||||
* At least one filer address (`-filer=`) must be configured; DLM is a
|
||||
filer-hosted service.
|
||||
* DLM is disabled when `-writebackCache` is set (single-writer mode).
|
||||
* DLM coordinates writes only. Concurrent readers on different mounts are
|
||||
never blocked, and they still see flushed data via the filer's metadata.
|
||||
* DLM does not replace POSIX `fcntl`/`flock` advisory locks; those remain
|
||||
available and are orthogonal (see the next section).
|
||||
|
||||
#### When to use it
|
||||
|
||||
Enable `-dlm` when multiple `weed mount` instances may write to the same
|
||||
files — e.g. a shared build cache mounted on several CI workers, or an
|
||||
application that runs on multiple nodes and writes to the same path. Leave
|
||||
it off for single-mount deployments or when each mount writes into its own
|
||||
subtree, since the extra round-trips add latency on every open-for-write.
|
||||
|
||||
### Advisory File Locking
|
||||
|
||||
SeaweedFS supports advisory file locking over FUSE:
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
# POSIX Compliance for `weed mount`
|
||||
|
||||
SeaweedFS runs the [pjdfstest](https://github.com/pjd/pjdfstest) POSIX
|
||||
compliance suite against `weed mount` on every change to `weed/mount/**`,
|
||||
`weed/filer/**`, or the test harness itself. The job is defined in
|
||||
[`.github/workflows/pjdfstest.yml`](https://github.com/seaweedfs/seaweedfs/blob/master/.github/workflows/pjdfstest.yml)
|
||||
and the harness lives under [`test/pjdfstest/`](https://github.com/seaweedfs/seaweedfs/tree/master/test/pjdfstest).
|
||||
|
||||
## How it runs
|
||||
|
||||
1. A SeaweedFS e2e image (master + volume + filer + mount) is built and
|
||||
started via `test/pjdfstest/docker-compose.yml`.
|
||||
2. `test/pjdfstest/run.sh` mounts the filesystem with `weed mount
|
||||
-allowOthers=true` and clones pjdfstest pinned to an immutable upstream
|
||||
commit for reproducibility.
|
||||
3. `prove -rv` runs every `tests/**/*.t` except the entries listed in
|
||||
[`known_failures.txt`](https://github.com/seaweedfs/seaweedfs/blob/master/test/pjdfstest/known_failures.txt).
|
||||
Any failure outside that allow-list fails CI.
|
||||
|
||||
## Tests passing
|
||||
|
||||
All pjdfstest syscall groups pass on `weed mount`, except for the three
|
||||
test files called out below. The groups exercised by the suite are:
|
||||
|
||||
| Group | Coverage |
|
||||
|-------|----------|
|
||||
| `tests/chflags/` | BSD file flags (skipped on Linux by the suite itself) |
|
||||
| `tests/chmod/` | `chmod(2)` permission, errno, and setuid/setgid behavior |
|
||||
| `tests/chown/` | `chown(2)` ownership, errno, and clearing of set\*id bits |
|
||||
| `tests/link/` | `link(2)` creation, permission, and errno cases (except `00.t`) |
|
||||
| `tests/mkdir/` | `mkdir(2)` creation and errno paths |
|
||||
| `tests/mkfifo/` | `mkfifo(2)` FIFO creation |
|
||||
| `tests/mknod/` | `mknod(2)` regular/FIFO node creation |
|
||||
| `tests/open/` | `open(2)` flags, permissions, `O_CREAT`, `O_TRUNC`, errno |
|
||||
| `tests/rename/` | `rename(2)` same/cross-directory, errno cases (except `21.t`) |
|
||||
| `tests/rmdir/` | `rmdir(2)` empty-directory and errno cases |
|
||||
| `tests/symlink/` | `symlink(2)` creation, traversal, errno cases |
|
||||
| `tests/truncate/` | `truncate(2)` resize, permission, and errno cases |
|
||||
| `tests/unlink/` | `unlink(2)` removal and errno cases (except `00.t`) |
|
||||
| `tests/utimensat/` | `utimensat(2)` atime/mtime/ctime semantics, errno paths |
|
||||
| `tests/granular/` | Granular timestamp update checks |
|
||||
|
||||
## Tests not passing
|
||||
|
||||
The following tests are listed in
|
||||
[`test/pjdfstest/known_failures.txt`](https://github.com/seaweedfs/seaweedfs/blob/master/test/pjdfstest/known_failures.txt)
|
||||
and are skipped in CI. They are deterministic and tracked as known issues:
|
||||
|
||||
| Test | Category | Reason |
|
||||
|------|----------|--------|
|
||||
| `tests/rename/21.t` | rename | Cross-directory rename of a subdirectory with restricted permissions triggers cascading failures in the rest of the test file. |
|
||||
| `tests/link/00.t` | link | `nlink` assertion mismatch (e.g. expected `nlink=2`, got `nlink=3`) after hard link creation. Filer-side hard-link counter issue. |
|
||||
| `tests/unlink/00.t` | unlink | Same filer-side hard-link counter issue surfaced on the removal path. |
|
||||
|
||||
The two `nlink` failures are filer-side hard-link counter bugs rather than
|
||||
FUSE mount bugs, and are deterministic under the current meta-cache load
|
||||
ordering. The rename failure is a permission edge case on cross-directory
|
||||
rename. Contributions that fix any of these are welcome — once a test is
|
||||
green, remove it from `known_failures.txt` so CI enforces it going forward.
|
||||
|
||||
## Running locally
|
||||
|
||||
```sh
|
||||
# Full suite against a local weed binary (requires sudo, fusermount3,
|
||||
# autoconf, make, perl TAP::Harness::Archive).
|
||||
test/pjdfstest/run.sh
|
||||
|
||||
# Subset, e.g. only chmod tests
|
||||
PJDFSTEST_TESTS=tests/chmod test/pjdfstest/run.sh
|
||||
|
||||
# Containerized (matches CI exactly)
|
||||
docker compose -f test/pjdfstest/docker-compose.yml up --wait
|
||||
docker compose -f test/pjdfstest/docker-compose.yml exec mount /run.sh
|
||||
```
|
||||
|
||||
See also: [FUSE Mount](FUSE-Mount).
|
||||
+1
@@ -58,6 +58,7 @@
|
||||
### [[FUSE Mount]]
|
||||
* [[FIO benchmark]]
|
||||
* [[fstab and systemd mount]]
|
||||
* [[POSIX Compliance]]
|
||||
|
||||
### [[WebDAV]]
|
||||
|
||||
|
||||
Reference in New Issue
Block a user