diff --git a/FUSE-Mount.md b/FUSE-Mount.md index 7ec0553..7cfb49f 100644 --- a/FUSE-Mount.md +++ b/FUSE-Mount.md @@ -16,6 +16,102 @@ With "weed mount", the files can be operated as a local file. The following oper * advisory file locking (`flock(2)` and POSIX `fcntl(2)` byte-range locks) * extended attributes (xattr) +### Distributed Lock (cross-mount write coordination) + +`weed mount` can coordinate writers across different mount instances using +the filer's distributed lock manager (DLM). When enabled, opening a file +for writing acquires a cluster-wide lock on the file's filer path, so only +one mount can write a given file at a time. Other mounts that open the same +file for writing block until the holder releases it. + +#### Enabling + +Pass `-dlm` when starting `weed mount`: + +```bash +weed mount -filer=localhost:8888 -dir=/mnt/seaweedfs -dlm +``` + +DLM is off by default. It is automatically disabled if `-writebackCache` is +also set, since writeback implies single-writer semantics and the extra +coordination would only add latency. When DLM is disabled, writers across +mounts are unordered — last writer wins on flush. + +#### Flow + +```mermaid +sequenceDiagram + participant A as App on Mount A + participant MA as weed mount A
(owner mount-1) + participant F as Filer DLM + participant MB as weed mount B
(owner mount-2) + participant B as App on Mount B + + A->>MA: open("/data/file", O_WRONLY) + MA->>F: Lock("/data/file", owner=mount-1, ttl=7s) + F-->>MA: granted + Note over MA,F: Heartbeat renews lock every ~7s + MA-->>A: fd + + B->>MB: open("/data/file", O_WRONLY) + MB->>F: Lock("/data/file", owner=mount-2, ttl=7s) + F-->>MB: blocked (held by mount-1) + + A->>MA: write / close + MA->>F: flush to filer, then Unlock + F-->>MB: granted (mount-1 released) + MB-->>B: fd + B->>MB: write / close + MB->>F: flush to filer, then Unlock +``` + +If mount A crashes without closing, the filer drops the lock after the +7-second TTL and mount B's pending acquire unblocks automatically. + +#### Semantics + +* **Scope.** The lock key is the file's filer path (not the local inode, + which is per-mount). Two mounts opening the same path for writing contend + on the same key. +* **Acquisition.** The lock is taken when a handle is opened with any write + flag (`O_WRONLY`, `O_RDWR`, `O_APPEND`, `O_CREAT`, `O_TRUNC`). Read-only + opens are not locked. Acquisition is blocking: a second mount's + `open(2)`-for-write blocks until the first mount closes the file or the + lock TTL expires. +* **Release.** The lock is held for the full lifetime of the file handle + and released on close. For handles running the writeback async flush + path, the lock is released after the background flush completes — so a + reader on another mount after a successful close always sees the flushed + data. +* **Auto-renewal.** The lock is renewed on a 7-second TTL heartbeat while + the handle is open. If a mount crashes, the lock frees on TTL expiry. +* **Rename and unlink.** `rename(2)` acquires DLM locks on both the source + and destination paths (sorted to avoid A→B / B→A deadlocks) so no other + mount can open either path for writing during the rename. `unlink(2)` of + a file that is currently open on this mount coordinates with the held + handle's lock. +* **Owner identifier.** Each mount is tagged with `mount-`, + which appears in filer lock logs and in `weed shell`'s lock commands for + debugging. + +#### Requirements and constraints + +* At least one filer address (`-filer=`) must be configured; DLM is a + filer-hosted service. +* DLM is disabled when `-writebackCache` is set (single-writer mode). +* DLM coordinates writes only. Concurrent readers on different mounts are + never blocked, and they still see flushed data via the filer's metadata. +* DLM does not replace POSIX `fcntl`/`flock` advisory locks; those remain + available and are orthogonal (see the next section). + +#### When to use it + +Enable `-dlm` when multiple `weed mount` instances may write to the same +files — e.g. a shared build cache mounted on several CI workers, or an +application that runs on multiple nodes and writes to the same path. Leave +it off for single-mount deployments or when each mount writes into its own +subtree, since the extra round-trips add latency on every open-for-write. + ### Advisory File Locking SeaweedFS supports advisory file locking over FUSE: diff --git a/POSIX-Compliance.md b/POSIX-Compliance.md new file mode 100644 index 0000000..c6d1b4a --- /dev/null +++ b/POSIX-Compliance.md @@ -0,0 +1,76 @@ +# POSIX Compliance for `weed mount` + +SeaweedFS runs the [pjdfstest](https://github.com/pjd/pjdfstest) POSIX +compliance suite against `weed mount` on every change to `weed/mount/**`, +`weed/filer/**`, or the test harness itself. The job is defined in +[`.github/workflows/pjdfstest.yml`](https://github.com/seaweedfs/seaweedfs/blob/master/.github/workflows/pjdfstest.yml) +and the harness lives under [`test/pjdfstest/`](https://github.com/seaweedfs/seaweedfs/tree/master/test/pjdfstest). + +## How it runs + +1. A SeaweedFS e2e image (master + volume + filer + mount) is built and + started via `test/pjdfstest/docker-compose.yml`. +2. `test/pjdfstest/run.sh` mounts the filesystem with `weed mount + -allowOthers=true` and clones pjdfstest pinned to an immutable upstream + commit for reproducibility. +3. `prove -rv` runs every `tests/**/*.t` except the entries listed in + [`known_failures.txt`](https://github.com/seaweedfs/seaweedfs/blob/master/test/pjdfstest/known_failures.txt). + Any failure outside that allow-list fails CI. + +## Tests passing + +All pjdfstest syscall groups pass on `weed mount`, except for the three +test files called out below. The groups exercised by the suite are: + +| Group | Coverage | +|-------|----------| +| `tests/chflags/` | BSD file flags (skipped on Linux by the suite itself) | +| `tests/chmod/` | `chmod(2)` permission, errno, and setuid/setgid behavior | +| `tests/chown/` | `chown(2)` ownership, errno, and clearing of set\*id bits | +| `tests/link/` | `link(2)` creation, permission, and errno cases (except `00.t`) | +| `tests/mkdir/` | `mkdir(2)` creation and errno paths | +| `tests/mkfifo/` | `mkfifo(2)` FIFO creation | +| `tests/mknod/` | `mknod(2)` regular/FIFO node creation | +| `tests/open/` | `open(2)` flags, permissions, `O_CREAT`, `O_TRUNC`, errno | +| `tests/rename/` | `rename(2)` same/cross-directory, errno cases (except `21.t`) | +| `tests/rmdir/` | `rmdir(2)` empty-directory and errno cases | +| `tests/symlink/` | `symlink(2)` creation, traversal, errno cases | +| `tests/truncate/` | `truncate(2)` resize, permission, and errno cases | +| `tests/unlink/` | `unlink(2)` removal and errno cases (except `00.t`) | +| `tests/utimensat/` | `utimensat(2)` atime/mtime/ctime semantics, errno paths | +| `tests/granular/` | Granular timestamp update checks | + +## Tests not passing + +The following tests are listed in +[`test/pjdfstest/known_failures.txt`](https://github.com/seaweedfs/seaweedfs/blob/master/test/pjdfstest/known_failures.txt) +and are skipped in CI. They are deterministic and tracked as known issues: + +| Test | Category | Reason | +|------|----------|--------| +| `tests/rename/21.t` | rename | Cross-directory rename of a subdirectory with restricted permissions triggers cascading failures in the rest of the test file. | +| `tests/link/00.t` | link | `nlink` assertion mismatch (e.g. expected `nlink=2`, got `nlink=3`) after hard link creation. Filer-side hard-link counter issue. | +| `tests/unlink/00.t` | unlink | Same filer-side hard-link counter issue surfaced on the removal path. | + +The two `nlink` failures are filer-side hard-link counter bugs rather than +FUSE mount bugs, and are deterministic under the current meta-cache load +ordering. The rename failure is a permission edge case on cross-directory +rename. Contributions that fix any of these are welcome — once a test is +green, remove it from `known_failures.txt` so CI enforces it going forward. + +## Running locally + +```sh +# Full suite against a local weed binary (requires sudo, fusermount3, +# autoconf, make, perl TAP::Harness::Archive). +test/pjdfstest/run.sh + +# Subset, e.g. only chmod tests +PJDFSTEST_TESTS=tests/chmod test/pjdfstest/run.sh + +# Containerized (matches CI exactly) +docker compose -f test/pjdfstest/docker-compose.yml up --wait +docker compose -f test/pjdfstest/docker-compose.yml exec mount /run.sh +``` + +See also: [FUSE Mount](FUSE-Mount). diff --git a/_Sidebar.md b/_Sidebar.md index c49bc63..c859c40 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -58,6 +58,7 @@ ### [[FUSE Mount]] * [[FIO benchmark]] * [[fstab and systemd mount]] +* [[POSIX Compliance]] ### [[WebDAV]]