diff --git a/Distributed-POSIX-Locks.md b/Distributed-POSIX-Locks.md new file mode 100644 index 0000000..d5efb99 --- /dev/null +++ b/Distributed-POSIX-Locks.md @@ -0,0 +1,243 @@ +# Distributed POSIX Locks + +`weed mount` supports cross-mount POSIX advisory locks — `flock(2)` and +`fcntl(F_SETLK/F_SETLKW)` — so a lock taken on one mount is honored by every +other mount of the same cluster. The feature is opt-in: pass `-dlm` to `weed +mount`. Without it, locks remain per-mount only (the historical behavior). + +This builds directly on [[Filer Operation Serialization]]: the same route-by-key +layer is reused, with a different authority (an in-memory POSIX lock table on +the owner filer) and a different RPC. + +## Why route, not replicate + +POSIX advisory locks are transient coordination, not durable data. Replicating +them through the metadata log would add write churn to every advisory lock +operation; failover would still race the application's expectations. + +The chosen shape — owner filer per inode + in-memory authority + client-side +polling for blocking acquires + session leases for dead-client cleanup — is +the established pattern for shared-store advisory locking. SeaweedFS already +has the routing layer (the lock ring built for the DLM and reused by +ObjectTransaction), so adding POSIX semantics on top is a small addition. + +## At a glance + +``` + app on mount A app on mount B + │ │ + │ flock(fd, LOCK_EX) │ flock(fd, LOCK_EX) + ▼ ▼ + weed mount A ─── PosixLock RPC ───► filer X (owner of this inode) + │ + ▼ posixlock.Manager + in-memory Set per inode + ▲ + weed mount B ─── PosixLock RPC ───► filer Y → forward (is_moved) → filer X +``` + +Every mount picks the filer it talks to (the `-filer=` argument). That filer +checks the lock ring, and if it is not the owner of this key, forwards the +RPC one hop to the owner. The owner's in-memory `posixlock.Manager` is the +single source of truth. + +Blocking acquires (`F_SETLKW`) are **client-side polling**: the mount +re-sends the non-blocking try with bounded backoff until it succeeds or the +syscall is cancelled. There is no server-side wait queue. + +## The key + +The mount converts a FUSE inode to a *cluster-stable* lock identity in +[`weed/mount/weedfs_posix_lock_routed.go: posixLockKeyForInode`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/mount/weedfs_posix_lock_routed.go): + +| Entry kind | Lock key | +|---|---| +| Regular file | `"s3.fuse.lock:" + path` | +| Hardlinked file | `"s3.fuse.lock:hl:" + hex(HardLinkId)` | + +POSIX locks are inode-scoped, not name-scoped. Using the `HardLinkId` for +linked files makes every name for the same inode share one lock table, which +matches what local POSIX gives you and means rename does not move locks. + +The FUSE NodeId is mount-local (`AsInode = hash(path) + time`), so it cannot +be used cross-mount. + +## Authority — `posixlock.Manager` + +The owner filer keeps the lock state in +[`weed/filer/posixlock/manager.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/filer/posixlock/manager.go): + +* `Set` — a per-inode collection of byte-range `Range`s with `Type` + (Read/Write), `Sid` (session id, unique per mount), `Owner` (the + application's lock owner value), and `IsFlock` (flock and fcntl live in + separate namespaces per POSIX). +* `TryLock` / `Unlock` — non-blocking acquire and release. +* `GetLk` — query without acquiring. +* `ReleasePosixOwner` / `ReleaseFlockOwner` — drop all of an owner's locks on + a key (close-on-fd semantics). +* `Reassert` — rebuild the lock set from a list a holder sends after an + ownership change or restart (used by KEEP_ALIVE). +* `Renew` / `ReapExpired` — session-lease bookkeeping. + +There is exactly one Manager per filer. Lock state is never written to disk; +it survives only as long as the owner filer keeps running, and the +re-assertion mechanism described below rebuilds it when ownership changes. + +## The RPC + +Defined in [`weed/pb/filer.proto`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/pb/filer.proto) +as `PosixLockRequest` / `PosixLockResponse`. The op enum: + +| Op | What it does | +|---|---| +| `TRY_LOCK` | Non-blocking acquire of one range | +| `UNLOCK` | Release one range | +| `GET_LK` | Query — returns the first conflicting range, if any | +| `RELEASE_POSIX_OWNER` | Release all fcntl ranges owned by `(Sid, Owner)` on this key | +| `RELEASE_FLOCK_OWNER` | Release all flock ranges owned by `(Sid, Owner)` on this key | +| `KEEP_ALIVE` | Renew the session lease; if `locks` is non-empty, re-assert held locks | + +A non-blocking grant returns `granted = true`. A conflict returns +`has_conflict = true` and the offending range as `conflict`. The handler +lives in +[`weed/server/filer_grpc_server_posix_lock.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/server/filer_grpc_server_posix_lock.go). + +## Mount-side flow + +Routed POSIX locking lives in +[`weed/mount/weedfs_posix_lock_routed.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/mount/weedfs_posix_lock_routed.go). +The mount runs with `-dlm`, which sets `wfs.lockClient`; the FUSE handlers +in [`weed/mount/weedfs_file_lock.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/mount/weedfs_file_lock.go) +route through the new path when `wfs.crossMountLocks()` is true. + +* **Session id (`Sid`)** — random 64-bit value per mount, namespaces lock + owners so the same FUSE `Owner` value on two mounts never aliases. +* **`SetLk` (non-blocking)** — one `TRY_LOCK` RPC; map `EWOULDBLOCK` + to/from `granted=false`. +* **`SetLkw` (blocking)** — `posixPollAcquire` loops `TRY_LOCK` with + exponential backoff bounded to `posixLockMaxBackoff = 200ms`; the syscall + cancellation (FUSE INT) translates to `EINTR`. +* **`GetLk`** — single `GET_LK` RPC. +* **`flush` / `release`** — POSIX requires that closing any fd to an inode + drops the calling process's fcntl locks on that inode. The mount tracks + `posixLockHint` (a per-inode set of owners we have taken locks for) so it + can fire `RELEASE_POSIX_OWNER` / `RELEASE_FLOCK_OWNER` on close without an + RPC on every close to a file we never locked. +* **Keepalive** — `loopRenewPosixLeases` (`posixKeepaliveInterval = 5s`) + sends KEEP_ALIVE per held key. The payload carries the held locks, so a + filer that just took over ownership (or just restarted) gets the holder's + state pushed to it — see "Re-assertion" below. + +## Sessions, leases, reaping + +Every mount has a 64-bit `Sid`. Every lock the mount takes carries that +`Sid`. The owner filer remembers, per `Sid`, the last time it saw a +KEEP_ALIVE. + +`startPosixLockSweeper` ([`weed/server/filer_grpc_server_posix_lock.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/server/filer_grpc_server_posix_lock.go)) +runs on every filer: + +* `posixLockSessionTTL = 15s` — sessions silent longer than this are reaped. +* `posixLockSweepInterval = 5s` — how often each filer checks. + +When a session is reaped, all of its locks across every key on this filer +are released. This is how a `kill -9`'d mount stops blocking other mounts — +nothing else does, because the kernel cannot tell the cluster that the FD +holding a flock just went away. + +Sessions that never call KEEP_ALIVE are never tracked (no resource cost), +so the sweeper is inert on a cluster without `-dlm` mounts. + +## Ring changes — re-assertion + cooling + warm-up + +The cooling-probe and warm-up machinery described in [[Filer Operation +Serialization]] applies here directly. The POSIX lock layer adds one piece +on top: + +* **Re-assertion via KEEP_ALIVE.** When a mount's keepalive fires, it sends + every held lock on that key in the request payload, not just a bare + renew. The owner's `posixlock.Manager.Reassert` rebuilds the lock set + from that payload and reports any range it could not reassert (a real + loss of lock to a different session). After at most one keepalive + interval following a ring change, every new owner has been told about + every lock its keys carry — and the warm-up window (10s) is sized to + cover that re-assertion round trip even under load. + +The combined picture across an ownership change: + +1. Master broadcasts the new filer set; every filer applies the new + snapshot to its `LockRing` (with the old snapshot retained for + `LockRing.snapshotInterval`). +2. New owner sees PosixLock RPCs for keys it now owns. Its in-memory + `posixlock.Manager` has no entries for those keys yet. +3. For each request, the new owner asks the prior owner (via + `PriorOwner(key)`) whether it sees a conflict — bounded by + `posixCoolingProbeTimeout = 2s`. The prior owner replies from its + in-memory state with `cooling_probe=true`, so the answer is local + and definitive. +4. Within `posixKeepaliveInterval = 5s`, every mount holding a lock on + one of the migrated keys re-asserts via KEEP_ALIVE. The new owner's + state for that key is now correct. +5. The cooling-snapshot ages out (typically `snapshotInterval` after the + ring update). After that, the new owner trusts its local state + unconditionally. + +A filer that just *started* is in its own warm-up window +(`posixLockWarmup = 10s`); during that window it fail-closes on +acquire requests whose "no conflict" answer it cannot verify yet, so +restarts cannot create double grants either. + +## Platform notes + +POSIX advisory locks are only forwarded to the FUSE server on Linux. The +macFUSE kernel module handles `flock` *in the kernel, per mount*, and does +not forward `SETLK` opcodes to the userspace filesystem at all — so two +`weed mount` instances on the same macOS machine cannot coordinate flocks +even with `-dlm`. This is a macFUSE behavior, not a SeaweedFS one. The +routed lock path itself works the same on macOS; there is just nothing for +it to handle. + +The integration test +[`test/fuse_dlm/posix_lock_ring_test.go`](https://github.com/seaweedfs/seaweedfs/blob/master/test/fuse_dlm/posix_lock_ring_test.go) +skips on non-Linux for this reason. + +## Configuration + +To enable cross-mount POSIX locks: + +``` +weed mount -dir=/mnt/sw -filer=filer1:8888 -dlm +``` + +`-dlm` enables the routed POSIX locks (and the whole-file write lock that +predates this work; see [[FUSE Mount]]). It is opt-in because: + +* `flock`/`fcntl` calls now make an RPC instead of touching a process-local + table — meaningful for applications that lock-on-every-write. +* Cluster operators that do not need cross-mount advisory locks are not + affected; the default keeps the per-mount table. + +No filer-side flag is needed. Every filer registers itself in the lock +ring as part of joining the cluster, and the sweeper is inert until a +`-dlm` mount calls KEEP_ALIVE. + +## What is not done by this layer + +* **Mandatory locks** (Linux `chmod g+s,o-x` mandatory mode) — advisory + only, as POSIX recommends. +* **Process-tree inheritance of fcntl owners** — handled by the kernel and + the application, not the filer. +* **Replication of lock state** — locks are kept in memory and rebuilt by + re-assertion. A filer crash with a `-dlm` mount holding locks on its + keys means those keys are unprotected for the cooling window plus the + re-assertion round trip; the design accepts that as the cost of keeping + the lock path off the metadata log. + +## See also + +* [[Filer Operation Serialization]] — the routing, per-path lock, and + ring-change machinery this page builds on. +* [[FUSE Mount]] +* [[POSIX Compliance]] +* [[S3 Object Lock and Retention]] — the unrelated S3 mechanism (object + metadata, not advisory locks). diff --git a/Filer-Operation-Serialization.md b/Filer-Operation-Serialization.md new file mode 100644 index 0000000..76ff8bc --- /dev/null +++ b/Filer-Operation-Serialization.md @@ -0,0 +1,212 @@ +# Filer Operation Serialization + +How SeaweedFS serializes multi-step operations across a filer cluster so that +read-modify-write sequences on the same key are atomic without a heavyweight +distributed lock held across RPCs. + +The architecture is a small, layered design: + +``` + caller (S3 gateway, mount, …) + │ + ▼ route-by-key + any filer in the cluster + │ + ▼ ring.GetPrimary(route_key) + owner filer ← single serialization point per key + │ + ▼ entryLockTable.AcquireLock(lock_key) + apply condition + mutations atomically +``` + +Everything that needs cross-filer atomicity — S3 conditional writes, version +pointer recomputes, multi-entry object transactions, [[Distributed POSIX Locks]] +— rides on these same three layers. + +## Why not just a distributed lock? + +The older mechanism (the `DistributedLock` RPC, still used for whole-file +write locking) hands the caller a lease token; the caller holds it across one +or more RPCs, then releases it. That works, but every read-modify-write step +costs an extra round trip, and a slow or crashed caller can leave a lease that +must be timed out. + +The serialization architecture inverts this: the caller sends *the whole +operation* (a precondition plus an ordered list of mutations) in one RPC to +the filer that owns the key. The lock is held only for that call. If the +caller dies, there is nothing to time out. + +## Layer 1 — Route by key + +Every filer registers with the master and joins a hash ring keyed by its +`ServerAddress` (see [`weed/cluster/lock_manager/lock_ring.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/cluster/lock_manager/lock_ring.go)). +The ring is a consistent hash with virtual nodes (`DefaultVnodeCount`); the +master broadcasts membership changes so every filer converges on the same +ring view. + +A *route key* is a stable string derived from the object the caller wants to +mutate (e.g. the object's path, or `"s3.fuse.lock:" + path` for an inode +lock). `LockRing.GetPrimary(route_key)` returns the filer that owns it. + +The caller does not have to talk to the owner directly. Any filer that +receives the request checks ownership and, if it is not the owner, forwards +the request one hop. Only one hop is allowed: a forwarded request carries +`is_moved=true` and is always applied locally on the receiver. This bounds +the forwarding cost and prevents a loop when two filers temporarily disagree +on the owner (a ring change in flight). + +See `ObjectTransaction` for the canonical example: +[`weed/server/filer_grpc_server.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/server/filer_grpc_server.go). + +``` +if req.RouteKey != "" && !req.IsMoved && fs.filer.Dlm != nil { + if owner := fs.filer.Dlm.LockRing.GetPrimary(req.RouteKey); owner != "" && owner != fs.option.Host { + // forward one hop with IsMoved=true + ... + } +} +``` + +## Layer 2 — Per-path lock on the owner + +Each filer keeps an in-memory `entryLockTable` keyed by `util.FullPath` +([`weed/server/filer_server.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/server/filer_server.go)): + +``` +// entryLockTable serializes mutations to the same entry path on this filer. +// ...the local serialization point for read-modify-write operations that +// replaces the distributed lock for that key. Idle keys are evicted +// automatically, so the table stays bounded. +entryLockTable *util.LockTable[util.FullPath] +``` + +`AcquireLock(name, fullpath, ExclusiveLock)` blocks until any other holder of +the same path releases. `CreateEntry`, `ObjectTransaction`, and the POSIX +lock layer all take it. Because routing always sends a given key's writes to +the same owner filer, this in-memory mutex is sufficient cluster-wide: there +is exactly one place in the system that holds the lock for that key. + +Idle entries are evicted by the `util.LockTable` itself, so the table size +tracks active concurrency, not total entries. + +## Layer 3 — `ObjectTransaction` (composite atomic op) + +`ObjectTransaction` is the canonical atomic primitive. One request describes: + +* `route_key` — used by Layer 1 to forward to the owner; +* `lock_key` — the path the per-path lock is taken on (Layer 2); +* `condition` (optional) — a precondition evaluated against `condition_key` + (or `lock_key`), e.g. "object exists with this ETag", or `If-None-Match: *`; +* `mutations` — an ordered list of CreateEntry / UpdateEntry / DeleteEntry + operations applied in order under the lock. + +If the condition fails, the request returns `FilerError_PRECONDITION_FAILED` +with no mutations applied. If any mutation fails, the response carries the +error and the rest are not applied. The whole sequence runs under one +exclusive hold of the per-path lock, so concurrent writers of the same +object cannot interleave. + +`ObjectTransactionBatch` lets a caller submit several independent +transactions in one round trip; each runs under its own per-path lock, and a +failure in one does not abort the rest (matching S3 multi-object semantics). + +This replaces the old pattern of "take a distributed lock → do RPC A → do +RPC B → release lock" with one RPC that the caller cannot drop mid-sequence. + +Typical callers: + +| Caller | Why a transaction | +|---|---| +| S3 versioned PUT/DELETE | Atomically write the version + flip the `latest` pointer + create/delete a marker | +| S3 conditional writes ([[S3 Conditional Operations]]) | Evaluate `If-Match`/`If-None-Match` against current state under lock | +| S3 Object Lock ([[S3 Object Lock and Retention]]) | Enforce WORM guards against the existing entry as a precondition | +| Lifecycle expirations | Delete an entry only if its metadata still matches the rule that selected it | + +The implementation lives at +[`weed/server/filer_grpc_server.go: ObjectTransaction`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/server/filer_grpc_server.go); +the proto is in [`weed/pb/filer.proto`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/pb/filer.proto) +(messages `ObjectTransactionRequest`, `ObjectTransactionResponse`, +`ObjectMutation`, `WriteCondition`). + +## Ring changes — the hard part + +A filer joining or leaving the ring shifts which filer owns a subset of +keys. For a single key, ownership transitions atomically the moment every +filer applies the new snapshot — but they do not all apply it at the same +instant. There is a brief window in which: + +1. The new owner has the snapshot and starts accepting writes, +2. The old owner has not yet seen the snapshot and still accepts writes, +3. The new owner's in-memory state for that key is empty (it never owned it + before). + +Two mechanisms keep this window safe. + +### Snapshot history + prior-owner cooling probe + +`LockRing` keeps the last few snapshots, not just the current one, with +timestamps ([`weed/cluster/lock_manager/lock_ring.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/cluster/lock_manager/lock_ring.go)). +`PriorOwner(key)` returns the previous snapshot's owner, but only while the +previous snapshot is still inside the cooling-off window (the snapshot +interval). Each snapshot prebuilds its own `HashRing` so prior-owner lookup +is O(1). + +When the new owner gets a non-blocking call that would normally answer +"no conflict, granted," it first checks: is there a prior owner in the +cooling window? If so, it sends a bounded probe to the prior owner asking +"do you hold a conflict for this key?" The probe is marked so the recipient +answers locally without re-forwarding. The probe is deadline-bounded +(`posixCoolingProbeTimeout = 2 * time.Second`) so a slow peer cannot stall +a non-blocking call. + +If the probe says "conflict," the new owner reports the conflict and lets +the caller retry. If the probe times out or errors, the new owner +fail-closes (treats it as a conflict) rather than risk a double grant. + +### Warm-up window on owner (re)start + +When a filer starts (or restarts), it has no in-memory state for any key. +For a short warm-up period it cannot trust "no local conflict" as the truth +even for keys it now owns — a holder from before the restart may still be +in the system. + +Each filer tracks `posixLockReadyAt` (atomic, set by the sweeper after the +first successful sweep). For `posixLockWarmup` (currently 10s) after that +timestamp, the owner defers granting non-blocking acquires whose +conflicts it cannot verify, instead returning the same fail-closed +conflict shape as the cooling probe. Holders re-assert their locks on the +next keepalive (see [[Distributed POSIX Locks]]), so the warm-up window +ends with the owner's state correctly rebuilt. + +This combination — snapshot history + cooling probe + warm-up — is what +lets the cluster admit and evict filers without dropping the +"single-serialization-point-per-key" guarantee that everything else +depends on. + +The cooling logic is in +[`weed/server/filer_grpc_server_posix_lock.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/server/filer_grpc_server_posix_lock.go); +ObjectTransaction reuses the same `LockRing.GetPrimary` / forwarding path +and inherits the snapshot-history protection. + +## What this lets you build + +Because the serialization layer is generic, anything routed by key inherits +the same atomicity guarantee. Two examples shipped in master: + +* **S3 conditional / versioned writes** ([[S3 Conditional Operations]], + [[S3 Object Versioning]], [[S3 Object Lock and Retention]]) — the S3 + gateway sends an `ObjectTransaction` with the conditional headers as the + precondition and the version-pointer recompute as the mutation list. No + distributed lock is held across the multi-entry update. +* **Cross-mount POSIX advisory locks** ([[Distributed POSIX Locks]]) — the + mount calls a dedicated `PosixLock` RPC that uses the same route-by-key, + one-hop forwarding, snapshot history, and warm-up logic; the authority is + an in-memory `posixlock.Manager` on the owner filer rather than the entry + lock table. + +## See also + +* [[Distributed POSIX Locks]] +* [[S3 Conditional Operations]] +* [[S3 Object Versioning]] +* [[Filer Server API]] diff --git a/_Sidebar.md b/_Sidebar.md index af0d365..e43bb83 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -56,11 +56,13 @@ * [[Filer as a Key-Large-Value Store]] * [[Path Specific Configuration]] * [[Filer Change Data Capture]] +* [[Filer Operation Serialization]] ### [[FUSE Mount]] * [[FIO benchmark]] * [[fstab and systemd mount]] * [[POSIX Compliance]] +* [[Distributed POSIX Locks]] * [[P2P reading in weed mount]] ### [[WebDAV]]