Github-Mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-20 13:30:46 +02:00
Code Issues Packages Projects Releases Wiki Activity

locks

Chris Lu
2026-05-25 21:36:51 -07:00
parent c53a479019
commit 72a4bb705f
3 changed files with 457 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Distributed-POSIX-Locks.md
+243
@@ -0,0 +1,243 @@
# Distributed POSIX Locks
`weed mount` supports cross-mount POSIX advisory locks — `flock(2)` and
`fcntl(F_SETLK/F_SETLKW)` — so a lock taken on one mount is honored by every
other mount of the same cluster. The feature is opt-in: pass `-dlm` to `weed
mount`. Without it, locks remain per-mount only (the historical behavior).
This builds directly on [[Filer Operation Serialization]]: the same route-by-key
layer is reused, with a different authority (an in-memory POSIX lock table on
the owner filer) and a different RPC.
## Why route, not replicate
POSIX advisory locks are transient coordination, not durable data. Replicating
them through the metadata log would add write churn to every advisory lock
operation; failover would still race the application's expectations.
The chosen shape — owner filer per inode + in-memory authority + client-side
polling for blocking acquires + session leases for dead-client cleanup — is
the established pattern for shared-store advisory locking. SeaweedFS already
has the routing layer (the lock ring built for the DLM and reused by
ObjectTransaction), so adding POSIX semantics on top is a small addition.
## At a glance
```
app on mount A app on mount B
│ │
│ flock(fd, LOCK_EX) │ flock(fd, LOCK_EX)
▼ ▼
weed mount A ─── PosixLock RPC ───► filer X (owner of this inode)
│
▼ posixlock.Manager
in-memory Set per inode
▲
weed mount B ─── PosixLock RPC ───► filer Y → forward (is_moved) → filer X
```
Every mount picks the filer it talks to (the `-filer=` argument). That filer
checks the lock ring, and if it is not the owner of this key, forwards the
RPC one hop to the owner. The owner's in-memory `posixlock.Manager` is the
single source of truth.
Blocking acquires (`F_SETLKW`) are **client-side polling**: the mount
re-sends the non-blocking try with bounded backoff until it succeeds or the
syscall is cancelled. There is no server-side wait queue.
## The key
The mount converts a FUSE inode to a *cluster-stable* lock identity in
[`weed/mount/weedfs_posix_lock_routed.go: posixLockKeyForInode`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/mount/weedfs_posix_lock_routed.go):
| Entry kind | Lock key |
|---|---|
| Regular file | `"s3.fuse.lock:" + path` |
| Hardlinked file | `"s3.fuse.lock:hl:" + hex(HardLinkId)` |
POSIX locks are inode-scoped, not name-scoped. Using the `HardLinkId` for
linked files makes every name for the same inode share one lock table, which
matches what local POSIX gives you and means rename does not move locks.
The FUSE NodeId is mount-local (`AsInode = hash(path) + time`), so it cannot
be used cross-mount.
## Authority — `posixlock.Manager`
The owner filer keeps the lock state in
[`weed/filer/posixlock/manager.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/filer/posixlock/manager.go):
* `Set` — a per-inode collection of byte-range `Range`s with `Type`
(Read/Write), `Sid` (session id, unique per mount), `Owner` (the
application's lock owner value), and `IsFlock` (flock and fcntl live in
separate namespaces per POSIX).
* `TryLock` / `Unlock` — non-blocking acquire and release.
* `GetLk` — query without acquiring.
* `ReleasePosixOwner` / `ReleaseFlockOwner` — drop all of an owner's locks on
a key (close-on-fd semantics).
* `Reassert` — rebuild the lock set from a list a holder sends after an
ownership change or restart (used by KEEP_ALIVE).
* `Renew` / `ReapExpired` — session-lease bookkeeping.
There is exactly one Manager per filer. Lock state is never written to disk;
it survives only as long as the owner filer keeps running, and the
re-assertion mechanism described below rebuilds it when ownership changes.
## The RPC
Defined in [`weed/pb/filer.proto`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/pb/filer.proto)
as `PosixLockRequest` / `PosixLockResponse`. The op enum:
| Op | What it does |
|---|---|
| `TRY_LOCK` | Non-blocking acquire of one range |
| `UNLOCK` | Release one range |
| `GET_LK` | Query — returns the first conflicting range, if any |
| `RELEASE_POSIX_OWNER` | Release all fcntl ranges owned by `(Sid, Owner)` on this key |
| `RELEASE_FLOCK_OWNER` | Release all flock ranges owned by `(Sid, Owner)` on this key |
| `KEEP_ALIVE` | Renew the session lease; if `locks` is non-empty, re-assert held locks |
A non-blocking grant returns `granted = true`. A conflict returns
`has_conflict = true` and the offending range as `conflict`. The handler
lives in
[`weed/server/filer_grpc_server_posix_lock.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/server/filer_grpc_server_posix_lock.go).
## Mount-side flow
Routed POSIX locking lives in
[`weed/mount/weedfs_posix_lock_routed.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/mount/weedfs_posix_lock_routed.go).
The mount runs with `-dlm`, which sets `wfs.lockClient`; the FUSE handlers
in [`weed/mount/weedfs_file_lock.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/mount/weedfs_file_lock.go)
route through the new path when `wfs.crossMountLocks()` is true.
* **Session id (`Sid`)** — random 64-bit value per mount, namespaces lock
owners so the same FUSE `Owner` value on two mounts never aliases.
* **`SetLk` (non-blocking)** — one `TRY_LOCK` RPC; map `EWOULDBLOCK`
to/from `granted=false`.
* **`SetLkw` (blocking)** — `posixPollAcquire` loops `TRY_LOCK` with
exponential backoff bounded to `posixLockMaxBackoff = 200ms`; the syscall
cancellation (FUSE INT) translates to `EINTR`.
* **`GetLk`** — single `GET_LK` RPC.
* **`flush` / `release`** — POSIX requires that closing any fd to an inode
drops the calling process's fcntl locks on that inode. The mount tracks
`posixLockHint` (a per-inode set of owners we have taken locks for) so it
can fire `RELEASE_POSIX_OWNER` / `RELEASE_FLOCK_OWNER` on close without an
RPC on every close to a file we never locked.
* **Keepalive** — `loopRenewPosixLeases` (`posixKeepaliveInterval = 5s`)
sends KEEP_ALIVE per held key. The payload carries the held locks, so a
filer that just took over ownership (or just restarted) gets the holder's
state pushed to it — see "Re-assertion" below.
## Sessions, leases, reaping
Every mount has a 64-bit `Sid`. Every lock the mount takes carries that
`Sid`. The owner filer remembers, per `Sid`, the last time it saw a
KEEP_ALIVE.
`startPosixLockSweeper` ([`weed/server/filer_grpc_server_posix_lock.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/server/filer_grpc_server_posix_lock.go))
runs on every filer:
* `posixLockSessionTTL = 15s` — sessions silent longer than this are reaped.
* `posixLockSweepInterval = 5s` — how often each filer checks.
When a session is reaped, all of its locks across every key on this filer
are released. This is how a `kill -9`'d mount stops blocking other mounts —
nothing else does, because the kernel cannot tell the cluster that the FD
holding a flock just went away.
Sessions that never call KEEP_ALIVE are never tracked (no resource cost),
so the sweeper is inert on a cluster without `-dlm` mounts.
## Ring changes — re-assertion + cooling + warm-up
The cooling-probe and warm-up machinery described in [[Filer Operation
Serialization]] applies here directly. The POSIX lock layer adds one piece
on top:
* **Re-assertion via KEEP_ALIVE.** When a mount's keepalive fires, it sends
every held lock on that key in the request payload, not just a bare
renew. The owner's `posixlock.Manager.Reassert` rebuilds the lock set
from that payload and reports any range it could not reassert (a real
loss of lock to a different session). After at most one keepalive
interval following a ring change, every new owner has been told about
every lock its keys carry — and the warm-up window (10s) is sized to
cover that re-assertion round trip even under load.
The combined picture across an ownership change:
1. Master broadcasts the new filer set; every filer applies the new
snapshot to its `LockRing` (with the old snapshot retained for
`LockRing.snapshotInterval`).
2. New owner sees PosixLock RPCs for keys it now owns. Its in-memory
`posixlock.Manager` has no entries for those keys yet.
3. For each request, the new owner asks the prior owner (via
`PriorOwner(key)`) whether it sees a conflict — bounded by
`posixCoolingProbeTimeout = 2s`. The prior owner replies from its
in-memory state with `cooling_probe=true`, so the answer is local
and definitive.
4. Within `posixKeepaliveInterval = 5s`, every mount holding a lock on
one of the migrated keys re-asserts via KEEP_ALIVE. The new owner's
state for that key is now correct.
5. The cooling-snapshot ages out (typically `snapshotInterval` after the
ring update). After that, the new owner trusts its local state
unconditionally.
A filer that just *started* is in its own warm-up window
(`posixLockWarmup = 10s`); during that window it fail-closes on
acquire requests whose "no conflict" answer it cannot verify yet, so
restarts cannot create double grants either.
## Platform notes
POSIX advisory locks are only forwarded to the FUSE server on Linux. The
macFUSE kernel module handles `flock` *in the kernel, per mount*, and does
not forward `SETLK` opcodes to the userspace filesystem at all — so two
`weed mount` instances on the same macOS machine cannot coordinate flocks
even with `-dlm`. This is a macFUSE behavior, not a SeaweedFS one. The
routed lock path itself works the same on macOS; there is just nothing for
it to handle.
The integration test
[`test/fuse_dlm/posix_lock_ring_test.go`](https://github.com/seaweedfs/seaweedfs/blob/master/test/fuse_dlm/posix_lock_ring_test.go)
skips on non-Linux for this reason.
## Configuration
To enable cross-mount POSIX locks:
```
weed mount -dir=/mnt/sw -filer=filer1:8888 -dlm
```
`-dlm` enables the routed POSIX locks (and the whole-file write lock that
predates this work; see [[FUSE Mount]]). It is opt-in because:
* `flock`/`fcntl` calls now make an RPC instead of touching a process-local
table — meaningful for applications that lock-on-every-write.
* Cluster operators that do not need cross-mount advisory locks are not
affected; the default keeps the per-mount table.
No filer-side flag is needed. Every filer registers itself in the lock
ring as part of joining the cluster, and the sweeper is inert until a
`-dlm` mount calls KEEP_ALIVE.
## What is not done by this layer
* **Mandatory locks** (Linux `chmod g+s,o-x` mandatory mode) — advisory
only, as POSIX recommends.
* **Process-tree inheritance of fcntl owners** — handled by the kernel and
the application, not the filer.
* **Replication of lock state** — locks are kept in memory and rebuilt by
re-assertion. A filer crash with a `-dlm` mount holding locks on its
keys means those keys are unprotected for the cooling window plus the
re-assertion round trip; the design accepts that as the cost of keeping
the lock path off the metadata log.
## See also
* [[Filer Operation Serialization]] — the routing, per-path lock, and
ring-change machinery this page builds on.
* [[FUSE Mount]]
* [[POSIX Compliance]]
* [[S3 Object Lock and Retention]] — the unrelated S3 mechanism (object
metadata, not advisory locks).
Filer-Operation-Serialization.md
+212
@@ -0,0 +1,212 @@
# Filer Operation Serialization
How SeaweedFS serializes multi-step operations across a filer cluster so that
read-modify-write sequences on the same key are atomic without a heavyweight
distributed lock held across RPCs.
The architecture is a small, layered design:
```
caller (S3 gateway, mount, …)
│
▼ route-by-key
any filer in the cluster
│
▼ ring.GetPrimary(route_key)
owner filer ← single serialization point per key
│
▼ entryLockTable.AcquireLock(lock_key)
apply condition + mutations atomically
```
Everything that needs cross-filer atomicity — S3 conditional writes, version
pointer recomputes, multi-entry object transactions, [[Distributed POSIX Locks]]
— rides on these same three layers.
## Why not just a distributed lock?
The older mechanism (the `DistributedLock` RPC, still used for whole-file
write locking) hands the caller a lease token; the caller holds it across one
or more RPCs, then releases it. That works, but every read-modify-write step
costs an extra round trip, and a slow or crashed caller can leave a lease that
must be timed out.
The serialization architecture inverts this: the caller sends *the whole
operation* (a precondition plus an ordered list of mutations) in one RPC to
the filer that owns the key. The lock is held only for that call. If the
caller dies, there is nothing to time out.
## Layer 1 — Route by key
Every filer registers with the master and joins a hash ring keyed by its
`ServerAddress` (see [`weed/cluster/lock_manager/lock_ring.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/cluster/lock_manager/lock_ring.go)).
The ring is a consistent hash with virtual nodes (`DefaultVnodeCount`); the
master broadcasts membership changes so every filer converges on the same
ring view.
A *route key* is a stable string derived from the object the caller wants to
mutate (e.g. the object's path, or `"s3.fuse.lock:" + path` for an inode
lock). `LockRing.GetPrimary(route_key)` returns the filer that owns it.
The caller does not have to talk to the owner directly. Any filer that
receives the request checks ownership and, if it is not the owner, forwards
the request one hop. Only one hop is allowed: a forwarded request carries
`is_moved=true` and is always applied locally on the receiver. This bounds
the forwarding cost and prevents a loop when two filers temporarily disagree
on the owner (a ring change in flight).
See `ObjectTransaction` for the canonical example:
[`weed/server/filer_grpc_server.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/server/filer_grpc_server.go).
```
if req.RouteKey != "" && !req.IsMoved && fs.filer.Dlm != nil {
if owner := fs.filer.Dlm.LockRing.GetPrimary(req.RouteKey); owner != "" && owner != fs.option.Host {
// forward one hop with IsMoved=true
...
}
}
```
## Layer 2 — Per-path lock on the owner
Each filer keeps an in-memory `entryLockTable` keyed by `util.FullPath`
([`weed/server/filer_server.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/server/filer_server.go)):
```
// entryLockTable serializes mutations to the same entry path on this filer.
// ...the local serialization point for read-modify-write operations that
// replaces the distributed lock for that key. Idle keys are evicted
// automatically, so the table stays bounded.
entryLockTable *util.LockTable[util.FullPath]
```
`AcquireLock(name, fullpath, ExclusiveLock)` blocks until any other holder of
the same path releases. `CreateEntry`, `ObjectTransaction`, and the POSIX
lock layer all take it. Because routing always sends a given key's writes to
the same owner filer, this in-memory mutex is sufficient cluster-wide: there
is exactly one place in the system that holds the lock for that key.
Idle entries are evicted by the `util.LockTable` itself, so the table size
tracks active concurrency, not total entries.
## Layer 3 — `ObjectTransaction` (composite atomic op)
`ObjectTransaction` is the canonical atomic primitive. One request describes:
* `route_key` — used by Layer 1 to forward to the owner;
* `lock_key` — the path the per-path lock is taken on (Layer 2);
* `condition` (optional) — a precondition evaluated against `condition_key`
(or `lock_key`), e.g. "object exists with this ETag", or `If-None-Match: *`;
* `mutations` — an ordered list of CreateEntry / UpdateEntry / DeleteEntry
operations applied in order under the lock.
If the condition fails, the request returns `FilerError_PRECONDITION_FAILED`
with no mutations applied. If any mutation fails, the response carries the
error and the rest are not applied. The whole sequence runs under one
exclusive hold of the per-path lock, so concurrent writers of the same
object cannot interleave.
`ObjectTransactionBatch` lets a caller submit several independent
transactions in one round trip; each runs under its own per-path lock, and a
failure in one does not abort the rest (matching S3 multi-object semantics).
This replaces the old pattern of "take a distributed lock → do RPC A → do
RPC B → release lock" with one RPC that the caller cannot drop mid-sequence.
Typical callers:
| Caller | Why a transaction |
|---|---|
| S3 versioned PUT/DELETE | Atomically write the version + flip the `latest` pointer + create/delete a marker |
| S3 conditional writes ([[S3 Conditional Operations]]) | Evaluate `If-Match`/`If-None-Match` against current state under lock |
| S3 Object Lock ([[S3 Object Lock and Retention]]) | Enforce WORM guards against the existing entry as a precondition |
| Lifecycle expirations | Delete an entry only if its metadata still matches the rule that selected it |
The implementation lives at
[`weed/server/filer_grpc_server.go: ObjectTransaction`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/server/filer_grpc_server.go);
the proto is in [`weed/pb/filer.proto`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/pb/filer.proto)
(messages `ObjectTransactionRequest`, `ObjectTransactionResponse`,
`ObjectMutation`, `WriteCondition`).
## Ring changes — the hard part
A filer joining or leaving the ring shifts which filer owns a subset of
keys. For a single key, ownership transitions atomically the moment every
filer applies the new snapshot — but they do not all apply it at the same
instant. There is a brief window in which:
1. The new owner has the snapshot and starts accepting writes,
2. The old owner has not yet seen the snapshot and still accepts writes,
3. The new owner's in-memory state for that key is empty (it never owned it
before).
Two mechanisms keep this window safe.
### Snapshot history + prior-owner cooling probe
`LockRing` keeps the last few snapshots, not just the current one, with
timestamps ([`weed/cluster/lock_manager/lock_ring.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/cluster/lock_manager/lock_ring.go)).
`PriorOwner(key)` returns the previous snapshot's owner, but only while the
previous snapshot is still inside the cooling-off window (the snapshot
interval). Each snapshot prebuilds its own `HashRing` so prior-owner lookup
is O(1).
When the new owner gets a non-blocking call that would normally answer
"no conflict, granted," it first checks: is there a prior owner in the
cooling window? If so, it sends a bounded probe to the prior owner asking
"do you hold a conflict for this key?" The probe is marked so the recipient
answers locally without re-forwarding. The probe is deadline-bounded
(`posixCoolingProbeTimeout = 2 * time.Second`) so a slow peer cannot stall
a non-blocking call.
If the probe says "conflict," the new owner reports the conflict and lets
the caller retry. If the probe times out or errors, the new owner
fail-closes (treats it as a conflict) rather than risk a double grant.
### Warm-up window on owner (re)start
When a filer starts (or restarts), it has no in-memory state for any key.
For a short warm-up period it cannot trust "no local conflict" as the truth
even for keys it now owns — a holder from before the restart may still be
in the system.
Each filer tracks `posixLockReadyAt` (atomic, set by the sweeper after the
first successful sweep). For `posixLockWarmup` (currently 10s) after that
timestamp, the owner defers granting non-blocking acquires whose
conflicts it cannot verify, instead returning the same fail-closed
conflict shape as the cooling probe. Holders re-assert their locks on the
next keepalive (see [[Distributed POSIX Locks]]), so the warm-up window
ends with the owner's state correctly rebuilt.
This combination — snapshot history + cooling probe + warm-up — is what
lets the cluster admit and evict filers without dropping the
"single-serialization-point-per-key" guarantee that everything else
depends on.
The cooling logic is in
[`weed/server/filer_grpc_server_posix_lock.go`](https://github.com/seaweedfs/seaweedfs/blob/master/weed/server/filer_grpc_server_posix_lock.go);
ObjectTransaction reuses the same `LockRing.GetPrimary` / forwarding path
and inherits the snapshot-history protection.
## What this lets you build
Because the serialization layer is generic, anything routed by key inherits
the same atomicity guarantee. Two examples shipped in master:
* **S3 conditional / versioned writes** ([[S3 Conditional Operations]],
[[S3 Object Versioning]], [[S3 Object Lock and Retention]]) — the S3
gateway sends an `ObjectTransaction` with the conditional headers as the
precondition and the version-pointer recompute as the mutation list. No
distributed lock is held across the multi-entry update.
* **Cross-mount POSIX advisory locks** ([[Distributed POSIX Locks]]) — the
mount calls a dedicated `PosixLock` RPC that uses the same route-by-key,
one-hop forwarding, snapshot history, and warm-up logic; the authority is
an in-memory `posixlock.Manager` on the owner filer rather than the entry
lock table.
## See also
* [[Distributed POSIX Locks]]
* [[S3 Conditional Operations]]
* [[S3 Object Versioning]]
* [[Filer Server API]]
_Sidebar.md
+2
@@ -56,11 +56,13 @@
* [[Filer as a Key-Large-Value Store]]
* [[Path Specific Configuration]]
* [[Filer Change Data Capture]]
* [[Filer Operation Serialization]]
### [[FUSE Mount]]
* [[FIO benchmark]]
* [[fstab and systemd mount]]
* [[POSIX Compliance]]
* [[Distributed POSIX Locks]]
* [[P2P reading in weed mount]]
### [[WebDAV]]