master,filer: fix lock ring poisoning after leader change (#11453)

* cluster: never broadcast an empty lock ring

An empty member list is never a usable ring state, but a delayed
RemoveServer on a former leader can fire after the new leader already
broadcast the recovered ring. That late broadcast carries a newer
wall-clock version, so clients accept the empty ring and permanently
reject the good one.

Skip the broadcast entirely when the member list is empty, keeping the
last non-empty snapshot for reconnecting clients.

* cluster: periodically rebroadcast the lock ring

Ring updates are purely event-driven, so one lost or poisoned update is
permanent until the next membership change — with a single filer that may
never come. Re-arm a per-group timer after every broadcast so the current
leader keeps re-sending the ring; clients reject nothing newer than their
last accepted version, so a re-sent snapshot always heals a stale view.

* filer,s3api: reset the lock ring on master change

Ring versions are per-master monotonic — each master stamps wall-clock
nanoseconds — so a late high-version update accepted from a former leader
makes the new leader's snapshot look stale forever. Detect a leader
change across the reconnect gap (currentMaster is cleared between
attempts, so remember the last served master) and reset the ring to
bootstrap state so the new leader's view always applies.

* cluster: fail lock acquisition when no lock server exists

retryUntilLocked loops forever, so a filer reporting an empty lock ring
wedges every append write indefinitely. Bound only the "no lock server
found" case — ordinary contention is still waited out since the holder
releases eventually. The constructors now return nil on failure: the
filer append path and S3 object writes fail fast, while mounts degrade
to their existing lockless mode.

* cluster: reset only the ring version on master change

Ring versions are per-master monotonic, so a version gate reset is all a
leader change needs. Clearing the whole ring made every filer its own
write owner until the next update and dropped the prior-owner window for
keys the new leader remaps; the last ring now keeps routing until the
new leader's snapshot transitions off it.

* cluster: skip redundant ring installs and defer rebroadcasts

An unchanged member list now only bumps the accepted version instead of
installing a snapshot: periodic rebroadcasts no longer fire the
topology-change callback or restart the prior-owner window. And a
rebroadcast that lands inside a membership stabilization window yields
to the pending timer rather than publishing an intermediate ring.

* cluster,mount: bound lock unavailability, fail ops that cannot lock

Only 'lock already owned' contention retries without bound now; every
other failure — no lock server, or a dead ring member refusing
connections — shares the same unavailability budget, so a ring naming
departed filers can no longer hang a lock forever. Mount open-write,
create, and rename fail with EAGAIN when the required lock cannot be
acquired instead of proceeding without cross-mount serialization.

* cluster: check pending stabilization inside the broadcast critical section

rebroadcast released the mutex between the pending-timer check and
nextBroadcastUpdate, so a membership change arriving in the gap could arm
a stabilization timer while the rebroadcast emitted an intermediate ring.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: acquire path locks before mutating create/rename state

Create took the DLM lock only after the filer create, so a lock failure
returned EAGAIN with an eagerly persisted file left behind. Rename marked
source handles renamed before acquiring locks, so a failed acquisition
left them suppressing old-path flushes for a rename that never happened.
Both now take the locks first; the create's lock is released again if the
entry race loses to another creator and AcquireHandle takes over.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: keep the old-path lock when rename lock migration fails

The migration stopped the handle's lock before acquiring the replacement,
so a nil result left the handle writing with no lock at all. Acquiring the
new-path lock first means failure keeps the existing lock instead of
reporting success with serialization dropped.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: skip new-path rename lock when a handle already holds it

A target file open for write on this mount already carries a lock on
newPath; the lock manager does not grant a second lock to the same
owner, so the rename would wait on itself until the handle closed.
Also avoid locking twice when old and new paths coincide.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: hand the rename's target lock to the migrating handle

The rename holds a lock on newPath for its duration, so the response
migration's fresh acquisition waited on that same lock until the handle
released — under fhLockTable, blocking the handle's own close. Adopt the
rename's lock directly; nested move responses still acquire their own.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: move the replaced target's lock to the renamed handle

When the target path was already locked by an open handle on this
mount, the migrated source handle kept only its stale old-path lock —
the target's close would then release the last lock on the new path
while the renamed handle was still open. Adopt the replaced handle's
lock instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: stop the handle lock inside the fh lock on release

ReleaseHandle stopped fh.dlmLock before taking the fhLockTable slot, so
a rename migration holding that slot could still observe and adopt a
lock that was already stopping. Stopping under the fh lock makes the
transfer serialize against the release.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: claim the replaced target's lock for the renamed handle

When the target path is already locked by an open handle on this mount,
adopting it at migration time keeps the renamed path protected after
that handle closes, without waiting on a lock this mount already holds.
If the handle was released mid-migration the claimed lock is stopped,
and a fresh acquire covers the case where it was already gone.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: claim the target handle's lock before the rename runs

Skipping the new-path lock when a handle already holds it let that
handle's close release the lock mid-rename, leaving the path unguarded
until the response migrated it. Take over the lock at check time and
hold it for the rename's duration: the response adopts it for the
migrating handle, or it returns to the target handle / is released on
failure. The target handle lookup also falls back to the entry's stored
inode for a forgotten path mapping.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: read handle locks only under the fh lock during rename

The loose dlmLock reads raced ReleaseHandle, which now mutates the lock
inside the handle lock; check and claim it under the same hold.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
Chris LuandDevin authored and GitHub committed 2026-09-26 11:57:00 +08:00
1 parent 4914c14982
commit f31a026b2a
17 files changed
+691 -106

No files matched your search

+3 -3
View File
@@ -211,6 +211,9 @@ func (fh *FileHandle) AddChunks(chunks []*filer_pb.FileChunk) {
}
func (fh *FileHandle) ReleaseHandle() {
fhActiveLock := fh.wfs.fhLockTable.AcquireLock("ReleaseHandle", fh.fh, util.ExclusiveLock)
defer fh.wfs.fhLockTable.ReleaseLock(fh.fh, fhActiveLock)
// Release distributed lock before cleaning up, so other mounts can
// proceed as soon as this handle is done flushing.
if fh.dlmLock != nil {
@@ -219,9 +222,6 @@ func (fh *FileHandle) ReleaseHandle() {
glog.V(1).Infof("DLM lock released for inode %d", fh.inode)
}
fhActiveLock := fh.wfs.fhLockTable.AcquireLock("ReleaseHandle", fh.fh, util.ExclusiveLock)
defer fh.wfs.fhLockTable.ReleaseLock(fh.fh, fhActiveLock)
if fh.entryChunkGroup != nil {
_ = fh.entryChunkGroup.Close()
}
+30 -10
View File
@@ -9,6 +9,7 @@ import (
"github.com/seaweedfs/go-fuse/v2/fuse"
"google.golang.org/protobuf/proto"
"github.com/seaweedfs/seaweedfs/weed/cluster"
"github.com/seaweedfs/seaweedfs/weed/cluster/lock_manager"
"github.com/seaweedfs/seaweedfs/weed/filer"
"github.com/seaweedfs/seaweedfs/weed/glog"
@@ -72,10 +73,26 @@ func (wfs *WFS) Create(cancel <-chan struct{}, in *fuse.CreateIn, name string, o
return code
}
// Acquire the DLM lock before the filer create: a lock failure after an
// eager create would return EAGAIN while the file stays persisted.
var dlmLock *cluster.LiveLock
if wfs.lockClient != nil {
owner := fmt.Sprintf("mount-%d", wfs.signature)
dlmLock = wfs.lockClient.NewBlockingLongLivedLock(
string(entryFullPath), owner, lock_manager.LiveLockTTL,
)
if dlmLock == nil {
return fuse.Status(syscall.EAGAIN)
}
}
inode, newEntry, code = wfs.createRegularFile(dirFullPath, name, in.Mode, in.Uid, in.Gid, 0, !wfs.option.EagerFilerCreate, true)
if code == fuse.Status(syscall.EEXIST) && in.Flags&syscall.O_EXCL == 0 {
// Race: another process created the file between our check and create.
// Reopen the winner's entry.
// Reopen the winner's entry; AcquireHandle takes its own lock.
if dlmLock != nil {
dlmLock.Stop()
}
newEntry, _, code = wfs.maybeLoadEntry(entryFullPath)
if code != fuse.OK {
return code
@@ -100,6 +117,9 @@ func (wfs *WFS) Create(cancel <-chan struct{}, in *fuse.CreateIn, name string, o
out.OpenFlags = 0
return fuse.OK
} else if code != fuse.OK {
if dlmLock != nil {
dlmLock.Stop()
}
return code
} else {
inode = wfs.inodeToPath.Lookup(entryFullPath, newEntry.Attributes.Crtime, false, false, inode, true)
@@ -122,15 +142,15 @@ func (wfs *WFS) Create(cancel <-chan struct{}, in *fuse.CreateIn, name string, o
// persisted the entry, so its handle starts clean.
fileHandle.dirtyMetadata = !wfs.option.EagerFilerCreate
// Acquire DLM lock for new file creation (Create bypasses AcquireHandle
// so we must acquire the lock here). Always lock on Create since file
// creation is inherently a write operation.
if wfs.lockClient != nil && fileHandle.dlmLock == nil {
owner := fmt.Sprintf("mount-%d", wfs.signature)
fileHandle.dlmLock = wfs.lockClient.NewBlockingLongLivedLock(
string(entryFullPath), owner, lock_manager.LiveLockTTL,
)
glog.V(1).Infof("DLM lock acquired for new file %s", entryFullPath)
// Create bypasses AcquireHandle, so attach the lock acquired above.
// A surviving handle may already hold one; ours is redundant then.
if dlmLock != nil {
if fileHandle.dlmLock == nil {
fileHandle.dlmLock = dlmLock
glog.V(1).Infof("DLM lock acquired for new file %s", entryFullPath)
} else {
dlmLock.Stop()
}
}
out.Fh = uint64(fileHandle.fh)
+7
View File
@@ -2,6 +2,7 @@ package mount
import (
"fmt"
"syscall"
"github.com/seaweedfs/go-fuse/v2/fuse"
"github.com/seaweedfs/seaweedfs/weed/cluster/lock_manager"
@@ -77,6 +78,12 @@ func (wfs *WFS) AcquireHandle(inode uint64, flags, uid, gid uint32) (fileHandle
fileHandle.dlmLock = wfs.lockClient.NewBlockingLongLivedLock(
string(path), owner, lock_manager.LiveLockTTL,
)
if fileHandle.dlmLock == nil {
// No lock server is reachable: proceeding would silently drop
// cross-mount write serialization, so fail the open instead.
wfs.fhMap.ReleaseByHandle(fileHandle.fh)
return nil, fuse.Status(syscall.EAGAIN)
}
glog.V(1).Infof("DLM lock acquired for %s", path)
}
return fileHandle, fuse.OK
+126 -48
View File
@@ -9,6 +9,7 @@ import (
"syscall"
"github.com/seaweedfs/go-fuse/v2/fuse"
"github.com/seaweedfs/seaweedfs/weed/cluster"
"github.com/seaweedfs/seaweedfs/weed/cluster/lock_manager"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
@@ -16,10 +17,10 @@ import (
)
// doRename tries the streaming mux first, falling back to unary on transport errors.
func (wfs *WFS) doRename(ctx context.Context, request *filer_pb.StreamRenameEntryRequest, oldPath, newPath util.FullPath) error {
func (wfs *WFS) doRename(ctx context.Context, request *filer_pb.StreamRenameEntryRequest, oldPath, newPath util.FullPath, newPathLock **cluster.LiveLock) error {
if wfs.streamMutate != nil && wfs.streamMutate.IsAvailable() {
err := wfs.streamMutate.Rename(ctx, request, func(resp *filer_pb.StreamRenameEntryResponse) error {
return wfs.handleRenameResponse(ctx, resp)
return wfs.handleRenameResponse(ctx, resp, newPath, newPathLock)
})
if err == nil || !errors.Is(err, ErrStreamTransport) {
return err // success or application error
@@ -39,7 +40,7 @@ func (wfs *WFS) doRename(ctx context.Context, request *filer_pb.StreamRenameEntr
}
return fmt.Errorf("dir Rename %s => %s receive: %v", oldPath, newPath, recvErr)
}
if err := wfs.handleRenameResponse(ctx, resp); err != nil {
if err := wfs.handleRenameResponse(ctx, resp, newPath, newPathLock); err != nil {
return err
}
}
@@ -240,6 +241,102 @@ func (wfs *WFS) Rename(cancel <-chan struct{}, in *fuse.RenameIn, oldName string
glog.V(4).Infof("dir Rename %s => %s", oldPath, newPath)
// Acquire DLM locks on both old and new paths to prevent another mount
// from opening either path for writing during the rename. Lock in
// sorted order to prevent deadlocks when two mounts rename in opposite
// directions (A→B vs B→A).
//
// Skip the old-path lock if this mount already holds it via an open
// file handle (otherwise we'd deadlock trying to re-acquire our own lock).
// Acquiring before the handle marks below keeps a lock failure from
// leaving source handles flagged for a rename that never happened.
var heldLocks []*cluster.LiveLock
var newPathLock *cluster.LiveLock
if wfs.lockClient != nil {
// A handle open on the target holds its lock; claiming it up front
// keeps the path locked through the rename (its close would otherwise
// release it mid-flight) without waiting on our own ownership.
var newPathLockFh *FileHandle
defer func() {
for _, l := range heldLocks {
l.Stop()
}
if newPathLock != nil {
// Never adopted by a migrated handle — return it to the
// handle it came from if that is still open.
if newPathLockFh != nil {
if cur, ok := wfs.fhMap.FindFileHandle(newPathLockFh.inode); ok && cur == newPathLockFh {
lk := wfs.fhLockTable.AcquireLock("renameDLM", cur.fh, util.ExclusiveLock)
if cur.dlmLock == nil {
cur.dlmLock = newPathLock
newPathLock = nil
}
wfs.fhLockTable.ReleaseLock(cur.fh, lk)
}
}
if newPathLock != nil {
newPathLock.Stop()
}
}
}()
owner := fmt.Sprintf("mount-%d", wfs.signature)
// Check if the source file handle already holds a DLM lock on oldPath
oldPathAlreadyLocked := false
sourceInode, sourceMapped := wfs.inodeToPath.GetInode(oldPath)
if !sourceMapped && oldEntry != nil && oldEntry.Attributes != nil {
sourceInode = oldEntry.Attributes.Inode
}
if sourceInode != 0 {
if fh, ok := wfs.fhMap.FindFileHandle(sourceInode); ok {
lk := wfs.fhLockTable.AcquireLock("renameDLM", fh.fh, util.ExclusiveLock)
oldPathAlreadyLocked = fh.dlmLock != nil
wfs.fhLockTable.ReleaseLock(fh.fh, lk)
}
}
targetInode, targetMapped := wfs.inodeToPath.GetInode(newPath)
if !targetMapped && newEntry != nil && newEntry.Attributes != nil {
targetInode = newEntry.Attributes.Inode
}
if targetInode != 0 && targetInode != sourceInode {
if targetFh, ok := wfs.fhMap.FindFileHandle(targetInode); ok {
targetFhLock := wfs.fhLockTable.AcquireLock("renameDLM", targetFh.fh, util.ExclusiveLock)
if targetFh.dlmLock != nil {
newPathLock = targetFh.dlmLock
targetFh.dlmLock = nil
newPathLockFh = targetFh
}
wfs.fhLockTable.ReleaseLock(targetFh.fh, targetFhLock)
}
}
// Determine which paths need new DLM locks
pathsToLock := []string{}
if newPathLock == nil {
pathsToLock = append(pathsToLock, string(newPath))
}
if !oldPathAlreadyLocked && string(oldPath) != string(newPath) {
pathsToLock = append(pathsToLock, string(oldPath))
}
// Sort for consistent lock ordering
if len(pathsToLock) == 2 && pathsToLock[0] > pathsToLock[1] {
pathsToLock[0], pathsToLock[1] = pathsToLock[1], pathsToLock[0]
}
for _, p := range pathsToLock {
dlmLock := wfs.lockClient.NewBlockingLongLivedLock(p, owner, lock_manager.LiveLockTTL)
if dlmLock == nil {
return fuse.Status(syscall.EAGAIN)
}
if p == string(newPath) {
newPathLock = dlmLock
} else {
heldLocks = append(heldLocks, dlmLock)
}
}
glog.V(1).Infof("DLM locks acquired for rename %s => %s (oldPathAlreadyLocked=%v)", oldPath, newPath, oldPathAlreadyLocked)
}
// Ensure the source file's metadata exists on the filer before renaming.
// Two cases can leave the entry only in the local cache:
// 1. deferFilerCreate=true — file handle still open, dirtyMetadata set.
@@ -300,41 +397,6 @@ func (wfs *WFS) Rename(cancel <-chan struct{}, in *fuse.RenameIn, oldName string
}
}
// Acquire DLM locks on both old and new paths to prevent another mount
// from opening either path for writing during the rename. Lock in
// sorted order to prevent deadlocks when two mounts rename in opposite
// directions (A→B vs B→A).
//
// Skip the old-path lock if this mount already holds it via an open
// file handle (otherwise we'd deadlock trying to re-acquire our own lock).
if wfs.lockClient != nil {
owner := fmt.Sprintf("mount-%d", wfs.signature)
// Check if the source file handle already holds a DLM lock on oldPath
oldPathAlreadyLocked := false
if sourceInode, found := wfs.inodeToPath.GetInode(oldPath); found {
if fh, ok := wfs.fhMap.FindFileHandle(sourceInode); ok && fh.dlmLock != nil {
oldPathAlreadyLocked = true
}
}
// Determine which paths need new DLM locks
pathsToLock := []string{string(newPath)}
if !oldPathAlreadyLocked {
pathsToLock = append(pathsToLock, string(oldPath))
}
// Sort for consistent lock ordering
if len(pathsToLock) == 2 && pathsToLock[0] > pathsToLock[1] {
pathsToLock[0], pathsToLock[1] = pathsToLock[1], pathsToLock[0]
}
for _, p := range pathsToLock {
dlmLock := wfs.lockClient.NewBlockingLongLivedLock(p, owner, lock_manager.LiveLockTTL)
defer dlmLock.Stop()
}
glog.V(1).Infof("DLM locks acquired for rename %s => %s (oldPathAlreadyLocked=%v)", oldPath, newPath, oldPathAlreadyLocked)
}
// update remote filer
request := &filer_pb.StreamRenameEntryRequest{
OldDirectory: string(oldDir),
@@ -345,7 +407,7 @@ func (wfs *WFS) Rename(cancel <-chan struct{}, in *fuse.RenameIn, oldName string
}
ctx := context.Background()
err := wfs.doRename(ctx, request, oldPath, newPath)
err := wfs.doRename(ctx, request, oldPath, newPath, &newPathLock)
if err != nil {
glog.V(0).Infof("Rename %s => %s: %v", oldPath, newPath, err)
// Map error strings to FUSE status codes. String matching is used
@@ -377,7 +439,7 @@ func (wfs *WFS) Rename(cancel <-chan struct{}, in *fuse.RenameIn, oldName string
}
func (wfs *WFS) handleRenameResponse(ctx context.Context, resp *filer_pb.StreamRenameEntryResponse) error {
func (wfs *WFS) handleRenameResponse(ctx context.Context, resp *filer_pb.StreamRenameEntryResponse, renameNewPath util.FullPath, renameNewPathLock **cluster.LiveLock) error {
// comes from filer StreamRenameEntry, can only be create or delete entry
glog.V(4).Infof("dir Rename %+v", resp.EventNotification)
@@ -411,16 +473,32 @@ func (wfs *WFS) handleRenameResponse(ctx context.Context, resp *filer_pb.StreamR
// Migrate the DLM lock from old path to new path so the
// lock key matches the current file location. Hold the
// fhLockTable to prevent ReleaseHandle from concurrently
// stopping the lock during migration.
if wfs.lockClient != nil {
// stopping the lock during migration. Acquire before
// releasing: a failed migration keeps the old lock rather
// than leaving the handle unlocked.
if wfs.lockClient != nil && oldPath != newPath {
fhActiveLock := wfs.fhLockTable.AcquireLock("renameDLM", fh.fh, util.ExclusiveLock)
if fh.dlmLock != nil {
owner := fmt.Sprintf("mount-%d", wfs.signature)
fh.dlmLock.Stop()
fh.dlmLock = wfs.lockClient.NewBlockingLongLivedLock(
string(newPath), owner, lock_manager.LiveLockTTL,
)
glog.V(1).Infof("DLM lock migrated from %s to %s", oldPath, newPath)
var newLock *cluster.LiveLock
if newPath == renameNewPath && renameNewPathLock != nil && *renameNewPathLock != nil {
// The rename already holds a lock on the target;
// adopt it — re-acquiring our own lock would block
// until this handle is released.
newLock = *renameNewPathLock
*renameNewPathLock = nil
} else {
owner := fmt.Sprintf("mount-%d", wfs.signature)
newLock = wfs.lockClient.NewBlockingLongLivedLock(
string(newPath), owner, lock_manager.LiveLockTTL,
)
}
if newLock != nil {
fh.dlmLock.Stop()
fh.dlmLock = newLock
glog.V(1).Infof("DLM lock migrated from %s to %s", oldPath, newPath)
} else {
glog.Warningf("DLM lock migration to %s failed; keeping lock on %s", newPath, oldPath)
}
}
wfs.fhLockTable.ReleaseLock(fh.fh, fhActiveLock)
}
+1 -1
View File
@@ -69,7 +69,7 @@ func TestHandleRenameResponseLeavesUncachedTargetOutOfCache(t *testing.T) {
},
}
if err := wfs.handleRenameResponse(context.Background(), resp); err != nil {
if err := wfs.handleRenameResponse(context.Background(), resp, targetPath, nil); err != nil {
t.Fatalf("handle rename response: %v", err)
}