mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-27 23:40:41 +01:00
b747e0d3b1
* cluster: bound each DistributedLock attempt with a timeout An unreachable lock owner blocked the attempt on the connection timeout. Bound it so the write fails fast; a timed-out attempt keeps no renew token. * cluster: keep renewToken across failed lock attempts A request may have renewed the lease server-side before its response was lost, so the last token can still be valid on the next attempt. Clearing it on any transport failure strands a live lease until its TTL expires. A definitive rejection still clears the token through the response's empty RenewToken, and falling back to the seed filer no longer abandons the lease either. * cluster: bound each lock attempt by the remaining route window The 15s unavailability window started only after the first failure, so three 10s attempts plus sleeps outlasted it: the third failure selected the seed fallback at the same moment the loop gave up, and a reachable seed route was never tried. Start the clock before the first attempt and cap each attempt at what the window has left. When the window is spent on a non-seed target, fall back to the seed filer — it can forward to the real owner — and give that route its own window, so the fallback is always exercised before acquisition fails. * cluster: bound DistributedUnlock by the lock RPC timeout The unlock used context.Background(), so Stop against an unreachable lock server could wait on the TCP connect timeout far past the renewal goroutine's bounded shutdown wait. A timed-out unlock is harmless: the lease expires via its TTL regardless. * dlm: replay the current token on a lost renewal's retry A renewal rotates the token server-side before the response is sent. When that response is lost the client only knows the superseded token: retries mismatched, the token was cleared, and the holder conflicted with its own lease until it expired. The lock now remembers the superseded token, so a retry from the same owner replays the current token, and a superseded token can still release its own lock. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * cluster: give the seed lock fallback one fresh window per acquisition The seed can redirect back to a dead primary via LockHostMovedTo, so refilling the retry window on every return to the seed made the primary->seed->primary cycle unbounded. The first seed fallback keeps its fresh window; later returns to the seed keep the running deadline. * cluster: advance the lock seq when a lost-renewal replay extends the lease A retry of a renewal whose response was lost updates ExpiredAtNs in place but returned the existing Seq, so the replicated extension was rejected by InsertBackupLock as a stale mutation. A promoted backup would then grant the lock when the old expiry passed even though the holder still believes the lease is renewed. Bump Seq on the replay so the backup accepts the new expiry. * cluster: carry the superseded lock token through replication and transfer A renewal rotates the lock token and keeps the superseded one in PrevToken so a holder whose renewal response was lost can still recover. That recovery broke as soon as the lock moved: neither the replicated backup copy nor a transferred lock carried PrevToken, so a new primary rejected retries of the old token until the lease expired. Add prev_renew_token to the Lock and ReplicateLockRequest messages and thread it through ReplicateLock, TransferLocks, InsertBackupLock and InsertLock. * cluster: count clean lock redirects against the retry window A bare LockHostMovedTo response (no error, no token) returns nil from attemptToLockWithin, and retryUntilLocked reset the retry window on every nil result. Two ring-disagreeing filers forwarding to each other therefore bounced the client in an unbounded loop; the spent-window exit could also return a nil error with no token, handing callers an unacquired lock. * cluster: do not mark a lock held or hot-loop on tokenless attempts A bare LockHostMovedTo redirect returns nil with no renewToken; the attempt previously flipped isLocked to 1 anyway and retried immediately. Gate the flag on a granted token and pace the retry like other outcomes. * cluster: keep renewToken across bare lock redirects A LockHostMovedTo response carries no token and no error; overwriting renewToken with it dropped a held token, so the next renewal re-acquired fresh and could contend against its own still-live lease until TTL. * cluster: poll for replicated backup state in lock tests Replication runs in a goroutine, so a fixed sleep followed by an immediate backup read is racy. Wait for the expected backup state with require.Eventually instead. * cluster: mark a lock held only on a grant verdict A bare LockHostMovedTo redirect with a saved token left isLocked set without the redirected filer granting anything, and retryUntilLocked could exit on a stale token with no attempt made. doLock now reports whether the response was a grant, and both the retry loop and isLocked follow that verdict. --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>