* cluster: bound each DistributedLock attempt with a timeout
An unreachable lock owner blocked the attempt on the connection timeout.
Bound it so the write fails fast; a timed-out attempt keeps no renew token.
* cluster: keep renewToken across failed lock attempts
A request may have renewed the lease server-side before its response was
lost, so the last token can still be valid on the next attempt. Clearing
it on any transport failure strands a live lease until its TTL expires.
A definitive rejection still clears the token through the response's
empty RenewToken, and falling back to the seed filer no longer abandons
the lease either.
* cluster: bound each lock attempt by the remaining route window
The 15s unavailability window started only after the first failure, so
three 10s attempts plus sleeps outlasted it: the third failure selected
the seed fallback at the same moment the loop gave up, and a reachable
seed route was never tried.
Start the clock before the first attempt and cap each attempt at what
the window has left. When the window is spent on a non-seed target,
fall back to the seed filer — it can forward to the real owner — and
give that route its own window, so the fallback is always exercised
before acquisition fails.
* cluster: bound DistributedUnlock by the lock RPC timeout
The unlock used context.Background(), so Stop against an unreachable
lock server could wait on the TCP connect timeout far past the renewal
goroutine's bounded shutdown wait. A timed-out unlock is harmless: the
lease expires via its TTL regardless.
* dlm: replay the current token on a lost renewal's retry
A renewal rotates the token server-side before the response is sent. When
that response is lost the client only knows the superseded token: retries
mismatched, the token was cleared, and the holder conflicted with its own
lease until it expired. The lock now remembers the superseded token, so a
retry from the same owner replays the current token, and a superseded
token can still release its own lock.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* cluster: give the seed lock fallback one fresh window per acquisition
The seed can redirect back to a dead primary via LockHostMovedTo, so
refilling the retry window on every return to the seed made the
primary->seed->primary cycle unbounded. The first seed fallback keeps
its fresh window; later returns to the seed keep the running deadline.
* cluster: advance the lock seq when a lost-renewal replay extends the lease
A retry of a renewal whose response was lost updates ExpiredAtNs in place
but returned the existing Seq, so the replicated extension was rejected
by InsertBackupLock as a stale mutation. A promoted backup would then
grant the lock when the old expiry passed even though the holder still
believes the lease is renewed. Bump Seq on the replay so the backup
accepts the new expiry.
* cluster: carry the superseded lock token through replication and transfer
A renewal rotates the lock token and keeps the superseded one in
PrevToken so a holder whose renewal response was lost can still recover.
That recovery broke as soon as the lock moved: neither the replicated
backup copy nor a transferred lock carried PrevToken, so a new primary
rejected retries of the old token until the lease expired.
Add prev_renew_token to the Lock and ReplicateLockRequest messages and
thread it through ReplicateLock, TransferLocks, InsertBackupLock and
InsertLock.
* cluster: count clean lock redirects against the retry window
A bare LockHostMovedTo response (no error, no token) returns nil from
attemptToLockWithin, and retryUntilLocked reset the retry window on every
nil result. Two ring-disagreeing filers forwarding to each other therefore
bounced the client in an unbounded loop; the spent-window exit could also
return a nil error with no token, handing callers an unacquired lock.
* cluster: do not mark a lock held or hot-loop on tokenless attempts
A bare LockHostMovedTo redirect returns nil with no renewToken; the
attempt previously flipped isLocked to 1 anyway and retried immediately.
Gate the flag on a granted token and pace the retry like other outcomes.
* cluster: keep renewToken across bare lock redirects
A LockHostMovedTo response carries no token and no error; overwriting
renewToken with it dropped a held token, so the next renewal re-acquired
fresh and could contend against its own still-live lease until TTL.
* cluster: poll for replicated backup state in lock tests
Replication runs in a goroutine, so a fixed sleep followed by an
immediate backup read is racy. Wait for the expected backup state
with require.Eventually instead.
* cluster: mark a lock held only on a grant verdict
A bare LockHostMovedTo redirect with a saved token left isLocked set
without the redirected filer granting anything, and retryUntilLocked
could exit on a stale token with no attempt made. doLock now reports
whether the response was a grant, and both the retry loop and isLocked
follow that verdict.
---------
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* dlm: replace modulo hashing with consistent hash ring
Introduce HashRing with virtual nodes (CRC32-based consistent hashing)
to replace the modulo-based hashKeyToServer. When a filer node is
removed, only keys that hashed to that node are remapped to the next
server on the ring, leaving all other mappings stable. This is the
foundation for backup replication — the successor on the ring is
always the natural takeover node.
* dlm: add Generation and IsBackup fields to Lock
Lock now carries IsBackup (whether this node holds the lock as a backup
replica) and Generation (a monotonic fencing token that increments on
each fresh acquisition, stays the same on renewal). Add helper methods:
AllLocks, PromoteLock, DemoteLock, InsertBackupLock, RemoveLock, GetLock.
* dlm: add ReplicateLock RPC and generation/is_backup proto fields
Add generation field to LockResponse for fencing tokens.
Add generation and is_backup fields to Lock message.
Add ReplicateLock RPC for primary-to-backup lock replication.
Add ReplicateLockRequest/ReplicateLockResponse messages.
* dlm: add async backup replication to DistributedLockManager
Route lock/unlock via consistent hash ring's GetPrimaryAndBackup().
After a successful lock or unlock on the primary, asynchronously
replicate the operation to the backup server via ReplicateFunc
callback. Single-server deployments skip replication.
* dlm: add ReplicateLock handler and backup-aware topology changes
Add ReplicateLock gRPC handler for primary-to-backup replication.
Revise OnDlmChangeSnapshot to handle three cases on topology change:
- Promote backup locks when this node becomes primary
- Demote primary locks when this node becomes backup
- Transfer locks when this node is neither primary nor backup
Wire up SetupDlmReplication during filer server initialization.
* dlm: expose generation fencing token in lock client
LiveLock now captures the generation from LockResponse and exposes it
via Generation() method. Consumers can use this as a fencing token to
detect stale lock holders.
* dlm: update empty folder cleaner to use consistent hash ring
Replace local modulo-based hashKeyToServer with LockRing.GetPrimary()
which uses the shared consistent hash ring for folder ownership.
* dlm: add unit tests for consistent hash ring
Test basic operations, consistency on server removal (only keys from
removed server move), backup-is-successor property (backup becomes
new primary when primary is removed), and key distribution balance.
* dlm: add integration tests for lock replication failure scenarios
Test cases:
- Primary crash with backup promotion (backup has valid token)
- Backup crash with primary continuing
- Both primary and backup crash (lock lost, re-acquirable)
- Rolling restart across all nodes
- Generation fencing token increments on new acquisition
- Replication failure (primary still works independently)
- Unlock replicates deletion to backup
- Lock survives server addition (topology change)
- Consistent hashing minimal disruption (only removed server's keys move)
* dlm: address PR review findings
1. Causal replication ordering: Add per-lock sequence number (Seq) that
increments on every mutation. Backup rejects incoming mutations with
seq <= current seq, preventing stale async replications from
overwriting newer state. Unlock replication also carries seq and is
rejected if stale.
2. Demote-after-handoff: OnDlmChangeSnapshot now transfers the lock to
the new primary first and only demotes to backup after a successful
TransferLocks RPC. If the transfer fails, the lock stays as primary
on this node.
3. SetSnapshot candidateServers leak: Replace the candidateServers map
entirely instead of appending, so removed servers don't linger.
4. TransferLocks preserves Generation and Seq: InsertLock now accepts
generation and seq parameters. After accepting a transferred lock,
the receiving node re-replicates to its backup.
5. Rolling restart test: Add re-replication step after promotion and
assert survivedCount > 0. Add TestDLM_StaleReplicationRejected.
6. Mixed-version upgrade note: Add comment on HashRing documenting that
all filer nodes must be upgraded together.
* dlm: serve renewals locally during transfer window on node join
When a new node joins and steals hash ranges from surviving nodes,
there's a window between ring update and lock transfer where the
client gets redirected to a node that doesn't have the lock yet.
Fix: if the ring says primary != self but we still hold the lock
locally (non-backup, matching token), serve the renewal/unlock here
rather than redirecting. The lock will be transferred by
OnDlmChangeSnapshot, and subsequent requests will go to the new
primary once the transfer completes.
Add tests:
- TestDLM_NodeDropAndJoin_OwnershipDisruption: measures disruption
when a node drops and a new one joins (14/100 surviving-node locks
disrupted, all handled by transfer logic)
- TestDLM_RenewalDuringTransferWindow: verifies renewal succeeds on
old primary during the transfer window
* dlm: master-managed lock ring with stabilization batching
The master now owns the lock ring membership. Instead of filers
independently reacting to individual ClusterNodeUpdate add/remove
events, the master:
1. Tracks filer membership in LockRingManager
2. Batches rapid changes with a 1-second stabilization timer
(e.g., a node drop + join within 1 second → single ring update)
3. Broadcasts the complete ring snapshot atomically via the new
LockRingUpdate message in KeepConnectedResponse
Filers receive the ring as a complete snapshot and apply it via
SetSnapshot, ensuring all filers converge to the same ring state
without intermediate churn.
This eliminates the double-churn problem where a rapid drop+join
would fire two separate ring mutations, each triggering lock
transfers and disrupting ownership on surviving nodes.
* dlm: track ring version, reject stale updates, remove dead code
SetSnapshot now takes a version parameter from the master. Stale
updates (version < current) are rejected, preventing reordered
messages from overwriting a newer ring state. Version 0 is always
accepted for bootstrap.
Remove AddServer/RemoveServer from LockRing — the ring is now
exclusively managed by the master via SetSnapshot. Remove the
candidateServers map that was only used by those methods.
* dlm: fix SelectLocks data race, advance generation on backup insert
- SelectLocks: change RLock to Lock since the function deletes map
entries, which is a write operation and causes a data race under RLock.
- InsertBackupLock: advance nextGeneration to at least the incoming
generation so that after failover promotion, new lock acquisitions
get a generation strictly greater than any replicated lock.
- Bump replication failure log from V(1) to Warningf for production
visibility.
* dlm: fix SetSnapshot race, test reliability, timer edge cases
- SetSnapshot: hold LockRing lock through both version update and
Ring.SetServers() so they're atomic. Prevents a concurrent caller
from seeing the new version but applying stale servers.
- Transfer window test: search for a key that actually moves primary
when filer4 joins, instead of relying on a fixed key that may not.
- renewLock redirect: pass the existing token to the new primary
instead of empty string, so redirected renewals work correctly.
- scheduleBroadcast: check timer.Stop() return value. If the timer
already fired, the callback picks up latest state.
- FlushPending: only broadcast if timer.Stop() returns true (timer
was still pending). If false, the callback is already running.
- Fix test comment: "idempotent" → "accepted, state-changing".
* dlm: use wall-clock nanoseconds for lock ring version
The lock ring version was an in-memory counter that reset to 0 on
master restart. A filer that had seen version 5 would reject version 1
from the restarted master.
Fix: use time.Now().UnixNano() as the version. This survives master
restarts without persistence — the restarted master produces a
version greater than any pre-restart value.
* dlm: treat expired lock owners as missing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dlm: reject stale lock transfers
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dlm: order replication by generation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dlm: bootstrap lock ring on reconnect
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
