mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-13 07:19:18 +01:00
c2b2ed769bbaa039ba83480e091b1dfc2f26a6aa
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
d7ae08ecad |
fix(mount): refuse stale whole-entry content flushes (#12594)
* fix(mount): condition content flushes on the observed entry * mount: stop retrying a flush the filer refused on its condition The filer reports a failed CreateEntry condition as FilerError_PRECONDITION_FAILED in the response, which surfaces as filer_pb.ErrPreconditionFailed rather than a FailedPrecondition status. The flush retry loop treated it as transient and replayed the refused flush three more times over 7 s before returning EIO. The test filer now answers like the real one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: take the handle base from the entry the filer stored After an acknowledged flush the handle base was the requested entry, but the filer stores it with a fresh commit_ts_ns (and fills crtime, inode, atime), so the next IF_ENTRY_EQUAL flush on the same handle never matched: write, fsync, write, fsync returned EIO, as did close after a periodic metadata flush and the first flush after an eager create. Use the stored entry from the acknowledgement's metadata event. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: condition the periodic metadata flush like the close flush The periodic metadata flush of an open, dirty handle sent the same whole entry through an unconditional CreateEntry, so a file held open past the flush interval still overwrote a peer's acknowledged append, and the close flush then matched the overwritten state. Both paths now publish through one helper that carries the handle's write condition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: republish a refused flush once when the stored entry is unchanged A handle's base can differ from the stored entry without any peer write: a rename while the file is open restamps the entry under its new name, and with -map.uid a filer id that is also a mapped local id does not map back to itself. Each made every flush of the handle fail with EIO. On a miss, read the stored entry and, if it still holds the observed state apart from name, commit stamp, id encoding and derived size, retry once with that entry as the condition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: take the filer's entry once a write open holds the DLM lock The lock orders this writer after the previous holder, but the holder's flush may not have reached the local cache yet, so the handle's base named the entry from before the wait and the conditional flush refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: share the open-existing path of Create Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: open the file a previous DLM holder created during a deferred create A deferred create never asks the filer, so a create that waited for the lock went on to publish over a file the previous holder had just made, and its flush expecting no entry was refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: send the lock ring snapshot to mounts A mount running -dlm is a lock client like the S3 gateway, and needs the ring on connect to reach a file's owner filer before the next membership change is broadcast. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: follow the masters' lock ring under -dlm The lock client only knew the seed filer, so every lock took the forward hop and the mount had no way to tell which filer owns a file. Subscribe to the masters named by the filer, like the S3 gateway, and wait briefly for the first ring before serving. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: read the DLM-ordered entry from the file's owner filer The filer checks a flush's condition on the owner of the path's write route key, and another filer may not have applied the previous holder's flush yet. Look the entry up on that owner once the ring is known. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
76fddab5ef |
master: keep a re-registered filer in the lock ring when its old stream ends (#12494)
* master: move the KeepConnected stream cleanup into removeClusterNode Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: drop a filer from the lock ring only with its last registration A filer's replacement can register under the same address before the master ends the old stream. The cluster list counts both registrations, but the old stream's cleanup removed the address from the lock ring outright, leaving the live filer out of the ring until it re-registered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: order a stream's cluster list and lock ring changes The cleanup of an ending stream could see its address removed from the cluster list, then a new registration of that address add it to the list and the ring, and only then remove it from the ring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: take a leaving filer's reconnect out of the lock ring A filer that reconnects while leaving registers with LeaveLockRing set. Its broken stream can still be open on the master, and since that stream's end no longer drops an address that is still registered, the reconnect removes the address from the ring itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Peter Dodd
|
52c6df3bec |
fix(filer): leave the lock ring before stopping gRPC on shutdown (#11604)
A draining filer stayed in the master's lock ring until its process exited, while gRPC GracefulStop was already refusing new connections. S3 gateways and peer filers kept routing object-write locks and owner-routed writes to it for the whole graceful-stop window. On shutdown the filer now sends leave_lock_ring on its open KeepConnected stream. The master removes it from the lock ring only; it stays a cluster member so peers keep following its metadata log through the drain. Once the ring update without it arrives, the filer has already transferred its locks to the new owners, and it keeps serving through the prior-owner window (plus a second for peers that apply the update later) before gRPC and HTTP begin draining. The whole leave is bounded at 10s so slow lock transfers or a stuck stream cannot hold up the drain. A lone filer, or a master that ignores the message, falls back to the previous behavior. |
||
|
Chris Lu
|
74038e1b14 |
master: don't let a dead KeepConnected handler close its successor's channel (#10900)
A client that reconnects before the old handler exits re-registers the same client name, and addClient overwrites the map entry. The old handler's deferred deleteClient then closed whatever channel the map held under that name: the new, live stream's. Receiving from a closed channel returns nil immediately and forever, so the new handler's send loop degenerated into sending empty responses at wire speed, pinning a core on each side until the client killed the connection. deleteClient now closes the channel its own handler registered and leaves the map entry alone unless it still points to that channel. This also closes the previously orphaned old channel, whose drain goroutine used to leak. The send loop treats a closed channel as an exit instead of a message stream. |
||
|
Neetika MittalandNeetika Mittal
|
45465e5a05 |
fix(master): notify clients after manual volume grow (#9656)
Co-authored-by: Neetika Mittal <mneetika@users.noreply.github.com> |
||
|
Chris LuandCopilot
|
75a6a34528 |
dlm: resilient distributed locks via consistent hashing + backup replication (#8860)
* dlm: replace modulo hashing with consistent hash ring Introduce HashRing with virtual nodes (CRC32-based consistent hashing) to replace the modulo-based hashKeyToServer. When a filer node is removed, only keys that hashed to that node are remapped to the next server on the ring, leaving all other mappings stable. This is the foundation for backup replication — the successor on the ring is always the natural takeover node. * dlm: add Generation and IsBackup fields to Lock Lock now carries IsBackup (whether this node holds the lock as a backup replica) and Generation (a monotonic fencing token that increments on each fresh acquisition, stays the same on renewal). Add helper methods: AllLocks, PromoteLock, DemoteLock, InsertBackupLock, RemoveLock, GetLock. * dlm: add ReplicateLock RPC and generation/is_backup proto fields Add generation field to LockResponse for fencing tokens. Add generation and is_backup fields to Lock message. Add ReplicateLock RPC for primary-to-backup lock replication. Add ReplicateLockRequest/ReplicateLockResponse messages. * dlm: add async backup replication to DistributedLockManager Route lock/unlock via consistent hash ring's GetPrimaryAndBackup(). After a successful lock or unlock on the primary, asynchronously replicate the operation to the backup server via ReplicateFunc callback. Single-server deployments skip replication. * dlm: add ReplicateLock handler and backup-aware topology changes Add ReplicateLock gRPC handler for primary-to-backup replication. Revise OnDlmChangeSnapshot to handle three cases on topology change: - Promote backup locks when this node becomes primary - Demote primary locks when this node becomes backup - Transfer locks when this node is neither primary nor backup Wire up SetupDlmReplication during filer server initialization. * dlm: expose generation fencing token in lock client LiveLock now captures the generation from LockResponse and exposes it via Generation() method. Consumers can use this as a fencing token to detect stale lock holders. * dlm: update empty folder cleaner to use consistent hash ring Replace local modulo-based hashKeyToServer with LockRing.GetPrimary() which uses the shared consistent hash ring for folder ownership. * dlm: add unit tests for consistent hash ring Test basic operations, consistency on server removal (only keys from removed server move), backup-is-successor property (backup becomes new primary when primary is removed), and key distribution balance. * dlm: add integration tests for lock replication failure scenarios Test cases: - Primary crash with backup promotion (backup has valid token) - Backup crash with primary continuing - Both primary and backup crash (lock lost, re-acquirable) - Rolling restart across all nodes - Generation fencing token increments on new acquisition - Replication failure (primary still works independently) - Unlock replicates deletion to backup - Lock survives server addition (topology change) - Consistent hashing minimal disruption (only removed server's keys move) * dlm: address PR review findings 1. Causal replication ordering: Add per-lock sequence number (Seq) that increments on every mutation. Backup rejects incoming mutations with seq <= current seq, preventing stale async replications from overwriting newer state. Unlock replication also carries seq and is rejected if stale. 2. Demote-after-handoff: OnDlmChangeSnapshot now transfers the lock to the new primary first and only demotes to backup after a successful TransferLocks RPC. If the transfer fails, the lock stays as primary on this node. 3. SetSnapshot candidateServers leak: Replace the candidateServers map entirely instead of appending, so removed servers don't linger. 4. TransferLocks preserves Generation and Seq: InsertLock now accepts generation and seq parameters. After accepting a transferred lock, the receiving node re-replicates to its backup. 5. Rolling restart test: Add re-replication step after promotion and assert survivedCount > 0. Add TestDLM_StaleReplicationRejected. 6. Mixed-version upgrade note: Add comment on HashRing documenting that all filer nodes must be upgraded together. * dlm: serve renewals locally during transfer window on node join When a new node joins and steals hash ranges from surviving nodes, there's a window between ring update and lock transfer where the client gets redirected to a node that doesn't have the lock yet. Fix: if the ring says primary != self but we still hold the lock locally (non-backup, matching token), serve the renewal/unlock here rather than redirecting. The lock will be transferred by OnDlmChangeSnapshot, and subsequent requests will go to the new primary once the transfer completes. Add tests: - TestDLM_NodeDropAndJoin_OwnershipDisruption: measures disruption when a node drops and a new one joins (14/100 surviving-node locks disrupted, all handled by transfer logic) - TestDLM_RenewalDuringTransferWindow: verifies renewal succeeds on old primary during the transfer window * dlm: master-managed lock ring with stabilization batching The master now owns the lock ring membership. Instead of filers independently reacting to individual ClusterNodeUpdate add/remove events, the master: 1. Tracks filer membership in LockRingManager 2. Batches rapid changes with a 1-second stabilization timer (e.g., a node drop + join within 1 second → single ring update) 3. Broadcasts the complete ring snapshot atomically via the new LockRingUpdate message in KeepConnectedResponse Filers receive the ring as a complete snapshot and apply it via SetSnapshot, ensuring all filers converge to the same ring state without intermediate churn. This eliminates the double-churn problem where a rapid drop+join would fire two separate ring mutations, each triggering lock transfers and disrupting ownership on surviving nodes. * dlm: track ring version, reject stale updates, remove dead code SetSnapshot now takes a version parameter from the master. Stale updates (version < current) are rejected, preventing reordered messages from overwriting a newer ring state. Version 0 is always accepted for bootstrap. Remove AddServer/RemoveServer from LockRing — the ring is now exclusively managed by the master via SetSnapshot. Remove the candidateServers map that was only used by those methods. * dlm: fix SelectLocks data race, advance generation on backup insert - SelectLocks: change RLock to Lock since the function deletes map entries, which is a write operation and causes a data race under RLock. - InsertBackupLock: advance nextGeneration to at least the incoming generation so that after failover promotion, new lock acquisitions get a generation strictly greater than any replicated lock. - Bump replication failure log from V(1) to Warningf for production visibility. * dlm: fix SetSnapshot race, test reliability, timer edge cases - SetSnapshot: hold LockRing lock through both version update and Ring.SetServers() so they're atomic. Prevents a concurrent caller from seeing the new version but applying stale servers. - Transfer window test: search for a key that actually moves primary when filer4 joins, instead of relying on a fixed key that may not. - renewLock redirect: pass the existing token to the new primary instead of empty string, so redirected renewals work correctly. - scheduleBroadcast: check timer.Stop() return value. If the timer already fired, the callback picks up latest state. - FlushPending: only broadcast if timer.Stop() returns true (timer was still pending). If false, the callback is already running. - Fix test comment: "idempotent" → "accepted, state-changing". * dlm: use wall-clock nanoseconds for lock ring version The lock ring version was an in-memory counter that reset to 0 on master restart. A filer that had seen version 5 would reject version 1 from the restarted master. Fix: use time.Now().UnixNano() as the version. This survives master restarts without persistence — the restarted master produces a version greater than any pre-restart value. * dlm: treat expired lock owners as missing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * dlm: reject stale lock transfers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * dlm: order replication by generation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * dlm: bootstrap lock ring on reconnect Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |