mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-07 06:47:51 +02:00
Chris Lu
dfecd664f9
* fix(master): do not re-enter warmup when a fresh cluster grows its first volume Follow-up investigation on #8777. After fixing the writable-chunk cap deadlock, a second issue surfaced on the same fio reproducer: the weed mount would log thousands of upload data X: filerGrpcAddress assign volume: assign volume failure count:1 path:"/test2/X": assign volume: rpc error: code = Canceled desc = grpc: the client connection is closing cascading from the filer's handler. The mount's own cached gRPC connection to the filer was never invalidated — the "client connection is closing" text is the FILER's cached connection to the MASTER going away, and the message is forwarded verbatim in the filer's AssignVolumeResponse.Error. Root cause: Topology.IsWarmingUp checks the *live* GetMaxVolumeId. On a fresh cluster the master is initialized with SetLastLeaderChangeTime (master_server.go:239 "Seed the warmup timestamp so IsWarmingUp() is active even if the leader change event hasn't fired yet"), but the MaxVolumeId==0 guard is supposed to short-circuit IsWarmingUp for bootstraps so there is no wait. That guard breaks the moment the first volume is grown inside the warmup window: MaxVolumeId flips from 0 to 1, the lastLeaderChangeTime is still within 3*pulse (15 s default), and IsWarmingUp retroactively returns true for the next several seconds. Every AssignVolume in that window returns codes.Unavailable, which trips the filer's shouldInvalidateConnection guard and tears down its cached master connection, which in turn surfaces as "client connection is closing" to every concurrent in-flight call from the mount's file-close flush storm. fio reports EIO on close and the user sees a thousand scary error lines in the mount log for an otherwise correct run. Fix: snapshot `hadVolumesAtLeaderChange` inside SetLastLeaderChangeTime and read that snapshot in IsWarmingUp instead of the live MaxVolumeId. A fresh cluster snapshots "no volumes at leader change" → IsWarmingUp stays false through the entire warmup window regardless of how fast the first grow lands. A real leader transition on a populated cluster still snapshots "has volumes" → IsWarmingUp behaves exactly as before until the 3*pulse window closes. The lock ordering in SetLastLeaderChangeTime reads MaxVolumeId before taking the lastLeaderChangeTimeLock so the two calls cannot interleave weirdly; IsWarmingUp reads both fields under a single RLock acquisition. Verified with the same containerized reproducer used for the deadlock fix: 4 jobs × 250 nrfiles × 40 MiB × 4k randwrite direct. - baseline (master): fio rc=1 (EIO on close), 2809 mount error lines matching "filerGrpcAddress assign volume ... client connection is closing", 788 filer lines matching "warming up", 331 "Removing cached gRPC connection to ...19333 due to error: master is warming up". - patched: fio rc=0 in 1.9 s at 79.2 MiB/s, 0 mount errors, 0 filer "warming up" lines, 0 cached-master-conn invalidations. go test ./weed/topology/... passes. * fix(topology): make NodeImpl.maxVolumeId atomic CodeRabbit flagged on #9092 that my new IsWarmingUp snapshot (hadVolumesAtLeaderChange) reads through GetMaxVolumeId(), which until now returned an unprotected int field on NodeImpl. UpAdjustMaxVolumeId is called from the volume server heartbeat path in parallel with GetMaxVolumeId reads on the assign path, and neither side had any synchronization — a long-standing data race the race detector would flag if the warmup test suite stressed both sides. Switch maxVolumeId to atomic.Uint32 (needle.VolumeId is uint32) and implement UpAdjustMaxVolumeId as a CAS loop so the check-then-set stays linearizable: two heartbeats racing to promote the field will land the higher value deterministically and propagate to the parent exactly once. GetMaxVolumeId is a single atomic load. Callers of both helpers are unchanged; the struct field comment documents why the atomic is necessary. go test -race ./weed/topology/... passes. * refactor(topology): use WarmupDuration helper in IsWarmingUp Gemini review on #9092 flagged that IsWarmingUp re-derives the warmup duration from pulse and WarmupPulseMultiplier instead of using the existing WarmupDuration() helper, which RemainingWarmupDuration already uses. Fold the duration calculation through the helper and short-circuit on lastChange.IsZero() before the time.Since call. No behavior change.