mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 23:50:43 +02:00
Chris Lu
2ef0e60aeb
* filer.sync: count received, processed, and failed events and export in-flight jobs The metadata processor admits at most -concurrency jobs and blocks the subscription stream past that, so the backlog lives in the source filer's metadata log and cannot be counted here. What can be measured honestly: events read off the stream, replication outcomes, and worker saturation. in_flight_jobs pinned at the concurrency limit means the sync itself is the bottleneck; near zero means it is caught up or starved by the source. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * filer.sync: export replication lag in seconds Lag is now minus the freshest of the processed watermark and the last idle heartbeat: the watermark stops at the last real event, so a quiet caught-up stream would otherwise show phantom lag. A ticker drives the gauge because the offset callback only fires while events flow and freezes exactly when the workers are saturated. Until the first event or heartbeat the gauge stays unset rather than reporting lag against a zero or stale resume offset. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * filer.sync: track replicated data sizes alongside event counts An event count hides that 32 in-flight jobs can be 32 KB or 300 GB. Byte counters mirror the event counters, and in_flight_bytes pairs with in_flight_jobs. An event's size is the chunk delta - new chunks the old entry does not already have - so deletes, renames, and attribute-only updates count zero and byte rates reflect data movement, not metadata churn. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * grafana: chart the new filer.sync metrics The lag panel reads lag_seconds directly instead of deriving it from sync_offset, and the sync row gains event rate, throughput, and the in-flight jobs and bytes gauges. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * filer.sync: a pinned failure keeps showing as lag An idle heartbeat means the stream is consumed, not that every event replicated. While a permanent failure pins the watermark, letting the heartbeat advance lag_seconds or the sync_offset gauge would report a caught-up stream with an unreplicated event in it, so both now ignore heartbeats until a restart replays the failure. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * filer.sync: in-flight gauges survive subscription retries A subscription retry builds a new processor sharing the gauge children while the old processor's jobs may still be draining, so setting the gauge from either side's local count clobbers the other. Each job now increments and decrements for itself, keeping the total truthful across generations. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw