* fix(s3): track filer joins and leaves pushed by the master
The S3 FilerClient replaced its -filer seed with a master snapshot of
filer IPs at boot and refreshed it only every 5 minutes. A rolling
restart replaces every filer well inside that window, leaving S3
servers with only dead addresses and failing every write until the
next poll.
Apply the master's ClusterNodeUpdate pushes to the filer list as they
arrive, keeping the poll as a backstop. The last filer is never
removed, and a poll snapshot requested before a push was applied is
discarded rather than overwriting newer membership.
* Defer last-filer leaves; bump the generation only on real changes
* fix(s3): cancel deferred filer leaves on rejoin and on discovery
A deferred last-filer leave outlived the filer it was recorded for: a
rejoin at the same address looked like a duplicate add, and a discovery
snapshot left the entry behind. The next join then removed a live
filer until the following poll.
A join now cancels any deferred leave for its address, and an applied
snapshot clears them, since it is the master's current membership.
* Bump the push generation when a rejoin cancels a deferred leave
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* wdclient: prune filers dropped from master discovery
Filer discovery only appended new addresses; it never removed ones that
disappeared from the master snapshot. After a K8s filer pod rolled to a
new IP the old address lingered in filerAddresses and got retried again
every resetTimeout window, stalling S3 uploads on i/o timeouts.
Treat the master snapshot as authoritative: keep survivors (preserving
their health counters and the active round-robin index), append newcomers
with fresh health, drop the rest. Empty snapshots are still ignored so a
transient master outage can't wipe the list.
* wdclient: skip discovery snapshots with no usable addresses
Guard against the defensive case where master returns updates whose
addresses are all empty; reconciling against an empty discovered set
would prune every filer.
