* wdclient: age vid map entries by generation instead of chaining snapshots
The vid map kept its history as a linked list of past snapshots, trimmed
in place by storing nil into a node's cache pointer. That cost up to six
full copies of the volume-location map, a recursive walk taking a
different lock per level, and deletes that had to cascade through every
generation. It also had to special-case explicitly-empty entries, or
fallback would resurrect locations a newer snapshot had cleared.
Keep one map instead, and stamp each entry with the generation it was
learned in. resetVidMap bumps the generation and drops entries that were
not relearned within the retained window, which is the same retention
the chain provided: an entry survives DefaultVidMapCacheSize resets.
The first write of a generation replaces an entry rather than merging
into it, so a volume that moved answers with where it is now — the
property a fresh map per reset used to give for free. Entries are
copy-on-write, so locations handed to a caller are no longer shifted
underneath it by a concurrent delete.
The map is never swapped now, so the client-side lock and its stable /
current accessors go away with it.
* wdclient: make vid map entries immutable and drop them once emptied
Review follow-up. Updating an entry in place left the copy-on-write
guarantee resting on callers never holding the entry pointer; install a
new entry instead, so the rule is simply that a stored entry never
changes.
Deleting a volume's last location now drops the entry rather than
keeping an empty one, which a client that never resets would otherwise
hold for every volume it ever saw deleted. Lookups already treat an
empty entry as a miss, so nothing observable changes.
* wdclient: let the newest generation decide between regular and EC locations
GetLocations checked the regular map first whatever its generation, so a
volume that was EC encoded kept answering with the regular copies the
previous master knew until they expired — for as long as the retained
window, since nothing relearns a copy that no longer exists.
The snapshot chain did not have this problem: the newest map was
consulted first and only a volume it knew nothing about fell through to
older ones. Restore that by comparing generations, with regular copies
winning a tie, since a tie means one generation reported both.
fix(wdclient): prevent stale cache fallback for empty volume locations
## Problem
During Kubernetes pod restarts, volume servers temporarily disconnect and their
locations are removed from vidMap. The deleteLocation function leaves an empty
array [] in vid2Locations map instead of removing the key entirely.
GetLocations() was checking 'if found && len(locations) > 0', which would fail
for empty arrays and fall back to the cache chain, returning STALE locations
from before the restart. This caused S3 gateway to try connecting to old pod
IPs that no longer exist, resulting in connection timeouts and hanging registry
sync jobs.
Example timeline:
1. Volume pod at 10.131.1.28:8081 registers volumes 10,12
2. S3 gateway caches: vid2Locations[10] = [10.131.1.28:8081]
3. Pod restarts, gets new IP 10.131.1.65:8081
4. Master sends delete → vid2Locations[10] = [] (empty, but key exists)
5. BUG: GetLocations(10) sees found=true, len=0 → falls back to cache
6. Returns stale 10.131.1.28:8081 instead of waiting for new location
7. S3 requests timeout trying to reach unreachable old IP
## Solution
Distinguish between two cases:
- found=true, locations=[] : Volume explicitly has no locations (e.g. restart)
→ Return nil, false (no fallback to cache)
- found=false : Volume never seen in current map
→ Check cache (preserve cache benefits for unknown volumes)
An empty array explicitly means 'this volume currently has no locations',
which is semantically different from 'volume unknown'. Don't fall back to
stale cache for explicitly empty volumes.
## Testing
Added comprehensive tests:
- TestGetLocationsEmptyArrayNoFallback: Verifies empty arrays don't use cache
- TestGetLocationsUnknownVolumeUsesCache: Verifies unknown volumes still use cache
- All existing tests pass
## Impact
Fixes registry sync job hangs during SeaweedFS upgrades/restarts. S3 gateway
will now correctly wait for updated volume locations instead of using stale
cached IPs.
Related: OutSystems.SeaWeedfs Helm chart, vega cluster incident 2026-06-24
Chris Lu