* wdclient: publish applied volume-location deletes to registered listeners
The master already tells each connected client which volumes a server
stopped hosting; only the MasterClient's own vidMap applied it. Expose
the same filtered deletions (moves between a server's disks excluded)
through OnVolumeLocationDelete so clients keeping a second location
cache can drop the retired server too.
* wdclient: drop retired volume servers from the FilerClient location cache
The S3 gateway's FilerClient keeps its own vidMap, filled by filer
lookups, and only forgot a retired volume server after a read against
it failed (a ~15s header-timeout stall when the host still resolves).
When built with a MasterClient, subscribe to its applied deletions so
the cache forgets the server as soon as the master reports it gone.
* wdclient: drop retired EC servers from the regular location map too
Filer lookups cache EC locations alongside regular ones in vid2Locations,
so an EC removal has to clear the retired server there as well as in the
EC map for the delete to reach what reads actually use.
* wdclient: tombstone pushed deletions against in-flight provider fills
A filer lookup already running when a delete lands can return the retired
location afterwards and re-cache it. Keep a short tombstone on each
deleted (vid, url) pair so fills neither cache nor return it while the
lookup source is still settling.
* wdclient: clear the FilerClient location cache on master reconnect
Removals the master pushed while the gateway link was down never arrive
as deletes, so on reconnect the lookup-fed cache is dropped outright
rather than kept generation-wise like the stream-fed one, and relearned
through filer lookups as before.
* wdclient: invoke push hooks outside their registration locks
A callback that registers or replaces a hook would wait on the read lock
the invocation already holds and stall the master stream. Copy the hook
under the lock and call it after the lock is released.
* wdclient: report pushed adds so confirmed re-adds clear the suppression
A server coming back at the same URL inside the tombstone window stayed
hidden from provider fills. updateVidMap now carries the additions it
applied through the same hook (renamed OnVolumeLocationChange), and the
registered cache untombstones them without eagerly caching them.
* wdclient: amortize tombstone expiry over a delete batch
A server leaving takes every volume it hosted with it, and scanning all
recentDeletes per removed vid squared the batch. Prune lazily on reads
and at most once per half TTL on writes.
* wdclient: let every location cache subscribe to pushed volume changes
The setters were single-slot, so a second FilerClient built on the same
MasterClient would have silently replaced the first one's subscription.
Registration now returns an unhook func, delivered to all listeners, and
FilerClient.Close releases its own.
* wdclient: hold the deletion fence until the master re-confirms a location
* wdclient: check the deletion fence and the cache add under one lock
* wdclient: drop provider answers that crossed a cache reset
* wdclient: bound the deletion fence by evicting its oldest mark
* wdclient: refresh a repeated removal instead of evicting another mark
* wdclient: evict the oldest deletion-fence mark in constant time
* wdclient: fence a server only once it serves the volume in no form
ec.encode registers the shards before the regular copy is deleted from the
same server, and ec.decode mounts the regular copy before the shards come
off it. Both arrive add-first in separate heartbeats, so the later removal
tombstoned a (vid, url) pair the server still serves and dropped it from
the FilerClient map; the filer's answer naming that server was then fenced
out and the volume read as not found until the gateway reconnected.
Tombstone and report a removal only when neither map still names the url
for that volume.
* wdclient: serve a provider answer that crossed a cache reset
The answer is as fresh as the provider's view; only the cache it would
land in has been re-told. Dropping it failed every read whose lookup was
in flight during a master reconnect with 'volume not found'. Return it
and leave the new view to be filled by the next lookup.
* wdclient: test the moved volume whose old server keeps proxying
* wdclient: stop caching inside the master lookup provider
The provider wrote every answered location through addLocation, which
clears the deletion fence: a lookup still naming a server a pushed delete
had just retired untombstoned and re-cached it. Fallback lookups already
gate caching through fillLocation, and authoritative lookups bypass the
cache by contract, so the provider no longer writes the map at all.
* wdclient: count only master-pushed locations when suppressing a removal
hasLocation could not tell a confirmed regular copy from a lookup fill:
the master answers an EC volume with its shard servers, so the regular map
holds shard locations that arrived by lookup, and entries an earlier
generation learned linger past a reconnect. An EC removal whose url still
showed either was suppressed, leaving the retired server served.
The vid map now records the generation each regular location was pushed
at, drops an unconfirmed regular entry alongside the shard that fed it,
and answers hasServingLocation from pushed marks and current-generation
entries only.
* wdclient: test removals against lookup-cached and stale locations
* wdclient: fence removals only in the gateway cache
The fence and the pushed-location marks ran inside every MasterClient,
including the filer's, whose map is also filled by lookups and cleared
by InvalidateCache. A removal reported while the server still served the
volume (e.g. a decoded copy whose mark a failed read had wiped) left the
pair fenced until the master pushed it again, with reads failing as
volume not found meanwhile.
Only the S3 gateway's FilerClient now tombstones, from the removals its
push-only MasterClient reports. That client decides a removal from the
current generation of both its maps after the whole message is applied,
so pushedLocations goes away, and the master lookup provider caches as
before. The hooks follow the existing single-slot SetOn...Fn pattern and
are wired next to SetOnPeerUpdateFn.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: let a removal fence expire after a minute
The master drops a broadcast when a client's channel is full, so the
re-add that would lift a fence can be lost while the gateway stays
connected. A mark held until that re-add then hid a live server, and a
volume whose only copy it held read as not found until the next master
reconnect. Bounding marks by a minute still outlasts a lookup in flight
when the removal landed; expiring early only falls back to learning by a
failed read, as before this change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
see https://blog.aqwari.net/xml-schema-go/
1. go get aqwari.net/xml/cmd/xsdgen
2. Add EncodingType element for ListBucketResult in AmazonS3.xsd
3. xsdgen -o s3api_xsd_generated.go -pkg s3api AmazonS3.xsd
4. Remove empty Grantee struct in s3api_xsd_generated.go
5. Remove xmlns: sed s'/http:\/\/s3.amazonaws.com\/doc\/2006-03-01\/\ //' s3api_xsd_generated.go
