* expose whether a volume replica is backed by remote storage
Volume locations returned by lookups do not indicate whether a replica
has been tiered to remote storage. Readers cannot distinguish a local
replica from a remote-backed one, so they may hit a remote-backed
replica first even when a local replica is available.
Add DataInRemote to the lookup location message, populate it from the
master's volume info, and carry it through the wdclient vid map so
clients can prefer local replicas when resolving chunk locations.
* wdclient: prefer local volume replicas over remote-tier replicas on lookup
LookupFileIdWithFallback (and the publicUrl variant in FilerClient)
didn't honor the DataInRemote flag when shuffling URLs, so the
DataInRemote patch only took effect in LookupVolumeServerUrl. Apply
the same ReorderToFront(localUrls) to sameDcUrls/otherDcUrls so
non-remote replicas stay at the front, matching the existing vidMap
convention.
* wdclient: propagate DataInRemote across tier transitions on existing replicas
When a volume is tiered to remote storage or a remote-backed replica is
restored locally, the cached DataInRemote on the same volume-server URL
stayed at its old value because two pieces of state never updated:
* master_grpc_server.go only split newVolumes and (already-tracked) volumes
into NewVids vs RemoteVids. ChangedVolumes went straight to NewVids, so
the broadcast announced the re-classified volume as a fresh arrival and
the client had no way to tell whether its existing cache was stale.
* vid_map.addLocationToMap early-returned when an entry already had the
same URL. A tier transition reports the same URL with DataInRemote
flipped, so the cached entry stayed at the old classification.
Wire both sides together: ChangedVolumes now go through the same IsRemote
split as newVolumes, and addLocationToMap replaces the existing entry in
place when the URL matches but DataInRemote has changed. The server
reference key only depends on URL/grpc port, so the refcount does not
move across the flip.
Adds vid_map_remote_transition_test.go covering the local->remote and
remote->local paths so the in-place update and the cache-key stability
are pinned by tests.
* wdclient: prefer local replicas across data-center boundaries
The previous local-first ordering hoisted local URLs to the front of each
data-center bucket separately, then concatenated same-DC before other-DC.
That meant a same-DC remote replica could still be tried before an
other-DC local replica even though the local one would answer cheaply.
Reorder once across the full candidate list: concatenate same-DC and
other-DC first, then ReorderToFront pulls every local replica to the very
front while preserving the DC preference inside each tier. Apply the same
ordering in all four lookup paths so the cached vidMap, the
LookupFileIdWithFallback provider path, FilerClient.GetLookupFileIdFunction
(PublicUrl-preferred variant), and the deprecated filer.LookupFn all agree:
- weed/wdclient/vid_map.go (LookupVolumeServerUrl)
- weed/wdclient/vidmap_client.go (LookupFileIdWithFallback)
- weed/wdclient/filer_client.go (LookupFileId)
- weed/filer/reader_at.go (LookupFn)
Strengthen the existing local-first tests: vidmap_client_localfirst_test
now asserts both endpoints are present (not just the local one is first),
and slice_test asserts an exact match instead of accepting two orderings.
Add TestLookupFileIdWithFallbackGlobalLocalFirst to pin the cross-DC
ordering invariant: any local replica (same or other DC) precedes every
remote-tier replica; within each tier DC1 precedes DC2.
Add docstrings to ToVolumeLocations, ReorderToFront, LookupVolumeServerUrl,
LookupFileId, GetVidLocations, GetLocations, LookupFileIdWithFallback, and
updateVidMap so the touched lookup paths are described in one place.
* topology: broadcast tier transitions on existing replicas
When a volume replica is tiered to remote storage or restored locally, the
wdclient's cached DataInRemote went stale: every connected client kept
preferring a remote-backed replica over a freshly restored local one, or
demoted a freshly tiered remote replica. The fix in commit 116982595 routed
ChangedVolumes to NewVids/RemoteVids on the master, but ApplyVolumeChanges
returned only fresh arrivals and previously servable replicas. An existing
replica whose IsRemote() classification flipped was neither, so it never
reached the broadcast loop and the wdclient never learned.
Make Disk.doAddOrUpdateVolume return a third signal -- tierTransition --
true exactly when an existing replica's IsRemote() flips. ApplyVolumeChanges
treats that as an arrival so the existing SendHeartbeat routing loop now
sees it. Add a master-side end-to-end test covering local->remote,
remote->local, no-op re-reports, and a mixed heartbeat that only announces
the tier transition.
Also add docstrings to LookupFileId, wdclientLocationsToPb, and
LookupVolume where the prior change touched their bodies.
* topology: broadcast tier transitions received through full reconciliation
The previous commit added tier-transition routing on the ChangedVolumes
delta path, but that is not the only way a re-tiered replica reaches the
master. After a digest mismatch the volume server resends a full Volumes
list, and SyncDataNodeRegistration applies the new IsRemote() classification
silently -- the changedVolumes return value was being thrown away. The
master therefore never broadcast NewVids/RemoteVids, and a wdclient connected
during the recovery kept the stale DataInRemote until it lost contact with
the master.
Surface the changed set through UpdateVolumes.changedVolumes (now covering
both ReadOnly flips and tier flips) and SyncDataNodeRegistration, then route
it through NewVids/RemoteVids in SendHeartbeat the same way the delta path
already does. Add an end-to-end test for the full reconciliation path.
* master: keep an EC volume's locations in the volume lookup
The nodes that answer for an EC volume hold shards, not a volume record,
so asking them for one fails. Dropping the location on that failure
emptied the result and turned every EC read through the master's HTTP
lookup and fid redirect into a 404.
Treat an absent volume record as a local read and keep the node in the
answer. The per-node conversion moves into topologyLocation so the EC
case is covered by a test.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
* wdclient: replace a tier-flipped location without writing under a reader
GetLocations hands back the entry's own slice and the caller walks it
after the read lock is dropped, which is why every other mutation here
builds a new slice. Writing the flipped replica into the array in place
raced LookupVolumeServerUrl, reported by -race.
Copy the slice, swap the one element, and publish it.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
* master: keep a remote volume on NewVids for older clients
Moving remote-tier volumes out of NewVids and into RemoteVids alone is a
wire break in the wrong direction. A master upgraded ahead of its filers
and mounts -- the usual order -- announces a tiered volume only on a
field the older client ignores, so the volume drops out of that client's
vid map entirely and reads for it fail.
Announce every volume on NewVids and repeat the remote-tier subset on
RemoteVids, so a new client still learns the tier and an old one keeps
the location. The routing moves into announceVolume, which the heartbeat
paths and their tests now share instead of each restating it.
On the client, RemoteVids no longer needs a second write per volume: the
tier is settled before anything is added.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
* topology: split the volume snapshot by tier without copying the records
ToVolumeLocations runs on every KeepConnected, so a filer or mount
connecting made the master allocate a full VolumeInfo per volume per node
just to read four bytes of id off each one. AppendVolumeIds exists to
avoid exactly that.
Extend it to fill the remote-tier list alongside the full one, and use it
again in the snapshot.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
* wdclient: keep the data-center preference ahead of the local-first ordering
Hoisting every local replica to the very front puts an other-DC local
read ahead of a same-DC remote one. When the remote tier sits in the same
region as the replicas -- the common arrangement -- that trades an
in-region GET for a WAN round trip and costs more than the remote read it
avoids.
Reorder inside each data-center bucket instead, so local still wins among
equals and the data-center preference still wins overall.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
* operation: pick the read replica from one list
The local-preferring lookup built a list of local URLs and then branched
on whether it was empty, duplicating the random pick. Fall back by
filling the same list with every replica instead.
Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW
---------
Co-authored-by: Bruce Zou <gift_secondst@msn.com>
Co-authored-by: bruce-zzz <bruce.zou@hhy-data.com>
* wdclient: age vid map entries by generation instead of chaining snapshots
The vid map kept its history as a linked list of past snapshots, trimmed
in place by storing nil into a node's cache pointer. That cost up to six
full copies of the volume-location map, a recursive walk taking a
different lock per level, and deletes that had to cascade through every
generation. It also had to special-case explicitly-empty entries, or
fallback would resurrect locations a newer snapshot had cleared.
Keep one map instead, and stamp each entry with the generation it was
learned in. resetVidMap bumps the generation and drops entries that were
not relearned within the retained window, which is the same retention
the chain provided: an entry survives DefaultVidMapCacheSize resets.
The first write of a generation replaces an entry rather than merging
into it, so a volume that moved answers with where it is now — the
property a fresh map per reset used to give for free. Entries are
copy-on-write, so locations handed to a caller are no longer shifted
underneath it by a concurrent delete.
The map is never swapped now, so the client-side lock and its stable /
current accessors go away with it.
* wdclient: make vid map entries immutable and drop them once emptied
Review follow-up. Updating an entry in place left the copy-on-write
guarantee resting on callers never holding the entry pointer; install a
new entry instead, so the rule is simply that a stored entry never
changes.
Deleting a volume's last location now drops the entry rather than
keeping an empty one, which a client that never resets would otherwise
hold for every volume it ever saw deleted. Lookups already treat an
empty entry as a miss, so nothing observable changes.
* wdclient: let the newest generation decide between regular and EC locations
GetLocations checked the regular map first whatever its generation, so a
volume that was EC encoded kept answering with the regular copies the
previous master knew until they expired — for as long as the retained
window, since nothing relearns a copy that no longer exists.
The snapshot chain did not have this problem: the newest map was
consulted first and only a volume it knew nothing about fell through to
older ones. Restore that by comparing generations, with regular copies
winning a tie, since a tie means one generation reported both.
resetVidMap trims the cache chain by storing nil into a node's cache
pointer once it ages past vidMapCacheSize. A lookup that missed in its
own map and only then loaded that pointer could find the link already
severed, reporting "not found" for a volume that stayed resolvable the
whole time.
Load the link first, while it is still guaranteed live. A published
vidMap's cache pointer only ever goes from its ancestor to nil, so
reading it earlier can never yield staler history.
fix(wdclient): prevent stale cache fallback for empty volume locations
## Problem
During Kubernetes pod restarts, volume servers temporarily disconnect and their
locations are removed from vidMap. The deleteLocation function leaves an empty
array [] in vid2Locations map instead of removing the key entirely.
GetLocations() was checking 'if found && len(locations) > 0', which would fail
for empty arrays and fall back to the cache chain, returning STALE locations
from before the restart. This caused S3 gateway to try connecting to old pod
IPs that no longer exist, resulting in connection timeouts and hanging registry
sync jobs.
Example timeline:
1. Volume pod at 10.131.1.28:8081 registers volumes 10,12
2. S3 gateway caches: vid2Locations[10] = [10.131.1.28:8081]
3. Pod restarts, gets new IP 10.131.1.65:8081
4. Master sends delete → vid2Locations[10] = [] (empty, but key exists)
5. BUG: GetLocations(10) sees found=true, len=0 → falls back to cache
6. Returns stale 10.131.1.28:8081 instead of waiting for new location
7. S3 requests timeout trying to reach unreachable old IP
## Solution
Distinguish between two cases:
- found=true, locations=[] : Volume explicitly has no locations (e.g. restart)
→ Return nil, false (no fallback to cache)
- found=false : Volume never seen in current map
→ Check cache (preserve cache benefits for unknown volumes)
An empty array explicitly means 'this volume currently has no locations',
which is semantically different from 'volume unknown'. Don't fall back to
stale cache for explicitly empty volumes.
## Testing
Added comprehensive tests:
- TestGetLocationsEmptyArrayNoFallback: Verifies empty arrays don't use cache
- TestGetLocationsUnknownVolumeUsesCache: Verifies unknown volumes still use cache
- All existing tests pass
## Impact
Fixes registry sync job hangs during SeaweedFS upgrades/restarts. S3 gateway
will now correctly wait for updated volume locations instead of using stale
cached IPs.
Related: OutSystems.SeaWeedfs Helm chart, vega cluster incident 2026-06-24
Ping previously dialled whatever host:port the caller asked for. Gate
each server's Ping handler on cluster membership: masters check the
topology, registered cluster nodes, and configured master peers; volume
servers only accept their seed/current masters; filers accept tracked
peer filers, the master-learned volume server set, and configured
masters.
Use address-indexed peer lookups to keep Ping target validation O(1):
- topology maintains a pb.ServerAddress -> *DataNode index alongside
the dc/rack/node tree, kept in sync from doLinkChildNode and
UnlinkChildNode plus the ip/port-rewrite branch in
GetOrCreateDataNode. GetTopology now returns nil on a detached
subtree instead of panicking, so the linkage hooks can no-op safely.
- vid_map tracks a refcount per volume-server address so
hasVolumeServer answers without scanning every vid location. The
add path skips empty-address entries the same way the delete path
already does, so a zero-value Location cannot leak a permanent
serverRefCount[""] bucket.
- masters reuse a cached master-address set from MasterClient instead
of walking the configured peer slice on every request.
- volume servers compare against a pre-built seed-master set and
protect currentMaster reads/writes with an RWMutex, fixing the
data race with the heartbeat goroutine. The seed slice is copied
on construction so external mutation cannot desync it from the
frozen lookup set.
- cluster.check drops the direct volume-to-volume sweep; volume
servers no longer carry a peer-volume list, and the note next to
the dropped probe is reworded to make clear that direct
volume-to-volume reachability is intentionally not validated by
this command.
Update the volume-server integration tests that drove Ping through the
new admission gate: success-path coverage now targets the master peer
(the only type a volume server tracks), and the unknown/unreachable
path asserts the InvalidArgument the gate now returns instead of the
old downstream dial error.
Mirror the same admission gate in the Rust volume server crate: a
seed-master HashSet built once at startup plus a tokio RwLock over the
heartbeat-tracked current master, both consulted in is_known_ping_target
on every Ping, with InvalidArgument returned for any target that isn't
a recognised master.
* chore(weed/wdclient): prune unused functions
* chore(weed/wdclient): prune test-only functions and associated tests
* chore(weed/wdclient): remove dead cursor field
The cursor field and its initialization are no longer used after
the removal of getLocationIndex.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fallback to check master
* clean up
* parsing
* refactor
* handle parse error
* return error
* avoid dup lookup
* use batch key
* dedup lookup logic
* address comments
* errors.Join(lookupErrors...)
* add a comment
* Fix: Critical data race in MasterClient vidMap
Fixes a critical data race where resetVidMap() was writing to the vidMap
pointer while other methods were reading it concurrently without synchronization.
Changes:
- Removed embedded *vidMap from MasterClient
- Added vidMapLock (sync.RWMutex) to protect vidMap pointer access
- Created safe accessor methods (GetLocations, GetDataCenter, etc.)
- Updated all direct vidMap accesses to use thread-safe methods
- Updated resetVidMap() to acquire write lock during pointer swap
The vidMap already has internal locking for its operations, but this fix
protects the vidMap pointer itself from concurrent read/write races.
Verified with: go test -race ./weed/wdclient/...
Impact:
- Prevents potential panics from concurrent pointer access
- No performance impact - uses RWMutex for read-heavy workloads
- Maintains backward compatibility through wrapper methods
* fmt
* Fix: Critical data race in MasterClient vidMap
Fixes a critical data race where resetVidMap() was writing to the vidMap
pointer while other methods were reading it concurrently without synchronization.
Changes:
- Removed embedded *vidMap from MasterClient struct
- Added vidMapLock (sync.RWMutex) to protect vidMap pointer access
- Created minimal public accessor methods for external packages:
* GetLocations, GetLocationsClone, GetVidLocations
* LookupFileId, LookupVolumeServerUrl
* GetDataCenter
- Internal code directly locks and accesses vidMap (no extra indirection)
- Updated resetVidMap() to acquire write lock during pointer swap
- Updated shell/commands.go to use GetDataCenter() method
Design Philosophy:
- vidMap already has internal locking for its map operations
- This fix specifically protects the vidMap *pointer* from concurrent access
- Public methods for external callers, direct locking for internal use
- Minimizes wrapper overhead while maintaining thread safety
Verified with: go test -race ./weed/wdclient/... (passes)
Impact:
- Prevents potential panics/crashes from data races
- Minimal performance impact (RWMutex for read-heavy workload)
- Maintains full backward compatibility
* fix more concurrent access
* reduce lock scope
* Optimize vidMap locking for better concurrency
Improved locking strategy based on the understanding that:
- vidMapLock protects the vidMap pointer from concurrent swaps
- vidMap has internal locks that protect its data structures
Changes:
1. Read operations: Grab pointer with RLock, release immediately, then operate
- Reduces lock hold time
- Allows resetVidMap to proceed sooner
- Methods: GetLocations, GetLocationsClone, GetVidLocations,
LookupVolumeServerUrl, GetDataCenter
2. Write operations: Changed from Lock() to RLock()
- RLock prevents pointer swap during operation
- Allows concurrent readers and other writers (serialized by vidMap's lock)
- Methods: addLocation, deleteLocation, addEcLocation, deleteEcLocation
Benefits:
- Significantly reduced lock contention
- Better concurrent performance under load
- Still prevents all race conditions
Verified with: go test -race ./weed/wdclient/... (passes)
* Further reduce lock contention in LookupVolumeIdsWithFallback
Optimized two loops that were holding RLock for extended periods:
Before:
- Held RLock during entire loop iteration
- Included string parsing and cache lookups
- Could block resetVidMap for significant time with large batches
After:
- Grab vidMap pointer with brief RLock
- Release lock immediately
- Perform all loop operations on local pointer
Impact:
- First loop: Cache check on initial volumeIds
- Second loop: Double-check after singleflight wait
Benefits:
- Minimal lock hold time (just pointer copy)
- resetVidMap no longer blocked by long loops
- Better concurrent performance with large volume ID lists
- Still thread-safe (vidMap methods have internal locks)
Verified with: go test -race ./weed/wdclient/... (passes)
* Add clarifying comments to vidMap helper functions
Added inline documentation to each helper function (addLocation, deleteLocation,
addEcLocation, deleteEcLocation) explaining the two-level locking strategy:
- RLock on vidMapLock prevents resetVidMap from swapping the pointer
- vidMap has internal locks that protect the actual map mutations
- This design provides optimal concurrency
The comments make it clear why RLock (not Lock) is correct and intentional,
preventing future confusion about the locking strategy.
* Improve encapsulation: Add shallowClone() method to vidMap
Added a shallowClone() method to vidMap to improve encapsulation and prevent
MasterClient from directly accessing vidMap's internal fields.
Changes:
1. Added vidMap.shallowClone() in vid_map.go
- Encapsulates the shallow copy logic within vidMap
- Makes vidMap responsible for its own state representation
- Documented that caller is responsible for thread safety
2. Simplified resetVidMap() in masterclient.go
- Uses tail := mc.vidMap.shallowClone() instead of manual field access
- Cleaner, more maintainable code
- Better adherence to encapsulation principles
Benefits:
- Improved code organization and maintainability
- vidMap internals are now properly encapsulated
- Easier to modify vidMap structure in the future
- More self-documenting code
Verified with: go test -race ./weed/wdclient/... (passes)
* Optimize locking: Reduce lock acquisitions and use helper methods
Two optimizations to further reduce lock contention and improve code consistency:
1. LookupFileIdWithFallback: Eliminated redundant lock acquisition
- Before: Two separate locks to get vidMap and dataCenter
- After: Single lock gets both values together
- Benefit: 50% reduction in lock/unlock overhead for this hot path
2. KeepConnected: Use GetDataCenter() helper for consistency
- Before: Manual lock/unlock to access DataCenter field
- After: Use existing GetDataCenter() helper method
- Benefit: Better encapsulation and code consistency
Impact:
- Reduced lock contention in high-traffic lookup path
- More consistent use of accessor methods throughout codebase
- Cleaner, more maintainable code
Verified with: go test -race ./weed/wdclient/... (passes)
* Refactor: Extract common locking patterns into helper methods
Eliminated code duplication by introducing two helper methods that encapsulate
the common locking patterns used throughout MasterClient:
1. getStableVidMap() - For read operations
- Acquires lock, gets pointer, releases immediately
- Returns stable snapshot for thread-safe reads
- Used by: GetLocations, GetLocationsClone, GetVidLocations,
LookupFileId, LookupVolumeServerUrl, GetDataCenter
2. withCurrentVidMap(f func(vm *vidMap)) - For write operations
- Holds RLock during callback execution
- Prevents pointer swap while allowing concurrent operations
- Used by: addLocation, deleteLocation, addEcLocation, deleteEcLocation
Benefits:
- Reduced code duplication (eliminated 48 lines of repetitive locking code)
- Centralized locking logic makes it easier to understand and maintain
- Self-documenting pattern through named helper methods
- Easier to modify locking strategy in the future (single point of change)
- Improved readability - accessor methods are now one-liners
Code size reduction: ~40% fewer lines for accessor/helper methods
Verified with: go test -race ./weed/wdclient/... (passes)
* consistent
* Fix cache pointer race condition with atomic.Pointer
Use atomic.Pointer for vidMap cache field to prevent data races
during cache trimming in resetVidMap. This addresses the race condition
where concurrent GetLocations calls could read the cache pointer while
resetVidMap is modifying it during cache chain trimming.
Changes:
- Changed cache field from *vidMap to atomic.Pointer[vidMap]
- Updated all cache access to use Load() and Store() atomic operations
- Updated shallowClone, GetLocations, deleteLocation, deleteEcLocation
- Updated resetVidMap to use atomic operations for cache trimming
* Merge: Resolve conflict in deleteEcLocation - keep atomic.Pointer and fix bug
Resolved merge conflict by combining:
1. Atomic pointer access pattern (from HEAD): cache.Load()
2. Correct method call (from fix): deleteEcLocation (not deleteLocation)
Resolution:
- Before (HEAD): cachedMap.deleteLocation() - WRONG, reintroduced bug
- Before (fix): vc.cache.deleteEcLocation() - RIGHT method, old pattern
- After (merged): cachedMap.deleteEcLocation() - RIGHT method, new pattern
This preserves both improvements:
✓ Thread-safe atomic.Pointer access pattern
✓ Correct recursive call to deleteEcLocation
Verified with: go test -race ./weed/wdclient/... (passes)
* Update vid_map.go
* remove shallow clone
* simplify
The vidMap structure is modified to a linked list structure (the length is limited to 5). When the vidMap is reset, the current vidMap is added to the new vidMap as a cache node. When the query locations is empty, the cache node is searched to avoid problems when the master switches leaders.
