2 Commits

Include renames

Author	SHA1	Message	Date
3 people	7620e96171	expose whether a volume replica is backed by remote storage, and prefer local replicas (#11105) ... * expose whether a volume replica is backed by remote storage Volume locations returned by lookups do not indicate whether a replica has been tiered to remote storage. Readers cannot distinguish a local replica from a remote-backed one, so they may hit a remote-backed replica first even when a local replica is available. Add DataInRemote to the lookup location message, populate it from the master's volume info, and carry it through the wdclient vid map so clients can prefer local replicas when resolving chunk locations. * wdclient: prefer local volume replicas over remote-tier replicas on lookup LookupFileIdWithFallback (and the publicUrl variant in FilerClient) didn't honor the DataInRemote flag when shuffling URLs, so the DataInRemote patch only took effect in LookupVolumeServerUrl. Apply the same ReorderToFront(localUrls) to sameDcUrls/otherDcUrls so non-remote replicas stay at the front, matching the existing vidMap convention. * wdclient: propagate DataInRemote across tier transitions on existing replicas When a volume is tiered to remote storage or a remote-backed replica is restored locally, the cached DataInRemote on the same volume-server URL stayed at its old value because two pieces of state never updated: * master_grpc_server.go only split newVolumes and (already-tracked) volumes into NewVids vs RemoteVids. ChangedVolumes went straight to NewVids, so the broadcast announced the re-classified volume as a fresh arrival and the client had no way to tell whether its existing cache was stale. * vid_map.addLocationToMap early-returned when an entry already had the same URL. A tier transition reports the same URL with DataInRemote flipped, so the cached entry stayed at the old classification. Wire both sides together: ChangedVolumes now go through the same IsRemote split as newVolumes, and addLocationToMap replaces the existing entry in place when the URL matches but DataInRemote has changed. The server reference key only depends on URL/grpc port, so the refcount does not move across the flip. Adds vid_map_remote_transition_test.go covering the local->remote and remote->local paths so the in-place update and the cache-key stability are pinned by tests. * wdclient: prefer local replicas across data-center boundaries The previous local-first ordering hoisted local URLs to the front of each data-center bucket separately, then concatenated same-DC before other-DC. That meant a same-DC remote replica could still be tried before an other-DC local replica even though the local one would answer cheaply. Reorder once across the full candidate list: concatenate same-DC and other-DC first, then ReorderToFront pulls every local replica to the very front while preserving the DC preference inside each tier. Apply the same ordering in all four lookup paths so the cached vidMap, the LookupFileIdWithFallback provider path, FilerClient.GetLookupFileIdFunction (PublicUrl-preferred variant), and the deprecated filer.LookupFn all agree: - weed/wdclient/vid_map.go (LookupVolumeServerUrl) - weed/wdclient/vidmap_client.go (LookupFileIdWithFallback) - weed/wdclient/filer_client.go (LookupFileId) - weed/filer/reader_at.go (LookupFn) Strengthen the existing local-first tests: vidmap_client_localfirst_test now asserts both endpoints are present (not just the local one is first), and slice_test asserts an exact match instead of accepting two orderings. Add TestLookupFileIdWithFallbackGlobalLocalFirst to pin the cross-DC ordering invariant: any local replica (same or other DC) precedes every remote-tier replica; within each tier DC1 precedes DC2. Add docstrings to ToVolumeLocations, ReorderToFront, LookupVolumeServerUrl, LookupFileId, GetVidLocations, GetLocations, LookupFileIdWithFallback, and updateVidMap so the touched lookup paths are described in one place. * topology: broadcast tier transitions on existing replicas When a volume replica is tiered to remote storage or restored locally, the wdclient's cached DataInRemote went stale: every connected client kept preferring a remote-backed replica over a freshly restored local one, or demoted a freshly tiered remote replica. The fix in commit 116982595 routed ChangedVolumes to NewVids/RemoteVids on the master, but ApplyVolumeChanges returned only fresh arrivals and previously servable replicas. An existing replica whose IsRemote() classification flipped was neither, so it never reached the broadcast loop and the wdclient never learned. Make Disk.doAddOrUpdateVolume return a third signal -- tierTransition -- true exactly when an existing replica's IsRemote() flips. ApplyVolumeChanges treats that as an arrival so the existing SendHeartbeat routing loop now sees it. Add a master-side end-to-end test covering local->remote, remote->local, no-op re-reports, and a mixed heartbeat that only announces the tier transition. Also add docstrings to LookupFileId, wdclientLocationsToPb, and LookupVolume where the prior change touched their bodies. * topology: broadcast tier transitions received through full reconciliation The previous commit added tier-transition routing on the ChangedVolumes delta path, but that is not the only way a re-tiered replica reaches the master. After a digest mismatch the volume server resends a full Volumes list, and SyncDataNodeRegistration applies the new IsRemote() classification silently -- the changedVolumes return value was being thrown away. The master therefore never broadcast NewVids/RemoteVids, and a wdclient connected during the recovery kept the stale DataInRemote until it lost contact with the master. Surface the changed set through UpdateVolumes.changedVolumes (now covering both ReadOnly flips and tier flips) and SyncDataNodeRegistration, then route it through NewVids/RemoteVids in SendHeartbeat the same way the delta path already does. Add an end-to-end test for the full reconciliation path. * master: keep an EC volume's locations in the volume lookup The nodes that answer for an EC volume hold shards, not a volume record, so asking them for one fails. Dropping the location on that failure emptied the result and turned every EC read through the master's HTTP lookup and fid redirect into a 404. Treat an absent volume record as a local read and keep the node in the answer. The per-node conversion moves into topologyLocation so the EC case is covered by a test. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW * wdclient: replace a tier-flipped location without writing under a reader GetLocations hands back the entry's own slice and the caller walks it after the read lock is dropped, which is why every other mutation here builds a new slice. Writing the flipped replica into the array in place raced LookupVolumeServerUrl, reported by -race. Copy the slice, swap the one element, and publish it. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW * master: keep a remote volume on NewVids for older clients Moving remote-tier volumes out of NewVids and into RemoteVids alone is a wire break in the wrong direction. A master upgraded ahead of its filers and mounts -- the usual order -- announces a tiered volume only on a field the older client ignores, so the volume drops out of that client's vid map entirely and reads for it fail. Announce every volume on NewVids and repeat the remote-tier subset on RemoteVids, so a new client still learns the tier and an old one keeps the location. The routing moves into announceVolume, which the heartbeat paths and their tests now share instead of each restating it. On the client, RemoteVids no longer needs a second write per volume: the tier is settled before anything is added. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW * topology: split the volume snapshot by tier without copying the records ToVolumeLocations runs on every KeepConnected, so a filer or mount connecting made the master allocate a full VolumeInfo per volume per node just to read four bytes of id off each one. AppendVolumeIds exists to avoid exactly that. Extend it to fill the remote-tier list alongside the full one, and use it again in the snapshot. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW * wdclient: keep the data-center preference ahead of the local-first ordering Hoisting every local replica to the very front puts an other-DC local read ahead of a same-DC remote one. When the remote tier sits in the same region as the replicas -- the common arrangement -- that trades an in-region GET for a WAN round trip and costs more than the remote read it avoids. Reorder inside each data-center bucket instead, so local still wins among equals and the data-center preference still wins overall. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW * operation: pick the read replica from one list The local-preferring lookup built a list of local URLs and then branched on whether it was empty, duplicating the random pick. Fall back by filling the same list with every replica instead. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW --------- Co-authored-by: Bruce Zou <gift_secondst@msn.com> Co-authored-by: bruce-zzz <bruce.zou@hhy-data.com>	2026-09-02 16:12:46 -07:00
Bruce Zou	1dd292fb84	batch drain delta heartbeat messages (#9914)	2026-06-10 13:33:45 -07:00