Federated volume-id bound via heartbeat stamps (#11763) (#11819)

* pb: heartbeat fields for the federated volume-id bound (#11763)

Heartbeat carries max_volume_id_seen and last_seen_topology_id up, and
HeartbeatResponse echoes max_volume_id and topology_id down, so the
allocation bound is replicated to every volume server.

* master: echo the volume-id bound and fold reported bounds in (#11763)

Every heartbeat response carries the leader current max volume id and
topology id; every heartbeat reported max_volume_id_seen is folded into
the topology bound via UpAdjustMaxVolumeId.

* volume: persist a per-dir vid stamp of the highest id ever seen (#11763)

The store keeps max_volume_id_seen + last_seen_topology_id in a
.vid_stamp file per volume dir, fed by heartbeat echoes, AllocateVolume,
and local volume/EC creation, seeded at boot from stamps and local
holdings, and reported in every heartbeat. A rebuilt master recovers
the bound from the first surviving veteran report.

* volume-rust: mirror the vid stamp and heartbeat echo handling (#11763)

Same mechanism as the Go volume server: .vid_stamp per dir, bound folded
in from add_location scans and allocate/mount calls, echoed fields
applied on heartbeat responses, reported on every heartbeat.

* volume: serialize vid stamp writes and fsync the directory (#11763)

* volume-rust: serialize vid stamp writes and fsync the directory (#11763)

* master: echo the allocation bound to connected servers on change

The bound was only sent at connect time or on digest mismatch, so a
healthy stream never learned later increases. If the high-id server then
went offline and the master lost state, survivors restored a stale bound
and the id could be reissued. Resend the heartbeat response whenever the
bound or topology id moved.

* volume: retry failed vid stamp saves on later updates

A transient write failure left the in-memory bound raised but the stamp
stale, and repeat updates at the same bound never retried. Track a dirty
flag so any later update re-attempts the save.

* volume: cover stamp bootstrap for pre-stamp dirs in test

* volume-rust: retry failed vid stamp saves, mirroring Go

* volume-rust: stamp every dir with the combined bound on add_location

A dir whose own stamp sat below the combined bound kept its stale value
after fold; losing the higher-stamped disk then restarted the server
below ids it had already learned. Rewrite all stamps after each location
folds in.

* volume: stamp mounted volume and ec shard ids (#11763)

* volume-rust: stamp mounted volume ids, mirroring Go (#11763)

* volume: retry dirty vid stamp save on unchanged topology id (#11763)

* volume-rust: retry dirty vid stamp save on unchanged topology id (#11763)

* master: record echoed bound from the sent heartbeat response (#11763)
This commit is contained in:
Chris Lu authored and GitHub committed 2026-10-19 17:01:19 -07:00
1 parent 117ea58a83
commit 4feb30deff
13 files changed
+659 -33

No files matched your search

+10
View File
@@ -474,6 +474,14 @@ async fn do_heartbeat(
)
.into());
}
let echo_max_volume_id = hb_resp.max_volume_id;
let echo_topology_id = hb_resp.topology_id.clone();
off_runtime(config, state, move |_, state| {
let store = state.store.read().unwrap();
store.update_max_volume_id_seen(echo_max_volume_id);
store.update_last_seen_topology_id(&echo_topology_id);
})
.await?;
let options_changed =
volume_options_pass(config, state, &hb_resp).await?;
if options_changed && maybe_adjust_volume_max(config, state).await? {
@@ -1360,6 +1368,8 @@ fn build_heartbeat_with_ec_status(
grpc_port: config.grpc_port as u32,
location_uuids,
disk_tags,
max_volume_id_seen: store.max_volume_id_seen(),
last_seen_topology_id: store.last_seen_topology_id(),
..Default::default()
};
(heartbeat, volumes, actions)