fix(ec): group orphan-source completeness by encode generation (topology encode_ts_ns) (#9952)

* feat(ec): carry the encode generation through the topology heartbeat

Add encode_ts_ns (field 14) to VolumeEcShardInformationMessage and
populate it from each EC volume's .vif identity. The volume server emits
it on the full and incremental heartbeats; the master stores it on
EcVolumeInfo and re-emits it via GetTopologyInfo, so the admin/worker
layer can see which encode run produced each shard set. Field 14 avoids
the enterprise fork's reserved 10-13. Mirror the proto field and both
heartbeat emit sites in the Rust volume server.

* fix(ec): group orphan-source shard completeness by encode generation

countExistingEcShardsForVolume ORed EcIndexBits across every disk, so two
interrupted encode runs whose shard sets overlap unioned into a
false-complete set -- triggering the orphaned-source delete while no
single generation was actually complete. Group shards by encode_ts_ns and
return the largest single generation's count, so the trigger fires only
when one run holds the full set. Shards from pre-upgrade servers
(encode_ts_ns==0) form their own bucket.

The heartbeat carries one encode_ts_ns per (volume, disk), so this
separates generations on different disks; same-disk mixing is prevented
upstream by the pre-encode artifact wipe and the cross-run read guard.

* fix(ec): guard against a nil Ec shard info entry in the generation count

Defensive: a manually-constructed or corrupted topology could carry a nil
entry in EcShardInfos. Skip it rather than dereference.

* fix(ec): carry the encode generation on the EC shard unmount delta

The mount delta sets EncodeTsNs; the unmount deletion delta left it 0.
Populate it from the Ec volume before unloading so both incremental
deltas are consistent (the Rust volume server already does this via its
snapshot diff).
This commit is contained in:
Chris Lu
2026-06-14 00:14:12 -07:00
committed by GitHub
parent 26754fca4d
commit da243b9423
12 changed files with 169 additions and 22 deletions
+5 -1
View File
@@ -147,7 +147,11 @@ message VolumeEcShardInformationMessage {
repeated int64 shard_sizes = 7; // optimized: sizes for shards in order of set bits in ec_index_bits
uint64 file_count = 8; // total needles in the .ecx index (live + tombstoned)
uint64 delete_count = 9; // node-local tombstones in the .ecj deletion journal
// fields 10-19 reserved for future upstream open-source additions.
// encode-run identity (unix nanos) from the .vif EcShardConfig; lets the admin
// group shards by encode generation. Numbered 14 (not 10) to skip the
// enterprise fork's reserved 10-13.
int64 encode_ts_ns = 14;
// fields 15-19 reserved for future upstream open-source additions.
// fields 20+ are owned by the enterprise fork (e.g. data_shards/parity_shards)
// and must not be used here without coordination.
}
+1
View File
@@ -314,6 +314,7 @@ fn collect_ec_shard_delta_messages(
disk_type: ec_vol.disk_type.to_string(),
expire_at_sec: ec_vol.expire_at_sec,
disk_id: disk_id as u32,
encode_ts_ns: ec_vol.encode_ts_ns,
..Default::default()
},
);
@@ -448,6 +448,7 @@ impl EcVolume {
disk_id,
file_count,
delete_count,
encode_ts_ns: self.encode_ts_ns,
..Default::default()
}]
}