mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 15:41:15 +02:00
fix(ec): group orphan-source completeness by encode generation (topology encode_ts_ns) (#9952)
* feat(ec): carry the encode generation through the topology heartbeat Add encode_ts_ns (field 14) to VolumeEcShardInformationMessage and populate it from each EC volume's .vif identity. The volume server emits it on the full and incremental heartbeats; the master stores it on EcVolumeInfo and re-emits it via GetTopologyInfo, so the admin/worker layer can see which encode run produced each shard set. Field 14 avoids the enterprise fork's reserved 10-13. Mirror the proto field and both heartbeat emit sites in the Rust volume server. * fix(ec): group orphan-source shard completeness by encode generation countExistingEcShardsForVolume ORed EcIndexBits across every disk, so two interrupted encode runs whose shard sets overlap unioned into a false-complete set -- triggering the orphaned-source delete while no single generation was actually complete. Group shards by encode_ts_ns and return the largest single generation's count, so the trigger fires only when one run holds the full set. Shards from pre-upgrade servers (encode_ts_ns==0) form their own bucket. The heartbeat carries one encode_ts_ns per (volume, disk), so this separates generations on different disks; same-disk mixing is prevented upstream by the pre-encode artifact wipe and the cross-run read guard. * fix(ec): guard against a nil Ec shard info entry in the generation count Defensive: a manually-constructed or corrupted topology could carry a nil entry in EcShardInfos. Skip it rather than dereference. * fix(ec): carry the encode generation on the EC shard unmount delta The mount delta sets EncodeTsNs; the unmount deletion delta left it 0. Populate it from the Ec volume before unloading so both incremental deltas are consistent (the Rust volume server already does this via its snapshot diff).
This commit is contained in:
@@ -147,7 +147,11 @@ message VolumeEcShardInformationMessage {
|
||||
repeated int64 shard_sizes = 7; // optimized: sizes for shards in order of set bits in ec_index_bits
|
||||
uint64 file_count = 8; // total needles in the .ecx index (live + tombstoned)
|
||||
uint64 delete_count = 9; // node-local tombstones in the .ecj deletion journal
|
||||
// fields 10-19 reserved for future upstream open-source additions.
|
||||
// encode-run identity (unix nanos) from the .vif EcShardConfig; lets the admin
|
||||
// group shards by encode generation. Numbered 14 (not 10) to skip the
|
||||
// enterprise fork's reserved 10-13.
|
||||
int64 encode_ts_ns = 14;
|
||||
// fields 15-19 reserved for future upstream open-source additions.
|
||||
// fields 20+ are owned by the enterprise fork (e.g. data_shards/parity_shards)
|
||||
// and must not be used here without coordination.
|
||||
}
|
||||
|
||||
@@ -314,6 +314,7 @@ fn collect_ec_shard_delta_messages(
|
||||
disk_type: ec_vol.disk_type.to_string(),
|
||||
expire_at_sec: ec_vol.expire_at_sec,
|
||||
disk_id: disk_id as u32,
|
||||
encode_ts_ns: ec_vol.encode_ts_ns,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
|
||||
@@ -448,6 +448,7 @@ impl EcVolume {
|
||||
disk_id,
|
||||
file_count,
|
||||
delete_count,
|
||||
encode_ts_ns: self.encode_ts_ns,
|
||||
..Default::default()
|
||||
}]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user