3 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	41d6c821ba	feat(topology): report empty disks (per-disk type + capacity in heartbeat) (#10166) ... * fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk DiskId 0 doubles as the first physical disk (Locations[0]) and the protobuf "unset" default. SplitByPhysicalDisk folded every DiskId-0 record onto the aggregate DiskId whenever that was non-zero, so on a multi-disk node the first disk's volumes merged into whichever disk held volumes[0]: the node reported one fewer disk, the sibling showed ~2x volumes, and per-disk max was smeared across the survivors. This surfaced as cluster.status and volume.list undercounting disks. Only treat 0 as unset when no record carries a non-zero DiskId; with a mix, 0 is a real disk and keeps its own entry. * fix(admin): resolve physical disk 0 in active-topology indexes rebuildIndexes re-derived each volume/EC record's physical disk id with the same "DiskId 0 means unset" heuristic SplitByPhysicalDisk used, so the two agreed only by sharing the bug. Now that SplitByPhysicalDisk keeps disk 0 distinct, the duplicated heuristic would fold disk-0 records onto a sibling while at.disks kept them on disk 0; GetVolumeLocations and GetECShardLocations then matched no record and silently dropped every volume and EC shard on the first disk, starving balance and EC tasks. Build the indexes from the same SplitByPhysicalDisk reconstruction that builds at.disks, so the keys always resolve. One source of truth instead of a parallel normalize. * fix(ec): allow physical disk 0 as preferred EC shard target pickBestDiskOnNode gated its result on bestDiskId != 0, but 0 is both a valid physical disk and the uint32 zero value, so a best-scoring disk 0 was discarded and the non-matching fallback returned instead. Gate on bestScore. * test(admin): cover EC-shard index resolution for physical disk 0 rebuildIndexes builds ecShardIndex the same way as volumeIndex; pin the EC path too so a shard on disk 0 keeps resolving via GetECShardLocations. * proto: per-disk type/capacity in DiskTag, DiskInfo.physical_disks DiskTag gains type + max_volume_count so the heartbeat can describe every physical disk, including ones holding no volumes or EC shards. DiskInfo gains physical_disks so the master can hand the full per-type disk set to per-physical-disk consumers. * feat(volume): report each physical disk's type and capacity CollectHeartbeat fills DiskTag.type and the per-disk effective max for every location, so the master can account for disks that hold no volumes or EC shards yet. Rust heartbeat mirrors it. * feat(master): surface empty disks in the per-physical-disk view The master records each disk's type and max from DiskTags and lists them on DiskInfo.physical_disks per type, including disks with no volumes or EC shards. SplitByPhysicalDisk enumerates that full set and gives each disk its exact max, so cluster.status, volume.list and the admin topology count and can target empty disks. Without physical_disks the even-split fallback is unchanged. * fix(master): clamp per-disk free at zero for over-allocated disks In the exact-max path FreeVolumeCount could go negative when a disk holds more volumes than its max; a negative would reduce the node's summed free and block placement on healthy disks. Clamp at 0. * fix(master): rebuild disk tags fresh each heartbeat DiskTags is the full authoritative per-disk list every heartbeat, so rebuild dn.diskTags from scratch like dn.diskBackends; merging left stale entries for removed disks. * fix(master): keep zero-capacity disks in physical_disks A disk reporting max 0 (an unavailable disk) is a valid physical disk, not a signal to drop it. List every disk of the type, but only emit physical_disks when the node reports real per-disk capacity, so an older server sending all zeros still falls back to the aggregate split. * test(volume): cover disk-space-low per-disk max in heartbeat Assert DiskTag.max_volume_count follows the used-slots override when a location is low on space, matching the per-type max_volume_counts. * chore: trim comments on the empty-disk change Drop narration; keep only the non-obvious why (disk-0 sentinel, exact-max free clamp, EC slots not subtracted, all-zeros fallback). * refactor(master): merge per-disk tags and capacity into one map diskTags and diskBackends were parallel maps keyed by the same DiskId and filled together from DiskTags. Fold them into one diskMetas map of {tags, type, max}. * refactor(proto): per-disk max as a map keyed by disk id physical_disks was a repeated {disk_id, max_volume_count} whose fields duplicated DiskInfo's own disk_id/max_volume_count. A map<uint32,int64> keyed by disk id expresses "max per disk" directly, drops the extra PhysicalDiskInfo message, and the consumer reads it as the disk set. * docs(proto): note DiskInfo.disk_id's two meanings Identity on a per-physical-disk DiskInfo (from SplitByPhysicalDisk), representative fallback on the type-keyed aggregate.	2026-06-30 18:45:44 -07:00
Chris Lu	a9c0ed91b5	fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk (#10161) ... * fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk DiskId 0 doubles as the first physical disk (Locations[0]) and the protobuf "unset" default. SplitByPhysicalDisk folded every DiskId-0 record onto the aggregate DiskId whenever that was non-zero, so on a multi-disk node the first disk's volumes merged into whichever disk held volumes[0]: the node reported one fewer disk, the sibling showed ~2x volumes, and per-disk max was smeared across the survivors. This surfaced as cluster.status and volume.list undercounting disks. Only treat 0 as unset when no record carries a non-zero DiskId; with a mix, 0 is a real disk and keeps its own entry. * fix(admin): resolve physical disk 0 in active-topology indexes rebuildIndexes re-derived each volume/EC record's physical disk id with the same "DiskId 0 means unset" heuristic SplitByPhysicalDisk used, so the two agreed only by sharing the bug. Now that SplitByPhysicalDisk keeps disk 0 distinct, the duplicated heuristic would fold disk-0 records onto a sibling while at.disks kept them on disk 0; GetVolumeLocations and GetECShardLocations then matched no record and silently dropped every volume and EC shard on the first disk, starving balance and EC tasks. Build the indexes from the same SplitByPhysicalDisk reconstruction that builds at.disks, so the keys always resolve. One source of truth instead of a parallel normalize. * fix(ec): allow physical disk 0 as preferred EC shard target pickBestDiskOnNode gated its result on bestDiskId != 0, but 0 is both a valid physical disk and the uint32 zero value, so a best-scoring disk 0 was discarded and the non-matching fallback returned instead. Gate on bestScore. * test(admin): cover EC-shard index resolution for physical disk 0 rebuildIndexes builds ecShardIndex the same way as volumeIndex; pin the EC path too so a shard on disk 0 keeps resolving via GetECShardLocations.	2026-06-30 15:35:27 -07:00
Chris Lu	41b6ad002b	fix(volume.list): show one entry per physical disk on multi-disk nodes (#9541) ... * fix(volume.list): show one entry per physical disk on multi-disk nodes DataNodeInfo.DiskInfos is keyed by disk type, so several same-type physical disks on one node collapse to a single map entry at the master. volume.list iterated that map directly and reported one "Disk hdd ... id:0" line per node, hiding the per-disk volume and shard layout. EC operators on multi-disk volume servers had no way to verify which physical disk a shard landed on. Lift the per-physical-disk split into a DiskInfo.SplitByPhysicalDisk() method on the proto type so consumers outside admin/topology can use it. Apply it in writeDataNodeInfo so the verbose Disk block shows one entry per physical disk, ordered by DiskId. Capacity counters are split evenly across reconstructed disks since the wire format doesn't carry per-disk capacity yet. This is a display-only change. ActiveTopology already did the split on its own and is now updated to call the shared helper. * fix(volume.list): preserve totals, count active/remote exactly, dedupe header Address review feedback on the per-physical-disk split: - share() truncated remainders so reconstructed per-disk counters could sum to less than the original aggregate (10 / 3 = 3+3+3). Distribute the remainder to the lowest disk ids so MaxVolumeCount and FreeVolumeCount sum exactly back to the node totals. - ActiveVolumeCount and RemoteVolumeCount are derivable per disk from the VolumeInfos already grouped by DiskId, so count them exactly (ReadOnly=false and RemoteStorageName!="" respectively) instead of approximating with an even split. - writeDataNodeInfo's per-disk callback fired the DataNode header on every iteration after the split, so a node with 6 physical disks emitted 6 DataNode headers. Guard the callback with headerPrinted so the header still appears at most once per node. - Sort split disks deterministically using explicit DiskId comparison to avoid int overflow risk on 32-bit systems. - Tighten the volume.list test substring to "id:N\n" so unrelated tokens like "ec volume id:101" don't accidentally match the id:1 needle, and assert the rack callback fires once.	2026-05-18 14:43:44 -07:00