master: export writable volumes per disk and reset leader-only layout gauges on followers (#12751)

* master: export writable volumes per disk for each volume layout

The layout gauges only report how many volumes of a collection take
writes, not where they sit, so two writable volumes sharing one disk
cannot be seen or alerted on. The leader now exports
writable_volumes_per_disk with the layout labels plus data center, rack,
node and disk id, counting the replicas of the layout's writable volumes
on each disk (state=writable) and those of them that are crowded
(state=crowded). Series for disks that no longer hold a writable volume of
the layout, and for layouts that are gone, are deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: reset the leader-only layout gauges on followers

The layout writable, crowded, under-replicated and per-disk gauges are only
set by the leader, so a master that lost leadership kept exporting the
values of its last term indefinitely, and dashboards that aggregate over
masters could show numbers days old. A master that is not the leader now
clears them on each periodic pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart disks holding several writable volumes of a collection

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: keep disks of different clusters apart in the per-disk panel

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chris LuandClaude Opus 5.5 authored and GitHub committed 2026-11-18 16:17:04 -08:00
1 parent 1fd5008d63
commit 7b4bdb77bb
6 files changed
+395 -1

No files matched your search

+107
View File
@@ -2302,6 +2302,113 @@
}
],
"pluginVersion": "10.3.1"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"title": "Writable Volumes per Disk",
"description": "Disks holding more than one writable volume of a collection, and the most writable volumes of a collection on one disk.",
"type": "timeseries",
"id": 262,
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 69
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
}
},
"overrides": []
},
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "count by (collection) (sum by (cluster, collection, data_center, rack, node, disk_id) (SeaweedFS_master_writable_volumes_per_disk{cluster=~\"$cluster\", state=\"writable\"}) > 1)",
"range": true,
"refId": "A",
"legendFormat": "{{collection}} disks with 2+"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "max by (collection) (sum by (cluster, collection, data_center, rack, node, disk_id) (SeaweedFS_master_writable_volumes_per_disk{cluster=~\"$cluster\", state=\"writable\"}))",
"range": true,
"refId": "B",
"legendFormat": "{{collection}} max on one disk"
}
],
"pluginVersion": "10.3.1"
}
]
},