feat: add Prometheus metrics for replication operations (#10006)

* feat: add Prometheus metrics for replication operations

Adds 5 metrics to instrument volume server replication (write/delete):
- Operations counter with success/failure labels
- Duration histogram for latency tracking
- Targets gauge for replica fanout
- Failures counter with error reason labels
- Under-replicated volumes gauge on master

* fix: record replication duration histogram only when replicaCount > 0

* fix: update replication targets gauge for all operations including zero

* fix: ensure symmetric replication success/failure counting and proper metrics updates

* fix: change VolumeServerReplicationTargets from Gauge to Histogram

- Replace .Set() with .Observe() in store_replicate.go (2 occurrences)
- Update test to use CollectAndCount for histogram assertion
- Rename TestReplicationTargetsGauge -> TestReplicationTargetsHistogram
- Update documentation to reflect Histogram type and PromQL examples

* Add comments to replication metrics and improve test coverage

* metrics: add replication panels to grafana dashboard

Master row gets an under-replicated volumes timeseries; Volume Servers
row gets replication operations, failures-by-reason, p99 duration, and
average fan-out panels for the new replication metrics.

* metrics: name the replication duration histogram replication_seconds

Match the volumeServer convention (request_seconds, vacuuming_seconds)
rather than the admin/lifecycle _duration_seconds spelling.

* metrics: guard replication fan-out panel against divide-by-zero

clamp_min the _count rate so the avg-targets ratio reads 0 instead of
NaN when there are no replication events in the window.

---------

Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>

weed/server/master_grpc_server_volume.go

@@ -68,8 +68,10 @@ func (ms *MasterServer) ProcessGrowRequest() {
 				writable, crowded := vl.GetWritableVolumeCount()
 				mustGrow := int(lastGrowCount) - writable
 				vgr := vlc.ToVolumeGrowRequest()
+				underReplicated := vl.CountUnderReplicatedVolumes()
 				stats.MasterVolumeLayoutWritable.WithLabelValues(vlc.Collection, vgr.DiskType, vgr.Replication, vgr.Ttl).Set(float64(writable))
 				stats.MasterVolumeLayoutCrowded.WithLabelValues(vlc.Collection, vgr.DiskType, vgr.Replication, vgr.Ttl).Set(float64(crowded))
+				stats.MasterUnderReplicatedVolumes.WithLabelValues(vlc.Collection, vgr.DiskType, vgr.Replication, vgr.Ttl).Set(float64(underReplicated))
 
 				switch {
 				case mustGrow > 0: