* Review comment removed unnecessary success and failure count
* fix: use Gather.Gather() with seeded counter for EC rebuild registration test
- Restore Gather.Gather() to verify MustRegister calls as requested in review
- Seed VolumeServerECRebuildCounter before gathering because CounterVec
only appears after at least one label value is observed
- Use correct fully-qualified metric names (SeaweedFS_volumeServer_*)
* fix: remove preflight checkEcVolumeStatus failure from ec_rebuild_total counter
ec_rebuild_total should only reflect actual rebuild execution failures
(from RebuildEcFiles / RebuildEcxFile), not scan/precheck failures in
the volume status loop. The error is still returned to the caller;
only the misleading counter increment was removed.
* Review comment removed unnecessary observe
* label EC rebuild duration histogram by result
Without a result label, fast failures pull down the success-latency
quantiles shown on the EC Rebuild Duration panel. Make the histogram a
HistogramVec keyed by result, record success/failure through one
recordEcRebuild helper, and split the Grafana quantiles by (le, result).
* reset EC rebuild metric vecs in registration test
The HistogramVec needs a child before Gather emits it, so the test must
observe once; reset both vecs in cleanup so that sample doesn't leak into
other tests.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* feat: add Prometheus metric for volume creation operations
Add VolumeServerVolumeCreationCounter metric to track volume creation
attempts by result (success/failure).
Changes:
- Add VolumeServerVolumeCreationCounter in weed/stats/metrics.go
- Instrument GrowByCountAndType in weed/topology/volume_growth.go
- Add unit tests in weed/stats/metrics_volume_creation_test.go
- Add Grafana dashboard panel for volume creation rate
This metric enables monitoring volume creation success/failure rates
in SeaweedFS clusters.
* fix: address CodeRabbit review - move failure counter before early return
- Move failure counter into loop else block before return statement
- Move success counter to after loop completion
- Strengthen test assertion from count < 1 to count != 2
- Add t.Cleanup() for test isolation in TestVolumeCreationCounterIncrement
* fix: count each volume create attempt and move metric to master subsystem
- topology runs on the master, so move volume_creation_total from the
volumeServer subsystem to master (SeaweedFS_master_volume_creation_total);
rename the var to MasterVolumeCreationCounter and group it with the other
master metrics.
- increment the counter per findAndGrow iteration instead of once per
GrowByCountAndType call: each logical volume is now counted, partial
successes before a failure are credited, and a targetCount==0 call no
longer records a spurious success.
- update the Grafana panel query and the unit tests; the registration test
now asserts via the shared Gather registry under the fully-qualified name.
* test: drop volume_creation metric tests
They only exercised the Prometheus client library (CounterVec increment and
registry collection), not any SeaweedFS behavior, so they carried maintenance
cost without verifying anything in this codebase.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* feat: add Prometheus metrics for replication operations
Adds 5 metrics to instrument volume server replication (write/delete):
- Operations counter with success/failure labels
- Duration histogram for latency tracking
- Targets gauge for replica fanout
- Failures counter with error reason labels
- Under-replicated volumes gauge on master
* fix: record replication duration histogram only when replicaCount > 0
* fix: update replication targets gauge for all operations including zero
* fix: ensure symmetric replication success/failure counting and proper metrics updates
* fix: change VolumeServerReplicationTargets from Gauge to Histogram
- Replace .Set() with .Observe() in store_replicate.go (2 occurrences)
- Update test to use CollectAndCount for histogram assertion
- Rename TestReplicationTargetsGauge -> TestReplicationTargetsHistogram
- Update documentation to reflect Histogram type and PromQL examples
* Add comments to replication metrics and improve test coverage
* metrics: add replication panels to grafana dashboard
Master row gets an under-replicated volumes timeseries; Volume Servers
row gets replication operations, failures-by-reason, p99 duration, and
average fan-out panels for the new replication metrics.
* metrics: name the replication duration histogram replication_seconds
Match the volumeServer convention (request_seconds, vacuuming_seconds)
rather than the admin/lifecycle _duration_seconds spelling.
* metrics: guard replication fan-out panel against divide-by-zero
clamp_min the _count rate so the avg-targets ratio reads 0 instead of
NaN when there are no replication events in the window.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix: return immediately on first error in DistributedOperation
* simplify DistributedOperation fail-fast to a single buffered channel
Drop the separate errCh: the collector now fails fast on the first error
it reads off the buffered resultCh and returns ret.Error(), so the early
return carries the same [host]: err annotation as the aggregated path and
there is no select race between two channels.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
