* filer.sync: count received, processed, and failed events and export in-flight jobs
The metadata processor admits at most -concurrency jobs and blocks the
subscription stream past that, so the backlog lives in the source filer's
metadata log and cannot be counted here. What can be measured honestly:
events read off the stream, replication outcomes, and worker saturation.
in_flight_jobs pinned at the concurrency limit means the sync itself is
the bottleneck; near zero means it is caught up or starved by the source.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* filer.sync: export replication lag in seconds
Lag is now minus the freshest of the processed watermark and the last
idle heartbeat: the watermark stops at the last real event, so a quiet
caught-up stream would otherwise show phantom lag. A ticker drives the
gauge because the offset callback only fires while events flow and
freezes exactly when the workers are saturated. Until the first event
or heartbeat the gauge stays unset rather than reporting lag against a
zero or stale resume offset.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* filer.sync: track replicated data sizes alongside event counts
An event count hides that 32 in-flight jobs can be 32 KB or 300 GB. Byte
counters mirror the event counters, and in_flight_bytes pairs with
in_flight_jobs. An event's size is the chunk delta - new chunks the old
entry does not already have - so deletes, renames, and attribute-only
updates count zero and byte rates reflect data movement, not metadata
churn.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* grafana: chart the new filer.sync metrics
The lag panel reads lag_seconds directly instead of deriving it from
sync_offset, and the sync row gains event rate, throughput, and the
in-flight jobs and bytes gauges.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* filer.sync: a pinned failure keeps showing as lag
An idle heartbeat means the stream is consumed, not that every event
replicated. While a permanent failure pins the watermark, letting the
heartbeat advance lag_seconds or the sync_offset gauge would report a
caught-up stream with an unreplicated event in it, so both now ignore
heartbeats until a restart replays the failure.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
* filer.sync: in-flight gauges survive subscription retries
A subscription retry builds a new processor sharing the gauge children
while the old processor's jobs may still be draining, so setting the
gauge from either side's local count clobbers the other. Each job now
increments and decrements for itself, keeping the total truthful across
generations.
Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
Nothing displayed the worker metrics, and the panels that look like they
did are about something else: Workers Connected, Worker Slots and Worker
Events in the Admin / Maintenance row read SeaweedFS_admin_*, which the
older maintenance queue feeds. A cluster running plugin workers - Go or
Rust - reads zero there while they are connected and busy.
A Plugin Workers row graphs what the workers themselves publish: how many
are connected, jobs and their failures, detection and proposal rates, job
duration, slot usage, stream events, and what the Lance jobs reclaimed.
The panel worth having is Objects Seen vs Skipped, since a sweep with
nothing to do and a sweep that could read nothing report the same number
of proposals.
Also a commented scrape target in the sample Prometheus config. It is 9328
rather than 9327: the sample compose already gives 9327 to the S3 gateway,
so the port the worker's own usage text suggests collides with it on a
single host.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* Review comment removed unnecessary success and failure count
* fix: use Gather.Gather() with seeded counter for EC rebuild registration test
- Restore Gather.Gather() to verify MustRegister calls as requested in review
- Seed VolumeServerECRebuildCounter before gathering because CounterVec
only appears after at least one label value is observed
- Use correct fully-qualified metric names (SeaweedFS_volumeServer_*)
* fix: remove preflight checkEcVolumeStatus failure from ec_rebuild_total counter
ec_rebuild_total should only reflect actual rebuild execution failures
(from RebuildEcFiles / RebuildEcxFile), not scan/precheck failures in
the volume status loop. The error is still returned to the caller;
only the misleading counter increment was removed.
* Review comment removed unnecessary observe
* label EC rebuild duration histogram by result
Without a result label, fast failures pull down the success-latency
quantiles shown on the EC Rebuild Duration panel. Make the histogram a
HistogramVec keyed by result, record success/failure through one
recordEcRebuild helper, and split the Grafana quantiles by (le, result).
* reset EC rebuild metric vecs in registration test
The HistogramVec needs a child before Gather emits it, so the test must
observe once; reset both vecs in cleanup so that sample doesn't leak into
other tests.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* feat: add Prometheus metric for volume creation operations
Add VolumeServerVolumeCreationCounter metric to track volume creation
attempts by result (success/failure).
Changes:
- Add VolumeServerVolumeCreationCounter in weed/stats/metrics.go
- Instrument GrowByCountAndType in weed/topology/volume_growth.go
- Add unit tests in weed/stats/metrics_volume_creation_test.go
- Add Grafana dashboard panel for volume creation rate
This metric enables monitoring volume creation success/failure rates
in SeaweedFS clusters.
* fix: address CodeRabbit review - move failure counter before early return
- Move failure counter into loop else block before return statement
- Move success counter to after loop completion
- Strengthen test assertion from count < 1 to count != 2
- Add t.Cleanup() for test isolation in TestVolumeCreationCounterIncrement
* fix: count each volume create attempt and move metric to master subsystem
- topology runs on the master, so move volume_creation_total from the
volumeServer subsystem to master (SeaweedFS_master_volume_creation_total);
rename the var to MasterVolumeCreationCounter and group it with the other
master metrics.
- increment the counter per findAndGrow iteration instead of once per
GrowByCountAndType call: each logical volume is now counted, partial
successes before a failure are credited, and a targetCount==0 call no
longer records a spurious success.
- update the Grafana panel query and the unit tests; the registration test
now asserts via the shared Gather registry under the fully-qualified name.
* test: drop volume_creation metric tests
They only exercised the Prometheus client library (CounterVec increment and
registry collection), not any SeaweedFS behavior, so they carried maintenance
cost without verifying anything in this codebase.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* feat: add Prometheus metrics for replication operations
Adds 5 metrics to instrument volume server replication (write/delete):
- Operations counter with success/failure labels
- Duration histogram for latency tracking
- Targets gauge for replica fanout
- Failures counter with error reason labels
- Under-replicated volumes gauge on master
* fix: record replication duration histogram only when replicaCount > 0
* fix: update replication targets gauge for all operations including zero
* fix: ensure symmetric replication success/failure counting and proper metrics updates
* fix: change VolumeServerReplicationTargets from Gauge to Histogram
- Replace .Set() with .Observe() in store_replicate.go (2 occurrences)
- Update test to use CollectAndCount for histogram assertion
- Rename TestReplicationTargetsGauge -> TestReplicationTargetsHistogram
- Update documentation to reflect Histogram type and PromQL examples
* Add comments to replication metrics and improve test coverage
* metrics: add replication panels to grafana dashboard
Master row gets an under-replicated volumes timeseries; Volume Servers
row gets replication operations, failures-by-reason, p99 duration, and
average fan-out panels for the new replication metrics.
* metrics: name the replication duration histogram replication_seconds
Match the volumeServer convention (request_seconds, vacuuming_seconds)
rather than the admin/lifecycle _duration_seconds spelling.
* metrics: guard replication fan-out panel against divide-by-zero
clamp_min the _count rate so the avg-targets ratio reads 0 instead of
NaN when there are no replication events in the window.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* filer: keep metadata-subscription send gauge fresh on idle heartbeat
last_send_timestamp_of_subscribe only advanced when a real matching
metadata event was streamed to a subscriber. On a quiet path an idle but
perfectly healthy subscriber therefore looked increasingly stale, and the
dashboard panel rendered a large, misleading 'lag'.
An idle heartbeat is a send too, so advance the gauge when one is emitted.
Subscribers that opt into idle heartbeats (filer.sync) now report true
freshness; the rest still show time since the last real event.
Rename the dashboard panel 'Metadata Subscription Lag' ->
'Time Since Last Subscription Send' and clarify its description to match.
* filer: guard nil option when advancing heartbeat gauge
maybeSendIdleHeartbeat is unit-tested with a bare &FilerServer{} (nil
option), so dereferencing fs.option.Host for the sourceFiler label
panicked. Guard it: production always has option set; the test now gets
an empty sourceFiler label instead of a nil-pointer panic.
* metrics: add per-bucket S3 panels and volume slot utilization to dashboard
The S3 Gateway/Buckets rows sliced requests only by type, never by bucket,
though SeaweedFS_s3_request_total and the request/TTFB histograms all carry a
bucket label. Add a Bucket template variable and four per-bucket panels to the
S3 Buckets row (request rate, response codes, request p95, TTFB p95), plus a
Volume Slots Utilization panel (volumes/max_volumes) to the Volume Servers row.
* metrics: scope bucket variable to cluster and guard slot utilization divide-by-zero
Source the Bucket variable from SeaweedFS_s3_bucket_size_bytes (a gauge emitted
for every bucket, including idle ones) scoped to the selected cluster, so the
dropdown lists all buckets in the cluster rather than only those that received
requests. Wrap the volume slot utilization denominator in clamp_min(..., 1) to
avoid +Inf/NaN when max_volumes is briefly zero or absent.
* metrics: guard time()-based panels against unset (zero) gauges
Several dashboard panels compute "time() - <gauge>" (uptime, time-since-
last-scrub, lifecycle cursor lag, last daily walk). When the underlying
gauge is unset it reports 0, so the panel rendered ~56 years (time since
the Unix epoch). This is the common case for "Volume Server Uptime": the
Rust volume server doesn't set SeaweedFS_volumeServer_start_time_seconds,
and Go components expose it registered-but-zero.
Guard each such expression with "> 0" so unset/zero series drop out (the
panel shows no data) instead of rendering a nonsensical epoch value.
Affected panels: Master Uptime, Volume Server Uptime, Time Since Last
Scrub, Lifecycle Cursor Lag, Time Since Last Daily Walk.
* metrics: guard Filer Sync Offset Lag and Metadata Subscription Lag panels too
Extend the > 0 guard to the two remaining time()-<gauge> panels that
share the same unset/zero failure mode (filerSync sync offset and the
metadata subscribe last-send timestamp), so they don't render ~56-year
lag when the gauge is absent.
The bundled dashboard (other/metrics/grafana_seaweedfs.json) covered only
18 of the 84 metrics weed/stats exposes and was a legacy Grafana 8 export
(graph panels, schemaVersion 30). Rebuild it as a modern dashboard
(timeseries panels, schemaVersion 39) with 100% metric coverage, targeting
the direct-scrape model used by Prometheus / seaweed-up / Kubernetes.
- Full coverage of every weed/stats metric: master, volume server, filer,
filer store/sync, s3, s3 buckets, s3 lifecycle, admin/maintenance, build,
wdclient, upload errors, plus Go runtime/process per component.
- Organized into collapsible rows with an always-on Overview.
- Scrape label model: group by `instance`; generic go_*/process_* panels
use `job=~"seaweedfs-.*"` to separate components; an optional `cluster`
template variable (from SeaweedFS_build_info, defaults to All) supports
multi-cluster setups and is transparent when no cluster label is present.
- Same uid (nh02dOVnz) and title so it upgrades in place; drops the dead
"AWS monthly cost" panel.
This is also the single source of truth bundled by seaweed-up's
`cluster dashboard install`.
* feat(filer): record object size distribution histogram
Add SeaweedFS_filer_object_size_bytes, a histogram sampled when an
object is first created in the filer namespace, covering every write
protocol (S3, WebDAV, FUSE mount, direct HTTP). Buckets follow the
1KB/100KB/1MB/100MB/1GB ranges operators use to size collections.
Directories, overwrites, and metadata-only updates are not sampled, so
the bucket counts track the size distribution of distinct objects.
* feat(metrics): add filer object size distribution dashboard panels
Add a write-rate-by-size-range graph and a size-distribution bar gauge,
driven by SeaweedFS_filer_object_size_bytes, to the standalone and Helm
Grafana dashboards. Per-range subtractions are clamped at zero so
transient negative rate() samples do not render below the axis.
* feat: add S3 bucket size and object count metrics
Adds periodic collection of bucket size metrics:
- SeaweedFS_s3_bucket_size_bytes: logical size (deduplicated across replicas)
- SeaweedFS_s3_bucket_physical_size_bytes: physical size (including replicas)
- SeaweedFS_s3_bucket_object_count: object count (deduplicated)
Collection runs every 1 minute via background goroutine that queries
filer Statistics RPC for each bucket's collection.
Also adds Grafana dashboard panels for:
- S3 Bucket Size (logical vs physical)
- S3 Bucket Object Count
* address PR comments: fix bucket size metrics collection
1. Fix collectCollectionInfoFromMaster to use master VolumeList API
- Now properly queries master for topology info
- Uses WithMasterClient to get volume list from master
- Correctly calculates logical vs physical size based on replication
2. Return error when filerClient is nil to trigger fallback
- Changed from 'return nil, nil' to 'return nil, error'
- Ensures fallback to filer stats is properly triggered
3. Implement pagination in listBucketNames
- Added listBucketPageSize constant (1000)
- Uses StartFromFileName for pagination
- Continues fetching until fewer entries than limit returned
4. Handle NewReplicaPlacementFromByte error and prevent division by zero
- Check error return from NewReplicaPlacementFromByte
- Default to 1 copy if error occurs
- Add explicit check for copyCount == 0
* simplify bucket size metrics: remove filer fallback, align with quota enforcement
- Remove fallback to filer Statistics RPC
- Use only master topology for collection info (same as s3.bucket.quota.enforce)
- Updated comments to clarify this runs the same collection logic as quota enforcement
- Simplified code by removing collectBucketSizeFromFilerStats
* use s3a.option.Masters directly instead of querying filer
* address PR comments: fix dashboard overlaps and improve metrics collection
Grafana dashboard fixes:
- Fix overlapping panels 55 and 59 in grafana_seaweedfs.json (moved 59 to y=30)
- Fix grid collision in k8s dashboard (moved panel 72 to y=48)
- Aggregate bucket metrics with max() by (bucket) for multi-instance S3 gateways
Go code improvements:
- Add graceful shutdown support via context cancellation
- Use ticker instead of time.Sleep for better shutdown responsiveness
- Distinguish EOF from actual errors in stream handling
* improve bucket size metrics: multi-master failover and proper error handling
- Initial delay now respects context cancellation using select with time.After
- Use WithOneOfGrpcMasterClients for multi-master failover instead of hardcoding Masters[0]
- Properly propagate stream errors instead of just logging them (EOF vs real errors)
* improve bucket size metrics: distributed lock and volume ID deduplication
- Add distributed lock (LiveLock) so only one S3 instance collects metrics at a time
- Add IsLocked() method to LiveLock for checking lock status
- Fix deduplication: use volume ID tracking instead of dividing by copyCount
- Previous approach gave wrong results if replicas were missing
- Now tracks seen volume IDs and counts each volume only once
- Physical size still includes all replicas for accurate disk usage reporting
* rename lock to s3.leader
* simplify: remove StartBucketSizeMetricsCollection wrapper function
* fix data race: use atomic operations for LiveLock.isLocked field
- Change isLocked from bool to int32
- Use atomic.LoadInt32/StoreInt32 for all reads/writes
- Sync shared isLocked field in StartLongLivedLock goroutine
* add nil check for topology info to prevent panic
* fix bucket metrics: use Ticker for consistent intervals, fix pagination logic
- Use time.Ticker instead of time.After for consistent interval execution
- Fix pagination: count all entries (not just directories) for proper termination
- Update lastFileName for all entries to prevent pagination issues
* address PR comments: remove redundant atomic store, propagate context
- Remove redundant atomic.StoreInt32 in StartLongLivedLock (AttemptToLock already sets it)
- Propagate context through metrics collection for proper cancellation on shutdown
- collectAndUpdateBucketSizeMetrics now accepts ctx
- collectCollectionInfoFromMaster uses ctx for VolumeList RPC
- listBucketNames uses ctx for ListEntries RPC
* fix: add S3 bucket traffic sent metric tracking
The BucketTrafficSent() function was defined but never called, causing
the S3 Bucket Traffic Sent Grafana dashboard panel to not display data.
Added BucketTrafficSent() calls in the streaming functions:
- streamFromVolumeServers: for inline and chunked content
- streamFromVolumeServersWithSSE: for encrypted range and full object requests
The traffic received metric already worked because BucketTrafficReceived()
was properly called in putToFiler() for both regular and multipart uploads.
* feat: add S3 API Calls per Bucket panel to Grafana dashboards
Added a new panel showing API calls per bucket using the existing
SeaweedFS_s3_request_total metric aggregated by bucket.
Updated all Grafana dashboard files:
- other/metrics/grafana_seaweedfs.json
- other/metrics/grafana_seaweedfs_k8s.json
- other/metrics/grafana_seaweedfs_heartbeat.json
- k8s/charts/seaweedfs/dashboards/seaweedfs-grafana-dashboard.json
* address PR comments: use actual bytes written for traffic metrics
- Use actual bytes written from w.Write instead of expected size for inline content
- Add countingWriter wrapper to track actual bytes for chunked content streaming
- Update streamDecryptedRangeFromChunks to return actual bytes written for SSE
- Remove redundant nil check that caused linter warning
- Fix duplicate panel id 86 in grafana_seaweedfs.json (changed to 90)
- Fix overlapping panel positions in grafana_seaweedfs_k8s.json (rebalanced x positions)
* fix grafana k8s dashboard: rebalance S3 panels to avoid overlap
- Panel 86 (S3 API Calls per Bucket): w:6, x:0, y:15
- Panel 67 (S3 Request Duration 95th): w:6, x:6, y:15
- Panel 68 (S3 Request Duration 80th): w:6, x:12, y:15
- Panel 65 (S3 Request Duration 99th): w:6, x:18, y:15
All four S3 panels now fit in a single row (y:15) with width 6 each.
Filer row header at y:22 and subsequent panels remain correctly positioned.
* add input validation and clarify comments in adjustRangeForPart
- Add validation that partStartOffset <= partEndOffset at function start
- Add clarifying comments for suffix-range handling where clientEnd
temporarily holds the suffix length before being reassigned
* align pluginVersion for panel 86 to 10.3.1 in k8s dashboard
* track partial writes for accurate egress traffic accounting
- Change condition from 'err == nil' to 'written > 0' for inline content
- Move BucketTrafficSent before error check for chunked content streaming
- Track traffic even on partial SSE range writes
- Track traffic even on partial full SSE object copies
This ensures egress traffic is counted even when writes fail partway through,
providing more accurate bandwidth metrics.
* metrics: add Prometheus metrics for concurrent upload tracking
Add Prometheus metrics to monitor concurrent upload activity for both
filer and S3 servers. This provides visibility into the upload limiting
feature added in the previous PR.
New Metrics:
- SeaweedFS_filer_in_flight_upload_bytes: Current bytes being uploaded to filer
- SeaweedFS_filer_in_flight_upload_count: Current number of uploads to filer
- SeaweedFS_s3_in_flight_upload_bytes: Current bytes being uploaded to S3
- SeaweedFS_s3_in_flight_upload_count: Current number of uploads to S3
The metrics are updated atomically whenever uploads start or complete,
providing real-time visibility into upload concurrency levels.
This helps operators:
- Monitor upload concurrency in real-time
- Set appropriate limits based on actual usage patterns
- Detect potential bottlenecks or capacity issues
- Track the effectiveness of upload limiting configuration
* grafana: add dashboard panels for concurrent upload metrics
Add 4 new panels to the Grafana dashboard to visualize the concurrent
upload metrics added in this PR:
Filer Section:
- Filer Concurrent Uploads: Shows current number of concurrent uploads
- Filer Concurrent Upload Bytes: Shows current bytes being uploaded
S3 Gateway Section:
- S3 Concurrent Uploads: Shows current number of concurrent uploads
- S3 Concurrent Upload Bytes: Shows current bytes being uploaded
These panels help operators monitor upload concurrency in real-time and
tune the upload limiting configuration based on actual usage patterns.
* more efficient
Corrected the dashboard to use the new metrics
The metric seems to have changed from SeaweedFS_filer_ to SeaweedFS_filerStore_. This commit replaces the ref in the dashboard with the new series name.
In the current dashboard there is no way to know the status of
volume servers, as prometheus only accepts push information, if
any server volume goes offline, the last information sent will
be read continuously by grafana.
Added a graph that calculates the delta between the information
of the last push timestamp with the timestamp of 15 seconds ago,
if the timestamp has not been changed, server state will be 0,
if volume server sent any information the server state will be 1.
-This graph will have to be modified if the time interval of pushs
is modified, a suggestion for improvement is to send the interval
in seconds along with the metrics.
-Only local tests were run, maybe if the latency between pushgateway
and volume server is too high, the delta calculation could be
affected.
Chris Lu