fix(admin): list all masters and dedupe EC file counts in dashboard (#9093)

* fix(admin): list all masters and dedupe EC file counts in dashboard

Dashboard -> Master Nodes only ever showed the currently connected master
because getMasterNodesStatus hard-coded a single entry. Replace it with a
RaftListClusterServers call that returns every master in the raft group and
tags the real leader, falling back to the current master only if the raft
call fails.

Buckets -> Object Store Buckets could render 0 objects for a bucket backed
by an EC volume. Every shard holder reports the same whole-volume
file_count (read from the replicated .ecx), so the first-seen value wins;
if that first node had not yet finished loading .ecx it reported 0 and
pinned the aggregate at 0. Take the max across reporting nodes instead.

The dashboard header total_files also dropped after volumes were converted
to erasure coding because getTopologyViaGRPC never folded EC file_count
into topology.TotalFiles. Aggregate it with the same max/sum dedupe.

* fix(admin): address PR review comments

- bound RaftListClusterServers with a 3s timeout so the dashboard endpoint
  cannot hang on a stalled master
- pre-validate raft addresses with net.SplitHostPort before calling
  pb.GrpcAddressToServerAddress, which otherwise glog.Fatalf's on a
  malformed entry and would crash the admin process
- when raft is unreachable, mark the fallback master as not-leader rather
  than claiming leadership the code cannot verify
- warn when summed EC delete_count exceeds file_count while folding into
  topology.TotalFiles, matching collectCollectionStats

* fix(admin): distinguish empty raft response from RPC failure

When RaftListClusterServers returns successfully with no servers, raft is
not initialized (standalone/non-raft cluster), so the single fallback
master is the leader. Only treat the fallback as a non-leader when the
RPC actually failed.

* fix(admin): remove misleading Objects column from S3 buckets page

The bucket "Objects" column displayed needle counts from volume
collection stats, not actual S3 object counts. This is confusing
because a single S3 object can span multiple needles (multipart
uploads, versions) and the count is inaccurate for EC volumes.

Remove the ObjectCount field from S3Bucket, the Objects table column,
the sort-by-objects handler, the detail-view row, and both CSV export
references.

* fix(admin): correct cell indexes in fallback bucket CSV export

After the Objects column was removed, the fallback CSV exporter in
admin.js still used stale cell indexes: cells[1] mapped to Owner
(not Created), cells[2] to Created (not Size), cells[3] to Logical
Size (not Quota). Align all indexes with the current table column
order and include Owner, Logical Size, and Physical Size.
This commit is contained in:
Chris Lu authored and GitHub committed 2026-04-15 22:28:54 -07:00
1 parent dfecd664f9
commit 46b801aedb
40 files changed
+1100 -1025

No files matched your search

+46 -16
View File
@@ -2,6 +2,7 @@ package dash
import (
"context"
"net"
"net/http"
"sort"
"time"
@@ -216,36 +217,65 @@ func (s *AdminServer) ShowOverview(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, topology)
}
// getMasterNodesStatus checks status of all master nodes
// getMasterNodesStatus returns the full set of master nodes in the cluster.
// It prefers the authoritative raft membership (RaftListClusterServers) and
// falls back to the currently-connected master if the raft call fails, so the
// dashboard never shows an empty list.
func (s *AdminServer) getMasterNodesStatus() []MasterNode {
var masterNodes []MasterNode
masterMap := make(map[string]MasterNode)
raftCallSucceeded := false
// Since we have a single master address, create one entry
var isLeader bool = true // Assume leader since it's the only master we know about
// Try to get leader info from this master
err := s.WithMasterClient(func(client master_pb.SeaweedClient) error {
_, err := client.GetMasterConfiguration(context.Background(), &master_pb.GetMasterConfigurationRequest{})
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
resp, err := client.RaftListClusterServers(ctx, &master_pb.RaftListClusterServersRequest{})
if err != nil {
return err
}
// For now, assume this master is the leader since we can connect to it
isLeader = true
raftCallSucceeded = true
for _, server := range resp.ClusterServers {
// pb.GrpcAddressToServerAddress calls glog.Fatalf on a parse
// error, so pre-validate the raft address with net.SplitHostPort
// and skip malformed entries instead of taking the process down.
if _, _, splitErr := net.SplitHostPort(server.Address); splitErr != nil {
glog.Warningf("skip master with invalid raft address %q: %v", server.Address, splitErr)
continue
}
httpAddress := pb.GrpcAddressToServerAddress(server.Address)
masterMap[httpAddress] = MasterNode{
Address: httpAddress,
IsLeader: server.IsLeader,
}
}
return nil
})
if err != nil {
isLeader = false
currentMaster := s.masterClient.GetMaster(context.Background())
glog.Errorf("Failed to list raft cluster masters from %s: %v", currentMaster, err)
}
currentMaster := s.masterClient.GetMaster(context.Background())
if currentMaster != "" {
masterNodes = append(masterNodes, MasterNode{
Address: pb.ServerAddress(currentMaster).ToHttpAddress(),
IsLeader: isLeader,
})
if len(masterMap) == 0 {
currentMaster := s.masterClient.GetMaster(context.Background())
if currentMaster != "" {
addr := pb.ServerAddress(currentMaster).ToHttpAddress()
// A successful empty raft response means raft is not initialized
// (standalone/non-raft cluster); the only master IS the leader.
// A failed RPC means connectivity issue; do not claim leadership.
masterMap[addr] = MasterNode{
Address: addr,
IsLeader: raftCallSucceeded,
}
}
}
masterNodes := make([]MasterNode, 0, len(masterMap))
for _, m := range masterMap {
masterNodes = append(masterNodes, m)
}
sort.Slice(masterNodes, func(i, j int) bool {
return masterNodes[i].Address < masterNodes[j].Address
})
return masterNodes
}