Enable filer discovery: pass master addresses to FilerClient

Fix two critical issues:

1. **Filer Discovery Not Working**: Master client was not being passed to
   FilerClient, so peer discovery couldn't work

2. **Credential Store Design**: Already uses FilerClient via GetCurrentFiler
   function - this is the correct design for HA

Changes:

**Command (s3.go):**
- Read master addresses from GetFilerConfiguration response
- Pass masterAddresses to S3ApiServerOption
- Log master addresses for visibility

**S3ApiServerOption:**
- Add Masters []pb.ServerAddress field for discovery

**S3ApiServer:**
- Create MasterClient from Masters when available
- Pass MasterClient + FilerGroup to FilerClient via options
- Enable discovery with 5-minute refresh interval
- Log whether discovery is enabled or disabled

**Credential Store:**
- Already correctly uses filerClient.GetCurrentFiler via function
- This provides HA without tight coupling to FilerClient struct
- Function-based design is clean and thread-safe

Discovery Flow:
1. S3 command reads filer config → gets masters + filer group
2. S3ApiServer creates MasterClient from masters
3. FilerClient uses MasterClient to query for peer filers
4. Background goroutine refreshes peer list every 5 minutes
5. Credential store uses GetCurrentFiler to get active filer

Now filer discovery actually works! ��
This commit is contained in:
Chris Lu committed 2025-11-25 22:54:04 -08:00
1 parent 8517cebb31
commit af3a0770ff
2 files changed
+28 -2

No files matched your search

+21 -2
View File
@@ -34,6 +34,7 @@ import (
type S3ApiServerOption struct {
Filers []pb.ServerAddress
Masters []pb.ServerAddress // For filer discovery
Port int
Config string
DomainName string
@@ -100,8 +101,26 @@ func NewS3ApiServerWithStore(router *mux.Router, option *S3ApiServerOption, expl
// Initialize FilerClient for volume location caching
// Uses the battle-tested vidMap with filer-based lookups
// Supports multiple filer addresses with automatic failover for high availability
filerClient := wdclient.NewFilerClient(option.Filers, option.GrpcDialOption, option.DataCenter)
glog.V(0).Infof("S3 API initialized FilerClient with %d filer(s) for volume location caching", len(option.Filers))
var filerClient *wdclient.FilerClient
if len(option.Masters) > 0 && option.FilerGroup != "" {
// Enable filer discovery via master
masterMap := make(map[string]pb.ServerAddress)
for i, addr := range option.Masters {
masterMap[fmt.Sprintf("master%d", i)] = addr
}
masterClient := wdclient.NewMasterClient(option.GrpcDialOption, option.FilerGroup, "s3", "", "", "", *pb.NewServiceDiscoveryFromMap(masterMap))
filerClient = wdclient.NewFilerClient(option.Filers, option.GrpcDialOption, option.DataCenter, &wdclient.FilerClientOption{
MasterClient: masterClient,
FilerGroup: option.FilerGroup,
DiscoveryInterval: 5 * time.Minute,
})
glog.V(0).Infof("S3 API initialized FilerClient with %d filer(s) and discovery enabled (group: %s, masters: %v)",
len(option.Filers), option.FilerGroup, option.Masters)
} else {
filerClient = wdclient.NewFilerClient(option.Filers, option.GrpcDialOption, option.DataCenter)
glog.V(0).Infof("S3 API initialized FilerClient with %d filer(s) (no discovery)", len(option.Filers))
}
// Update credential store to use FilerClient's current filer for HA
if store := iam.credentialManager.GetStore(); store != nil {