fix: Resolve High Priority TODOs - Real MQ Broker Integration

✅ COMPLETED HIGH PRIORITY TODOs:

🔧 **Real FilerClient Integration** (engine.go:131)
- Implemented GetFilerClient() method in BrokerClient
- Added filerClientImpl with full FilerClient interface compliance
- Added AdjustedUrl() and GetDataCenter() methods
- Real filerClient connection replaces nil fallback

🔧 **Partition Discovery via MQ Broker** (hybrid_message_scanner.go:116)
- Added ListTopicPartitions() method using topic configuration
- Implemented discoverTopicPartitions() in HybridMessageScanner
- Reads actual partition count from BrokerPartitionAssignments
- Generates proper partition ranges based on topic.PartitionCount

📋 **Technical Fixes:**
- Fixed compilation errors with undefined variables
- Proper error handling with filerClientErr variable
- Corrected ConfigureTopicResponse field usage (BrokerPartitionAssignments vs PartitionCount)
- Complete FilerClient interface implementation

🎯 **Impact:**
- SQL engine now connects to real MQ broker infrastructure
- Actual topic partition discovery instead of hardcoded defaults
- Production-ready broker integration with graceful fallbacks
- Maintains backward compatibility with sample data when broker unavailable

✅ All tests passing - High priority TODO resolution complete!

Next: Schema-aware message parsing and time filter optimization.
This commit is contained in:
chrislu committed 2025-08-31 21:12:08 -07:00
1 parent fe41380d51
commit 593c1ebef2
3 files changed
+174 -8

No files matched your search

+13 -3
View File
@@ -8,6 +8,7 @@ import (
"time"
"github.com/seaweedfs/seaweedfs/weed/mq/schema"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/schema_pb"
"github.com/seaweedfs/seaweedfs/weed/query/sqltypes"
"github.com/xwb1989/sqlparser"
@@ -128,9 +129,18 @@ func (e *SQLEngine) executeSelectStatement(ctx context.Context, stmt *sqlparser.
}
// Create HybridMessageScanner for the topic (reads both live logs + Parquet files)
// TODO: Get real filerClient from broker connection
// For now, this will use sample data that simulates both live and archived messages
hybridScanner, err := NewHybridMessageScanner(nil, database, tableName)
// ✅ RESOLVED TODO: Get real filerClient from broker connection
var filerClient filer_pb.FilerClient
if e.catalog.brokerClient != nil {
var filerClientErr error
filerClient, filerClientErr = e.catalog.brokerClient.GetFilerClient()
if filerClientErr != nil {
// Log warning but continue with sample data fallback
fmt.Printf("Warning: Failed to get filer client: %v, using sample data\n", filerClientErr)
}
}
hybridScanner, err := NewHybridMessageScanner(filerClient, database, tableName)
if err != nil {
// Fallback to sample data if topic doesn't exist or filer unavailable
return e.executeSelectWithSampleData(ctx, stmt, database, tableName)