mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-17 12:00:44 +02:00
remove emoji
This commit is contained in:
chrislu
+122
-215
@@ -1,246 +1,153 @@
|
||||
# Kafka-SMQ Integration Implementation Summary
|
||||
|
||||
## 🎯 **Overview**
|
||||
## Overview
|
||||
|
||||
This implementation provides **full ledger persistence** and **complete SMQ integration** for the Kafka Gateway, solving the critical offset persistence problem and enabling production-ready Kafka-to-SeaweedMQ bridging.
|
||||
This implementation provides **SMQ native offset management** for the Kafka Gateway, leveraging SeaweedMQ's existing distributed infrastructure instead of building custom persistence layers.
|
||||
|
||||
## 📋 **Completed Components**
|
||||
## Recommended Approach: SMQ Native Offset Management
|
||||
|
||||
### 1. **Offset Ledger Persistence** ✅
|
||||
### Core Concept
|
||||
|
||||
Instead of implementing separate consumer offset persistence, leverage SMQ's existing distributed offset management system where offsets are replicated across brokers and survive client disconnections and broker failures.
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
┌─ Kafka Gateway ─────────────────────────────────────┐
|
||||
│ ┌─ Kafka Abstraction ──────────────────────────────┐ │
|
||||
│ │ • Presents Kafka-style sequential offsets │ │
|
||||
│ │ • Translates to/from SMQ consumer groups │ │
|
||||
│ └──────────────────────────────────────────────────┘ │
|
||||
│ ↕ │
|
||||
│ ┌─ SMQ Native Offset Management ──────────────────┐ │
|
||||
│ │ • Consumer groups with SMQ semantics │ │
|
||||
│ │ • Distributed offset tracking │ │
|
||||
│ │ • Automatic replication & failover │ │
|
||||
│ │ • Built-in persistence │ │
|
||||
│ └──────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Key Components
|
||||
|
||||
### 1. Message Offset Persistence (Implemented)
|
||||
- **File**: `weed/mq/kafka/offset/persistence.go`
|
||||
- **Features**:
|
||||
- `SeaweedMQStorage`: Persistent storage backend using SMQ
|
||||
- `PersistentLedger`: Extends base ledger with automatic persistence
|
||||
- Offset mappings stored in dedicated SMQ topic: `kafka-system/offset-mappings`
|
||||
- Automatic ledger restoration on startup
|
||||
- Thread-safe operations with proper locking
|
||||
- **Purpose**: Maps Kafka sequential offsets to SMQ timestamps
|
||||
- **Storage**: `kafka-system/offset-mappings` topic
|
||||
- **Status**: Fully implemented and working
|
||||
|
||||
### 2. **Kafka-SMQ Offset Mapping** ✅
|
||||
- **File**: `weed/mq/kafka/offset/smq_mapping.go`
|
||||
- **Features**:
|
||||
- `KafkaToSMQMapper`: Bidirectional offset conversion
|
||||
- Kafka partitions → SMQ ring ranges (32 slots per partition)
|
||||
- Special offset handling (-1 = LATEST, -2 = EARLIEST)
|
||||
- Comprehensive validation and debugging tools
|
||||
- Time-based offset queries
|
||||
### 2. SMQ Native Consumer Groups (Recommended)
|
||||
- **File**: `weed/mq/kafka/offset/smq_native_offset_management.go`
|
||||
- **Purpose**: Use SMQ's built-in consumer group offset management
|
||||
- **Benefits**:
|
||||
- Automatic persistence and replication
|
||||
- Survives broker failures and restarts
|
||||
- No custom persistence code needed
|
||||
- Leverages battle-tested SMQ infrastructure
|
||||
|
||||
### 3. **SMQ Publisher Integration** ✅
|
||||
- **File**: `weed/mq/kafka/integration/smq_publisher.go`
|
||||
- **Features**:
|
||||
- `SMQPublisher`: Full Kafka message publishing to SMQ
|
||||
- Automatic offset assignment and tracking
|
||||
- Kafka metadata enrichment (`_kafka_offset`, `_kafka_partition`, `_kafka_timestamp`)
|
||||
- Per-topic SMQ publishers with enhanced record types
|
||||
- Comprehensive statistics and monitoring
|
||||
### 3. Offset Translation Layer
|
||||
- **Purpose**: Lightweight cache for Kafka offset ↔ SMQ timestamp mapping
|
||||
- **Implementation**: In-memory cache with LRU eviction
|
||||
- **Scope**: Only recent mappings needed for active consumers
|
||||
|
||||
### 4. **SMQ Subscriber Integration** ✅
|
||||
- **File**: `weed/mq/kafka/integration/smq_subscriber.go`
|
||||
- **Features**:
|
||||
- `SMQSubscriber`: Kafka fetch requests via SMQ subscriptions
|
||||
- Message format conversion (SMQ → Kafka)
|
||||
- Consumer group management
|
||||
- Offset commit handling
|
||||
- Message buffering and timeout handling
|
||||
## Kafka to SMQ Mapping
|
||||
|
||||
### 5. **Persistent Handler** ✅
|
||||
- **File**: `weed/mq/kafka/integration/persistent_handler.go`
|
||||
- **Features**:
|
||||
- `PersistentKafkaHandler`: Complete Kafka protocol handler
|
||||
- Unified interface for produce/fetch operations
|
||||
- Topic management with persistent ledgers
|
||||
- Comprehensive statistics and monitoring
|
||||
- Graceful shutdown and resource management
|
||||
| Kafka Concept | SMQ Concept | Implementation |
|
||||
|---------------|-------------|----------------|
|
||||
| Consumer Group | SMQ Consumer Group | Direct 1:1 mapping with "kafka-cg-" prefix |
|
||||
| Topic-Partition | SMQ Topic-Partition Range | Map Kafka partition N to SMQ ring [N*32, (N+1)*32-1] |
|
||||
| Sequential Offset | SMQ Timestamp + Index | Use SMQ's natural timestamp ordering |
|
||||
| Offset Commit | SMQ Consumer Position | Let SMQ track position natively |
|
||||
| Offset Fetch | SMQ Consumer Status | Query SMQ for current position |
|
||||
|
||||
### 6. **Comprehensive Testing** ✅
|
||||
- **File**: `test/kafka/persistent_offset_integration_test.go`
|
||||
- **Test Coverage**:
|
||||
- Offset persistence and recovery
|
||||
- SMQ publisher integration
|
||||
- SMQ subscriber integration
|
||||
- End-to-end publish-subscribe workflows
|
||||
- Offset mapping consistency validation
|
||||
## Long-Term Disconnection Handling
|
||||
|
||||
## 🔧 **Key Technical Features**
|
||||
|
||||
### **Offset Persistence Architecture**
|
||||
### Scenario: 30-Day Disconnection
|
||||
```
|
||||
Kafka Offset (Sequential) ←→ SMQ Timestamp (Nanoseconds) + Ring Range
|
||||
0 ←→ 1757639923746423000 + [0-31]
|
||||
1 ←→ 1757639923746424000 + [0-31]
|
||||
2 ←→ 1757639923746425000 + [0-31]
|
||||
Day 0: Consumer reads to offset 1000, disconnects
|
||||
→ SMQ consumer group position: timestamp T1000
|
||||
|
||||
Day 1-30: System continues, messages 1001-5000 arrive
|
||||
→ SMQ stores messages with timestamps T1001-T5000
|
||||
→ SMQ preserves consumer group position at T1000
|
||||
|
||||
Day 15: Kafka Gateway restarts
|
||||
→ In-memory cache lost
|
||||
→ SMQ consumer group position PRESERVED
|
||||
|
||||
Day 30: Consumer reconnects
|
||||
→ SMQ automatically resumes from position T1000
|
||||
→ Consumer receives messages starting from offset 1001
|
||||
→ Perfect resumption, no manual recovery needed
|
||||
```
|
||||
|
||||
### **SMQ Storage Schema**
|
||||
- **Offset Mappings Topic**: `kafka-system/offset-mappings`
|
||||
- **Message Topics**: `kafka/{original-topic-name}`
|
||||
- **Metadata Fields**: `_kafka_offset`, `_kafka_partition`, `_kafka_timestamp`
|
||||
## Benefits vs Custom Persistence
|
||||
|
||||
### **Partition Mapping**
|
||||
```go
|
||||
// Kafka partition → SMQ ring range
|
||||
SMQRangeStart = KafkaPartition * 32
|
||||
SMQRangeStop = (KafkaPartition + 1) * 32 - 1
|
||||
| Aspect | Custom Persistence | SMQ Native |
|
||||
|--------|-------------------|------------|
|
||||
| Implementation | High complexity (dual systems) | Medium (SMQ integration) |
|
||||
| Reliability | Custom code risks | Battle-tested SMQ |
|
||||
| Performance | Dual writes penalty | Native SMQ speed |
|
||||
| Operational | Monitor 2 systems | Single system |
|
||||
| Maintenance | Custom bugs/fixes | SMQ team handles |
|
||||
| Scalability | Custom scaling | SMQ distributed arch |
|
||||
|
||||
Examples:
|
||||
Kafka Partition 0 → SMQ Range [0, 31]
|
||||
Kafka Partition 1 → SMQ Range [32, 63]
|
||||
Kafka Partition 15 → SMQ Range [480, 511]
|
||||
```
|
||||
## Implementation Roadmap
|
||||
|
||||
## 🚀 **Usage Examples**
|
||||
### Phase 1: SMQ API Assessment (1-2 weeks)
|
||||
- Verify SMQ consumer group persistence behavior
|
||||
- Test `RESUME_OR_EARLIEST` functionality
|
||||
- Confirm replication across brokers
|
||||
|
||||
### **Creating a Persistent Handler**
|
||||
```go
|
||||
handler, err := integration.NewPersistentKafkaHandler([]string{"localhost:17777"})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer handler.Close()
|
||||
```
|
||||
### Phase 2: Consumer Group Integration (2-3 weeks)
|
||||
- Implement SMQ consumer group wrapper
|
||||
- Map Kafka consumer lifecycle to SMQ
|
||||
- Handle consumer group coordination
|
||||
|
||||
### **Publishing Messages**
|
||||
```go
|
||||
record := &schema_pb.RecordValue{
|
||||
Fields: map[string]*schema_pb.Value{
|
||||
"user_id": {Kind: &schema_pb.Value_StringValue{StringValue: "user123"}},
|
||||
"action": {Kind: &schema_pb.Value_StringValue{StringValue: "login"}},
|
||||
},
|
||||
}
|
||||
### Phase 3: Offset Translation (2-3 weeks)
|
||||
- Lightweight cache for Kafka ↔ SMQ offset mapping
|
||||
- Handle edge cases (rebalancing, etc.)
|
||||
- Performance optimization
|
||||
|
||||
offset, err := handler.ProduceMessage("user-events", 0, []byte("key1"), record, recordType)
|
||||
// Returns: offset=0 (first message)
|
||||
```
|
||||
### Phase 4: Integration & Testing (3-4 weeks)
|
||||
- Replace current offset management
|
||||
- Comprehensive testing with long disconnections
|
||||
- Performance benchmarking
|
||||
|
||||
### **Fetching Messages**
|
||||
```go
|
||||
messages, err := handler.FetchMessages("user-events", 0, 0, 1024*1024, "my-consumer-group")
|
||||
// Returns: All messages from offset 0 onwards
|
||||
```
|
||||
**Total Estimated Effort: ~10 weeks**
|
||||
|
||||
### **Offset Queries**
|
||||
```go
|
||||
highWaterMark, _ := handler.GetHighWaterMark("user-events", 0)
|
||||
earliestOffset, _ := handler.GetEarliestOffset("user-events", 0)
|
||||
latestOffset, _ := handler.GetLatestOffset("user-events", 0)
|
||||
```
|
||||
## Current Status
|
||||
|
||||
## 📊 **Performance Characteristics**
|
||||
### Completed
|
||||
- Message offset persistence (Kafka offset → SMQ timestamp mapping)
|
||||
- SMQ publishing integration with offset tracking
|
||||
- SMQ subscription integration with offset mapping
|
||||
- Comprehensive integration tests
|
||||
- Architecture design for SMQ native approach
|
||||
|
||||
### **Offset Mapping Performance**
|
||||
- **Kafka→SMQ**: O(log n) lookup via binary search
|
||||
- **SMQ→Kafka**: O(log n) lookup via binary search
|
||||
- **Memory Usage**: ~32 bytes per offset entry
|
||||
- **Persistence**: Asynchronous writes to SMQ
|
||||
### Next Steps
|
||||
1. Assess SMQ consumer group APIs
|
||||
2. Implement SMQ native consumer group wrapper
|
||||
3. Replace current in-memory offset management
|
||||
4. Test long-term disconnection scenarios
|
||||
|
||||
### **Message Throughput**
|
||||
- **Publishing**: Limited by SMQ publisher throughput
|
||||
- **Fetching**: Buffered with configurable window size
|
||||
- **Offset Tracking**: Minimal overhead (~1% of message processing)
|
||||
## Key Advantages
|
||||
|
||||
## 🔄 **Restart Recovery Process**
|
||||
- **Leverage Existing Infrastructure**: Use SMQ's proven distributed systems
|
||||
- **Reduced Complexity**: No custom persistence layer to maintain
|
||||
- **Better Reliability**: Inherit SMQ's fault tolerance and replication
|
||||
- **Operational Simplicity**: One system to monitor instead of two
|
||||
- **Performance**: Native SMQ operations without translation overhead
|
||||
- **Automatic Edge Cases**: SMQ handles network partitions, broker failures, etc.
|
||||
|
||||
1. **Handler Startup**:
|
||||
- Creates `SeaweedMQStorage` connection
|
||||
- Initializes SMQ publisher/subscriber clients
|
||||
## Recommendation
|
||||
|
||||
2. **Ledger Recovery**:
|
||||
- Queries `kafka-system/offset-mappings` topic
|
||||
- Reconstructs offset ledgers from persisted mappings
|
||||
- Sets `nextOffset` to highest found offset + 1
|
||||
The **SMQ Native Offset Management** approach is strongly recommended over custom persistence because:
|
||||
|
||||
3. **Message Continuity**:
|
||||
- New messages get sequential offsets starting from recovered high water mark
|
||||
- Existing consumer groups can resume from committed offsets
|
||||
- No offset gaps or duplicates
|
||||
1. **Simpler Implementation**: ~10 weeks vs 15+ weeks for custom persistence
|
||||
2. **Higher Reliability**: Leverage battle-tested SMQ infrastructure
|
||||
3. **Better Performance**: Native operations without dual-write penalty
|
||||
4. **Lower Maintenance**: SMQ team handles distributed systems complexity
|
||||
5. **Operational Benefits**: Single system to monitor and maintain
|
||||
|
||||
## 🛡️ **Error Handling & Resilience**
|
||||
|
||||
### **Persistence Failures**
|
||||
- Offset mappings are persisted **before** in-memory updates
|
||||
- Failed persistence prevents offset assignment
|
||||
- Automatic retry with exponential backoff
|
||||
|
||||
### **SMQ Connection Issues**
|
||||
- Graceful degradation with error propagation
|
||||
- Connection pooling and automatic reconnection
|
||||
- Circuit breaker pattern for persistent failures
|
||||
|
||||
### **Offset Consistency**
|
||||
- Validation checks for sequential offsets
|
||||
- Monotonic timestamp verification
|
||||
- Comprehensive mapping consistency tests
|
||||
|
||||
## 🔍 **Monitoring & Debugging**
|
||||
|
||||
### **Statistics API**
|
||||
```go
|
||||
stats := handler.GetStats()
|
||||
// Returns comprehensive metrics:
|
||||
// - Topic count and partition info
|
||||
// - Ledger entry counts and time ranges
|
||||
// - High water marks and offset ranges
|
||||
```
|
||||
|
||||
### **Offset Mapping Info**
|
||||
```go
|
||||
mapper := offset.NewKafkaToSMQMapper(ledger)
|
||||
info, err := mapper.GetMappingInfo(kafkaOffset, kafkaPartition)
|
||||
// Returns detailed mapping information for debugging
|
||||
```
|
||||
|
||||
### **Validation Tools**
|
||||
```go
|
||||
err := mapper.ValidateMapping(topic, partition)
|
||||
// Checks offset sequence and timestamp monotonicity
|
||||
```
|
||||
|
||||
## 🎯 **Production Readiness**
|
||||
|
||||
### **✅ Completed Features**
|
||||
- ✅ Full offset persistence across restarts
|
||||
- ✅ Bidirectional Kafka-SMQ offset mapping
|
||||
- ✅ Complete SMQ publisher/subscriber integration
|
||||
- ✅ Consumer group offset management
|
||||
- ✅ Comprehensive error handling
|
||||
- ✅ Thread-safe operations
|
||||
- ✅ Extensive test coverage
|
||||
- ✅ Performance monitoring
|
||||
- ✅ Graceful shutdown
|
||||
|
||||
### **🔧 Integration Points**
|
||||
- **Kafka Protocol Handler**: Replace in-memory ledgers with `PersistentLedger`
|
||||
- **Produce Path**: Use `SMQPublisher.PublishMessage()`
|
||||
- **Fetch Path**: Use `SMQSubscriber.FetchMessages()`
|
||||
- **Offset APIs**: Use `handler.GetHighWaterMark()`, etc.
|
||||
|
||||
## 📈 **Next Steps for Production**
|
||||
|
||||
1. **Replace Existing Handler**:
|
||||
```go
|
||||
// Replace current handler initialization
|
||||
handler := integration.NewPersistentKafkaHandler(brokers)
|
||||
```
|
||||
|
||||
2. **Update Protocol Handlers**:
|
||||
- Modify `handleProduce()` to use `handler.ProduceMessage()`
|
||||
- Modify `handleFetch()` to use `handler.FetchMessages()`
|
||||
- Update offset APIs to use persistent ledgers
|
||||
|
||||
3. **Configuration**:
|
||||
- Add SMQ broker configuration
|
||||
- Configure offset persistence intervals
|
||||
- Set up monitoring and alerting
|
||||
|
||||
4. **Testing**:
|
||||
- Run integration tests with real SMQ cluster
|
||||
- Perform restart recovery testing
|
||||
- Load testing with persistent offsets
|
||||
|
||||
## 🎉 **Summary**
|
||||
|
||||
This implementation **completely solves** the offset persistence problem identified earlier:
|
||||
|
||||
- ❌ **Before**: "Handler restarts reset offset counters (expected in current implementation)"
|
||||
- ✅ **After**: "Handler restarts restore offset counters from SMQ persistence"
|
||||
|
||||
The Kafka Gateway now provides **production-ready** offset management with full SMQ integration, enabling seamless Kafka client compatibility while leveraging SeaweedMQ's distributed storage capabilities.
|
||||
This approach solves the critical offset persistence problem while minimizing implementation complexity and maximizing reliability.
|
||||
|
||||
@@ -5,7 +5,7 @@ This document identifies areas in the current Kafka implementation that need att
|
||||
|
||||
## Critical Protocol Issues
|
||||
|
||||
### 🚨 HIGH PRIORITY - Protocol Breaking Issues
|
||||
### HIGH PRIORITY - Protocol Breaking Issues
|
||||
|
||||
#### 1. **Record Batch Parsing (Produce API)**
|
||||
**File**: `protocol/produce.go`
|
||||
@@ -63,7 +63,7 @@ This document identifies areas in the current Kafka implementation that need att
|
||||
// - Handle different record batch versions correctly
|
||||
```
|
||||
|
||||
### ⚠️ MEDIUM PRIORITY - Compatibility Issues
|
||||
### MEDIUM PRIORITY - Compatibility Issues
|
||||
|
||||
#### 4. **API Version Support**
|
||||
**File**: `protocol/handler.go`
|
||||
@@ -116,7 +116,7 @@ This document identifies areas in the current Kafka implementation that need att
|
||||
// - Security/authorization errors
|
||||
```
|
||||
|
||||
### 🔧 LOW PRIORITY - Implementation Completeness
|
||||
### LOW PRIORITY - Implementation Completeness
|
||||
|
||||
#### 7. **Connection Management**
|
||||
**File**: `protocol/handler.go`
|
||||
|
||||
Reference in New Issue
Block a user