mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-20 13:30:46 +02:00
chrislu
2ffdda2661
This commit adds explicit offset commit in the ConsumerGroupHandler.Cleanup() method, which is called during consumer group rebalancing. This ensures all marked offsets are committed BEFORE partitions are reassigned to other consumers, significantly reducing duplicate message consumption during rebalancing. Problem: - Cleanup() was not committing offsets before rebalancing - When partition reassigned to another consumer, it started from last committed offset - Uncommitted messages (processed but not yet committed) were read again by new consumer - This caused ~100-200% duplicate messages during rebalancing in tests Solution: - Add session.Commit() in Cleanup() method - This runs after all ConsumeClaim goroutines have exited - Ensures all MarkMessage() calls are committed before partition release - New consumer starts from the last processed offset, not an older committed offset Benefits: - Dramatically reduces duplicate messages during rebalancing - Improves at-least-once semantics (closer to exactly-once for normal cases) - Better performance (less redundant processing) - Cleaner test results (expected duplicates only from actual failures) Kafka Rebalancing Lifecycle: 1. Rebalance triggered (consumer join/leave, timeout, etc.) 2. All ConsumeClaim goroutines cancelled 3. Cleanup() called ← WE COMMIT HERE NOW 4. Partitions reassigned to other consumers 5. New consumer starts from last committed offset ← NOW MORE UP-TO-DATE Expected Results: - Before: ~100-200% duplicates during rebalancing (2-3x reads) - After: <10% duplicates (only from uncommitted in-flight messages) This is a critical fix for production deployments where consumer churn (scaling, restarts, failures) causes frequent rebalancing.
Kafka Gateway Tests with SMQ Integration
This directory contains tests for the SeaweedFS Kafka Gateway with full SeaweedMQ (SMQ) integration.
Test Types
Unit Tests (./unit/)
- Basic gateway functionality
- Protocol compatibility
- No SeaweedFS backend required
- Uses mock handlers
Integration Tests (./integration/)
- Mock Mode (default): Uses in-memory handlers for protocol testing
- SMQ Mode (with
SEAWEEDFS_MASTERS): Uses real SeaweedFS backend for full integration
E2E Tests (./e2e/)
- End-to-end workflows
- Automatically detects SMQ availability
- Falls back to mock mode if SMQ unavailable
Running Tests Locally
Quick Protocol Testing (Mock Mode)
# Run all integration tests with mock backend
cd test/kafka
go test ./integration/...
# Run specific test
go test -v ./integration/ -run TestClientCompatibility
Full Integration Testing (SMQ Mode)
Requires running SeaweedFS instance:
- Start SeaweedFS with MQ support:
# Terminal 1: Start SeaweedFS server
weed server -ip="127.0.0.1" -ip.bind="0.0.0.0" -dir=/tmp/seaweedfs-data -master.port=9333 -volume.port=8081 -filer.port=8888 -filer=true
# Terminal 2: Start MQ broker
weed mq.broker -master="127.0.0.1:9333" -ip="127.0.0.1" -port=17777
- Run tests with SMQ backend:
cd test/kafka
SEAWEEDFS_MASTERS=127.0.0.1:9333 go test ./integration/...
# Run specific SMQ integration tests
SEAWEEDFS_MASTERS=127.0.0.1:9333 go test -v ./integration/ -run TestSMQIntegration
Test Broker Startup
If you're having broker startup issues:
# Debug broker startup locally
./scripts/test-broker-startup.sh
CI/CD Integration
GitHub Actions Jobs
- Unit Tests - Fast protocol tests with mock backend
- Integration Tests - Mock mode by default
- E2E Tests (with SMQ) - Full SeaweedFS + MQ broker stack
- Client Compatibility (with SMQ) - Tests different Kafka clients against real backend
- Consumer Group Tests (with SMQ) - Tests consumer group persistence
- SMQ Integration Tests - Dedicated SMQ-specific functionality tests
What Gets Tested with SMQ
When SEAWEEDFS_MASTERS is available, tests exercise:
- Real Message Persistence - Messages stored in SeaweedFS volumes
- Offset Persistence - Consumer group offsets stored in SeaweedFS filer
- Topic Persistence - Topic metadata persisted in SeaweedFS filer
- Consumer Group Coordination - Distributed coordinator assignment
- Cross-Client Compatibility - Sarama, kafka-go with real backend
- Broker Discovery - Gateway discovers MQ brokers via masters
Test Infrastructure
testutil.NewGatewayTestServerWithSMQ(t, mode)
Smart gateway creation that automatically:
- Detects SMQ availability via
SEAWEEDFS_MASTERS - Uses production handler when available
- Falls back to mock when unavailable
- Provides timeout protection against hanging
Modes:
SMQRequired- Skip test if SMQ unavailableSMQAvailable- Use SMQ if available, otherwise mockSMQUnavailable- Always use mock
Timeout Protection
Gateway creation includes timeout protection to prevent CI hanging:
- 20 second timeout for
SMQRequiredmode - 15 second timeout for
SMQAvailablemode - Clear error messages when broker discovery fails
Debugging Failed Tests
CI Logs to Check
- "SeaweedFS master is up" - Master started successfully
- "SeaweedFS filer is up" - Filer ready
- "SeaweedFS MQ broker is up" - Broker started successfully
- Broker/Server logs - Shown on broker startup failure
Local Debugging
- Run
./scripts/test-broker-startup.shto test broker startup - Check logs at
/tmp/weed-*.log - Test individual components:
# Test master curl http://127.0.0.1:9333/cluster/status # Test filer curl http://127.0.0.1:8888/status # Test broker nc -z 127.0.0.1 17777
Common Issues
- Broker fails to start: Check filer is ready before starting broker
- Gateway timeout: Broker discovery fails, check broker is accessible
- Test hangs: Timeout protection not working, reduce timeout values
Architecture
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Kafka Client │───▶│ Kafka Gateway │───▶│ SeaweedMQ Broker│
│ (Sarama, │ │ (Protocol │ │ (Message │
│ kafka-go) │ │ Handler) │ │ Persistence) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ SeaweedFS Filer │ │ SeaweedFS Master│
│ (Offset Storage)│ │ (Coordination) │
└─────────────────┘ └─────────────────┘
│ │
▼ ▼
┌─────────────────────────────────────────┐
│ SeaweedFS Volumes │
│ (Message Storage) │
└─────────────────────────────────────────┘
This architecture ensures full integration testing of the entire Kafka → SeaweedFS message path.