Commit Graph
2 Commits
Author SHA1 Message Date
chrislu 5595dfd476 mq(kafka): Add comprehensive protocol compatibility review and TODOs
- Create PROTOCOL_COMPATIBILITY_REVIEW.md documenting all compatibility issues
- Add critical TODOs to most problematic protocol implementations:
  * Produce: Record batch parsing is simplified, missing compression/CRC
  * Offset management: Hardcoded 'test-topic' parsing breaks real clients
  * JoinGroup: Consumer subscription extraction hardcoded, incomplete parsing
  * Fetch: Fake record batch construction with dummy data
  * Handler: Missing API version validation across all endpoints
- Identify high/medium/low priority fixes needed for real client compatibility
- Document specific areas needing work:
  * Record format parsing (v0/v1/v2, compression, CRC validation)
  * Request parsing (topics arrays, partition arrays, protocol metadata)
  * Consumer group protocol metadata parsing
  * Connection metadata extraction
  * Error code accuracy
- Add testing recommendations for kafka-go, Sarama, Java clients
- Provide roadmap for Phase 4 protocol compliance improvements

This review is essential before attempting integration with real Kafka clients
as current simplified implementations will fail with actual client libraries.
2025-09-10 13:56:02 -07:00
chrislu 26acff4373 mq(kafka): Phase 3 Step 2 - Offset Management
- Implement OffsetCommit API (key 8) for consumer offset persistence
- Implement OffsetFetch API (key 9) for consumer offset retrieval
- Add comprehensive offset management with group-level validation
- Integrate offset storage with existing consumer group coordinator
- Support offset retention, metadata, and leader epoch handling
- Add partition assignment validation for offset commits
- Update ApiVersions to advertise 11 APIs total (was 9)
- Complete test suite with 14 new test cases covering:
  * Basic offset commit/fetch operations
  * Error conditions (invalid group, wrong generation, unknown member)
  * End-to-end offset persistence workflows
  * Request parsing and response building
- All integration tests pass with updated API count (11 APIs)
- E2E tests show '84 bytes' response (increased from 72 bytes)

This completes consumer offset management, enabling Kafka clients to
reliably track and persist their consumption progress across sessions.
2025-09-10 13:39:20 -07:00