mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-03 02:32:35 +01:00
chrislu
82705a214f
- Phase 1 ✅: SeaweedMQ record retrieval implemented - Phase 2 ✅: CreateTopics v0-v5 compliance implemented - Phase 3 ✅: ApiVersions matrix accuracy fixed - Phase 4 ✅: Consumer group protocol metadata parsing enhanced - Phase 5 ✅: Multi-batch Fetch concatenation support implemented (MaxBytes compliance, 17 tests) - Ready to start Phase 6: Basic flexible versions support
118 lines
4.6 KiB
Markdown
118 lines
4.6 KiB
Markdown
# Kafka Gateway Implementation Phases
|
|
|
|
## Phase 1: Core SeaweedMQ Integration (COMPLETED ✅)
|
|
**Goal**: Enable real message retrieval from SeaweedMQ storage
|
|
|
|
### Tasks:
|
|
- [x] Implement `integration.SeaweedMQHandler.GetStoredRecords()` to return actual records
|
|
- [x] Add proper SMQ record conversion from SeaweedMQ format to Kafka format
|
|
- [x] Wire Fetch API to use real SMQ records instead of synthetic batches
|
|
- [x] Add integration tests for end-to-end message storage and retrieval
|
|
|
|
**Files modified**:
|
|
- `weed/mq/kafka/integration/seaweedmq_handler.go`
|
|
- `weed/mq/kafka/protocol/fetch.go` (verification)
|
|
- Added test file: `weed/mq/kafka/integration/record_retrieval_test.go`
|
|
|
|
**Verification**: E2E tests show "Found X SMQ records" - real data retrieval working
|
|
|
|
## Phase 2: CreateTopics Protocol Compliance (COMPLETED ✅)
|
|
**Goal**: Fix CreateTopics API parsing and partition handling
|
|
|
|
### Tasks:
|
|
- [x] Implement `handleCreateTopicsV0V1` request parsing
|
|
- [x] Add support for partition count and basic topic configurations
|
|
- [x] Wire CreateTopics to actual SeaweedMQ topic creation
|
|
- [x] Add CreateTopics integration tests
|
|
|
|
**Files modified**:
|
|
- `weed/mq/kafka/protocol/handler.go`
|
|
- Added test file: `weed/mq/kafka/protocol/create_topics_test.go`
|
|
|
|
**Verification**: All v0-v5 CreateTopics tests pass; proper partition handling
|
|
|
|
## Phase 3: ApiVersions Matrix Accuracy (COMPLETED ✅)
|
|
**Goal**: Ensure advertised API versions match actual implementation
|
|
|
|
### Tasks:
|
|
- [x] Audit current `handleApiVersions` response against implemented features
|
|
- [x] Correct max versions for APIs with higher implementations than advertised
|
|
- [x] Update version validation in request handlers to match advertisements
|
|
- [x] Document supported vs unsupported features per API version
|
|
|
|
**Files modified**:
|
|
- `weed/mq/kafka/protocol/handler.go` (`handleApiVersions` & `validateAPIVersion`)
|
|
- Added file: `weed/mq/kafka/API_VERSION_MATRIX.md`
|
|
- Added test file: `weed/mq/kafka/protocol/api_versions_test.go`
|
|
|
|
**Verification**: Critical fixes for OffsetFetch v0-v5 and CreateTopics v0-v5 accuracy
|
|
|
|
## Phase 4: Consumer Group Protocol Metadata (COMPLETED ✅)
|
|
**Goal**: Proper JoinGroup protocol metadata parsing
|
|
|
|
### Tasks:
|
|
- [x] Implement consumer protocol metadata parsing in JoinGroup
|
|
- [x] Extract subscription topics and user data from metadata
|
|
- [x] Populate ClientHost from connection information
|
|
- [x] Support multiple assignment strategies properly
|
|
|
|
**Files modified**:
|
|
- `weed/mq/kafka/protocol/joingroup.go`
|
|
- `weed/mq/kafka/protocol/handler.go` (connection context)
|
|
- Added file: `weed/mq/kafka/protocol/consumer_group_metadata.go`
|
|
- Added test file: `weed/mq/kafka/protocol/consumer_group_metadata_test.go`
|
|
|
|
**Verification**: ClientHost shows real IPs; enhanced protocol metadata parsing with 17 tests
|
|
|
|
## Phase 5: Multi-Batch Fetch Support (COMPLETED ✅)
|
|
**Goal**: Support multiple record batch concatenation in Fetch responses
|
|
|
|
### Tasks:
|
|
- [x] Implement proper record batch concatenation
|
|
- [x] Handle batch size limits and chunking
|
|
- [x] Add support for compressed record batches
|
|
- [x] Performance optimization for large batch responses
|
|
|
|
**Files modified**:
|
|
- `weed/mq/kafka/protocol/fetch.go` (integrated multi-batch fetcher)
|
|
- Added file: `weed/mq/kafka/protocol/fetch_multibatch.go`
|
|
- Added test file: `weed/mq/kafka/protocol/fetch_multibatch_test.go`
|
|
|
|
**Verification**: MaxBytes compliance, multi-batch concatenation, 17 comprehensive tests, E2E compatibility
|
|
|
|
## Phase 6: Flexible Versions Support (PRIORITY LOW)
|
|
**Goal**: Basic support for flexible versions and tagged fields
|
|
|
|
### Tasks:
|
|
- [ ] Add flexible version detection in request headers
|
|
- [ ] Implement tagged field parsing/skipping
|
|
- [ ] Update response encoders for flexible versions
|
|
- [ ] Add flexible version tests
|
|
|
|
**Files to modify**:
|
|
- `weed/mq/kafka/protocol/handler.go`
|
|
- `weed/mq/kafka/protocol/flexible_versions.go` (new file)
|
|
- Add test file: `weed/mq/kafka/protocol/flexible_versions_test.go`
|
|
|
|
## Phase 7: Error Handling and Edge Cases (PRIORITY LOW)
|
|
**Goal**: Comprehensive error handling and Kafka spec compliance
|
|
|
|
### Tasks:
|
|
- [ ] Audit all error codes against Kafka specification
|
|
- [ ] Add missing error codes for network/timeout scenarios
|
|
- [ ] Implement proper connection timeout handling
|
|
- [ ] Add comprehensive error handling tests
|
|
|
|
**Files to modify**:
|
|
- `weed/mq/kafka/protocol/errors.go`
|
|
- All protocol handler files
|
|
- Add test file: `weed/mq/kafka/protocol/error_handling_test.go`
|
|
|
|
## Current Status: Phase 1-5 completed, ready for Phase 6
|
|
|
|
### Implementation Notes:
|
|
- Each phase should include comprehensive tests
|
|
- Commit after each phase completion
|
|
- Backward compatibility must be maintained
|
|
- Integration tests should verify end-to-end functionality
|