Files
seaweedfs/weed/mq/kafka/IMPLEMENTATION_PHASES.md
T
chrislu 5d0c45c9dc Phase 2: Implement CreateTopics protocol compliance for v0/v1
CreateTopics Protocol Compliance completed:

## Implementation
- Implement handleCreateTopicsV0V1() with proper v0/v1 request parsing
- Support regular array/string format (not compact) for v0/v1
- Parse topic name, partitions, replication factor, assignments, configs
- Handle timeout_ms and validate_only fields correctly
- Maintain existing v2+ compact format support
- Wire to SeaweedMQ handler for actual topic creation

## Key Features
- Full v0-v5 CreateTopics API version support
- Proper error handling (TOPIC_ALREADY_EXISTS, INVALID_PARTITIONS, etc.)
- Partition count validation and enforcement
- Compatible with existing SeaweedMQ topic management

## Tests
- Comprehensive unit tests for v0/v1/v2+ parsing
- Error condition testing (duplicate topics, invalid partitions)
- Multi-topic creation support
- Integration tests across all API versions
- Performance benchmarks for CreateTopics operations

## Verification
- All protocol tests pass (v0-v5 CreateTopics)
- E2E Sarama tests continue to work
- Real topics created with specified partition counts
- Proper error responses for edge cases

Ready for Phase 3: ApiVersions matrix accuracy
2025-09-13 13:18:54 -07:00

109 lines
4.1 KiB
Markdown

# Kafka Gateway Implementation Phases
## Phase 1: Core SeaweedMQ Integration (COMPLETED ✅)
**Goal**: Enable real message retrieval from SeaweedMQ storage
### Tasks:
- [x] Implement `integration.SeaweedMQHandler.GetStoredRecords()` to return actual records
- [x] Add proper SMQ record conversion from SeaweedMQ format to Kafka format
- [x] Wire Fetch API to use real SMQ records instead of synthetic batches
- [x] Add integration tests for end-to-end message storage and retrieval
**Files modified**:
- `weed/mq/kafka/integration/seaweedmq_handler.go`
- `weed/mq/kafka/protocol/fetch.go` (verification)
- Added test file: `weed/mq/kafka/integration/record_retrieval_test.go`
**Verification**: E2E tests show "Found X SMQ records" - real data retrieval working
## Phase 2: CreateTopics Protocol Compliance (PRIORITY HIGH)
**Goal**: Fix CreateTopics API parsing and partition handling
### Tasks:
- [ ] Implement `handleCreateTopicsV0V1` request parsing
- [ ] Add support for partition count and basic topic configurations
- [ ] Wire CreateTopics to actual SeaweedMQ topic creation
- [ ] Add CreateTopics integration tests
**Files to modify**:
- `weed/mq/kafka/protocol/handler.go`
- `weed/mq/kafka/protocol/create_topics.go` (new file)
- Add test file: `weed/mq/kafka/protocol/create_topics_test.go`
## Phase 3: ApiVersions Matrix Accuracy (PRIORITY MEDIUM)
**Goal**: Ensure advertised API versions match actual implementation
### Tasks:
- [ ] Audit current `handleApiVersions` response against implemented features
- [ ] Lower max versions for APIs with incomplete implementations
- [ ] Add version validation in request handlers
- [ ] Document supported vs unsupported features per API version
**Files to modify**:
- `weed/mq/kafka/protocol/handler.go` (`handleApiVersions`)
- Add file: `weed/mq/kafka/API_VERSION_MATRIX.md`
- Add test file: `weed/mq/kafka/protocol/api_versions_test.go`
## Phase 4: Consumer Group Protocol Metadata (PRIORITY MEDIUM)
**Goal**: Proper JoinGroup protocol metadata parsing
### Tasks:
- [ ] Implement consumer protocol metadata parsing in JoinGroup
- [ ] Extract subscription topics and user data from metadata
- [ ] Populate ClientHost from connection information
- [ ] Support multiple assignment strategies properly
**Files to modify**:
- `weed/mq/kafka/protocol/joingroup.go`
- `weed/mq/kafka/protocol/consumer_group_metadata.go` (new file)
- Add test file: `weed/mq/kafka/protocol/consumer_group_metadata_test.go`
## Phase 5: Multi-Batch Fetch Support (PRIORITY MEDIUM)
**Goal**: Support multiple record batch concatenation in Fetch responses
### Tasks:
- [ ] Implement proper record batch concatenation
- [ ] Handle batch size limits and chunking
- [ ] Add support for compressed record batches
- [ ] Performance optimization for large batch responses
**Files to modify**:
- `weed/mq/kafka/protocol/fetch.go`
- Add test file: `weed/mq/kafka/protocol/fetch_multi_batch_test.go`
## Phase 6: Flexible Versions Support (PRIORITY LOW)
**Goal**: Basic support for flexible versions and tagged fields
### Tasks:
- [ ] Add flexible version detection in request headers
- [ ] Implement tagged field parsing/skipping
- [ ] Update response encoders for flexible versions
- [ ] Add flexible version tests
**Files to modify**:
- `weed/mq/kafka/protocol/handler.go`
- `weed/mq/kafka/protocol/flexible_versions.go` (new file)
- Add test file: `weed/mq/kafka/protocol/flexible_versions_test.go`
## Phase 7: Error Handling and Edge Cases (PRIORITY LOW)
**Goal**: Comprehensive error handling and Kafka spec compliance
### Tasks:
- [ ] Audit all error codes against Kafka specification
- [ ] Add missing error codes for network/timeout scenarios
- [ ] Implement proper connection timeout handling
- [ ] Add comprehensive error handling tests
**Files to modify**:
- `weed/mq/kafka/protocol/errors.go`
- All protocol handler files
- Add test file: `weed/mq/kafka/protocol/error_handling_test.go`
## Current Status: Ready to start Phase 1
### Implementation Notes:
- Each phase should include comprehensive tests
- Commit after each phase completion
- Backward compatibility must be maintained
- Integration tests should verify end-to-end functionality