Commit Graph
12033 Commits
Author SHA1 Message Date
chrislu dec63c22e4 Add Kafka gateway implementation phases roadmap
- Document 7 phases of implementation from high to low priority
- Phase 1: Core SeaweedMQ integration for real message retrieval
- Phase 2: CreateTopics protocol compliance
- Phase 3: ApiVersions matrix accuracy
- Phase 4: Consumer group protocol metadata parsing
- Phase 5: Multi-batch Fetch support
- Phase 6: Flexible versions support
- Phase 7: Error handling and edge cases

Ready to start Phase 1 implementation.
2025-09-13 13:09:44 -07:00
chrislu c0b15ed489 refactoring 2025-09-13 13:03:51 -07:00
chrislu 964d1d06e4 fix TestSaramaProduceConsume 2025-09-13 11:06:29 -07:00
chrislu 42aea1dc68 align package decoding 2025-09-13 10:23:14 -07:00
chrislu ba48ea9c4c fix samara 2025-09-13 09:58:53 -07:00
chrislu 445d7343d7 fix v7 samara 2025-09-13 09:06:02 -07:00
chrislu 4f0dd2e527 fix kafka tests 2025-09-13 08:55:18 -07:00
chrislu 6eafc87413 fix remaining tests 2025-09-13 08:47:46 -07:00
chrislu ba1599b1e9 fix tests 2025-09-13 08:31:16 -07:00
chrislu 8743c5453a clean up connections 2025-09-13 08:17:20 -07:00
chrislu 9ea6ef0bf8 fix tests 2025-09-13 08:06:36 -07:00
chrislu ccd48feefb fix test errors 2025-09-13 00:17:54 -07:00
chrislu cce9d8a365 Remove s3tests patches - upstream repo now fixed
- Remove _create_objects patches since upstream s3-tests resolved NameError
- SeaweedFS strong consistency should eliminate 404 errors
- Let tests run naturally to detect real issues
2025-09-12 23:52:01 -07:00
chrislu c67e3255d9 Revert "Update s3tests.yml"
This reverts commit 117bcaa4b1.
2025-09-12 23:51:07 -07:00
chrislu e6f7e7efb5 fix in-memory variables 2025-09-12 23:50:07 -07:00
chrislu 78382b380e Merge branch 'master' into feature/mq-kafka-gateway-m1 2025-09-12 23:14:14 -07:00
chrislu 56aa5278af single mode 2025-09-12 23:13:41 -07:00
chrislu 3fa024479e remove TestMode 2025-09-12 23:04:41 -07:00
chrislu ba73939ca2 fixes 2025-09-12 23:02:10 -07:00
chrislu 16e014f360 start tests 2025-09-12 22:45:06 -07:00
chrislu 30b21abab9 fix kafka tests 2025-09-12 22:44:56 -07:00
chrislu c951fc03c1 Update smq_mapping_test.go 2025-09-12 22:44:36 -07:00
chrislu a342ede4cd Update go.sum 2025-09-12 22:11:04 -07:00
chrislu 117bcaa4b1 Update s3tests.yml 2025-09-12 22:10:08 -07:00
chrislu a424bfa3ce docker compose 2025-09-12 22:08:00 -07:00
chrislu e21c89d062 fix tests 2025-09-12 22:04:18 -07:00
chrislu dbac0c45d8 Update broker_offset_manager.go
Fixed GetHighWaterMark() to use correct partition managers
Fixed GetPartitionOffsetInfo() with proper struct fields
Fixed GetOffsetMetrics() with correct types and system
2025-09-12 22:04:05 -07:00
chrislu 7d9029f647 Update s3tests.yml 2025-09-12 22:00:16 -07:00
chrislu 76ddaa8b84 Update go.mod 2025-09-12 21:43:20 -07:00
chrislu fd235505f5 fmt 2025-09-12 21:34:18 -07:00
chrislu e41c31c88e Fix all critical test errors
- Fix gateway tests: Replace AgentAddress with Masters in Options struct
- Fix consumer test: Correct GenerateMemberID test to expect deterministic behavior
- Fix schema tests: Remove incorrect error assertions for mock broker scenarios
- All core offset management and protocol tests now pass
- Gateway, consumer, protocol, and offset packages compile and test successfully
2025-09-12 21:33:46 -07:00
chrislu 8de1ce5497 Fix compilation errors in integration modules
- Fix NewPersistentLedger calls (returns 1 value, not 2)
- Fix GetStats calls (returns 3 values, not 4)
- Remove error handling for NewPersistentLedger since it doesn't return errors
- All Kafka integration modules now compile successfully
2025-09-12 21:30:14 -07:00
chrislu cd2264ffc7 Update persistence.go 2025-09-12 21:23:04 -07:00
chrislu b5eb16a1a1 Phase 4: Clean up old SMQIntegratedStorage and fix compilation
- Remove old SMQIntegratedStorage implementation from persistence.go
- Update all integration modules to use SMQOffsetStorage instead
- Add delegation methods to PersistentLedger for backward compatibility
- Fix method signatures and compilation errors
- Maintain support for legacy offset operations through SeaweedMQStorage
2025-09-12 21:03:46 -07:00
chrislu 56ba8ce219 Phase 3: Add comprehensive integration tests
- Add end-to-end flow tests for Kafka OffsetCommit to SMQ storage
- Test multiple consumer groups with independent offset tracking
- Validate SMQ file path and format compatibility
- Test error handling and edge cases (negative, zero, max offsets)
- Verify offset encoding/decoding matches SMQ broker format
- Ensure consumer group isolation and proper key generation
2025-09-12 21:00:30 -07:00
chrislu ac436eac94 Phase 2: Wire OffsetCommit/OffsetFetch to SMQ storage
- Update Kafka protocol handler to use SMQOffsetStorage for consumer offsets
- Modify OffsetCommit to save consumer offsets using SMQ's filer format
- Modify OffsetFetch to read consumer offsets from SMQ's filer location
- Add proper ConsumerOffsetKey creation with consumer group and instance ID
- Maintain backward compatibility with in-memory storage fallback
- Include comprehensive test coverage for offset handler integration
2025-09-12 20:58:43 -07:00
chrislu c7b483442d Phase 1: Implement SMQ-compatible offset storage
- Add SMQOffsetStorage that uses same filer locations and format as SMQ brokers
- Store offsets in <topic-dir>/<partition-dir>/<consumerGroup>.offset files
- Use 8-byte big-endian format matching SMQ broker implementation
- Include comprehensive test coverage for core functionality
- Maintain backward compatibility through legacy method support
2025-09-12 20:54:10 -07:00
chrislu e71c6d1e48 ringsize 2025-09-12 18:51:51 -07:00
chrislu 6eb1da41d4 fix issues 2025-09-12 18:44:29 -07:00
chrislu bd7b07e90c use broker client 2025-09-12 18:35:07 -07:00
chrislu 969ca60b6f change to connect to mq brokers instead of agents 2025-09-12 18:30:35 -07:00
chrislu b7514c4ab0 SeaweedMQ is Now the Only Mode 2025-09-12 18:17:50 -07:00
chrislu 09568a6f4f real data from SeaweedMQ instead of stub/placeholder data 2025-09-12 18:12:33 -07:00
chrislu 5c17bba00b ring size MaxPartitionCount 2025-09-12 17:59:56 -07:00
chrislu d9d099744d use customer request values 2025-09-12 17:56:48 -07:00
chrislu cd6a55533a fmt 2025-09-12 17:56:34 -07:00
chrislu eeb5f62d74 fix host 2025-09-12 17:52:41 -07:00
chrislu b2016ab9b6 fix build 2025-09-12 17:49:29 -07:00
chrislu 4e4e3ce1a8 tests: align ApiVersions test expectations with advertised ranges (ListOffsets v0-2, Fetch v0-7) 2025-09-12 17:43:57 -07:00
chrislu a5f330ad17 kafka protocol: align advertised and validated API version ranges with implemented handlers (Fetch<=v7, ListOffsets<=v2, FindCoordinator<=v2, OffsetCommit/OffsetFetch<=v2); keep Metadata<=v7, JoinGroup<=v7, SyncGroup<=v5 2025-09-12 17:42:32 -07:00