- Design comprehensive SQL schema for offset storage with future _index column support
- Implement SQLOffsetStorage with full database operations:
- Partition offset checkpoints with UPSERT functionality
- Detailed offset mappings with range queries and statistics
- Database migration system with version tracking
- Performance optimizations with proper indexing
- Add database migration manager with automatic schema updates
- Create comprehensive test suite with 11 test cases covering:
- Schema initialization and table creation
- Checkpoint save/load operations with error handling
- Offset mapping storage and retrieval with sorting
- Range queries and highest offset detection
- Partition statistics with NULL value handling
- Cleanup operations for old data retention
- Concurrent access safety and database vacuum
- Extend BrokerOffsetManager with SQL storage integration:
- NewBrokerOffsetManagerWithSQL for database-backed storage
- Configurable storage backends (in-memory fallback, SQL preferred)
- Database connection management and error handling
- Add SQLite driver dependency and configure for optimal performance
- Support for future database types (PostgreSQL, MySQL) with abstraction layer
Key TODOs and Assumptions:
- TODO: Add _index as computed column when database supports it
- TODO: Implement database backup and restore functionality
- TODO: Add configuration for database path and connection parameters
- ASSUMPTION: Using SQLite for now, extensible to other databases
- ASSUMPTION: WAL mode and performance pragmas for production use
- ASSUMPTION: Migration system handles schema evolution gracefully
All 11 SQL storage tests pass, providing robust persistent offset management.
- Add PartitionOffsetManager for sequential offset assignment per partition
- Implement OffsetStorage interface with in-memory and SQL storage backends
- Add PartitionOffsetRegistry for managing multiple partition offset managers
- Implement offset recovery from checkpoints and storage scanning
- Add OffsetAssigner for high-level offset assignment operations
- Support both single and batch offset assignment with timestamps
- Add comprehensive tests covering:
- Basic and batch offset assignment
- Offset recovery from checkpoints and storage
- Multi-partition offset management
- Concurrent offset assignment safety
- All tests pass, offset assignment is thread-safe and recoverable
chrislu