- Fix TestKafkaGateway_SchemaPerformance: Update test schema to match registered schema with email field
- Fix TestSchematizedMessageToSMQ: Always store records in ledger regardless of schema processing
- Fix persistent_offset_integration_test.go: Remove unused subscription variable
- Improve error handling for schema registry connection failures
- All schema integration tests now pass successfully
Issues Fixed:
1. Avro decoding failure due to schema mismatch (missing email field)
2. Offset retrieval failure due to records not being stored in ledger
3. Compilation error with unused variable
4. Graceful handling of schema registry unavailability
Test Results:
✅ TestKafkaGateway_SchemaIntegration - All subtests pass
✅ TestKafkaGateway_SchemaPerformance - Performance test passes (avg: 9.69µs per decode)
✅ TestSchematizedMessageToSMQ - Offset management and Avro workflow pass
✅ TestCompressionWithSchemas - Compression integration passes
Schema registry integration is now robust and handles both connected and disconnected scenarios.
Phase E2: Integrate Protobuf descriptor parser with decoder
- Update NewProtobufDecoder to use ProtobufDescriptorParser
- Add findFirstMessageName helper for automatic message detection
- Fix ParseBinaryDescriptor to return schema even on resolution failure
- Add comprehensive tests for protobuf decoder integration
- Improve error handling and caching behavior
This enables proper binary descriptor parsing in the protobuf decoder,
completing the integration between descriptor parsing and decoding.
Phase E3: Complete Protobuf message descriptor resolution
- Implement full protobuf descriptor resolution using protoreflect API
- Add buildFileDescriptor and findMessageInFileDescriptor methods
- Support nested message resolution with findNestedMessageDescriptor
- Add proper mutex protection for thread-safe cache access
- Update all test data to use proper field cardinality labels
- Update test expectations to handle successful descriptor resolution
- Enable full protobuf decoder creation from binary descriptors
Phase E (Protobuf Support) is now complete:
✅ E1: Binary descriptor parsing
✅ E2: Decoder integration
✅ E3: Full message descriptor resolution
Protobuf messages can now be fully parsed and decoded
Phase F: Implement Kafka record batch compression support
- Add comprehensive compression module supporting gzip/snappy/lz4/zstd
- Implement RecordBatchParser with full compression and CRC validation
- Support compression codec extraction from record batch attributes
- Add compression/decompression for all major Kafka codecs
- Integrate compression support into Produce and Fetch handlers
- Add extensive unit tests for all compression codecs
- Support round-trip compression/decompression with proper error handling
- Add performance benchmarks for compression operations
Key features:
✅ Gzip compression (ratio: 0.02)
✅ Snappy compression (ratio: 0.06, fastest)
✅ LZ4 compression (ratio: 0.02)
✅ Zstd compression (ratio: 0.01, best compression)
✅ CRC32 validation for record batch integrity
✅ Proper Kafka record batch format v2 parsing
✅ Backward compatibility with uncompressed records
Phase F (Compression Handling) is now complete.
Phase G: Implement advanced schema compatibility checking and migration
- Add comprehensive SchemaEvolutionChecker with full compatibility rules
- Support BACKWARD, FORWARD, FULL, and NONE compatibility levels
- Implement Avro schema compatibility checking with field analysis
- Add JSON Schema compatibility validation
- Support Protobuf compatibility checking (simplified implementation)
- Add type promotion rules (int->long, float->double, string<->bytes)
- Integrate schema evolution into Manager with validation methods
- Add schema evolution suggestions and migration guidance
- Support schema compatibility validation before evolution
- Add comprehensive unit tests for all compatibility scenarios
Key features:
✅ BACKWARD compatibility: New schema can read old data
✅ FORWARD compatibility: Old schema can read new data
✅ FULL compatibility: Both backward and forward compatible
✅ Type promotion support for safe schema evolution
✅ Field addition/removal validation with default value checks
✅ Schema evolution suggestions for incompatible changes
✅ Integration with schema registry for validation workflows
Phase G (Schema Evolution) is now complete.
fmt
chrislu