Commit Graph
4 Commits
Author SHA1 Message Date
chrislu 0e481cf97a fmt 2025-10-15 21:37:07 -07:00
chrislu c5634470ed feat: add disk I/O fallback for historical offset reads
This commit implements async disk I/O fallback to handle cases where:
1. Data is flushed from memory before consumers can read it (CI issue)
2. Consumers request historical offsets not in memory
3. Small LogBuffer retention in resource-constrained environments

Changes:
- Add readHistoricalDataFromDisk() helper function
- Update ReadMessagesAtOffset() to call ReadFromDiskFn when offset < bufferStartOffset
- Properly handle maxMessages and maxBytes limits during disk reads
- Return appropriate nextOffset after disk reads
- Log disk read operations at V(2) and V(3) levels

Benefits:
- Fixes CI test failures where data is flushed before consumption
- Enables consumers to catch up even if they fall behind memory retention
- No blocking on hot path (disk read only for historical data)
- Respects existing ReadFromDiskFn timeout handling

How it works:
1. Try in-memory read first (fast path)
2. If offset too old and ReadFromDiskFn configured, read from disk
3. Return disk data with proper nextOffset
4. Consumer continues reading seamlessly

This fixes the 'offset 0 too old (earliest in-memory: 5)' error in
TestOffsetManagement where messages were flushed before consumer started.
2025-10-15 21:35:39 -07:00
chrislu 1807b8093c debug fetch offset APIs 2025-10-15 18:59:46 -07:00
chrislu ba1a8aed64 log read stateless 2025-10-15 16:08:21 -07:00