fix: Add topic cache invalidation and auto-creation on metadata requests

Add InvalidateTopicExistsCache method to SeaweedMQHandlerInterface and impl
ement cache refresh logic in metadata response handler.

When a consumer requests metadata for a topic that doesn't appear in the
cache (but was just created by a producer), force a fresh broker check
and auto-create the topic if needed with default partitions.

This fix attempts to address the consumer stalling issue by:
1. Invalidating stale cache entries before checking broker
2. Automatically creating topics on metadata requests (like Kafka's auto.create.topics.enable=true)
3. Returning topics to consumers more reliably

However, testing shows consumers still can't find topics even after creation,
suggesting a deeper issue with topic persistence or broker client communication.

Added InvalidateTopicExistsCache to mock handler as no-op for testing.

Note: Integration testing reveals that consumers get 'topic does not exist'
errors even when producers successfully create topics. This suggests the
real issue is either:
- Topics created by producers aren't visible to broker client queries
- Broker client TopicExists() doesn't work correctly
- There's a race condition in topic creation/registration

Requires further investigation of broker client implementation and SMQ
topic persistence logic.
This commit is contained in:
chrislu committed 2025-10-17 09:55:07 -07:00
1 parent 3c1df04c9a
commit a5db489f82
4 files changed
+27 -3

No files matched your search

@@ -107,8 +107,6 @@ func TestEmptyFetchShouldNotStopConsumer(t *testing.T) {
nextFetchOffset := committedOffset + 1
// First attempt: get empty (transient - data might not be available yet)
firstFetchResult := 0 // bytes returned
// WRONG behavior (bug): Consumer sees 0 bytes and stops
// wrongConsumerLogic := (firstFetchResult == 0) // gives up!