debug: add critical diagnostics for EOFException (78 bytes missing)

The persistent EOFException shows Parquet expects 78 more bytes than exist.
This suggests a mismatch between what was written vs what's in chunks.

Added logging to track:
1. Buffer state at close (position before flush)
2. Stream position when flushing metadata
3. Chunk count vs file size in attributes
4. Explicit fileSize setting from stream position

Key hypothesis:
- Parquet writes N bytes total (e.g., 762)
- Stream.position tracks all writes
- But only (N-78) bytes end up in chunks
- This causes Parquet read to fail with 'Still have: 78 bytes left'

If buffer.position() = 78 at close, the buffer wasn't flushed.
If position != chunk total, write submission failed.
If attr.fileSize != position, metadata is inconsistent.

Next run will show which scenario is happening.
This commit is contained in:
chrislu committed 2025-11-23 11:39:47 -08:00
1 parent d9ab0721b9
commit a5bccca443
3 files changed
+50 -21

No files matched your search

+1 -1
View File
@@ -572,7 +572,7 @@
</dependency>
</dependencies>
<properties>
<seaweedfs.client.version>3.80</seaweedfs.client.version>
<seaweedfs.client.version>3.80.1-SNAPSHOT</seaweedfs.client.version>
<hadoop.version>3.4.0</hadoop.version>
</properties>
</project>