debug: track stream lifecycle and total bytes written

Added comprehensive logging to identify why Parquet files fail with
'EOFException: Still have: 78 bytes left'.

Key additions:
1. SeaweedHadoopOutputStream constructor logging with 🔧 marker
   - Shows when output streams are created
   - Logs path, position, bufferSize, replication

2. totalBytesWritten counter in SeaweedOutputStream
   - Tracks cumulative bytes written via write() calls
   - Helps identify if Parquet wrote 762 bytes but only 684 reached chunks

3. Enhanced close() logging with 🔒 and  markers
   - Shows totalBytesWritten vs position vs buffer.position()
   - If totalBytesWritten=762 but position=684, write submission failed
   - If buffer.position()=78 at close, buffer wasn't flushed

Expected scenarios in next run:
A) Stream never created → No 🔧 log for .parquet files
B) Write failed → totalBytesWritten=762 but position=684
C) Buffer not flushed → buffer.position()=78 at close
D) All correct → totalBytesWritten=position=684, but Parquet expects 762

This will pinpoint whether the issue is in:
- Stream creation/lifecycle
- Write submission
- Buffer flushing
- Or Parquet's internal state
This commit is contained in:
chrislu
2025-11-23 11:54:01 -08:00
parent a5bccca443
commit a3cf4eb843
7 changed files with 22 additions and 7 deletions
+1
View File
@@ -272,3 +272,4 @@ For persistent issues:
- Note if it passes locally
+1
View File
@@ -146,3 +146,4 @@ trap cleanup INT
main
+1
View File
@@ -43,3 +43,4 @@ echo "✓ Test run completed"
echo "View detailed reports in: target/surefire-reports/"
@@ -142,3 +142,4 @@ public class SparkSeaweedFSExample {
}
@@ -235,3 +235,4 @@ public class SparkSQLTest extends SparkTestBase {
}