mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-10 00:20:42 +02:00
debug: track stream lifecycle and total bytes written
Added comprehensive logging to identify why Parquet files fail with 'EOFException: Still have: 78 bytes left'. Key additions: 1. SeaweedHadoopOutputStream constructor logging with 🔧 marker - Shows when output streams are created - Logs path, position, bufferSize, replication 2. totalBytesWritten counter in SeaweedOutputStream - Tracks cumulative bytes written via write() calls - Helps identify if Parquet wrote 762 bytes but only 684 reached chunks 3. Enhanced close() logging with 🔒 and ✅ markers - Shows totalBytesWritten vs position vs buffer.position() - If totalBytesWritten=762 but position=684, write submission failed - If buffer.position()=78 at close, buffer wasn't flushed Expected scenarios in next run: A) Stream never created → No 🔧 log for .parquet files B) Write failed → totalBytesWritten=762 but position=684 C) Buffer not flushed → buffer.position()=78 at close D) All correct → totalBytesWritten=position=684, but Parquet expects 762 This will pinpoint whether the issue is in: - Stream creation/lifecycle - Write submission - Buffer flushing - Or Parquet's internal state
This commit is contained in:
@@ -272,3 +272,4 @@ For persistent issues:
|
||||
- Note if it passes locally
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -146,3 +146,4 @@ trap cleanup INT
|
||||
main
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -43,3 +43,4 @@ echo "✓ Test run completed"
|
||||
echo "View detailed reports in: target/surefire-reports/"
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -142,3 +142,4 @@ public class SparkSeaweedFSExample {
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -235,3 +235,4 @@ public class SparkSQLTest extends SparkTestBase {
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user