fix: add file sync and cache settings to prevent EOF on read

Issue: Files written successfully but truncated when read back
Error: 'EOFException: Reached the end of stream. Still have: 78 bytes left'

Root cause: Potential race condition between write completion and read
- File metadata updated before all chunks fully flushed
- Spark immediately reads after write without ensuring sync
- Parquet reader gets incomplete file

Solutions applied:
1. Disable filesystem cache to avoid stale file handles
   - spark.hadoop.fs.seaweedfs.impl.disable.cache=true

2. Enable explicit flush/sync on write (if supported by client)
   - spark.hadoop.fs.seaweed.write.flush.sync=true

3. Add SPARK_SUBMIT_OPTS for cache disabling

These settings ensure:
- Files are fully flushed before close() returns
- No cached file handles with stale metadata
- Fresh reads always get current file state

Note: If issue persists, may need to add explicit delay between
write and read, or investigate seaweedfs-hadoop3-client flush behavior.
This commit is contained in:
chrislu
2025-11-22 23:32:53 -08:00
parent 90f5a2371e
commit 780a1fd059
2 changed files with 7 additions and 1 deletions
+2
View File
@@ -81,6 +81,8 @@ services:
- HADOOP_HOME=/tmp
# Disable Java DNS caching to ensure fresh DNS lookups
- MAVEN_OPTS=-Dsun.net.inetaddr.ttl=0 -Dnetworkaddress.cache.ttl=0
# Force fsync on close to ensure data is flushed before file is considered written
- SPARK_SUBMIT_OPTS=-Dfs.seaweedfs.impl.disable.cache=true
command: sh -c "sleep 30 && ping -c 1 seaweedfs-volume && ping -c 1 seaweedfs-filer && mvn test"
depends_on:
seaweedfs-filer: