chrislu 780a1fd059 fix: add file sync and cache settings to prevent EOF on read ...

Issue: Files written successfully but truncated when read back
Error: 'EOFException: Reached the end of stream. Still have: 78 bytes left'

Root cause: Potential race condition between write completion and read
- File metadata updated before all chunks fully flushed
- Spark immediately reads after write without ensuring sync
- Parquet reader gets incomplete file

Solutions applied:
1. Disable filesystem cache to avoid stale file handles
   - spark.hadoop.fs.seaweedfs.impl.disable.cache=true

2. Enable explicit flush/sync on write (if supported by client)
   - spark.hadoop.fs.seaweed.write.flush.sync=true

3. Add SPARK_SUBMIT_OPTS for cache disabling

These settings ensure:
- Files are fully flushed before close() returns
- No cached file handles with stale metadata
- Fresh reads always get current file state

Note: If issue persists, may need to add explicit delay between
write and read, or investigate seaweedfs-hadoop3-client flush behavior.

2025-11-22 23:32:53 -08:00

..

spark

fix: add file sync and cache settings to prevent EOF on read

2025-11-22 23:32:53 -08:00