mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-17 20:10:51 +02:00
fix: add file sync and cache settings to prevent EOF on read
Issue: Files written successfully but truncated when read back Error: 'EOFException: Reached the end of stream. Still have: 78 bytes left' Root cause: Potential race condition between write completion and read - File metadata updated before all chunks fully flushed - Spark immediately reads after write without ensuring sync - Parquet reader gets incomplete file Solutions applied: 1. Disable filesystem cache to avoid stale file handles - spark.hadoop.fs.seaweedfs.impl.disable.cache=true 2. Enable explicit flush/sync on write (if supported by client) - spark.hadoop.fs.seaweed.write.flush.sync=true 3. Add SPARK_SUBMIT_OPTS for cache disabling These settings ensure: - Files are fully flushed before close() returns - No cached file handles with stale metadata - Fresh reads always get current file state Note: If issue persists, may need to add explicit delay between write and read, or investigate seaweedfs-hadoop3-client flush behavior.
This commit is contained in:
@@ -81,6 +81,8 @@ services:
|
||||
- HADOOP_HOME=/tmp
|
||||
# Disable Java DNS caching to ensure fresh DNS lookups
|
||||
- MAVEN_OPTS=-Dsun.net.inetaddr.ttl=0 -Dnetworkaddress.cache.ttl=0
|
||||
# Force fsync on close to ensure data is flushed before file is considered written
|
||||
- SPARK_SUBMIT_OPTS=-Dfs.seaweedfs.impl.disable.cache=true
|
||||
command: sh -c "sleep 30 && ping -c 1 seaweedfs-volume && ping -c 1 seaweedfs-filer && mvn test"
|
||||
depends_on:
|
||||
seaweedfs-filer:
|
||||
|
||||
Reference in New Issue
Block a user