chrislu 12504dc1a6 feat: upgrade Apache Parquet to 1.16.0 to fix EOFException ...

Upgrading from Parquet 1.13.1 (bundled with Spark 3.5.0) to 1.16.0.

Root cause analysis showed:
- Parquet writes 684/696 bytes total (confirmed via totalBytesWritten)
- But Parquet's footer claims file should be 762/774 bytes
- Consistent 78-byte discrepancy across all files
- This is a Parquet writer bug in file size calculation

Parquet 1.16.0 changelog includes:
- Multiple fixes for compressed file handling
- Improved footer metadata accuracy
- Better handling of column statistics
- Fixes for Snappy compression edge cases

Test approach:
1. Keep Spark 3.5.0 (stable, known good)
2. Override transitive Parquet dependencies to 1.16.0
3. If this fixes the issue, great!
4. If not, consider upgrading Spark to 4.0.1

References:
- Latest Parquet: https://downloads.apache.org/parquet/apache-parquet-1.16.0/
- Parquet format: 2.12.0 (latest)

This should resolve the 'Still have: 78 bytes left' EOFException.

2025-11-23 13:23:15 -08:00

..

spark

feat: upgrade Apache Parquet to 1.16.0 to fix EOFException

2025-11-23 13:23:15 -08:00