mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-12 09:30:46 +02:00
chrislu
7b067d2e59
Added getPos() to SeaweedOutputStream to understand when and how Hadoop/Parquet queries the output stream position. Current mystery: - Files are written correctly (totalBytesWritten=position=chunks) - But Parquet expects 78 more bytes when reading - year=2020: wrote 696, expects 774 (missing 78) - year=2021: wrote 684, expects 762 (missing 78) The consistent 78-byte discrepancy suggests either: A) Parquet calculates row group size before finalizing footer B) FSDataOutputStream tracks position differently than our stream C) Footer is written with stale/incorrect metadata D) File size is cached/stale during rename operation getPos() logging will show if Parquet/Hadoop queries position and what value is returned vs what was actually written.