Files
seaweedfs/weed/worker
Chris Lu 97b54adcf6 iceberg: sort compaction bins on disk instead of in memory (#11112)
A sorted rewrite collected every row of a bin into one slice and sorted it
there, so a bin larger than the worker's heap could not be sorted at all.
sort_max_input_mb existed for that reason and skipped the bins it capped.

parquet-go's SortingWriter buffers sort_buffer_rows rows, encodes each buffer
as a sorted run, and merges the runs at close; backing those runs with a
FileBufferPool keeps them in files rather than on the heap. sort_spill_dir says
where, defaulting to the system temp directory — NewFileBufferPool resolves an
empty path to the working directory, which is not what an unset setting means.

The output now also declares its sorting columns, which the plain writer the
sorted path used never did.

Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y
2026-09-02 20:17:30 -07:00
..