* volume: run the durable sync outside the reader-visible lock
Reads of a volume waited on that volume's fsync because the write lock
was held across Sync()/sync_all. A new per-volume lock (Go
dataFileSyncLock, Rust data_file_sync_lock) now serializes every .dat
mutation and lifecycle op across append+publish+sync+rollback, while the
fsync itself runs with dataFileAccessLock/the io write guard dropped —
so readers proceed during a sync, and a failed sync's rollback still
cannot clobber another write or a compaction swap.
Same change in Go (volume_write.go syncWrite/processBatch, SyncToDisk)
and Rust (write_needles_grouped/write_synced_run).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: publish after sync; fix Close/CommitCompact lock order
Address review on #11863:
- Close took dataFileSyncLock before waiting on isCompactionInProgress
while CommitCompact claims the flag then takes the lock — a deadlock.
Close now claims the flag before taking the locks (flag -> sync -> access).
- Go now publishes needle-map entries only after the .dat sync, matching
Rust's ordering: appendWriteRequest/appendDeleteRequest stage the bytes,
publishWriteRequest/publishDeleteRequest index them after Sync. A failed
sync rolls back bytes no reader ever resolved — the streamed-read
offset-invalidation window is gone. processBatch splits runs at repeated
needle ids (Rust parity) so dedup/cookie checks stay sequential; the old
snapshot/restore rollback machinery is removed.
- Rust commit_compact (store-lock caller) takes data_file_sync_lock via
try_lock -> WouldBlock before the write lease; commit_compact_held and
the tier-down _held paths document/require the caller-held sync lock,
and the two vol.instance().write_lock() grpc sites take it first. The
swap can no longer land between a durable write's append and publish.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: restore write watermarks when an inline fsync fails
The syncWrite rollback truncated the .dat but left lastAppendAtNs,
lastWriteAppendAtNs, lastWriteNeedleKey, lastWriteDeleted and
lastModifiedTsSeconds describing the rejected write; recovery and
compaction could read a watermark for a record that no longer exists.
Snapshot and restore them on failure, as processBatchRun already does.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
