volume server: group-commit fsync writes in the write queue (#11543)

* volume: split the write path into reusable steps

do_write_request ran its pre-append checks, the append, the sync
rollback, the index publish and the post-write bookkeeping inline, so a
batched write could only reuse it one needle at a time. Pull the steps
out (check_writable, prepare_write, undo_unsynced_append, publish_write,
finish_write) and the store's volume lookup plus disk-space check
(writable_volume_mut). do_write_request composes them in the same order
with the same early returns; no behaviour change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: group-commit fsync writes in the write queue

The write queue holds one store lock for a batch of up to 128 needles
but wrote them one at a time, so every fsync needle paid its own .dat
sync and its own .idx sync: 2N syncs per batch.

Add Volume::write_needles_grouped, after Go's processBatch. A volume's
entries are split into runs of distinct needle ids (a repeated id starts
a new run, so its dedup and cookie checks see the earlier write). A run
with a durable entry appends everything with append_at_ns chained
through a local, syncs the .dat once, and only then publishes the
entries and syncs the .idx once. A failed .dat sync truncates the .dat
back to the run start (marking the volume unavailable if that fails),
leaves last_append_at_ns and last_modified untouched, and fails every
entry of the run. Runs with no durable entry go through the unchanged
per-needle path. Store::write_volume_needles is the queue's entry point;
the handlers' non-queue path is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: fail closed when a failed append's rollback fails

append_needle discarded the truncate-back result, so a partial write
that could not be rolled back left unindexed bytes on the .dat while
the volume stayed writable; the next append would bury them mid-file,
past the load-time tail check. Route the rollback through
undo_unsynced_append, which marks the volume unavailable when the
truncate fails, so nothing more is appended over an unverified tail.

* volume server: keep a grouped run's I/O error streak from later appends

A synced run stages every append before any entry finishes, so the
success reset in finish_write ran after the failed appends queued
behind the last write to land and erased their media-error streak.
Sent one at a time, those errors would have counted and quarantined
the volume. Skip the reset when an append after the last landed write
added to the streak.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: replay a grouped run's I/O error streak in queue order

Skipping the run's success reset whenever an append after the last
landed write failed kept the errors from before that write as well,
so a run like [EIO, EIO, landed, EIO] reached the quarantine count
that the same writes one at a time (one error) do not. Mark the
streak where each entry is staged and record the run's success at
the last landed write's mark: errors before it are cleared, the ones
after it still count.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* volume server: replay a run's I/O error streak in one atomic step

Reads record their outcomes on the tracker without the volume's write
lock, so record_success_at's separate load and store could drop an
error a read counted in between, or restore errors a read had just
cleared. Keep the count and the clear counter in one atomic word and
apply the replay with a single fetch_update.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>

seaweed-volume/src/server/write_queue.rs

@@ -3,8 +3,8 @@
 //! Instead of each upload handler directly calling `write_needle`, writes are
 //! submitted to a queue. A background worker drains the queue in batches (up to
 //! 128 entries), groups them by volume ID, and processes them together under a
-//! single store lock. Requests that asked for `fsync` are flushed by
-//! `write_needle` itself, one flush per durable write.
+//! single store lock. Durable writes to a volume share their .dat and .idx
+//! flushes (see `Volume::write_needles_grouped`).
 
 use std::sync::Arc;
 
@@ -159,8 +159,12 @@ fn process_batch(state: Arc<VolumeServerState>, batch: Vec<WriteRequest>) {
     let mut store = state.store.write().unwrap();
 
     for (vid, entries) in groups {
-        for (mut needle, fsync, response_tx) in entries {
-            let result = store.write_volume_needle(vid, &mut needle, fsync);
+        let (mut writes, senders): (Vec<_>, Vec<_>) = entries
+            .into_iter()
+            .map(|(needle, fsync, response_tx)| ((needle, fsync), response_tx))
+            .unzip();
+        let results = store.write_volume_needles(vid, &mut writes);
+        for (response_tx, result) in senders.into_iter().zip(results) {
             // Send result back; ignore error if receiver dropped.
             let _ = response_tx.send(result);
         }
@@ -312,6 +316,63 @@ mod tests {
         }
     }
 
+    /// The queue hands a volume's batch to the grouped path, so ten durable
+    /// writes cost one .dat sync and one .idx sync, not ten of each.
+    #[test]
+    fn test_process_batch_group_commits_fsync_writes() {
+        use crate::config::MinFreeSpace;
+        use crate::storage::types::{DiskType, NeedleId};
+        use crate::storage::volume::VolumeSpec;
+
+        let tmp = tempfile::TempDir::new().unwrap();
+        let dir = tmp.path().to_str().unwrap();
+        let state = make_test_state();
+        {
+            let mut store = state.store.write().unwrap();
+            store
+                .add_location(
+                    dir,
+                    dir,
+                    10,
+                    DiskType::HardDrive,
+                    MinFreeSpace::Percent(1.0),
+                    Vec::new(),
+                )
+                .unwrap();
+            store
+                .add_volume(VolumeId(1), DiskType::HardDrive, &VolumeSpec::default())
+                .unwrap();
+        }
+
+        let mut receivers = Vec::new();
+        let batch = (1..=10u64)
+            .map(|id| {
+                let (response_tx, response_rx) = oneshot::channel();
+                receivers.push(response_rx);
+                WriteRequest {
+                    volume_id: VolumeId(1),
+                    needle: Needle {
+                        id: NeedleId(id),
+                        cookie: 0x1111.into(),
+                        data: vec![id as u8; 8],
+                        data_size: 8,
+                        ..Needle::default()
+                    },
+                    fsync: true,
+                    response_tx,
+                }
+            })
+            .collect();
+        process_batch(state.clone(), batch);
+
+        for mut rx in receivers {
+            assert!(matches!(rx.try_recv().unwrap(), Ok((_, _, false))));
+        }
+        let store = state.store.read().unwrap();
+        let (_, vol) = store.find_volume(VolumeId(1)).unwrap();
+        assert_eq!(vol.sync_counts_for_test(), (1, 1));
+    }
+
     #[tokio::test]
     async fn test_write_queue_dropped_sender() {
         // When the queue is dropped, subsequent submits should fail gracefully.