rust volume: share the I/O-error tracker between Volume and EcVolume (#11351)

* rust volume: share the I/O-error tracker between Volume and EcVolume

Volume and EcVolume each carried the same three fields - a mutex-held
last error, a consecutive count and a sticky quarantine flag - and the
same four methods over them, identical except for the path qualifier on
is_storage_io_error. The tolerance the count is compared against was a
fourth copy: heartbeat.rs held VOLUME_IO_ERROR_TOLERANCE for volumes,
ec_volume.rs held IO_ERROR_TOLERANCE for EC, and the volume test helper
open-coded the same 3, so the two paths could drift apart silently.

Go keeps this in one place already: weed/storage/io_error.go holds
IoErrorTracker, IoErrorTolerance and isStorageIoError, and Volume embeds
the tracker. Go's EcVolume has to re-implement it only because those
fields are unexported and EC lives in another package.

storage::io_error::IoErrorTracker now owns that state, with record /
state / should_quarantine / mark_quarantined / reset and the single
IO_ERROR_TOLERANCE. is_storage_io_error moves into the same file, so it
sits with the tracker that is now its only caller, the way io_error.go
is laid out. Both volume kinds embed one tracker and keep their existing
method names as delegates, so the ~16 internal call sites and the
readers in heartbeat.rs, store.rs and grpc_server.rs change only where
the two threshold comparisons become should_quarantine().

Volume::last_io_error and EcVolume::reset_io_error_state had no callers
and are gone.

Unchanged: what counts as a storage-media error - is_storage_io_error
changed file, not body, and is still the single predicate both volume
kinds share, where Go's EcVolume tests EIO directly and so misses the
Windows codes. Also unchanged: the tolerance value, the metric increment
on every counted error, and the sticky quarantine - a success clears the
count and the last error but never the flag, which only reset lifts. In
the heartbeat the state read moved inside the quarantine branch, so the
common path no longer takes the tracker's mutex or clones the last-error
string; should_quarantine's two relaxed loads run either way.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* rust volume: hoist absolute_display_path into server

handlers.rs and ui.rs each held a byte-identical copy of the helper that
turns a configured -dir into an absolute path for display. The status
JSON and the status page are meant to show the same directory, so the
two copies had to be edited together to stay that way.

The helper now lives in server/mod.rs as pub(crate) and both callers use
it. No behaviour change: same body, same call sites.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* rust volume: keep EcVolume::reset_io_error_state

Moving both volume types onto the shared IoErrorTracker dropped
EcVolume's public reset while Volume kept its own, so the two sides of
the tracker drifted apart.

mark_quarantined is sticky: a later successful read clears the error
count through record(), but the quarantine flag only comes down through
reset(). Without the delegate an EC volume that hit sustained media
errors could not be returned to service in place once the storage was
repaired. Go exposes the same method as EcVolume.ResetIoErrorState
(weed/storage/erasure_coding/ec_volume.go:114).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* rust volume: name the shared tracker after Go's IoErrorTracker

- check_read_write_error, get_io_error_state, mark_io_quarantined,
  reset_io_error_state match weed/storage/io_error.go one to one
- io_error module is pub(crate) like the io module beside it
- restore EcVolume::reset_io_error_state so both volume kinds expose the
  same recovery surface
- trim comments that restate the code

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

seaweed-volume/src/storage/erasure_coding/ec_volume.rs

@@ -13,13 +13,11 @@ use crate::pb::master_pb;
 use crate::storage::erasure_coding::ec_locate;
 use crate::storage::erasure_coding::ec_shard::*;
 use crate::storage::io::read_exact_at;
+use crate::storage::io_error::IoErrorTracker;
 use crate::storage::needle::needle::{Needle, NeedleError, get_actual_size};
 use crate::storage::types::*;
 use crate::storage::volume_open::open_volume_file;
 
-/// An erasure-coded volume managing its local shards and index.
-pub const IO_ERROR_TOLERANCE: i32 = 3;
-
 /// The shard-location cache: where each shard lives, when that was last learned
 /// from the master, and whether a read has since disproved it. One struct behind
 /// one lock, because the freshness heuristic judges all three together (the map
@@ -46,6 +44,7 @@ pub(crate) struct ShardLocationCache {
     stale: bool,
 }
 
+/// An erasure-coded volume managing its local shards and index.
 pub struct EcVolume {
     pub volume_id: VolumeId,
     pub collection: String,
@@ -104,9 +103,9 @@ pub struct EcVolume {
     /// sidecar other than the one those two fields actually hold.
     pub(crate) bitrot_source_dir: String,
 
-    io_error_count: std::sync::atomic::AtomicI32,
-    io_error_quarantined: std::sync::atomic::AtomicBool,
-    last_io_error: std::sync::Mutex<Option<String>>,
+    /// Consecutive storage-media errors and the quarantine they lead to,
+    /// for EC volume health monitoring.
+    io_errors: IoErrorTracker,
 }
 
 /// Locate the `.vif` for a (collection, vid) by preferring the data dir
@@ -459,9 +458,7 @@ impl EcVolume {
             bitrot: None,
             bitrot_status: crate::storage::erasure_coding::ec_bitrot::BitrotStatus::Off,
             bitrot_source_dir: String::new(),
-            io_error_count: std::sync::atomic::AtomicI32::new(0),
-            io_error_quarantined: std::sync::atomic::AtomicBool::new(false),
-            last_io_error: std::sync::Mutex::new(None),
+            io_errors: IoErrorTracker::default(),
         };
 
         // Open .ecx file (sorted index) in read/write mode for in-place deletion marking.
@@ -1006,48 +1003,26 @@ impl EcVolume {
         refresh
     }
 
-    // ---- I/O error tracking (mirrors Go's EcVolume IoErrorTracker) ----
+    // ---- I/O error tracking ----
 
     pub fn check_read_write_error(&self, err: Option<&io::Error>) {
-        use std::sync::atomic::Ordering;
-        if let Some(e) = err
-            && crate::storage::volume::is_storage_io_error(e)
-        {
-            self.io_error_count.fetch_add(1, Ordering::Relaxed);
-            if let Ok(mut guard) = self.last_io_error.lock() {
-                *guard = Some(e.to_string());
-            }
-            crate::metrics::STORAGE_IO_ERROR_COUNTER.inc();
-            return;
-        }
-        self.io_error_count.store(0, Ordering::Relaxed);
-        if let Ok(mut guard) = self.last_io_error.lock()
-            && guard.is_some()
-        {
-            *guard = None;
-        }
+        self.io_errors.check_read_write_error(err);
     }
 
     pub fn get_io_error_state(&self) -> (Option<String>, i32, bool) {
-        use std::sync::atomic::Ordering;
-        let err = self.last_io_error.lock().ok().and_then(|g| g.clone());
-        let count = self.io_error_count.load(Ordering::Relaxed);
-        let quarantined = self.io_error_quarantined.load(Ordering::Relaxed);
-        (err, count, quarantined)
+        self.io_errors.get_io_error_state()
+    }
+
+    pub fn should_quarantine(&self) -> bool {
+        self.io_errors.should_quarantine()
     }
 
     pub fn mark_io_quarantined(&self) {
-        self.io_error_quarantined
-            .store(true, std::sync::atomic::Ordering::Relaxed);
+        self.io_errors.mark_io_quarantined();
     }
 
     pub fn reset_io_error_state(&self) {
-        use std::sync::atomic::Ordering;
-        self.io_error_count.store(0, Ordering::Relaxed);
-        self.io_error_quarantined.store(false, Ordering::Relaxed);
-        if let Ok(mut guard) = self.last_io_error.lock() {
-            *guard = None;
-        }
+        self.io_errors.reset_io_error_state();
     }
 
     // ---- Index operations ----