rust volume: share the I/O-error tracker between Volume and EcVolume (#11351)

* rust volume: share the I/O-error tracker between Volume and EcVolume

Volume and EcVolume each carried the same three fields - a mutex-held
last error, a consecutive count and a sticky quarantine flag - and the
same four methods over them, identical except for the path qualifier on
is_storage_io_error. The tolerance the count is compared against was a
fourth copy: heartbeat.rs held VOLUME_IO_ERROR_TOLERANCE for volumes,
ec_volume.rs held IO_ERROR_TOLERANCE for EC, and the volume test helper
open-coded the same 3, so the two paths could drift apart silently.

Go keeps this in one place already: weed/storage/io_error.go holds
IoErrorTracker, IoErrorTolerance and isStorageIoError, and Volume embeds
the tracker. Go's EcVolume has to re-implement it only because those
fields are unexported and EC lives in another package.

storage::io_error::IoErrorTracker now owns that state, with record /
state / should_quarantine / mark_quarantined / reset and the single
IO_ERROR_TOLERANCE. is_storage_io_error moves into the same file, so it
sits with the tracker that is now its only caller, the way io_error.go
is laid out. Both volume kinds embed one tracker and keep their existing
method names as delegates, so the ~16 internal call sites and the
readers in heartbeat.rs, store.rs and grpc_server.rs change only where
the two threshold comparisons become should_quarantine().

Volume::last_io_error and EcVolume::reset_io_error_state had no callers
and are gone.

Unchanged: what counts as a storage-media error - is_storage_io_error
changed file, not body, and is still the single predicate both volume
kinds share, where Go's EcVolume tests EIO directly and so misses the
Windows codes. Also unchanged: the tolerance value, the metric increment
on every counted error, and the sticky quarantine - a success clears the
count and the last error but never the flag, which only reset lifts. In
the heartbeat the state read moved inside the quarantine branch, so the
common path no longer takes the tracker's mutex or clones the last-error
string; should_quarantine's two relaxed loads run either way.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* rust volume: hoist absolute_display_path into server

handlers.rs and ui.rs each held a byte-identical copy of the helper that
turns a configured -dir into an absolute path for display. The status
JSON and the status page are meant to show the same directory, so the
two copies had to be edited together to stay that way.

The helper now lives in server/mod.rs as pub(crate) and both callers use
it. No behaviour change: same body, same call sites.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* rust volume: keep EcVolume::reset_io_error_state

Moving both volume types onto the shared IoErrorTracker dropped
EcVolume's public reset while Volume kept its own, so the two sides of
the tracker drifted apart.

mark_quarantined is sticky: a later successful read clears the error
count through record(), but the quarantine flag only comes down through
reset(). Without the delegate an EC volume that hit sustained media
errors could not be returned to service in place once the storage was
repaired. Go exposes the same method as EcVolume.ResetIoErrorState
(weed/storage/erasure_coding/ec_volume.go:114).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* rust volume: name the shared tracker after Go's IoErrorTracker

- check_read_write_error, get_io_error_state, mark_io_quarantined,
  reset_io_error_state match weed/storage/io_error.go one to one
- io_error module is pub(crate) like the io module beside it
- restore EcVolume::reset_io_error_state so both volume kinds expose the
  same recovery surface
- trim comments that restate the code

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

seaweed-volume/src/server/handlers.rs

@@ -15,6 +15,7 @@ use axum::http::{HeaderMap, Method, Request, StatusCode, header};
 use axum::response::{IntoResponse, Response};
 use serde::{Deserialize, Serialize};
 
+use super::absolute_display_path;
 use super::grpc_client::{GRPC_MAX_MESSAGE_SIZE, build_grpc_endpoint};
 use super::volume_server::{VolumeServerState, normalize_outgoing_http_url, to_http_address};
 use crate::config::ReadMode;
@@ -3621,16 +3622,6 @@ async fn read_remote_chunk_needle(
 // Helpers
 // ============================================================================
 
-fn absolute_display_path(path: &str) -> String {
-    let p = std::path::Path::new(path);
-    if p.is_absolute() {
-        return path.to_string();
-    }
-    std::env::current_dir()
-        .map(|cwd| cwd.join(p).to_string_lossy().to_string())
-        .unwrap_or_else(|_| path.to_string())
-}
-
 fn build_disk_statuses(store: &crate::storage::store::Store) -> Vec<serde_json::Value> {
     let mut disk_statuses = Vec::new();
     for loc in &store.locations {

seaweed-volume/src/server/heartbeat.rs

@@ -24,7 +24,6 @@ use crate::storage::volume_report::VolumeReportKey;
 use crate::storage::volume_report_hash::report_hash;
 
 const DUPLICATE_UUID_RETRY_MESSAGE: &str = "duplicate UUIDs detected, retrying connection";
-const VOLUME_IO_ERROR_TOLERANCE: i32 = 3;
 const MAX_DUPLICATE_UUID_RETRIES: u32 = 3;
 
 /// Configuration for the heartbeat client.
@@ -952,8 +951,8 @@ fn build_heartbeat_with_ec_status(
             let volume_size = vol.dat_file_size().unwrap_or(0);
             let mut should_delete_volume = false;
 
-            let (_, io_count, io_quarantined) = vol.get_io_error_state();
-            if io_quarantined || io_count >= VOLUME_IO_ERROR_TOLERANCE {
+            if vol.should_quarantine() {
+                let (_, io_count, io_quarantined) = vol.get_io_error_state();
                 if !io_quarantined {
                     vol.mark_io_quarantined();
                     warn!(

seaweed-volume/src/server/mod.rs

@@ -38,6 +38,19 @@ pub fn status_with_context(context: &str, err: VolumeError) -> Status {
     Status::new(status.code(), format!("{context}: {}", status.message()))
 }
 
+/// Render a configured disk directory as an absolute path for display, so the
+/// status JSON and the UI show the same thing for a relative `-dir`. Falls
+/// back to the configured spelling when the current directory cannot be read.
+pub(crate) fn absolute_display_path(path: &str) -> String {
+    let p = std::path::Path::new(path);
+    if p.is_absolute() {
+        return path.to_string();
+    }
+    std::env::current_dir()
+        .map(|cwd| cwd.join(p).to_string_lossy().to_string())
+        .unwrap_or_else(|_| path.to_string())
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;

seaweed-volume/src/server/ui.rs

@@ -1,5 +1,6 @@
 use std::fmt::Write as _;
 
+use crate::server::absolute_display_path;
 use crate::server::server_stats;
 use crate::server::volume_server::VolumeServerState;
 use crate::storage::store::Store;
@@ -450,16 +451,6 @@ fn collect_ui_data(
     (disk_rows, volumes, remote_volumes, ec_volumes)
 }
 
-fn absolute_display_path(path: &str) -> String {
-    let p = std::path::Path::new(path);
-    if p.is_absolute() {
-        return path.to_string();
-    }
-    std::env::current_dir()
-        .map(|cwd| cwd.join(p).to_string_lossy().to_string())
-        .unwrap_or_else(|_| path.to_string())
-}
-
 fn join_i64(values: &[i64]) -> String {
     values
         .iter()