* fix(volume): don't nuke local data on transient IO error (#9378)
A single syscall.EIO from any read/write/delete set v.lastIoError, and
the next CollectHeartbeat then called Volume.Destroy on the replica —
removing the .dat/.idx/.vif/.sdx/.ldb/.rdb files. A brief NFS / fabric
/ controller blip hitting several replicas at once could cascade into
removal of the last healthy copy, with no recovery for non-tiered
volumes.
Now require IoErrorTolerance (3) consecutive EIOs before acting, and on
that threshold mark the volume read-only and stop announcing it to the
master so re-replication kicks in from healthy peers — never delete
the data files. The on-disk copy stays for operator inspection /
recovery.
* review: fix race, accounting, recovery, non-EIO streak break
Addressing PR #9382 review:
- Data race on lastIoError: guard lastIoError + lastIoErrorCount with a
RWMutex and expose them through note/clear/get helpers so the
heartbeat reader sees a consistent snapshot. Verified with -race.
- Collection-size accounting: when a volume is quarantined for sustained
EIO, skip the entire per-volume bookkeeping (`continue`) instead of
flipping shouldDeleteVolume — the old branch subtracted a size that
was never added, dragging the collection gauge to zero / negative.
- Recoverability: MarkVolumeWritable now also calls clearIoError so an
operator can rejoin a quarantined replica. The next failed op
re-arms the streak if the disk is still bad.
- Non-EIO streak break: a non-EIO error (e.g. ENOSPC) now resets the
consecutive-EIO counter, so a sequence EIO,EIO,ENOSPC,EIO is treated
as a streak of one — the counter only tracks consecutive EIOs.
Reads already call checkReadWriteError (volume_read.go), so successful
reads also clear the streak — no change needed there.
