Two follow-ups on PR #9382:
1. Quarantine wasn't sticky. Once CollectHeartbeat crossed the streak
threshold and hid the replica, a subsequent successful read called
checkReadWriteError(nil), wiping the streak; the next heartbeat
then re-announced the suspect replica as read-only and master could
send reads back to a disk that already failed IoErrorTolerance.
Added an ioErrorQuarantined sticky flag set on the first heartbeat
that observes the threshold and cleared only by MarkVolumeWritable
(resetIoErrorState). clearIoError continues to reset just the
streak so successful ops don't accumulate phantom errors.
2. Streamed reads bypassed the EIO counter. readNeedleDataInto and
ReadNeedleBlob — the hot paths for large/range GETs — returned
ReadNeedleData / needle.ReadNeedleBlob errors without threading
them through checkReadWriteError, so a disk failing only on those
paths would never trip IoErrorTolerance. Both now route the
backend error through the tracker, and a fully clean
readNeedleDataInto call clears the streak.
Tests cover the sticky flag (TestQuarantineIsSticky) and the streamed
read path (TestReadNeedleBlobTracksEIO via a fake EIO backend).
* fix(volume): don't nuke local data on transient IO error (#9378)
A single syscall.EIO from any read/write/delete set v.lastIoError, and
the next CollectHeartbeat then called Volume.Destroy on the replica —
removing the .dat/.idx/.vif/.sdx/.ldb/.rdb files. A brief NFS / fabric
/ controller blip hitting several replicas at once could cascade into
removal of the last healthy copy, with no recovery for non-tiered
volumes.
Now require IoErrorTolerance (3) consecutive EIOs before acting, and on
that threshold mark the volume read-only and stop announcing it to the
master so re-replication kicks in from healthy peers — never delete
the data files. The on-disk copy stays for operator inspection /
recovery.
* review: fix race, accounting, recovery, non-EIO streak break
Addressing PR #9382 review:
- Data race on lastIoError: guard lastIoError + lastIoErrorCount with a
RWMutex and expose them through note/clear/get helpers so the
heartbeat reader sees a consistent snapshot. Verified with -race.
- Collection-size accounting: when a volume is quarantined for sustained
EIO, skip the entire per-volume bookkeeping (`continue`) instead of
flipping shouldDeleteVolume — the old branch subtracted a size that
was never added, dragging the collection gauge to zero / negative.
- Recoverability: MarkVolumeWritable now also calls clearIoError so an
operator can rejoin a quarantined replica. The next failed op
re-arms the streak if the disk is still bad.
- Non-EIO streak break: a non-EIO error (e.g. ENOSPC) now resets the
consecutive-EIO counter, so a sequence EIO,EIO,ENOSPC,EIO is treated
as a streak of one — the counter only tracks consecutive EIOs.
Reads already call checkReadWriteError (volume_read.go), so successful
reads also clear the streak — no change needed there.
Chris Lu