Two follow-ups on PR #9382:
1. Quarantine wasn't sticky. Once CollectHeartbeat crossed the streak
threshold and hid the replica, a subsequent successful read called
checkReadWriteError(nil), wiping the streak; the next heartbeat
then re-announced the suspect replica as read-only and master could
send reads back to a disk that already failed IoErrorTolerance.
Added an ioErrorQuarantined sticky flag set on the first heartbeat
that observes the threshold and cleared only by MarkVolumeWritable
(resetIoErrorState). clearIoError continues to reset just the
streak so successful ops don't accumulate phantom errors.
2. Streamed reads bypassed the EIO counter. readNeedleDataInto and
ReadNeedleBlob — the hot paths for large/range GETs — returned
ReadNeedleData / needle.ReadNeedleBlob errors without threading
them through checkReadWriteError, so a disk failing only on those
paths would never trip IoErrorTolerance. Both now route the
backend error through the tracker, and a fully clean
readNeedleDataInto call clears the streak.
Tests cover the sticky flag (TestQuarantineIsSticky) and the streamed
read path (TestReadNeedleBlobTracksEIO via a fake EIO backend).
* fix(volume): don't panic on read when needle map is nil
A failed CommitCompact reload (and #9335's new error path for a
remote-tiered volume with a stray .vif but no .idx) leaves v.nm == nil
on a volume that's still in the store. readNeedle / readNeedleDataInto
dereferenced v.nm with no guard, so the next GET segfaulted the
http handler instead of returning an error the client could retry on
another replica.
Add the same v.nm == nil check the other Volume accessors already use,
including the slow-read inner loop where the lock is released between
iterations and a failed reload can race in.
Fixes#9339.
* match rust nm-nil read behavior; trim comments
seaweed-volume's read_needle_with_option / re_lookup_needle_data_offset
already lift Option<NeedleMap> through ok_or(NotFound). Use ErrorNotFound
on the Go side too instead of a generic 500-mapped error so both volume
servers respond identically when v.nm is nil.
* log once when reads hit nil needle map
ErrorNotFound alone hides the real cause: a half-loaded volume just
returns 404s and the operator has nothing to grep for. Add a once-per-
volume Errorf on the nil path, reset on successful load. Mirror the
same in seaweed-volume via nm_or_not_found().
* trim comments
* drop once-flag, log inline on every nil-nm read
* Fix disk errors handling in vacuum compaction
When a disk reports IO errors during vacuum compaction (e.g., 'read /mnt/d1/weed/oc_xyz.dat: input/output error'), the vacuum task should signal the error to the master so it can:
1. Drop the faulty volume replica
2. Rebuild the replica from healthy copies
Changes:
- Add checkReadWriteError() calls in vacuum read paths (ReadNeedleBlob, ReadData, ScanVolumeFile) to flag EIO errors in volume.lastIoError
- Preserve error wrapping using %w format instead of %v so EIO propagates correctly
- The existing heartbeat logic will detect lastIoError and remove the bad volume
Fixes issue #8237
* error
* s3: fix health check endpoints returning 404 for HEAD requests #8243
When a disk reports IO errors during vacuum compaction (e.g., 'read /mnt/d1/weed/oc_xyz.dat: input/output error'), the vacuum task should signal the error to the master so it can:
1. Drop the faulty volume replica
2. Rebuild the replica from healthy copies
Changes:
- Add checkReadWriteError() calls in vacuum read paths (ReadNeedleBlob, ReadData, ScanVolumeFile) to flag EIO errors in volume.lastIoError
- Preserve error wrapping using %w format instead of %v so EIO propagates correctly
- The existing heartbeat logic will detect lastIoError and remove the bad volume
Fixes issue #8237
* simplify a bit
* feat: volume: add "readBufSize" option to customize read optimization
* refactor : redbufSIze -> readBufferSize
* simplify a bit
* simplify a bit
Chris Lu