3 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	12dac68d66	seaweed-volume: replace deprecated AtomicU64::fetch_update with try_update	2026-10-30 01:37:42 -07:00
3 people	bd953b0f84	volume server: group-commit fsync writes in the write queue (#11543) ... * volume: split the write path into reusable steps do_write_request ran its pre-append checks, the append, the sync rollback, the index publish and the post-write bookkeeping inline, so a batched write could only reuse it one needle at a time. Pull the steps out (check_writable, prepare_write, undo_unsynced_append, publish_write, finish_write) and the store's volume lookup plus disk-space check (writable_volume_mut). do_write_request composes them in the same order with the same early returns; no behaviour change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: group-commit fsync writes in the write queue The write queue holds one store lock for a batch of up to 128 needles but wrote them one at a time, so every fsync needle paid its own .dat sync and its own .idx sync: 2N syncs per batch. Add Volume::write_needles_grouped, after Go's processBatch. A volume's entries are split into runs of distinct needle ids (a repeated id starts a new run, so its dedup and cookie checks see the earlier write). A run with a durable entry appends everything with append_at_ns chained through a local, syncs the .dat once, and only then publishes the entries and syncs the .idx once. A failed .dat sync truncates the .dat back to the run start (marking the volume unavailable if that fails), leaves last_append_at_ns and last_modified untouched, and fails every entry of the run. Runs with no durable entry go through the unchanged per-needle path. Store::write_volume_needles is the queue's entry point; the handlers' non-queue path is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: fail closed when a failed append's rollback fails append_needle discarded the truncate-back result, so a partial write that could not be rolled back left unindexed bytes on the .dat while the volume stayed writable; the next append would bury them mid-file, past the load-time tail check. Route the rollback through undo_unsynced_append, which marks the volume unavailable when the truncate fails, so nothing more is appended over an unverified tail. * volume server: keep a grouped run's I/O error streak from later appends A synced run stages every append before any entry finishes, so the success reset in finish_write ran after the failed appends queued behind the last write to land and erased their media-error streak. Sent one at a time, those errors would have counted and quarantined the volume. Skip the reset when an append after the last landed write added to the streak. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: replay a grouped run's I/O error streak in queue order Skipping the run's success reset whenever an append after the last landed write failed kept the errors from before that write as well, so a run like [EIO, EIO, landed, EIO] reached the quarantine count that the same writes one at a time (one error) do not. Mark the streak where each entry is staged and record the run's success at the last landed write's mark: errors before it are cleared, the ones after it still count. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * volume server: replay a run's I/O error streak in one atomic step Reads record their outcomes on the tracker without the volume's write lock, so record_success_at's separate load and store could drop an error a read counted in between, or restore errors a read had just cleared. Keep the count and the clear counter in one atomic word and apply the replay with a single fetch_update. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-10-01 20:59:40 +08:00
4 people	818f3bb71b	rust volume: share the I/O-error tracker between Volume and EcVolume (#11351) ... * rust volume: share the I/O-error tracker between Volume and EcVolume Volume and EcVolume each carried the same three fields - a mutex-held last error, a consecutive count and a sticky quarantine flag - and the same four methods over them, identical except for the path qualifier on is_storage_io_error. The tolerance the count is compared against was a fourth copy: heartbeat.rs held VOLUME_IO_ERROR_TOLERANCE for volumes, ec_volume.rs held IO_ERROR_TOLERANCE for EC, and the volume test helper open-coded the same 3, so the two paths could drift apart silently. Go keeps this in one place already: weed/storage/io_error.go holds IoErrorTracker, IoErrorTolerance and isStorageIoError, and Volume embeds the tracker. Go's EcVolume has to re-implement it only because those fields are unexported and EC lives in another package. storage::io_error::IoErrorTracker now owns that state, with record / state / should_quarantine / mark_quarantined / reset and the single IO_ERROR_TOLERANCE. is_storage_io_error moves into the same file, so it sits with the tracker that is now its only caller, the way io_error.go is laid out. Both volume kinds embed one tracker and keep their existing method names as delegates, so the ~16 internal call sites and the readers in heartbeat.rs, store.rs and grpc_server.rs change only where the two threshold comparisons become should_quarantine(). Volume::last_io_error and EcVolume::reset_io_error_state had no callers and are gone. Unchanged: what counts as a storage-media error - is_storage_io_error changed file, not body, and is still the single predicate both volume kinds share, where Go's EcVolume tests EIO directly and so misses the Windows codes. Also unchanged: the tolerance value, the metric increment on every counted error, and the sticky quarantine - a success clears the count and the last error but never the flag, which only reset lifts. In the heartbeat the state read moved inside the quarantine branch, so the common path no longer takes the tracker's mutex or clones the last-error string; should_quarantine's two relaxed loads run either way. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * rust volume: hoist absolute_display_path into server handlers.rs and ui.rs each held a byte-identical copy of the helper that turns a configured -dir into an absolute path for display. The status JSON and the status page are meant to show the same directory, so the two copies had to be edited together to stay that way. The helper now lives in server/mod.rs as pub(crate) and both callers use it. No behaviour change: same body, same call sites. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * rust volume: keep EcVolume::reset_io_error_state Moving both volume types onto the shared IoErrorTracker dropped EcVolume's public reset while Volume kept its own, so the two sides of the tracker drifted apart. mark_quarantined is sticky: a later successful read clears the error count through record(), but the quarantine flag only comes down through reset(). Without the delegate an EC volume that hit sustained media errors could not be returned to service in place once the storage was repaired. Go exposes the same method as EcVolume.ResetIoErrorState (weed/storage/erasure_coding/ec_volume.go:114). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * rust volume: name the shared tracker after Go's IoErrorTracker - check_read_write_error, get_io_error_state, mark_io_quarantined, reset_io_error_state match weed/storage/io_error.go one to one - io_error module is pub(crate) like the io module beside it - restore EcVolume::reset_io_error_state so both volume kinds expose the same recovery surface - trim comments that restate the code Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-20 16:57:23 -07:00