2 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Tim Olowandtimolow	331c6c3642	shell: volume.check.disk — actionable verdict for diverged vacuumed replicas (#11197) ... * shell: volume.check.disk — actionable verdict for diverged vacuumed replicas When -resurrectMissingNeedles is gated off because both replicas have been vacuumed (compaction revision > 0) — the normal state of any production cluster — check.disk previously stopped at 'cannot prove they are missing writes vs vacuumed deletes' and did nothing, leaving a diverged replica with no repair path. volume.fix.replication does not catch it either: it only acts when the replica COUNT is below the expected replication, never when two replicas are both present but hold different live data. Classify the divergence instead of dead-ending: liveDivergence() counts live (non-deleted) needles present on one replica's index but entirely absent from the other, in both directions. Tombstones are excluded, so vacuum asymmetry (a compacted replica that dropped deleted entries) is not mistaken for divergence. reportDivergenceVerdict() turns the count into an operator action: - one-sided (one replica has all the live data, the other has no unique live needles) -> print the exact safe repair: volume.copy -source <complete> -target <lagging> -volumeId <id> Re-copying the complete replica is safe precisely because the lagging side holds no unique live data; VolumeCopy's verify-before-destroy gate independently confirms the source holds the volume before deleting the target. - two-sided (split-brain, both sides have unique live data) -> warn and do NOT emit an auto repair; point to volume.fsck -findMissingChunksInFiler to confirm the 'missing' needles are orphans before converging. Report-only: no data is modified and the resurrection safety gate is untouched. This is what lets a 13-volume diverged cluster be diagnosed and repaired in minutes instead of by hand-diffing every index. Observed motivating case: home SeaweedFS 4.45 cluster, 13 010 cross-rack volumes diverged after failed replicate writes (ReplicatedWrite MaxAttempts=1 fire-and-forget), 11 one-sided + 2 two-sided, all repaired via volume.copy. * shell: volume.check.disk — address review nits on divergence verdict - Use pb.NewServerAddressFromDataNode (dialable ip:port, Address with Id fallback) for the advertised volume.copy -source/-target instead of the logical node Id, which may not be dialable. - Make the one-sided verdict tombstone-aware: when the lagging replica has been vacuumed, absent live needles may be valid deletions whose tombstones were dropped, so a whole-volume re-copy would resurrect them. The command is only advertised as safe when the lagging side is proven never-vacuumed (compaction revision 0 read under -resurrectMissingNeedles); otherwise a caveat is printed pointing at fsck/needle-level repair. - Fix reversed copy direction when the source replica is the lagging one (must copy complete -> lagging in both cases). - Test: real tombstone (negative size) with the correct 0/0 expectation and || assertion; verdict test now covers dialable address, corrected direction, and caveat on/off. * shell: volume.check.disk — per-replica revision knowledge, no copy command for vacuumed lagging side - Track srcRevKnown/tgtRevKnown separately: in unidirectional mode the target revision IS read, so a proven never-vacuumed target no longer gets a false resurrection warning (regression: bidi=false, target rev 0). - A one-sided verdict now only emits the volume.copy command when the lagging side is proven never-vacuumed; when it is vacuumed (or unproven) the verdict refuses to print the destructive command and points at fsck/needle-level repair instead — an appended caveat next to a ready-to- paste copy command was still inviting the resurrection. - deletionCaveat now returns the boolean safety decision. * shell: volume.check.disk — preserve gRPC port, proven two-sided is not split-brain - Emit the raw ServerAddress string (host:port.grpcPort) instead of String()/ToHttpAddress(), which drops the custom gRPC port and would make the suggested volume.copy dial the default port and fail. - Two-sided divergence with both replicas proven never-vacuumed under -resurrectMissingNeedles is mutually missed writes, not split-brain: recommend re-running with -apply (in-place resurrection both directions) instead of the split-brain no-auto-repair warning. - Tests: case F (proven two-sided -> -apply, no split-brain warning), case G (custom gRPC ports preserved in emitted addresses). --------- Co-authored-by: timolow <tim@timolow.dev>	2026-09-06 23:25:28 -07:00
4 people	ab79d1f680	operation: re-assign chunk upload when replica volume is full (#10588) ... * operation: re-assign chunk upload when replica volume is full When a volume reaches MaxPossibleVolumeSize, needle writes return 'Volume Size Exceeded' and the fan-out in uploadChunkToHolders fails. Previously the error propagated immediately, killing the entire chunked upload even though the master has other writable volumes. Fix: detect 'Volume Size' errors on the fan-out path, call AssignFunc again to get a fresh volume, and retry (up to 3 attempts). This avoids backup failures while a single replica volume is full and waiting for GC. Also add 'volume size exceeded' to the transient error messages so any retry path that checks IsTransientError recognises it. * util: fix transient error pattern for volume size errors The actual error message is 'Volume Size 34361499680 Exceeded 34359738368' where the numeric size separates 'Volume Size' from 'Exceeded'. The previous pattern 'volume size exceeded' would never match. Change to 'volume size' which correctly matches any capacity-full error. * operation: fix reassignment loop nits - Case-insensitive volume size match (strings.ToLower) - Propagate AssignFunc error so caller sees reassignment failure - Reset JWT fallback before each reassignment to avoid retaining stale auth * util: stop classifying a full volume as a transient error A volume at capacity does not become writable on the next attempt, so the entry only bought a retry loop's worth of sleeping before the same failure. It also reached four consumers that all retry the same target — the deletion queue, the replication sink, volume lookups, and Retry/MultiRetry — none of which reassign, and the widened "volume size" substring swallowed the replica-receive rejection from WriteNeedleBlob too. The chunked upload path recovers by asking for a different volume instead. * operation: reassign a chunk with the shared upload gate shouldReassignUpload already answers this question for the non-chunked path, keyed off the status the volume server returned rather than its message text. Reusing it covers a lost replica peer and an unreachable target as well as a full volume, and it stops a 4xx from being retried on a second volume that would reject it identically. Pulling the single attempt out into uploadChunk keeps the retry loop readable now that it wraps both the fan-out and the relay path. * test: cover chunk reassignment across volumes Pins the three outcomes the gate decides: a full volume moves the chunk to a fresh assignment, a 4xx stays put, and the loop gives up after chunkAssignAttempts volumes. * operation: roll back the fid a reassigned chunk abandons ReplicatedWrite commits the needle locally before it replicates, so a 5xx can leave a copy behind on a volume the chunk is about to walk away from. Nothing will ever reference that fid, and an unreferenced needle is not garbage vacuum can find — it is dead space until the volume is destroyed. uploadChunkToHolders rolls back only the holders that reported success, which misses the one whose write landed but whose response did not, and the relay path had no rollback at all. Delete from every holder of the abandoned assignment instead; deleting a needle that never landed is a no-op. * operation: stop the reassignment loop from multiplying work retriedUploadData already retries a chunk three times against the same URL, so wrapping it in three assignments made nine POSTs for one chunk. On the relay path that inner retry is redundant — the loop retries everything it would, and on a different volume — so cap it at one attempt per assignment and leave the budget where it was. The fan-out path keeps its inner retries: absorbing a blip on one holder beats cancelling the rest and re-uploading the whole chunk. Nothing bounded any of it by time. weed/s3api passes context.Background() so a chunk survives client disconnect, which also means no deadline cuts the loop short, and a chunk goroutine holds one of four buffer slots while it spins. Break out once another chunk has already failed the object. * operation: keep the reassignment gate's inputs deterministic uploadChunkToHolders reported whichever holder error won the channel race. That was cosmetic while the value was only logged; now it decides whether the chunk moves to another volume, so a 400 and a 500 arriving in either order made the retry behavior depend on scheduling. Prefer an error the caller can act on, and the same failure always retries the same way. A failed reassignment also overwrote the upload error that prompted it, which buried a full volume behind whatever the filer happened to say. Keep both in the chain. The tests grew a JWT per assignment, since the loop re-derives one and nothing covered it, and the bound is now spelled out rather than compared against the constant that defines it. * operation: roll back the last abandoned fid too The rollback ran only on the path that goes on to reassign, so the attempt that exhausts the budget — or stops because another chunk already failed the object, or because the error is not one a different volume fixes — left its fid behind. That is the case that matters most: no chunk names it, the caller gets no fid to clean up, and a 5xx can still mean the needle was committed. Roll back on every failed attempt instead, before deciding whether to retry. --------- Co-authored-by: timolow <timolow@users.noreply.github.com> Co-authored-by: timolow <tim@timolow.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-05 17:43:37 -07:00