mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-20 13:30:46 +02:00
c2cdefd06d89f4510dfe06508eee7ea5aa49d607
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Lu
|
23241cf0f1 |
Let filer.sync move past a chunk the source cluster no longer has (#11019)
* Name the failure when the source cluster cannot locate a chunk's volume LookupFileId formatted a nil err into the message it returned, so the only thing a caller could do with "no locations for this volume" was match on the text. Return a typed error instead. Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK * Fail a source chunk read on a failure status instead of copying the error page ReadPart never looked at the response status, so a volume server answering 404 for a needle vacuum had removed came back as a successful read whose body was the error page. The caller counted those bytes as file content and reported a size mismatch — a corruption claim about data the source had simply lost — and a 404 from one replica ended the search instead of trying the next. Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK * Stop retrying a chunk the source cluster can no longer produce A chunk whose volume vacuum has removed fails the same way on every attempt, but the retry loop had no way to say so and kept going forever. The sync job holding it never finished, so it pinned the offset watermark at the event ahead of it and filer.sync never checkpointed again — alive, quiet, and permanently behind. Wait the source out for a grace period long enough to cover a volume server restart or a master failover, then give up and mark the failure permanent. Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK * Let replication continue past an entry whose source data is gone An entry the source can no longer read holds the sync offset forever: the event fails on every replay, so the checkpoint never moves past it and every later event stays uncheckpointed, however long the sync keeps running. Nothing brings those bytes back, so skip the entry with an error naming it and carry on. Skip only while the source is demonstrably still serving other chunks. A volume with no locations reads the same whether it was vacuumed away or every replica is down, and during a cluster-wide outage that answer comes back for every chunk — skipping then would drop live files wholesale. Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK * Propagate a missing source chunk instead of waiting when supersession is unverifiable An incremental sink's dated target keys cannot be mapped back to a source path, so nothing here can tell a chunk the source lost from one a later version already replaced. Waiting out the grace period would stall every vacuumed needle for half an hour; hand the failure to the caller, which has the event's real source key. Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK * Wait out a gone volume once, not once per file it held A volume vacuum removed took every file it held with it, and each chunk was timing its own grace period. With a bounded chunk executor those waits serialize, so one gone volume holding many files stalls the sync for far longer than the grace period — the wedge again, only slower. Track the wait per source volume on the sink instead: the first chunk to find it unlocatable starts the clock, every later chunk inherits it and gives up as soon as it has run out, and a chunk the source does serve clears it. Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK * Probe the source with a read, not a lookup, before writing an entry off A lookup only proves the source master still has the topology. If every volume server is unreachable while the master still lists them, the probe passed and the sink wrote off an entry whose data was merely out of reach. Read the probe chunk instead, and say in the log that the entry stays unreplicated. Claude-Session: https://claude.ai/code/session_01SRPEP4jRu29FbLSN6bjLaK |
||
|
Jaehoon KimandClaude Fable 5
|
c17aebeecf |
fix(filer.backup): prevent silent backup data loss on transient not-found (#10295)
* fix(filer.backup): stop silently dropping events on transient not-found Under a write burst, filer.backup could consume a metadata event (advancing the persisted offset) without replicating the file, with no error logged: 1. filersink CreateEntry/UpdateEntry swallowed replicateChunks errors (glog.Warningf + return nil), so the offset advanced past entries that were never written. 2. The manifest-chunk branch of replicateChunks resolved via LookupFileId with no retry, unlike the data-chunk branch — transient lookup races dropped exactly the large manifest-backed files while small inline-content siblings landed. 3. isIgnorable404 matched "LookupFileId" / "volume id ... not found", misclassifying those races as genuine source 404s at the backup layer. Fix: on a replicateChunks failure the filer sink now skips only when the live source has moved past the replayed version (deleted or strictly-newer mtime) — lossless, a later event carries the current content — and propagates otherwise so the event is retried. The manifest resolve retries transient errors like the data-chunk path. isIgnorable404 is narrowed to genuine 404s; non-filer sinks and the initial-snapshot walk, which relied on the broad match as their only lossless-skip valve, now make the same live-source decision (filersink.SourceSupersedes) instead of retrying forever on a permanently gone volume. Tests cover propagation of unconfirmed lookup failures, the narrowed 404 classification, and the supersession guards. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(filer.backup): derive supersession path with directory fallback eventSourceSuperseded built the source path from NewParentPath alone. Legacy metadata events (persisted by older filers) carry an empty NewParentPath, so the probe looked up "/<name>", read the miss as "source gone", and skipped a live file on a transient lookup error — the silent drop this change is meant to eliminate. Derive the path via MetadataEventTargetFullPath (the same directory fallback genProcessFunction uses) and cover both event shapes with TestEventSupersessionProbe_PathDerivation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(filersink): retry manifest resolve only for transient errors, bounded when unverifiable The manifest-resolve retry stopped only when hasSourceNewerVersion proved the source moved past the replayed version, which wedged the sink in two cases: incremental sinks use dated target keys that cannot map back to a source path (supersession never provable), and permanent resolve errors (corrupt manifest data, bad file ids) fail forever while the source entry stays live. Gate the retry instead: keep retrying only transient errors (volume-lookup races, network interruptions), stop after a few attempts when supersession cannot be checked, and propagate everything else immediately so the configured metadata error policy applies (-disableErrorRetry included). Propagation is lossless: filer.backup's fallback decides with the event's real source key, and both filer.backup and filer.sync re-deliver the event (RetryForeverOnError) without advancing the offset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(filer.backup): make error classifiers nil-safe isIgnorable404, isSourceLookupError, and isTransientResolveError called err.Error() without a nil guard. All current call sites pass a non-nil error, but the guard is free and matches isRetryableNetworkError. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Jaehoon KimandChris Lu
|
202517c02a |
fix(filer.backup): skip replay events whose source chunk was superseded or deleted (#9886)
* fix(filer.backup): skip replay events whose chunk no longer exists on the source "Source" is the filer we replicate FROM (e.g. green in a green->blue backup). Replaying the metadata log from a checkpoint can hit an event whose chunk was since overwritten/deleted and garbage-collected on the source volume. Fetching it returns 0 bytes (a permanent size mismatch), which the sink propagated to the subscription — so the same offset retried forever and replication stalled. Skip the event only when proven stale; otherwise keep refusing so genuine loss of a live file still halts loudly: - onCorruptChunk centralizes the three errChunkSizeMismatch sites. - getEntryMtimeNs compares mtime at nanosecond precision so same-second rewrites (git's config.lock dance) are ordered correctly. - sourceSupersedes re-reads the entry's current state on the source: gone (ErrNotFound) or a strictly-newer mtime than the replayed version -> skip; any other lookup error keeps the entry. Skipping is lossless: events are full-entry snapshots, so a later event re-carries the current chunks and a delete event reconciles a removed file. * test(filer.backup): cover the superseded-chunk skip decision - TestSourceSupersedes: not-found (sentinel / wrapped / gRPC string) and nil entry -> skip; network error -> keep; source newer -> skip; same/older -> keep. - TestGetEntryMtimeNs: nanosecond precision, same-second ordering, nil safety. - TestOnCorruptChunkRefusesWhenSupersessionUnconfirmed: never skip silently when supersession cannot be confirmed. * fix(filer.backup): don't infer supersession for incremental sinks In incremental mode the sink key carries a date prefix (sinkDir/YYYY-MM-DD/relPath) that cannot be reversed to a real source path, so a source lookup would always be ErrNotFound and wrongly classify a live entry as deleted — skipping it. Make targetPathToSourcePath report "unmappable" in incremental mode; hasSourceNewerVersion already declines to skip when the source path cannot be mapped. Found in code review. Non-incremental sinks (filer.backup green->blue) are unaffected. * refactor(filer.backup): name the mtime param sourceMtimeNs; note ns overflow bound - Rename the threaded sourceMtime parameter to sourceMtimeNs across the internal replicate/fetch helpers so the unit is explicit (it only feeds hasSourceNewerVersion, which compares in nanoseconds). - Document that getEntryMtimeNs's int64 ns arithmetic is safe until ~year 2262. No behavior change. * fix(filer.backup): order same-second versions in the CreateEntry skip and update gates The CreateEntry already-replicated short-circuit and chooseUpdateAction still compared second-grained mtime, so a newer version written within the same second could be skipped as already-replicated or overwritten by an older same-second replay. Route both through getEntryMtimeNs, matching the precision the chunk-replication path already uses. * test(filer.backup): cover same-second update-action ordering * docs(filer.backup): trim verbose comments to terse why * fix(filer.backup): check supersession against the rename's new path For a rename the filer sink updates in place (the delete+create branch is skipped for sink name "filer"), so the corrupt-chunk supersession check queried the pre-rename key. Its source-side ErrNotFound was read as "superseded", silently advancing the checkpoint without applying the rename. Map the incoming entry's new path (newParentPath/newEntry.Name) for both update branches. * fix(filer.backup): detect a deleted source even when the replayed mtime is epoch hasSourceNewerVersion returned early when sourceMtimeNs <= 0, skipping the source lookup, so a deleted entry with mtime 0 (a valid epoch timestamp) never got the gone verdict and wedged on permanent retries. Always look up; gate only the newer-mtime comparison on a valid replayed mtime. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
2e3fabbf24 |
filer.sync: back off on transient upload errors (#9777)
A destination volume server that hits its idle deadline while reading a large upload body under load returns 400 "read tcp ...: i/o timeout". fetchAndWrite retried that on the flat ~1s RetryUntil backoff, hammering the already-overloaded destination. Route i/o timeout, connection reset, broken pipe and net.Error timeouts through the same escalating 10s-2min backoff already used for EOF so it can recover. |
||
|
|
675020b342 |
fix(filer.sync): validate chunk size in FilerSink to prevent 0-byte propagation (#9701)
* fix(filer.sync): validate chunk size in FilerSink to prevent 0-byte propagation
FilerSink.fetchAndWrite previously trusted the source response and the
upload result blindly: a 200 OK / Content-Length: 0 reply from a broken
source volume was happily uploaded as a 0-byte needle to the destination,
and the destination filer metadata was then written with the source
chunk size. The result was permanent silent corruption -- ls shows the
file at its original size but reads fail with EIO.
Add two cheap defenses inside fetchAndWrite:
1. After assembling fullData, compare its length against sourceChunk.Size.
2. After a successful upload, compare uploadResult.Size against
sourceChunk.Size.
Both checks wrap a new sentinel errChunkSizeMismatch that the retry
callback recognizes and refuses to retry -- needle.size=0 on disk is a
persistent state, not a transient network error, so the sync should stop
loudly on the affected entry instead of looping or, worse, silently
propagating it.
Tests:
* TestValidateReplicatedChunkSize -- table-driven coverage of healthy,
legitimately empty, zero-byte read, short read, and truncated upload
cases.
* TestFetchAndWriteRejectsZeroByteSource -- end-to-end: an httptest
source that returns 200 OK with an empty body must cause fetchAndWrite
to return errChunkSizeMismatch after exactly one source hit (fail
fast, no retry storm).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* filer.sync: bubble size-mismatch past CreateEntry/UpdateEntry
Three follow-ups on the chunk-size validation:
- Use %w in replicateOneChunk so the errChunkSizeMismatch sentinel
survives the wrap and reaches errors.Is callers up the stack.
- In FilerSink.CreateEntry/UpdateEntry, surface errChunkSizeMismatch
instead of warning-and-nil. Other errors (deleted source chunk,
transient network) keep the existing swallow so a hiccup doesn't
stall the stream.
- Drop validateReplicatedUploadSize: uploadResult.Size is set
client-side from the same len(fullData) we already validated
pre-upload, so the second check can't fail.
Test: scope the RetryWaitTime override to the one test that needs it,
add a regression that locks in the errors.Is chain through
replicateChunks.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
|
||
|
Chris Lu
|
be0379f6fd |
Fix filer.sync retry on stale chunk (#8298)
* Fix filer.sync stale chunk uploads * Tweak filersink stale logging |