mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-19 02:15:02 +01:00
94de65b3b5d79b85e1112321199e81ccc1273cc1
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
a91b333658 |
s3 lifecycle: persist daily walk checkpoints and resume them safely (#12650)
* s3 lifecycle: walk checkpoints take each entry's resume position Versions of one object share a Path, so a checkpoint set from Path after the first version made a resume skip the rest. The ListFunc now gives each entry the start value that continues after it, and leaves it empty on all but an object's last version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: resume a bucket listing where the walk stopped The filer listing skipped entries with key <= start, but its order is not key order: each directory emits its .versions groups before its files, and a directory's children come before a sibling like dir.txt. Positions are now one (pass, name) frame per directory level, so a resume seeds each listing and skips exactly what was finished. A version group is one unit, and a skipped group still keeps its bare null version out of the second pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: content hash of a walk view Hashes every non-disabled action of a view and the versioning of their buckets, so a saved walk position can be tied to the walk it came from: another rule set, another walk mode, or a bucket that turned versioned gives another hash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: persist daily walk progress and resume it A walk that failed or was cut short started over on the next pass, listing again every bucket it had already finished. WalkBuckets now keeps the finished buckets and each unfinished bucket's position in a checkpoint file next to the shard cursors, one per shard set, and the next walk of the same view and shards resumes from it. The record is saved every 30s and when the walk stops, and cleared when a walk completes, so the next walk is full again and sees keys added behind the position; a clear that fails fails the walk rather than leave a record that would skip buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a stuck resumed walk starts over A bucket that fails at the same place on every pass kept the walk from completing, so the buckets it had finished were never walked again and their due objects waited forever. A resumed walk that fails again without finishing anything clears its checkpoint, and the next walk is full. A canceled walk keeps it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: do not resume a walk checkpoint left for a week A checkpoint is cleared only by a later walk of the same shard set, so a set that stopped walking, such as a one-off per-shard walk, left its record behind, and a walk of that set much later would skip buckets finished long before. Records now carry their save time and are not resumed after seven days; a walk that keeps retrying saves on every attempt and stays fresh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a failed walk without progress leaves its checkpoint alone A pass canceled while it read the checkpoint failed the read, then saved its own empty progress over the record it never loaded, so the next pass started every bucket again. A failed walk that finished nothing has nothing new to record and no longer writes; the stuck resume reset still clears. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: space out walk checkpoint saves after a failed one The save time advanced only on success, so once the filer refused a periodic save every following entry tried again and logged another warning. The interval now counts from the last attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a walk checkpoint save that errors may still be on disk A save whose reply was lost left onDisk false, so a walk that then completed skipped the clear and the next walk resumed from the stale record instead of walking everything. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: keep a stalled walk checkpoint for a day A resumed walk that failed again without progress dropped its record right away, so an outage spanning two attempts restarted the whole walk. Treat the walk as stuck only once its record has not advanced for a day. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a resumed walk counts its earlier buckets as walked Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: replay a resumed recovery walk from the walk that began it A recovery or cold-start pass rewinds replay to runNow - maxTTL. When the walk resumed a checkpoint, its finished buckets were listed in an earlier attempt, so an object there that was not yet due then and whose write is now older than maxTTL was neither expired by the walk nor replayed. The checkpoint now keeps the start of the walk that began it, WalkBuckets returns that time (its own start for a fresh walk), and runShard floors the recovery rewind and the cold-start replay at it. A record without a start is not resumed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a cold start that resumed a walk replays on the next pass The pass's shared subscription starts at runNow - maxTTL for a shard without a cursor, before its walk reports an earlier start, so flooring the cold-start drain at the walk start could not deliver the older events. Such a shard now saves its cursor at the walk start minus maxTTL and returns, like the recovery branch, and the next pass replays the window from that cursor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test that a cursor below the floor opens the next subscription Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test the cold-start replay across two Run passes A cold start whose walk resumed an earlier attempt sees an older event only on the next pass. The test runs Run twice against a filer that serves events past the requested floor, with a later in-window event that would otherwise move the cursor past the older one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c819e3fbea |
s3 lifecycle: per-pass run results and last full-pass status (#12643)
* s3 lifecycle: collect per-pass bucket results and work counters WalkBuckets kept only the first bucket error and logged the rest. A RunStats shared by the walker, the walker dispatcher and the replay drain now records every bucket's walk result (walked, failed, skipped on cancellation), entry and object visits, replay events, and LifecycleDelete outcomes under the dispatch_total outcome labels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: persist the last fully successful pass Shard cursors and walk timestamps advance on partial passes as well, so neither says when lifecycle last covered every policy. Keep that in its own record next to the cursors, plus a worker gauge for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: report each pass's result in the job record The job ended with a fixed success summary, or with only the first error, and malformed lifecycle policies were V(1) logs. The completion now carries the pass status (complete, partial, failed, canceled), bucket totals and failures, invalid policies, entry, object and event counts, LifecycleDelete outcomes, the last phase and bucket, the elapsed time and the last full pass, on failed passes as well. Only a complete pass advances the full-pass record; a malformed policy makes the pass partial. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep a failed job's result in its record Completions with Success=false dropped Result.Summary and Result.OutputValues, so a handler could not report what a failed run had done, which buckets failed, or how far it got. Keep them; the error message still drives the job's message and run record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grafana: chart the time since the last full lifecycle pass Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a replay halted on an unresolved delete is a partial pass BLOCKED, RETRY_LATER and exhausted transport retries stop a shard's replay without an error, leaving its due events for a later pass. Count those shards and report the pass as partial, so it no longer advances the full-pass record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a pass with throttled walks is not a full pass WalkerInterval can skip the walk for walk-only rules (ExpirationDate, ExpiredDeleteMarker, NewerNoncurrent) with no error. Count those shards and keep the full-pass record for passes that evaluated every rule; the job reports full_pass and walks_deferred. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: keep the saved full-pass time when the save fails Publishing a timestamp that never reached the filer let the gauge move back on the next job's reload. Keep the previous value and report the save error in the job result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: drop a failed attempt's summary when the retry reports none Retries reuse the job record, and failed completions now keep their result summary, so a retry that completes without a Result showed the failed attempt's summary, and a successful one used it as its message. Clear the summary when a completion carries no Result, as the output values already are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
443ae1cd83 |
s3 lifecycle: share daily bucket walk and prune listings (#11797)
* s3 lifecycle: share the daily bucket walk across shards The daily pass invoked the walker once per shard, and every shard's walk listed the full bucket tree while keeping only its own 1/16 of the entries — 16 full listings per pass by default (#11791). Run now groups the shards by the walk branch they'll take (recovery vs steady, using the same cursor data and hash checks runShard applies) and starts one shared walk per group, so a pass lists each bucket at most twice. WalkerFunc and WalkBuckets take a shard set instead of a single shard. runShard keeps its own decision and waits on its group's walk; a shard whose freshly-loaded cursor disagrees with the plan falls back to a per-shard walk, so mid-pass cursor rewrites can't under-cover a shard. * s3 lifecycle: skip the .versions/ pre-pass on non-versioned buckets walkBucketTree listed every directory twice — once looking for .versions/ dirs, once for regular entries. On a bucket that never had versioning enabled the first pass can only find coincidentally-named user folders, so the extra listing is pure cost (#11791). FilerListFunc now takes the walk view and consults Snapshot.BucketVersioned; non-versioned buckets skip pass 1 entirely. A *-suffixed .versions user folder on such a bucket is recursed into as a regular directory so real objects inside still surface; a nil view keeps the two-pass behavior. * s3 lifecycle: prune walk directories to active rule prefixes The daily walk listed every directory under a bucket even when no rule prefix could reach it. Collect the active rule prefixes of each bucket from the walk view (new Snapshot.BucketPrefixes) and skip recursing into directories whose keys can never match; a bucket with no active actions is skipped entirely, saving the listing. .uploads stays in scope regardless of prefixes because AbortMPU rules match DestKey, and .versions expansion is pruned only conservatively (the logical key is a string-prefix of the dir key). * s3 lifecycle: join planned walks before Run returns planWalks starts a shared walk goroutine per group, but a shard whose authoritative cursor load then fails exits runShard without waiting on it. Run returned while the walk still listed or dispatched deletes against connections the caller closes on return. Join every planned walk after the shard waitgroup. * s3 lifecycle: treat object-locked buckets as versioned in the walk Object Lock requires versioning and every writer sets both extended attributes, but IsBucketVersioned only read ExtVersioningKey. A locked bucket missing that attribute would have its .versions dirs walked as ordinary folders, hiding version siblings from lifecycle. Classify ExtObjectLockEnabledKey as versioned too. |
||
|
|
f954781169 |
feat(s3/lifecycle): Phase 4b — daily walker for recovery and steady state (#9459)
* feat(s3/lifecycle): plumb RetentionWindow into dailyrun.Config
Adds a Config.RetentionWindow field that runShard threads into
engine.PromotedHash. Zero (the default) falls back to maxTTL, which
matches Phase 4a behavior — PromotedHash stays empty and the
partition-flip recovery trigger stays dormant.
Pure plumbing. The handler still passes zero so nothing changes at
runtime. The walker work (Phase 4b proper) sets a real retention from
the meta-log boundary and the partition-flip trigger starts firing.
* feat(s3/lifecycle): WalkerDispatcher adapter for the daily-run walker
Phase 4b prep. Implements bootstrap.Dispatcher on top of LifecycleClient
so the same LifecycleDelete RPC drives both the meta-log replay path
and the walker. No CAS witness — the server's identityMatches treats
nil ExpectedIdentity as a bootstrap call and rebuilds the witness from
the live entry, which is the right contract for a full-tree walk.
Adds VersionID to bootstrap.Entry so versioned-bucket walks address
the right version. MPU init uses DestKey for ObjectPath (matching the
prefix-match contract); rejecting empty DestKey keeps malformed init
records out of the dispatch path.
Not wired yet — runShard still doesn't invoke the walker. Follow-up
commits add the ListFunc adapter and the recovery-branch wiring.
* feat(s3/lifecycle): wire Walker hook into runShard's recovery branch
Adds a Config.Walker callback that fires on rule-content edit /
partition flip BEFORE the cursor rewinds, so already-due objects across
the rewritten rule set get caught instead of waiting on meta-log
replay alone. The callback receives engine.RecoveryView(snap) and the
per-shard ID; nil disables it (Phase 4a behavior preserved).
Decoupling the wiring from the implementation: the handler-side
WalkerFunc that drives bootstrap.Walk via the filer is the follow-up
commit, and tests can stub the callback without standing up the full
filer/client/lister harness.
Tests pin: walker fires exactly once on hash mismatch, walker error
propagates and leaves the cursor unchanged, nil Walker is a no-op.
* feat(s3/lifecycle): WalkBuckets composes ListFunc + Dispatcher per shard
Adds dailyrun.WalkBuckets — the composable driver the handler-side
WalkerFunc will call. Iterates a bucket list, wraps the supplied
bootstrap.ListFunc with a per-shard filter (Path for non-MPU, DestKey
for MPU init), and runs bootstrap.Walk per bucket using the supplied
Dispatcher. First bucket error wins; remaining buckets log and run to
completion so one filer flake doesn't kill the shard.
Composable rather than monolithic so callers and tests can swap parts:
production uses a filer-backed ListFunc + WalkerDispatcher; tests use
bootstrap.EntryCallback + a stub. The filer-backed ListFunc is the
next commit.
Tests pin: shard filter routes only matching entries, MPU shard uses
DestKey not the .uploads/<id> path, single-bucket error propagates
while other buckets still run, ctx cancellation short-circuits between
buckets, nil guards on view/list/dispatch.
* feat(s3/lifecycle): filer-backed ListFunc for the daily-run walker
Phase 4b: dailyrun.FilerListFunc returns a bootstrap.ListFunc that
streams entries under <bucketsPath>/<bucket> by paginated SeaweedList.
Recurses into regular directories; .versions/ and .uploads/ are
skipped at this stage so they don't surface as raw children — the
sibling expansion (versioned NoncurrentDays state, MPU init dispatch)
lands in the next commit.
listAll and isVersionsDir are ported from scheduler/bootstrap.go's
same-named helpers. Phase 5 deletes the scheduler copies along with
the streaming path.
Tests pin: flat listing, recursion through nested directories,
.versions/ and .uploads/ skipped, kill-resume via the start path
contract, nil-client error, attribute propagation (mtime / size /
IsLatest default).
* feat(s3/lifecycle): versioned-sibling expansion in FilerListFunc
Adds the .versions/<key>/ expansion to the daily-run's filer-backed
ListFunc. Each call emits one bootstrap.Entry per sibling (real
version files + the bare null version, when found) with the same
sibling state the streaming bootstrap injects via reader.Event:
- Path = logical key (not the .versions/<file> physical path), so
bootstrap.Walk's MatchPath uses the user's intended path.
- VersionID per sibling (version_id or "null").
- IsLatest resolved via parent's ExtLatestVersionIdKey, falling back
to explicit-null-bare, falling back to newest-by-mtime.
- NoncurrentIndex rank computed against the latest's position.
- SuccessorModTime: SuccessorFromEntryStamp if stamped, else the
previous-newer sibling's mtime (legacy derivation).
- IsDeleteMarker from ExtDeleteMarkerKey.
- NumVersions = len(siblings).
Two-pass walk so .versions/ dirs run before regular files; the bare
null-version path is recorded in skipBare so pass 2 doesn't emit it
twice.
expandVersionsDir and lookupNullVersion are ported from
scheduler/bootstrap.go. Sort order, latest resolution, and successor
derivation must agree with that path verbatim so streaming and walker
reach the same verdict on the same objects. Phase 5 deletes the
scheduler copy.
MPU init (.uploads/<id>) remains skipped — the dedicated commit emits
it with IsMPUInit and DestKey.
Tests pin: pointer-wins latest resolution, no-pointer newest-sibling
fallback, explicit-null-is-latest with skipBare suppression of the
bare emission, coincidentally-named .versions folder recursing as a
regular subdir, delete-marker propagation.
* feat(s3/lifecycle): emit MPU init records from FilerListFunc
Last gap in the filer-backed ListFunc. A directory at .uploads/<id>
carrying ExtMultipartObjectKey is the MPU init record; emit one
bootstrap.Entry with IsMPUInit=true and DestKey set to the user's
intended path. The walker's MatchPath uses DestKey for prefix
matching; the WalkerDispatcher uses it for the LifecycleDelete RPC's
ObjectPath. .uploads/<id> directories without the extended key are
mid-write before metadata landed and stay skipped.
isMPUInitDir is upgraded from the path-shape-only stub to the full
shape + extended-attr check that mirrors router.mpuInitInfo and
scheduler/bootstrap.go's same-named helper.
Tests pin: valid init record emits with the right DestKey, missing
ExtMultipartObjectKey skips the directory.
* feat(s3/lifecycle): wire walker into executeDailyReplay
Activates the recovery-branch walker. The handler composes the three
Phase 4b building blocks — FilerListFunc + WalkerDispatcher + WalkBuckets
— into a dailyrun.WalkerFunc and passes it via Config.Walker. The
bucket list is derived from the compiled inputs so it matches the
engine snapshot exactly.
Effect on master behavior: when a worker observes a RuleSetHash or
PromotedHash mismatch on its persisted cursor (rule content edited /
partition flip), runShard now walks the live filer tree under the
RecoveryView before rewinding the cursor. Already-due objects across
the rewritten rule set fire immediately instead of waiting on the
sliding meta-log replay.
Still scoped to replay-eligible action kinds because
checkSnapshotForUnsupported continues to reject walker-bound rules
(ExpirationDate / ExpiredDeleteMarker / NewerNoncurrent) and
scan_only-promoted rules at the top of Run. The follow-up commit
relaxes the gate once the steady-state walker over RulesForShard's
walk view is wired so those rules fire every day, not just on rule
edits.
* feat(s3/lifecycle): steady-state walker + drop unsupported-rule gate
Adds the second walker invocation in runShard. After the recovery
check passes, runShard derives the walk view via snap.RulesForShard
(using the same retentionWindow PromotedHash used, so the partition
is consistent) and runs the walker over it. The view holds
walker-bound action kinds (ExpirationDate / ExpiredDeleteMarker /
NewerNoncurrent) plus any replay-eligible rules promoted to walk by
retention shortage; an empty view skips the call so non-versioned,
replay-only deployments don't pay an O(N) bucket walk per run.
With the walker now servicing every rule kind, checkSnapshotForUnsupported
and its UnsupportedRuleError type are obsolete. router.Route gates
replay on Mode == ModeEventDriven, so walker-bound and scan_only
rules are silently dropped by replay and picked up by the walker
instead — no double-dispatch. Drop the gate, delete replayability.go
+ replayability_test.go, and remove the handler's redundant
IsUnsupportedRule branch.
* fix(s3/lifecycle): walker dispatcher nil-response guard + retention-comment
Two PR-review fixes on 9459:
1. WalkerDispatcher.Delete used to panic on a (nil, nil) RPC return —
add a defensive nil-response check so the walk halts cleanly
instead. Spotted by coderabbit.
2. The retentionWindow=maxTTL comment in runShard claimed PromotedHash
"stays empty" in fallback mode, which gemini correctly pointed out
is only true once rules are active. During bootstrap (rules
compiled but IsActive=false) MaxEffectiveTTL is 0 while
PromotedHash counts every non-disabled rule, so promoted becomes
non-empty and the next post-activation run hits the recovery
branch. That's the intended bootstrap walk — rewrite the comment
to explain it rather than misstate the invariant.
Test: pins nil-response → error path on WalkerDispatcher.
* fix(s3/lifecycle): explicit stale-pointer fallback in versioned expansion
Reviewer caught a structural bug in expandVersionsDir's latest
resolution: when ExtLatestVersionIdKey was set but no scanned sibling
carried that id (stale pointer), the code left latestPos at the
default 0 without ever entering the no-pointer fallback. Today the
two paths yield the same value (newest sibling wins), but the
implicit fall-through makes the intent unclear and would break
silently if the no-pointer branch ever did anything more than
latestPos=0.
Track a pointerResolved flag explicitly so the no-pointer branch
(including the explicit-null-bare check) re-runs on a stale pointer.
Behavior unchanged today.
Test pins: stale pointer + two real versions falls back to
newest-sibling (vnew, not vold).
* feat(s3/lifecycle): walker-side dispatch metrics in WalkerDispatcher
Mirrors the Phase 6 instrumentation already on the replay side
(processMatches) onto the walker's Delete dispatch. Every walker
dispatch now bumps S3LifecycleDispatchCounter with the resolved
outcome (or TRANSPORT_ERROR / NIL_RESPONSE for the failure paths) so
streaming, daily_replay's replay drain, and daily_replay's walker
share a single per-(bucket, kind, outcome) counter view.
Lands together with the rest of Phase 4b — no new metric, just an
extra observation site for the existing one.
|