Commit Graph
4 Commits
Author SHA1 Message Date
Chris LuandClaude Opus 5.5 a91b333658 s3 lifecycle: persist daily walk checkpoints and resume them safely (#12650)
* s3 lifecycle: walk checkpoints take each entry's resume position

Versions of one object share a Path, so a checkpoint set from Path after
the first version made a resume skip the rest. The ListFunc now gives
each entry the start value that continues after it, and leaves it empty
on all but an object's last version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: resume a bucket listing where the walk stopped

The filer listing skipped entries with key <= start, but its order is
not key order: each directory emits its .versions groups before its
files, and a directory's children come before a sibling like dir.txt.
Positions are now one (pass, name) frame per directory level, so a
resume seeds each listing and skips exactly what was finished. A version
group is one unit, and a skipped group still keeps its bare null
version out of the second pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: content hash of a walk view

Hashes every non-disabled action of a view and the versioning of their
buckets, so a saved walk position can be tied to the walk it came from:
another rule set, another walk mode, or a bucket that turned versioned
gives another hash.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: persist daily walk progress and resume it

A walk that failed or was cut short started over on the next pass,
listing again every bucket it had already finished. WalkBuckets now
keeps the finished buckets and each unfinished bucket's position in a
checkpoint file next to the shard cursors, one per shard set, and the
next walk of the same view and shards resumes from it. The record is
saved every 30s and when the walk stops, and cleared when a walk
completes, so the next walk is full again and sees keys added behind
the position; a clear that fails fails the walk rather than leave a
record that would skip buckets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a stuck resumed walk starts over

A bucket that fails at the same place on every pass kept the walk from
completing, so the buckets it had finished were never walked again and
their due objects waited forever. A resumed walk that fails again
without finishing anything clears its checkpoint, and the next walk is
full. A canceled walk keeps it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: do not resume a walk checkpoint left for a week

A checkpoint is cleared only by a later walk of the same shard set, so a
set that stopped walking, such as a one-off per-shard walk, left its
record behind, and a walk of that set much later would skip buckets
finished long before. Records now carry their save time and are not
resumed after seven days; a walk that keeps retrying saves on every
attempt and stays fresh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a failed walk without progress leaves its checkpoint alone

A pass canceled while it read the checkpoint failed the read, then
saved its own empty progress over the record it never loaded, so the
next pass started every bucket again. A failed walk that finished
nothing has nothing new to record and no longer writes; the stuck
resume reset still clears.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: space out walk checkpoint saves after a failed one

The save time advanced only on success, so once the filer refused a
periodic save every following entry tried again and logged another
warning. The interval now counts from the last attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a walk checkpoint save that errors may still be on disk

A save whose reply was lost left onDisk false, so a walk that then
completed skipped the clear and the next walk resumed from the stale
record instead of walking everything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: keep a stalled walk checkpoint for a day

A resumed walk that failed again without progress dropped its record
right away, so an outage spanning two attempts restarted the whole
walk. Treat the walk as stuck only once its record has not advanced
for a day.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a resumed walk counts its earlier buckets as walked

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: replay a resumed recovery walk from the walk that began it

A recovery or cold-start pass rewinds replay to runNow - maxTTL. When the
walk resumed a checkpoint, its finished buckets were listed in an earlier
attempt, so an object there that was not yet due then and whose write is
now older than maxTTL was neither expired by the walk nor replayed.

The checkpoint now keeps the start of the walk that began it, WalkBuckets
returns that time (its own start for a fresh walk), and runShard floors
the recovery rewind and the cold-start replay at it. A record without a
start is not resumed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a cold start that resumed a walk replays on the next pass

The pass's shared subscription starts at runNow - maxTTL for a shard
without a cursor, before its walk reports an earlier start, so flooring
the cold-start drain at the walk start could not deliver the older
events. Such a shard now saves its cursor at the walk start minus maxTTL
and returns, like the recovery branch, and the next pass replays the
window from that cursor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: test that a cursor below the floor opens the next subscription

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: test the cold-start replay across two Run passes

A cold start whose walk resumed an earlier attempt sees an older event
only on the next pass. The test runs Run twice against a filer that
serves events past the requested floor, with a later in-window event
that would otherwise move the cursor past the older one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:35:49 -08:00
Chris LuandClaude Opus 5.5 c819e3fbea s3 lifecycle: per-pass run results and last full-pass status (#12643)
* s3 lifecycle: collect per-pass bucket results and work counters

WalkBuckets kept only the first bucket error and logged the rest. A
RunStats shared by the walker, the walker dispatcher and the replay
drain now records every bucket's walk result (walked, failed, skipped
on cancellation), entry and object visits, replay events, and
LifecycleDelete outcomes under the dispatch_total outcome labels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: persist the last fully successful pass

Shard cursors and walk timestamps advance on partial passes as well, so
neither says when lifecycle last covered every policy. Keep that in its
own record next to the cursors, plus a worker gauge for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: report each pass's result in the job record

The job ended with a fixed success summary, or with only the first error,
and malformed lifecycle policies were V(1) logs. The completion now
carries the pass status (complete, partial, failed, canceled), bucket
totals and failures, invalid policies, entry, object and event counts,
LifecycleDelete outcomes, the last phase and bucket, the elapsed time
and the last full pass, on failed passes as well.

Only a complete pass advances the full-pass record; a malformed policy
makes the pass partial.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep a failed job's result in its record

Completions with Success=false dropped Result.Summary and
Result.OutputValues, so a handler could not report what a failed run
had done, which buckets failed, or how far it got. Keep them; the error
message still drives the job's message and run record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart the time since the last full lifecycle pass

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a replay halted on an unresolved delete is a partial pass

BLOCKED, RETRY_LATER and exhausted transport retries stop a shard's
replay without an error, leaving its due events for a later pass. Count
those shards and report the pass as partial, so it no longer advances
the full-pass record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a pass with throttled walks is not a full pass

WalkerInterval can skip the walk for walk-only rules (ExpirationDate,
ExpiredDeleteMarker, NewerNoncurrent) with no error. Count those shards
and keep the full-pass record for passes that evaluated every rule; the
job reports full_pass and walks_deferred.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: keep the saved full-pass time when the save fails

Publishing a timestamp that never reached the filer let the gauge move
back on the next job's reload. Keep the previous value and report the
save error in the job result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: drop a failed attempt's summary when the retry reports none

Retries reuse the job record, and failed completions now keep their
result summary, so a retry that completes without a Result showed the
failed attempt's summary, and a successful one used it as its message.
Clear the summary when a completion carries no Result, as the output
values already are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 10:50:37 -08:00
Chris Lu 443ae1cd83 s3 lifecycle: share daily bucket walk and prune listings (#11797)
* s3 lifecycle: share the daily bucket walk across shards

The daily pass invoked the walker once per shard, and every shard's
walk listed the full bucket tree while keeping only its own 1/16 of
the entries — 16 full listings per pass by default (#11791).

Run now groups the shards by the walk branch they'll take (recovery
vs steady, using the same cursor data and hash checks runShard applies)
and starts one shared walk per group, so a pass lists each bucket at
most twice. WalkerFunc and WalkBuckets take a shard set instead of a
single shard. runShard keeps its own decision and waits on its group's
walk; a shard whose freshly-loaded cursor disagrees with the plan falls
back to a per-shard walk, so mid-pass cursor rewrites can't under-cover
a shard.

* s3 lifecycle: skip the .versions/ pre-pass on non-versioned buckets

walkBucketTree listed every directory twice — once looking for
.versions/ dirs, once for regular entries. On a bucket that never had
versioning enabled the first pass can only find coincidentally-named
user folders, so the extra listing is pure cost (#11791).

FilerListFunc now takes the walk view and consults
Snapshot.BucketVersioned; non-versioned buckets skip pass 1 entirely.
A *-suffixed .versions user folder on such a bucket is recursed into
as a regular directory so real objects inside still surface; a nil
view keeps the two-pass behavior.

* s3 lifecycle: prune walk directories to active rule prefixes

The daily walk listed every directory under a bucket even when no
rule prefix could reach it. Collect the active rule prefixes of each
bucket from the walk view (new Snapshot.BucketPrefixes) and skip
recursing into directories whose keys can never match; a bucket
with no active actions is skipped entirely, saving the listing.

.uploads stays in scope regardless of prefixes because AbortMPU
rules match DestKey, and .versions expansion is pruned only
conservatively (the logical key is a string-prefix of the dir key).

* s3 lifecycle: join planned walks before Run returns

planWalks starts a shared walk goroutine per group, but a shard whose
authoritative cursor load then fails exits runShard without waiting on
it. Run returned while the walk still listed or dispatched deletes
against connections the caller closes on return. Join every planned
walk after the shard waitgroup.

* s3 lifecycle: treat object-locked buckets as versioned in the walk

Object Lock requires versioning and every writer sets both extended
attributes, but IsBucketVersioned only read ExtVersioningKey. A locked
bucket missing that attribute would have its .versions dirs walked as
ordinary folders, hiding version siblings from lifecycle. Classify
ExtObjectLockEnabledKey as versioned too.
2026-10-17 15:21:53 -07:00
Chris Lu f954781169 feat(s3/lifecycle): Phase 4b — daily walker for recovery and steady state (#9459)
* feat(s3/lifecycle): plumb RetentionWindow into dailyrun.Config

Adds a Config.RetentionWindow field that runShard threads into
engine.PromotedHash. Zero (the default) falls back to maxTTL, which
matches Phase 4a behavior — PromotedHash stays empty and the
partition-flip recovery trigger stays dormant.

Pure plumbing. The handler still passes zero so nothing changes at
runtime. The walker work (Phase 4b proper) sets a real retention from
the meta-log boundary and the partition-flip trigger starts firing.

* feat(s3/lifecycle): WalkerDispatcher adapter for the daily-run walker

Phase 4b prep. Implements bootstrap.Dispatcher on top of LifecycleClient
so the same LifecycleDelete RPC drives both the meta-log replay path
and the walker. No CAS witness — the server's identityMatches treats
nil ExpectedIdentity as a bootstrap call and rebuilds the witness from
the live entry, which is the right contract for a full-tree walk.

Adds VersionID to bootstrap.Entry so versioned-bucket walks address
the right version. MPU init uses DestKey for ObjectPath (matching the
prefix-match contract); rejecting empty DestKey keeps malformed init
records out of the dispatch path.

Not wired yet — runShard still doesn't invoke the walker. Follow-up
commits add the ListFunc adapter and the recovery-branch wiring.

* feat(s3/lifecycle): wire Walker hook into runShard's recovery branch

Adds a Config.Walker callback that fires on rule-content edit /
partition flip BEFORE the cursor rewinds, so already-due objects across
the rewritten rule set get caught instead of waiting on meta-log
replay alone. The callback receives engine.RecoveryView(snap) and the
per-shard ID; nil disables it (Phase 4a behavior preserved).

Decoupling the wiring from the implementation: the handler-side
WalkerFunc that drives bootstrap.Walk via the filer is the follow-up
commit, and tests can stub the callback without standing up the full
filer/client/lister harness.

Tests pin: walker fires exactly once on hash mismatch, walker error
propagates and leaves the cursor unchanged, nil Walker is a no-op.

* feat(s3/lifecycle): WalkBuckets composes ListFunc + Dispatcher per shard

Adds dailyrun.WalkBuckets — the composable driver the handler-side
WalkerFunc will call. Iterates a bucket list, wraps the supplied
bootstrap.ListFunc with a per-shard filter (Path for non-MPU, DestKey
for MPU init), and runs bootstrap.Walk per bucket using the supplied
Dispatcher. First bucket error wins; remaining buckets log and run to
completion so one filer flake doesn't kill the shard.

Composable rather than monolithic so callers and tests can swap parts:
production uses a filer-backed ListFunc + WalkerDispatcher; tests use
bootstrap.EntryCallback + a stub. The filer-backed ListFunc is the
next commit.

Tests pin: shard filter routes only matching entries, MPU shard uses
DestKey not the .uploads/<id> path, single-bucket error propagates
while other buckets still run, ctx cancellation short-circuits between
buckets, nil guards on view/list/dispatch.

* feat(s3/lifecycle): filer-backed ListFunc for the daily-run walker

Phase 4b: dailyrun.FilerListFunc returns a bootstrap.ListFunc that
streams entries under <bucketsPath>/<bucket> by paginated SeaweedList.
Recurses into regular directories; .versions/ and .uploads/ are
skipped at this stage so they don't surface as raw children — the
sibling expansion (versioned NoncurrentDays state, MPU init dispatch)
lands in the next commit.

listAll and isVersionsDir are ported from scheduler/bootstrap.go's
same-named helpers. Phase 5 deletes the scheduler copies along with
the streaming path.

Tests pin: flat listing, recursion through nested directories,
.versions/ and .uploads/ skipped, kill-resume via the start path
contract, nil-client error, attribute propagation (mtime / size /
IsLatest default).

* feat(s3/lifecycle): versioned-sibling expansion in FilerListFunc

Adds the .versions/<key>/ expansion to the daily-run's filer-backed
ListFunc. Each call emits one bootstrap.Entry per sibling (real
version files + the bare null version, when found) with the same
sibling state the streaming bootstrap injects via reader.Event:

  - Path = logical key (not the .versions/<file> physical path), so
    bootstrap.Walk's MatchPath uses the user's intended path.
  - VersionID per sibling (version_id or "null").
  - IsLatest resolved via parent's ExtLatestVersionIdKey, falling back
    to explicit-null-bare, falling back to newest-by-mtime.
  - NoncurrentIndex rank computed against the latest's position.
  - SuccessorModTime: SuccessorFromEntryStamp if stamped, else the
    previous-newer sibling's mtime (legacy derivation).
  - IsDeleteMarker from ExtDeleteMarkerKey.
  - NumVersions = len(siblings).

Two-pass walk so .versions/ dirs run before regular files; the bare
null-version path is recorded in skipBare so pass 2 doesn't emit it
twice.

expandVersionsDir and lookupNullVersion are ported from
scheduler/bootstrap.go. Sort order, latest resolution, and successor
derivation must agree with that path verbatim so streaming and walker
reach the same verdict on the same objects. Phase 5 deletes the
scheduler copy.

MPU init (.uploads/<id>) remains skipped — the dedicated commit emits
it with IsMPUInit and DestKey.

Tests pin: pointer-wins latest resolution, no-pointer newest-sibling
fallback, explicit-null-is-latest with skipBare suppression of the
bare emission, coincidentally-named .versions folder recursing as a
regular subdir, delete-marker propagation.

* feat(s3/lifecycle): emit MPU init records from FilerListFunc

Last gap in the filer-backed ListFunc. A directory at .uploads/<id>
carrying ExtMultipartObjectKey is the MPU init record; emit one
bootstrap.Entry with IsMPUInit=true and DestKey set to the user's
intended path. The walker's MatchPath uses DestKey for prefix
matching; the WalkerDispatcher uses it for the LifecycleDelete RPC's
ObjectPath. .uploads/<id> directories without the extended key are
mid-write before metadata landed and stay skipped.

isMPUInitDir is upgraded from the path-shape-only stub to the full
shape + extended-attr check that mirrors router.mpuInitInfo and
scheduler/bootstrap.go's same-named helper.

Tests pin: valid init record emits with the right DestKey, missing
ExtMultipartObjectKey skips the directory.

* feat(s3/lifecycle): wire walker into executeDailyReplay

Activates the recovery-branch walker. The handler composes the three
Phase 4b building blocks — FilerListFunc + WalkerDispatcher + WalkBuckets
— into a dailyrun.WalkerFunc and passes it via Config.Walker. The
bucket list is derived from the compiled inputs so it matches the
engine snapshot exactly.

Effect on master behavior: when a worker observes a RuleSetHash or
PromotedHash mismatch on its persisted cursor (rule content edited /
partition flip), runShard now walks the live filer tree under the
RecoveryView before rewinding the cursor. Already-due objects across
the rewritten rule set fire immediately instead of waiting on the
sliding meta-log replay.

Still scoped to replay-eligible action kinds because
checkSnapshotForUnsupported continues to reject walker-bound rules
(ExpirationDate / ExpiredDeleteMarker / NewerNoncurrent) and
scan_only-promoted rules at the top of Run. The follow-up commit
relaxes the gate once the steady-state walker over RulesForShard's
walk view is wired so those rules fire every day, not just on rule
edits.

* feat(s3/lifecycle): steady-state walker + drop unsupported-rule gate

Adds the second walker invocation in runShard. After the recovery
check passes, runShard derives the walk view via snap.RulesForShard
(using the same retentionWindow PromotedHash used, so the partition
is consistent) and runs the walker over it. The view holds
walker-bound action kinds (ExpirationDate / ExpiredDeleteMarker /
NewerNoncurrent) plus any replay-eligible rules promoted to walk by
retention shortage; an empty view skips the call so non-versioned,
replay-only deployments don't pay an O(N) bucket walk per run.

With the walker now servicing every rule kind, checkSnapshotForUnsupported
and its UnsupportedRuleError type are obsolete. router.Route gates
replay on Mode == ModeEventDriven, so walker-bound and scan_only
rules are silently dropped by replay and picked up by the walker
instead — no double-dispatch. Drop the gate, delete replayability.go
+ replayability_test.go, and remove the handler's redundant
IsUnsupportedRule branch.

* fix(s3/lifecycle): walker dispatcher nil-response guard + retention-comment

Two PR-review fixes on 9459:

1. WalkerDispatcher.Delete used to panic on a (nil, nil) RPC return —
   add a defensive nil-response check so the walk halts cleanly
   instead. Spotted by coderabbit.

2. The retentionWindow=maxTTL comment in runShard claimed PromotedHash
   "stays empty" in fallback mode, which gemini correctly pointed out
   is only true once rules are active. During bootstrap (rules
   compiled but IsActive=false) MaxEffectiveTTL is 0 while
   PromotedHash counts every non-disabled rule, so promoted becomes
   non-empty and the next post-activation run hits the recovery
   branch. That's the intended bootstrap walk — rewrite the comment
   to explain it rather than misstate the invariant.

Test: pins nil-response → error path on WalkerDispatcher.

* fix(s3/lifecycle): explicit stale-pointer fallback in versioned expansion

Reviewer caught a structural bug in expandVersionsDir's latest
resolution: when ExtLatestVersionIdKey was set but no scanned sibling
carried that id (stale pointer), the code left latestPos at the
default 0 without ever entering the no-pointer fallback. Today the
two paths yield the same value (newest sibling wins), but the
implicit fall-through makes the intent unclear and would break
silently if the no-pointer branch ever did anything more than
latestPos=0.

Track a pointerResolved flag explicitly so the no-pointer branch
(including the explicit-null-bare check) re-runs on a stale pointer.
Behavior unchanged today.

Test pins: stale pointer + two real versions falls back to
newest-sibling (vnew, not vold).

* feat(s3/lifecycle): walker-side dispatch metrics in WalkerDispatcher

Mirrors the Phase 6 instrumentation already on the replay side
(processMatches) onto the walker's Delete dispatch. Every walker
dispatch now bumps S3LifecycleDispatchCounter with the resolved
outcome (or TRANSPORT_ERROR / NIL_RESPONSE for the failure paths) so
streaming, daily_replay's replay drain, and daily_replay's walker
share a single per-(bucket, kind, outcome) counter view.

Lands together with the rest of Phase 4b — no new metric, just an
extra observation site for the existing one.
2026-05-12 11:39:15 -07:00