seaweedfs/weed/s3api/s3lifecycle/DESIGN.md

S3 Lifecycle (Design)

Replaces the current streaming + heap design with a daily meta-log replay. The worker becomes "start, do today's work, stop" — matching the operator's mental model and removing the per-shard heap that grows O(events × TTL_days).

Goal

For each bucket lifecycle rule with TTL D days, delete every object whose age exceeds D, exactly once per day, with the worker exiting when the pass completes. No future-buffered Matches in memory. No tick-driven dispatch loop. Cluster-wide delete rate cap allocated per worker.

Algorithm

One pass per day, per shard:

daily_run(shardID):
    persisted             = load_cursor(shardID)                // { TsNs, rule_set_hash, promoted_hash }
    earliest_available    = metalog.EarliestAvailableTsNs(shardID)
    retention_window      = duration(now - earliest_available)   // time.Duration; sole window input to engine
    snap                  = engine.CurrentSnapshot()             // base; per-action active/Mode reflect compile-time state, not partition
    replay, walk          = snap.RulesForShard(shardID, retention_window)
    if replay == nil and walk == nil: return                    // bucket has no rules
    rsh                   = engine.ReplayContentHash(snap)      // content over ExpirationDays/NoncurrentDays/AbortMPU rules; partition-independent
    promoted              = engine.PromotedHash(snap, retention_window)  // rules currently in walk due to scan_only

    needs_recovery_walk = (
        persisted is fresh                                      // cold start
        or persisted.rule_set_hash != rsh                       // replay-rule edit
        or persisted.promoted_hash != promoted                  // partition flip in either direction
        or (replay != nil and persisted.TsNs < earliest_available)  // retention loss; meaningless if pure walker bucket
    )

    if needs_recovery_walk:
        // Catch already-due objects across ALL rules — both walker-bound and
        // replay-bound. Replay-bound rules may have objects whose mtime is
        // older than (now - max_ttl) that the steady-state replay window cannot
        // reach. The recovery view force-activates every action so the walker
        // doesn't skip event-driven rules that the base snapshot has marked
        // inactive (e.g., pre-BootstrapComplete).
        walker.RunForShard(ctx, shardID, engine.RecoveryView(snap), limiter)
        if replay == nil:
            save_cursor(shardID, { TsNs: 0, rule_set_hash: rsh, promoted_hash: promoted })
        else:
            save_cursor(shardID, { TsNs: now - engine.MaxEffectiveTTL(replay), rule_set_hash: rsh, promoted_hash: promoted })
        return

    // --- Steady-state path. ----------------------------------------------
    if walk != nil:
        walker.RunForShard(ctx, shardID, walk, limiter)         // walker-bound actions only

    if replay == nil:
        save_cursor(shardID, { TsNs: 0, rule_set_hash: rsh, promoted_hash: promoted })
        return

    max_ttl  = engine.MaxEffectiveTTL(replay)
    cursor   = max(persisted.TsNs, now - max_ttl)
    done     = {r: false for r in replay.activeRules()}
    last_ok  = persisted.TsNs

    for ev in metalog.from(shardID, cursor):                    // ascending TsNs
        if all(done.values()): break
        halt    = false
        matches = router.Route(ctx, replay, ev, now, lister)    // existing signature; replay is *engine.Snapshot
        for m in matches:
            r = replay.lookup(m.RuleHash)
            if r == nil or done[r]: continue
            if m.DueTime > now:
                done[r] = true                                  // monotonic for replay kinds
                continue
            limiter.wait()
            outcome = dispatch_delete(m.Bucket, m.ObjectKey, m.VersionID,
                                      expected_mtime=m.Identity.mtime, ruleHash=m.RuleHash)
            if outcome is unresolved (RETRY_LATER, transport error after in-run retries, BLOCKED):
                halt = true; break
        if halt: break
        last_ok = ev.TsNs

    save_cursor(shardID, { TsNs: last_ok, rule_set_hash: rsh, promoted_hash: promoted })

Two distinct walker invocations, deliberately:

Branch	Walker input	Why
Recovery (cold-start, replay-rule edit, retention loss, partition flip)	engine.RecoveryView(snap) — every action clone force-markActive()'d, Mode preserved	Catch already-due objects across all rules — including replay-bound rules whose subjects are older than max_ttl (replay scan window cannot reach), and event-driven rules the base snapshot may mark inactive (compile.go:73-74 requires BootstrapComplete).
Steady-state	walk (walker-only actions)	Replay handles its own rules via the meta-log scan. Walker only runs for actions that have no replay path.

router.Route and walker.RunForShard keep their existing *engine.Snapshot parameter; replay, walk, and snap are all *Snapshot values.

The single API surface is:

// In engine:
func CurrentSnapshot() *Snapshot                                                  // base
func (s *Snapshot) RulesForShard(shardID int, retentionWindow time.Duration) (replay, walk *Snapshot)
func RecoveryView(s *Snapshot) *Snapshot                                          // every action force-activated; recovery walker only
func ReplayContentHash(s *Snapshot) [32]byte                                      // content over replay-eligible rules; partition-independent
func PromotedHash(s *Snapshot, retentionWindow time.Duration) [32]byte            // hash of replay-eligible rules currently in walk (scan_only)
func MaxEffectiveTTL(s *Snapshot) time.Duration                                   // max over active replay actions

RulesForShard and RecoveryView both return new *Snapshot instances with cloned *CompiledAction objects and shared (by pointer) rule definitions. The fields that differ from the base are:

• active (per clone) — set per view.
• Mode (per clone) — rewritten to ModeEventDriven on replay clones; preserved as-is on walk and recovery clones. The rewrite is required because router.Route gates on Mode == ModeEventDriven (router.go:134, 253, 413, 479, 566, 623, 844), and today's compile preserves a persistent prior.Mode = ModeScanOnly that would otherwise lock the action out of replay even after retention rehabilitates it.
• Action-map membership — replay's map contains only replay-eligible clones; walk's contains only walker-bound clones; recovery's contains every action.

Per-view settings are summarized in the table under "router.Route integration."

router.Route is the existing entry point — it computes ComputeDueAt, applies predicate filters, expands one event into 0+ Matches (e.g., a demoting PUT produces one for the new version's lifecycle and another for the displaced version's NoncurrentDays). The replay path delegates due-time math to the engine so all replay-eligible action kinds (ExpirationDays, NoncurrentDays, AbortMPU) are handled uniformly.

router.Route integration

Today's router.Route(ctx, snap *engine.Snapshot, ev, now, lister) iterates every action with IsActive() == true in the snapshot. IsActive is instance-level state on *CompiledAction (engine/compile.go:73-85, ca.markActive()), so two snapshots can't disagree on activation if they share the same *CompiledAction pointers.

Approach: views are independent *Snapshot instances with cloned CompiledAction objects.

Two gating fields are instance-level on *CompiledAction and must both match what the downstream call site expects:

• active (engine/compile.go:73-85) — gates routing and walker iteration.
• Mode — router.Route requires Mode == engine.ModeEventDriven at every emit site (router.go:134, 253, 413, 479, 566, 623, 844); the walker rejects only Mode == engine.ModeDisabled (walker.go:147). Today's compile preserves prior.Mode, so a rule previously promoted to ModeScanOnly will still be skipped by router.Route even if active=true.

engine.Snapshot.RulesForShard(shardID, retentionWindow) (replay, walk *Snapshot) returns two new *Snapshot instances. Each has its own actions map[ActionKey]*CompiledAction containing clones of the originals with both active and Mode rewritten per view. The partition predicate is "TTL ≤ retentionWindow" for replay membership; the engine does no clock arithmetic of its own — retentionWindow is the only window input it receives:

View	Clone settings	Why
replay	active = true, Mode = ModeEventDriven	router.Route requires ModeEventDriven. Forced regardless of prior.Mode, so a rule previously stuck in ModeScanOnly is rehabilitated when retention now fits its TTL.
walk	active = true, Mode preserved from original (any non-ModeDisabled is accepted by the walker)	Walker doesn't gate on a specific Mode value, so no rewrite needed beyond ensuring active.
recovery (see below)	active = true, Mode preserved	Recovery walker iterates all action clones; preserving Mode lets each rule's evaluator do its normal predicate work.

Shared-by-pointer with the base: Rule definitions, predicate maps, RuleHash table, and other immutable data — memory overhead is one CompiledAction struct per action per view (tens of bytes × dozens of actions, not megabytes).

Membership:

• replay: clones of ExpirationDays, NoncurrentDays, AbortMPU actions whose TTL ≤ retentionWindow. The Mode rewrite is what makes "rehabilitation" work — retention recovers, a previously ModeScanOnly-stuck rule clone now reads as ModeEventDriven in the replay view.
• walk: clones of ExpirationDate, ExpiredDeleteMarker, NewerNoncurrent, plus scan_only-promoted clones whose TTL exceeds retentionWindow.
• Either return value may be nil if its partition is empty.

The daily replay loop calls router.Route(ctx, replay, ev, now, lister). Steady-state walker calls walker.RunForShard(ctx, shardID, walk, limiter). Both call sites are drop-in — they see the correct, per-partition set with the correct (active, Mode) for their gating logic.

Recovery view (separate). The recovery branch needs the walker to see all replay-eligible + walker-eligible actions as active, including event-driven actions that the base snapshot may have marked inactive (compile.go:73-74: event-driven actions require BootstrapComplete). Add:

func RecoveryView(s *Snapshot) *Snapshot

RecoveryView is a third *Snapshot instance whose action clones are all markActive()'d, with Mode preserved from the original. The walker accepts any non-ModeDisabled Mode, so no rewrite is needed. The recovery branch calls walker.RunForShard(ctx, shardID, engine.RecoveryView(snap), limiter). Without this, recovery would silently skip any rule the base snapshot considers inactive — defeating the recovery branch's purpose of catching already-due objects across the full rule set.

RecoveryView does not participate in steady-state. It exists only for the recovery walker invocation.

scan_only input path. The retention boundary used for promotion is supplied explicitly via retentionWindow time.Duration, not derived inside the engine. daily_run computes it once at run start (retention_window = duration(now - metalog.EarliestAvailableTsNs(shardID))) and passes it to RulesForShard and PromotedHash. The promotion decision is therefore stable across one daily_run invocation — a rule that flips between replay and walk mid-run does not re-route; it gets its new bucket on the next day's RulesForShard call. RecoveryView takes only the base snapshot — it activates everything, partition-independent.

This approach keeps router.Route and walker.Run signatures unchanged. The trade-off is cloning CompiledAction per view (and rewriting Mode in the replay view); the alternative (adding a filter parameter to Route or restructuring IsActive/Mode to be view-scoped) would force every call site — including the streaming dispatcher during Phase 2/3 — to thread the filter, while clone-per-view localizes the partition decision to the engine. The Mode rewrite also bypasses today's persistent prior.Mode = ModeScanOnly lock-in, which is now obsoleted by the new partition logic.

Why correct:

• Meta-log is TsNs-ordered. For any rule R, due_time is also monotonic in TsNs (see "Due-time source"), so the first not-yet-due event for R proves every later event is also not-yet-due. Hence the done flag.
• Multiple rules fan out on the same scan: one meta-log read, N predicate checks per event.
• Identity drift (object overwritten between event and delete) is caught at the RPC by the existing LifecycleDelete identity-CAS, which returns NOOP_RESOLVED for stale events. The algorithm dispatches optimistically and lets the server filter.

Action kinds and dispatch paths

The engine defines six action kinds (weed/s3api/s3lifecycle/action_kind.go). Each has a distinct trigger and due-time semantics, and they split into two dispatch paths. Replay path uses meta-log iteration with the done early-stop; walker path lists current bucket state.

ActionKind	Trigger	Due time	Path	Early-stop?
ExpirationDays	Latest-version PUT	ev.TsNs + r.ExpirationDays	Replay	Yes — due strictly increasing in iteration order
NoncurrentDays	Demotion (next PUT for same key)	entry.noncurrent_since + r.NoncurrentDays	Replay	Yes — noncurrent_since is the demoting event's TsNs (see "noncurrent_since" below), monotonic
AbortMPU	MPU init	mpu_init.TsNs + r.AbortMPUDaysAfterInitiation	Replay	Yes — same monotonicity as ExpirationDays
ExpirationDate	Latest-version PUT, fired on now >= r.ExpirationDate	r.ExpirationDate (constant)	Walker	n/a — fixed wall-clock date, fires once per object
ExpiredDeleteMarker	Delete marker with NumVersions == 1 (no remaining siblings)	"now if orphaned, else never" — depends on current bucket state, not event age	Walker	n/a — requires sibling lookup, not derivable from event TsNs
NewerNoncurrent	Version becomes noncurrent AND total noncurrents > r.NewerNoncurrentVersions	"now if over the count cap, else never" — count is current state	Walker	n/a — requires version-list lookup

Why ExpiredDeleteMarker and NewerNoncurrent are walker-only:

Both rules' "due time" depends on the current state of sibling versions, not on any single event's timestamp. Modeling them as replay would force a sibling-list lookup on every meta-log event for the matching keys (today's router.Route does exactly this), and the done early-stop would never engage because the trigger condition isn't event-time-monotonic. They fit the walker far better: list each key's versions once per day, evaluate the rule against current state, emit deletes. Today's bootstrap/walker.go already handles them via EvaluateAction.

expected_mtime passed to LifecycleDelete is always the entry's original mtime (used for identity CAS), regardless of action kind. The TTL clock and the CAS identity are separate concerns.

Rule changes

Rule changes invalidate the persisted cursor's assumption that "everything older has already been processed under the same rules." The algorithm tracks rule_set_hash alongside the cursor and forces a bootstrap walk on mismatch. Cases this covers:

• New rule added → walker discovers existing objects under the new rule.
• TTL increased (e.g. 30d → 60d) → walker discovers objects in [now-60d, now-30d] that were skipped while the old rule ran.
• Predicate widened (prefix/tag filter change) → walker re-evaluates everything.
• TTL decreased / predicate narrowed → walker still runs (harmless re-evaluation), but the next daily run picks up steady-state quickly.

Bootstrap walker on rule change is a one-shot full bucket list. Cost is proportional to bucket size, not history; acceptable as an infrequent event.

Bootstrap retention window

The walker only deletes objects whose age already exceeds their TTL. Not-yet-due objects are intentionally left for the meta-log path. If the post-walker cursor were set to now, the next daily replay would scan from now forward and never see the PUT event for, say, a 20-day-old object under a 30-day rule — that event sits at TsNs ≈ now - 20d, before the cursor.

The cursor is therefore reset to now - max_ttl (excluding ScanAtDate rules from max_ttl, since those don't use the replay path). The clamp max(persisted, now - max_ttl) in the steady-state algorithm preserves the sliding window: each day the effective scan range is [now - max_ttl, now], and not-yet-due events surface naturally as they age in.

Invariant this depends on: meta-log retention ≥ max_ttl. If retention is shorter, events for some still-pending objects age out before they become due. Detected at run start by comparing the meta-log's earliest available TsNs against now - max_ttl; if the gap is non-empty, the affected rules are flagged scan_only for that run and routed to the walker (see "Scan-at-date and scan-only paths").

Replay vs walker: single source of truth

Snapshot.RulesForShard(shardID, retentionWindow) (replay, walk *Snapshot) is the only place that decides which path each compiled action takes during steady state. Both return values are new *Snapshot instances containing cloned *CompiledAction values with per-partition active flags (see "router.Route integration" for why cloning is required); either may be nil when its partition is empty. Every other component (daily_run, the walker invocation, the cursor's RuleSetHash, the max_ttl calculation) reads through these views. A given compiled action's clone has active=true in exactly one of replay or walk:

Reason for walker	Engine signal
ExpirationDate (ModeScanAtDate)	compile.go:73 sets mode == ModeScanAtDate
ExpiredDeleteMarker	Action kind requires sibling count
NewerNoncurrent	Action kind requires version count
scan_only (TTL > meta-log retention)	event_log_horizon.go check against earliest available TsNs at run start

Everything else (ExpirationDays, NoncurrentDays, AbortMPU when TTL fits in retention) is on the replay list.

The effective_ttl per active replay action — used by engine.MaxEffectiveTTL(replay) — is:

• ExpirationDays.ExpirationDays
• NoncurrentDays.NoncurrentDays
• AbortMPU.AbortMPUDaysAfterInitiation

Walker actions don't appear as active in replay, so they naturally contribute nothing to MaxEffectiveTTL — the cursor sliding window only governs replay.

If snap.replay is empty (e.g., a bucket whose only rule is ExpirationDate), the cursor's TsNs is recorded as 0 (or any sentinel) and never read; only RuleSetHash matters, so a future addition of a replay rule is detected and triggers the rewind branch. The walker still runs every day for the walker rules.

Cursor hashes: RuleSetHash and PromotedHash

The cursor stores two hashes. Together they detect every situation that invalidates the cursor's "everything before persisted.TsNs has been processed under the same rules" assumption.

RuleSetHash = engine.ReplayContentHash(snap) — content over the rule definitions (action kind, predicate, TTL value) of replay-eligible action kinds (ExpirationDays, NoncurrentDays, AbortMPU) in the base snapshot. Partition-independent.

PromotedHash = engine.PromotedHash(snap, retentionWindow) — hash of the set of replay-eligible rules that are currently classified as walk due to scan_only (their TTL exceeds retentionWindow). Changes when retention shifts move rules between partitions, even if XML rules and their content are unchanged.

Why two hashes: a replay-cursor rewind is needed when either the rule definitions change or when a rule transitions between partitions in either direction.

• Promotion (replay → walk, retention dropped): the promoted rule's meta-log events for in-flight objects are no longer reachable. Walker must take over for that rule. Cursor for other replay rules is fine, but the recovery branch's full-rule-set walk is still the safest catch-up — PromotedHash change triggers it.
• Demotion (walk → replay, retention recovered): the rule's events are now reachable again in meta-log. But persisted.TsNs may be forward of where the rule's relevant events live, because other replay rules kept the cursor advancing while this rule was in walk. The persisted.TsNs < earliest_available check does not catch this — that check fires only on retention regression. Demotion is detected by PromotedHash change.

Recovery triggers, complete list:

Trigger	Detection	Why
Cold start	No persisted cursor	First run
Replay-rule edit	RuleSetHash mismatch	Content of replay-eligible rules changed
Retention loss	replay != nil && persisted.TsNs < earliest_available	Worker was down or retention shrank past cursor. Gated on replay != nil so the TsNs sentinel (0) used by pure walker buckets doesn't perpetually trigger recovery.
Partition flip	PromotedHash mismatch	A replay-eligible rule moved between replay and walk

Walker-only rule edits (ExpirationDate, ExpiredDeleteMarker, NewerNoncurrent, or any rule with no replay-eligible action kind) do not change either hash. They are picked up by the steady-state walk invocation; replay cursor is untouched.

Practical consequences:

• Adding/removing/modifying a walker-only rule: both hashes unchanged, replay cursor untouched.
• Retention drops (replay rule promotes to walk): RuleSetHash unchanged, PromotedHash changes → recovery fires. Walker over base catches already-due objects; cursor rewinds for replay continuity.
• Retention recovers (rule demotes back to replay): RuleSetHash unchanged, PromotedHash changes → recovery fires. The demoted rule's events get a fresh scan window via the rewind.
• Editing a replay-eligible XML rule: RuleSetHash changes → recovery fires. Walker covers the full rule set; already-due objects under the new rule are caught.

Shard-aware walker

Today's walker (bootstrap/walker.go) is bucket-wide: it lists every entry under a bucket and emits matches regardless of ShardID. Shards are derived as sha256(bucket||"/"||key) >> 252 (weed/s3api/s3lifecycle/shard.go:23), so a single bucket's objects spread across all 16 shards.

Daily replay is per-shard, so a per-shard daily_run(shardID) invoking the bucket-wide walker would have all 16 shards re-walk every bucket — 16× the filer listing cost, 16× the duplicate delete RPCs (idempotent but wasteful and noisy in metrics).

Two paths are workable; the design picks the simpler one and notes the optimization:

1. Shard-filtering walker (chosen). Add walker.RunForShard(shardID int, ...). The walker still does one full bucket listing, but for each entry it computes ShardID(bucket, key) and skips entries not in the target shard. Cost: still 1 listing per shard per bucket = 16× listing total. Simpler; no coordination across workers. Acceptable for buckets up to ~100M entries where listing is cheap relative to deletes.
2. Bucket-coordinated walker (future). A separate coordinator (e.g., the worker that owns shard 0 for that bucket) runs the walker once, routing emitted matches to each shard's local injection point. 1 listing per bucket total. Requires a coordination layer that today's plugin/worker model doesn't have. Tracked as an optimization, not blocking.

Path #1 is wire-compatible with today's walker — only the iteration filter changes. RunForShard is a thin wrapper that takes a shard predicate; the existing Run(bucket) becomes RunForShard(shard, bucket) with shard == any.

Delete failure handling

Cursor advance is gated on success. The cursor only moves past events whose deletes returned DONE, NOOP_RESOLVED, or SKIPPED_OBJECT_LOCK (lock-protected, but rule semantics are satisfied). Any other outcome — RETRY_LATER, BLOCKED, or transport error after in-run retries — halts the run and persists the cursor at the last fully-processed event.

Semantics:

• Head-of-line blocking is intentional. A transient filer error stalls the rest of today's pass; tomorrow's run resumes at the same cursor and retries from the failing event. This is loud — operator sees a stuck cursor in metrics — and idempotent (identity-CAS makes any redundant delete a no-op).
• In-run retry with backoff for transport errors only (default 3 attempts, exponential backoff). Server-side outcomes (RETRY_LATER, BLOCKED) are not retried in-run; the server already classified them as "wait until next run."
• No separate retry queue. Removing the per-key freeze state was the whole point — adding it back would re-introduce the state machine this design replaces.

Data model changes

noncurrent_since on version entries

A non-current version's TTL clock starts when the next version was written, not at its own mtime. Today there's no field for that timestamp; the engine has to derive it by joining successive PUTs for the same key.

Change: at the S3 PUT handler that demotes a prior version (current → noncurrent), write noncurrent_since on the demoted entry to the TsNs of the demoting event, not local wall-clock. The demoting event is itself a meta-log PUT with a well-defined TsNs; using that value keeps noncurrent_since strictly monotonic in meta-log order across all replicas and immune to wall-clock skew between S3 handler nodes.

The lifecycle evaluator uses ev.TsNs for current-version rules and entry.noncurrent_since for noncurrent rules — both monotonic in the iteration order, so the algorithm's done flag still holds. expected_mtime (passed to LifecycleDelete for identity CAS) remains the entry's own mtime; CAS identity and TTL clock are kept separate.

Cursor

Per-shard cursor is { TsNs int64, RuleSetHash [32]byte, PromotedHash [32]byte }. No frozen-cursor set, no per-key freeze state — failed deletes halt the run and leave the cursor at the last successful event (see "Delete failure handling"). RuleSetHash and PromotedHash together gate recovery — mismatch on either triggers the recovery branch on the next run. See "Cursor hashes."

Components

Component	Status
engine/ (rule compilation, snapshot)	Keep
evaluate.go, due_at.go, rule_hash.go, tags.go	Keep
reader/ (meta-log subscribe)	Keep, used once per daily pass instead of continuously
router/router.go (rule evaluation per event)	Keep, callable in the per-event loop
router/schedule.go (heap)	Remove
dispatcher/dispatcher.go (tick, retries, freeze)	Remove; per-call LifecycleDelete moves into the daily loop
dispatcher/pipeline.go	Replace with daily_run per shard
scheduler/scheduler.go (long-running goroutines + refresh ticker)	Replace with one-shot per-shard fan-out
bootstrap/walker.go	Keep, with shard filter added (RunForShard). Two invocation modes: (a) steady-state daily against the walk view (ExpirationDate, ExpiredDeleteMarker, NewerNoncurrent, scan_only rules); (b) recovery against engine.RecoveryView(snap) (cold start, replay-rule edit, retention loss, partition flip). No longer feeds an in-memory schedule.

Rate limiting

Cluster-wide deletes-per-second cap, set as admin config. Admin computes per-worker share at job-dispatch time:

• Admin counts workers capable of s3_lifecycle from the registry, divides cluster_deletes_per_second by the count, writes the share into ExecuteJobRequest.ClusterContext.Metadata["s3_lifecycle.deletes_per_second"].
• Worker reads the share and constructs one golang.org/x/time/rate.Limiter shared across all shard goroutines. dispatch_delete calls limiter.Wait(ctx) before each RPC.
• Brief over/undershoot during worker join/leave is acceptable for a bulk workload.

No proto changes: ClusterContext.Metadata already exists.

Failure & recovery

• Worker crashes mid-run. Cursor advances only past successfully-deleted events (see "Delete failure handling"); on restart the next run resumes at the same cursor and re-attempts. Identity-CAS makes redundant deletes no-ops.

• Transient delete failure. Run halts at the failing event, cursor stays. Tomorrow's run retries from the same point. Stuck cursor is visible in metrics — operators see head-of-line blocking and address root cause. Four triggers route into the same recovery branch (cold-start, replay-rule edit, retention loss, partition flip). All four run the walker against engine.RecoveryView(snap) (every action force-markActive()'d, regardless of compile-time IsActive/BootstrapComplete) to catch already-due objects across the full rule set, then rewind the cursor to now - MaxEffectiveTTL(replay) (or sentinel 0 if replay is empty). The merge is by design: distinguishing the four would require independent state that isn't worth the complexity, and the walker is idempotent.

• Cold start (no persisted cursor). Recovery branch.

• Replay-rule edit (new/removed/changed ExpirationDays, NoncurrentDays, or AbortMPU rule). Detected by RuleSetHash mismatch. Recovery branch.

• Retention loss (replay != nil && persisted.TsNs < earliest_available). Worker was down longer than meta-log retention, or retention itself shrunk. Gated on replay != nil so the sentinel TsNs=0 used by pure walker buckets doesn't perpetually re-trigger recovery. Recovery branch.

• Partition flip (PromotedHash mismatch — a replay-eligible rule moved between replay and walk since last run). Covers both directions:

  • Promotion (replay → walk, retention dropped): walker takes over for that rule. Recovery fires to refresh state.
  • Demotion (walk → replay, retention recovered): persisted.TsNs is likely forward of the rule's relevant events (other replay rules kept advancing during the walk era). Recovery rewinds the cursor so the demoted rule's events get a fresh scan window.

• Walker-only rule edit (added/removed/modified ExpirationDate, ExpiredDeleteMarker, NewerNoncurrent, or any rule with no replay-eligible action kind). Not a recovery trigger — both hashes unchanged, cursor untouched. Walker re-reads walk on every steady-state run and applies the new rule automatically.

• ExpirationDate rules. Always routed to the walker — fixed wall-clock due time can't be derived from event TsNs, and objects created before the date may have aged out of meta-log. See "Replay vs walker."

• Identity drift. Handled by LifecycleDelete RPC as today. No worker-side state needed.

Comparison

	Streaming (today)	Daily replay (this design)
Worker lifetime	Long-running, capped by MaxRuntime	Starts, completes pass, exits
Memory	O(events with DueTime in [now, now+TTL])	O(in-flight delete RPCs)
Disk reads	Continuous meta-log subscribe + periodic bootstrap	One meta-log scan per day, length ≤ max_ttl
Latency	DispatchTick (~1 min)	Up to 24h
Future-buffering	Yes, in heap	No
State persisted	Cursor + frozen set + heap (rebuilt from meta-log on restart)	Cursor only
Dispatch tick / checkpoint tick / refresh tick	Required	Removed
Bootstrap walker	Hot path (every run, feeds heap)	Daily for walk rules; recovery for full rule set
Failure recovery	Heap rebuild from meta-log replay	Cursor advance

Configuration changes

The admin and worker forms collapse significantly. New worker config:

deletes_per_run_concurrency    // optional, in-pass parallelism (default 1)
max_runtime_minutes            // safety cap; default 1440 (24h)

Removed worker config (no longer meaningful):

dispatch_tick_minutes
checkpoint_tick_seconds
refresh_interval_minutes
bootstrap_interval_minutes

New admin config (added to the existing form):

cluster_deletes_per_second     // 0 = unlimited
cluster_deletes_burst          // 0 = 2× rps

Admin runtime stays at DetectionIntervalMinutes=1440 (daily detection produces one execution per worker).

---

Implementation phases

Sequenced so each phase compiles and ships independently. Phase 1 lands first because it's a prerequisite the algorithm relies on; phases 2–4 can be authored in parallel but should merge in order.

Phase 1 — noncurrent_since on version entries

Smallest, isolated, prerequisite.

• Add NoncurrentSinceNs int64 to the filer entry extended attributes (or as a top-level field if the proto allows non-breaking add).
• At the S3 PUT handler in weed/s3api/s3api_object_handlers_put.go (or whichever site demotes the prior current version), set NoncurrentSinceNs to the TsNs of the demoting meta-log event, not time.Now().UnixNano(). Read the value from the in-flight write's TsNs (the filer assigns this) — coordinate with the filer write API to make that value available to the demotion path. If the API can't surface it cleanly, fall back to a single time.Now().UnixNano() captured at the start of the demoting RPC, accepting that this introduces sub-millisecond skew relative to meta-log TsNs (still monotonic per node, weakly monotonic across nodes within clock-sync tolerance).
• Backfill: nil/zero on existing entries — the lifecycle evaluator treats NoncurrentSinceNs == 0 as "fall back to entry mtime" so legacy data still expires, just with the old (slightly looser) semantics.
• Unit tests in the s3api package covering PUT-over-existing and PUT-with-versioning-suspended.
• Verify with a clock-skew test: two S3 nodes with ±5s clock drift produce a sequence of PUT/demotion events; assert noncurrent_since is non-decreasing in meta-log iteration order.

Phase 2 — Daily-replay worker (parallel to today's, behind a flag)

Add the new code path without disturbing the existing one. Allows side-by-side smoke testing.

• New file weed/s3api/s3lifecycle/dailyrun/dailyrun.go with func Run(ctx, shardID, ...) error implementing the algorithm.
• Reuses engine.Snapshot, evaluate.go, due_at.go, the existing reader.Reader, and the existing LifecycleDelete RPC. No new RPCs.
• Per-shard cursor type dailyrun.Cursor { TsNs int64; RuleSetHash [32]byte; PromotedHash [32]byte }, JSON-persisted under the same filer path as today's cursor, keyed by daily/ prefix to avoid collision. Cursor load returns (cursor, isFresh bool) so the algorithm can distinguish first-run from a rule-change rewrite. In Phase 2 (no walker plumbing), PromotedHash is always the empty hash; Phase 4 wires engine.PromotedHash(snap, retentionWindow) and the partition-flip recovery trigger.
• Replay/walker partition: full RulesForShard / RecoveryView plumbing lands in Phase 4. For Phase 2 alone, since walker rules are rejected (next bullet), Phase 2 is replay-only: pass the base snapshot directly to router.Route (no view, no clone, no Mode rewrite). This is acceptable for Phase 2 because the unsupported-rule check guarantees no walker-bound actions are present, and the absence of walker rules means there's no risk of router.Route emitting walker-bound matches. Mode rewrite is also unnecessary in Phase 2: any rule that compiled to ModeScanOnly (legacy persistent state) falls into the unsupported-rule path along with scan_only-promoted rules. Phase 4 then introduces clone-per-view (including Mode rewrite) and the recovery branch.
• Refuse to run on unsupported rule sets. Before invoking the replay loop, dailyrun.Run checks the compiled rules for any action of kind ExpirationDate, ExpiredDeleteMarker, NewerNoncurrent, or any rule promoted to scan_only. If any are present, the handler:
  • Logs at error level which rules are unsupported.
  • Returns a typed error from Execute so admin marks the run failed (visible in the activity log).
  • Does not advance the cursor. This makes flipping algorithm: daily_replay before Phase 4 a loud failure, not a silent dropped rule. Operators see the error and either revert to streaming or wait for Phase 4.
• Rule-change handling: compute rsh = engine.ReplayContentHash(snap) at run start (partition-independent — see "Cursor hashes"). If rsh != cursor.RuleSetHash and replay is non-nil, persist { TsNs: now - MaxEffectiveTTL(replay), RuleSetHash: rsh, PromotedHash: promoted } and return. If replay is nil, persist { TsNs: 0, RuleSetHash: rsh, PromotedHash: promoted } and return. In Phase 2, promoted is the empty hash (no walker plumbing); Phase 4 wires real values. All cursor writes carry all three fields so Phase 4 doesn't have to migrate existing Phase 2 cursors. Tested explicitly: add a replay-eligible rule mid-test, assert cursor TsNs rewinds to now - max_ttl and RuleSetHash updates.
• Delete failure handling: track last_ok TsNs as the algorithm progresses; only set last_ok = ev.TsNs after a successful or NOOP_RESOLVED outcome for all matches produced from that event. On any unresolved outcome (RETRY_LATER, BLOCKED, transport error after in-run retries), break the loop and persist { TsNs: last_ok, RuleSetHash: rsh, PromotedHash: promoted }. In-run retry policy for transport errors: 3 attempts with exponential backoff capped at 5s.
• Match expansion: invoke router.Route(ctx, snap, ev, now, lister) per event, where snap is the base snapshot directly (Phase 2 is replay-only — see "Replay/walker partition" bullet above; the view-based call site router.Route(ctx, replay, ev, ...) lands in Phase 4 when the partition is wired). Handles ExpirationDays, NoncurrentDays, AbortMPU correctly out of the box (already in due_at.go).
• Rate limiter accepted as a parameter; constructed once per Execute() and shared across all shard goroutines.
• New admin config flag algorithm: streaming | daily_replay in weed/worker/tasks/s3_lifecycle/handler.go. When daily_replay, the handler invokes dailyrun.Run and exits. When streaming (default), today's code path runs unchanged.
• Unit tests:
  • Per-rule done flag correctness (single rule, multi-rule).
  • Multiple-match fan-out from one event (PUT-over-existing emits both ExpirationDays for the new version and NoncurrentDays for the displaced one).
  • Cursor advance halts at the first unresolved failure; subsequent run resumes there.
  • Identity-CAS skip path (event mtime stale → NOOP_RESOLVED still advances cursor).
  • Rule-set-hash mismatch rewinds cursor to now - max_ttl (Phase 4 adds the walker call on this path; Phase 2 just rewinds).
  • replay empty case: cursor persists { TsNs: 0, RuleSetHash: rsh, PromotedHash: <empty in Phase 2> }, no meta-log read attempted.
  • NoncurrentDays rule uses entry.noncurrent_since, ExpirationDays rule uses ev.TsNs; both monotonic in iteration order.
  • AbortMPU test: MPU init event ages past AbortMPUDaysAfterInitiation, delete emitted.
  • Unsupported-rule rejection: bucket has an ExpirationDate (or ExpiredDeleteMarker or NewerNoncurrent) rule and the flag is set to daily_replay. Assert Execute returns the typed unsupported-rule error, the cursor is not advanced, and the streaming code path was not invoked. Repeat for a rule promoted to scan_only.

Phase 3 — Cluster rate limit allocation

Admin-side share computation. Lands once Phase 2 has a working Limiter parameter to receive it.

• weed/worker/tasks/s3_lifecycle/handler.go: add cluster_deletes_per_second, cluster_deletes_burst to AdminConfigForm.Sections["scope"].
• weed/admin/plugin/plugin.go near RunDetectionRequest/ExecuteJobRequest build sites: count capable workers, divide rps, write s3_lifecycle.deletes_per_second and s3_lifecycle.deletes_burst into ClusterContext.Metadata.
• Worker Execute: parse metadata keys, build rate.Limiter, pass to dailyrun.Run.
• New histogram S3LifecycleDispatchLimiterWaitSeconds for operator visibility.
• Tests: admin allocation math under join/leave, worker correctly slows under low share.

Phase 4 — Shard-aware walker + scan-at-date / scan-only routing

Reposition the walker and add shard-filtering + per-rule routing.

• Shard filter on the walker. Add walker.RunForShard(ctx context.Context, shardID int, view *engine.Snapshot, limiter *rate.Limiter) error. view is one of walk/replay/recovery views — its action set already filters to the right partition (via per-clone active flags), so the walker iterates the bucket and evaluates each entry against view exactly as today's Run does, with the only added step being if ShardID(bucket, entry.Name) != shardID { continue }. The existing bucket-wide Run becomes RunForShard with shardID = -1 (any) for backward compatibility, or is deleted outright in Phase 5.
• Engine surface. Four new exports on the engine package, all consumed by daily_run:
  • func (s *Snapshot) RulesForShard(shardID int, retentionWindow time.Duration) (replay, walk *Snapshot). Returns two new *Snapshot instances with cloned *CompiledAction values; per-view active flags (and Mode rewrite on replay) determine partition membership. Shared-by-pointer with the base: Rule definitions, predicate maps, RuleHash table. Either return value may be nil when its partition is empty. The retention window is an explicit input — daily_run computes it once at run start as duration(now - metalog.EarliestAvailableTsNs(shardID)) and passes it in. The engine does no clock arithmetic of its own. The partition is stable across one daily_run invocation. Membership:
    • replay: cloned ExpirationDays, NoncurrentDays, AbortMPU actions, active=true, where the action's TTL ≤ retentionWindow.
    • walk: cloned ExpirationDate (ModeScanAtDate), ExpiredDeleteMarker, NewerNoncurrent actions, plus any of the above promoted to scan_only because their TTL exceeds the retention boundary. All active=true.
    • A single XML rule that compiles to multiple action kinds (e.g., ExpirationDays + NewerNoncurrent on the same rule) is split: each compiled action lands in replay or walk independently. ReplayContentHash only hashes replay-eligible rule definitions from the base snapshot, so neither the NewerNoncurrent action nor any walker-only edit affects the hash.
  • func RecoveryView(s *Snapshot) *Snapshot. Returns a third *Snapshot instance with every action cloned and markActive()'d — regardless of Mode or prior BootstrapComplete state. The recovery walker uses this view to see all replay-eligible + walker-eligible rules. Without it, recovery would skip event-driven actions the base snapshot marks inactive (compile.go:73-74).
  • func ReplayContentHash(s *Snapshot) [32]byte. Stable across reorderings, partition-independent. Used as RuleSetHash = ReplayContentHash(snap). Returns the empty hash when no replay-eligible rules exist. See "Cursor hashes."
  • func PromotedHash(s *Snapshot, retentionWindow time.Duration) [32]byte. Hash of the set of replay-eligible rules currently classified as walk due to scan_only (TTL > retentionWindow). Detects partition flips in either direction. Takes the same retentionWindow value as RulesForShard so the two cannot disagree about partition membership. See "Cursor hashes."
  • func MaxEffectiveTTL(s *Snapshot) time.Duration. Max over active replay actions; returns 0 for nil or empty replay (caller must already be in the empty-replay branch).
  • func CurrentSnapshot() *Snapshot. The base snapshot before partitioning. Recovery branch passes this to RecoveryView.
• dailyrun.Run flow. Cold-start, replay-rule edit (RuleSetHash mismatch), retention loss, and partition flip (PromotedHash mismatch) all route into the recovery branch: walker over engine.RecoveryView(snap) so already-due objects across all rules are caught (including event-driven actions the base snapshot may mark inactive), then cursor rewinds. Steady-state path runs walker over walk only, then replay over replay. Cursor stores { TsNs, RuleSetHash, PromotedHash }; steady-state writes both hashes through unchanged.
• Recovery branch. Triggered by any of: no persisted cursor, RuleSetHash mismatch, PromotedHash mismatch, or replay != nil && persisted.TsNs < earliest_available. The retention-loss trigger uses the absolute earliest_available TsNs (the oldest available meta-log TsNs from metalog.EarliestAvailableTsNs(shardID)), not the retentionWindow duration that RulesForShard/PromotedHash consume — they're different shapes for different purposes: the retention check on the cursor compares two absolute timestamps, while the engine partition check compares a per-action TTL to a duration. Gated on replay != nil because pure walker buckets persist a sentinel TsNs=0 that would otherwise re-fire recovery every run. Invokes walker.RunForShard(ctx, shardID, engine.RecoveryView(snap), limiter). Persists cursor at:
  • { TsNs: now - engine.MaxEffectiveTTL(replay), RuleSetHash: rsh, PromotedHash: promoted } if replay is non-nil.
  • { TsNs: 0, RuleSetHash: rsh, PromotedHash: promoted } if replay is nil (pure walker bucket). MaxEffectiveTTL is undefined over an empty set, so the sentinel form is required when replay is nil. Both hashes are written on every recovery persistence — failing to update PromotedHash here would cause the partition-flip trigger to re-fire on every subsequent run. See "Bootstrap retention window."
• Steady-state walker invocation. Distinct from recovery: walker.RunForShard(ctx, shardID, walk, limiter). walk's cloned actions are masked to walker-bound kinds only, so the walker doesn't re-process replay rules that the meta-log scan already handles on the same run.
• Drop the bootstrap walker's call from scheduler.go (removed entirely in Phase 5).
• Tests:
  • Cold start with mixed walker-only + replay rules in same bucket: recovery branch fires once (walker over engine.RecoveryView(snap), then cursor rewind). Next day's run is steady-state (walker over walk only, replay over replay).
  • ExpirationDate rule fires only on/after the date (walker path).
  • ExpiredDeleteMarker rule deletes orphaned markers (walker path, sibling-aware).
  • NewerNoncurrent count rule deletes oldest noncurrent versions beyond the cap (walker path, version-list-aware).
  • Bucket with only walker rules (e.g., pure ExpirationDate): replay is nil, cursor persisted with sentinel TsNs=0, no meta-log subscribe attempted. Critically — assert that across 5 consecutive daily runs, recovery fires exactly once (on the first run); subsequent runs see no recovery despite persisted.TsNs (0) < earliest_available, because the retention-loss check is gated on replay != nil.
  • Replay-rule edit triggers recovery: walker runs over RecoveryView(snap) (catches an object already 60d old under a brand-new 30d rule), then cursor rewinds. Object aged 20d at edit time is later deleted at age 30 via the replay path.
  • RecoveryView activates event-driven actions: simulate a base snapshot where an ExpirationDays action has IsActive=false (e.g., BootstrapComplete=false); assert the recovery walker still sees and evaluates the rule via the recovery view, while the steady-state walker over walk does not.
  • Replay view rewrites Mode: simulate a base snapshot with a ModeScanOnly-locked ExpirationDays action (carried over from prior.Mode); assert that replay's clone of the action has Mode == ModeEventDriven, and that router.Route(ctx, replay, ev, ...) emits a Match for it. The test should fail if Mode rewrite is skipped — confirming the rehabilitation path is wired.
  • Cursor persistence after recovery: assert PromotedHash is written through on every recovery branch persistence; simulate a partition flip across two runs and verify the second run does not re-fire recovery because the prior promoted_hash was correctly stored.
  • Adding a walker-only rule to an existing replay-only bucket: ReplayContentHash(snap) is unchanged, replay cursor TsNs unchanged, walker picks up the new rule on its next steady-state pass.
  • scan_only promotion (retention drops, replay-eligible rule moves to walk): ReplayContentHash unchanged, PromotedHash changes → recovery branch fires. Walker over base catches in-flight objects under the promoted rule; cursor rewinds; PromotedHash is persisted.
  • Retention recovery (scan_only rule demotes back to replay while other replay rules' cursor has advanced past the rule's relevant events): ReplayContentHash unchanged, persisted.TsNs > earliest_available so retention-loss check does NOT fire, PromotedHash changes → recovery branch fires. Asserts the demoted rule's older events get processed (this is the test that would have failed before adding PromotedHash).
  • PromotedHash persistence: simulate a partition flip in either direction across two runs, assert cursor's promoted_hash field carries through and gates the next run's recovery decision correctly.
  • Shard filter: 2 shards × 1 bucket walker run leaves no double-deletes (assert via RPC counter).
  • scan_only promotion: rule with TTL > meta-log retention moves from replay to walk at run start.

Phase 5 — Switch default and delete streaming path

Cut over and clean up. Should be a single PR that's mostly deletions.

• Default algorithm: daily_replay and remove the streaming alternative from the descriptor.
• Delete:
  • weed/s3api/s3lifecycle/router/schedule.go (+ tests)
  • weed/s3api/s3lifecycle/dispatcher/dispatcher.go (+ tests), freeze-related code, retry state
  • weed/s3api/s3lifecycle/dispatcher/pipeline.go
  • weed/s3api/s3lifecycle/scheduler/scheduler.go (+ tests), refresh ticker
  • Removed worker config fields and their ParseConfig handling
• Rewrite weed/s3api/s3lifecycle/DESIGN.md to remove "streaming" sections.
• One full end-to-end test on a bucket with 1M objects, mixed-TTL rules, with cluster rate cap engaged.

Phase 6 — Observability polish

After cutover, before announcing.

• Daily-run summary log line: shard, rules-touched, events-scanned, deletes-issued, walker-fallback (y/n), duration.
• Prometheus: S3LifecycleDailyRunDurationSeconds, S3LifecycleDailyRunEventsScanned, S3LifecycleDailyRunDeletesIssued, all labelled by shard_id and outcome.
• Admin UI: a single "Last Run" card per bucket showing the above summary line. Removes the now-meaningless "Next Run / Detection Results" widgets we already hid for s3_lifecycle.

Open questions

1. In-pass concurrency. Today's worker has 1–16 shard goroutines via workers config. Daily replay can either (a) keep that knob, fanning out shard scans across goroutines that share the limiter, or (b) drop it since the limiter is now the throughput governor. Lean toward (a) — shards are independent meta-log subscriptions and parallel reads cut wall-clock significantly.
2. Cursor format compatibility. Worth deciding whether the new cursor coexists at a separate filer path (clean cutover, no migration) or overwrites the old one (single source of truth, requires a one-time read-then-rewrite during cutover). Lean toward separate paths during phases 2–4, unified at Phase 5.
3. Per-shard rule snapshots vs. global. Engine snapshots are bucket-scoped today; Phase 4 adds Snapshot.RulesForShard(shardID, retentionWindow) (replay, walk *Snapshot). Open: should RulesForShard also gate by which buckets a shard "owns" (every bucket is on every shard via ShardID(bucket, key)), or just return the global rule set partitioned by mode? Current lean: return global partition, since every shard worker runs against every bucket and the actual shard filter happens at the entry-iteration site (meta-log subscription and walker entry loop).
4. Bucket-coordinated walker. Phase 4 has each shard walk the full bucket and filter by ShardID — simple, but 16× the listing cost. A future optimization is a per-bucket coordinator (the worker owning shard 0 for that bucket lists once, routes matches to other shards via the existing injection point). Worth doing if listing cost becomes the bottleneck for very large buckets. Not blocking initial cutover.
5. noncurrent_since source from filer write API. Phase 1 prefers the demoting event's TsNs over wall-clock. Whether the existing filer write path surfaces that TsNs to the S3 handler without a proto change is a Phase 1 design decision; the fallback (handler-local time.Now()) is documented but loses cross-node monotonicity guarantees and should be a temporary measure.
6. Retry budget for RETRY_LATER. The current dispatcher tracks per-key retry counts; this design removes that state and lets head-of-line blocking surface the issue. Worth confirming that operators are OK with a stuck cursor as the failure signal vs. silent retry. If not, add a bounded retry counter on the cursor (one per stuck event) — small state, but inches back toward the model we're replacing.