seaweedfs/weed/s3api/s3lifecycle/DESIGN.md

S3 Lifecycle (Design)

The S3 lifecycle worker replaces the streaming + heap design with a daily meta-log replay. The worker runs as a scheduled job: "start, do today's work, stop" — no long-running per-shard goroutines, no future-buffered match heap.

This document is the as-built reference. For operator-facing guides, see the SeaweedFS wiki.

Goal

For each bucket lifecycle rule with TTL D days, the worker processes every object whose age exceeds D during each scheduled run, dispatching deletes via LifecycleDelete. Events whose dispatch returned a retryable / blocked outcome are reprocessed from the persisted cursor on later runs — head-of-line blocking is intentional rather than a per-key retry queue. No future-buffered Matches in memory. Cluster-wide delete rate cap allocated per worker. The worker exits when the pass completes.

Algorithm

One pass per scheduled invocation. The pass spawns one filer meta-log subscription, fans events out to per-shard processors by ShardID(bucket, key), and drains each shard's events through the router + dispatcher:

dailyrun.Run(ctx, cfg):
    runNow             = cfg.Now()                                  // frozen for the whole pass
    snap               = cfg.Engine.Snapshot()
    rsh                = engine.ReplayContentHash(snap)
    maxTTL             = engine.MaxEffectiveTTL(snap)

    cursors            = load every shard's persisted cursor
    walks              = planWalks(cursors) → at most 2 shared walks (recovery, steady)

    if rsh != [32]byte{}:                                           // replay-eligible rules present
        globalStartTsNs = min over cfg.Shards of (persisted cursor or runNow - maxTTL)
        reader          = subscribeMeta(ShardPredicate ∈ cfg.Shards, StartTsNs = globalStartTsNs)
        fanOut(reader.Events → shardEvents[shardID]) until ev.TsNs > runNow

    spawn one goroutine per shard:
        runShard(ctx, cfg, snap, runNow, shardID, shardEvents[shardID], walks)

    wait all
    teardown reader + fan-out

runShard(ctx, cfg, snap, runNow, shardID, events):
    persisted, found   = cfg.Persister.Load(shardID)
    retentionWindow    = cfg.RetentionWindow or maxTTL              // see "retention" below
    promoted           = engine.PromotedHash(snap, retentionWindow)

    if rsh == [32]byte{}:                                           // pure walker bucket
        if walkerDue and Walker:
            walkForShard(walkView)                                // shared walk, per-shard fallback
            lastWalkedNs = runNow.UnixNano()
        save cursor (TsNs=0, rsh, promoted, lastWalkedNs)
        return

    mustWalkRecovery   = found && (persisted.RuleSetHash != rsh || persisted.PromotedHash != promoted)
    mustWalkColdStart  = !found

    if mustWalkRecovery or mustWalkColdStart:
        walkForShard(RecoveryView(snap))                          // every rule, force-active
        walkedThisPass = true
        lastWalkedNs   = runNow.UnixNano()
        if mustWalkRecovery:
            save cursor (TsNs = runNow - maxTTL, rsh, promoted, lastWalkedNs)  // rewind
            return

    if Walker and !walkedThisPass and walkerDue:
        walkForShard(walkView)                                    // steady-state walker
        lastWalkedNs = runNow.UnixNano()

    startTsNs = !found ? runNow - maxTTL : persisted.TsNs           // steady state honors cursor
    replayView = snap.ReplayView(retentionWindow)                   // minus walk-bound rules
    lastOK, _, drainErr = drainShardEvents(ctx, cfg, runNow, shardID, replayView, startTsNs, events)
    save cursor (TsNs = lastOK, rsh, promoted, lastWalkedNs)

Walker invocations — three call sites, distinct conditions. cfg.Walker takes a shard set, not a single shard: planWalks predicts each shard's branch from the pre-loaded cursors and runs one shared walk per group, so a pass lists each bucket at most twice (recovery group + steady group) instead of once per shard. runShard waits on its group's walk; a shard whose own freshly-loaded cursor disagrees with the plan falls back to a per-shard walk, so the plan can't under-cover a shard after a cursor is rewritten mid-pass.

Branch	View	Trigger	Throttle
Recovery	engine.RecoveryView(snap)	mustWalkColdStart or mustWalkRecovery	Unconditional
Steady state	RulesForShard.walk	!walkedThisPass && walkerDue	WalkerInterval
Empty replay	RulesForShard.walk	rsh == [32]byte{} and walkerDue	WalkerInterval

walkerDue returns true when WalkerInterval == 0, or when no walk has happened yet (LastWalkedNs == 0), or when runNow - LastWalkedNs >= WalkerInterval. Within-pass double-fire suppression lives in runShard's walkedThisPass flag, not in walkerDue — the recovery branch fires the walker with RecoveryView (a superset of every per-shard partition), so the steady-state branch must not re-walk in the same pass.

Engine surface

// In engine:
func (e *Engine) Snapshot() *Snapshot
func (s *Snapshot) RulesForShard(shardID int, retentionWindow time.Duration) (replay, walk *Snapshot)
func (s *Snapshot) ReplayView(retentionWindow time.Duration) *Snapshot
func RecoveryView(s *Snapshot) *Snapshot
func ReplayContentHash(s *Snapshot) [32]byte
func PromotedHash(s *Snapshot, retentionWindow time.Duration) [32]byte
func MaxEffectiveTTL(s *Snapshot) time.Duration

RulesForShard and RecoveryView return new *Snapshot instances with cloned *CompiledAction objects and shared (by pointer) rule definitions. Fields that differ from the base:

• active — per clone, set per view.
• Mode — rewritten to ModeEventDriven on replay clones; preserved on walk and recovery clones. The rewrite is required because router.Route gates on Mode == ModeEventDriven, and today's compile preserves a persistent prior.Mode = ModeScanOnly that would otherwise lock a rule out of replay even after retention rehabilitates it.
• Action-map membership — replay contains only replay-eligible clones; walk contains only walker-bound clones; recovery contains every action.

Shared-by-pointer with the base: Rule definitions, predicate maps, RuleHash table.

router.Route integration

router.Route(ctx, snap, ev, now, lister) iterates every action with IsActive() == true in the snapshot. Two snapshots can't disagree on activation if they share the same *CompiledAction pointers, which is why RulesForShard clones.

View	Clone settings	Why
replay	active = true, Mode = ModeEventDriven	router.Route requires ModeEventDriven. Forced regardless of prior.Mode.
replayView	active preserved, Mode = ModeEventDriven	The drain's routing set: replay partition minus promoted rules, but without force-activating pre-bootstrap actions. Walk-bound kinds are shared by pointer.
walk	active = true, Mode preserved	Walker accepts any non-ModeDisabled Mode.
recovery	active = true, Mode preserved	Walker iterates all action clones.

Subscription model

One filer SubscribeMetadata stream per dailyrun.Run() call, covering every shard in cfg.Shards. The Reader carries a ShardPredicate func(int) bool that accepts the shard set; a fan-out goroutine routes events to per-shard channels by ev.ShardID. This replaces the earlier model (16 separate per-shard subscriptions per pass).

globalStartTsNs = min(per-shard cursor, runNow - maxTTL). Pre-loaded once at pass start so the subscription's StartTsNs covers every shard's needed range; per-shard drains then filter ev.TsNs <= shard.startTsNs locally.

The subscription is bounded: UntilNs = runNow, so the filer ends the stream once it has delivered everything up to the pass boundary and the pass ends on its own. Fan-out also cancels the reader on the first ev.TsNs > runNow as a backstop (meta-log events arrive in TsNs order; everything after is past the boundary) — that used to be the only way a pass ended, which wedged the job for as long as the cluster stayed quiet. Per-shard channels are buffered to 256 events — large enough to absorb bursts without back-pressuring the fan-out.

Action kinds and dispatch paths

Table uses the S3-spec rule names (what operators type in lifecycle XML). The corresponding engine constants in weed/s3api/s3lifecycle/action_kind.go are ActionKindExpirationDays, ActionKindNoncurrentDays, ActionKindAbortMPU, ActionKindExpirationDate, ActionKindExpiredDeleteMarker, ActionKindNewerNoncurrent — same one-to-one mapping, shorter spelling.

ActionKind	Trigger	Due time	Path	Early-stop in replay?
ExpirationDays	Latest-version PUT	ev.TsNs + r.ExpirationDays	Replay	Yes
NoncurrentDays	Demotion (next PUT for same key)	entry.NoncurrentSince + r.NoncurrentDays	Replay	Yes
AbortIncompleteMultipartUpload	MPU init	mpu_init.TsNs + r.AbortMPUDaysAfterInitiation	Replay	Yes
ExpirationDate	Latest-version PUT, fires on now >= r.ExpirationDate	r.ExpirationDate (constant)	Walker	n/a
ExpiredObjectDeleteMarker	Delete marker with NumVersions == 1	"now if orphaned, else never"	Walker	n/a
NewerNoncurrentVersions	Version becomes noncurrent AND total noncurrents > r.NewerNoncurrentVersions	"now if over the cap, else never"	Walker	n/a

ExpiredObjectDeleteMarker and NewerNoncurrentVersions are walker-only because their due-time depends on current sibling state, not on any event's TsNs. The done early-stop in replay can't engage — there's nothing event-time-monotonic to early-stop on.

Cursor

Persisted per shard at /etc/s3/lifecycle/daily-cursors/shard-NN.json. Shape (weed/s3api/s3lifecycle/dailyrun/cursor.go):

type Cursor struct {
    TsNs         int64       // last meta-log event whose matches all dispatched
    RuleSetHash  [32]byte    // ReplayContentHash of the rule set that wrote this
    PromotedHash [32]byte    // PromotedHash with retentionWindow at write time
    LastWalkedNs int64       // wall-clock of the last successful walker fire
}

LastWalkedNs is JSON-omitempty, so cursor files written before that field existed decode cleanly as zero (treated as "never walked steady-state" → next pass seeds the anchor).

Cursor save uses a fresh context.Background() with a 5s timeout because a caller-imposed wall-clock cap on the pass (the shell driver's -runtime) cancels the drain's context. Saving with the canceled context would silently drop the cursor and the next pass would re-replay from the same floor.

In steady state the start position honors the cursor verbatim — the floor runNow - maxTTL is applied only on cold start (!found). The drain freezes the cursor at the last pre-skip event so pending matches with DueTime == TsNs + maxTTL stay in scope across passes; bumping forward in steady state would orphan exactly those events.

Cursor hashes

The cursor stores two hashes that together detect every situation invalidating "everything before persisted.TsNs has been processed under the same rules":

RuleSetHash = engine.ReplayContentHash(snap) — content over the rule definitions (action kind, predicate, TTL value) of replay-eligible action kinds. Partition-independent.

PromotedHash = engine.PromotedHash(snap, retentionWindow) — hash of replay-eligible rules currently classified as walk due to scan_only promotion (their TTL exceeds retentionWindow).

Recovery triggers, complete list:

Trigger	Detection	Why
Cold start	No persisted cursor	First run for this shard
Replay-rule edit	RuleSetHash mismatch	Replay-eligible rule content changed
Partition flip	PromotedHash mismatch	A replay-eligible rule moved between replay and walk

Retention loss as a recovery trigger is a known gap: in stock SeaweedFS the filer's meta-log is effectively never GC'd (no on-disk retention policy for /topics/.system/log), so cfg.RetentionWindow defaults to maxTTL and PromotedHash stays empty. When operators add explicit meta-log retention, the cursor-vs-earliest-available check becomes load-bearing again.

Walker throttle

cfg.WalkerInterval decouples the walker's cadence from Run() invocation cadence. The s3tests CI workflow invokes the worker every 2s; a single daily admin schedule invokes it once per day. The walker fire rate should be set by walk cost, not by the invocation interval — so:

• Steady-state and empty-replay walker fires gate on walkerDue(persisted.LastWalkedNs, runNow, WalkerInterval).
• 0 keeps the prior "fire every pass" behavior (back-compat for tests).
• Production: pick the walk cost budget per shard per cluster. Small cluster: 1h. Large cluster: 6h+.
• Recovery walker fires (cold-start, hash mismatch) are unconditional — these are bounded events that must run once.

Walker fires update Cursor.LastWalkedNs so the next pass's throttle has a fresh anchor. The recovery walker also updates it, so a steady-state branch in the same pass doesn't double-walk over the same superset.

Delete failure handling

Cursor advance is gated on success. The cursor only moves past events whose matches all returned DONE, NOOP_RESOLVED, or SKIPPED_OBJECT_LOCK. Any other outcome (RETRY_LATER, BLOCKED, transport error after in-run retries) halts the run and persists the cursor at the last fully-processed event.

• Head-of-line blocking is intentional. A transient filer error stalls today's pass; tomorrow's run resumes at the same cursor. Loud (operator sees stuck cursor in metrics) and idempotent (identity-CAS makes redundant deletes no-ops).
• In-run retry with backoff for transport errors only — default 3 attempts, exponential backoff capped at 5s. Server-side outcomes are not retried in-run.
• No retry queue. Removing the per-key freeze state was the whole point; adding it back would re-introduce the state machine the redesign replaces.

Rate limiting

Cluster-wide deletes-per-second cap, set in admin config. The admin allocator:

1. Counts workers capable of s3_lifecycle from the registry.
2. Divides cluster_deletes_per_second by the count.
3. Writes the per-worker share into ExecuteJobRequest.ClusterContext.Metadata["s3_lifecycle.deletes_per_second"].

The worker reads the share and constructs one golang.org/x/time/rate.Limiter shared across all shard goroutines. dispatchWithRetry calls limiter.Wait(ctx) before each LifecycleDelete RPC.

Observability

Per-shard Prometheus gauges (weed/stats/metrics.go):

Metric	What it tells you
s3_lifecycle_cursor_min_ts_ns{shard}	now - this is the per-shard replay lag
s3_lifecycle_daily_run_last_walked_ns{shard}	now - this is walker freshness; stuck = throttle misconfigured or walker failing
s3_lifecycle_daily_run_shard_duration_seconds{shard}	Wall-clock per shard pass
s3_lifecycle_daily_run_events_scanned_total{shard}	Counter of meta-log events drainShardEvents processed
s3_lifecycle_dispatch_limiter_wait_seconds	Per-dispatch wait time on the cluster rate limiter
s3_lifecycle_dispatch_total{bucket,kind,outcome}	Per-bucket dispatch counter

Heartbeat log line, emitted once per Run():

daily_run: status=ok shards=16 errors=0 duration=7s cursor_lag_max=2h walked_max_age=3m

Tokens status, shards, errors, duration are stable for grep. cursor_lag_max=cold and walked_max_age=cold distinguish "not started yet" from "0s caught up".

Data model

noncurrent_since on version entries

A non-current version's TTL clock starts when the next version was written, not at its own mtime. The demoting PUT writes NoncurrentSinceNs on the demoted entry, set to the TsNs of the demoting meta-log event. Using the meta-log TsNs keeps noncurrent_since strictly monotonic in meta-log order across all replicas, immune to wall-clock skew.

The lifecycle evaluator uses ev.TsNs for current-version rules and entry.NoncurrentSinceNs for noncurrent rules — both monotonic in iteration order. Legacy entries with NoncurrentSinceNs == 0 fall back to entry mtime.

expected_mtime passed to LifecycleDelete (for identity CAS) is always the entry's own mtime. CAS identity and TTL clock are separate concerns.

Components

Path	Role
engine/	Rule compilation, snapshot, partition views
evaluate.go, due_at.go, rule_hash.go, tags.go	Engine-side rule evaluation
reader/	Meta-log subscribe; one subscription per dailyrun.Run pass
router/router.go	Per-event rule evaluation
bootstrap/walker.go	Bucket walker; shard-set entry filter
dispatcher/filer_persister.go	Filer-backed cursor I/O
dailyrun/run.go	Main pass orchestrator: subscription, fan-out, per-shard runShard
dailyrun/walk_plan.go	Groups shards into shared recovery/steady walks (one listing per group)
dailyrun/cursor.go	Cursor type + filer JSON serializer
dailyrun/walker_dispatcher.go	Adapter from walker to LifecycleDelete RPC

Configuration

Admin config (weed/worker/tasks/s3_lifecycle/):

Key	Type	Default	What
cluster_deletes_per_second	int64	0 (unlimited)	Cluster-wide ceiling on lifecycle delete RPCs/s. Allocated per worker.
cluster_deletes_burst	int64	0 (= 2× rate)	Token-bucket burst across cluster.
meta_log_retention_days	int64	0 (unbounded)	How far back the filer's meta-log can reach. Rules with TTL > retention promote to walker.
walker_interval_minutes	int64	0 (fire every pass)	Minimum time between steady-state walker fires per shard. Set positive when worker runs at tighter cadence than the desired walk frequency.

Worker config:

Key	Default	What
max_runtime_minutes	60	Wall-clock cap per dailyrun.Run call

Failure & recovery

• Worker crashes mid-run. Cursor advances only past successfully-deleted events. On restart, the next pass resumes at the same cursor and re-attempts. Identity-CAS makes redundant deletes no-ops.
• Transient delete failure. Pass halts at the failing event, cursor stays. Tomorrow's pass retries from the same point. Stuck cursor is visible in s3_lifecycle_cursor_min_ts_ns; operators see head-of-line blocking and address the root cause.
• Identity drift (object overwritten between event and delete). Handled by LifecycleDelete RPC's identity-CAS, which returns NOOP_RESOLVED for stale events. The algorithm dispatches optimistically and lets the server filter.
• Cold start, rule edit, partition flip. All route into the recovery branch. The walker over engine.RecoveryView(snap) catches already-due objects across the full rule set, then the cursor rewinds (rule edit) or stays at the cold-start floor.

Future work

Tracked as optimizations rather than blockers:

1. Long-lived subscription across passes. Today the subscription is rebuilt per Run(). Keeping it alive across passes would eliminate the 7s ctx-timeout per pass and the start/teardown overhead. Requires per-shard pending heap (events whose DueTime > runNow would be parked in-memory instead of replayed) and a hot-swappable snapshot for mid-pass config changes. Multi-day refactor; current model works.
2. Bucket-coordinated walker. Done: planWalks shares one listing across the recovery group and one across the steady group (issue #11791). Remaining headroom: directories could be pruned to rule prefixes, and per-shard views could partition work rather than fanning every entry to its shard's dispatcher.
3. Per-bucket dispatch lag metric. Currently only per-shard lag is exposed. Per-bucket would require a per-bucket cursor or a derived metric from s3_lifecycle_dispatch_total{bucket,kind,outcome}. Punted on cardinality concerns; revisit when an operator asks for it.
4. Meta-log retention plumbing. If the filer adds GC for /topics/.system/log, the PromotedHash partition flip needs to consume the filer's actual retention horizon (currently dormant because retention is effectively infinite).