# S3 Lifecycle Redesign — Event-Driven Expiration

## Why

`PutBucketLifecycleConfiguration` today walks every entry under the bucket inside the request handler (`weed/s3api/s3api_bucket_handlers.go:900-1068` → `weed/s3api/filer_util.go:191`), so it times out on buckets with tens of thousands of objects. Worse, only one rule shape actually expires anything: simple `Expiration.Days` + prefix-only filter on a non-versioned bucket. Tag/size filters, `Expiration.Date`, versioned buckets, `NoncurrentVersionExpiration`, `AbortIncompleteMultipartUpload`, and `ExpiredObjectDeleteMarker` are silent no-ops. There is no scan-time evaluator anywhere — `weed/s3api/s3lifecycle/rule.go` defines `Rule`, `ObjectInfo`, `EvalResult`, `Action` but no `Evaluate` and no callers.

Three independent mechanisms exist today, only one of which is real enforcement:

1. Read-triggered TTL check on the filer (`weed/filer/filer.go:396-408`, `:435-449`). Lazy: cold objects never expire.
2. Volume-level TTL: `filer.conf` + TTL volumes drop chunks wholesale, but only for writes that arrive after the rule exists.
3. The synchronous "back-stamp" inside the PUT handler that walks existing entries to bridge #1 and #2. This is what's slow.

## Design

### Core idea

Drive lifecycle expiration off the persistent metadata change log at `/topics/.system/log/<YYYY-MM-DD>/<HH-MM>.<filerId>` (`weed/filer/topics.go:5`, `weed/filer/filer_notify.go:166`). The same log powers `filer.sync`, `filer.backup`, and `filer.meta.tail`. Each event carries `OldEntry` and `NewEntry` with full attributes and `Extended`. The day/hour-minute layout makes time-windowed reads cheap. Cost scales with **change rate**, not bucket size.

### Repo facts the design depends on

Verified in the codebase:

- Meta events are persisted under `SystemLogDir = TopicsDir + "/.system/log"` and accessed via `Filer.ReadPersistedLogBuffer` and `Filer.CollectLogFileRefs` (`weed/filer/filer_notify_read.go`).
- Each event's `EventNotification.OldEntry` and `NewEntry` are `*filer_pb.Entry` with full `Attributes` and `Extended` (`weed/pb/filer_pb/filer.pb.go:622-672`).
- Versioning constants (`weed/s3api/s3_constants/`):
  - `VersionsFolder = ".versions"` — appended as a suffix, not a child. A versioned object at key `foo/bar.txt` has its versions under directory `foo/bar.txt.versions/` (`weed/s3api/filer_multipart.go:578`).
  - `ExtDeleteMarkerKey = "Seaweed-X-Amz-Delete-Marker"` — a delete marker is a version file in the `.versions` directory whose `Extended[ExtDeleteMarkerKey] == "true"`. There is **no** `IsDeleteMarker` field on `filer_pb.Entry`.
  - `ExtLatestVersionIdKey`, `ExtLatestVersionFileNameKey`, `ExtLatestVersionMtimeKey`, `ExtLatestVersionIsDeleteMarker` — the *current-version pointer* lives on the `.versions` directory entry's `Extended`.
- Multipart uploads live under `<bucket>/.uploads/<uploadId>/` (`s3a.genUploadsFolder` in `weed/s3api/s3api_object_handlers_multipart.go:475`).
- Existing internal delete helpers we will reuse: `s3a.createDeleteMarker(bucket, object)` (`weed/s3api/s3api_object_versioning.go:162`), `s3a.deleteSpecificObjectVersion(bucket, object, versionId)` (`:968`), `s3a.deleteUnversionedObjectWithClient(client, bucket, object)` (`weed/s3api/s3api_object_handlers_delete.go:169`).

### Phase 0 — Verified assumptions

Each open assumption resolved against the codebase at `sulfuric-podium`:

**1. Persisted log payload includes `Extended`.** `Filer.logMetaEvent` (`weed/filer/filer_notify.go:110-119`) marshals `*filer_pb.SubscribeMetadataResponse{EventNotification:{OldEntry, NewEntry, …}}` into `LogEntry.data` via `entry.ToProtoEntry()`, and `Entry.extended` (`weed/pb/filer.proto:135`) is a `map<string,bytes>` serialized as part of the proto. `ReadPersistedLogBuffer` (`weed/filer/filer_notify.go:203`) hands back the same `*filer_pb.LogEntry` whose `data` deserializes back to the full event with `Extended` intact. Lifecycle's predicate evaluation against tags/`Extended` keys is therefore directly available from the meta log without any side fetch.

**2. Meta-log retention is unbounded by default.** Persisted log files at `<SystemLogDir>/YYYY-MM-DD/HH-MM.<filerId>` are written via `Filer.appendToFile` (`weed/filer/filer_notify_append.go:14-49`). The created `Entry` has no `TtlSec` field set, only `Crtime`/`Mtime`/`Mode`/`Uid`/`Gid`. There is no built-in cleaner: nothing under `weed/filer/` or `weed/server/` deletes `topics/.system/log/` files. The volume-level TTL only kicks in if an operator configures a `filer.conf` rule whose `LocationPrefix` matches `/topics/.system/log/` and whose `Ttl` is set, in which case `Filer.appendToFile` → `assignAndUpload` (`weed/filer/filer_notify_append.go:51-94`) routes through `MatchStorageRule` and the assigned volume carries the rule's `Ttl`. Without that explicit configuration, retention is operator-bounded only (manual deletion of old day directories).

This means:
- For default deployments, `metaLogRetention = ∞` and the retention mode gate (`metaLogRetention < eventLogHorizon(rule) + bootstrapLookbackMin`) never trips — every reader-driven kind runs `event_driven`.
- For deployments that opt into volume-TTL pruning of the meta log, operators must configure `metaLogRetention` to match the configured TTL; the gate then promotes long-horizon rules to `scan_only` automatically.
- Phase 2 will read `metaLogRetention` from a cluster-config knob (default: a sentinel meaning "unbounded"). When the operator sets a TTL on `topics/.system/log/`, they must also set `metaLogRetention` to the same value; the design doc surfaces this as a required-coupled configuration in Phase 8 docs.

**3. `.versions/` filename scheme.** Inside `<object>.versions/`, version files are named `v_<32-hex>` where the 32-hex string is `<16-hex-timestamp><16-hex-random>` (`weed/s3api/s3api_version_id.go:30-55`, `:148-150`). Two formats coexist, distinguished by the timestamp portion's value relative to threshold `0x4000000000000000` (`weed/s3api/s3api_version_id.go:24`):
- **New (inverted) format**: timestamp portion = `MaxInt64 - now_ns`, value > threshold (~0x68… in 2025). Newer versions sort **earlier** lexicographically. Used for new `.versions/` directories.
- **Old (raw) format**: timestamp portion = `now_ns`, value < threshold. Older versions sort earlier lexicographically. Detected by reading `Extended[ExtLatestVersionIdKey]` on the `.versions/` directory entry and applying `isNewFormatVersionId` (`weed/s3api/s3api_version_id.go:58-70`).

For the design's "successor" version computation (the version that replaced this one to make it non-current):
- New format: the lex-immediate-predecessor in `.versions/` (sorts earlier = newer).
- Old format: the lex-immediate-successor (sorts later = newer).
- Universal fallback: `getVersionTimestamp(versionId)` (`weed/s3api/s3api_version_id.go:74-92`) returns the actual ns timestamp regardless of format; `compareVersionIds(a, b)` (`:97-140`) gives a format-agnostic newest-first comparator. Phase 5 will use these helpers when discovering successor non-current time.

The `.versions/` directory entry's `Extended` carries the current-version pointer (`ExtLatestVersionIdKey`, `ExtLatestVersionFileNameKey`, `ExtLatestVersionMtimeKey`, `ExtLatestVersionIsDeleteMarker`) — already documented above as repo facts.

### Storage

**Three storage areas, separated by ownership and write-concurrency boundaries.**

The bucket directory's xattrs continue to hold the policy itself:

- `Extended["s3-bucket-lifecycle-configuration-xml"]` — original XML (existing key, see `weed/s3api/s3api_bucket_lifecycle_config.go:11`).

**Multi-action rules.** A single AWS lifecycle XML `<Rule>` may declare multiple actions in parallel — for example `Expiration.Days=90` together with `AbortIncompleteMultipartUpload.DaysAfterInitiation=7` and `NoncurrentVersionExpiration.NoncurrentDays=30`. Each action has its **own** delay/horizon/mode and must drive its own pending stream and cursor independently. Modeling a rule as one compiled entry with one `kind` and one delay collapses these — e.g. picking the smallest delay (7d MPU) means the 90d expiration cursor "advances past" objects that aren't yet due, and the 90d action never re-fires for them.

The engine therefore **expands every XML rule into N compiled actions** at compile time, where N is the count of action sub-elements actually populated. Each compiled action has its own state, its own pending file, its own delay group, and its own mode. The shared filter (Prefix, Tags, Sizes, Status) is copied to each action — actions of the same rule are evaluated in parallel against the same filter.

**Per-action state** lives outside the bucket under `/etc/s3/lifecycle/<bucket>/<rule_hash_hex>/<action_kind>/`. One subdirectory per action; one writer per directory (the per-action cluster lock). The intermediate `<rule_hash_hex>` keeps a rule's actions grouped so operators can list "rule R's status" by enumerating its action subdirectories:

- `state` — protobuf, one record:
  - `rule_hash` (8 bytes; matches the parent rule directory)
  - `action_kind`: `EXPIRATION_DAYS` | `EXPIRATION_DATE` | `NONCURRENT_DAYS` | `NEWER_NONCURRENT` | `ABORT_MPU` | `EXPIRED_DELETE_MARKER` (matches the leaf directory name)
  - `rule_id` (display only; can be empty/duplicate; identical across sibling actions of the same XML rule)
  - `mode`: `event_driven` | `scan_at_date` | `scan_only` | `disabled` (scheduling intent — per action, not per rule)
  - `degraded_reason`: `NONE` | `LAG_HIGH` | `PENDING_FULL` | `DELETE_FAILURES` | `OPERATOR_PAUSED` | `RETENTION_BELOW_HORIZON` | `LOST_LOG` (orthogonal health signal; a rule's individual action can be `event_driven AND LAG_HIGH`, with auto-promotion to `scan_only` deferred until the next threshold; `RETENTION_BELOW_HORIZON` and `LOST_LOG` are set together with `mode = scan_only` by the retention gate and lost-log GC respectively)
  - `degraded_since_ns`: when the current `degraded_reason` was set; cleared together with reason
  - `bootstrap_complete`
  - `bootstrap_started_at_ns`
  - `bootstrap_completed_at_ns`
  - `last_safety_scan_ts_ns`
  - `next_safety_scan_ts_ns`
  - counters: `evaluated_total`, `expired_total`, `metadata_only_total`, `error_total`
- `pending` — protobuf, append-with-tombstones: `repeated PendingItem { path, version_id, due_at_ns, expected_identity }`. **One use only**: late predicate changes that create not-yet-due eligibility for *this* action (tag added at age 30d on a 60d rule's `EXPIRATION_DAYS` action). Dedupe key `(path, version_id)`. Sibling actions on the same rule have their own pending files.

No per-rule `blocked` file (compliance ledger). Object-lock and retain-until are the operator's concern — see "Object lock and compliance" below. (Cluster-level `_reader/blockers` is unrelated — it tracks paused cursor positions for un-processable events, not retained objects.)

Per-rule state does **not** include reader watermarks. Watermarks belong to the shared reader, not to individual rules — see below.

**Cluster reader state** lives at `/etc/s3/lifecycle/_reader/`. **A single reader task** (`s3.lifecycle.read`, locked by one cluster-wide lock) is subscribed to **one** filer endpoint at a time. That endpoint is just a routing convenience: `Filer.CollectLogFileRefs(start_position, stop_ts)` returns chunk refs from per-filer log files in directory order, and the worker heap-merges them client-side. The single subscription is for connection management; cross-filer event ordering happens in the worker.

Files:

- `reader_state` — protobuf, single record (no reader groups):
  - `primary_filer_endpoint` — current subscription target. Workers fail over to another filer if this one is unreachable.
  - `last_processed_original: map<delay_group_seconds, map<filer_id, MessagePosition>>` — per delay group, per per-filer log shard.
  - `last_processed_predicate: map<filer_id, MessagePosition>` — per per-filer shard.
  - `tail_drained_streams: set<TailDrainedStreamKey>` — stream-specific markers for departed shards already observed at their retained tail. Keys are `(ORIGINAL, delay_seconds, filer_id)` or `(PREDICATE, filer_id)`, never just `filer_id`; one delay group being drained does not prove another delay group is safe. The marker is cleared when that stream is lazily seeded again after the shard reappears.
  - `last_engine_compiled_at_ns: int64`.
- `blockers` — durable per-shard blocker records. When a deterministic per-event failure (or a sustained transient one) is observed, the worker pauses that shard at its current `MessagePosition` and records a `BlockerRecord` here; the cursor does **not** advance. Operators inspect via `weed shell s3.lifecycle.blockers list` and resolve manually (`retry`, `resume`, or `quarantine`). See "Blocked cursor handling" below. There is no automatic dead-letter — for lifecycle, silently routing a failed event aside would mean losing a delete decision; the design pauses instead.
- `retry_budget` — durable per-stream retry counter file. Tracks `consecutive_retries` for `(stream_kind, key)` tuples that produce repeated `RETRY_LATER`. Compacts on success; promotes to a `BlockerRecord` once the configured budget trips. See "Sustained-RETRY_LATER promotion" below.

**Why per-filer cursors are unavoidable.** The persisted log is partitioned by filer (`<HH-MM>.<filerId>` files). The client-side heap merge produces a single ordered stream from N per-filer streams, but resuming after a partial-batch failure requires knowing each per-filer stream's position — `LogEntry.Offset` is per-buffer (per filerId), not globally unique. A single `(ts, offset)` cursor across the merged stream would conflate offsets from different filers. Therefore:

- The cursor type per delay group is `map<filer_id, MessagePosition>`.
- `MessagePosition = {ts_ns, offset}` (the existing log-buffer type, `weed/util/log_buffer/log_read.go:38`) is the per-shard tie-breaker.
- The worker maintains the heap merge over per-shard `MessagePosition`s; on resume, each filer's stream restarts at its own cursor position.
- Watermark exposed to operators / metrics is the low-water-mark: `min over filer_ids of last_processed.ts_ns` per delay group. Lag = `now - watermark`.

This is more state than the previous "single subscription, single cursor" claim implied, but it's what correctness requires across a client-side multi-filer merge. The single subscription still gives connection-level simplicity (one filer endpoint, easy failover); the per-filer cursor map is internal bookkeeping.

**Cursor-skip semantics.** `last_processed[filer_id]` points at the *last resolved event* for that shard. Resume must skip events with position `<= cursor[filer_id]` (using the `(ts, offset)` order per shard) — strict `<=`, not `<`, otherwise the last processed event is replayed on every resume. Equivalently: deliver events with `(event.ts, event.offset) > cursor[filer_id]`. Phase 3 introduces a new API variant that takes per-shard `MessagePosition`s and applies this comparator — see "API change" below. The existing `pb.ReadLogFileRefs` filter `logEntry.TsNs <= startTsNs` (`weed/pb/filer_pb_direct_read.go:333`) is the wrong shape because it can't disambiguate equal-ts events.

> **Implementation note:** the API addition described below was not built. The shipped reader uses `SubscribeMetadata` with a single global cursor (the filer's server-side `MetaAggregator` already does the merge). The pseudocode is preserved as design context for a future per-shard architecture.

**~~Required API addition~~ (Phase 3 dependency, not built)**: `pb.ReadLogFileRefsWithPosition` extending the current `ReadLogFileRefs`:

- Take a per-shard `start_positions: map<filer_id, MessagePosition>` instead of a single `startTsNs`.
- Per-event callback signature `eachLogEntry(event, filer_id, position) → CallbackResult`.
- Skip predicate uses `(ts, offset)` ordering per shard, not `TsNs <= startTsNs`.
- Return per-shard `last_positions: map<filer_id, MessagePosition>`.

`CallbackResult` (explicit enum):

```
enum CallbackResult {
    Continue                      // proceed to next event
    HaltAll                       // stop merge entirely; return last_positions for ALL shards
    PauseShard(filer_id)          // stop delivering events from THIS shard for the rest of this call
                                  // (other shards continue); subsequent merge ticks within this call
                                  // never invoke the callback again for filer_id
}
```

`last_positions` semantics on return:

- For shards that returned `Continue` until exhaustion: `last_positions[filer_id]` = the last delivered event's position.
- For shards paused via `PauseShard(filer_id)`: `last_positions[filer_id]` = the position of the event whose callback returned `PauseShard` (the *paused-on* event, not the next un-delivered one). The lifecycle reader does **not** advance the cursor in this case — the BlockerRecord already records the failing position, and the cursor stays at the last *resolved* position recorded by the previous successful callback for that shard.
- For shards untouched (no events delivered before the call ended): `last_positions[filer_id]` = the value passed in via `start_positions[filer_id]` (unchanged).

`HaltAll` exits the function immediately; `last_positions` reflects the state up to the point of halt, with the halting event's shard's `last_positions` entry equal to the position that returned `HaltAll`.

No `afterChunk` hook. Under client-side heap merge, a chunk's events are interleaved with events from other shards, so "chunk read complete" does not mean "all events from this chunk delivered and resolved." A hook firing on chunk-read-completion could checkpoint past unresolved events. Checkpointing happens only inside the per-event callback (see "Periodic checkpointing" in the reader pseudocode below).

The existing `pb.ReadLogFileRefs` stays unchanged for other consumers; the new variant is additive.

Why one reader, one subscription:

- **Watermark correctness is well-defined.** Each per-filer cursor is its own monotonic stream; the heap merge produces a single ordered delivery; cursors persist per shard so resume after partial failure is correct on each shard. Watermark surfaced externally is the low-water-mark across the per-filer cursors per delay group.
- **Filer load is metadata-only.** `CollectLogFileRefs` returns chunk file IDs and their time ranges — no event bodies are deserialized at the filer. Worker reads bodies from volume servers, which parallelizes naturally across the chunks of the merged stream.
- **No partitioning, no epochs, no migration.** Adding parallelism is not a primary scaling story; if a single worker can't keep up after raising in-worker chunk-read concurrency and `lifecycle.delete.concurrency`, the next-stage answer is segment-level intra-stream parallelism (see "Future scaling" below). Explicitly **non-v1**.

**Flush-safety lag.** The filer's persisted log files appear on disk after `LogFlushInterval` (current constant: `time.Minute` — `weed/filer/filer.go:33`). The worker reads only events at `event.ts <= now - flushSafetyLag` (default `2 × LogFlushInterval`, so **≈2 minutes** with current defaults; reduces if `LogFlushInterval` is shortened) so a late-flushing filer cannot produce events older than the cursor after the cursor has advanced. All cutoffs are clamped: `effectiveCutoff = min(kindCutoff, now - flushSafetyLag)`. Phase 0 confirms the active value of `LogFlushInterval` and pins `flushSafetyLag` to it; the design treats `flushSafetyLag` as derived, not as a hardcoded number.

**Primary filer failover.** Workers reach the primary filer via a standard filer client; on connection failure, fall back to the next healthy filer endpoint (discovered via the master). The cursor is portable — every filer reads the same persisted logs from the shared namespace, so the worker can resume against a different primary without state loss.

Per-bucket bootstrap progress (used by bucket-level bootstrap tasks):

- `/etc/s3/lifecycle/<bucket>/_bootstrap` — protobuf: `last_scanned_path`, `bootstrap_started_at_ns`, the engine snapshot ID it was launched against. One writer (the bucket's bootstrap task).

Why `/etc/...` rather than inside the bucket: keeps system state out of bucket listings without needing list-time filters; matches existing convention (`/etc/seaweedfs/`, `/etc/...`).

`rule_hash` is `sha256(canonicalize(rule))[:8]` over a length-prefixed canonical form (sorted tag map, prefix verbatim, every action's parameters — days/date/count/flags, filter fields). Length prefixing prevents delimiter forgery between adjacent fields. Stable across reorder and resilient to empty or duplicate `Rule.ID`. Prefix `"logs"` and `"logs/"` hash differently because they match different objects under literal `strings.HasPrefix` semantics.

**Policy CAS is per-rule via the `rule_hash` directory layout.** Pending items live under `<rule_hash>/pending`; the drain pass and the `LifecycleDelete` server check whether `rule_hash` is still in the current policy's hash set. Edits to other rules don't affect this rule's pending. See "Policy-version CAS" section below for full mechanics.

### Tick (default 24h, configurable, jittered, per bucket)

For each bucket with `lifecycle.xml`:

1. Parse XML; compute `rule_hash` per rule. Expand each rule into its compiled actions (one per populated action element); compute `(rule_hash, action_kind)` keys.
2. Reconcile per-action `state` files under `/etc/s3/lifecycle/<bucket>/<rule_hash_hex>/<action_kind>/`: drop directories for `(rule_hash, action_kind)` pairs no longer present (after a grace period to absorb policy edit-and-revert); create directories for new pairs (`bootstrap_complete=false`).
3. Per **compiled action** (not per rule), decide mode in this order:
   - `action_kind == EXPIRATION_DATE` → `mode = SCAN_AT_DATE`. Detector schedules a single bucket-level bootstrap at the action's `rule.date`. No per-object pending. No reader involvement.
   - Else compute `eventLogHorizon(rule, action_kind)` — the age of an event the reader needs to be able to observe for *this specific action*:
     - `EXPIRATION_DAYS`     → `rule.Days`              (age-origin events at `now - days`)
     - `NONCURRENT_DAYS`     → `rule.NoncurrentDays`    (version-flip events at `now - days`)
     - `ABORT_MPU`           → `rule.DaysAfterInitiation` (MPU init events at `now - days`)
     - `NEWER_NONCURRENT`    → `smallDelay`             (count-based; reader observes version-flip events near `now - smallDelay`)
     - `EXPIRED_DELETE_MARKER` → `smallDelay`           (immediate; reader observes the marker creation event near `now - smallDelay`)
     If `metaLogRetention < eventLogHorizon(rule, action_kind) + bootstrapLookbackMin` → this action's `mode = scan_only` (with `degraded_reason = RETENTION_BELOW_HORIZON`). Event-driven would silently miss events that aged out of the log. Sibling actions on the same XML rule are evaluated independently — a 90d `EXPIRATION_DAYS` action may degrade to `scan_only` while its 7d `ABORT_MPU` sibling stays `event_driven`.
   - Else if `bootstrap_complete == false` → run bootstrap (the same walker; on completion this action transitions to its target mode).
   - Otherwise → `event_driven`.
4. Run `event_driven` actions: trigger pass, then drain `pending`.
5. Run `scan_only` actions: re-run bootstrap on the per-kind safety-scan cadence (see "Safety-scan cadence per rule kind" table). Date and count kinds don't have a TTL-style `MinTriggerAge`; their cadences are defined explicitly in that table. Bootstrap is idempotent; this is the same code path.

`metaLogRetention` is read from cluster config (Phase 0 confirms the knob). If unconfigured, default to a conservative value (e.g. 30d) and require operators to extend it explicitly for longer-TTL rules.

### Rule kinds and `dueAt`

Each rule kind defines `dueAt(rule, live)` — the earliest wall-clock time the entry becomes eligible. The trigger pass uses a kind-specific cutoff to bound work; eligibility is always checked against `dueAt`.

| Rule kind | Mode | `dueAt(rule, live)` | Original-write trigger cutoff | Predicate-change trigger cutoff |
|---|---|---|---|---|
| `EXPIRATION_DAYS` | `event_driven` | `live.mtime + days` | `now - days` | `now - smallDelay` |
| `EXPIRATION_DATE` | `scan_at_date` | n/a (single date-triggered sweep) | n/a | n/a |
| `NONCURRENT_DAYS` | `event_driven` | `successorMtime + days` | `now - days` (version-flip events) | `now - smallDelay` |
| `NEWER_NONCURRENT` | `event_driven` | `now` (count-based, immediate at flip) | `now - smallDelay` | `now - smallDelay` |
| `ABORT_MPU` | `event_driven` | `mpuInitMtime + days` | `now - days` (MPU init events) | n/a |
| `EXPIRED_DELETE_MARKER` | `event_driven` | `marker.mtime` (immediate when sole survivor) | `now - smallDelay` | n/a |

`smallDelay` is a single global, default 1 minute, just to avoid racing in-flight writes.

**Safety-scan cadence per rule kind.** A periodic re-bootstrap catches drift the reader cannot recover. Cadence depends on rule kind because `MinTriggerAge` is undefined for some:

| Rule kind | Safety-scan cadence (default) |
|---|---|
| `EXPIRATION_DAYS` | `max(MinTriggerAge, 7d)` capped at 30d |
| `NONCURRENT_DAYS` | `max(MinTriggerAge, 7d)` capped at 30d |
| `ABORT_MPU` | `max(MinTriggerAge, 1d)` |
| `EXPIRATION_DATE` | `24h` (no `MinTriggerAge`; daily scan picks up new objects matching the prefix) |
| `NEWER_NONCURRENT` | `7d` (count-based; periodic catches missed version-flip events) |
| `EXPIRED_DELETE_MARKER` | `7d` |

Operators can override per rule via shell command. `scan_only` mode (when retention < trigger window) uses the same cadence as the safety scan but it's the *only* enforcement path.

The trigger pass uses two cutoffs **per rule** (not one):

- **Age-origin events** (the "original-write" sweep covers all of these — the term "original-write" is a slight misnomer kept for continuity): cutoff = kind's "age-origin" column. These represent any event that establishes a new "object age" for AWS-semantic purposes (`LastModified` reset). Concretely:
  - `OldEntry == nil && NewEntry != nil` — fresh PUT.
  - `OldEntry != nil && NewEntry != nil` AND any of `Mtime`, `FileSize`, or chunk fileIds differ — overwrite or content rewrite.
  - MPU init events (initialize a new upload, not yet completed).
  An overwrite resets the lifecycle clock; the engine must re-evaluate. If we excluded overwrites from this stream, a 60d rule on a 30d-old object that was just overwritten would still see the original PUT event (already past the watermark) and never re-trigger; the rewritten content would be deleted at the original PUT's age + 60d, not the rewrite's age + 60d.
- **Predicate-change-only events** (`OldEntry != nil && NewEntry != nil` AND `Mtime/FileSize/chunks unchanged` AND `Extended` differs): cutoff = `now - smallDelay`. These represent tag/metadata edits that can change rule applicability *without* resetting the lifecycle clock. We must look at them quickly so a tag added at age 30d on a 60d rule doesn't wait until age 90d. `dueAt` is computed from `live.mtime + TTL`, so the object's eligibility instant is unchanged — but it gets on the radar at the right time. Events where both content and Extended changed are classified as age-origin, not predicate-change.

Events whose `OldEntry.Extended` and `NewEntry.Extended` are byte-identical (and no other rule-relevant fields changed) are filtered out at ingestion — most internal `UpdateEntry` traffic is irrelevant.

### Bootstrap — bucket-level, inline-delete

One walk per bucket evaluates every applicable rule per object via the engine's bucket index. Currently-due objects are deleted **inline** (no per-object pending writes); not-yet-due entries are skipped (the reader picks them up via the original-write watermark sweep). The walker checkpoints `last_scanned_path` per page in `/etc/s3/lifecycle/<bucket>/_bootstrap`; on `TRANSPORT_ERROR` it stops and the next bootstrap task resumes from the cursor.

```
T_start := now()
snapshot := engine.Snapshot()

// Refuse to start if a BOOTSTRAP blocker is active for this bucket — operator
// must clear it via blockers retry|resume|quarantine first. Without this guard
// the walker would re-attempt the same failing entry every tick and the cursor
// could advance past the blocked path if the underlying issue self-heals,
// without the blocker ever being explicitly resolved.
active_blockers := load("/etc/s3/lifecycle/_reader/blockers")
if any b in active_blockers where b.stream_kind == BOOTSTRAP and b.bucket == bucket:
    return ContinuationHint{ next_run_after_ns: now + 60s, reason: "BOOTSTRAP blocker active" }

for each entry under /buckets/<bucket>/ starting from last_scanned_path:
    if entry is internal (.uploads/, .versions/ for non-versioned, ...): continue
    candidates := snapshot.MatchPath(bucket, entry.path)
    if len(candidates) == 0: continue
    info := buildObjectInfo(entry)
    for each rule in candidates:
        if rule.kind == EXPIRATION_DATE && rule.mode == SCAN_AT_DATE && now() < rule.date:
            continue   // wait for the scheduled date-triggered bootstrap
        if !ruleAppliesToEntryShape(rule, entry): continue
        if now() < computeDueAt(rule, info):
            continue   // not-yet-due: reader's original-write sweep will pick this up later
        action := s3lifecycle.EvaluateAction(rule, action_kind, info, now())
        // Use the same deleteAndResolve contract as the reader/drain paths.
        // Outcomes: DONE/NOOP_RESOLVED → walker advances; RETRY_LATER → walker
        // halts at this entry, last_scanned_path stays at the previous successful
        // entry, next bootstrap task resumes there; BLOCKED → blocker record was
        // written and the walker halts (operator unblocks).
        outcome := deleteAndResolve(DeleteContext{
            bucket:            bucket,
            object_path:       entry.path,
            version_id:        entry.version_id,
            rule_hash:         rule.hash,
            rule:              rule,
            stream_kind:       BOOTSTRAP,
            shard:             "",                  // bootstrap is not shard-bound
            delay_seconds:     0,                   // unused for BOOTSTRAP
            position:          MessagePosition{},   // unused for BOOTSTRAP
            action:            action,
            expected_identity: identityFromLive(entry),
        })
        stream_key := StreamKey{ stream_kind: BOOTSTRAP, bootstrap: BootstrapKey{
            bucket: bucket, object_path: entry.path,
            version_id: entry.version_id, rule_hash: rule.hash } }
        retry_target := RetryTarget{
            key:        stream_key,
            bucket:     bucket,
            object_path: entry.path,
            version_id: entry.version_id,
            rule_hash:  rule.hash,
        }
        switch outcome {
        case DONE, NOOP_RESOLVED:
            clearRetryBudget(stream_key)
            continue
        case RETRY_LATER:
            promoted := recordRetryLater(retry_target)
            if promoted == BLOCKED:
                return ErrBootstrapBlocked        // promotion wrote BlockerRecord; halt and surface
            return ErrBootstrapRetryLater         // halt; resume from last_scanned_path next task
        case BLOCKED:
            clearRetryBudget(stream_key)
            return ErrBootstrapBlocked            // halt; operator must clear blocker
        }
    checkpoint last_scanned_path = entry.path     // only after DONE/NOOP_RESOLVED for every rule

// Walk completed for this snapshot. Commit cursor seed/rewind FIRST, then per-action
// completion. Order matters: engine.compile(action) keys activation off
// state[ActionKey].bootstrap_complete && state[ActionKey].mode == EVENT_DRIVEN;
// if either flips before the cursor seed commits, the engine could mark an action
// active while its cursor is still uninitialized.

// Compute target modes UPFRONT, per ActionKey, so the seeder can filter on them.
// The seeder looks at target_mode to decide whether to rewind a delay group's
// cursor for this action (only EVENT_DRIVEN actions participate in reader sweeps
// and need the cursor floor). Computing modes after seeding would leave the
// seeder reading whatever default value the state field has — wrong.
target_modes := {}
for each ActionKey k in snapshot.action_keys_for_bucket(bucket):       // expanded per-action
    target_modes[k] = decideMode(snapshot.actions[k])

newly_completed := { k for k in snapshot.action_keys_for_bucket(bucket)
                       where state[k].bootstrap_complete == false }

seedReaderCursorsForNewDelayGroups(snapshot, newly_completed, target_modes, T_start)
                                                                          // (1) durable cluster write

// (2) durable per-action writes — bootstrap_complete=true AND mode are committed
// atomically per ActionKey (single proto write per action), using the SAME
// target_modes computed upfront. Activation needs both: engine.compile activates
// an action iff bootstrap_complete && mode == EVENT_DRIVEN.
for each ActionKey k in snapshot.action_keys_for_bucket(bucket):
    state[k].mode                       = target_modes[k]
    state[k].bootstrap_started_at_ns    = T_start
    state[k].bootstrap_completed_at_ns  = now()
    state[k].bootstrap_complete         = true                            // committed AFTER (1)

engine.markActive(snapshot, action_keys_for_bucket)                       // (3) in-memory hint
remove _bootstrap (last_scanned_path) for this bucket
```

`decideMode(action)` is the same predicate used by the tick-time mode decision (date kind → `SCAN_AT_DATE`; reader-driven kind with `metaLogRetention < eventLogHorizon(rule, kind) + bootstrapLookbackMin` → `SCAN_ONLY`; rule explicitly disabled by operator → `DISABLED`; otherwise `EVENT_DRIVEN`). Computing it once at bootstrap completion and persisting it durably means subsequent engine refreshes don't have to recompute or guess. Computing it **before** seeding (rather than inside the per-action loop) lets the seeder see consistent target modes for every `ActionKey` it must consider.

Note the `newly_completed` filter passed to `seedReaderCursorsForNewDelayGroups`: rewind only applies to actions transitioning `bootstrap_complete=false → true`. Routine safety-scan re-bootstraps for already-active actions do **not** rewind shared cursors. See "Cursor seeding/rewind" below for the rationale.

**State transitions, explicit.** Bootstrap is bucket-level execution; completion is per-action:
- `_bootstrap` (per bucket): only the run cursor (`last_scanned_path`, `bootstrap_started_at_ns`, `snapshot_id`). Removed on completion.
- `state[ActionKey].bootstrap_complete` (per action): set to true for every `ActionKey` in the snapshot when the bucket walk finishes. Actions added after the walk started keep `bootstrap_complete=false` and trigger a new bucket walk on the next detector tick.
- `engine.markActive(ActionKey)` (in-memory): the action becomes a candidate in reader sweeps only after both (1) and (2) durable writes have committed for that key.

Bootstrap **never writes to per-action pending** in the normal path. Pending exists only for late-predicate-change exceptions, which are observed by the reader, not by bootstrap.

**Policy change during bootstrap.** Bootstrap binds to a `snapshot_id`. If lifecycle XML changes mid-walk:
- The current walk finishes (it's evaluating against the old snapshot, which is consistent with what the reader's cursor floors expect).
- The engine compiles a new snapshot. New `ActionKey`s start `pending_bootstrap`; unchanged `ActionKey`s carry their `bootstrap_complete=true`. (An XML edit that adds a new action sub-element to an existing rule introduces a new `ActionKey` for that kind only; the rule's other actions stay `active`.)
- Detector emits a new bucket bootstrap task on the next tick for the new snapshot. It walks again from the beginning (idempotent — inline deletes are CAS-protected).
- Removed `ActionKey`s' directories enter grace cleanup. Removing an action sub-element from an XML rule removes only that `ActionKey`; siblings under the same `rule_hash` survive.

**Cursor seeding/rewind on bootstrap completion.** Bootstrap must touch the shared cursors for the **actions** that just transitioned `bootstrap_complete=false → true` — not every action in the snapshot, and not on every safety-scan re-bootstrap. Scoping by `newly_completed` `ActionKey`s is critical: an action joining an existing shared delay group needs the shared cursor pulled back to its safe floor (the cursor reflects work done for sibling actions and has advanced past the new action's floor), but routine safety scans for already-active actions must not rewind that same cursor — doing so would force every safety-scan tick to replay big shared windows.

Rewind rule (Option B):

- For each newly-completing `ActionKey` whose `targetOriginalDelayGroup(action)` is non-nil (computed from `action_kind`/delay, *not* from `engine.originalDelayGroups` membership — that collection only contains actions already activated, which excludes the action we're now completing): set `reader_state.last_processed_original[D.seconds][filer_id] = min(existing_cursor, MessagePosition{T_start - max(D, bootstrapLookbackMin), BEFORE_FIRST_OFFSET})` for every shard. The `targetOriginalDelayGroup` filter covers `EXPIRATION_DAYS`, `NONCURRENT_DAYS`, `ABORT_MPU`, `NEWER_NONCURRENT`, `EXPIRED_DELETE_MARKER`, and any future event-driven kind. `EXPIRATION_DATE` returns nil and is skipped. Comparison is the lex `(ts, offset)` order — same shape as the strict-`<=` skip predicate — not bare ts; otherwise an existing cursor at `(T, k)` with `k > BEFORE_FIRST_OFFSET` and `T == new_floor.ts_ns` would be left in place even though the new action's floor sits at `(T, BEFORE_FIRST_OFFSET)` and an event at `(T, 0)` should be delivered.
- For event-driven actions with effectively-zero delay (`NEWER_NONCURRENT`, `EXPIRED_DELETE_MARKER`): the delay group is `0` (or `smallDelay`); the cursor floor is `T_start - bootstrapLookbackMin`. Same min-comparison and rewind logic apply.
- For predicate-sensitive actions (those whose rule has tag/size filters): also `min(existing_cursor, MessagePosition{T_start - bootstrapLookbackMin, BEFORE_FIRST_OFFSET})` on `last_processed_predicate[filer_id]`.

Sibling actions already in the same delay group will replay the rewind window. That replay is idempotent — pending upserts collapse on `(path, version_id)`, CAS on identity rejects redundant deletes — but operators should know it costs them work proportional to events in the rewind window. The reseed shell command and the bootstrap completion path both surface this in the status output.

When a bucket bootstrap completes, it calls `seedReaderCursorsForNewDelayGroups(snapshot, newly_completed, target_modes, T_start, force_reseed_delays={}, force_reseed_predicate=false)`. The `newly_completed` argument is a set of `ActionKey`s, restricting rewind scope to actions transitioning `bootstrap_complete=false → true`. Routine safety-scan re-bootstraps for already-active actions pass empty `newly_completed`, empty `force_reseed_delays`, and `force_reseed_predicate=false`, so they don't rewind cursors.

The optional **`force_reseed_delays`** and **`force_reseed_predicate`** parameters together form the **operator-driven scan_only re-enable path**. When `s3.lifecycle.reseed` flips an action from `scan_only` back to `event_driven`, the action already has `bootstrap_complete=true`, so it does not enter `newly_completed`. Without explicit force arguments, the seeder would skip it and the previously-deleted/stale cursors would never get re-seeded. The reseed shell command:

1. Computes `force_reseed_delays = { D : D = targetOriginalDelayGroup(action), for action in affected_actions where target_mode == EVENT_DRIVEN }`.
2. Sets `force_reseed_predicate = any action in affected_actions where action.predicateSensitive`. The predicate cursor is **shared across the whole engine** (one `map<filer_id, MessagePosition>`), so the seeder operates on it whenever any predicate-sensitive action is being re-enabled — independent of which delay group the action belongs to. While the action was in `scan_only`, predicate-change events for matching tags/sizes were not fed to the engine; without rewind, the action would silently miss any tag/metadata change that happened during the `scan_only` window.
3. Deletes `reader_state.last_processed_original[D.seconds]` for each `D` in `force_reseed_delays`. If `force_reseed_predicate` is true and the operator wants a hard reseed of the predicate cursor, also deletes `reader_state.last_processed_predicate` (otherwise the seeder will pull it back via `min(existing, new_floor)`, which is sufficient).
4. Runs a fresh bucket bootstrap that calls the seeder with the populated force arguments. The seeder treats them as needing seeding/rewind even when `newly_completed` is empty.
5. After the seeder commits, flips affected rules' `mode` from `scan_only` back to `event_driven` durably.

Without this scoping, every safety scan would silently rewind every shared cursor and force unrelated rules to repeatedly replay big windows; without the force-delay path, scan_only re-enable would be silently broken because the deleted delay-cursor never gets re-seeded; without the force-predicate path, predicate-sensitive rules re-enabling from scan_only would silently miss tag/metadata changes that occurred during the scan_only window.

```
seedReaderCursorsForNewDelayGroups(snapshot, newly_completed, target_modes, T_start,
                                   force_reseed_delays={}, force_reseed_predicate=false):
    if newly_completed is empty AND force_reseed_delays is empty AND !force_reseed_predicate:
        return     // safety-scan re-bootstrap; no rewind, no force-seed

    acquire cluster lock /etc/s3/lifecycle/_reader/seeding.lock
    rs := load("/etc/s3/lifecycle/_reader/reader_state")

    // Derive affected delay groups from each newly-completing rule's TARGET
    // delay group (computed from rule.kind/delay), NOT from engine.originalDelayGroups
    // membership: that collection only includes rules with bootstrap_complete &&
    // mode == EVENT_DRIVEN, so a rule that is *just now completing* is not yet a
    // member when this seeding runs.
    //
    // target_modes is the precomputed map<rule_hash, RuleMode> from the caller.
    //
    // targetOriginalDelayGroup(rule) returns the delay-group key the rule would
    // join if event-driven:
    //   EXPIRATION_DAYS / NONCURRENT_DAYS / ABORT_MPU      → rule.delay
    //   NEWER_NONCURRENT / EXPIRED_DELETE_MARKER           → smallDelay (delay = 0)
    //   EXPIRATION_DATE                                    → nil (not reader-driven)
    affected_delays := {}
    affects_predicate := false

    // Path 1: rules transitioning bootstrap_complete=false → true.
    for rule in snapshot.rules where rule.hash in newly_completed:
        if target_modes[rule.hash] != EVENT_DRIVEN:         continue   // SCAN_*/DISABLED skip rewind
        if D := targetOriginalDelayGroup(rule); D != nil:
            affected_delays.add(D)
        if rule.predicateSensitive:
            affects_predicate = true

    // Path 2: operator-driven scan_only re-enable. Force-seed each delay in
    // force_reseed_delays even if no rule is in newly_completed. This is the
    // only path that re-seeds an already-deleted cursor for an already-active rule.
    for D in force_reseed_delays:
        affected_delays.add(D)

    // Predicate cursor: rewound if any newly_completed predicate-sensitive rule
    // OR if force_reseed_predicate is set. The latter handles scan_only re-enable
    // for predicate-sensitive rules: while the rule was scan_only, predicate-change
    // events weren't applied; without rewind we'd silently miss any tag/metadata
    // change from that window.
    if force_reseed_predicate:
        affects_predicate = true

    known_filer_ids    := list_filer_ids_in("/topics/.system/log/")
    floor_ts_for(D)    := T_start - max(D, bootstrapLookbackMin)
    predicate_floor_ts := T_start - bootstrapLookbackMin

    for each delay D in affected_delays:
        if rs.last_processed_original[D.seconds] is unset:
            rs.last_processed_original[D.seconds] = {}
        for filer_id in known_filer_ids:
            new_floor := MessagePosition{ ts_ns: floor_ts_for(D), offset: BEFORE_FIRST_OFFSET }
            existing  := rs.last_processed_original[D.seconds][filer_id]
            if existing is unset:
                rs.last_processed_original[D.seconds][filer_id] = new_floor
            else if compareMessagePosition(new_floor, existing) < 0:
                // REWIND. compareMessagePosition is the lex (ts, offset) comparator —
                // matching the strict <= skip semantics. Bare-ts comparison would
                // miss boundary events when timestamps are equal but offsets differ.
                rs.last_processed_original[D.seconds][filer_id] = new_floor

    if affects_predicate:
        if rs.last_processed_predicate is unset:
            rs.last_processed_predicate = {}
        for filer_id in known_filer_ids:
            new_floor := MessagePosition{ ts_ns: predicate_floor_ts, offset: BEFORE_FIRST_OFFSET }
            existing  := rs.last_processed_predicate[filer_id]
            if existing is unset or compareMessagePosition(new_floor, existing) < 0:
                rs.last_processed_predicate[filer_id] = new_floor

save rs
release seeding.lock
```

Anchored to `T_start` (when the walk began), not `T_completed`. Replaying the past `D` window catches objects bootstrap missed under walk-cursor drift. Pre-existing delay-group / filer-id entries are pulled back if the new rule's safe floor is earlier than the current cursor; never advanced past their existing position. Sibling rules in the same delay group replay the rewind window — that's a known cost of cursor sharing; surfaced in the bootstrap-completion status.

**New filer shards** discovered during steady-state operation (a filer joins the cluster, or a chunk ref names a `filer_id` not yet in `last_processed_*`) are seeded by the reader at task start via `seedNewlyDiscoveredShardsLazily` (see reader pseudocode). For each newly-discovered `filer_id`, the cursor floor is set to `min(earliest_retained_log_position[filer_id].ts_ns, safe_floor_ts)` with `offset = BEFORE_FIRST_OFFSET`, and any matching `tail_drained_streams` marker is cleared because this is a new retained-log epoch for that stream, where:

- `earliest_retained_log_position[filer_id]` is read once via a filer probe (single RPC: `EarliestRetainedPositionPerShard`).
- `safe_floor_ts` is the kind-specific floor: `T_now - max(D, bootstrapLookbackMin)` for age-origin sweeps, `T_now - bootstrapLookbackMin` for predicate.

Picking the *earlier* of the two ensures no still-retained event can be skipped: if the earliest retained log for that shard predates the safe floor, the cursor sits at that earlier position so the events are delivered. The `BEFORE_FIRST_OFFSET` sentinel ensures the earliest event itself isn't filtered out by the strict `<=` skip predicate.

**Safety scan.** `SeaweedList` is not snapshot-isolated; an entry whose original-write event is older than `D` may be missed by the walk and the reader's replay alike. Defense: a periodic re-bootstrap on event-driven rules at the per-kind cadence (default `max(MinTriggerAge, 7d)` capped at 30d for age rules; 24h for date rules; 7d for count rules). Re-running is idempotent.

**Date-based rules** (`EXPIRATION_DATE`): no per-object pending. `mode = SCAN_AT_DATE`. The detector schedules a single bucket-level bootstrap at `rule.date`; that bootstrap deletes all matches inline. No reader involvement for date rules.

Before `rule.date`, no work happens. New objects PUT during the wait period don't need handling — the date-triggered bootstrap evaluates current state. Tag changes are likewise reflected. After `rule.date`, the bootstrap sweeps everything in one pass via inline `LifecycleDelete` calls (same as any other bootstrap); no per-object pending is written.

Bootstrap is checkpointable: persist `last_scanned_path` after each page. On worker death, the next bootstrap task resumes from there.

### Compiled policy engine

A naive design — per-rule reader tasks, each scanning the meta log filtered to its bucket prefix — multiplies log I/O by `N_rules` and event-evaluation by `N_events × N_rules`. Wrong shape.

The right shape: **one shared cluster-level reader, one compiled policy engine, one event read = one router lookup**. All rules across all buckets are evaluated by routing each event through the engine; rules with the same trigger delay share their cutoff sweep.

**ActionKey is the engine's primary identity.** Every per-action data structure — engine indexes, target modes, newly-completed sets, bootstrap completion, drain/locks/metrics, status — is keyed by `ActionKey{rule_hash, action_kind}`, not by `rule_hash` alone. A single XML rule with two action sub-elements appears as **two** `ActionKey`s through the entire pipeline: separate delay group memberships, separate cursors, separate pending files, separate locks, separate completion bits. Sibling actions can complete bootstrap on different schedules and degrade independently.

```go
type ActionKey struct {
    RuleHash   [8]byte
    ActionKind ActionKind
}
```

**Engine shape** (rebuilt periodically and on lifecycle PUT/DELETE):

```
type Engine struct {
    snapshot_id   uint64                                 // monotonic; bumps on every rebuild

    // Per-bucket prefix trie -> candidate ActionKeys.
    buckets       map[bucket]*BucketIndex

    // Trigger-delay groups. Original-write events for these (rule, action)
    // pairs become eligible at the same cutoff; the reader sweeps each group
    // in one pass. Only ActionKeys with mode = EVENT_DRIVEN AND
    // bootstrap_complete are included here — engine.compile filters before
    // adding to these collections so the reader cannot route events to
    // scan_only / scan_at_date / disabled / pending actions.
    originalDelayGroups map[time.Duration][]ActionKey   // e.g. 7d -> [{rA, ABORT_MPU}], 90d -> [{rA, EXPIRATION_DAYS}, {rB, NONCURRENT_DAYS}]

    // Predicate-change actions — single near-now sweep, no group needed.
    // Same EVENT_DRIVEN + bootstrap_complete filter.
    predicateActions    []ActionKey

    // Date-driven actions — handled by SCAN_AT_DATE bootstrap, not the reader.
    dateActions         map[ActionKey]time.Time

    // Definitions (for evaluation).
    actions       map[ActionKey]*CompiledAction
}

type BucketIndex struct {
    prefixTrie  *PrefixTrie                              // path prefix -> []ActionKey
    tagIndex    map[tagKey]map[tagValue][]ActionKey      // optional: speeds up tag-filter actions
    versioned   bool
}

type CompiledAction struct {
    raw         *s3lifecycle.Rule    // shared with sibling actions of the same XML rule
    key         ActionKey            // (rule_hash, action_kind)
    bucket      string
    delay       time.Duration        // MinTriggerAge(rule, kind) for age kinds; zero for date/count
    predicateSensitive bool           // true if the rule's filter has tag/size predicates
    mode        RuleMode              // EVENT_DRIVEN | SCAN_AT_DATE | SCAN_ONLY | DISABLED
                                       // Mirrored from durable state[ActionKey].mode at compile time;
                                       // engine.compile uses it to decide whether to register the
                                       // action in originalDelayGroups / predicateActions.
}
```

The engine is built once per worker process from all bucket lifecycle xml; rebuilt on policy change events observed in the meta log itself. The `snapshot_id` stamps each evaluation; pending items recorded under one snapshot are still valid as long as their `ActionKey` survives in the next snapshot.

**Two-phase action activation.** When the engine compiles a new `ActionKey` from a freshly-PUT lifecycle XML, the action starts in state `pending_bootstrap` and is **excluded from reader sweeps**. Only after the bucket's bootstrap task runs and seeds the cluster cursors for that action's delay group does the action transition to `active`. Activation is per `ActionKey`, not per rule — a rule's 7d `ABORT_MPU` action can become active before its 90d `EXPIRATION_DAYS` sibling finishes bootstrap. Concretely:

```
engine.compile(bucket, rule, kind):                    // called once per (rule, kind) pair
    key := ActionKey{rule.hash, kind}
    // Activation requires BOTH:
    //   (a) bootstrap_complete (durable; cursor seed already committed before this).
    //   (b) mode allows event-driven processing.
    // The reader only routes events to actions with mode = EVENT_DRIVEN. SCAN_AT_DATE
    // actions are handled by their date-triggered bootstrap; SCAN_ONLY actions are
    // handled by periodic safety-scan bootstraps; DISABLED actions do nothing.
    // Each CompiledAction carries its mode so reader matching can filter.
    if state[key].bootstrap_complete && state[key].mode == EVENT_DRIVEN:
        action.engine_state = active
    else:
        action.engine_state = inactive   // pending_bootstrap, scan_only, scan_at_date, or disabled

reader.MatchOriginalWrite/MatchPredicateChange/MatchPath:
    return only ActionKeys with engine_state == active

bootstrap.complete (REQUIRED ORDER — same as the main bootstrap pseudocode above):
    target_modes := { key: decideMode(snapshot.actions[key])
                      for key in snapshot.action_keys_for_bucket }
    seedReaderCursorsForNewDelayGroups(snapshot, newly_completed, target_modes, T_start)
                                                                                // (1) durable cluster
    // (2) per-action durable writes — mode AND bootstrap_complete committed atomically per ActionKey.
    for key in newly_completed:
        state[key].mode               = target_modes[key]
        state[key].bootstrap_complete = true                                    // AFTER (1)
        engine.markActive(key)                                                  // (3) in-memory hint
```

The transition order matters: cursor seed must commit **before** the per-action durable write that sets both `mode` and `bootstrap_complete=true`. If the engine refreshes between (1) and (2) for some key, it sees `bootstrap_complete=false` and keeps that action `pending_bootstrap` — correct. If the engine refreshes between (2) and (3), it sees `bootstrap_complete=true && mode == EVENT_DRIVEN` and activates the action — also correct, because (1) already committed. The in-memory `markActive` step is only an optimization to avoid waiting for the next engine refresh; it does not gate correctness. Writing `mode` and `bootstrap_complete` in a single proto write per action (atomic at the file level) avoids any window where one is set and the other isn't.

Policy changes during a bucket bootstrap: the bootstrap is bound to a specific `snapshot_id` (recorded in `_bootstrap`). If the policy changes mid-walk, the in-progress bootstrap finishes for its snapshot; the new policy compiles a fresh engine snapshot; new/changed `ActionKey`s start in `pending_bootstrap`; the next detector tick emits a fresh bucket bootstrap for the new snapshot. `ActionKey`s whose `(rule_hash, action_kind)` is unchanged carry over their `bootstrap_complete=true` and stay `active`. An XML edit that adds a new action sub-element to an existing rule produces a new `ActionKey` (same rule_hash, new action_kind) that goes through `pending_bootstrap` while the rule's other actions stay active.

### Reader — single subscription, client-side merge

> **Implementation note (post-Phase 3):** the production reader uses `client.SubscribeMetadata(...)` directly with a single global cursor. The filer's `SubscribeMetadata` server (`weed/server/filer_grpc_server_sub_meta.go`) already aggregates across peer filers — it reads disk-persisted logs from every filer via `sendLogFileRefs` then drains `MetaAggregator.MetaLogBuffer` for the in-memory tail — so the worker doesn't have to assemble its own multi-filer merge. The per-filer-shard cursor model below was designed before this was wired and was not built; the proto fields (`last_processed_original` map keyed by `filer_id`, `tail_drained_streams`) remain as forward-compat stubs. The pseudocode below describes the alternative the design considered, kept here for context. See "Multi-filer durability" and the obsoleted Phase 6 section.

The reader is one cluster-singleton task subscribed to one filer at a time. `Filer.CollectLogFileRefs(start_position, stop_ts)` returns chunk refs from log files in directory order. The worker passes those refs to `pb.ReadLogFileRefsWithPosition` (the new API — see Task #19), which heap-merges per-filer chunks by event ts client-side and applies the per-shard `(ts, offset)` skip filter. Chunk *bodies* are read in parallel from volume servers via the returned fileIds.

Cursors are per-filer-shard. `last_processed_original[D.seconds]` is `map<filer_id, MessagePosition>`; `MessagePosition = {ts_ns, offset}` disambiguates equal-ts events within a per-filer chunk. Resume skips events with `(event.ts, event.offset) <= cursor[filer_id]` — strict `<=` so the last resolved event isn't replayed.

```
reader_state  := load("/etc/s3/lifecycle/_reader/reader_state")
primary_filer := pickHealthyFiler(reader_state.primary_filer_endpoint)
flush_lag     := 2 * LogFlushInterval     // currently ~2 minutes; derived from filer config

// One probe per task entry: which per-filer log shards exist, what's the earliest
// retained position on each, and what's the latest? `earliest` is used by
// lazy-seeding to safely floor new shards' cursors; `latest` is used for tail-drain
// GC immediately below.
retained_range  := primary_filer.RetainedLogRangePerShard()   // map<filer_id, {earliest, latest}>
earliest_retained := { fid: r.earliest for fid, r in retained_range }

// TAIL-DRAIN / LOST-LOG GC. Run BEFORE blocker load and seeding so the rest of
// the pass operates on the post-GC cursor map (no stale departed-filer entries
// pinning low_water_ts to ancient timestamps; correctly-degraded streams when
// logs were pruned ahead of the cursor).
//
// For each cursor entry (delay group D, filer_id F) — and analogously for the
// predicate cursor map:
//
//   range := retained_range[F]
//
//   case A — F has a retained range AND cursor[F] >= range.latest:
//     The shard is tail-drained. Cursor caught up to (or past) the last
//     retained event for F; no further events from F can arrive without
//     reseeding via lazy-seeding when F reappears. Safe to delete cursor[F].
//
//   case B — F has a retained range AND cursor[F] < range.latest:
//     Not tail-drained. Keep cursor[F]; the regular pass will continue draining.
//
//   case C — F has NO retained range AND we have a durable
//   `tail_drained_streams[stream_key]` marker (recorded the last time we observed
//   case A and pruned this exact stream cursor): safe to skip — F's logs were pruned
//   after this stream cursor caught up. No action; cursor was already removed in
//   case A's pass.
//
//   case D — F has NO retained range AND no `tail_drained_streams[stream_key]`
//   marker AND cursor[F] is present: lost-log scenario. The shard's logs
//   were pruned before the cursor reached `range.latest` — events were
//   silently dropped. We must not GC the cursor as if it were safe; doing so
//   would erase the only signal that those events ever existed.
//     1. Mark every reader-driven rule as `degraded_reason = LOST_LOG` and
//        downgrade `mode = scan_only` until the next safety scan re-establishes
//        correctness. The log shard is global metadata history; once it is pruned,
//        we cannot prove which buckets were unaffected.
//     2. Surface `lifecycle_lost_log_total{filer_id}` and
//        `lifecycle_degraded_streams{reason="LOST_LOG"}` metrics; emit a
//        warning log including F and the cursor position.
//     3. Leave cursor[F] in place (don't GC) until an operator runs
//        `s3.lifecycle.reseed -ack-lost-log --reason <text>`, which clears
//        lost-log cursor entries and the degraded flag together. Deleting silently is
//        forbidden — the operator must explicitly acknowledge the data loss.
//
// `tail_drained_streams` is a small durable set in `reader_state` updated
// each time a stream cursor is removed under case A, so case C can distinguish
// "we already GC'd this exact stream safely" from "logs vanished without this
// stream catching up." The key is `(ORIGINAL, D, F)` or `(PREDICATE, F)`.
gcDepartedShards(reader_state, retained_range)
save reader_state                                    // persist GC results before reads

// LOAD ACTIVE BLOCKERS once at task entry. The reader uses this set to skip
// blocked (shard, stream, [delay]) tuples without re-reading their events
// each batch. Without this, a stuck shard would pull events repeatedly across
// every tick — wasted I/O and noisy "blocked again" log spam.
active_blockers := load("/etc/s3/lifecycle/_reader/blockers")
blocked_original := { (b.shard, b.delay_seconds) for b in active_blockers
                                                  where b.stream_kind == ORIGINAL }
blocked_predicate := { b.shard for b in active_blockers
                                where b.stream_kind == PREDICATE }

// SEED FIRST, PERSIST, THEN READ.
for each delay group D in engine.originalDelayGroups:
    seedNewlyDiscoveredShardsLazily(
        reader_state.last_processed_original[D.seconds],
        /*kind=*/ ageOrigin(D), earliest_retained,
        reader_state.tail_drained_streams, now)
seedNewlyDiscoveredShardsLazily(
    reader_state.last_processed_predicate,
    /*kind=*/ predicateChange, earliest_retained,
    reader_state.tail_drained_streams, now)
save reader_state                                    // durable seed before any reads

// Periodic checkpoint: persist reader_state every checkpointEveryNEvents (default 1000)
// or checkpointEveryT (default 5s), whichever first.

// Pass A: age-origin events, one sweep per delay group.
for each delay group D in engine.originalDelayGroups:
    effective_cutoff := min(now - D, now - flush_lag)
    starts := reader_state.last_processed_original[D.seconds]    // already seeded, persisted

    // Build the per-shard start positions to pass into the API. For shards with
    // an active blocker, we pass MaxMessagePosition so ReadLogFileRefsWithPosition
    // delivers nothing for them — the shard is paused at its previous cursor.
    pass_starts := {}
    for filer_id, pos in starts:
        if (filer_id, D.seconds) in blocked_original:
            pass_starts[filer_id] = MaxMessagePosition       // skips all events for this shard
        else:
            pass_starts[filer_id] = pos

    // Listing covers shards we will actually process. We compute the low-water
    // ONLY across non-blocked shards.
    active_shards := { fid for fid in starts if (fid, D.seconds) not in blocked_original }
    if active_shards is empty:                              // every shard for this delay is blocked
        continue                                            // skip to next delay group
    low_water_ts := min over fid in active_shards: starts[fid].ts_ns
    chunkRefs    := primary_filer.CollectLogFileRefs(start_position: ts_only(low_water_ts),
                                                     stop_ts: effective_cutoff)

    halt_batch := false
    pb.ReadLogFileRefsWithPosition(
        chunkRefs,
        readerFn,
        start_positions: pass_starts,
        stop_ts:         effective_cutoff,
        eachLogEntry(event, filer_id, position):
            // First-statement guard: if this (shard, delay) was blocked earlier
            // in THIS batch by a sibling event, the merge engine may still
            // deliver later events from the same shard before our pauseShard()
            // call propagates. Drop them without mutating the cursor — the
            // cursor must stay at the position recorded in the BlockerRecord.
            if (filer_id, D.seconds) in blocked_original:
                return continueLoop                              // no cursor update
            if !event.isOriginalWrite:
                reader_state.last_processed_original[D.seconds][filer_id] = position
                periodicCheckpoint()
                return continueLoop
            candidates := engine.MatchOriginalWrite(event, delayGroup=D)
            if len(candidates) == 0:
                reader_state.last_processed_original[D.seconds][filer_id] = position
                periodicCheckpoint()
                return continueLoop
            // handleEvent owns the retry-budget interaction (recordRetryLater /
            // clearRetryBudget) for the exact failing branch, so the reader
            // callback only sees the final outcome. handleEvent already returned
            // BLOCKED if its own recordRetryLater call promoted internally.
            outcome := handleEvent(event, candidates,
                                   /*stream_kind=*/ ORIGINAL, /*shard=*/ filer_id,
                                   /*delay_seconds=*/ D.seconds, /*position=*/ position)
            switch outcome:
                case DONE, NOOP_RESOLVED:
                    reader_state.last_processed_original[D.seconds][filer_id] = position
                    periodicCheckpoint()
                    return continueLoop
                case RETRY_LATER:
                    save reader_state
                    halt_batch = true
                    return halt
                case BLOCKED:
                    blocked_original.add((filer_id, D.seconds))
                    save reader_state
                    return pauseShard(filer_id)
    )
    save reader_state
    if halt_batch: return ContinuationHint{ next_run_after_ns: now + 60s, reason: "RETRY_LATER" }

// Pass B: predicate-change events.
predicate_cutoff := now - max(smallDelay, flush_lag)
starts := reader_state.last_processed_predicate                    // already seeded

pass_starts := {}
for filer_id, pos in starts:
    if filer_id in blocked_predicate:
        pass_starts[filer_id] = MaxMessagePosition        // skip all events for this shard
    else:
        pass_starts[filer_id] = pos

active_shards := { fid for fid in starts if fid not in blocked_predicate }
if active_shards is not empty:
    low_water_ts := min over fid in active_shards: starts[fid].ts_ns
    chunkRefs    := primary_filer.CollectLogFileRefs(start_position: ts_only(low_water_ts),
                                                     stop_ts: predicate_cutoff)

    halt_batch := false
    pb.ReadLogFileRefsWithPosition(
        chunkRefs, readerFn,
        start_positions: pass_starts, stop_ts: predicate_cutoff,
        eachLogEntry(event, filer_id, position):
            // First-statement guard: same-batch BLOCKED protection.
            if filer_id in blocked_predicate:
                return continueLoop                              // no cursor update
            if !event.isPredicateChange:
                reader_state.last_processed_predicate[filer_id] = position
                periodicCheckpoint()
                return continueLoop
            candidates := engine.MatchPredicateChange(event)
            if len(candidates) == 0:
                reader_state.last_processed_predicate[filer_id] = position
                periodicCheckpoint()
                return continueLoop
            // handleEvent owns retry-budget calls; reader callback only acts on outcome.
            outcome := handleEvent(event, candidates,
                                   /*stream_kind=*/ PREDICATE, /*shard=*/ filer_id,
                                   /*delay_seconds=*/ 0, /*position=*/ position)
            switch outcome:
                case DONE, NOOP_RESOLVED:
                    reader_state.last_processed_predicate[filer_id] = position
                    periodicCheckpoint()
                    return continueLoop
                case RETRY_LATER:
                    save reader_state
                    halt_batch = true
                    return halt
                case BLOCKED:
                    blocked_predicate.add(filer_id)
                    save reader_state
                    return pauseShard(filer_id)
    )
    save reader_state
    if halt_batch: return ContinuationHint{ next_run_after_ns: now + 60s, reason: "RETRY_LATER" }
```

`periodicCheckpoint` is `if eventCount % checkpointEveryNEvents == 0 || timeSinceLastSave > checkpointEveryT { save reader_state }`. It batches per-event mutations into chunked durable writes; on crash, redo is bounded by the configured budget. Defaults: 1000 events or 5s.

Notes on the listing call: `CollectLogFileRefs` accepts a single `MessagePosition` start. We pass the **low-water-mark ts across all per-filer cursors** so the listing returns chunks covering every shard's still-unread tail. `ReadLogFileRefsWithPosition` then applies the per-shard `start_positions` filter so each shard only delivers events strictly past its own cursor. This means we may stream a few already-processed events from shards that are ahead, but the per-shard filter drops them before the callback.

`seedNewlyDiscoveredShardsLazily` handles the case where a chunk ref names a `filer_id` we've never seen before. The caller passes in the precomputed `earliest_per_shard` (probed once at task entry) so the helper does not re-issue the RPC:

```
seedNewlyDiscoveredShardsLazily(cursors, kind, earliest_per_shard, tail_drained_streams, now):
    // earliest_per_shard: map<filer_id, MessagePosition>, from
    //   primary_filer.EarliestRetainedPositionPerShard() probed once per task.
    // kind:               ageOrigin(D) or predicateChange.
    // now:                wall clock at task start (for the safe floor).

    for filer_id in earliest_per_shard:
        if filer_id not in cursors:
            // Floor at the earlier of (a) the earliest retained log for this shard,
            // and (b) the kind-specific safe floor (now - max(kind.delay, bootstrapLookbackMin)
            // for age-origin; now - bootstrapLookbackMin for predicate). The EARLIER
            // of the two ensures retained events that predate the safe floor are
            // delivered; BEFORE_FIRST_OFFSET ensures the earliest event itself is
            // not filtered out by the strict <= skip predicate.
            safe_floor_ts := safeFloorTsFor(kind, now)
            shard_floor   := min(earliest_per_shard[filer_id].ts_ns, safe_floor_ts)
            cursors[filer_id] = MessagePosition{ ts_ns: shard_floor, offset: BEFORE_FIRST_OFFSET }
            delete tail_drained_streams[tailDrainedStreamKey(kind, filer_id)]
```

The probe is a single filer RPC at task start, cheap. Passing the result in (rather than re-probing inside the helper) means a single task only issues one `EarliestRetainedPositionPerShard` call no matter how many delay groups call the seeder.

Properties:

- **Cursors are per-filer-shard**; cursor state per delay group is `map<filer_id, MessagePosition>`. Resume after partial failure is correct on each shard independently.
- **Listing uses the low-water-mark ts**; per-shard filtering inside the new API. Single subscription endpoint; per-shard correctness.
- **Watermark exposed externally** is the low-water-mark across per-filer cursors per delay group. Lag = `now - watermark`.
- **Cost ≈ events × distinct_delay_groups** for age-origin passes; the per-delay sweeps each pull a time window from the merged stream. Real configs have a handful of distinct day values.
- For configs with many distinct delays (rare), the engine should optionally **merge nearby delay groups** into bins to bound the number of passes.
- **Filer is not the bottleneck** — it serves chunk metadata only. Volume reads parallelize per chunk.
- **Failover** moves the worker to a different `primary_filer_endpoint`; per-filer cursor map is portable because every filer reads the same persisted logs from the shared namespace.

If a single worker cannot keep up after raising in-worker chunk-read concurrency and `lifecycle.delete.concurrency`, the next-stage answer is segment-level intra-stream parallelism (see "Future scaling" — explicitly non-v1).

`handleEvent(event, candidates, stream_kind, shard, delay_seconds, position) → outcome`:

Single contract. Fetch the live entry once, evaluate every candidate rule against it, return the aggregate outcome. The caller passes the stream context (`stream_kind`, `shard`, `delay_seconds`, `position`) that `handleEvent` forwards to `DeleteContext` and any `BlockerRecord` writes. Reader passes `ORIGINAL` or `PREDICATE` plus `(filer_id, D.seconds, position)` or `(filer_id, 0, position)` respectively; bootstrap and pending drain construct `DeleteContext` directly without going through `handleEvent`.

The cursor (the per-shard `MessagePosition` for the event's `filer_id`) only advances past this event when **every** candidate returns `DONE`/`NOOP_RESOLVED`. If any candidate returns `RETRY_LATER` or `BLOCKED`, that outcome propagates up and the caller halts (or pauses the shard) per the reader pseudocode.

```
handleEvent(event, candidates, stream_kind, shard, delay_seconds, position):
    // handleEvent owns its retry-budget interactions. Building a RetryTarget
    // outside (in the reader callback) would have to guess "which candidate
    // failed" — wrong if the failure is fetchLive ERR (no candidates evaluated
    // yet) or an early candidate (not the last one). Doing it here means the
    // RetryTarget reflects the actual failing branch.
    stream_key := buildStreamKey(stream_kind, shard, delay_seconds, position)
    base_target := RetryTarget{
        key:         stream_key,
        bucket:      event.bucket,
        object_path: event.path,
        version_id:  extractVersionId(event.path),    // best effort; refined when live is available
        // rule_hash filled in per branch below (nil before first candidate evaluation)
    }

    // fetchLiveEntry is tri-state. Conflating not-found with errors would either
    // advance the cursor over un-evaluated events (transport blip) or stall on a
    // legitimately-deleted entry. Distinguish:
    //   FOUND     → proceed.
    //   NOT_FOUND → object is gone (live state = absent). Resolved.
    //   ERR       → transient transport/RPC failure → RETRY_LATER + retry budget.
    //   FATAL     → deterministic decode/schema failure → BLOCKED. writeBlocker.
    live, status := fetchLiveEntry(event.path)
    switch status:
        case NOT_FOUND:
            clearRetryBudget(stream_key)
            return NOOP_RESOLVED
        case ERR:
            return recordRetryLater(base_target)         // returns RETRY_LATER or promoted BLOCKED
        case FATAL:
            // Persistent failure. Persist the blocker durably BEFORE returning
            // BLOCKED — otherwise the cursor would pause without a record for
            // the operator to inspect or clear, and the lag-fallback would
            // eventually demote silently. If the durable write itself fails,
            // return RETRY_LATER and let the next batch retry; the cursor stays.
            ok := writeBlocker(BlockerRecord{
                stream_kind:      stream_kind,    // ORIGINAL or PREDICATE for reader callers
                shard:            shard,
                delay_seconds:    delay_seconds,  // 0 for PREDICATE
                position:         position,
                rule_hash:        nil,            // unset — failure is pre-evaluation, not bound to a rule
                bucket:           event.bucket,
                object_path:      event.path,
                version_id:       extractVersionId(event.path), // may be ""
                reason:           "fetch_live_FATAL",
                last_error:       reason,
                first_seen_at_ns: now(),
                last_retry_at_ns: now(),
                retry_count:      1,
            })
            if !ok: return RETRY_LATER            // blocker write failed; treat like outage
            // Clear any retry_budget entry that may have accumulated from prior
            // transient ERRs at this same position. Without this, a stream that
            // bounced through ERR → ERR → FATAL would leave a stale budget entry
            // that confuses subsequent budget accounting.
            clearRetryBudget(stream_key)
            return BLOCKED
        case FOUND:                               // continue
    info := buildObjectInfo(live)

    aggregate := NOOP_RESOLVED                    // start from "nothing required"
    for each rule in candidates:
        if !ruleAppliesToEntryShape(rule, live):  continue
        action := s3lifecycle.EvaluateAction(rule, action_kind, info, now)
        if action == ActionNone:                  continue
        dueAt := computeDueAt(rule, info)

        var per_rule_outcome eventOutcome
        ctx := DeleteContext{
            bucket:            event.bucket,
            object_path:       event.path,
            version_id:        live.version_id,
            rule_hash:         rule.hash,
            rule:              rule,
            stream_kind:       stream_kind,        // ORIGINAL or PREDICATE — passed in by the reader pass
            shard:             shard,               // == event.filer_id when called from reader
            delay_seconds:     delay_seconds,       // populated only when stream_kind == ORIGINAL
            position:          event.position,
            action:            action,
            expected_identity: identityFromLive(live),
        }
        if event.isOriginalWrite():
            // event.ts crossed the original cutoff, but live.mtime may be newer
            // (overwrite resets the clock). The overwrite produces its own
            // age-origin event; this older event is stale.
            if now < dueAt:
                per_rule_outcome = NOOP_RESOLVED
            else:
                per_rule_outcome = deleteAndResolve(ctx)
        else if event.isPredicateChange():
            if now >= dueAt:
                per_rule_outcome = deleteAndResolve(ctx)
            else:
                // Late predicate change made the object newly match, but mtime+TTL
                // is in the future. The original-write event is already past the
                // watermark, so we'd miss this without an exception record.
                err := upsertPending(rule, PendingItem{
                    path: event.path, version_id: extractVersionId(...),
                    due_at_ns: dueAt.UnixNano(),
                    expected_identity: identityFromLive(live),
                })
                if err == ErrPendingFull or err == ErrPendingWriteFailed:
                    // Demote the rule to scan_only durably. If the demote also
                    // fails (filer IO unhealthy), feed the retry budget for THIS
                    // rule's slot and return — sustained demote-failures will
                    // promote to BLOCKED.
                    if !durably_demote_rule_to_scan_only(rule.hash, reason: err):
                        rule_target := base_target
                        rule_target.rule_hash = rule.hash
                        return recordRetryLater(rule_target)
                    counter.pending_full_total{rule_hash}++; log_warn(...)
                    per_rule_outcome = NOOP_RESOLVED
                else:
                    per_rule_outcome = NOOP_RESOLVED   // exception recorded in pending
        else:
            per_rule_outcome = NOOP_RESOLVED       // unknown event class

        if per_rule_outcome == RETRY_LATER:
            // deleteAndResolve produced RETRY_LATER for THIS specific candidate
            // rule. Feed the retry budget with a target that reflects the
            // actual failing branch (this rule's hash + this rule's action).
            rule_target := base_target
            rule_target.rule_hash = rule.hash
            return recordRetryLater(rule_target)
        if per_rule_outcome == BLOCKED:
            // deleteAndResolve already wrote a BlockerRecord with this rule's
            // hash. Clear any retry budget that was tracking earlier transient
            // failures for this stream key.
            clearRetryBudget(stream_key)
            return BLOCKED
        aggregate = max(aggregate, per_rule_outcome)   // DONE > NOOP_RESOLVED for telemetry

    // All candidates returned DONE/NOOP_RESOLVED. Clear any in-flight retry
    // budget for this stream key — success at this position resets the budget.
    clearRetryBudget(stream_key)
    return aggregate                              // safe to advance cursor past event.ts
```

**Processing outcomes (single contract used by reader, bootstrap walker, and pending drain):**

| Outcome | Cursor advance? | Meaning |
|---|---|---|
| `DONE` | yes | Action resolved (delete succeeded, marker created, abort completed). |
| `NOOP_RESOLVED` | yes | Object already deleted / changed / no longer applicable / object-lock skipped. No further work needed. |
| `RETRY_LATER` | **no** | Transient failure (S3 unreachable, network blip, filer IO error). The batch halts at this position; the next task retries from the same `MessagePosition`. |
| `BLOCKED` | **no** | Deterministic or persistent failure (`FATAL_EVENT_ERROR`, malformed entry, `TRANSPORT_ERROR` exceeded its retry budget). The shard is paused and a `BlockerRecord` is written. The cursor stays at the failing position until an operator intervenes. |

**No automatic dead-lettering.** A dead letter would mean "this record is poison, skip it so the worker can keep moving." For lifecycle that's wrong: skipping a record means losing a delete decision (correctness). The worker pauses instead and surfaces the blocker. Operators decide whether to `retry`, `resume`, or explicitly `quarantine`.

`deleteAndResolve`:

```
type StreamKind enum { ORIGINAL = 0; PREDICATE = 1; BOOTSTRAP = 2; PENDING = 3; }

type DeleteContext struct {
    // Identifying:
    bucket            string
    object_path       string
    version_id        string          // optional; "" for non-versioned
    rule_hash         bytes
    rule              *CompiledRule

    // Stream / shard / position context (used by BlockerRecord scoping):
    stream_kind       StreamKind      // ORIGINAL | PREDICATE for reader; BOOTSTRAP / PENDING otherwise
    shard             string          // filer_id; "" for BOOTSTRAP / PENDING (not shard-bound)
    delay_seconds     int64           // populated when stream_kind == ORIGINAL; 0 otherwise
    position          MessagePosition // populated when stream_kind == ORIGINAL or PREDICATE

    // What to do:
    action            Action          // CURRENT | VERSION | DELETE_MARKER | ABORT_MPU
    expected_identity EntryIdentity   // CAS
}

deleteAndResolve(ctx DeleteContext):
    // K in-memory retries against the same event for FATAL_EVENT_ERROR / worker-local
    // deterministic errors. TRANSPORT_ERROR is NEVER counted here — it returns
    // RETRY_LATER immediately. The caller feeds the durable retry_budget for this
    // stream key; sustained TRANSPORT_ERROR promotes to BLOCKED, while watermark-lag
    // fallback remains the retention-safety backstop.
    for attempt in 1..K (default 5):
        outcome := lifecycleDeleteRPC(ctx.bucket, ctx.object_path, ctx.version_id,
                                      ctx.action, ctx.expected_identity)
        switch outcome {
        case DELETED:
            return DONE
        case NOT_FOUND, STALE_IDENTITY, STALE_POLICY, NO_LONGER_ELIGIBLE:
            return NOOP_RESOLVED
        case SKIPPED_OBJECT_LOCK:
            counter.skipped_object_lock_total++; log_warn(...); return NOOP_RESOLVED
        case TRANSPORT_ERROR:
            return RETRY_LATER                 // batch halts; cursor stays
        case FATAL_EVENT_ERROR:
            // retry deterministically; if still FATAL after K attempts, fall through to BLOCKED
            continue
        }

    // K in-memory retries exhausted. The event is deterministically un-processable.
    // Write a BlockerRecord scoped by stream_kind so the right cursor pauses; only
    // return BLOCKED after the durable write commits. If the write itself fails
    // (filer IO unhealthy), return RETRY_LATER instead — without a durable
    // blocker, BLOCKED would silently halt the stream with nothing for the
    // operator to inspect.
    ok := writeBlocker(BlockerRecord{
        stream_kind:      ctx.stream_kind,
        shard:            ctx.shard,
        delay_seconds:    ctx.delay_seconds,    // populated only for ORIGINAL
        position:         ctx.position,         // populated for ORIGINAL/PREDICATE
        rule_hash:        ctx.rule_hash,
        bucket:           ctx.bucket,
        object_path:      ctx.object_path,
        version_id:       ctx.version_id,
        reason:           "FATAL_EVENT_ERROR after K retries",
        last_error:       last_outcome.reason,
        first_seen_at_ns: now(),
        last_retry_at_ns: now(),
        retry_count:      K,
    })
    if !ok: return RETRY_LATER                  // blocker write failed; treat like outage
    // Clear any retry_budget entry that earlier RETRY_LATER attempts at this
    // same key may have left; promotion to BLOCKED via FATAL is a different
    // path from sustained-RETRY_LATER promotion, but they share the stream key.
    clearRetryBudget(buildStreamKeyFromCtx(ctx))
    return BLOCKED
```

The function takes a single `DeleteContext` because four callers (reader-original, reader-predicate, bootstrap walker, pending drain) all populate the same shape. The `stream_kind` field tells `writeBlocker` which cursor to pause: ORIGINAL pauses `(shard, delay_seconds)`; PREDICATE pauses `shard` predicate stream; BOOTSTRAP pauses the bucket walker; PENDING pauses the rule's drain.

**Watermark advance, per the outcome enum above.** The per-shard cursor `last_processed_*[filer_id]` advances past an event whose `handleEvent` returns `DONE` or `NOOP_RESOLVED`. It does **not** advance on `RETRY_LATER` (transient — next batch retries) or `BLOCKED` (persistent — operator intervention required). The meta log itself is the durable queue; the cursor's persisted position **is** the retry pointer.

`drainPending` runs per `(bucket, rule_hash)`. At task entry it loads `_reader/blockers` and refuses to drain if a `PENDING` blocker is active for this rule:

```
// Task-start guard: if a PENDING blocker is active for this (bucket, rule_hash),
// don't drain. Operator must clear via blockers retry|resume|quarantine.
active_blockers := load("/etc/s3/lifecycle/_reader/blockers")
if any b in active_blockers where b.stream_kind == PENDING
                                  and b.bucket == bucket
                                  and b.rule_hash == rule.hash:
    return ContinuationHint{ next_run_after_ns: now + 60s, reason: "PENDING blocker active" }

// Per-item processing.
// Build the stream key + retry target BEFORE fetchLive, so that an ERR can
// participate in the retry budget. Without this, repeated transient fetchLive
// failures for the same pending item would never promote to BLOCKED.
stream_key := StreamKey{ stream_kind: PENDING, pending: PendingKey{
    bucket: bucket, rule_hash: rule.hash,
    object_path: item.path, version_id: item.version_id } }
retry_target := RetryTarget{
    key:        stream_key,
    bucket:     bucket,
    object_path: item.path,
    version_id: item.version_id,
    rule_hash:  rule.hash,
}
// RetryTarget intentionally carries no action / expected_identity — operator-driven
// retry replays drainPending against current live state, which re-fetches and
// re-evaluates from scratch.

live, status := fetchLiveEntry(item.path)
switch status:
    case NOT_FOUND:
        clearRetryBudget(stream_key)                   // resolved (object gone)
        pending.delete(item); return
    case ERR:
        // Transient failure — feed the retry budget. Sustained ERR promotes to BLOCKED.
        promoted := recordRetryLater(retry_target)
        if promoted == BLOCKED:
            return                                     // BlockerRecord written; drain pauses
        return                                         // leave pending intact; next drain retries
    case FATAL:
        // Deterministic per-event failure. Block this rule's drain at this item;
        // skip the retry budget — FATAL is not a transient outage.
        if !writeBlocker(BlockerRecord{
            stream_kind:      PENDING,
            shard:            "",                    // PENDING is not shard-bound
            delay_seconds:    0,
            position:         MessagePosition{},     // unused for PENDING
            rule_hash:        rule.hash,
            bucket:            bucket,
            object_path:       item.path,
            version_id:        item.version_id,
            reason:            "fetch_live_FATAL",
            last_error:        reason,
            first_seen_at_ns:  now(),
            last_retry_at_ns:  now(),
            retry_count:       1,
        }): return                                   // blocker write failed; leave pending intact
        clearRetryBudget(stream_key)                  // any prior ERR-driven budget is superseded
        return                                       // pending stays; drain pauses for this rule
    case FOUND:            // continue
        clearRetryBudget(stream_key)                  // fetchLive succeeded; reset any pre-existing budget

if !identityMatch(live, item.expected_identity):
    // object moved since we deferred. The newer event already created its own pending
    // or already deleted; drop this stale entry.
    pending.delete(item); return

info  := buildObjectInfo(live)
dueAt := computeDueAt(rule, info)
if now < dueAt:
    item.due_at_ns = dueAt.UnixNano()      // mtime advanced (rare but possible via metadata update)
    pending.upsert(item); return

action := s3lifecycle.EvaluateAction(rule, action_kind, info, now)
if action == ActionNone:                   pending.delete(item); return

// Same deleteAndResolve contract as reader and bootstrap.
outcome := deleteAndResolve(DeleteContext{
    bucket:            bucket,
    object_path:       item.path,
    version_id:        item.version_id,
    rule_hash:         rule.hash,
    rule:              rule,
    stream_kind:       PENDING,
    shard:             "",                  // PENDING is not shard-bound
    delay_seconds:     0,                   // unused for PENDING
    position:          MessagePosition{},   // unused for PENDING
    action:            action,
    expected_identity: item.expected_identity,
})
switch outcome {
case DONE, NOOP_RESOLVED:
    clearRetryBudget(stream_key)
    pending.delete(item)
case RETRY_LATER:
    promoted := recordRetryLater(retry_target)
    if promoted == BLOCKED:
        // BlockerRecord already written by recordRetryLater on threshold trip.
        // Leave pending intact; drain pauses on this rule until operator clears.
        return
    // Plain transient: leave pending intact; next drain retries.
    return
case BLOCKED:
    // BlockerRecord already written by deleteAndResolve. Leave pending intact;
    // drain pauses on this rule until the operator clears the blocker.
    clearRetryBudget(stream_key)        // any in-flight retry budget is moot
    return
}
```

### Why current state, not event payload

Tags and other predicates can change after the original PUT. Binding the predicate to the event payload means a tag added at age 30d for a 60d rule isn't picked up until 60d after the tag-add — up to one TTL late. Live evaluation handles this in O(1) extra filer lookups.

### CAS identity

```
type entryIdentity struct {
    Mtime        time.Time   // entry.Attributes.Mtime
    Size         int64       // entry.Attributes.FileSize
    HeadFid      string      // entry.Chunks[0].FileId, "" if zero-byte
    ExtendedHash [8]byte     // sha256(canonicalize(entry.Extended))[:8]
}
```

`ExtendedHash` covers tags and any other Extended-stored predicate input. Without it, a tag flipped between worker evaluation and server-side delete would let the server proceed against stale eligibility.

Mtime alone is fooled by snapshot-restore. Head fid resolves it: a re-uploaded object lands on a fresh fid.

### Policy-version CAS — per ActionKey, not per bucket

A single bucket-wide etag would invalidate this action's pending whenever any *other* rule (or sibling action) on the same bucket is edited. Wrong: an unrelated edit shouldn't dump pending work that's still valid under this action. Worse, because pending wouldn't be re-enqueued unless the object's events fire again (and many objects' relevant events are already in the past), the work would simply be lost.

Use the **`ActionKey{rule_hash, action_kind}`** as the CAS token. Pending items live under `/etc/s3/lifecycle/<bucket>/<rule_hash_hex>/<action_kind>/pending`, which already binds them to a specific (rule version, action kind) pair (the rule_hash changes when the rule's filter or any action's parameters change; the action_kind selects the sibling under that rule). On drain and on `LifecycleDelete`:

- Server reads current `lifecycle.xml`, computes the `ActionKey` set (one per populated action sub-element), and checks whether `request.ActionKey` is a member.
- Member → proceed.
- Not a member → `STALE_POLICY`. Either the rule was removed / had its definition changed (so a new `rule_hash` now exists), or the specific action sub-element was removed from an otherwise-unchanged rule. The pending item is genuinely orphaned; drop it. The new `ActionKey` (if any) will go through bootstrap and pick up still-relevant work.

Per-`ActionKey` CAS:
- An edit to rule A leaves rule B's actions untouched.
- An edit to rule A's filter or any action parameter creates a new `rule_hash`; the old `<old_hash_A>/` directory tree (with all its action sub-directories) is orphaned. The orphan is cleaned up after a grace period (so an edit-and-revert preserves progress).
- Removing one action sub-element from rule A while keeping others: only the corresponding `<rule_hash_A>/<removed_kind>/` directory becomes orphaned; siblings stay live. (`rule_hash` is computed over the full action set, so removing or adding an action sub-element produces a new `rule_hash` for the rule. Both the new and old rule_hash directory trees coexist during the grace period; the new one is bootstrapped from scratch.)
- A removal of rule A entirely: the `<hash_A>/` tree becomes orphaned, cleaned up after grace.

### Versioned buckets — corrected model

Versioning facts (verified in repo):
- A versioned object at key `foo/bar.txt` lives under directory `foo/bar.txt.versions/` (suffix `.versions`, not a `.versions` subdirectory).
- Each version is its own file inside that directory; filename encodes timestamp + version id.
- A delete marker is a version file with `Extended[ExtDeleteMarkerKey] == "true"`.
- The current-version pointer is stored as `Extended[ExtLatestVersionFileNameKey]` (and friends) on the `.versions` *directory entry* itself.

Worker handling:

| Event path shape | Meaning | Worker action |
|---|---|---|
| `<bucket>/.../<key>.versions/<versionFile>` (NewEntry, no OldEntry) | New version added (PUT or DeleteMarker). | Trigger evaluation for the previously-latest version (now non-current). Use the parent dir's `ExtLatestVersionMtimeKey` *before* this event as the successor's becoming-current time → no, simpler: when handling, list the `.versions` dir and determine the just-became-non-current version; its `successorMtime = event.TsNs`. |
| `<bucket>/.../<key>.versions` (Old/NewEntry, attrs change on the directory) | Latest pointer flipped. | Same trigger as above, redundantly; idempotent. |
| `<bucket>/.../<key>.versions/<versionFile>` where Extended has `ExtDeleteMarkerKey=true` and that file is now the only entry in the dir | A delete marker is the sole survivor. | `ExpiredObjectDeleteMarker` evaluation. |

For each versioned event, `ruleAppliesToEntryShape` filters out entries that don't match the rule's target (e.g. `Expiration.Days` rules target the current version on a non-versioned bucket; on a versioned bucket they create a new delete marker via `createDeleteMarker`). `NoncurrentVersionExpiration` targets non-current versions only.

Action dispatch on versioned buckets:
- `ActionDeleteObject` (current version of versioned bucket) → `s3a.createDeleteMarker(bucket, key)`.
- `ActionDeleteVersion` (a specific non-current version) → `s3a.deleteSpecificObjectVersion(bucket, key, versionId)`.
- `ActionExpireDeleteMarker` → `s3a.deleteSpecificObjectVersion(bucket, key, versionId)` for the marker.

### AbortIncompleteMultipartUpload

Multipart uploads live under `<bucket>/.uploads/<uploadId>/`. Treat creation events for `<uploadId>` directories under `.uploads/` as triggers; eligibility is `now - mtime > rule.AbortMPUDays`. On expiry, call the existing abort-MPU path (which removes the upload directory).

### Worker-to-S3 discovery and failover

The worker process (`weed/worker/`) connects to the admin server today (`weed/worker/worker.go:81 SendHeartbeat`). The admin server is the natural discovery point for S3 endpoints.

- **Registry.** S3 servers register their gRPC listen address with admin on startup, the same way they register HTTP today. New admin RPC `ListS3Endpoints() repeated S3Endpoint`.
- **Worker cache.** Each worker fetches the S3 endpoint list at startup and refreshes every 30s in the heartbeat path. Cache survives admin reachability blips.
- **Selection.** For each `LifecycleDelete`, pick a healthy endpoint round-robin. On RPC failure (DEADLINE_EXCEEDED, UNAVAILABLE), mark that endpoint unhealthy for 60s and retry on the next.
- **All endpoints unhealthy.** Reader and bootstrap halt at the next `LifecycleDelete` they need to make. The reader's per-shard cursor map stays at the last resolved `(filer_id → MessagePosition)` entries; the bootstrap walker's `last_scanned_path` stays at the last successful entry. The task returns `ContinuationHint{ next_run_after_ns: now + 60s }` and exits. No pending is appended for the halted batch — the meta log is the durable retry queue. No data loss; lifecycle pauses cleanly. When S3 returns, the next read/bootstrap task resumes from exactly the same per-shard cursors.
- **Co-located deployments.** When workers and S3 servers run on the same nodes, the worker can prefer its local S3 endpoint by IP affinity to keep traffic on-host.
- **S3 is not a hard dependency for the worker process to start.** The worker boots and registers; the reader and bootstrap tasks halt on first `LifecycleDelete` if no endpoint is healthy. Engine compilation and policy reconciliation continue (no S3 needed). Once S3 returns, the next scheduled read/bootstrap task picks up where the previous halted, replaying events from the stalled cursor.

### Internal service-principal delete API — concrete shape

The worker is in a separate process (`weed/worker/`) and must not depend on user IAM signatures. We add an internal gRPC method on the S3 server, callable only from authorized workers:

```
service S3InternalService {
  rpc LifecycleDelete(LifecycleDeleteRequest) returns (LifecycleDeleteResponse);
}

message EntryIdentity {
  int64 mtime_ns = 1;
  int64 size = 2;
  string head_fid = 3;
  bytes extended_hash = 4;        // sha256(canonical Extended)[:8]
}

message LifecycleDeleteRequest {
  string bucket = 1;
  string object_path = 2;          // path under bucket dir
  string version_id = 3;           // optional, for ActionDeleteVersion
  enum Action { CURRENT = 0; VERSION = 1; DELETE_MARKER = 2; ABORT_MPU = 3; }
  Action action = 4;
  EntryIdentity expected_identity = 5;
  string trigger_rule_id = 6;
  bytes  trigger_rule_hash = 7;    // server checks membership in current policy's rule-hash set
  bool   skip_chunk_delete = 8;
}

message LifecycleDeleteResponse {
  enum Outcome {
    DELETED              = 0;
    NOT_FOUND            = 1;
    STALE_IDENTITY       = 2;     // CAS failed
    STALE_POLICY         = 3;     // rule_hash no longer in current policy
    NO_LONGER_ELIGIBLE   = 4;     // server-side re-evaluation says rule no longer applies
    SKIPPED_OBJECT_LOCK  = 5;     // object under legal hold or retain-until — skip silently
    TRANSPORT_ERROR      = 6;     // transient (network, RPC); retry by stalling watermark
    FATAL_EVENT_ERROR    = 7;     // deterministic per-event failure (corrupted entry, internal panic);
                                  // server's best attempt at distinguishing transient from permanent
  }
  Outcome outcome = 1;
  string  reason = 2;             // human-readable detail; populated for FATAL_EVENT_ERROR and TRANSPORT_ERROR
}
```

**Worker-local error classifications.** Some failures occur entirely on the worker side and never reach the server: malformed event payload that fails to deserialize, identity construction failure, evaluator panic recovered via `defer recover()`. These are not RPC outcomes — they're worker errors. The worker treats them under the same K-retry-then-`BLOCKED` rule as `FATAL_EVENT_ERROR` (5 in-memory retries, then write a `BlockerRecord` and pause the shard). Server-side `FATAL_EVENT_ERROR` is for cases the server detects (e.g., dispatch helper returns an error not classifiable as transport or object-lock).

Server handler (`weed/s3api/s3api_internal_lifecycle.go`, new file):

1. Authn: shared HMAC token from cluster config or mTLS — same scheme used by other intra-cluster RPCs in `weed/security/`.
2. Re-fetch live entry and verify `expected_identity` matches in full (mtime, size, head fid, extended hash). Mismatch → `STALE_IDENTITY`.
3. Re-fetch bucket lifecycle xml; compute the rule-hash set; if `request.trigger_rule_hash` is not a member → `STALE_POLICY`. (Per-rule CAS, not per-bucket. Other rules' edits don't poison this rule's deletes.)
4. Re-evaluate the rule against the live entry. If `Evaluate(rule, info, now) == ActionNone` → `NO_LONGER_ELIGIBLE`. (Defence in depth: tags or filters could have just changed.)
5. **Object lock**: explicitly call `s3a.enforceObjectLockProtections(bucket, object, versionId, governanceBypassAllowed=false)` (defined in `weed/s3api/s3api_object_retention.go:570`). The existing low-level helpers (`deleteUnversionedObjectWithClient`, `deleteSpecificObjectVersion`, `createDeleteMarker`) do **not** check object lock — the check lives in the user-facing HTTP handler (`weed/s3api/s3api_object_handlers_delete.go:225,124,147,368`) and must be reproduced here. If `enforceObjectLockProtections` returns an error, return `SKIPPED_OBJECT_LOCK`. Lifecycle never bypasses governance retention.

`enforceObjectLockProtections` currently takes `*http.Request` to extract the `x-amz-bypass-governance-retention` header; for lifecycle we hard-code `governanceBypassAllowed=false`. Phase 2 introduces a thin wrapper `enforceObjectLockForLifecycle(bucket, object, versionId)` that calls the same logic without needing a request, by either passing a synthetic request or refactoring the request-extraction step out of the core check.
6. Dispatch via the same internal helpers user deletes use:
   - `CURRENT` on non-versioned → `deleteUnversionedObjectWithClient`.
   - `CURRENT` on versioned → `createDeleteMarker`.
   - `VERSION` → `deleteSpecificObjectVersion`.
   - `DELETE_MARKER` → `deleteSpecificObjectVersion` (for the marker).
   - `ABORT_MPU` → existing abort-upload code.
7. Emit access log + Prometheus counter with principal `lifecycle:<rule_hash>:<rule_id>`.

**Worker handling per outcome:**

| Outcome | Worker action |
|---|---|
| `DELETED` | Advance cursor / drop pending item. Increment `expired_total`. |
| `NOT_FOUND` | Advance cursor / drop pending item. |
| `STALE_IDENTITY` | Advance cursor / drop pending. The newer event (if any) re-triggers via the reader. |
| `STALE_POLICY` | Advance cursor / drop pending. New rule's directory bootstraps. |
| `NO_LONGER_ELIGIBLE` | Advance cursor / drop pending. |
| `SKIPPED_OBJECT_LOCK` | **Advance cursor / drop pending. Increment `skipped_object_lock_total` counter; log at warn.** No retry, no blocked file. The next safety-scan bootstrap evaluates again; if the lock has been released, the object deletes then. |
| `TRANSPORT_ERROR` | **Halt the batch.** Cursor stays at the last resolved event. Retry on next read task. The meta log is the durable retry queue. |

This keeps replication, audit, object-lock, and metrics consistent without bypassing them and without piping the worker through the public IAM stack.

### Metadata-only deletion when chunks are on TTL volumes

When the entry's chunks were placed on TTL volumes, the volume server reclaims them wholesale on volume drop. Per-chunk `DeleteFile` RPCs are wasted work. The filer wire is already there: `Filer.DeleteChunks` honours `rule.DisableChunkDeletion` (`weed/filer/filer_deletion.go:588-594`).

The worker computes `skip_chunk_delete` per object before calling `LifecycleDelete`:

```go
ruleConf := f.FilerConf.MatchStorageRule(path)
skipChunkDelete :=
    live.Attributes.TtlSec > 0 &&    // chunks were placed on TTL volumes
    ruleConf.Ttl != ""                // path is still TTL-routed
```

The S3 server passes the flag to the filer delete path; per-chunk RPCs are skipped; the entry is removed; chunks are reclaimed on the volume's natural TTL drop. Storage reclamation lags by up to the volume's natural TTL boundary (bounded by the latest needle in that volume) — minutes-to-hours in practice, fine for lifecycle.

Versioned buckets: `createDeleteMarker` writes a marker (no chunks deleted in either path; flag is moot). `deleteSpecificObjectVersion` honours `skip_chunk_delete` for non-current versions or expired markers.

This composes with the operator-driven volume-TTL routing that replaces today's PUT back-stamp: an operator runs `weed shell s3.bucket.ttl.set -bucket X -ttl 60d` to add the `filer.conf` route. New writes flow to TTL volumes immediately. Existing entries can be back-stamped via an async maintenance task on `LaneLifecycle` (same code as today's `updateEntriesTTL`, parallelized and restartable). Once back-stamped, lifecycle deletes go through the metadata-only path.

### Scheduling

Existing maintenance scheduler in `weed/admin/maintenance/` and `weed/worker/`. The repo already anticipates this work: `weed/admin/plugin/scheduler_lane.go` defines `LaneLifecycle` (line 24), maps job type `s3_lifecycle` to it (line 83), exempts the lane from the default-lane admin lock (`laneRequiresLock[LaneLifecycle] = false`), and gives it a 5-minute idle sleep. Per-lane execution slots (`execRes` in `schedulerLaneState`) mean lanes don't starve each other for worker capacity.

Concrete plumbing:

- **Lane: `LaneLifecycle`, not `LaneDefault`.** Job-type string `s3_lifecycle` with **three subtypes** carried in params: `READ` (cluster singleton), `BOOTSTRAP` (per bucket), `DRAIN` (per `(bucket, ActionKey)`). The default lane is serialised under one admin lock and is reserved for vacuum/balance/EC/admin scripts; lifecycle must not enter it.
- Task type constants in `weed/worker/types/task_types.go`: `S3LifecycleReadTaskType`, `S3LifecycleBootstrapTaskType`, `S3LifecycleDrainTaskType`. All bound to lane `LaneLifecycle` via `jobTypeLaneMap`.
- Cluster-lock keys per subtype:
  - `READ` → `s3.lifecycle.read` (cluster singleton).
  - `BOOTSTRAP` → `s3.lifecycle.bootstrap:<bucket>` (per bucket; bootstrap walks all `ActionKey`s for the bucket in one pass).
  - `DRAIN` → `s3.lifecycle.drain:<bucket>:<rule_hash_hex>:<action_kind>` (per `ActionKey` — sibling actions of one rule have separate drain locks so their pending streams advance independently).
- Task params proto (extend `weed/pb/worker_pb/worker.proto`):
  ```
  message S3LifecycleParams {
    enum Subtype { SUBTYPE_UNSPECIFIED = 0; READ = 1; BOOTSTRAP = 2; DRAIN = 3; }
    Subtype subtype = 1;

    // bucket is required for BOOTSTRAP and DRAIN. rule_hash is optional for
    // BOOTSTRAP (omitting it means walk-all-actions-for-this-bucket) and
    // required for DRAIN. action_kind is required for DRAIN; ignored by
    // BOOTSTRAP (the walk evaluates every ActionKey per object). READ
    // ignores all three.
    string     bucket      = 2;
    bytes      rule_hash   = 3;       // 8 bytes when present
    ActionKind action_kind = 7;       // DRAIN only

    bool   force = 4;
    int64  batch_time_budget_ns = 5;
    int32  batch_event_budget = 6;
  }
  ```
- Continuation hint on task result (extend the result message returned to admin):
  ```
  message ContinuationHint {
    int64  next_run_after_ns = 1;     // 0 = ASAP, >0 = earliest re-emit
    int64  expected_work_size = 2;    // pending+events_remaining
    string prefer_worker_id = 3;      // cache-locality hint
  }
  ```
  Lane scheduler honours `next_run_after_ns` as a per-`(bucket, ActionKey)` re-arm timer for DRAIN, per-`bucket` for BOOTSTRAP; biases routing by `prefer_worker_id`. Other lanes ignore the field.
- Handler registration in `weed/worker/tasks/s3lifecycle/`. `Register()` wired from `weed/worker/worker.go`.
- Detector in `weed/admin/maintenance/`: per `LaneLifecycle` tick, list buckets, walk each per-action `state` file at `/etc/s3/lifecycle/<bucket>/<rule_hash_hex>/<action_kind>/state`, emit `read` / `bootstrap` / `drain` tasks, respecting `next_run_after_ns` and the optional off-peak window (config knob `lifecycle.run_hours`, e.g. `"01-06"`). DRAIN scheduling is per `ActionKey`; sibling actions' next-run timers are independent.
- Per-`(bucket, ActionKey)` DRAIN singleton via cluster lock `s3.lifecycle.drain:<bucket>:<rule_hash_hex>:<action_kind>`. Detector skips emitting while a lock is held.
- Manual triggers via new shell commands:
  - `weed/shell/command_s3_lifecycle_run.go` → enqueue with `force=true`.
  - `weed/shell/command_s3_lifecycle_status.go` → print state.

Why a separate lane matters here:
- Long-running batches don't take the default-lane admin lock and therefore never block vacuum/balance/EC.
- Per-lane execution reservation means lifecycle can saturate its own slot pool without starving volume management.
- Operators can dedicate workers to lifecycle by labelling capability `s3.lifecycle.scan` only on those workers.

### Multi-filer durability

> **Implementation note (post-Phase 3):** the worker did not end up building its own multi-filer merge. `SubscribeMetadata` on the filer server (`weed/server/filer_grpc_server_sub_meta.go:154`) routes through `MetaAggregator` when peer filers exist: it reads from every filer's disk-persisted logs first (via `sendLogFileRefs`/`Filer.CollectLogFileRefs`), then drains the aggregator's in-memory tail. Workers get an aggregated, durable stream from a single subscription. The per-filer-shard cursor architecture below describes what would be needed if the worker had to assemble the merge itself; we kept it here as design context and as a reference for the proto stubs (`LOST_LOG`, `tail_drained_streams`) that remain forward-compatible if the architecture ever changes. The genuine residual gap — auto-degrading the worker on `ResumeFromDiskError` — is tracked separately as a narrow follow-up rather than a Phase.

`MetaAggregator` is **in-memory only** — `weed/filer/meta_aggregator.go:39`: *"MetaAggregator only aggregates data 'on the fly'. The logs are not re-persisted to disk. The old data comes from what each LocalMetadata persisted on disk."*

So the aggregator can't be the worker's source of truth: a worker restart, a crash, or a peer outage all lose unpersisted aggregated events. The persistent source is per-filer logs at `/topics/.system/log/<date>/<hour-min>.<filerId>` — written by each filer for its own events, durable, and queryable via `Filer.CollectLogFileRefs` and `Filer.ReadPersistedLogBuffer`.

Worker design for multi-filer:

- The persisted log lives in the **shared filer namespace**. Listing `/topics/.system/log/` returns chunk references for every filer's segments; reading is via `Filer.CollectLogFileRefs` and `Filer.ReadPersistedLogBuffer` against any filer endpoint. There is **no need to subscribe to each filer or run one task per filer.**
- Single cluster-singleton reader subscribed to one filer at a time. `Filer.CollectLogFileRefs` returns chunk refs in directory order; the worker heap-merges per-filer chunks client-side via `pb.ReadLogFileRefsWithPosition` (Task #19). Filer forwards metadata only — not a bottleneck.
- Cursors in `reader_state` (one writer):
  - `last_processed_original: map<delay_seconds, map<filer_id, MessagePosition>>` — per delay group, per per-filer log shard.
  - `last_processed_predicate: map<filer_id, MessagePosition>` — per per-filer shard.
  - `MessagePosition = {ts_ns, offset}` is the per-shard tie-breaker; `LogEntry.Offset` is per-buffer (per filerId), not globally unique, so cursor must track each shard.
- Cross-filer event ordering inside the heap is `(event.ts, filer_id, event.offset)`. `filer_id` is the secondary key because `event.offset` is per-filer-buffer and not globally unique; without `filer_id` in the order, two equal-ts events from different filers could swap relative positions across runs. The skip predicate stays per-shard: `(event.ts, event.offset) <= cursor[filer_id]` evaluated against that shard's own cursor.
- Filer-set discovery: list `/topics/.system/log/<date>/` and observe filerId suffixes that appear; new shards get lazy-seeded cursors via `seedNewlyDiscoveredShardsLazily`.
- **Departed-filer cursor GC and lost-log handling.** A filer is "tail-drained" for shard `F` at delay group `D` when there are **no retained log refs** for `F` with position `> reader_state.last_processed_original[D.seconds][F]` (and analogously for the predicate stream). `EarliestRetainedPositionPerShard()` is not sufficient on its own — knowing the earliest retained position doesn't prove there's nothing newer than the cursor. Phase 3 adds `RetainedLogRangePerShard()` returning `map<filer_id, {earliest, latest}>`. Tail-drained cursor entries (`cursor >= range.latest`) are removed at task entry, right after the probe, and the stream key is recorded in `reader_state.tail_drained_streams` so that future probes can distinguish "this exact stream was safely drained then pruned" from "F's logs were pruned before this stream caught up." The marker must be stream-specific: `(ORIGINAL, delay_seconds, filer_id)` or `(PREDICATE, filer_id)`. A drained 30-day stream does not make the 60-day stream safe. Without this GC, an old departed filer would pin the low-water-mark forever, forcing every batch to issue a `CollectLogFileRefs` listing starting from an ancient ts and inflating both listing cost and reader watermark-lag.

  Critically, the **absence** of a retained range for `F` is not sufficient evidence that GC is safe. If `F` has no retained range AND this stream cursor was never observed to reach `range.latest` (no entry in `tail_drained_streams`), the shard's logs were pruned before catch-up — events between `cursor[F]` and the (now-gone) `range.latest` were silently dropped. The reader **must not** GC such cursor entries: they're the only durable evidence that those events existed. Instead, the reader downgrades all reader-driven rules to `mode = scan_only` with `degraded_reason = LOST_LOG`, surfaces `lifecycle_lost_log_total{filer_id}` / `lifecycle_degraded_streams{reason="LOST_LOG"}`, and emits a warning. The cursor stays put until an operator runs `s3.lifecycle.reseed -ack-lost-log --reason <text>`, which clears the lost-log cursor entries and the degraded flag together after safety scans have covered the affected buckets — the operator must explicitly acknowledge the data loss before the system silently advances.

`LogFlushInterval` (the period before in-memory events flush to disk) bounds how stale the persistent view can be. Current constant: `time.Minute` (`weed/filer/filer.go:33`), so the flush gap is on the order of one minute. Workers process events with a `2 × LogFlushInterval` lag from real time (default ~2 minutes), so flushes always complete before the cursor reaches them.

The aggregated in-memory stream is still useful for the *bootstrap detector* (notifying the worker that a fresh policy was just PUT and a tick should run sooner than the next scheduled interval) — but it's a hint, never a substitute for the persisted log when computing watermarks or replaying history.

### Object lock and compliance — out of scope as a special case

Lifecycle does **not** model object lock state. Concretely:

- No tracking of legal-hold or retain-until-date in lifecycle storage.
- No per-rule `blocked` file (compliance ledger). No retain-until rescheduling. No re-evaluation triggered by Extended-change events that release a hold. (The cluster-level `_reader/blockers` file is unrelated — it tracks unprocessable cursor positions, not retained objects.)
- The `LifecycleDelete` server explicitly invokes `s3a.enforceObjectLockProtections` (in `weed/s3api/s3api_object_retention.go`) with `governanceBypassAllowed=false` before dispatching the low-level delete helper. If that check refuses, the server returns `SKIPPED_OBJECT_LOCK`. The worker logs, increments a counter, and advances.

What this means in practice:

- A legal-hold-released object will not be re-attempted on the original-write watermark schedule (the cursor has passed). It is re-attempted on the next periodic safety-scan bootstrap (per-kind cadence). For age-based rules that's `max(MinTriggerAge, 7d)` capped at 30d, which is acceptable for non-compliance lifecycle and irrelevant for compliance flows that should not be expecting lifecycle-driven cleanup anyway.
- A persistently-locked object surfaces as a sustained `skipped_object_lock_total` counter for that bucket/rule. Operators with compliance workloads handle removal of locks separately from lifecycle.

This is a deliberate scope cut. S3 object lock is a separate consistency surface; mixing its scheduling into lifecycle was producing complexity (legal-hold blocked file, retain-until rescheduling, re-eval on Extended change, K-retry guards, status surfacing) disproportionate to the value for typical workloads.

What remains:

- The S3 server's own object-lock enforcement is unchanged. Locked objects are not deleted.
- Lifecycle does not corrupt or interfere with object lock state.
- `skipped_object_lock_total` is exported as a Prometheus metric in Phase 7 so operators see when lifecycle and locks intersect.

If a future need arises for compliance-aware lifecycle (e.g., automated retain-until-expiry deletion), it can be a separate feature with its own data model — not bolted into the event-driven path.

### Blocked cursor handling — pause, don't dead-letter

For lifecycle, automatic dead-lettering is the wrong default. A dead letter would mean "this record is poison, skip it so the worker can keep moving" — but skipping a record means losing a delete decision (correctness). Instead, when an event cannot be processed, the worker **pauses that shard at its current `MessagePosition`** and surfaces a blocker for the operator to resolve.

Rules:

- **`RETRY_LATER`** (transient — `TRANSPORT_ERROR`, network, filer IO): the batch halts at the failing event. Cursor stays at the last resolved position. Next batch retries from the same `MessagePosition`. No `BlockerRecord` is written; the durable `retry_budget` entry for this stream key may be created or incremented (see "Sustained-RETRY_LATER promotion" below).
- **`BLOCKED`** (deterministic — `FATAL_EVENT_ERROR` after K=5 in-memory retries, or `RETRY_LATER` against the same `(stream_kind, shard, delay_seconds, position)` past the durable retry budget — see "Sustained-RETRY_LATER promotion" below): the worker writes a `BlockerRecord`, marks the affected stream paused at `failing_position`, and stops processing that stream until an operator intervenes. Other shards and other streams continue.

```
/etc/s3/lifecycle/_reader/blockers   (chunked, append-and-compact-by-stream)
  enum StreamKind { ORIGINAL = 0; PREDICATE = 1; BOOTSTRAP = 2; PENDING = 3; }

  repeated BlockerRecord {
    StreamKind stream_kind = 1;      // distinguishes the four blockable streams
    string shard           = 2;      // filer_id; populated when stream_kind == ORIGINAL or PREDICATE
    int64  delay_seconds   = 3;      // populated when stream_kind == ORIGINAL only
    MessagePosition position = 4;    // populated when stream_kind == ORIGINAL or PREDICATE

    // Common context. bucket/object_path/version_id are populated for all
    // streams. rule_hash is OPTIONAL: it is populated when the failure is
    // bound to a specific rule (the typical case). For pre-evaluation
    // failures — e.g. handleEvent's fetchLive FATAL where no rule has been
    // evaluated yet — rule_hash is left empty (nil/zero-length bytes).
    // Operators interpreting blockers should treat empty rule_hash as
    // "applies to whatever rule the next replay selects" rather than as a
    // missing field. See handleEvent fetchLive FATAL path for the one site
    // that produces this.
    bytes  rule_hash   = 5;     // optional; empty for pre-evaluation failures
    string bucket      = 6;
    string object_path = 7;
    string version_id  = 8;

    string reason       = 9;         // human-readable cause (e.g. "FATAL_EVENT_ERROR: malformed entry")
    string last_error   = 10;        // raw error string from the last attempt
    int64  first_seen_at_ns = 11;
    int64  last_retry_at_ns = 12;
    int32  retry_count      = 13;
  }
```

**StreamKind semantics:**

- `ORIGINAL`: the reader's per-shard cursor for `(shard, delay_seconds)` is paused at `position`. Subsequent ticks skip that `(shard, delay_seconds)` combination until the blocker is cleared. Other delay groups on the same shard, and the same delay group on other shards, continue.
- `PREDICATE`: the reader's per-shard predicate cursor for `shard` is paused at `position`. Other shards' predicate streams continue. Note that `delay_seconds` is not meaningful for the predicate stream — `BlockerRecord.delay_seconds` is only populated for `ORIGINAL`.
- `BOOTSTRAP`: the bucket bootstrap walker pauses; the bucket's `_bootstrap.last_scanned_path` is the resume point. The walker writes a blocker for the failing `(bucket, object_path)` and refuses to advance until cleared. Other buckets continue.
- `PENDING`: the rule's drain task pauses on the failing `(rule_hash, path, version_id)`. Other rules' drains continue.

**Operator commands:**

- `weed shell s3.lifecycle.blockers list [--stream ...] [--shard ...] [--bucket ...]` — show all blockers.
- `weed shell s3.lifecycle.blockers retry <id>` — re-attempt the failing event/path **right now**, in the shell command itself, bypassing the task-start guard. The shell synchronously re-runs the appropriate primitive (`handleEvent` for `ORIGINAL`/`PREDICATE`, the walker step for `BOOTSTRAP`, `drainPending` step for `PENDING`) against current live state. Per-stream retry-success effects:
  - `ORIGINAL`: on `DONE`/`NOOP_RESOLVED`, remove blocker record AND advance `reader_state.last_processed_original[delay_seconds][shard]` past `position`.
  - `PREDICATE`: on `DONE`/`NOOP_RESOLVED`, remove blocker record AND advance `reader_state.last_processed_predicate[shard]` past `position`.
  - `BOOTSTRAP`: on `DONE`/`NOOP_RESOLVED` for the recorded `(bucket, object_path, version_id, rule_hash)`, remove blocker record. **Do not** advance `_bootstrap.last_scanned_path` directly — the next scheduled bootstrap task picks up from `last_scanned_path` (which already sits at the previous successful entry) and re-walks normally; if the entry is still due it will be re-attempted, if not it'll be skipped. This avoids the operator command racing with concurrent walker progress.
  - `PENDING`: on `DONE`/`NOOP_RESOLVED`, remove blocker record AND tombstone the matching `PendingItem` in the rule's `pending` file (the live retry might have already deleted/marked the object). On `NOOP_RESOLVED` due to `STALE_IDENTITY`, also tombstone the pending entry — it referred to a stale identity.
  - All streams, on failure (`RETRY_LATER` or `BLOCKED` again): update `retry_count` and `last_retry_at_ns` on the blocker record; leave it in place; do not advance any cursor or tombstone any pending. The next scheduled task remains gated by the task-start guard until the blocker clears.

  Idempotency: the live re-attempt uses the same `expected_identity` / `MessagePosition` CAS path as a normal worker invocation, so a concurrent successful delete by something else just produces `NOOP_RESOLVED` (`STALE_IDENTITY` / `NOT_FOUND`) and the blocker resolves cleanly.
- `weed shell s3.lifecycle.blockers resume <id>` — clear the blocker record AND its corresponding `retry_budget` entry, but **leave the cursor at the failing position**. The next reader/drain/bootstrap pass will encounter the event and re-process it normally with a fresh retry-budget counter. Use this when the operator has fixed the underlying cause (corrupted entry repaired, upstream bug patched, etc.) and wants the worker to attempt the same record on its own schedule rather than synchronously via `retry`. Resume does *not* advance past the event — that's `quarantine`.
- `weed shell s3.lifecycle.blockers quarantine <id> --reason <text>` — manual decision to **skip the event and advance the cursor without processing it**. Clears the blocker AND its `retry_budget` entry. Logged with operator identity, reason, and a `lifecycle_quarantined_total` metric increment. This is the **only** path that intentionally drops a delete decision; only the operator can take it; the action is auditable.

In all three resolution paths (`retry` success, `resume`, `quarantine`), the corresponding `retry_budget` entry is cleared along with the `BlockerRecord`. Stale retry-budget state is never left behind.

The cursor advances **only** in two ways: a worker-initiated `DONE`/`NOOP_RESOLVED` outcome, or an operator-initiated `quarantine`. `resume` is "clear the diagnostic, let the worker retry on its own" — it never moves the cursor.

**No automatic skip.** The worker does not advance past a `BLOCKED` event on its own. Watermark lag for that `(shard, stream)` accumulates and triggers the lag-correctness fallback (degraded → scan_only) at the usual thresholds. Persistent operator inattention falls back to safety scans, which is the correctness floor.

**Sustained-RETRY_LATER promotion.** A `RETRY_LATER` outcome (`TRANSPORT_ERROR`, network blip, filer IO error) by itself doesn't write a `BlockerRecord` — the cursor stays and the next batch retries. But if the same logical retry target keeps producing `RETRY_LATER` across many batches, the underlying issue is no longer "transient." The stream would silently lag forever without operator attention.

A small durable counter file at `/etc/s3/lifecycle/_reader/retry_budget` tracks consecutive retries per stream-specific key (the same key shape used by `BlockerRecord` for that stream):

```
// RetryTarget carries the routing/identification fields needed to (a) write a
// BlockerRecord on promotion and (b) point the operator's `blockers retry`
// at the right handler.
//
// We deliberately do NOT carry expected_identity or action. Retry semantics
// are: re-invoke the handler implied by stream_kind. The handler re-fetches
// live state and re-evaluates from scratch:
//   ORIGINAL  / PREDICATE  → re-run handleEvent at (shard, position) — fetches
//                            live, evaluates each candidate rule against current
//                            state, attempts deletes with current-identity CAS.
//   BOOTSTRAP              → re-run walker step at (bucket, object_path) —
//                            fetches the entry, evaluates rule, attempts delete.
//   PENDING                → re-run drainPending step for the matching
//                            PendingItem (which itself carries expected_identity
//                            in the pending file).
// If the object changed between the original failure and the retry, CAS returns
// STALE_IDENTITY → NOOP_RESOLVED and the blocker resolves cleanly. Capturing
// the original action would just be a stale hint.
//
// This also means RetryTarget works uniformly for failure paths that AREN'T
// delete RPCs — fetchLive ERR, pending upsert/demotion failure — because all
// of them are handled by the same per-stream handler on retry.
message RetryTarget {
  StreamKey key                  = 1;        // oneof, matches BlockerRecord stream-specific compaction keys

  // Common context (populated for every stream).
  string    bucket               = 2;
  string    object_path          = 3;
  string    version_id           = 4;
  bytes     rule_hash             = 5;       // for BOOTSTRAP/PENDING this is the rule the walker/drain
                                              // was working under. For ORIGINAL/PREDICATE the failure
                                              // can span multiple candidate rules; rule_hash is the
                                              // one whose evaluation triggered the RETRY_LATER, but
                                              // retry re-evaluates ALL current candidates against
                                              // current state, so a stale rule_hash here doesn't
                                              // mislead replay.
}

message RetryBudget {
  RetryTarget target              = 1;
  int32       consecutive_retries = 2;
  int64       first_retry_at_ns   = 3;
  int64       last_retry_at_ns    = 4;
}

message StreamKey {
  StreamKind stream_kind = 1;
  oneof key {
    OriginalKey  original  = 2;
    PredicateKey predicate = 3;
    BootstrapKey bootstrap = 4;
    PendingKey   pending   = 5;
  }
}

message OriginalKey  { string shard = 1; int64 delay_seconds = 2; MessagePosition position = 3; }
message PredicateKey { string shard = 1; MessagePosition position = 2; }
message BootstrapKey { string bucket = 1; string object_path = 2; string version_id = 3; bytes rule_hash = 4; }
message PendingKey   { string bucket = 1; bytes  rule_hash   = 2; string object_path = 3; string version_id = 4; }
```

Note on `BootstrapKey`/`PendingKey`: these duplicate `bucket`/`object_path`/`version_id`/`rule_hash` from `RetryTarget`'s common context. The duplication is intentional — `StreamKey` is the **lookup/dedup key** for blockers and retry-budget entries (so two callers can find the same record); `RetryTarget`'s common-context fields are populated identically and used for the BlockerRecord/retry payload. For `ORIGINAL`/`PREDICATE`, the common context fields *are* the only place bucket/object info exists (the `StreamKey` carries shard/position only).

Helper functions used by reader/bootstrap/drain:

```
recordRetryLater(target RetryTarget):
    rb := load_or_create(target.key)
    rb.target = target                                    // refresh; the failing target should be stable
    rb.consecutive_retries += 1
    if rb.first_retry_at_ns == 0: rb.first_retry_at_ns = now()
    rb.last_retry_at_ns = now()

    if rb.consecutive_retries < retryBudgetMax and
       now() - rb.first_retry_at_ns < retryBudgetMaxAge:
        save retry_budget
        return RETRY_LATER

    // Threshold tripped — promote to BLOCKED. The blocker must commit durably
    // BEFORE we return BLOCKED, otherwise the caller would pause the stream
    // with no operator-visible record. If the write fails, leave the
    // retry_budget entry in place and return RETRY_LATER; the next batch will
    // re-attempt the promotion.
    ok := writeBlocker(BlockerRecord{
        stream_kind:      target.key.stream_kind,
        ...key fields from target.key...,
        rule_hash:        target.rule_hash,
        bucket:           target.bucket,
        object_path:      target.object_path,
        version_id:       target.version_id,
        reason:           "sustained RETRY_LATER",
        last_error:       "promoted from retry_budget",
        first_seen_at_ns: rb.first_retry_at_ns,
        last_retry_at_ns: rb.last_retry_at_ns,
        retry_count:      rb.consecutive_retries,
    })
    if !ok:
        save retry_budget                                 // keep the budget; try promotion again next batch
        return RETRY_LATER
    delete retry_budget entry for target.key
    return BLOCKED                                        // caller treats as BLOCKED

clearRetryBudget(streamKey):
    delete retry_budget entry for streamKey if present    // success → reset budget
```

Each caller passes a fully-populated `RetryTarget`. The reader path builds it **inside `handleEvent`** (per failing branch — fetch_live FATAL/ERR, deleteAndResolve outcome) so the retry target reflects which branch actually failed; the reader callback itself doesn't construct `RetryTarget`. Bootstrap builds it directly from the entry being walked; pending drain builds it directly from `item`. RetryTarget intentionally carries no `action` or `expected_identity`: operator-driven retry (and shell-side `blockers retry`) replays the appropriate primitive against current live state, which re-fetches and re-evaluates from scratch.

Defaults: `retryBudgetMax = 30` (≈30 minutes at 1-batch-per-minute reader cadence), `retryBudgetMaxAge = 4h`, whichever comes first. The retry_budget file is bounded by the count of positions *actively retrying* (typically 0 or a small handful) and compacts on success.

This gives sustained-transient → BLOCKED a concrete mechanism without a per-event durable record on every `RETRY_LATER`.

The blocker file's compaction key is **stream-specific**, since the identifying tuple differs per stream:

- `ORIGINAL`: `(stream_kind, shard, delay_seconds, position)` — `position` is required because the same shard can have multiple blockers at different `MessagePosition`s within the same delay group (sequential failures the operator hasn't cleared yet); without `position` they'd collapse.
- `PREDICATE`: `(stream_kind, shard, position)` — same reasoning; `delay_seconds` not used.
- `BOOTSTRAP`: `(stream_kind, bucket, object_path, version_id, rule_hash)` — without `version_id` and `rule_hash`, two different rules' walks failing on the same `(bucket, object_path)` would collapse into one record and the operator couldn't distinguish them.
- `PENDING`: `(stream_kind, bucket, rule_hash, object_path, version_id)` — different versions of the same key under the same rule are distinct pending items; without `version_id` they'd collapse.

A re-blocking of the same key overwrites the previous record (preserves `first_seen_at_ns`, refreshes `last_retry_at_ns`, increments `retry_count`) rather than appending a new one. Resolution (via `retry` success, `resume`, or `quarantine`) removes the record by its full key.

### Failure / restart

- The meta log is the durable queue. Watermarks advance only past events that returned `DONE` or `NOOP_RESOLVED`.
- On `RETRY_LATER` (transient — S3 unreachable, network, filer IO): the batch stops at that event; the per-shard cursor stays at the last resolved `(filer_id → MessagePosition)`; the next batch retries the same event from the per-shard skip predicate.
- On `BLOCKED` (deterministic — `FATAL_EVENT_ERROR` after K=5 retries, or sustained `RETRY_LATER` past a per-shard threshold): the worker writes a `BlockerRecord` and pauses that `(shard, delay_seconds)` combination. Other shards and other delay groups continue. The cursor stays at `failing_position` until an operator runs `s3.lifecycle.blockers retry|resume|quarantine`.
- On bootstrap `RETRY_LATER`/`BLOCKED`: the walker stops; `last_scanned_path` stays at the previous successful entry; a `BlockerRecord` is written for the `BLOCKED` case.
- On pending drain `RETRY_LATER`/`BLOCKED`: the pending item stays in the rule's `pending` file; for `BLOCKED` a `BlockerRecord` is written and the rule's drain pauses.
- Mid-tick crash → next tick re-runs from last persisted state.
- Idempotent throughout: identity CAS in the server prevents double-delete; pending upsert keyed by `(path, version_id)` within each rule's directory.

For permanently-failing events: see "Blocked cursor handling" above. The worker never auto-skips a failing event; it pauses and surfaces. Operator decides resume / retry / quarantine.

### Scalability — pending is for exceptions, not normal scheduling

The meta log + per-filer watermarks **are the scheduler** for age-based rules. Per-object pending is reserved for cases the watermark can't express:

| Reason for pending | Bounded by |
|---|---|
| Late predicate change creates new eligibility on a not-yet-due object | rate of late-arriving tag/metadata changes that newly match a rule |

What does **not** go into pending in normal operation:

- **Future age-based deletes.** Discovered by the reader when their original-write event ages past the cutoff. The event log is the durable schedule.
- **Currently-due objects from bootstrap.** Bootstrap walks and deletes them inline; the walk cursor (`last_scanned_path`) is the resume point.
- **Date-rule matches.** `SCAN_AT_DATE` mode runs a single bootstrap at `rule.date`; no pending.

**Steady-state pending size** in a normally-operating cluster: typically zero or low single-digit items per rule, populated only by late predicate-change exceptions (a tag added long after the original PUT that newly matches a rule, where `live.mtime + TTL` is still in the future). Independent of bucket size and write rate. Object-lock skips do *not* go to pending — they're logged-and-counted, with a periodic safety scan handling re-evaluation if the lock is later released.

**S3 unavailable**: reader/bootstrap stop on `TRANSPORT_ERROR`, watermark stalls at the last resolved event, no pending growth. Next read task retries from the same watermark. Events are durable in the meta log for the configured retention. Operators see an alerting signal via watermark-lag metrics.

**High write rate** (e.g. 10k/s log ingest with age=1d): the reader processes events as their `now - MinTriggerAge` cutoff sweeps past them. Each event is one synchronous `LifecycleDelete`. Throughput is bounded by `LifecycleDelete` parallelism (configurable via `lifecycle.delete.concurrency`, default 16 concurrent calls). Chunk reads from the merged log stream parallelize across volume servers (`lifecycle.reader.chunk_concurrency`, default 8). Pending stays empty in normal flow.

**Delete throughput is coupled to reader progress.** Cursor advance is gated on per-event resolution; if `LifecycleDelete` throughput is slower than the rate at which events age into the original-write cutoff, the watermark lags behind real time. If the lag grows past `metaLogRetention`, events would be pruned from the log before they can be processed — without a fallback, silent under-deletion.

Operators have three levers in order of cost:

1. **Raise `lifecycle.delete.concurrency`** to fan out more in-flight `LifecycleDelete` RPCs. Effective until the S3 server pool is the bottleneck.
2. **Raise `lifecycle.reader.chunk_concurrency`** to fan out more in-flight chunk reads from volume servers. Effective until network or volume servers are the bottleneck.
3. **Scale S3 servers / network**.

If those levers don't suffice, see "Future scaling" below.

**Future scaling (non-v1, no committed phase).** If single-worker reader throughput is insufficient even after raising delete concurrency, chunk-read concurrency, and S3 server capacity, the design admits one extension: time-segment leases on the merged log stream. The merged stream's time range `[low_water, cutoff]` is partitioned into N time segments; each segment leased to a separate worker process; each worker processes its segment independently and reports `last_processed_position_per_shard` on completion; a coordinator gates **contiguous watermark commit** — a segment's results only commit (and its watermark advance only persists) once *every earlier segment has committed*. Per-bucket pending writes still use the same `(path, version_id)` dedup key, so cross-segment race on a path is idempotent. This stays worker-side (no per-filer fanout, no filer-side changes); the filer's role is unchanged. Concrete design and implementation are deferred — there is no committed phase for it. Operators hitting this scale should bring it up; until then v1 ships with one reader.

**Correctness fallback when the lag approaches retention.** Alerting alone is not enough — by the time an operator scales capacity, events may already be pruned. The reader monitors **two parallel lag signals**:

1. **Per-delay-group original-write lag** — `original_lag_seconds[delay_seconds] = now - low_water(reader_state.last_processed_original[delay_seconds])`. Affects every rule that uses that delay.
2. **Per-shard predicate lag** — `predicate_lag_seconds[filer_id] = now - reader_state.last_processed_predicate[filer_id].ts_ns`, with the engine-wide aggregate `predicate_lag_seconds = max over filer_id`. Affects every **predicate-sensitive** rule (any rule with tag/size filters), regardless of its delay group. Without this check, predicate-stream blockage would let tag/metadata changes age out of the meta log without any rule entering `scan_only`.

Per-stream thresholds:

| Lag (per stream) | Action |
|---|---|
| `< metaLogRetention × 0.5` | Healthy. Clear `degraded_reason = LAG_HIGH` if previously set. |
| `≥ × 0.5` and `< × 0.8` | Emit `LIFECYCLE_LAG_WARNING` metric + log; surface in `s3.lifecycle.status`. Set `degraded_reason = LAG_HIGH` if not already. |
| `≥ × 0.8` | Affected rules keep `mode = event_driven` AND `degraded_reason = LAG_HIGH`; force the next safety-scan bootstrap to run *immediately* without waiting for the per-kind cadence. Continue event-driven processing in parallel. |
| `≥ × 0.95` | Promote affected rules to `mode = scan_only` durably (set `degraded_reason = LAG_HIGH`, `degraded_since_ns = now`). The reader stops processing the affected stream — the affected delay group, or all predicate-sensitive rules' predicate ingestion, respectively. The safety scan is the only enforcement until the operator explicitly re-enables event-driven mode. |

Affected-rules scope:

- Original-write lag at threshold for delay `D` → rules whose `delay == D`.
- Predicate lag at threshold → all rules with `predicateSensitive == true` (scoped per bucket — promotion to `scan_only` is per-rule, but the trigger is engine-wide).

**Re-enable** after a predicate-stream-induced `scan_only` uses the existing `force_reseed_predicate` path. The `s3.lifecycle.reseed` shell command takes `-predicate` to flag predicate-stream-driven re-enables: it sets `force_reseed_predicate=true` on the seed call, rewinding `last_processed_predicate` to `T_start - bootstrapLookbackMin` per filer, then flips affected rules back to `event_driven`.

**Re-enabling event-driven mode after a `scan_only` fallback** is a delay-group operation, not a per-rule state flip. The cursor `last_processed_original[D.seconds]` is shared across **all rules with delay D** (e.g., two rules with 60-day expiration but different prefixes both consume the 60d cursor's stream). Reseeding for rule A's sake forces rules B, C, … on the same delay D to replay the reseed window too. That replay is idempotent (CAS on identity, pending upserts collapse) and the cost is bounded by the lookback window's events, but it must be acknowledged.

Operator workflow:

1. `weed shell s3.lifecycle.reseed -bucket X -delay <days>` — operates on the delay group, not a single rule. Lists all rules currently in `scan_only` mode for that delay and reports which buckets/rules will replay.
2. The command:
   - Computes `force_reseed_delays = { D }` for the delay being re-enabled. (For multiple delays in one invocation, the set has multiple entries.)
   - Deletes `reader_state.last_processed_original[D.seconds]` (the entire per-filer map for that delay group).
   - Runs a fresh bucket-level bootstrap for **every bucket whose policy includes a rule with delay D** (idempotent, in parallel). Each bootstrap, on completion, calls **`seedReaderCursorsForNewDelayGroups(snapshot, newly_completed, target_modes, T_start, force_reseed_delays={D}, force_reseed_predicate=<true if any affected rule is predicateSensitive else false>)`** — the explicit force arguments are required because the affected rules already have `bootstrap_complete=true` and therefore are NOT in `newly_completed`; without them the seeder's early-exit would leave both the deleted delay cursor un-seeded AND the predicate cursor un-rewound (silently missing tag/metadata changes that happened during the scan_only window for predicate-sensitive rules). The seeder seeds each `D` in `force_reseed_delays` at `T_start - max(D, bootstrapLookbackMin)` per filer with the before-first sentinel; if `force_reseed_predicate=true` it also rewinds the predicate cursor to `T_start - bootstrapLookbackMin` per filer.
   - After the seeder commits, flips affected rules' `mode` from `scan_only` back to `event_driven` durably and clears `degraded_reason`.
3. The reader resumes processing the delay group; rules that were not in scan_only also re-receive the lookback window of events, idempotently. CAS on identity ensures no duplicate deletes; pending upserts collapse on `(path, version_id)`.

Lost-log acknowledgement is a separate, explicit workflow because it admits that some retained metadata history was lost:

1. `weed shell s3.lifecycle.reseed -ack-lost-log --reason <text>` — cluster-wide, only allowed while some rules have `degraded_reason = LOST_LOG`.
2. The command verifies an immediate safety-scan bootstrap has completed for every bucket with reader-driven lifecycle rules after `degraded_since_ns`; if not, it runs those bootstraps first and waits for completion.
3. It deletes the lost-log cursor entries for vanished shard streams, clears any matching `tail_drained_streams` markers, then force-reseeds all reader delay groups and the predicate cursor using the same `force_reseed_delays` / `force_reseed_predicate` path above.
4. After seeding commits, it clears `LOST_LOG`; rules return to `event_driven` only if the normal retention gate still passes. Rules whose `metaLogRetention < eventLogHorizon + bootstrapLookbackMin` remain `scan_only` with `RETENTION_BELOW_HORIZON`.

If only one rule is to be reactivated and the operator wants to avoid replay for sibling rules, the alternative is to *split* the rule's delay into a unique value (e.g., `60d` → `61d`), which gives it its own delay-group cursor at the cost of changing AWS-visible semantics. Generally, delay-group-wide replay is acceptable.

Without this protocol, naive per-rule re-enable would either replay too much (cursor at the old position before the lag spike) or skip newly-eligible events (cursor naively bumped to "now"). The reseed pins the cursor exactly at the safe replay floor.

This guarantees no silent loss: by the time events would be pruned, the affected rules are already running on the bucket-walk fallback, with explicit signal in `status`. `degraded` and `scan_only` are surfaced via:

- `lifecycle_rule_mode{rule_hash, mode}` — `event_driven | scan_at_date | scan_only | disabled`.
- `lifecycle_rule_degraded{rule_hash, reason}` — `NONE | LAG_HIGH | PENDING_FULL | DELETE_FAILURES | OPERATOR_PAUSED`.
- `watermark_lag_seconds{delay_group}` — primary indicator.
- `lifecycle_delete_concurrency_saturation` — fraction of the concurrent-call budget in use.
- `lifecycle_delete_p99_latency` — server-side delete latency.

We deliberately do **not** introduce a durable delete-handoff queue. That would decouple cursor advance from delete completion at the cost of bringing back per-object pending growth and the scalability problems it causes. The simpler answer — auto-fallback to scan_only when lag threatens retention — is what this design commits to.

**Bootstrap of a stale bucket** (lifecycle freshly enabled on a 60k-object bucket where all are currently due): bootstrap walks pages and deletes inline. No pending growth. The walk pauses on `TRANSPORT_ERROR` and resumes from `last_scanned_path`. Total wall time = `bucket_size / parallel_delete_throughput`.

**Pending file format.** Append-only with tombstones; in-memory `(path, version_id) → offset` index for upsert lookups. Compaction triggered when tombstone ratio exceeds 50%. Because pending is bounded by exceptions, the file rarely grows beyond a few MB and compaction is cheap.

**Backpressure cap on pending** (default 50k items per rule): triggered only by an unusual workload — typically a sudden burst of late predicate-change events on a long-TTL rule. The predicate-change stream is **shared across all rules** (one `last_processed_predicate: map<filer_id, MessagePosition>` per shard, advanced by every predicate event the engine routes to *any* rule's candidates). We cannot selectively halt it for a single rule without halting predicate-change processing for all rules. The mechanism is therefore:

1. On the first pending write for the affected rule that would exceed the cap (or any pending write that fails), the reader treats the rule as **degraded** and **atomically switches it to `scan_only` mode** (durable per-rule state write). The current event is then "resolved" for that rule — the next safety-scan bootstrap will re-evaluate any objects whose predicates changed late.
2. Other rules continue their predicate-change processing normally; the global predicate cursor keeps advancing.
3. The reader emits `PENDING_FULL{rule_hash}` (metric + log) so operators see why the rule degraded.
4. The same mechanism handles any unrecoverable `pending` write error (filer IO failure, etc.): attempt the durable demotion to `scan_only`. The cursor only advances after that demotion write *commits*. If the demotion write itself also fails (filer IO continues to be unhealthy), `handleEvent` returns `RETRY_LATER`; the batch halts with the per-shard cursor at the previous event, and the next read task retries. Advance is gated on durable degraded-state persistence, never on alerting alone.

This preserves correctness without coupling one rule's exception load to the global predicate stream. Demoting a rule to `scan_only` is reversible — operators can manually return it to `event_driven` after the load subsides.

Object-lock skips never enter pending; they're logged-and-counted only. Pending is exclusively the late-predicate-change exception path.

**Health signals.** Watermark lag (per delay group AND per shard for predicate), pending size, S3 RPC error rate, `skipped_object_lock_total`, `blocked_streams{stream_kind}` (gauge of active blocker records). Sustained watermark lag is the primary indicator. Active blockers are the operator-attention indicator.

### Observability

CLI:
- `weed shell s3.lifecycle.status -bucket X` — mode, watermark, pending size, last tick, counts.

Headers:
- `X-SeaweedFS-Lifecycle-LastTick` on `GetBucketLifecycleConfiguration`.

Prometheus:
- `seaweedfs_s3_lifecycle_evaluated_total{bucket,rule_hash}`
- `seaweedfs_s3_lifecycle_expired_total{bucket,rule_hash,action}`
- `seaweedfs_s3_lifecycle_metadata_only_total{bucket,rule_hash}` — count of skip-chunk-delete deletes
- `seaweedfs_s3_lifecycle_skipped_object_lock_total{bucket,rule_hash}` — locked objects encountered
- `seaweedfs_s3_lifecycle_blocked_streams{stream_kind,bucket}` — gauge: number of active blocker records by stream_kind (ORIGINAL / PREDICATE / BOOTSTRAP / PENDING)
- `seaweedfs_s3_lifecycle_quarantined_total{stream_kind,bucket}` — counter: operator-quarantined records (auditable; only path that intentionally drops a delete decision)
- `seaweedfs_s3_lifecycle_predicate_lag_seconds{filer_id}` — primary indicator for predicate-stream health
- `seaweedfs_s3_lifecycle_errors_total{bucket,rule_hash,kind}`
- `seaweedfs_s3_lifecycle_pending_size{bucket,rule_hash}`
- `seaweedfs_s3_lifecycle_watermark_lag_seconds{bucket,rule_hash,delay_group}`

Admin UI (the existing plugin lane filter at `weed/admin/view/app/plugin_lane.templ` + `weed/admin/dash/plugin_api.go:filterActivitiesByLane` already surfaces job type `s3_lifecycle` — confirmed by `dash/plugin_api_test.go:204-230`):

1. **Per-bucket lifecycle panel on `s3_buckets.templ`.** Per rule:
   - rule_id (or hash prefix if blank)
   - mode: `event_driven` | `scan_only`
   - bootstrap state: `complete` | `% scanned`
   - watermark lag (`now - last_processed_ts`)
   - pending size
   - counters: evaluated / expired / errors / metadata-only
   - actions: `Run now` (force batch), `Re-bootstrap`, `Pause`

2. **Lifecycle lane summary on `plugin_lane.templ`** when `?lane=lifecycle`:
   - active batches
   - cluster-wide pending sum
   - watermark lag p50/p99
   - off-peak window indicator if `lifecycle.run_hours` is set

3. **JSON API endpoints under `weed/admin/dash/`:**
   - `GET  /api/s3/buckets/{name}/lifecycle/status` — decoded per-rule `state` files
   - `POST /api/s3/buckets/{name}/lifecycle/run` — enqueue forced batch (optional `?rule_hash=`)
   - `POST /api/s3/buckets/{name}/lifecycle/rebootstrap` — reset bootstrap for a rule

4. **PUT XML editor (if/when it lands)**: warn on rules whose TTL exceeds meta-log retention with "Will run in scan_only mode at slower cadence."

### PUT handler becomes (post-Phase 4 below)

1. Validate XML.
2. Persist `lifecycle.xml` xattr.
3. Reconcile per-rule directories under `/etc/s3/lifecycle/<bucket>/`: create new dirs with `bootstrap_complete=false`; preserve `bootstrap_complete=true` and counters for unchanged rule hashes; mark removed rules' dirs for grace-period cleanup. (Reader watermarks live in cluster `reader_state`, not per-rule; unchanged.)
4. Wake the scan scheduler.
5. Return 200.

Constant time, no bucket walk, no `filer.conf` mutation.

### Migration

- **Cannot strip the back-stamp until the worker can take over.** Phasing reflects this.
- Existing entries with `TtlSec` stamps from the old back-stamp keep working via the read-time filer check. Worker plus those stamps converge.
- First worker tick on each existing bucket runs the bootstrap scan once.
- `DeleteBucketLifecycle` clears xattrs; existing `TtlSec` stamps stay (don't aggressively unstamp — would resurrect objects).

## Comparison

| | Today | Proposed |
|---|---|---|
| `PUT` latency | O(N entries) | O(1) |
| Steady-state cost per tick | n/a (no scanner) | O(events × distinct_delay_groups) + pending drain (exceptions only) + amortized periodic safety-scan walk per bucket per safety cadence |
| Bootstrap cost | Hidden in `PUT` | Once per bucket per snapshot, in background; re-runs on safety cadence |
| Tag/size filters | Silent no-op | Live-evaluated; late-tag-add uses pending exception |
| Date filters | Silent no-op | `SCAN_AT_DATE` mode: scheduled bucket-level bootstrap at `rule.date` (no per-object pending) |
| Versioned buckets | Silent no-op | Driven by `.versions` events |
| `AbortIncompleteMultipartUpload` | Silent no-op | Triggered on `.uploads/` events |
| Cold-object handling | Never | Bounded by tick + due-time |
| Source of truth | `filer.conf` + xattr | xattr only |
| Failure recovery | None | Watermark + pending + CAS |

---

# Dev Plan

Phasing changed: **the back-stamp cannot be removed until the worker can take over.** The first behavior-changing PR is the worker, not the PUT slim-down.

## Phase 0 — Confirm assumptions (research, no code)

- Verify `Filer.ReadPersistedLogBuffer` returns events including `Extended` on `NewEntry`.
- Document retention policy of `/topics/.system/log/`. If pruned, find the knob and document it; this sets the threshold for `scan_only` mode.
- Confirm the exact filename scheme inside `<key>.versions/` (sortable timestamp prefix? base32 of ts? affects how we determine "successor" version).
- Capture findings inline in this design doc.

## Phase 1 — `s3lifecycle.Evaluate` (no callers)

- Add `EvaluateAction(rule *Rule, kind ActionKind, info *ObjectInfo, now time.Time) EvalResult`, `ComputeDueAt(rule *Rule, kind ActionKind, info *ObjectInfo) time.Time`, `MinTriggerAge(rule *Rule, kind ActionKind) time.Duration`, `EventLogHorizon(rule *Rule, kind ActionKind) time.Duration`, and `RuleActionKinds(rule *Rule) []ActionKind` in `weed/s3api/s3lifecycle/`. All kind-aware: a single XML rule expands into N compiled actions and each helper operates on one (rule, kind) pair so sibling actions are scheduled, gated, and evaluated independently. `EventLogHorizon` returns `rule.Days` / `rule.NoncurrentDays` / `rule.DaysAfterInitiation` for age-based kinds, and `smallDelay` for `NEWER_NONCURRENT` and `EXPIRED_DELETE_MARKER`. `EXPIRATION_DATE` is not a reader-driven kind and the gate skips it.
- Cover: `ExpirationDays`, `ExpirationDate`, `ExpiredObjectDeleteMarker`, `NoncurrentVersionExpirationDays`, `NewerNoncurrentVersions`, `AbortMPUDaysAfterInitiation`, prefix + tag + size filters, And-combinations.
- Add `RuleHash(rule *Rule) [8]byte` over a length-prefixed canonical form (sorted tags, prefix verbatim, every action's parameters, days/date). Stable across XML reordering, ID renames, and Status flips; resistant to delimiter forgery.
- Unit tests in `evaluate_test.go` per rule shape, including AWS edge cases (date in future, zero days, empty prefix, And).
- No callers yet. **Independently mergeable.**

## Phase 2 — LifecyclePolicyEngine + bucket-level bootstrap + LifecycleDelete RPC + scan_only gate (non-versioned)

- Define protos in `weed/pb/s3_lifecycle.proto`: per-action `LifecycleState` keyed by `(rule_hash, action_kind)` (no watermarks), `PendingItem` (late-predicate exceptions only — no `BlockedItem`, no compliance ledger), `EntryIdentity`. Cluster-level `ReaderState` (watermark maps + `tail_drained_streams`). Top-level `ActionKind` enum referenced by `LifecycleState`, `BootstrapKey`, `PendingKey`, `BlockerRecord`, `RetryTarget`.
- Define `S3LifecycleParams` (subtype `READ` | `BOOTSTRAP` | `DRAIN`) and `ContinuationHint` in `weed/pb/worker_pb/worker.proto`. `S3LifecycleParams.action_kind` is required for DRAIN, ignored by BOOTSTRAP.
- Storage:
  - Per-action: `/etc/s3/lifecycle/<bucket>/<rule_hash_hex>/<action_kind>/{state, pending}`. Each XML rule expands into N action subdirectories under one rule_hash dir.
  - Per-bucket bootstrap progress: `/etc/s3/lifecycle/<bucket>/_bootstrap`.
  - Cluster reader: `/etc/s3/lifecycle/_reader/{reader_state, blockers, retry_budget}`. Single reader task; `reader_state` holds per-filer-shard cursors (`map<delay_seconds, map<filer_id, MessagePosition>>` for originals and `map<filer_id, MessagePosition>` for predicate) — required for resume correctness across the client-side merge. `blockers` is the durable record of paused stream entries (operator-resolvable). `retry_budget` tracks consecutive `RETRY_LATER` per stream key for promotion to BLOCKED at threshold. No reader-group partitioning, no assignment epochs.
- New package `weed/s3api/s3lifecycle/engine/` — `LifecyclePolicyEngine`. Compiles all bucket lifecycle xml into:
  - `bucket → BucketIndex { prefixTrie, tagIndex, versioned }` keyed by `ActionKey`.
  - `originalDelayGroups: map[duration][]ActionKey` for original-write sweeps.
  - `predicateActions: []ActionKey`, `dateActions: map[ActionKey]time.Time`.
  - Snapshot ID; rebuild on observed lifecycle xattr change events.
- Per-`ActionKey` policy CAS via `(rule_hash, action_kind)` membership — no per-bucket etag scheme.
- New package `weed/s3api/s3lifecycle/worker/`:
  - **Bucket-level** bootstrap walker: walks each bucket once per run, evaluates every applicable `ActionKey` per object via the engine (`EvaluateAction(rule, kind, info, now)` for each compiled action). Inline delete for currently-due age actions; skip not-yet-due (log replay handles); skip date actions (SCAN_AT_DATE handles). `last_scanned_path` checkpointed in `_bootstrap`. Per-action completion writes commit *after* the walk.
  - On `TRANSPORT_ERROR` the walker stops; resume from `last_scanned_path` next task.
  - Skip versioned buckets with a logged warning. Implementation lands in Phase 5.
- **Mode gate (lands here):** at engine compile time, for each `ActionKey`, compute `eventLogHorizon(rule, action_kind)` per the table in §"Per-rule mode decision"; if `metaLogRetention < eventLogHorizon(rule, action_kind) + bootstrapLookbackMin`, mark *that action* `scan_only` with `degraded_reason = RETENTION_BELOW_HORIZON`. Applies to all reader-driven kinds (`EXPIRATION_DAYS`, `NONCURRENT_DAYS`, `ABORT_MPU`, `NEWER_NONCURRENT`, `EXPIRED_DELETE_MARKER`); date kind bypasses this gate. Sibling actions of the same rule are evaluated independently — a 90d `EXPIRATION_DAYS` may degrade while a 7d `ABORT_MPU` stays event-driven. Same bootstrap code path; detector emits at the per-kind cadence.
- `SCAN_AT_DATE` mode: detector schedules a single bootstrap at the action's `rule.date`.
- Worker-to-S3 discovery via admin's `ListS3Endpoints` RPC; cache 30s; rotate on RPC failure.
- New gRPC `LifecycleDelete` (`weed/s3api/s3api_internal_lifecycle.go`). Server handler steps 1–7 above; the request carries `(rule_hash, action_kind)` for ActionKey CAS.
- Object lock: `LifecycleDelete` server explicitly calls `s3a.enforceObjectLockProtections` (in `weed/s3api/s3api_object_retention.go`) with `governanceBypassAllowed=false` BEFORE dispatching low-level helpers. If the check refuses, returns `SKIPPED_OBJECT_LOCK`. Worker logs, increments counter, advances. No retain-until rescheduling, no blocked file.
- Admin server: register S3 endpoints; serve `ListS3Endpoints`.
- Wire task types in `weed/worker/tasks/s3lifecycle/`:
  - `s3.lifecycle.bootstrap` per-bucket (not per-action).
  - `s3.lifecycle.read` cluster-singleton (Phase 3 fills in).
  - `s3.lifecycle.drain` per-`ActionKey` for exception drains.
  All bound to `LaneLifecycle`.
- Cluster locks: `s3.lifecycle.read` (singleton), `s3.lifecycle.bootstrap:<bucket>`, `s3.lifecycle.drain:<bucket>:<rule_hash_hex>:<action_kind>`, `s3.lifecycle._reader.seeding` (global, serializes seed writes across bucket bootstraps).
- Detector in `weed/admin/maintenance/`.
- Shell commands `s3.lifecycle.run`, `s3.lifecycle.status`.
- Worker runs alongside the back-stamp.
- Tests: bootstrap fixture bucket with multiple rules — verify one walk evaluates all; not-yet-due not enqueued; date rules deferred to SCAN_AT_DATE; transport error pauses then resumes; scan_only gate triggers on short retention.
- **Regression tests**:
  - `target_modes` computed before seeding: inject a fixture where the seeder is called with rules whose `state.mode` field is at proto zero value; assert the seeder filters using the precomputed `target_modes` map (not the unwritten state field), so EVENT_DRIVEN rules trigger rewind and SCAN_AT_DATE/SCAN_ONLY/DISABLED rules do not. Without this, the seeder's filter would silently fail.
  - `force_reseed_delays` for scan_only re-enable: start with a rule in `mode = scan_only` and `bootstrap_complete = true` (so it never enters `newly_completed`); delete `reader_state.last_processed_original[D.seconds]`; call the seeder with `newly_completed = {}` and `force_reseed_delays = {D}`; assert the cursor is re-seeded at `T_start - max(D, bootstrapLookbackMin)`. Then assert the same call WITHOUT `force_reseed_delays` does **not** re-seed (verifies the early-exit path is correct for routine safety scans).
  - `force_reseed_predicate` for predicate-sensitive scan_only re-enable: start with a predicate-sensitive rule (e.g. `Filter.Tag` filter) in `mode = scan_only`; advance the predicate cursor in `reader_state.last_processed_predicate` past `T_start - bootstrapLookbackMin` to simulate normal reader progress while the rule was scanned-only; emit a tag-add meta event in that interval (would be predicate-relevant if the rule were active); flip the rule via the reseed workflow with `force_reseed_predicate = true`; assert the predicate cursor was rewound to `T_start - bootstrapLookbackMin` per shard; assert the previously-emitted tag-add event is now picked up by the next reader pass and routed through `handleEvent` for the re-enabled rule. Without `force_reseed_predicate` the test must show the cursor un-rewound and the tag-add event silently missed — this is the regression that motivates the parameter.
  - Bootstrap fatal-error blocker: inject a rule with a corrupted entry that causes `LifecycleDelete` to return `FATAL_EVENT_ERROR` deterministically. Verify bootstrap walker calls `deleteAndResolve` (not `lifecycleDeleteRPC` directly), retries K=5 times, writes a durable `BlockerRecord` with `stream_kind = BOOTSTRAP`, returns `BLOCKED`, and `last_scanned_path` is left at the previous successful entry (NOT advanced past the failure). Verify the next bootstrap task refuses to advance past the blocker until the operator clears it.
  - Reader fatal-error blocker: similar, with `stream_kind = ORIGINAL` and `(shard, delay_seconds, position)` populated for the original-write pass; with `stream_kind = PREDICATE` and `(shard, position)` for the predicate pass. The blocked `(shard, stream)` pauses; other shards/streams continue processing in the SAME batch — the BLOCKED outcome adds to in-memory blocked sets and the callback returns `PauseShard(filer_id)`, which removes that shard from the merge heap so subsequent buffered events from it are not delivered (verify via spy on the merge engine). The next reader task loads `_reader/blockers` and uses `MaxMessagePosition` for blocked shards' `start_positions` so no events are re-read until the blocker is cleared.
  - Pending fatal-error blocker: similar, with `stream_kind = PENDING`; the rule's drain pauses; the pending item stays.
  - `writeBlocker` failure path: if the durable blocker append fails, `deleteAndResolve` (and the inline FATAL fetchLive path in `handleEvent`) returns `RETRY_LATER` instead of `BLOCKED`. Verify the cursor stays at the failing position and the next batch retries.
  - Operator commands:
    - `blockers list` filters by `--stream`, `--shard`, `--bucket`.
    - `blockers retry <id>` re-attempts synchronously, with stream-specific success effects:
      - `ORIGINAL`: on `DONE`/`NOOP_RESOLVED` removes the record AND advances `reader_state.last_processed_original[delay_seconds][shard]` past the recorded `position`.
      - `PREDICATE`: on `DONE`/`NOOP_RESOLVED` removes the record AND advances `reader_state.last_processed_predicate[shard]` past the recorded `position`.
      - `BOOTSTRAP`: on `DONE`/`NOOP_RESOLVED` removes the record only; **does not** mutate `_bootstrap.last_scanned_path`. The next scheduled bootstrap task picks up from `last_scanned_path` and re-walks normally.
      - `PENDING`: on `DONE`/`NOOP_RESOLVED` removes the record AND tombstones the matching `PendingItem` in the rule's `pending` file. `STALE_IDENTITY` outcomes also tombstone (the pending entry referred to a stale identity).
      - All streams: on `RETRY_LATER`/`BLOCKED` again, update `retry_count` and `last_retry_at_ns`, leave the blocker, leave any cursors / pending items unchanged.
    - `blockers resume <id>` clears the blocker AND its retry_budget entry but **leaves the cursor at the failing position**; the next reader/drain/bootstrap pass re-processes the event normally with a fresh retry-budget counter. Resume does **not** advance the cursor.
    - `blockers quarantine <id> --reason <text>` is the **only** path that advances the cursor without processing — auditable, increments `lifecycle_quarantined_total`. Clears the blocker AND its retry_budget entry.
  - retry_budget cleanup: any direct-FATAL blocker write (fetchLive FATAL in handleEvent, K-exhausted FATAL_EVENT_ERROR in deleteAndResolve, fetchLive FATAL in drainPending) calls `clearRetryBudget(stream_key)` after the durable blocker write commits — verify no stale retry_budget entries remain after a sequence like ERR → ERR → FATAL.
  - Operator resolution paths all clear retry_budget alongside the blocker record (verify via post-resolution inspection of `_reader/retry_budget`).
  - Predicate-stream lag fallback: simulate predicate-cursor lag past `metaLogRetention × 0.95` for some shard while original-write streams remain healthy; verify all predicate-sensitive rules in affected buckets are durably promoted to `mode = scan_only` with `degraded_reason = LAG_HIGH`; verify rules without tag/size filters are unaffected. Verify `s3.lifecycle.reseed -predicate` re-enables them via the `force_reseed_predicate` path.
  - Sustained `RETRY_LATER` (transient repeated past a threshold) eventually upgrades to `BLOCKED` so the shard doesn't silently lag forever; verify the upgrade fires and the lag-correctness fallback then promotes the rule to scan_only.

## Phase 3 — Shared cluster reader + delay-group sweeps (non-versioned)

> **Implementation note:** the shipped reader uses `client.SubscribeMetadata(...)` with a single global `(ts_ns, offset)` cursor (`weed/s3api/s3lifecycle/reader/reader.go`). The filer's `SubscribeMetadata` server already aggregates persisted + in-memory logs across peer filers, so the per-filer-shard cursor map and the `RetainedLogRangePerShard` / `EarliestRetainedPositionPerShard` filer RPCs (Task #19) were not built. The tail-drain matrix and lost-log tests below were written against that unbuilt architecture and are obsoleted; the only residual gap — auto-degrading on `ResumeFromDiskError` — is tracked as a separate narrow follow-up. The retention mode-gate, engine routing, predicate-change pass, SKIPPED_OBJECT_LOCK, TRANSPORT_ERROR, and cursor-trail tests remain valid and were exercised against the production reader/router/engine stack.

- Implement `s3.lifecycle.read` task: cluster singleton, subscribed to one filer at a time. ~~`Filer.CollectLogFileRefs` returns chunk refs in directory order; the worker uses `pb.ReadLogFileRefsWithPosition` (Task #19) to heap-merge per-filer chunks by event ts client-side, with per-shard `MessagePosition` skip filtering.~~ The shipped task uses a direct `SubscribeMetadata` subscription; the filer server-side does the multi-filer merge. Cursor is one global `(ts_ns, offset)` position resumed via `SinceNs`. Failover to another filer endpoint on connection error; cursor is portable (one ts). Periodic checkpointing bounds crash redo.
- ~~**Tail-drain tests required in this phase**~~ — obsoleted by the single-cursor architecture; see Phase 6 for context.
- **Retention mode-gate tests** (per `eventLogHorizon` table, kinds beyond age-based): inject a rule of each reader-driven kind and toggle `metaLogRetention` above and below `eventLogHorizon(rule) + bootstrapLookbackMin`; verify the rule is `event_driven` above and `scan_only` with `degraded_reason = RETENTION_BELOW_HORIZON` below. Cover `EXPIRATION_DAYS`, `NONCURRENT_DAYS`, `ABORT_MPU`, `NEWER_NONCURRENT`, `EXPIRED_DELETE_MARKER`. Verify `EXPIRATION_DATE` bypasses the gate regardless of retention.
- Engine drives event routing: `engine.MatchOriginalWrite(event, delay=D)` and `engine.MatchPredicateChange(event)`.
- **One sweep per delay group** (e.g. `1d`, `30d`, `60d`, `90d`); each sweep advances `reader_state.last_processed_original[D.seconds]` (a single global `(ts_ns, offset)` `MessagePosition`). Single predicate-change sweep clamped to `min(now - smallDelay, now - flushSafetyLag)` advances `reader_state.last_processed_predicate` (also a single global position).
- One filer fetch per event (`fetchLiveEntryOnce`), reused across all candidate rules.
- Per-event resolution-or-stop: `Resolved` outcomes advance the cursor; `TRANSPORT_ERROR` halts the batch and the cursor stays.
- Engine refreshes when the reader observes a lifecycle-xattr `UpdateEntry` event.
- `s3.lifecycle.drain` task drains per-rule pending exceptions.
- Tests with fixture meta logs:
  - 100 buckets × 5 rules — engine routes events with O(matched-rules) per-event candidate lookup. Verify log I/O scales with `events × distinct_delay_groups` (a small handful of passes), not `events × rule_count` (500x).
  - Original-write events sweep per delay group; rules with same delay share a sweep.
  - Predicate-change pass uses one near-now cutoff for all rules.
  - Engine rebuild on lifecycle PUT observed in log.
  - Late tag-add → predicate pass picks up promptly; rare exception path writes pending.
  - SCAN_AT_DATE rule never adds to pending until rule.date.
  - Rule edit creates new `<new_hash>` dir; unrelated rule's pending unaffected.
  - `SKIPPED_OBJECT_LOCK` → counter increment, log, advance (no pending, no blocked).
  - `TRANSPORT_ERROR` → batch stops; reader resumes from same cursor on next task.
  - Original cursor trails `now - delay`; predicate cursor stays at `now - smallDelay`.

## Phase 4 — Strip `updateEntriesTTL` from `PutBucketLifecycleConfiguration`

- **Now safe**: worker has been the source of truth for at least one release.
- Remove `updateEntriesTTL` and `filer.conf` mutation from `PutBucketLifecycleConfigurationHandler`.
- Reconcile-on-PUT: preserve `bootstrap_complete=true` and counters for unchanged `rule_hash`; new hashes start with `bootstrap_complete=false` (engine_state=pending_bootstrap). Reader cursors live in cluster `reader_state` and are unaffected.
- Regression test: PUT on a 10k-entry bucket completes under 1s.
- Old `TtlSec`-stamped entries continue to work via the filer read-time check; worker takes over going forward.

## Phase 5 — Versioning + MPU

- Implement `ruleAppliesToEntryShape` for `.versions/` paths and the current-pointer directory entry.
- Discover successor's becoming-non-current time from `event.TsNs` (or by listing the `.versions` dir for the next-newer version's mtime if event came from the dir attr update).
- `ExpiredObjectDeleteMarker` triggered when a delete-marker is the sole survivor.
- `AbortIncompleteMultipartUpload` driven by events under `.uploads/`.
- Tests covering: new version PUT, delete marker creation, non-current expiration, expired-delete-marker, MPU abort.

## Phase 6 — Multi-filer durable replay (obsoleted)

**Status: not built. Architecture moved on; the production reader uses the filer's existing aggregation surface.**

What the original Phase 6 set out to add — per-filer-shard cursor maps, `EarliestRetainedPositionPerShard` and `RetainedLogRangePerShard` filer RPCs, `tail_drained_streams` GC, `CollectLogFileRefs` heap-merge, lost-log degraded promotion — was designed under the assumption that the lifecycle worker would have to assemble its own multi-filer view because `MetaAggregator` is in-memory only.

`SubscribeMetadata` already does that work server-side: when the filer has peer filers it reads disk-persisted logs from every filer first (via `sendLogFileRefs` → `Filer.CollectLogFileRefs`) and only then drains the in-memory aggregator. A single subscription returns a durable, multi-filer-merged stream. The Phase 3 shipped reader (`weed/s3api/s3lifecycle/reader/reader.go`) consumes that stream with one global `(ts_ns, offset)` cursor and never needed the per-shard cursor map.

What the design's per-shard architecture would have bought us, and why each piece is no longer required:

| Phase 6 mechanism | Why it's no longer required |
|---|---|
| Per-filer-shard cursor map | Single global cursor suffices because the filer presents one merged stream. Resume-replay overhead is bounded by `LogFlushInterval` (~1m) and the dispatcher's tick interval (~200ms), so the worst-case redundant work per batch is one minute of events that the cursor compare immediately skips. |
| `RetainedLogRangePerShard` / `EarliestRetainedPositionPerShard` filer RPCs | No per-shard cursor → no need for per-shard probe. `SubscribeMetadata` already returns `ResumeFromDiskError` when the cursor predates retained logs. |
| Tail-drain GC + `tail_drained_streams` | No per-shard cursor → no entries to GC. Departed filers' events are absorbed into the aggregated stream and the global cursor advances past them. |
| Lazy seeding `seedNewlyDiscoveredShardsLazily` | No per-shard cursor → no shards to seed. New filers' events appear automatically through the aggregator. |

Forward-compat stubs that stay in the proto: `LifecycleState.DegradedReason.LOST_LOG`, `ReaderState.tail_drained_streams`, `last_processed_original` keyed by `filer_id`. They cost nothing and keep the on-disk state file readable if a future deployment ever needs a per-shard architecture.

The single residual gap — and the only piece worth building — is auto-degrading the reader when `SubscribeMetadata` returns `ResumeFromDiskError`. Tracked as a separate narrow follow-up (catch the error in `reader.Run`, mark reader-driven actions `scan_only` with `degraded_reason = LOST_LOG`, increment the counter, advance the cursor to retained-earliest, expose `s3.lifecycle.reseed -ack-lost-log` to flip back to event_driven once safety scans cover the gap). It is a ~50-line change, not a Phase.

## Phase 7 — Observability

- Shell `s3.lifecycle.status` command.
- Admin UI tile.
- `X-SeaweedFS-Lifecycle-LastTick` header.
- Prometheus metrics.

## Phase 8 — `scan_only` cadence tuning + operator hooks

(The gate itself lands in Phase 2; this phase is observability and tuning.)

- Surface `scan_only` mode in shell `s3.lifecycle.status` and admin UI.
- Configurable cadence per rule via shell command (operator override of the per-kind safety-scan cadence default).
- Document in bucket-lifecycle docs: when retention < TTL, the rule runs as periodic scan; trade-offs explained.
- Manual "scan now" via shell command for unscheduled runs.

## Cut points

- Phase 0–2 alone delivers, on **non-versioned** buckets only:
  - Currently-due deletions for age-based rules via bucket-level bootstrap that calls `LifecycleDelete` inline (no pending/drain in this path).
  - Date-rule deletions (`SCAN_AT_DATE` mode: a single scheduled bucket-level bootstrap at `rule.date` deletes all matches inline).
  - Periodic safety scans (re-run bootstrap on the per-kind cadence) catch drift over time.
  Not-yet-due age-based discovery via meta-log events arrives in Phase 3 (the reader). Versioned buckets remain unhandled until Phase 5; the worker skips them with a logged warning until then. The existing back-stamp keeps simple `Expiration.Days` working in parallel.
- Phase 3 brings steady-state event-driven expiration. The reader is a cluster singleton subscribed to one filer via `SubscribeMetadata`, which the filer's `MetaAggregator` already merges across peer filers. Single global `(ts_ns, offset)` cursor; multi-filer clusters work the same as single-filer clusters from this point forward.
- Phase 4 makes `PUT` constant-time. Required dependency: Phase 2 + Phase 3 shipped and stable.
- Phases 5–8 complete the surface.

---

# Test Plan

Four layers, mapped to existing repo patterns. Each layer has a different cost/scope. Test work lands in the same PR as the code it covers; CI gates are by layer.

## Layer 1 — Unit (pure Go, milliseconds)

Location: `weed/s3api/s3lifecycle/*_test.go` next to source. Standard table-driven Go tests. No fakes, no goroutines.

| File | Coverage |
|---|---|
| `evaluate_test.go` | `EvaluateAction(rule, kind, info, now) → action`: every action kind × (eligible, not-eligible) × every filter shape (prefix, tag, size, And-combinations). AWS edge cases: zero days, future date, empty prefix, empty/duplicate Rule.ID. Multi-action rule: each kind independently. |
| `rule_hash_test.go` | Semantically-equivalent rules hash equal. Reordered tags hash equal. Different rules hash different. ID changes don't affect hash. |
| `due_at_test.go` | `computeDueAt(rule, info)` per kind. Boundary: mtime exactly at threshold, date in past, NewerNoncurrent at exactly N versions. |
| `min_trigger_age_test.go` | `MinTriggerAge` per kind. `EXPIRATION_DATE` returns sentinel; `NEWER_NONCURRENT` returns small. |
| `event_log_horizon_test.go` | `EventLogHorizon` per kind. Age kinds return their day count; `NEWER_NONCURRENT` and `EXPIRED_DELETE_MARKER` return `smallDelay`; `EXPIRATION_DATE` is not called and panics or returns sentinel. |
| `identity_test.go` | `EntryIdentity` builder produces stable hash for same Extended; differs when tags change. Snapshot-restore (different head fid, same mtime) detected. |

Run: `go test ./weed/s3api/s3lifecycle/...`. Phase 1 lands these alongside the function.

## Layer 2 — Component (in-process harness, seconds–minutes)

Location: `test/plugin_workers/lifecycle/` — new package, mirrors `test/plugin_workers/vacuum/` and `test/plugin_workers/volume_balance/`.

Reuses `test/plugin_workers/framework.go` (in-process plugin admin gRPC + worker). Extends it with:

1. **Meta-log fixture loader** — takes a Go slice of `{filerId, tsNs, eventType, oldEntry, newEntry}`, writes into a test filer's `/topics/.system/log/` in the on-disk format. Used by all reader/watermark tests.
2. **Controllable clock** — `s3lifecycle.Now` wired to a test func; advance without sleeping.
3. **Fake S3 LifecycleDelete server** — records calls, returns scripted outcomes, so component tests assert exact RPC sequences.

Files and the cases each owns:

```
test/plugin_workers/lifecycle/
  framework.go                  # harness extensions
  detection_test.go             # detector emits the right tasks given state
  bootstrap_test.go             # Phase 2
  reader_test.go                # Phase 3
  drain_test.go                 # Phase 3
  versioning_test.go            # Phase 5
  multi_filer_test.go           # Phase 6
  metadata_only_test.go         # Phase 2b
  s3_discovery_test.go          # Phase 2
```

Cases:

**Bootstrap** — inline-delete for age rules (currently-due deleted, not-yet-due skipped), `SCAN_AT_DATE` for date rules (date-triggered bucket walk), kill-resume via `last_scanned_path`, walk-cursor drift covered by past-`MinTriggerAge` cursor-seed, scan_only gate, per-rule `bootstrap_complete` set on completion before engine activation.

**Reader / event-driven** — original-write threshold, late tag-add via predicate pass, two watermarks advance independently, cross-filer heap-merge, Extended-equal events filtered, lifecycle-XML xattr update reloads trie.

**Drain / `LifecycleDelete`** — identity CAS reject, per-rule policy CAS (unrelated edit doesn't poison), `SKIPPED_OBJECT_LOCK` → log + counter + advance (no pending, no blocked).

**Metadata-only delete** — `TtlSec > 0` + matching `filer.conf` rule → server invoked with `skip_chunk_delete=true`; intercept fake-filer chunk-deletion queue to assert no enqueue.

**Worker–S3 discovery** — all endpoints unhealthy → continuation-hint backoff, no progress, no data loss; flapping endpoint rotates; S3 down at startup → worker boots, drains catch up on recovery.

Run: `go test ./test/plugin_workers/lifecycle/...`. Phase mapping: each phase ships its half.

## Layer 3 — Real-server integration (AWS SDK + live `weed server`, minutes per scenario)

Location: `test/s3/lifecycle/` — new, copy of `test/s3/retention/` shape (Makefile-driven binary build, server start/stop, AWS SDK clients).

```
test/s3/lifecycle/
  Makefile                          # mirrors test/s3/retention/Makefile
  test_config.json
  s3_lifecycle_basic_test.go              # PUT/GET/DELETE
  s3_lifecycle_put_perf_test.go           # 60k-object PUT < 1s — original ticket regression
  s3_lifecycle_age_test.go                # ExpirationDays end-to-end
  s3_lifecycle_date_test.go               # ExpirationDate end-to-end
  s3_lifecycle_tag_filter_test.go         # tag/size filters
  s3_lifecycle_late_tag_test.go           # tag added at age 30d on 60d rule expires at AWS-correct time
  s3_lifecycle_versioning_test.go         # versioned bucket flows
  s3_lifecycle_mpu_test.go                # AbortIncompleteMultipartUpload
  s3_lifecycle_object_lock_test.go        # retain-until + legal hold interactions
  s3_lifecycle_metadata_only_test.go      # volume-TTL fast path
  s3_lifecycle_scan_only_test.go          # short-retention forces scan_only
```

Time compression: existing build tag `lifecycle_10sec` (`weed/util/constants_lifecycle_interval_10sec.go`) treats 1 day as 10 seconds. Verify it covers `MinTriggerAge` math; add a `bootstrapLookbackMin` constant in the same files so it scales too. Makefile target:

```
test-fast: build-weed-10sec
	go test -tags lifecycle_10sec -timeout 15m ./...
```

Run: `make -C test/s3/lifecycle test-with-server`. Phase mapping: end-to-end tests land in the same phase as the capability they cover.

## Layer 4 — Multi-filer convergence (obsoleted)

**Not built.** Phase 6 was obsoleted by `SubscribeMetadata`'s server-side aggregation (see the Phase 6 section). With a single global cursor on a server-merged stream, there are no per-filer-shard convergence properties for the worker to test — any merge correctness lives inside the filer's `MetaAggregator`, which has its own coverage in `weed/filer/`.

If `ResumeFromDiskError` auto-degrade lands as a follow-up (task #21), its tests live with the reader package, not in a separate multi-filer harness.

## CI gating

| Layer | When | Time |
|---|---|---|
| `make test-unit` | every push | < 10s |
| `make test-component` | every push | < 2m |
| `make test-s3-lifecycle` (Layer 3, 10sec build tag) | PRs touching `weed/s3api/s3lifecycle/**`, `weed/worker/tasks/s3lifecycle/**`, or `weed/s3api/s3api_internal_lifecycle.go` | ~20m |
| `make test-multi-filer-lifecycle` | nightly | ~30m |

CI must require `test-component` and `test-s3-lifecycle` green before merging Phase 4 (the PUT-strip). All layers green required for Phase 5+.
