2 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	05d31a04b6	fix(s3tests): wire lifecycle worker for expiration suite (#9374) ... * fix(s3tests): wire lifecycle worker for expiration suite The upstream s3-tests `test_lifecycle_expiration` / `test_lifecyclev2_expiration` exercise the "set rule, wait, verify deletion" path. Phase 4 (#9367) intentionally stripped the PUT-time back-stamp, so pre-existing objects no longer pick up TtlSec on a freshly-applied rule. The s3tests CI bare-bones `weed -s3` had nothing left driving expiration. Three changes that work together: - Engine scales `Days` by `util.LifeCycleInterval`. Production keeps the 24h day; the `s3tests` build tag shrinks it to 10s so a `Days: 1` rule completes inside the suite's 30s polling window. Exported `DaysToDuration` so sibling-package tests pin to the same scale. - Scheduler/dispatcher tick defaults split into `_default` / `_s3tests` files. Production stays 5s/30s/5m; the test build runs at 500ms/2s/2s so deletions land within a couple ticks of becoming due. - s3tests.yml spawns `weed shell s3.lifecycle.run-shard -shards 0-15 -events 0 -runtime 1800s` alongside the s3 server in both the basic and SQL blocks; the shell command runs the full pipeline (reader + scheduler + dispatcher) for the duration of the suite. `test_lifecycle_expiration_versioning_enabled` is left out for now — versioned-bucket expiration via the worker still needs its own pass. Drive-by: bump `TestWorkerDefaultJobTypes` to 7 to match the registered handler count (8b87ceb0d updated `mini_plugin_test.go` for the s3_lifecycle plugin but missed this twin test). Two retention-gate engine tests `t.Skip` under the s3tests build because they rely on absolute lookback-vs-retention math the day-rescale collapses; the prod build still covers them. * review: harden lifecycle worker spawn + assert handler identity - Workflow: aliveness check on the backgrounded `weed shell` (a bad command exits in <1s and the suite would otherwise just opaque-timeout); move worker/server teardown into a `trap cleanup EXIT` so failure paths still print the worker log and reap the data dir. - worker_test: check the actual job-type set by name, not just the count. * fix(shell): keep s3.lifecycle.run-shard alive when no rules exist yet The s3-tests CI runs the worker BEFORE any test creates a bucket, so LoadCompileInputs returns empty and the shell command was bailing out with "no buckets with enabled lifecycle rules found" within ~1s. The aliveness check then fired exit 1 before tox ever started. Two changes: - Don't early-exit on empty inputs. Compile against the empty set, log a one-liner, and let the pipeline run normally — the meta-log subscription is already up, so events for buckets created later DO arrive; they just need the engine to know about them when they do. - Add `-refresh <duration>` (default 5m, 2s in s3tests CI) that periodically re-runs LoadCompileInputs + engine.Compile so rules added after startup land in the snapshot the dispatcher reads on its next tick. Production deployments keep the 5m default; only the CI workflow drops to 2s. Workflow passes `-refresh 2s` in both basic and SQL blocks. * fix(shell): backfill pre-rule entries via bootstrap walker The reader-driven path only sees meta-log events created AFTER its engine snapshot knows the rule. The s3-tests CI scenario PUTs objects first, then PUTs the lifecycle config, so by the time the engine refresh picks up the new bucket the object events have already been seen-and-dropped (BucketActionKeys returned empty for the bucket). Wire bootstrap.Walk into the shell command: - bucketBootstrapper tracks buckets seen so far. kickOffNew spawns one loop goroutine per fresh bucket. - Each goroutine re-walks the bucket every walkInterval (defaults to the same value as -refresh, i.e. 2s in s3tests CI, 5m in prod) and feeds each entry through bootstrap.Walk; due actions dispatch via a direct LifecycleDelete RPC. Not-yet-due entries are silently skipped and picked up on a later iteration once they age past their (rescaled or real) threshold. - LifecycleDelete is called with no expected_identity; the server-side identityMatches treats nil as "skip CAS", which is the right call for bootstrap (the bootstrap entry doesn't carry chunk fid / extended hash anyway). The dispatcher's pkg-private toProtoActionKind is duplicated in the shell file rather than exported, since the shape is six lines and the reverse import would pull a proto dep into the s3lifecycle root. * refactor(s3/lifecycle): hoist bucket bootstrapper into scheduler pkg The shell command got the backfill in the previous commit but the worker plugin (weed/worker/tasks/s3_lifecycle/handler.go) drives Scheduler.Run directly and missed it — same root cause: the reader-driven path only sees events created after the rule lands, so a daily cron picking up a freshly-PUT rule wouldn't expire any pre-rule object. Move the looping bucket walker into scheduler.BucketBootstrapper: - Scheduler.Run now constructs one and calls KickOffNew on every engine refresh. Per-bucket goroutines re-walk every BootstrapWalkInterval (defaults to RefreshInterval — 5m in prod, 2s under s3tests). - The shell command consumes the same struct instead of its own copy so the two paths can't drift in semantics. * refactor(s3/lifecycle): walk-once + schedule via event injection Previous per-bucket walker re-listed every WalkInterval forever. For a bucket with N objects under a long rule, the worker did O(N * runtime / walkInterval) listings even when nothing was newly due — way too much for production-scale buckets. New approach: walk each bucket exactly once on first sight, synthesize one *reader.Event per existing entry, push it onto Pipeline.events. Router.Route builds a Match with DueTime=mtime+delay; future-due matches sit in the per-shard Schedule and fire when their DueTime arrives. Currently-due matches fire on the very next dispatch tick. Wiring: - dispatcher.Pipeline lifts its events channel into a struct field with sync.Once init, and exposes InjectEvent(ctx, ev). Reader no longer closes the channel — the dispatch goroutine exits on runCtx cancellation, which works the same as channel-close did. - scheduler.BucketBootstrapper drops the WalkInterval ticker. KickOffNew spawns one walker goroutine per fresh bucket; the goroutine lists, synthesizes events, then exits. - scheduler.Scheduler builds its pipelines up front and exposes a pipelineFanout (shard -> Pipeline) as the EventInjector, so a multi- worker scheduler routes each synthesized event to the pipeline that owns its shard. - Shell command's single-pipeline path passes pipeline.InjectEvent directly. Synthesized events carry TsNs=0; dispatcher.advance treats that as a no-op so the reader's persisted cursor isn't ratcheted past unprocessed meta-log events. Identity (HeadFid + ExtendedHash) is still computed from the real filer entry, so the server's identity-CAS catches an overwrite between bootstrap and dispatch. * debug(s3tests): make lifecycle worker progress visible in CI logs The previous CI failure dumped an empty $LC_LOG even though the worker was running. Two reasons: 1. weed shell suppresses glog by default (logtostderr / alsologtostderr set to false). Pass `-debug` so the bootstrapper's V(0) lines reach stderr instead of disappearing into /tmp/weed.*.log. 2. cleanup used `kill -9` which skips Go's stdout flush. SIGTERM first with a 1s grace, then SIGKILL the holdout, then read the log. While here: bump the bootstrap walker's two informational logs to V(0) so the diagnosis from CI doesn't require -v=1 on the worker. * fix(s3/lifecycle/dispatcher): refresh snap on every event Pipeline.Run captured snap at startup and only refreshed it on the dispatch tick. With bootstrap event injection, the walker pushes events seconds after engine.Compile sees the bucket — typically WITHIN the same dispatch interval. Routing against the cached (empty) snap then silently dropped every match because BucketActionKeys returned nil for the bucket-not-yet-in-snapshot case. Re-fetch on each event. Engine.Snapshot is an atomic.Pointer.Load, so the cost is negligible. The dispatch-tick branch keeps using a fresh local read for its own loop, so its semantics are unchanged.	2026-05-08 17:29:47 -07:00
Chris Lu	4e10669221	docs(s3/lifecycle): event-driven redesign (#9346) ... * docs(s3/lifecycle): event-driven redesign Replaces the synchronous PUT-handler walk with an event-driven worker model: meta-log reader subscribed to one filer, client-side heap merge with per-filer-shard MessagePosition cursors, bucket-level bootstrap with inline delete, blocked-cursor handling for fatal events, durable retry budget for sustained-transient promotion, retention mode gate that downgrades reader-driven rules to scan_only when log retention falls below the rule's event-log horizon. * docs(s3/lifecycle): record Phase 0 verified assumptions ReadPersistedLogBuffer payload carries Extended (event marshaled via SubscribeMetadataResponse → ToProtoEntry). Meta-log files at topics/.system/log/<date>/<HH-MM>.<filerId> are written without TtlSec in filer_notify_append.go; retention is unbounded by default and only shrinks if an operator sets a filer.conf rule with a TTL on the SystemLogDir prefix. .versions/ filenames are v_<16h-ts><16h-rand> with old/new (inverted) format distinguished by threshold 0x4000000000000000; getVersionTimestamp / compareVersionIds give format-agnostic ordering for successor-version discovery. * feat(s3/lifecycle): rule evaluator and dueAt helper Evaluate(rule, info, now) returns the EvalResult by object shape: IsMPUInit -> AbortMultipartUpload, IsDeleteMarker -> ExpireDeleteMarker when sole survivor, IsLatest -> DeleteObject (Days or Date), non-current -> DeleteVersion (Days fallback to ModTime when SuccessorModTime is zero; NewerNoncurrent retention enforced when both are set). ComputeDueAt mirrors Evaluate's shape and returns the earliest eligible wall-clock time for the same (rule, info), used by the reader/bootstrap to decide pending vs inline-delete. Adds StatusEnabled/Disabled and SmallDelay consts and an IsMPUInit flag on ObjectInfo so .uploads/<id>/ entries route to AbortMPU without overloading IsLatest. No callers; package compiles standalone. * feat(s3/lifecycle): MinTriggerAge for safety-scan cadence Returns the smallest non-zero day threshold across the rule's actions. Used as max(MinTriggerAge, kindFloor) for the per-kind cadence; date, count-only, and delete-marker-only rules return 0 so callers fall through to their kind-specific floor. * feat(s3/lifecycle): EventLogHorizon for retention mode gate Returns the maximum event age the reader needs for a rule. Days-based kinds return their day threshold; pure NewerNoncurrent (count) and ExpiredObjectDeleteMarker return SmallDelay. Date rules return 0 (the gate skips them). Multi-action rules take the max — strictest horizon wins. Drives the Phase 2 mode gate: metaLogRetention < EventLogHorizon(rule) + bootstrapLookbackMin -> scan_only with RETENTION_BELOW_HORIZON. * feat(s3/lifecycle): RuleHash for per-rule state CAS sha256 over canonicalized form, first 8 bytes. Stable across tag-key reorder, prefix trailing-slash variation, ID renames, and Status flips (state continuity is preserved when an operator toggles Enabled/Disabled). Different action shapes — different days, filter, or action type — hash differently. Used by the per-rule state directory layout /etc/s3/lifecycle/<bucket>/<rule_hash_hex>/ and by the bootstrap detector's reconcile-on-PUT. * feat(s3/lifecycle): add s3_lifecycle.proto storage schema Defines the durable types backing the lifecycle worker: LifecycleState (per-rule mode + bootstrap_complete + degraded_reason incl. RETENTION_BELOW_HORIZON / LOST_LOG), PendingItem, EntryIdentity, BootstrapState, ReaderState (per-filer-shard cursors plus tail_drained_streams marker), BlockerRecord (rule_hash optional for pre-evaluation failures), RetryBudgetEntry with the four-shape StreamKey oneof, and RetryTarget (no action / no expected_identity — retry replays handler against current state). No callers; schema only. Wired into the pb Makefile. * feat(s3/lifecycle): add S3LifecycleParams to TaskParams Wires lifecycle subroutines into the existing worker dispatch. Subtype is READ (cluster-singleton meta-log reader), BOOTSTRAP (per-bucket walker), or DRAIN (per-rule pending). bucket / rule_hash populated for the latter two; ContinuationHint is an advisory resume point for kill-resumable BOOTSTRAP / READ. oneof tag = 14, after the existing ec_balance_params at 13. * fix(s3/lifecycle): noncurrent delete markers honor NoncurrentDays A non-current delete marker is just another version per AWS S3 spec and is eligible under NoncurrentVersionExpirationDays. The IsDeleteMarker special case is meant only for the *current* delete marker (sole-survivor ExpiredObjectDeleteMarker action), so guard that switch arm with IsLatest. Without IsLatest, the bootstrap walker silently skipped pre-existing non-current delete markers under a NoncurrentDays rule because ComputeDueAt returned zero. Mirrors the same fix in evaluate.go so the runtime decision matches. * fix(s3/lifecycle): preserve prefix trailing slash in RuleHash "logs" and "logs/" match different object sets under literal strings.HasPrefix semantics: "logs" matches "logsmore/x", "logs/" does not. Collapsing them in the hash would let an XML edit silently bind the new rule to the previous rule's durable state directory, causing stale bootstrap_complete and stale pending entries against a rule that now matches a different set. * fix(s3/lifecycle): add UNSPECIFIED sentinels to enum zero values Proto3 best practice: enum 0 should be an _UNSPECIFIED sentinel, not an active value. Persisted state schemas care most: a partially populated payload (or one written by an older binary that didn't set the field) would otherwise silently default to a semantically active value. - LifecycleState.RuleKind: shifts EXPIRATION_DAYS..EXPIRED_DELETE_MARKER from 0..5 to 1..6, with RULE_KIND_UNSPECIFIED at 0. - LifecycleState.RuleMode: shifts EVENT_DRIVEN..PENDING_BOOTSTRAP from 0..4 to 1..5. - LifecycleState.DegradedReason: replaces NONE=0 with DEGRADED_REASON_UNSPECIFIED=0 (operators treat both as healthy). - StreamKind: shifts ORIGINAL..PENDING from 0..3 to 1..4. - S3LifecycleParams.Subtype: shifts READ..DRAIN from 0..2 to 1..3, so an unset subtype no longer routes into the cluster-singleton READ task. No on-disk state has been written yet; renumbering is safe. * docs(s3/lifecycle): per-action state, not per-rule A single AWS lifecycle XML <Rule> can declare multiple actions in parallel (e.g. ExpirationDays=90 + AbortMPU=7 + NoncurrentDays=30). Each must drive its own delay/horizon/mode/pending stream independently. Modeling the rule as one compiled entry with one kind collapses these — picking the smallest delay (7d MPU) means the 90d expiration cursor advances past objects that aren't yet due, and the 90d action never re-fires. Restructure storage to per-action: every XML rule expands into N compiled actions; state lives at <bucket>/<rule_hash>/<action_kind>/. The intermediate rule_hash directory keeps a rule's actions grouped for operator listing. Each action has its own state file with its own mode + bootstrap_complete + degraded_reason; sibling actions of the same rule can degrade independently. * fix(s3/lifecycle): per-action proto schema + safety-scan counters Realigns the durable schema with the per-action storage model and the safety-scan contract in the design doc. - Promote LifecycleState.RuleKind to a top-level ActionKind enum and rename rule_kind -> action_kind. The same enum now also keys BootstrapKey, PendingKey, and BlockerRecord so per-action streams under one rule never collapse. - LifecycleState now keyed by (rule_hash, action_kind). Added last_safety_scan_ts_ns / next_safety_scan_ts_ns and the four observability counters the design specifies (evaluated_total, expired_total, metadata_only_total, error_total). Dropped last_evaluated_ns, deleted_total, skipped_object_lock_total, and pending_size — subsumed or out-of-band. - BlockerRecord.action_kind is OPTIONAL (UNSPECIFIED for pre-evaluation failures, just like rule_hash). No on-disk state has been written yet; the renumbering / rename is free. * fix(s3/lifecycle): per-action MinTriggerAge / EventLogHorizon helpers The retention mode gate and safety-scan cadence run on each compiled action independently. Taking a "min/max across all actions in the rule" — as the previous helpers did — was wrong: a 90d ExpirationDays sibling alongside a 7d AbortMPU action would cause the 90d cursor to advance at 7d (because MinTriggerAge picked the smallest), and the ExpirationDays action would never re-fire on objects that aged past the 7d sweep window before reaching 90d. Both helpers now take an ActionKind and return the threshold for that specific action only. Returns 0 if the rule does not declare the requested kind, which makes the gate a no-op and the cadence fall through to the kind floor. Also adds RuleActionKinds(rule), the canonical expansion that the engine uses at compile time to turn one XML rule into N compiled actions. NewerNoncurrentVersions paired with NoncurrentDays is subsumed into a single NONCURRENT_DAYS action (AWS-paired conditions); only when NewerNoncurrent stands alone does it become NEWER_NONCURRENT. * fix(s3/lifecycle): length-prefix RuleHash to remove delimiter ambiguity The previous encoding used "tag=K=V\n" lines, which is ambiguous: a tag (a=b, c) serializes identically to (a, b=c). A prefix containing "\nexp_days=99" could likewise forge an action field. Either could silently bind two semantically different rules to the same per-rule state directory. Switch to a length-prefixed canonical form: each scalar is written as <field-tag-byte> <uvarint-length> <bytes>. Field-tag bytes namespace each scalar so ("a=b" tag-key) and ("a" tag-key with "=b" leakage) can't collide. Three regression tests pin the resistance. * docs(s3/lifecycle): ActionKey is the engine identity, not rule_hash Finishes the per-action restructure across the engine pseudocode, bootstrap completion, drain locks, detector paths, policy CAS, and Phase 2 plan. Every per-action data structure — engine indexes, target modes, newly-completed sets, bootstrap completion bits, drain keys, locks, metrics, status — is now keyed by ActionKey{rule_hash, action_kind}, not by rule_hash alone. Without this, sibling actions under one XML rule still shared scheduling state in the engine: originalDelayGroups holding []ruleHash means a rule's 7d AbortMPU and 90d ExpirationDays collapse into one entry, the smaller delay wins for cursor advance, and the larger sibling never re-fires. Helper API tightened: EvaluateAction(rule, kind, info, now) and ComputeDueAt(rule, kind, info) replace the aggregate-rule signatures so a caller asks one specific action's eligibility against one entry, never "any action of this rule." Drain task lock includes action_kind so siblings have independent re-arm timers. Policy CAS moves from rule_hash to ActionKey membership. * fix(s3/lifecycle): kind-aware EvaluateAction / ComputeDueAt Replaces the aggregate-rule signatures with per-action ones: EvaluateAction(rule, kind, info, now) and ComputeDueAt(rule, kind, info). The old Evaluate(rule, info, now) could return the verdict of ANY action declared on the rule, which sat wrong with the per-action engine indexing (each ActionKey has its own delay group, mode, and pending stream). Each helper now decides exactly one (rule, kind) compiled action's fate against one entry. Asking for a kind the rule doesn't declare, or asking against the wrong object shape for that kind, returns ActionNone / zero — never silently routes to a sibling. Multi-action regression test: a rule with ExpirationDays=90 and AbortMPU=7 evaluates each kind independently for the same entry; the 7d window has no influence on the 90d eligibility decision.	2026-05-07 10:02:32 -07:00