* feat(s3/lifecycle): filer-backed cursor Persister
FilerPersister persists per-shard cursor maps as JSON to
/etc/s3/lifecycle/cursors/shard-NN.json via filer.SaveInsideFiler.
One file per shard keeps Save atomic — the filer writes the entry
in a single mutation, so a crash mid-write doesn't leak partial
state. Pipeline.Run loads on start; the periodic checkpoint and
graceful-shutdown save go through this implementation.
A small FilerStore interface wraps the SeaweedFilerClient surface
the persister needs, so tests inject an in-memory fake instead of
mocking the full gRPC client.
* refactor(s3/lifecycle): drop BlockerStore — durable cursor IS the block
A frozen cursor doesn't advance, so the durable cursor (FilerPersister)
encodes the blocked state on its own. On worker restart the reader
re-encounters the poison event at MinTsNs, the dispatcher walks the
same retry budget to BLOCKED, and the cursor freezes at the same
EventTs. Other in-flight events between freeze tsNs and prior cursor
positions self-resolve via NOOP_RESOLVED (STALE_IDENTITY) since the
underlying objects were already deleted on the prior pass.
Removed:
- BlockerStore interface + InMemoryBlockerStore + BlockerRecord
- Dispatcher.Blockers + Dispatcher.ReplayBlockers
- the BlockerStore.Put call in handleBlocked
- Pipeline.Blockers field + the ReplayBlockers call on startup
Added a TestDispatchRestartReFreezesNaturally that pins the
self-recovery property: a fresh Dispatcher with a fresh Cursor, fed
the same poison event, reaches the same frozen state at the same
EventTs without any durable blocker store.
Operator visibility: a cursor whose MinTsNs hasn't advanced is the
signal — surfaced via the durable cursor file.
* refactor(filer): SaveInsideFiler accepts ctx
ReadInsideFiler already takes ctx; SaveInsideFiler used context.Background()
internally and silently dropped the caller's ctx. Symmetric API now;
cancellation/deadlines propagate through LookupEntry / CreateEntry /
UpdateEntry. Mechanical update of all callers — most pass
context.Background() since the existing call sites have no ctx in scope.
* fix(s3/lifecycle): deterministic order in cursor save
Iterating Go maps yields random order, so json.Encode produced a different
byte sequence on each save even when the state hadn't changed. Sort
entries by (Bucket, ActionKind, RuleHash) before encoding so the on-disk
file diffs cleanly. New test pins byte-identical output across two saves
of the same map.
* fix(s3/lifecycle): log reason when freezing cursor in handleBlocked
handleBlocked dropped the reason via _ = reason with a comment claiming
the caller logged it; none of the three callers do. A frozen cursor is
the only surface where the operator finds out something stuck, so the
reason has to land somewhere. glog.Warningf with shard, key, eventTs,
and the original reason — same shape the rest of the package uses.
* fix(s3/lifecycle): reader handles bare /buckets parent and pre-normalizes prefix
extractBucketKey accepted /buckets/ but rejected /buckets (no trailing
slash); some delete events emit the bare form, so bucket-root events
were silently dropped. Pre-normalize BucketsPath once on Run instead
of recomputing per event.
* perf(s3/lifecycle): pool sha256 hashers in ShardID
ShardID runs on every meta-log event before the shard filter; a fresh
sha256.New per call produces measurable allocator pressure under load.
sync.Pool reuses hashers across calls.
* fix(s3/lifecycle): router skips hard deletes and missing-attribute events
A hard delete carries no schedule-relevant state — Expiration would hit
NOOP_RESOLVED at dispatch and ExpiredObjectDeleteMarker fires from a
Create on the latest version. Skip rather than burn a schedule slot.
Missing Attributes leaves ModTime at year 0001, which makes
ExpirationDays fire immediately at dispatch. Skip the event instead.
Drop the unused 'versioned' parameter from buildObjectInfo; the
dispatcher's identity-CAS handles version drift in Phase 5.
* fix(s3/lifecycle): EntryIdentity.MtimeNs holds true nanoseconds
Both computeEntryIdentity (server) and buildIdentity (router) wrote
entry.Attributes.Mtime (seconds) into a field named MtimeNs. The CAS
worked because both sides agreed, but the encoding contradicted the
field name and would break if either side later started using true
nanoseconds. Combine Mtime*1e9 + the FuseAttributes.MtimeNs nanosecond
component on both sides; the test was updated to match.
* fix(s3/lifecycle): dispatcher distinguishes ctx cancel from transport errors
A canceled or deadline-exceeded RPC is shutdown, not a transport
failure: re-queue the Match at its original DueTime with no retry-budget
burn so a quick restart can't escalate it to BLOCKED.
* fix(s3/lifecycle): reader fallback prefix normalization mirrors Run
The fallback path that builds prefix from r.BucketsPath when
bucketsPathSlash is empty (test-only entry into extractBucketKey) was
appending an unconditional '/', producing '//' if BucketsPath already
ended with one. Use the same normalization Run does.
* fix(s3/lifecycle): ObjectInfo.ModTime carries the nanosecond component
ModTime dropped FuseAttributes.MtimeNs, leaving ExpirationDays one
nanosecond off relative to EntryIdentity.MtimeNs. Pass both to
time.Unix so the precision matches the CAS witness.
feat(s3/lifecycle): dispatcher + blocker store + per-shard pipeline
Dispatcher consumes due Matches from the schedule, calls LifecycleDelete,
and routes outcomes:
DONE / NOOP_RESOLVED / SKIPPED_OBJECT_LOCK -> Cursor.Advance
RETRY_LATER (within budget) -> re-schedule with backoff
RETRY_LATER (budget exhausted) / BLOCKED -> BlockerStore.Put + Freeze
BlockerStore is a small interface with InMemoryBlockerStore for tests;
the filer-backed impl follows when the worker task registration lands.
Pipeline composes Reader + Router + Dispatcher into a single Run loop
keyed by shard. Cursor is restored on start, blockers are replayed as
freezes, checkpoints write at a configurable cadence, and a final save
fires on shutdown. The meta-log itself is the durable buffer for in-flight
schedule entries — restart re-derives them from the cursor's MinTsNs.
Chris Lu