Commit Graph
4 Commits
Author SHA1 Message Date
Chris LuandClaude Opus 5.5 a91b333658 s3 lifecycle: persist daily walk checkpoints and resume them safely (#12650)
* s3 lifecycle: walk checkpoints take each entry's resume position

Versions of one object share a Path, so a checkpoint set from Path after
the first version made a resume skip the rest. The ListFunc now gives
each entry the start value that continues after it, and leaves it empty
on all but an object's last version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: resume a bucket listing where the walk stopped

The filer listing skipped entries with key <= start, but its order is
not key order: each directory emits its .versions groups before its
files, and a directory's children come before a sibling like dir.txt.
Positions are now one (pass, name) frame per directory level, so a
resume seeds each listing and skips exactly what was finished. A version
group is one unit, and a skipped group still keeps its bare null
version out of the second pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: content hash of a walk view

Hashes every non-disabled action of a view and the versioning of their
buckets, so a saved walk position can be tied to the walk it came from:
another rule set, another walk mode, or a bucket that turned versioned
gives another hash.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: persist daily walk progress and resume it

A walk that failed or was cut short started over on the next pass,
listing again every bucket it had already finished. WalkBuckets now
keeps the finished buckets and each unfinished bucket's position in a
checkpoint file next to the shard cursors, one per shard set, and the
next walk of the same view and shards resumes from it. The record is
saved every 30s and when the walk stops, and cleared when a walk
completes, so the next walk is full again and sees keys added behind
the position; a clear that fails fails the walk rather than leave a
record that would skip buckets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a stuck resumed walk starts over

A bucket that fails at the same place on every pass kept the walk from
completing, so the buckets it had finished were never walked again and
their due objects waited forever. A resumed walk that fails again
without finishing anything clears its checkpoint, and the next walk is
full. A canceled walk keeps it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: do not resume a walk checkpoint left for a week

A checkpoint is cleared only by a later walk of the same shard set, so a
set that stopped walking, such as a one-off per-shard walk, left its
record behind, and a walk of that set much later would skip buckets
finished long before. Records now carry their save time and are not
resumed after seven days; a walk that keeps retrying saves on every
attempt and stays fresh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a failed walk without progress leaves its checkpoint alone

A pass canceled while it read the checkpoint failed the read, then
saved its own empty progress over the record it never loaded, so the
next pass started every bucket again. A failed walk that finished
nothing has nothing new to record and no longer writes; the stuck
resume reset still clears.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: space out walk checkpoint saves after a failed one

The save time advanced only on success, so once the filer refused a
periodic save every following entry tried again and logged another
warning. The interval now counts from the last attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a walk checkpoint save that errors may still be on disk

A save whose reply was lost left onDisk false, so a walk that then
completed skipped the clear and the next walk resumed from the stale
record instead of walking everything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: keep a stalled walk checkpoint for a day

A resumed walk that failed again without progress dropped its record
right away, so an outage spanning two attempts restarted the whole
walk. Treat the walk as stuck only once its record has not advanced
for a day.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a resumed walk counts its earlier buckets as walked

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: replay a resumed recovery walk from the walk that began it

A recovery or cold-start pass rewinds replay to runNow - maxTTL. When the
walk resumed a checkpoint, its finished buckets were listed in an earlier
attempt, so an object there that was not yet due then and whose write is
now older than maxTTL was neither expired by the walk nor replayed.

The checkpoint now keeps the start of the walk that began it, WalkBuckets
returns that time (its own start for a fresh walk), and runShard floors
the recovery rewind and the cold-start replay at it. A record without a
start is not resumed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a cold start that resumed a walk replays on the next pass

The pass's shared subscription starts at runNow - maxTTL for a shard
without a cursor, before its walk reports an earlier start, so flooring
the cold-start drain at the walk start could not deliver the older
events. Such a shard now saves its cursor at the walk start minus maxTTL
and returns, like the recovery branch, and the next pass replays the
window from that cursor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: test that a cursor below the floor opens the next subscription

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: test the cold-start replay across two Run passes

A cold start whose walk resumed an earlier attempt sees an older event
only on the next pass. The test runs Run twice against a filer that
serves events past the requested floor, with a later in-window event
that would otherwise move the cursor past the older one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:35:49 -08:00
Chris Lu e6633146d8 filer: shrink zip spool threshold in tests to fix 32-bit OOM (#11801)
* filer: shrink zip spool threshold in tests to fix 32-bit OOM

TestSpoolZipReaderTempFile and TestSpoolZipReaderLimit each allocate
~33-70 MiB to cross the 32 MiB in-memory spool threshold. On linux/386 the
weed/filer test binary already runs near the 4 GiB address-space limit, so
these allocations reliably crash it with "out of memory" mid-suite.

Make zipSpoolMemoryLimit a var and drop it to 1 KiB in those tests, which
exercises the same memory-vs-temp-file spool paths with KiB-sized buffers.

* s3 lifecycle: update stale runShard and WalkerFunc call sites in tests

Recent master changes added a walks param to runShard (#11797) and made WalkerFunc take []int, but replay_view_test.go and run_bounded_test.go still use the old signatures, breaking go vet and test builds on merge.
2026-10-17 20:30:21 -07:00
74ed12d6b3 s3 lifecycle: replay only the rules that fit the retention window (#11789)
* s3 lifecycle: replay only the rules that fit the retention window

runShard handed drainShardEvents the full snapshot, so rules the retention
window assigns to the walk were also matched during meta-log replay. Their
matches are not due yet, and drainShardEvents freezes the cursor at the first
event with a not-yet-due match. With a 3-day and a 30-day expiration rule on one
bucket and RetentionWindow of 8 days, every shard's cursor stopped at the first
30-day object's PUT and stayed there until that object came due. Each pass
then re-read the log from that point and re-dispatched every 3-day expiration
already handled, which surfaces as NOOP_RESOLVED dispatches, a growing
cursor_lag_max, and passes that still report status=ok.

Snapshot.ReplayView drops the replay-kind actions promoted to the walk and
keeps every other action as compiled, and the drain uses it. Promoted rules
are served by the walk as before. RulesForShard, PromotedHash and ReplayView
share one isPromoted predicate. With RetentionWindow unset no active rule is
promoted, so behaviour without a configured window is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: scale the replay-view test with DaysToDuration

The test took its retention window from DaysToDuration but dated its PUTs
in hours, so under the s3tests build tag (one lifecycle day = 10s) every
event fell before the seeded cursor and replay never ran. Express the
event ages in DaysToDuration too, so the test exercises replay at both
scales.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: rehabilitate scan_only mode in the replay drain view

ReplayView shared the base CompiledAction pointers, so a replay-kind
rule locked into ModeScanOnly by a prior compile stayed invisible to
router.Route's event_driven gate even when the runtime retention window
kept it in the replay partition - and it is absent from the walk view
too, so the rule would be served by neither.

Clone non-promoted replay-kind actions the way RulesForShard does
(Mode forced to event_driven), reactivating the scan_only case whose
bootstrap rendezvous already completed, while preserving the base
active bit for genuinely pre-bootstrap actions. Disabled actions are
excluded for parity with the other views, and DESIGN.md now documents
the view.

* s3 lifecycle: keep drain activation observable, require walker with retention

ReplayView now shares pointers for event-driven actions so a mid-pass
MarkActive flip remains visible to the drain; only a scan_only-locked
action is cloned to event_driven, and reactivated only when its
bootstrap rendezvous completed (tracked via a new bootstrapComplete
field on CompiledAction).

validate() rejects RetentionWindow without Walker: promoted rules
would lose their only dispatch path while the drain cursor still
advanced past their events.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-17 15:24:35 -07:00
Chris Lu 7063b3e14c s3 lifecycle: bound the daily-replay pass so a quiet cluster stops wedging the job (#10578)
* s3 lifecycle: bound the daily-replay subscription at the pass boundary

A pass opens one meta-log subscription and 16 shard drains, then waits
on all of them. Nothing told the subscription where the pass ends, so
the only exit was the fan-out spotting an event past runNow — i.e. some
unrelated write landing under /buckets after the pass started. On a
cluster that goes quiet the reader parks in Recv, every shard drain
starves on an empty channel, and Run never returns. The job sits at
stage "starting" with the executor slot held and no log line, so
expiry stops cluster-wide until someone restarts the worker.

The pass covers (globalStartTsNs, runNow], so say that: UntilNs on the
subscribe request makes the filer end the stream once it has shipped
that range. The reader then closes the event channel on the way out,
which is what unblocks the fan-out and the drains when the stream
finishes on its own rather than by cancellation.

Same fix retires the other silent hang: a reader that failed early
(subscribe error, stream error) also left every drain waiting forever.

* s3 lifecycle: keep a halted shard from starving the shared fan-out

A drain that halts mid-stream (BLOCKED / RETRY_LATER / an RPC error on
dispatch) returns while the fan-out is still routing that shard's
events. After 256 of them the per-shard buffer is full and the fan-out
blocks on the send, so no other shard sees another event. Run's
WaitGroup never drains, and the teardown that would cancel the reader
sits behind that wait — the pass wedges exactly like an idle
subscription did, with one S3 hiccup as the trigger.

Keep discarding the channel after runShard returns. The events are
past this shard's saved cursor and get re-scanned next pass anyway.

* s3 lifecycle: assert the starved shard actually made progress

The fan-out test only checked that Run returned, which a version that
quietly dropped the second shard's events would also satisfy. Assert
the dispatch landed and the cursor moved.

recordingClient gains a per-object outcome map: the two shards dispatch
from separate goroutines, so pinning BLOCKED by call index was a race
waiting to pick the wrong shard.

* s3 lifecycle: fail the pass when the shared subscription dies

Closing the event channel on reader exit is what unblocks the shard
drains, but it also means a subscribe that never opened, or a stream
that broke mid-pass, now ends every drain cleanly. Run logged that at
V(2) and returned the shard result — so a filer failure produced a
green lifecycle job that had processed nothing.

Surface it as the pass error. Cursors still hold what was processed and
tomorrow resumes there; what changes is that the job stops claiming
success.

Cancellation has to stay a non-error — the shell driver's -runtime cap
is a truncated pass, not a failed one — and a canceled gRPC stream
arrives as a status code, not a wrapped context.Canceled, so isCanceled
checks both forms the way the rest of the tree does.

* s3 lifecycle: decide reader cancellation by intent, not status code

A stream we cancel and a stream the filer cancels both arrive as
codes.Canceled, so classifying the reader's exit by its error let a
truncated pass report success whenever the failure happened to carry a
cancellation status.

Intent is knowable exactly, so read that instead: the pass stops on
purpose only when the caller's context ended (the shell driver's
-runtime cap) or the fan-out hit the pass boundary itself. Everything
else is a broken subscription and fails the pass.

TestRun_ServerSideCancelFailsThePass and TestRun_CappedPassIsNotAFailure
are the same codes.Canceled from the reader with opposite verdicts —
the pair only passes because the decision no longer looks at the error.

* s3 lifecycle: time out a subscription that stops delivering

UntilNs ends a healthy stream and gRPC keepalive catches a dead
connection, but neither reaches a filer that keeps answering pings while
its handler has stopped producing. The pass would wait on that forever,
since s3_lifecycle is the one job type with no execution timeout.

Bound the wait for each response at 20 minutes, and opt into the filer's
idle heartbeats so a caught-up stream proves liveness instead of looking
stalled. The default sits above the filer's 15-minute metadata-gap
recovery budget, so a subscriber legitimately parked on a gap is never
mistaken for a stalled one.

Recv is only interruptible by killing the RPC, so it moves to its own
goroutine behind a per-response deadline. The timer covers only the wait
on the filer — dispatch to Events happens outside it, so a slow consumer
can't trip the watchdog.

Approach and the 20-minute figure are from #10577 by way of comparing
the two fixes; the wiring differs because the reader here ends the pass
by closing its event channel rather than cancelling the fan-out.

* s3 lifecycle: trim the comments added by this branch

Keep the non-obvious why, drop the prose restating what the code says.

* s3 lifecycle: snapshot reader intent where the reader stops

Sampling ctx.Err() during teardown reads it after the drains and cursor
saves have run. A reader that failed while the deadline was still live,
on a pass whose teardown then outlives that deadline, was classified as
an intentional stop and reported success.

Sampling earlier in Run is not the fix either: before the shard wait, a
legitimately capped pass has not reached its deadline yet and would be
misclassified the other way. Intent belongs where the reader actually
stops, so the reader goroutine records it next to the error it returns.

Reported by greptile on #10578.

* s3 lifecycle: cover the worker-dispatched pass with nothing due

The e2e suite drives the shell command in 14 of 15 files; the one test
on the real admin->worker path backdates an object, so its own delete
pushes a meta-log event past the pass boundary and ends the pass. The
branch where a pass has nothing to dispatch was never exercised through
the worker.

Cover it, asserting the pass returns on its own: no admin cancellation,
and the executor slot free for the next one.

This is not a regression test for the wedge. A pass used to end when any
write landed past its boundary, and on a shared test cluster something
usually does — the whole suite passes on the unfixed build, verified.
The deterministic guards stay the dailyrun unit tests; this one would
catch a pass that hangs unconditionally.
2026-08-05 08:41:37 -07:00