3 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	e6633146d8	filer: shrink zip spool threshold in tests to fix 32-bit OOM (#11801) ... * filer: shrink zip spool threshold in tests to fix 32-bit OOM TestSpoolZipReaderTempFile and TestSpoolZipReaderLimit each allocate ~33-70 MiB to cross the 32 MiB in-memory spool threshold. On linux/386 the weed/filer test binary already runs near the 4 GiB address-space limit, so these allocations reliably crash it with "out of memory" mid-suite. Make zipSpoolMemoryLimit a var and drop it to 1 KiB in those tests, which exercises the same memory-vs-temp-file spool paths with KiB-sized buffers. * s3 lifecycle: update stale runShard and WalkerFunc call sites in tests Recent master changes added a walks param to runShard (#11797) and made WalkerFunc take []int, but replay_view_test.go and run_bounded_test.go still use the old signatures, breaking go vet and test builds on merge.	2026-10-17 20:30:21 -07:00
3 people	74ed12d6b3	s3 lifecycle: replay only the rules that fit the retention window (#11789) ... * s3 lifecycle: replay only the rules that fit the retention window runShard handed drainShardEvents the full snapshot, so rules the retention window assigns to the walk were also matched during meta-log replay. Their matches are not due yet, and drainShardEvents freezes the cursor at the first event with a not-yet-due match. With a 3-day and a 30-day expiration rule on one bucket and RetentionWindow of 8 days, every shard's cursor stopped at the first 30-day object's PUT and stayed there until that object came due. Each pass then re-read the log from that point and re-dispatched every 3-day expiration already handled, which surfaces as NOOP_RESOLVED dispatches, a growing cursor_lag_max, and passes that still report status=ok. Snapshot.ReplayView drops the replay-kind actions promoted to the walk and keeps every other action as compiled, and the drain uses it. Promoted rules are served by the walk as before. RulesForShard, PromotedHash and ReplayView share one isPromoted predicate. With RetentionWindow unset no active rule is promoted, so behaviour without a configured window is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: scale the replay-view test with DaysToDuration The test took its retention window from DaysToDuration but dated its PUTs in hours, so under the s3tests build tag (one lifecycle day = 10s) every event fell before the seeded cursor and replay never ran. Express the event ages in DaysToDuration too, so the test exercises replay at both scales. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: rehabilitate scan_only mode in the replay drain view ReplayView shared the base CompiledAction pointers, so a replay-kind rule locked into ModeScanOnly by a prior compile stayed invisible to router.Route's event_driven gate even when the runtime retention window kept it in the replay partition - and it is absent from the walk view too, so the rule would be served by neither. Clone non-promoted replay-kind actions the way RulesForShard does (Mode forced to event_driven), reactivating the scan_only case whose bootstrap rendezvous already completed, while preserving the base active bit for genuinely pre-bootstrap actions. Disabled actions are excluded for parity with the other views, and DESIGN.md now documents the view. * s3 lifecycle: keep drain activation observable, require walker with retention ReplayView now shares pointers for event-driven actions so a mid-pass MarkActive flip remains visible to the drain; only a scan_only-locked action is cloned to event_driven, and reactivated only when its bootstrap rendezvous completed (tracked via a new bootstrapComplete field on CompiledAction). validate() rejects RetentionWindow without Walker: promoted rules would lose their only dispatch path while the drain cursor still advanced past their events. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-10-17 15:24:35 -07:00
Chris Lu	7063b3e14c	s3 lifecycle: bound the daily-replay pass so a quiet cluster stops wedging the job (#10578) ... * s3 lifecycle: bound the daily-replay subscription at the pass boundary A pass opens one meta-log subscription and 16 shard drains, then waits on all of them. Nothing told the subscription where the pass ends, so the only exit was the fan-out spotting an event past runNow — i.e. some unrelated write landing under /buckets after the pass started. On a cluster that goes quiet the reader parks in Recv, every shard drain starves on an empty channel, and Run never returns. The job sits at stage "starting" with the executor slot held and no log line, so expiry stops cluster-wide until someone restarts the worker. The pass covers (globalStartTsNs, runNow], so say that: UntilNs on the subscribe request makes the filer end the stream once it has shipped that range. The reader then closes the event channel on the way out, which is what unblocks the fan-out and the drains when the stream finishes on its own rather than by cancellation. Same fix retires the other silent hang: a reader that failed early (subscribe error, stream error) also left every drain waiting forever. * s3 lifecycle: keep a halted shard from starving the shared fan-out A drain that halts mid-stream (BLOCKED / RETRY_LATER / an RPC error on dispatch) returns while the fan-out is still routing that shard's events. After 256 of them the per-shard buffer is full and the fan-out blocks on the send, so no other shard sees another event. Run's WaitGroup never drains, and the teardown that would cancel the reader sits behind that wait — the pass wedges exactly like an idle subscription did, with one S3 hiccup as the trigger. Keep discarding the channel after runShard returns. The events are past this shard's saved cursor and get re-scanned next pass anyway. * s3 lifecycle: assert the starved shard actually made progress The fan-out test only checked that Run returned, which a version that quietly dropped the second shard's events would also satisfy. Assert the dispatch landed and the cursor moved. recordingClient gains a per-object outcome map: the two shards dispatch from separate goroutines, so pinning BLOCKED by call index was a race waiting to pick the wrong shard. * s3 lifecycle: fail the pass when the shared subscription dies Closing the event channel on reader exit is what unblocks the shard drains, but it also means a subscribe that never opened, or a stream that broke mid-pass, now ends every drain cleanly. Run logged that at V(2) and returned the shard result — so a filer failure produced a green lifecycle job that had processed nothing. Surface it as the pass error. Cursors still hold what was processed and tomorrow resumes there; what changes is that the job stops claiming success. Cancellation has to stay a non-error — the shell driver's -runtime cap is a truncated pass, not a failed one — and a canceled gRPC stream arrives as a status code, not a wrapped context.Canceled, so isCanceled checks both forms the way the rest of the tree does. * s3 lifecycle: decide reader cancellation by intent, not status code A stream we cancel and a stream the filer cancels both arrive as codes.Canceled, so classifying the reader's exit by its error let a truncated pass report success whenever the failure happened to carry a cancellation status. Intent is knowable exactly, so read that instead: the pass stops on purpose only when the caller's context ended (the shell driver's -runtime cap) or the fan-out hit the pass boundary itself. Everything else is a broken subscription and fails the pass. TestRun_ServerSideCancelFailsThePass and TestRun_CappedPassIsNotAFailure are the same codes.Canceled from the reader with opposite verdicts — the pair only passes because the decision no longer looks at the error. * s3 lifecycle: time out a subscription that stops delivering UntilNs ends a healthy stream and gRPC keepalive catches a dead connection, but neither reaches a filer that keeps answering pings while its handler has stopped producing. The pass would wait on that forever, since s3_lifecycle is the one job type with no execution timeout. Bound the wait for each response at 20 minutes, and opt into the filer's idle heartbeats so a caught-up stream proves liveness instead of looking stalled. The default sits above the filer's 15-minute metadata-gap recovery budget, so a subscriber legitimately parked on a gap is never mistaken for a stalled one. Recv is only interruptible by killing the RPC, so it moves to its own goroutine behind a per-response deadline. The timer covers only the wait on the filer — dispatch to Events happens outside it, so a slow consumer can't trip the watchdog. Approach and the 20-minute figure are from #10577 by way of comparing the two fixes; the wiring differs because the reader here ends the pass by closing its event channel rather than cancelling the fan-out. * s3 lifecycle: trim the comments added by this branch Keep the non-obvious why, drop the prose restating what the code says. * s3 lifecycle: snapshot reader intent where the reader stops Sampling ctx.Err() during teardown reads it after the drains and cursor saves have run. A reader that failed while the deadline was still live, on a pass whose teardown then outlives that deadline, was classified as an intentional stop and reported success. Sampling earlier in Run is not the fix either: before the shard wait, a legitimately capped pass has not reached its deadline yet and would be misclassified the other way. Intent belongs where the reader actually stops, so the reader goroutine records it next to the error it returns. Reported by greptile on #10578. * s3 lifecycle: cover the worker-dispatched pass with nothing due The e2e suite drives the shell command in 14 of 15 files; the one test on the real admin->worker path backdates an object, so its own delete pushes a meta-log event past the pass boundary and ends the pass. The branch where a pass has nothing to dispatch was never exercised through the worker. Cover it, asserting the pass returns on its own: no admin cancellation, and the executor slot free for the next one. This is not a regression test for the wedge. A pass used to end when any write landed past its boundary, and on a shared test cluster something usually does — the whole suite passes on the unfixed build, verified. The deterministic guards stay the dailyrun unit tests; this one would catch a pass that hangs unconditionally.	2026-08-05 08:41:37 -07:00