mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-06 20:17:38 +01:00
* docs(s3lifecycle): design for daily-replay worker Captures the algorithm and dev plan iterated on in PR #9431 and the discussion leading up to it: per-shard daily meta-log replay, walker as a per-day pass for ExpirationDate/ExpiredDeleteMarker/NewerNoncurrent plus a recovery branch over engine.RecoveryView(snap), explicit retention-window input to RulesForShard, two cursor hashes (ReplayContentHash + PromotedHash) that together detect every invalidation case. Implementation phases are sequenced so each can ship independently — Phase 1 (noncurrent_since stamp) just landed. * feat(s3/lifecycle): daily-replay worker behind algorithm flag (Phase 2) New weed/s3api/s3lifecycle/dailyrun package implementing the bounded daily meta-log scan from the design doc. One pass per Execute per shard: load cursor, scan events forward, route each through router.Route, dispatch any due Match, advance the cursor on success. Halt-on-failure keeps the cursor at the last fully-processed event so tomorrow resumes from the same point — head-of-line blocking is the deliberate failure signal. Replay-only in this phase. Phase 4 wires the walker for ExpirationDate, ExpiredDeleteMarker, NewerNoncurrent, and scan_only-promoted rules. Until then a typed UnsupportedRuleError refuses runs on those buckets: operators see the rejection in the activity log rather than silently losing rules. Behavior: - Per-shard cursor {TsNs, RuleSetHash, PromotedHash} JSON-persisted under /etc/s3/lifecycle/daily-cursors/. PromotedHash always-empty in Phase 2; Phase 4 turns it on. - Rule-change branch rewinds cursor to now - max_ttl when the replay-content hash mismatches. Cold start uses the same floor. - Transport errors retry 3x with exponential backoff capped at 5s; server outcomes (RETRY_LATER / BLOCKED) halt the run without retry. - Empty-replay sentinel: cursor TsNs=0 when no replay-eligible rules exist, only the hash gates a future addition. Worker shape: - New admin config field "algorithm" with enum streaming|daily_replay, default streaming. Existing deployments are unaffected. - handler.Execute branches on the flag: streaming routes through the current scheduler.Scheduler, daily_replay routes through dailyrun.Run. - dispatcher.NewFilerSiblingLister exported so both paths share the same .versions/ + null-bare lookup. Engine integration: - Local replayContentHash + maxEffectiveTTL helpers in dailyrun. Phase 4's engine surface (ReplayContentHash, MaxEffectiveTTL) will replace them with one-line redirects; the local versions hash the same fields so the cursor stays valid across the swap. Tests cover cursor persistence, unsupported-rule rejection, hash stability under rule reordering, hash sensitivity to TTL edits, max-TTL aggregation, dispatch retry budget, and request shape including the identity-CAS witness. Includes the design doc at weed/s3api/s3lifecycle/DESIGN.md so reviewers and future phases share the same spec. * feat(s3/lifecycle): default to daily_replay; streaming becomes the fallback knob The streaming dispatcher hasn't shipped to users yet, so there's no backward-compat surface to preserve. Flip the algorithm default from streaming to daily_replay so the new path is the standard from day one. Streaming stays as an explicit opt-in escape hatch during the Phase 4 walker rollout; Phase 5 deletes both the flag and the streaming code. Buckets whose lifecycle rules require walker-bound dispatch (ExpirationDate, ExpiredDeleteMarker, NewerNoncurrent, scan_only) will fail the daily_replay run with the existing UnsupportedRuleError until Phase 4 walker integration ships. Operators hitting that case can set algorithm=streaming until the follow-up lands. Updates the test for the default value and renames the unknown-value-fallback case to reflect the new default. * fix(s3/lifecycle/dailyrun): drop per-rule done flag — it suppressed due matches The done map was keyed by ActionKey = {Bucket, RuleHash, ActionKind}. That's only safe when each event produces at most one match per ActionKey with a single deterministic due-time formula — ExpirationDays and AbortMPU fit that shape because due_time = ev.TsNs + r.days is monotonic in event TsNs. But NoncurrentDays paired with NewerNoncurrentVersions > 0 (allowed in Phase 2 since it compiles to ActionKindNoncurrentDays) routes through routePointerTransitionExpand, which emits matches for every noncurrent sibling — each with its own SuccessorModTime taken from the demoting event for that specific sibling. A single event can therefore produce two matches for the same ActionKey on different objects with wildly different DueTimes. With the old code, a not-yet-due sibling encountered first would set done[ActionKey] = true and then the next sibling — even though its DueTime had already passed — would be skipped. Future events for the same rule would also be suppressed for the rest of the run. Objects that should have been deleted weren't. Fix: drop the early-stop optimization. Process every match independently. A future-DueTime match is now silently skipped without affecting any later match. The performance hit is small (Phase 2 is a single bounded daily pass, and the rate limiter is the real throughput governor); the correctness gain is non-negotiable. Also fixes the inverted comment in processMatches that described the old check as "due_time is past now" when it actually checked DueTime.After(now) (i.e., NOT yet due). Adds four targeted tests: - not-yet-due match first in slice does not suppress two later due matches for the same rule; - reversed slice ordering produces identical dispatch; - BLOCKED outcome halts the loop before later due matches are sent; - empty match slice is a no-op. Phase 4's walker-and-recovery integration can revisit a per-(rule, object) memoization if profiling argues for it. * fix(s3/lifecycle/dailyrun): address PR review — cursor advance, mode gate, ctx cancel, snapshot consistency Addresses PR #9446 review feedback. Eight distinct fixes: 1. CURSOR ADVANCEMENT (gemini, critical). The old code advanced the persisted cursor to lastOK = TsNs of the last event processed, including events whose matches were skipped as not-yet-due. Those skipped matches would never be re-scanned, so objects under long-TTL rules would never expire. Track a "stuck" flag in drainShardEvents: the first event with a skipped (future-DueTime) match stops cursorAdvanceTo from rising, but the loop keeps processing later events to dispatch any that ARE due. The persisted cursor sits at the last fully-processed event so tomorrow's run re-scans from the skipped event onward and the future-due matches get re-evaluated when they age in. processMatches now returns (skippedAny, halted, err) so the drain loop can tell apart "event fully drained" from "event had pending future-due matches." 2. MODE GATE (gemini). checkSnapshotForUnsupported only checked the ActionKind. A replay-eligible kind with Mode != ModeEventDriven (e.g. ModeScanOnly via retention promotion) passed the check but then got silently ignored by router.Route, which gates dispatch on Mode == ModeEventDriven. Reject loudly with the typed error so admin sees the rejection in the activity log. 3. WORKERS CONFIG (gemini). The handler hardcoded 16 concurrent shard goroutines regardless of cfg.Workers. Add a Workers field to dailyrun.Config and gate the goroutine fan-out on a semaphore of that size; the handler now passes cfg.Workers through. 4. SINGLE SNAPSHOT PER RUN (coderabbit). Run() validated against one snapshot but runShard() pulled a fresh cfg.Engine.Snapshot() per shard. Mid-run Compile would let shards process different rule sets. Capture snap at the top of Run, pass it down to every shard. 5. FROZEN runNow (coderabbit). drainShardEvents and processMatches accepted a `now func() time.Time` and called it multiple times. DueTime comparisons would slip as the run wore on. Capture runNow once at the top of Run and thread it through as a time.Time value. 6. CTX CANCELLATION (coderabbit). The drain loop's <-ctx.Done() case broke out of the loop and returned nil, marking interrupted runs as successful. Return ctx.Err() instead so the caller propagates the interrupt; cursorAdvanceTo carries whatever progress was made. 7. CURSOR LOAD VALIDATION (coderabbit + gemini). The persister silently accepted empty files, mismatched shard_ids, and hash slices shorter than 32 bytes (copy() would zero-pad). Each now returns a typed error so the run halts and an operator investigates rather than silently re-scanning from time zero or persisting a zero-padded hash that masks corruption forever. 8. DEAD BRANCH (coderabbit). The "lastOK < startTsNs → keep persisted" guard in runShard was unreachable because drainShardEvents initialized lastOK := startTsNs and only ever raised it. Removed along with the new cursor-advancement semantics that handle the "no events processed" case implicitly. Plus markdown lint: DESIGN.md fenced code blocks now carry a `text` language identifier to satisfy MD040. Skipped from the review: - gemini's "maxTTL == 0 incorrectly skips immediate expirations": actions with Days <= 0 don't compile to a CompiledAction (see weed/s3api/s3lifecycle/action_kind.go: `if rule.X > 0`). The new empty-replay sentinel uses `rsh == [32]byte{}` for clarity per gemini's suggested form, but the behavior is equivalent. Tests added/updated: - TestProcessMatches_AllDueNoSkippedFlag pins skippedAny=false when all matches are past their DueTime. - TestCheckSnapshotForUnsupported_NonEventDrivenModeRejected pins the new Mode check. - TestFilerCursorPersister_EmptyFileReturnsError, _ShardIDMismatchReturnsError, _HashLengthMismatchReturnsError pin the new validation rules. - Existing process-matches tests reshaped for the (skippedAny, halted, err) return tuple. Full build clean. Dailyrun + worker test packages green.
130 lines
5.0 KiB
Go
130 lines
5.0 KiB
Go
package s3_lifecycle
|
|
|
|
import (
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
|
|
)
|
|
|
|
func TestParseConfigDefaults(t *testing.T) {
|
|
cfg := ParseConfig(nil, nil)
|
|
if cfg.Workers != defaultWorkers {
|
|
t.Errorf("Workers default=%d, want %d", cfg.Workers, defaultWorkers)
|
|
}
|
|
if cfg.DispatchTick != 1*time.Minute {
|
|
t.Errorf("DispatchTick default=%v, want 1m", cfg.DispatchTick)
|
|
}
|
|
if cfg.CheckpointTick != 30*time.Second {
|
|
t.Errorf("CheckpointTick default=%v, want 30s", cfg.CheckpointTick)
|
|
}
|
|
if cfg.RefreshInterval != 5*time.Minute {
|
|
t.Errorf("RefreshInterval default=%v, want 5m", cfg.RefreshInterval)
|
|
}
|
|
if cfg.MaxRuntime != 60*time.Minute {
|
|
t.Errorf("MaxRuntime default=%v, want 60m", cfg.MaxRuntime)
|
|
}
|
|
if cfg.BootstrapInterval != 0 {
|
|
t.Errorf("BootstrapInterval default=%v, want 0 (walk-once-per-process)", cfg.BootstrapInterval)
|
|
}
|
|
if cfg.Algorithm != AlgorithmDailyReplay {
|
|
t.Errorf("Algorithm default=%q, want %q", cfg.Algorithm, AlgorithmDailyReplay)
|
|
}
|
|
}
|
|
|
|
func TestParseConfig_AlgorithmStreamingExplicit(t *testing.T) {
|
|
// Streaming stays available as a rollout escape hatch until Phase 5
|
|
// deletes it. Operators must be able to opt back into it explicitly.
|
|
admin := map[string]*plugin_pb.ConfigValue{
|
|
"algorithm": {Kind: &plugin_pb.ConfigValue_StringValue{StringValue: AlgorithmStreaming}},
|
|
}
|
|
cfg := ParseConfig(admin, nil)
|
|
if cfg.Algorithm != AlgorithmStreaming {
|
|
t.Errorf("Algorithm=%q, want %q", cfg.Algorithm, AlgorithmStreaming)
|
|
}
|
|
}
|
|
|
|
func TestParseConfig_AlgorithmUnknownValueFallsBackToDefault(t *testing.T) {
|
|
// Operators should not be able to silently activate a future
|
|
// algorithm value by typo. Anything not in the enum falls back to
|
|
// the default (daily_replay).
|
|
admin := map[string]*plugin_pb.ConfigValue{
|
|
"algorithm": {Kind: &plugin_pb.ConfigValue_StringValue{StringValue: "future_algo"}},
|
|
}
|
|
cfg := ParseConfig(admin, nil)
|
|
if cfg.Algorithm != AlgorithmDailyReplay {
|
|
t.Errorf("unknown algorithm=%q must fall back to %q, got %q", "future_algo", AlgorithmDailyReplay, cfg.Algorithm)
|
|
}
|
|
}
|
|
|
|
func TestParseConfigOverrides(t *testing.T) {
|
|
admin := map[string]*plugin_pb.ConfigValue{
|
|
"workers": {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 4}},
|
|
}
|
|
worker := map[string]*plugin_pb.ConfigValue{
|
|
"dispatch_tick_minutes": {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 2}},
|
|
"checkpoint_tick_seconds": {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 15}},
|
|
"refresh_interval_minutes": {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 10}},
|
|
"bootstrap_interval_minutes": {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 120}},
|
|
"max_runtime_minutes": {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 120}},
|
|
}
|
|
cfg := ParseConfig(admin, worker)
|
|
if cfg.Workers != 4 {
|
|
t.Errorf("Workers=%d, want 4", cfg.Workers)
|
|
}
|
|
if cfg.DispatchTick != 2*time.Minute {
|
|
t.Errorf("DispatchTick=%v, want 2m", cfg.DispatchTick)
|
|
}
|
|
if cfg.CheckpointTick != 15*time.Second {
|
|
t.Errorf("CheckpointTick=%v, want 15s", cfg.CheckpointTick)
|
|
}
|
|
if cfg.RefreshInterval != 10*time.Minute {
|
|
t.Errorf("RefreshInterval=%v, want 10m", cfg.RefreshInterval)
|
|
}
|
|
if cfg.BootstrapInterval != 120*time.Minute {
|
|
t.Errorf("BootstrapInterval=%v, want 120m", cfg.BootstrapInterval)
|
|
}
|
|
if cfg.MaxRuntime != 120*time.Minute {
|
|
t.Errorf("MaxRuntime=%v, want 120m", cfg.MaxRuntime)
|
|
}
|
|
}
|
|
|
|
func TestParseConfigClampsZeroAndNegative(t *testing.T) {
|
|
worker := map[string]*plugin_pb.ConfigValue{
|
|
"dispatch_tick_minutes": {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
|
|
"checkpoint_tick_seconds": {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
|
|
"refresh_interval_minutes": {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
|
|
"bootstrap_interval_minutes": {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: -5}},
|
|
"max_runtime_minutes": {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: -5}},
|
|
}
|
|
cfg := ParseConfig(nil, worker)
|
|
if cfg.DispatchTick != 1*time.Minute {
|
|
t.Errorf("zero DispatchTick should clamp to default, got %v", cfg.DispatchTick)
|
|
}
|
|
if cfg.CheckpointTick != 30*time.Second {
|
|
t.Errorf("zero CheckpointTick should clamp to default, got %v", cfg.CheckpointTick)
|
|
}
|
|
if cfg.RefreshInterval != 5*time.Minute {
|
|
t.Errorf("zero RefreshInterval should clamp to default, got %v", cfg.RefreshInterval)
|
|
}
|
|
if cfg.BootstrapInterval != 0 {
|
|
t.Errorf("negative BootstrapInterval should clamp to 0, got %v", cfg.BootstrapInterval)
|
|
}
|
|
if cfg.MaxRuntime != 60*time.Minute {
|
|
t.Errorf("negative MaxRuntime should clamp to default, got %v", cfg.MaxRuntime)
|
|
}
|
|
}
|
|
|
|
func TestParseConfigBootstrapIntervalZeroIsLegacy(t *testing.T) {
|
|
// Explicit zero stays zero (walk-once-per-process). Existing
|
|
// deployments that don't set bootstrap_interval_minutes get the
|
|
// legacy behavior unchanged.
|
|
worker := map[string]*plugin_pb.ConfigValue{
|
|
"bootstrap_interval_minutes": {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
|
|
}
|
|
cfg := ParseConfig(nil, worker)
|
|
if cfg.BootstrapInterval != 0 {
|
|
t.Errorf("zero BootstrapInterval must stay zero, got %v", cfg.BootstrapInterval)
|
|
}
|
|
}
|