mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-11 17:10:40 +02:00
* docs(s3lifecycle): design for daily-replay worker Captures the algorithm and dev plan iterated on in PR #9431 and the discussion leading up to it: per-shard daily meta-log replay, walker as a per-day pass for ExpirationDate/ExpiredDeleteMarker/NewerNoncurrent plus a recovery branch over engine.RecoveryView(snap), explicit retention-window input to RulesForShard, two cursor hashes (ReplayContentHash + PromotedHash) that together detect every invalidation case. Implementation phases are sequenced so each can ship independently — Phase 1 (noncurrent_since stamp) just landed. * feat(s3/lifecycle): daily-replay worker behind algorithm flag (Phase 2) New weed/s3api/s3lifecycle/dailyrun package implementing the bounded daily meta-log scan from the design doc. One pass per Execute per shard: load cursor, scan events forward, route each through router.Route, dispatch any due Match, advance the cursor on success. Halt-on-failure keeps the cursor at the last fully-processed event so tomorrow resumes from the same point — head-of-line blocking is the deliberate failure signal. Replay-only in this phase. Phase 4 wires the walker for ExpirationDate, ExpiredDeleteMarker, NewerNoncurrent, and scan_only-promoted rules. Until then a typed UnsupportedRuleError refuses runs on those buckets: operators see the rejection in the activity log rather than silently losing rules. Behavior: - Per-shard cursor {TsNs, RuleSetHash, PromotedHash} JSON-persisted under /etc/s3/lifecycle/daily-cursors/. PromotedHash always-empty in Phase 2; Phase 4 turns it on. - Rule-change branch rewinds cursor to now - max_ttl when the replay-content hash mismatches. Cold start uses the same floor. - Transport errors retry 3x with exponential backoff capped at 5s; server outcomes (RETRY_LATER / BLOCKED) halt the run without retry. - Empty-replay sentinel: cursor TsNs=0 when no replay-eligible rules exist, only the hash gates a future addition. Worker shape: - New admin config field "algorithm" with enum streaming|daily_replay, default streaming. Existing deployments are unaffected. - handler.Execute branches on the flag: streaming routes through the current scheduler.Scheduler, daily_replay routes through dailyrun.Run. - dispatcher.NewFilerSiblingLister exported so both paths share the same .versions/ + null-bare lookup. Engine integration: - Local replayContentHash + maxEffectiveTTL helpers in dailyrun. Phase 4's engine surface (ReplayContentHash, MaxEffectiveTTL) will replace them with one-line redirects; the local versions hash the same fields so the cursor stays valid across the swap. Tests cover cursor persistence, unsupported-rule rejection, hash stability under rule reordering, hash sensitivity to TTL edits, max-TTL aggregation, dispatch retry budget, and request shape including the identity-CAS witness. Includes the design doc at weed/s3api/s3lifecycle/DESIGN.md so reviewers and future phases share the same spec. * feat(s3/lifecycle): default to daily_replay; streaming becomes the fallback knob The streaming dispatcher hasn't shipped to users yet, so there's no backward-compat surface to preserve. Flip the algorithm default from streaming to daily_replay so the new path is the standard from day one. Streaming stays as an explicit opt-in escape hatch during the Phase 4 walker rollout; Phase 5 deletes both the flag and the streaming code. Buckets whose lifecycle rules require walker-bound dispatch (ExpirationDate, ExpiredDeleteMarker, NewerNoncurrent, scan_only) will fail the daily_replay run with the existing UnsupportedRuleError until Phase 4 walker integration ships. Operators hitting that case can set algorithm=streaming until the follow-up lands. Updates the test for the default value and renames the unknown-value-fallback case to reflect the new default. * fix(s3/lifecycle/dailyrun): drop per-rule done flag — it suppressed due matches The done map was keyed by ActionKey = {Bucket, RuleHash, ActionKind}. That's only safe when each event produces at most one match per ActionKey with a single deterministic due-time formula — ExpirationDays and AbortMPU fit that shape because due_time = ev.TsNs + r.days is monotonic in event TsNs. But NoncurrentDays paired with NewerNoncurrentVersions > 0 (allowed in Phase 2 since it compiles to ActionKindNoncurrentDays) routes through routePointerTransitionExpand, which emits matches for every noncurrent sibling — each with its own SuccessorModTime taken from the demoting event for that specific sibling. A single event can therefore produce two matches for the same ActionKey on different objects with wildly different DueTimes. With the old code, a not-yet-due sibling encountered first would set done[ActionKey] = true and then the next sibling — even though its DueTime had already passed — would be skipped. Future events for the same rule would also be suppressed for the rest of the run. Objects that should have been deleted weren't. Fix: drop the early-stop optimization. Process every match independently. A future-DueTime match is now silently skipped without affecting any later match. The performance hit is small (Phase 2 is a single bounded daily pass, and the rate limiter is the real throughput governor); the correctness gain is non-negotiable. Also fixes the inverted comment in processMatches that described the old check as "due_time is past now" when it actually checked DueTime.After(now) (i.e., NOT yet due). Adds four targeted tests: - not-yet-due match first in slice does not suppress two later due matches for the same rule; - reversed slice ordering produces identical dispatch; - BLOCKED outcome halts the loop before later due matches are sent; - empty match slice is a no-op. Phase 4's walker-and-recovery integration can revisit a per-(rule, object) memoization if profiling argues for it. * fix(s3/lifecycle/dailyrun): address PR review — cursor advance, mode gate, ctx cancel, snapshot consistency Addresses PR #9446 review feedback. Eight distinct fixes: 1. CURSOR ADVANCEMENT (gemini, critical). The old code advanced the persisted cursor to lastOK = TsNs of the last event processed, including events whose matches were skipped as not-yet-due. Those skipped matches would never be re-scanned, so objects under long-TTL rules would never expire. Track a "stuck" flag in drainShardEvents: the first event with a skipped (future-DueTime) match stops cursorAdvanceTo from rising, but the loop keeps processing later events to dispatch any that ARE due. The persisted cursor sits at the last fully-processed event so tomorrow's run re-scans from the skipped event onward and the future-due matches get re-evaluated when they age in. processMatches now returns (skippedAny, halted, err) so the drain loop can tell apart "event fully drained" from "event had pending future-due matches." 2. MODE GATE (gemini). checkSnapshotForUnsupported only checked the ActionKind. A replay-eligible kind with Mode != ModeEventDriven (e.g. ModeScanOnly via retention promotion) passed the check but then got silently ignored by router.Route, which gates dispatch on Mode == ModeEventDriven. Reject loudly with the typed error so admin sees the rejection in the activity log. 3. WORKERS CONFIG (gemini). The handler hardcoded 16 concurrent shard goroutines regardless of cfg.Workers. Add a Workers field to dailyrun.Config and gate the goroutine fan-out on a semaphore of that size; the handler now passes cfg.Workers through. 4. SINGLE SNAPSHOT PER RUN (coderabbit). Run() validated against one snapshot but runShard() pulled a fresh cfg.Engine.Snapshot() per shard. Mid-run Compile would let shards process different rule sets. Capture snap at the top of Run, pass it down to every shard. 5. FROZEN runNow (coderabbit). drainShardEvents and processMatches accepted a `now func() time.Time` and called it multiple times. DueTime comparisons would slip as the run wore on. Capture runNow once at the top of Run and thread it through as a time.Time value. 6. CTX CANCELLATION (coderabbit). The drain loop's <-ctx.Done() case broke out of the loop and returned nil, marking interrupted runs as successful. Return ctx.Err() instead so the caller propagates the interrupt; cursorAdvanceTo carries whatever progress was made. 7. CURSOR LOAD VALIDATION (coderabbit + gemini). The persister silently accepted empty files, mismatched shard_ids, and hash slices shorter than 32 bytes (copy() would zero-pad). Each now returns a typed error so the run halts and an operator investigates rather than silently re-scanning from time zero or persisting a zero-padded hash that masks corruption forever. 8. DEAD BRANCH (coderabbit). The "lastOK < startTsNs → keep persisted" guard in runShard was unreachable because drainShardEvents initialized lastOK := startTsNs and only ever raised it. Removed along with the new cursor-advancement semantics that handle the "no events processed" case implicitly. Plus markdown lint: DESIGN.md fenced code blocks now carry a `text` language identifier to satisfy MD040. Skipped from the review: - gemini's "maxTTL == 0 incorrectly skips immediate expirations": actions with Days <= 0 don't compile to a CompiledAction (see weed/s3api/s3lifecycle/action_kind.go: `if rule.X > 0`). The new empty-replay sentinel uses `rsh == [32]byte{}` for clarity per gemini's suggested form, but the behavior is equivalent. Tests added/updated: - TestProcessMatches_AllDueNoSkippedFlag pins skippedAny=false when all matches are past their DueTime. - TestCheckSnapshotForUnsupported_NonEventDrivenModeRejected pins the new Mode check. - TestFilerCursorPersister_EmptyFileReturnsError, _ShardIDMismatchReturnsError, _HashLengthMismatchReturnsError pin the new validation rules. - Existing process-matches tests reshaped for the (skippedAny, halted, err) return tuple. Full build clean. Dailyrun + worker test packages green.
156 lines
5.2 KiB
Go
156 lines
5.2 KiB
Go
package dailyrun
|
|
|
|
import (
|
|
"context"
|
|
"sync"
|
|
"testing"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
|
"github.com/stretchr/testify/assert"
|
|
"github.com/stretchr/testify/require"
|
|
)
|
|
|
|
// fakeStore is a minimal in-memory dispatcher.FilerStore for cursor tests.
|
|
type fakeStore struct {
|
|
mu sync.Mutex
|
|
files map[string][]byte
|
|
}
|
|
|
|
func newFakeStore() *fakeStore {
|
|
return &fakeStore{files: map[string][]byte{}}
|
|
}
|
|
|
|
func (s *fakeStore) key(dir, name string) string { return dir + "/" + name }
|
|
|
|
func (s *fakeStore) Read(_ context.Context, dir, name string) ([]byte, error) {
|
|
s.mu.Lock()
|
|
defer s.mu.Unlock()
|
|
b, ok := s.files[s.key(dir, name)]
|
|
if !ok {
|
|
return nil, filer_pb.ErrNotFound
|
|
}
|
|
return append([]byte(nil), b...), nil
|
|
}
|
|
|
|
func (s *fakeStore) Save(_ context.Context, dir, name string, content []byte) error {
|
|
s.mu.Lock()
|
|
defer s.mu.Unlock()
|
|
s.files[s.key(dir, name)] = append([]byte(nil), content...)
|
|
return nil
|
|
}
|
|
|
|
func TestFilerCursorPersister_EmptyLoadReturnsNotFound(t *testing.T) {
|
|
p := &FilerCursorPersister{Store: newFakeStore()}
|
|
c, found, err := p.Load(context.Background(), 0)
|
|
require.NoError(t, err)
|
|
assert.False(t, found)
|
|
assert.Equal(t, Cursor{}, c)
|
|
}
|
|
|
|
func TestFilerCursorPersister_SaveLoadRoundTrip(t *testing.T) {
|
|
p := &FilerCursorPersister{Store: newFakeStore()}
|
|
in := Cursor{
|
|
TsNs: 1700000000_000000123,
|
|
}
|
|
for i := range in.RuleSetHash {
|
|
in.RuleSetHash[i] = byte(i + 1)
|
|
}
|
|
for i := range in.PromotedHash {
|
|
in.PromotedHash[i] = byte(0xff - i)
|
|
}
|
|
require.NoError(t, p.Save(context.Background(), 3, in))
|
|
|
|
out, found, err := p.Load(context.Background(), 3)
|
|
require.NoError(t, err)
|
|
require.True(t, found)
|
|
assert.Equal(t, in, out)
|
|
}
|
|
|
|
func TestFilerCursorPersister_IsolatesShards(t *testing.T) {
|
|
p := &FilerCursorPersister{Store: newFakeStore()}
|
|
a := Cursor{TsNs: 100}
|
|
b := Cursor{TsNs: 200}
|
|
require.NoError(t, p.Save(context.Background(), 0, a))
|
|
require.NoError(t, p.Save(context.Background(), 1, b))
|
|
|
|
got0, _, err := p.Load(context.Background(), 0)
|
|
require.NoError(t, err)
|
|
got1, _, err := p.Load(context.Background(), 1)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, int64(100), got0.TsNs)
|
|
assert.Equal(t, int64(200), got1.TsNs)
|
|
}
|
|
|
|
func TestFilerCursorPersister_CorruptDataReturnsError(t *testing.T) {
|
|
store := newFakeStore()
|
|
store.files[store.key(CursorDir, "shard-00.json")] = []byte("not json")
|
|
p := &FilerCursorPersister{Store: store}
|
|
_, _, err := p.Load(context.Background(), 0)
|
|
require.Error(t, err)
|
|
}
|
|
|
|
func TestFilerCursorPersister_EmptyFileReturnsError(t *testing.T) {
|
|
// A cursor file that exists but is empty signals a partial write
|
|
// or external truncation. Treating it as "cold start" would silently
|
|
// re-scan from time zero and burn through a meta-log window for no
|
|
// reason. Pin that we error instead.
|
|
store := newFakeStore()
|
|
store.files[store.key(CursorDir, "shard-00.json")] = []byte{}
|
|
p := &FilerCursorPersister{Store: store}
|
|
_, _, err := p.Load(context.Background(), 0)
|
|
require.Error(t, err)
|
|
}
|
|
|
|
func TestFilerCursorPersister_WrongVersionReturnsError(t *testing.T) {
|
|
// Future schema bumps must not silently overwrite a cursor written
|
|
// by a newer worker. Pin that contract here.
|
|
store := newFakeStore()
|
|
store.files[store.key(CursorDir, "shard-00.json")] = []byte(`{"version":42,"shard_id":0,"ts_ns":0,"rule_set_hash":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=","promoted_hash":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="}`)
|
|
p := &FilerCursorPersister{Store: store}
|
|
_, _, err := p.Load(context.Background(), 0)
|
|
require.Error(t, err)
|
|
}
|
|
|
|
func TestFilerCursorPersister_ShardIDMismatchReturnsError(t *testing.T) {
|
|
// A file declaring shard=1 read by a shard=0 caller is corruption
|
|
// (or a misroute); applying that cursor would advance the wrong
|
|
// shard's state. Reject loudly.
|
|
store := newFakeStore()
|
|
// Build a valid v1 payload but with shard_id=1 in the body.
|
|
in := Cursor{TsNs: 42}
|
|
for i := range in.RuleSetHash {
|
|
in.RuleSetHash[i] = byte(i + 1)
|
|
}
|
|
for i := range in.PromotedHash {
|
|
in.PromotedHash[i] = byte(i)
|
|
}
|
|
p := &FilerCursorPersister{Store: store}
|
|
require.NoError(t, p.Save(context.Background(), 1, in))
|
|
// Reparent that file as if it lived at shard 0 — the test's fake
|
|
// store only keys by name, so we copy the bytes manually.
|
|
store.mu.Lock()
|
|
store.files[store.key(CursorDir, "shard-00.json")] = store.files[store.key(CursorDir, "shard-01.json")]
|
|
store.mu.Unlock()
|
|
_, _, err := p.Load(context.Background(), 0)
|
|
require.Error(t, err)
|
|
}
|
|
|
|
func TestFilerCursorPersister_HashLengthMismatchReturnsError(t *testing.T) {
|
|
// Hand-rolled JSON with hashes shorter than 32 bytes. copy() would
|
|
// silently zero-pad which makes the hash comparison match a value
|
|
// that doesn't actually exist on disk. Reject loudly so an operator
|
|
// fixes the persisted state.
|
|
store := newFakeStore()
|
|
store.files[store.key(CursorDir, "shard-00.json")] = []byte(`{"version":1,"shard_id":0,"ts_ns":0,"rule_set_hash":"AA==","promoted_hash":"AA=="}`)
|
|
p := &FilerCursorPersister{Store: store}
|
|
_, _, err := p.Load(context.Background(), 0)
|
|
require.Error(t, err)
|
|
}
|
|
|
|
func TestFilerCursorPersister_NilStoreErrors(t *testing.T) {
|
|
p := &FilerCursorPersister{}
|
|
_, _, err := p.Load(context.Background(), 0)
|
|
require.Error(t, err)
|
|
require.Error(t, p.Save(context.Background(), 0, Cursor{}))
|
|
}
|