mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-16 00:38:53 +01:00
* s3 lifecycle: walk checkpoints take each entry's resume position Versions of one object share a Path, so a checkpoint set from Path after the first version made a resume skip the rest. The ListFunc now gives each entry the start value that continues after it, and leaves it empty on all but an object's last version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: resume a bucket listing where the walk stopped The filer listing skipped entries with key <= start, but its order is not key order: each directory emits its .versions groups before its files, and a directory's children come before a sibling like dir.txt. Positions are now one (pass, name) frame per directory level, so a resume seeds each listing and skips exactly what was finished. A version group is one unit, and a skipped group still keeps its bare null version out of the second pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: content hash of a walk view Hashes every non-disabled action of a view and the versioning of their buckets, so a saved walk position can be tied to the walk it came from: another rule set, another walk mode, or a bucket that turned versioned gives another hash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: persist daily walk progress and resume it A walk that failed or was cut short started over on the next pass, listing again every bucket it had already finished. WalkBuckets now keeps the finished buckets and each unfinished bucket's position in a checkpoint file next to the shard cursors, one per shard set, and the next walk of the same view and shards resumes from it. The record is saved every 30s and when the walk stops, and cleared when a walk completes, so the next walk is full again and sees keys added behind the position; a clear that fails fails the walk rather than leave a record that would skip buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a stuck resumed walk starts over A bucket that fails at the same place on every pass kept the walk from completing, so the buckets it had finished were never walked again and their due objects waited forever. A resumed walk that fails again without finishing anything clears its checkpoint, and the next walk is full. A canceled walk keeps it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: do not resume a walk checkpoint left for a week A checkpoint is cleared only by a later walk of the same shard set, so a set that stopped walking, such as a one-off per-shard walk, left its record behind, and a walk of that set much later would skip buckets finished long before. Records now carry their save time and are not resumed after seven days; a walk that keeps retrying saves on every attempt and stays fresh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a failed walk without progress leaves its checkpoint alone A pass canceled while it read the checkpoint failed the read, then saved its own empty progress over the record it never loaded, so the next pass started every bucket again. A failed walk that finished nothing has nothing new to record and no longer writes; the stuck resume reset still clears. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: space out walk checkpoint saves after a failed one The save time advanced only on success, so once the filer refused a periodic save every following entry tried again and logged another warning. The interval now counts from the last attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a walk checkpoint save that errors may still be on disk A save whose reply was lost left onDisk false, so a walk that then completed skipped the clear and the next walk resumed from the stale record instead of walking everything. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: keep a stalled walk checkpoint for a day A resumed walk that failed again without progress dropped its record right away, so an outage spanning two attempts restarted the whole walk. Treat the walk as stuck only once its record has not advanced for a day. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a resumed walk counts its earlier buckets as walked Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: replay a resumed recovery walk from the walk that began it A recovery or cold-start pass rewinds replay to runNow - maxTTL. When the walk resumed a checkpoint, its finished buckets were listed in an earlier attempt, so an object there that was not yet due then and whose write is now older than maxTTL was neither expired by the walk nor replayed. The checkpoint now keeps the start of the walk that began it, WalkBuckets returns that time (its own start for a fresh walk), and runShard floors the recovery rewind and the cold-start replay at it. A record without a start is not resumed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a cold start that resumed a walk replays on the next pass The pass's shared subscription starts at runNow - maxTTL for a shard without a cursor, before its walk reports an earlier start, so flooring the cold-start drain at the walk start could not deliver the older events. Such a shard now saves its cursor at the walk start minus maxTTL and returns, like the recovery branch, and the next pass replays the window from that cursor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test that a cursor below the floor opens the next subscription Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test the cold-start replay across two Run passes A cold start whose walk resumed an earlier attempt sees an older event only on the next pass. The test runs Run twice against a filer that serves events past the requested floor, with a later in-window event that would otherwise move the cursor past the older one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
280 lines
11 KiB
Go
280 lines
11 KiB
Go
package dailyrun
|
|
|
|
import (
|
|
"context"
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
|
"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
|
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
|
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
|
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/reader"
|
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/router"
|
|
"github.com/stretchr/testify/assert"
|
|
"github.com/stretchr/testify/require"
|
|
)
|
|
|
|
// readerEventAlias keeps the test free of the reader import collision
|
|
// in the recovery test file when we need a typed nil channel.
|
|
type readerEventAlias = reader.Event
|
|
|
|
// TestWalkerDue covers the throttle decision matrix in isolation.
|
|
func TestWalkerDue(t *testing.T) {
|
|
runNow := time.Unix(1_700_000_000, 0).UTC()
|
|
cases := []struct {
|
|
name string
|
|
lastWalkedNs int64
|
|
interval time.Duration
|
|
want bool
|
|
}{
|
|
// interval=0 keeps the prior "fire every pass" semantics — the
|
|
// in-repo integration tests and the s3tests fast driver rely on
|
|
// this so the rule-change-then-walk behavior surfaces within a
|
|
// single test runtime. Within-pass double-fire suppression
|
|
// lives in runShard's walkedThisPass local; walkerDue answers
|
|
// the persisted-state question only.
|
|
{"interval zero always due", runNow.UnixNano(), 0, true},
|
|
// LastWalkedNs=0 marks "never walked steady-state" — the post-
|
|
// upgrade cold-start case for cursor files that predate the
|
|
// LastWalkedNs field. Fire so the throttle anchor gets seeded.
|
|
{"never walked is due", 0, time.Hour, true},
|
|
// Throttle on: interval elapsed → due.
|
|
{"interval elapsed", runNow.Add(-2 * time.Hour).UnixNano(), time.Hour, true},
|
|
{"interval exactly elapsed", runNow.Add(-time.Hour).UnixNano(), time.Hour, true},
|
|
// Throttle on: not yet → skip.
|
|
{"interval not yet elapsed", runNow.Add(-30 * time.Minute).UnixNano(), time.Hour, false},
|
|
// Future LastWalkedNs (clock skew or replay of an older runNow
|
|
// against a newer cursor) must not panic and must not fire —
|
|
// treat as "throttled" because the cursor claims a future walk.
|
|
{"future lastWalked", runNow.Add(time.Hour).UnixNano(), time.Hour, false},
|
|
}
|
|
for _, tc := range cases {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
assert.Equal(t, tc.want, walkerDue(tc.lastWalkedNs, runNow, tc.interval))
|
|
})
|
|
}
|
|
}
|
|
|
|
// TestRunShard_WalkerThrottle confirms two back-to-back runShard
|
|
// invocations only fire the steady-state walker once when the second
|
|
// pass starts inside the WalkerInterval window. The interval-zero
|
|
// control case keeps firing on every pass — same shape, different
|
|
// expectation — pinning the prior behavior so a default-zero config
|
|
// (tests, in-repo integration) doesn't regress.
|
|
func TestRunShard_WalkerThrottle(t *testing.T) {
|
|
cases := []struct {
|
|
name string
|
|
interval time.Duration
|
|
secondPassAfter time.Duration
|
|
wantTotalCalls int
|
|
wantSecondAdvNs bool // did LastWalkedNs change between pass 1 and pass 2?
|
|
}{
|
|
{"interval=0 fires every pass", 0, 30 * time.Second, 2, true},
|
|
{"throttled: second pass within interval", time.Hour, 30 * time.Second, 1, false},
|
|
{"throttled: second pass past interval", time.Hour, 2 * time.Hour, 2, true},
|
|
}
|
|
for _, tc := range cases {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
snap := snapshotWithRule(t, 30) // replay-eligible only; steady-state walker would normally skip
|
|
// Force a walker-only partition so the steady-state branch
|
|
// has something to walk: set retentionWindow=0 in cfg, which
|
|
// makes promoted non-empty and walkView non-empty.
|
|
p := newMemPersister()
|
|
calls := 0
|
|
cfg := Config{
|
|
Persister: p,
|
|
Walker: func(_ context.Context, _ *engine.Snapshot, _ []int) (time.Time, error) {
|
|
calls++
|
|
return time.Time{}, nil
|
|
},
|
|
RetentionWindow: -1, // negative sentinel: falls back to maxTTL — leaves walkView empty
|
|
WalkerInterval: tc.interval,
|
|
}
|
|
// Pre-seed cursor matching snap's hashes so cold-start /
|
|
// recovery branches don't fire and we measure ONLY the
|
|
// steady-state walker.
|
|
rsh := engine.ReplayContentHash(snap)
|
|
promoted := engine.PromotedHash(snap, engine.MaxEffectiveTTL(snap))
|
|
runNow := time.Unix(1_700_000_000, 0).UTC()
|
|
require.NoError(t, p.Save(context.Background(), 0, Cursor{
|
|
TsNs: runNow.UnixNano(),
|
|
RuleSetHash: rsh,
|
|
PromotedHash: promoted,
|
|
}))
|
|
|
|
// With the snapshot's only rule being replay-eligible, the
|
|
// steady-state walkView is empty and the walker won't fire
|
|
// regardless of throttle. Use a snapshot that has BOTH a
|
|
// replay rule AND a scan-only walker rule by forcing
|
|
// retentionWindow to a very small value via cfg.
|
|
cfg.RetentionWindow = time.Nanosecond
|
|
// Recompute promoted with the test retention so persisted
|
|
// hashes match what runShard sees on each pass.
|
|
snapPromoted := engine.PromotedHash(snap, time.Nanosecond)
|
|
require.NoError(t, p.Save(context.Background(), 0, Cursor{
|
|
TsNs: runNow.UnixNano(),
|
|
RuleSetHash: rsh,
|
|
PromotedHash: snapPromoted,
|
|
}))
|
|
|
|
// runShard reaches drainShardEvents after the steady-state
|
|
// walker fires (replay-eligible rules present). A nil events
|
|
// channel would block forever; a closed one returns
|
|
// immediately so drain exits cleanly and the cursor save
|
|
// still records LastWalkedNs.
|
|
closedEvents := make(chan *readerEventAlias)
|
|
close(closedEvents)
|
|
|
|
// Pass 1.
|
|
require.NoError(t, runShard(context.Background(), cfg, snap, runNow, 0, closedEvents, nil))
|
|
afterPass1, _, _ := p.Load(context.Background(), 0)
|
|
require.Equal(t, 1, calls, "pass 1 must fire the walker (cold-start anchor)")
|
|
require.Equal(t, runNow.UnixNano(), afterPass1.LastWalkedNs)
|
|
|
|
// Pass 2.
|
|
runNow2 := runNow.Add(tc.secondPassAfter)
|
|
closedEvents2 := make(chan *readerEventAlias)
|
|
close(closedEvents2)
|
|
cfg.Stats = &RunStats{}
|
|
require.NoError(t, runShard(context.Background(), cfg, snap, runNow2, 0, closedEvents2, nil))
|
|
afterPass2, _, _ := p.Load(context.Background(), 0)
|
|
assert.Equal(t, tc.wantTotalCalls, calls, "walker calls after 2 passes")
|
|
if tc.wantSecondAdvNs {
|
|
assert.Equal(t, runNow2.UnixNano(), afterPass2.LastWalkedNs, "throttle allowed the second walk")
|
|
assert.Zero(t, cfg.Stats.Summary().WalksDeferred)
|
|
} else {
|
|
assert.Equal(t, runNow.UnixNano(), afterPass2.LastWalkedNs, "throttle suppressed the second walk; anchor unchanged")
|
|
assert.Equal(t, 1, cfg.Stats.Summary().WalksDeferred, "a throttled walk leaves its rules unevaluated")
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
// TestValidate_RejectsNegativeWalkerInterval pins the loud-failure
|
|
// contract for an embedder or test that constructs dailyrun.Config
|
|
// directly with a negative WalkerInterval. The admin-config parser
|
|
// clamps negative input to zero (worker/tasks/s3_lifecycle/config.go),
|
|
// but a caller bypassing that parser would otherwise get the silent
|
|
// fall-through-to-walk-every-pass behavior the throttle is trying to
|
|
// prevent.
|
|
func TestValidate_RejectsNegativeWalkerInterval(t *testing.T) {
|
|
// validate only checks for nil fields, not behavior — type-assert
|
|
// each interface to its zero value via a stub. The stubs never run.
|
|
cfg := validatableConfig()
|
|
cfg.WalkerInterval = -time.Hour
|
|
err := validate(cfg)
|
|
require.Error(t, err)
|
|
assert.Contains(t, err.Error(), "WalkerInterval")
|
|
// And the zero sentinel still passes.
|
|
cfg.WalkerInterval = 0
|
|
require.NoError(t, validate(cfg))
|
|
}
|
|
|
|
// validatableConfig builds a minimal dailyrun.Config that passes the
|
|
// nil checks in validate() so individual fields can be poked. The
|
|
// stubs intentionally don't implement any meaningful behavior — they
|
|
// only need to be non-nil interface values.
|
|
func validatableConfig() Config {
|
|
return Config{
|
|
Engine: engine.New(),
|
|
FilerClient: stubFilerClient{},
|
|
Client: stubLifecycleClient{},
|
|
Persister: newMemPersister(),
|
|
Lister: stubSiblingLister{},
|
|
BucketsPath: "/buckets",
|
|
}
|
|
}
|
|
|
|
type stubFilerClient struct{ filer_pb.SeaweedFilerClient }
|
|
type stubLifecycleClient struct{}
|
|
|
|
func (stubLifecycleClient) LifecycleDelete(_ context.Context, _ *s3_lifecycle_pb.LifecycleDeleteRequest) (*s3_lifecycle_pb.LifecycleDeleteResponse, error) {
|
|
return nil, nil
|
|
}
|
|
|
|
type stubSiblingLister struct{}
|
|
|
|
func (stubSiblingLister) Survivors(_ context.Context, _, _ string) (router.Survivors, error) {
|
|
return router.Survivors{}, nil
|
|
}
|
|
func (stubSiblingLister) LookupVersion(_ context.Context, _, _, _ string) (*filer_pb.Entry, error) {
|
|
return nil, nil
|
|
}
|
|
func (stubSiblingLister) ListVersions(_ context.Context, _, _ string) ([]*filer_pb.Entry, error) {
|
|
return nil, nil
|
|
}
|
|
func (stubSiblingLister) LookupNullVersion(_ context.Context, _, _ string) (*filer_pb.Entry, bool, error) {
|
|
return nil, false, nil
|
|
}
|
|
|
|
// TestRunShard_ColdStartDoesNotDoubleWalk pins the within-pass guard
|
|
// in runShard's walkedThisPass local. Before the guard, a cold-start
|
|
// pass with WalkerInterval=0 fired the walker twice in one pass:
|
|
// once via the mustWalkColdStart branch (with RecoveryView) and again
|
|
// immediately via the steady-state branch (with the per-shard walk
|
|
// view, which is a subset of RecoveryView). The second walk added no
|
|
// coverage and burned a full bucket scan. Guard against regression.
|
|
func TestRunShard_ColdStartDoesNotDoubleWalk(t *testing.T) {
|
|
snap := snapshotWithRule(t, 30)
|
|
p := newMemPersister()
|
|
// No persisted cursor → mustWalkColdStart=true. WalkerInterval=0
|
|
// would, before the fix, allow the steady-state branch to fire
|
|
// again with no time elapsed.
|
|
calls := 0
|
|
cfg := Config{
|
|
Persister: p,
|
|
Walker: func(_ context.Context, _ *engine.Snapshot, _ []int) (time.Time, error) {
|
|
calls++
|
|
return time.Time{}, nil
|
|
},
|
|
WalkerInterval: 0,
|
|
}
|
|
runNow := time.Unix(1_700_000_000, 0).UTC()
|
|
|
|
closedEvents := make(chan *readerEventAlias)
|
|
close(closedEvents)
|
|
require.NoError(t, runShard(context.Background(), cfg, snap, runNow, 0, closedEvents, nil))
|
|
|
|
assert.Equal(t, 1, calls, "cold-start walker must fire exactly once per pass even with interval=0")
|
|
}
|
|
|
|
// TestRunShard_RecoveryWalkerSetsLastWalkedAnchor pins that the
|
|
// unconditional recovery-branch walker fire still updates the
|
|
// LastWalkedNs anchor so the NEXT pass's throttle starts counting
|
|
// from this walk rather than re-firing immediately.
|
|
func TestRunShard_RecoveryWalkerSetsLastWalkedAnchor(t *testing.T) {
|
|
snap := snapshotWithRule(t, 30)
|
|
p := newMemPersister()
|
|
var stale [32]byte
|
|
for i := range stale {
|
|
stale[i] = 0xAA
|
|
}
|
|
require.NoError(t, p.Save(context.Background(), 5, Cursor{
|
|
TsNs: 1234,
|
|
RuleSetHash: stale,
|
|
LastWalkedNs: 0, // never walked anchor
|
|
}))
|
|
|
|
calls := 0
|
|
cfg := Config{
|
|
Persister: p,
|
|
Walker: func(_ context.Context, _ *engine.Snapshot, _ []int) (time.Time, error) {
|
|
calls++
|
|
return time.Time{}, nil
|
|
},
|
|
WalkerInterval: time.Hour,
|
|
}
|
|
runNow := time.Unix(1_700_000_000, 0).UTC()
|
|
require.NoError(t, runShard(context.Background(), cfg, snap, runNow, 5, nil, nil))
|
|
|
|
require.Equal(t, 1, calls, "recovery branch fires the walker unconditionally")
|
|
got, ok, _ := p.Load(context.Background(), 5)
|
|
require.True(t, ok)
|
|
assert.Equal(t, runNow.UnixNano(), got.LastWalkedNs,
|
|
"recovery walk must update LastWalkedNs so the steady-state pass after it can throttle")
|
|
}
|
|
|
|
// Compile-time check that the documented sentinel passes the type system.
|
|
var _ = s3lifecycle.ShardCount
|