Files
seaweedfs/weed/shell/command_s3_lifecycle_run_shard.go
T
Chris LuandClaude Opus 5.5 a91b333658 s3 lifecycle: persist daily walk checkpoints and resume them safely (#12650)
* s3 lifecycle: walk checkpoints take each entry's resume position

Versions of one object share a Path, so a checkpoint set from Path after
the first version made a resume skip the rest. The ListFunc now gives
each entry the start value that continues after it, and leaves it empty
on all but an object's last version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: resume a bucket listing where the walk stopped

The filer listing skipped entries with key <= start, but its order is
not key order: each directory emits its .versions groups before its
files, and a directory's children come before a sibling like dir.txt.
Positions are now one (pass, name) frame per directory level, so a
resume seeds each listing and skips exactly what was finished. A version
group is one unit, and a skipped group still keeps its bare null
version out of the second pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: content hash of a walk view

Hashes every non-disabled action of a view and the versioning of their
buckets, so a saved walk position can be tied to the walk it came from:
another rule set, another walk mode, or a bucket that turned versioned
gives another hash.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: persist daily walk progress and resume it

A walk that failed or was cut short started over on the next pass,
listing again every bucket it had already finished. WalkBuckets now
keeps the finished buckets and each unfinished bucket's position in a
checkpoint file next to the shard cursors, one per shard set, and the
next walk of the same view and shards resumes from it. The record is
saved every 30s and when the walk stops, and cleared when a walk
completes, so the next walk is full again and sees keys added behind
the position; a clear that fails fails the walk rather than leave a
record that would skip buckets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a stuck resumed walk starts over

A bucket that fails at the same place on every pass kept the walk from
completing, so the buckets it had finished were never walked again and
their due objects waited forever. A resumed walk that fails again
without finishing anything clears its checkpoint, and the next walk is
full. A canceled walk keeps it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: do not resume a walk checkpoint left for a week

A checkpoint is cleared only by a later walk of the same shard set, so a
set that stopped walking, such as a one-off per-shard walk, left its
record behind, and a walk of that set much later would skip buckets
finished long before. Records now carry their save time and are not
resumed after seven days; a walk that keeps retrying saves on every
attempt and stays fresh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a failed walk without progress leaves its checkpoint alone

A pass canceled while it read the checkpoint failed the read, then
saved its own empty progress over the record it never loaded, so the
next pass started every bucket again. A failed walk that finished
nothing has nothing new to record and no longer writes; the stuck
resume reset still clears.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: space out walk checkpoint saves after a failed one

The save time advanced only on success, so once the filer refused a
periodic save every following entry tried again and logged another
warning. The interval now counts from the last attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a walk checkpoint save that errors may still be on disk

A save whose reply was lost left onDisk false, so a walk that then
completed skipped the clear and the next walk resumed from the stale
record instead of walking everything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: keep a stalled walk checkpoint for a day

A resumed walk that failed again without progress dropped its record
right away, so an outage spanning two attempts restarted the whole
walk. Treat the walk as stuck only once its record has not advanced
for a day.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a resumed walk counts its earlier buckets as walked

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: replay a resumed recovery walk from the walk that began it

A recovery or cold-start pass rewinds replay to runNow - maxTTL. When the
walk resumed a checkpoint, its finished buckets were listed in an earlier
attempt, so an object there that was not yet due then and whose write is
now older than maxTTL was neither expired by the walk nor replayed.

The checkpoint now keeps the start of the walk that began it, WalkBuckets
returns that time (its own start for a fresh walk), and runShard floors
the recovery rewind and the cold-start replay at it. A record without a
start is not resumed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a cold start that resumed a walk replays on the next pass

The pass's shared subscription starts at runNow - maxTTL for a shard
without a cursor, before its walk reports an earlier start, so flooring
the cold-start drain at the walk start could not deliver the older
events. Such a shard now saves its cursor at the walk start minus maxTTL
and returns, like the recovery branch, and the next pass replays the
window from that cursor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: test that a cursor below the floor opens the next subscription

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: test the cold-start replay across two Run passes

A cold start whose walk resumed an earlier attempt sees an older event
only on the next pass. The test runs Run twice against a filer that
serves events past the requested floor, with a later in-window event
that would otherwise move the cursor past the older one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:35:49 -08:00

343 lines
11 KiB
Go

package shell
import (
"context"
"errors"
"flag"
"fmt"
"io"
"sort"
"strconv"
"strings"
"time"
"github.com/seaweedfs/seaweedfs/weed/credential"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/dailyrun"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/dispatcher"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/scheduler"
)
func init() {
Commands = append(Commands, &commandS3LifecycleRunShard{})
}
type commandS3LifecycleRunShard struct{}
func (c *commandS3LifecycleRunShard) Name() string {
return "s3.lifecycle.run-shard"
}
func (c *commandS3LifecycleRunShard) Help() string {
return `manually run one daily-replay pass for the given shards
Drives dailyrun.Run once against the live filer + S3 server: builds
the engine snapshot from filer-backed bucket configs, opens the
meta-log subscription per shard, dispatches due actions via
LifecycleDelete, and walks the live tree for any walker-bound rules.
Persists each shard's cursor to /etc/s3/lifecycle/daily-cursors/
so subsequent runs resume.
Used by the s3-tests CI workflow and the test/s3/lifecycle/
integration tests to drive expirations on demand without standing up
the full admin+worker plugin stack.
# single shard
s3.lifecycle.run-shard -shard 0 -s3 localhost:8333 -events 100
# contiguous range
s3.lifecycle.run-shard -shards 0-15 -s3 localhost:8333 -events 5000
# explicit set
s3.lifecycle.run-shard -shards 0,3,7 -s3 localhost:8333
# bounded wall-clock
s3.lifecycle.run-shard -shards 0-15 -s3 localhost:8333 -runtime 10s
`
}
func (c *commandS3LifecycleRunShard) HasTag(CommandTag) bool { return false }
func (c *commandS3LifecycleRunShard) Do(args []string, env *CommandEnv, writer io.Writer) error {
fs := flag.NewFlagSet(c.Name(), flag.ContinueOnError)
shard := fs.Int("shard", -1, "single shard id in [0, 16); use -shards for a range or set")
shardsSpec := fs.String("shards", "", "shard range \"lo-hi\" or comma list \"a,b,c\"; mutually exclusive with -shard")
s3Endpoint := fs.String("s3", "", "s3 server gRPC endpoint, host:port")
eventBudget := fs.Int("events", 1000, "max in-shard events per pass (0 = drain to now)")
runtime := fs.Duration("runtime", 0, "wall-clock cap on the whole run; 0 = no timeout")
// -refresh drives the inter-pass cadence when the command runs as a
// long-lived worker (the s3tests CI workflow case): every refresh
// the engine snapshot is re-loaded and dailyrun.Run fires another
// pass. 0 means "run once and exit" (the integration-test case).
cadence := fs.Duration("refresh", 0, "inter-pass interval; 0 = single pass, then exit")
// Obsolete flags kept for back-compat with existing CI scripts and
// integration tests. Accept and ignore.
_ = fs.Duration("dispatch", 0, "ignored (legacy streaming flag)")
_ = fs.Duration("checkpoint", 0, "ignored (legacy streaming flag)")
_ = fs.Duration("bootstrap-interval", 0, "ignored (legacy streaming flag)")
if err := fs.Parse(args); err != nil {
return err
}
shards, err := resolveShardSelection(*shard, *shardsSpec)
if err != nil {
return err
}
if *s3Endpoint == "" {
return fmt.Errorf("-s3 required (host:port of s3 server gRPC)")
}
if *eventBudget < 0 {
return fmt.Errorf("-events must be >= 0 (0 = unbounded)")
}
bucketsPath, err := resolveBucketsPath(env)
if err != nil {
return fmt.Errorf("resolve buckets path: %w", err)
}
fmt.Fprintf(writer, "buckets path: %s\n", bucketsPath)
dialCtx, dialCancel := context.WithTimeout(context.Background(), 30*time.Second)
conn, err := pb.GrpcDial(dialCtx, *s3Endpoint, false, env.option.GrpcDialOption)
dialCancel()
if err != nil {
return fmt.Errorf("dial s3 %s: %w", *s3Endpoint, err)
}
defer conn.Close()
rpcClient := s3_lifecycle_pb.NewSeaweedS3LifecycleInternalClient(conn)
return env.WithFilerClient(true, func(filerClient filer_pb.SeaweedFilerClient) error {
ctx := context.Background()
var cancel context.CancelFunc
if *runtime > 0 {
ctx, cancel = context.WithTimeout(ctx, *runtime)
defer cancel()
}
client := &lifecycleClientCallable{c: rpcClient}
walkerDispatch := &dailyrun.WalkerDispatcher{Client: client}
fmt.Fprintf(writer, "running shards %s (event budget=%d, runtime=%s, refresh=%s)…\n",
formatShardLabel(shards), *eventBudget, *runtime, *cadence)
announcedLoad := false
runPass := func() error {
// When -refresh is set we MUST cap each pass; otherwise
// drainShardEvents blocks until the outer runtime ctx
// expires and the loop's ticker never fires. With cadence=0
// (one-shot) the pass uses the full ctx and returns when
// the runtime cap hits.
passCtx := ctx
if *cadence > 0 {
var passCancel context.CancelFunc
// Pass budget = cadence + grace. Grace gives the
// drain time to actually process events that arrived
// during the cadence window.
passCtx, passCancel = context.WithTimeout(ctx, *cadence+5*time.Second)
defer passCancel()
}
eng := engine.New()
inputs, parseErrors, err := scheduler.LoadCompileInputs(passCtx, filerClient, bucketsPath)
if err != nil {
return fmt.Errorf("load lifecycle configs: %w", err)
}
for i, pe := range parseErrors {
if i < 3 {
fmt.Fprintf(writer, "warning: %s: %v\n", pe.Bucket, pe.Err)
}
}
if extra := len(parseErrors) - 3; extra > 0 {
fmt.Fprintf(writer, "warning: %d additional bucket(s) had malformed lifecycle config\n", extra)
}
eng.Compile(inputs, engine.CompileOptions{PriorStates: scheduler.AllActivePriorStates(inputs)})
if len(inputs) == 0 {
return nil
}
if !announcedLoad {
fmt.Fprintf(writer, "loaded lifecycle for %d bucket(s)\n", len(inputs))
announcedLoad = true
}
buckets := make([]string, 0, len(inputs))
for _, in := range inputs {
if in.Bucket != "" {
buckets = append(buckets, in.Bucket)
}
}
persister := &dailyrun.FilerCursorPersister{Store: dispatcher.NewFilerStoreClient(filerClient)}
walker := dailyrun.WalkerFunc(func(walkCtx context.Context, view *engine.Snapshot, shards []int) (time.Time, error) {
return dailyrun.WalkBuckets(walkCtx, view, shards, buckets,
dailyrun.FilerListFunc(filerClient, bucketsPath, view), walkerDispatch, nil, persister)
})
return dailyrun.Run(passCtx, dailyrun.Config{
Shards: shards,
BucketsPath: bucketsPath,
Engine: eng,
FilerClient: filerClient,
Client: client,
Persister: persister,
Lister: dispatcher.NewFilerSiblingLister(filerClient, bucketsPath),
// The shell command is used for bounded one-shot sweeps in
// integration tests and CI. Fan out across the selected shards
// so recovery walks do not serialize 16 shard scans into a 10s
// timeout budget.
Workers: len(shards),
Walker: walker,
EventBudget: *eventBudget,
ClientName: fmt.Sprintf("shell-lifecycle-%s", formatShardLabel(shards)),
})
}
if err := runPass(); err != nil && !errors.Is(err, context.Canceled) && !errors.Is(err, context.DeadlineExceeded) {
return fmt.Errorf("daily_run: %w", err)
}
// cadence=0 → one-shot (test/s3/lifecycle/ uses this).
// cadence>0 → loop until ctx done (s3tests CI workflow uses this).
if *cadence > 0 {
ticker := time.NewTicker(*cadence)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
fmt.Fprintf(writer, "shards %s complete; ctx done\n", formatShardLabel(shards))
return nil
case <-ticker.C:
if err := runPass(); err != nil && !errors.Is(err, context.Canceled) && !errors.Is(err, context.DeadlineExceeded) {
fmt.Fprintf(writer, "shards %s pass error: %v\n", formatShardLabel(shards), err)
}
}
}
}
fmt.Fprintf(writer, "shards %s complete; cursors checkpointed\n", formatShardLabel(shards))
return nil
})
}
// resolveShardSelection turns the -shard / -shards flags into a sorted,
// deduplicated []int. Exactly one form must be specified.
func resolveShardSelection(singleShard int, shardsSpec string) ([]int, error) {
if singleShard >= 0 && shardsSpec != "" {
return nil, fmt.Errorf("-shard and -shards are mutually exclusive")
}
if singleShard < 0 && shardsSpec == "" {
return nil, fmt.Errorf("specify -shard <id> or -shards <range|set>")
}
if singleShard >= 0 {
if singleShard >= s3lifecycle.ShardCount {
return nil, fmt.Errorf("-shard %d out of [0,%d)", singleShard, s3lifecycle.ShardCount)
}
return []int{singleShard}, nil
}
return parseShardsSpec(shardsSpec)
}
// parseShardsSpec accepts "lo-hi" (inclusive) or "a,b,c" and returns a
// sorted, deduplicated, in-range []int.
func parseShardsSpec(spec string) ([]int, error) {
spec = strings.TrimSpace(spec)
seen := map[int]struct{}{}
add := func(v int) error {
if v < 0 || v >= s3lifecycle.ShardCount {
return fmt.Errorf("shard %d out of [0,%d)", v, s3lifecycle.ShardCount)
}
seen[v] = struct{}{}
return nil
}
if strings.Contains(spec, "-") && !strings.Contains(spec, ",") {
parts := strings.SplitN(spec, "-", 2)
lo, err := strconv.Atoi(strings.TrimSpace(parts[0]))
if err != nil {
return nil, fmt.Errorf("range lo: %w", err)
}
hi, err := strconv.Atoi(strings.TrimSpace(parts[1]))
if err != nil {
return nil, fmt.Errorf("range hi: %w", err)
}
if lo > hi {
return nil, fmt.Errorf("range lo %d > hi %d", lo, hi)
}
for v := lo; v <= hi; v++ {
if err := add(v); err != nil {
return nil, err
}
}
} else {
for _, part := range strings.Split(spec, ",") {
part = strings.TrimSpace(part)
if part == "" {
continue
}
v, err := strconv.Atoi(part)
if err != nil {
return nil, fmt.Errorf("shard list: %w", err)
}
if err := add(v); err != nil {
return nil, err
}
}
}
if len(seen) == 0 {
return nil, fmt.Errorf("empty shard set")
}
out := make([]int, 0, len(seen))
for v := range seen {
out = append(out, v)
}
sort.Ints(out)
return out, nil
}
func formatShardLabel(shards []int) string {
if len(shards) == 1 {
return fmt.Sprintf("%d", shards[0])
}
contiguous := true
for i := 1; i < len(shards); i++ {
if shards[i] != shards[i-1]+1 {
contiguous = false
break
}
}
if contiguous {
return fmt.Sprintf("%d-%d", shards[0], shards[len(shards)-1])
}
parts := make([]string, len(shards))
for i, v := range shards {
parts[i] = strconv.Itoa(v)
}
return strings.Join(parts, ",")
}
// lifecycleClientCallable adapts the generated grpc client (variadic
// CallOption tail) to dailyrun.LifecycleClient.
type lifecycleClientCallable struct {
c s3_lifecycle_pb.SeaweedS3LifecycleInternalClient
}
func (l *lifecycleClientCallable) LifecycleDelete(ctx context.Context, req *s3_lifecycle_pb.LifecycleDeleteRequest) (*s3_lifecycle_pb.LifecycleDeleteResponse, error) {
ctx, _ = credential.WithS3InternalAdminAuth(ctx)
return l.c.LifecycleDelete(ctx, req)
}
// resolveBucketsPath fetches the filer's configured buckets directory.
// Falls back to /buckets when the filer doesn't return one.
func resolveBucketsPath(env *CommandEnv) (string, error) {
var path string
err := env.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
resp, err := client.GetFilerConfiguration(context.Background(), &filer_pb.GetFilerConfigurationRequest{})
if err != nil {
return err
}
path = resp.GetDirBuckets()
return nil
})
if err != nil {
return "", err
}
if path == "" {
path = "/buckets"
}
return path, nil
}