Files
seaweedfs/weed/worker/tasks/s3_lifecycle/handler.go
T
Chris LuandClaude Opus 5.5 a91b333658 s3 lifecycle: persist daily walk checkpoints and resume them safely (#12650)
* s3 lifecycle: walk checkpoints take each entry's resume position

Versions of one object share a Path, so a checkpoint set from Path after
the first version made a resume skip the rest. The ListFunc now gives
each entry the start value that continues after it, and leaves it empty
on all but an object's last version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: resume a bucket listing where the walk stopped

The filer listing skipped entries with key <= start, but its order is
not key order: each directory emits its .versions groups before its
files, and a directory's children come before a sibling like dir.txt.
Positions are now one (pass, name) frame per directory level, so a
resume seeds each listing and skips exactly what was finished. A version
group is one unit, and a skipped group still keeps its bare null
version out of the second pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: content hash of a walk view

Hashes every non-disabled action of a view and the versioning of their
buckets, so a saved walk position can be tied to the walk it came from:
another rule set, another walk mode, or a bucket that turned versioned
gives another hash.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: persist daily walk progress and resume it

A walk that failed or was cut short started over on the next pass,
listing again every bucket it had already finished. WalkBuckets now
keeps the finished buckets and each unfinished bucket's position in a
checkpoint file next to the shard cursors, one per shard set, and the
next walk of the same view and shards resumes from it. The record is
saved every 30s and when the walk stops, and cleared when a walk
completes, so the next walk is full again and sees keys added behind
the position; a clear that fails fails the walk rather than leave a
record that would skip buckets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a stuck resumed walk starts over

A bucket that fails at the same place on every pass kept the walk from
completing, so the buckets it had finished were never walked again and
their due objects waited forever. A resumed walk that fails again
without finishing anything clears its checkpoint, and the next walk is
full. A canceled walk keeps it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: do not resume a walk checkpoint left for a week

A checkpoint is cleared only by a later walk of the same shard set, so a
set that stopped walking, such as a one-off per-shard walk, left its
record behind, and a walk of that set much later would skip buckets
finished long before. Records now carry their save time and are not
resumed after seven days; a walk that keeps retrying saves on every
attempt and stays fresh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a failed walk without progress leaves its checkpoint alone

A pass canceled while it read the checkpoint failed the read, then
saved its own empty progress over the record it never loaded, so the
next pass started every bucket again. A failed walk that finished
nothing has nothing new to record and no longer writes; the stuck
resume reset still clears.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: space out walk checkpoint saves after a failed one

The save time advanced only on success, so once the filer refused a
periodic save every following entry tried again and logged another
warning. The interval now counts from the last attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a walk checkpoint save that errors may still be on disk

A save whose reply was lost left onDisk false, so a walk that then
completed skipped the clear and the next walk resumed from the stale
record instead of walking everything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: keep a stalled walk checkpoint for a day

A resumed walk that failed again without progress dropped its record
right away, so an outage spanning two attempts restarted the whole
walk. Treat the walk as stuck only once its record has not advanced
for a day.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a resumed walk counts its earlier buckets as walked

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: replay a resumed recovery walk from the walk that began it

A recovery or cold-start pass rewinds replay to runNow - maxTTL. When the
walk resumed a checkpoint, its finished buckets were listed in an earlier
attempt, so an object there that was not yet due then and whose write is
now older than maxTTL was neither expired by the walk nor replayed.

The checkpoint now keeps the start of the walk that began it, WalkBuckets
returns that time (its own start for a fresh walk), and runShard floors
the recovery rewind and the cold-start replay at it. A record without a
start is not resumed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a cold start that resumed a walk replays on the next pass

The pass's shared subscription starts at runNow - maxTTL for a shard
without a cursor, before its walk reports an earlier start, so flooring
the cold-start drain at the walk start could not deliver the older
events. Such a shard now saves its cursor at the walk start minus maxTTL
and returns, like the recovery branch, and the next pass replays the
window from that cursor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: test that a cursor below the floor opens the next subscription

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: test the cold-start replay across two Run passes

A cold start whose walk resumed an earlier attempt sees an older event
only on the next pass. The test runs Run twice against a filer that
serves events past the requested floor, with a later in-window event
that would otherwise move the cursor past the older one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:35:49 -08:00

514 lines
21 KiB
Go
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
package s3_lifecycle
import (
"context"
"errors"
"fmt"
"math"
"strconv"
"time"
"github.com/seaweedfs/seaweedfs/weed/credential"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
pluginworker "github.com/seaweedfs/seaweedfs/weed/plugin/worker"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/dailyrun"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/dispatcher"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/scheduler"
"github.com/seaweedfs/seaweedfs/weed/stats"
"golang.org/x/time/rate"
"google.golang.org/grpc"
)
func init() {
pluginworker.RegisterHandler(pluginworker.HandlerFactory{
JobType: jobType,
Category: pluginworker.CategoryDefault,
Aliases: []string{"s3-lifecycle", "s3.lifecycle", "lifecycle"},
Build: func(opts pluginworker.HandlerBuildOptions) (pluginworker.JobHandler, error) {
return NewHandler(opts.GrpcDialOption), nil
},
})
}
// Handler is the worker-side runner for S3 object lifecycle expiration.
// One Execute call drives one bounded dailyrun.Run pass against the S3
// endpoints discovered from the master; admin caps concurrency at one
// job per worker so a fresh proposal only spawns a new run after the
// prior one exits.
type Handler struct {
grpcDialOption grpc.DialOption
}
func NewHandler(grpcDialOption grpc.DialOption) *Handler {
return &Handler{grpcDialOption: grpcDialOption}
}
func (h *Handler) Capability() *plugin_pb.JobTypeCapability {
return &plugin_pb.JobTypeCapability{
JobType: jobType,
CanDetect: true,
CanExecute: true,
MaxDetectionConcurrency: 1,
MaxExecutionConcurrency: 1,
DisplayName: "S3 Lifecycle",
Description: "Daily batch: scan the filer meta-log and delete objects whose lifecycle rule has fired.",
Weight: 20,
}
}
func (h *Handler) Descriptor() *plugin_pb.JobTypeDescriptor {
return &plugin_pb.JobTypeDescriptor{
JobType: jobType,
DisplayName: "S3 Lifecycle",
Description: "Daily S3 object expiration scan with per-shard cursors and bounded retry.",
Icon: "fas fa-recycle",
DescriptorVersion: 1,
AdminConfigForm: &plugin_pb.ConfigForm{
FormId: "s3-lifecycle-admin",
Title: "S3 Lifecycle",
Description: "Cluster-wide delete-throughput cap.",
Sections: []*plugin_pb.ConfigSection{
{
SectionId: "scope",
Title: "Scope",
Description: "Cluster-wide delete-throughput cap.",
Fields: []*plugin_pb.ConfigField{
{
Name: ClusterDeletesPerSecondAdminKey,
Label: "Cluster Delete Rate (per second)",
Description: "Cluster-wide ceiling on lifecycle delete RPCs per second, divided evenly across active s3_lifecycle workers at job-dispatch time. 0 = unlimited.",
FieldType: plugin_pb.ConfigFieldType_CONFIG_FIELD_TYPE_INT64,
Widget: plugin_pb.ConfigWidget_CONFIG_WIDGET_NUMBER,
MinValue: &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
},
{
Name: ClusterDeletesBurstAdminKey,
Label: "Cluster Delete Burst",
Description: "Token-bucket burst capacity across the cluster (max simultaneous deletes). 0 = 2 × rate. Same allocation rule as the rate.",
FieldType: plugin_pb.ConfigFieldType_CONFIG_FIELD_TYPE_INT64,
Widget: plugin_pb.ConfigWidget_CONFIG_WIDGET_NUMBER,
MinValue: &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
},
{
Name: MetaLogRetentionDaysAdminKey,
Label: "Meta-Log Retention (days)",
Description: "How far back the filer's meta-log subscription can reach. Rules whose TTL exceeds this run via the walker; shrinking this value will trigger a one-time recovery walk on the next run for any rule that's now too old to replay. 0 = unbounded (no partition; every rule serviced by replay).",
FieldType: plugin_pb.ConfigFieldType_CONFIG_FIELD_TYPE_INT64,
Widget: plugin_pb.ConfigWidget_CONFIG_WIDGET_NUMBER,
MinValue: &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
},
{
Name: WalkerIntervalMinutesAdminKey,
Label: "Walker Interval (minutes)",
Description: "Minimum time between steady-state walker fires per shard. Cold-start and rule-change recovery walks ignore this — they run unconditionally. 0 = fire on every run (use when the worker is scheduled at the desired walk cadence, e.g. hourly). Set to a positive value when the worker runs at a tighter cadence than the desired walk frequency, to avoid hammering filer with a full subtree scan per run.",
FieldType: plugin_pb.ConfigFieldType_CONFIG_FIELD_TYPE_INT64,
Widget: plugin_pb.ConfigWidget_CONFIG_WIDGET_NUMBER,
MinValue: &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
},
},
},
},
DefaultValues: map[string]*plugin_pb.ConfigValue{
ClusterDeletesPerSecondAdminKey: {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
ClusterDeletesBurstAdminKey: {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
MetaLogRetentionDaysAdminKey: {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
WalkerIntervalMinutesAdminKey: {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
},
},
// WorkerConfigForm intentionally absent — the prior
// "Per-Run Time Limit (minutes)" knob duplicated the admin
// scheduler's Execution Timeout (both wall-clock caps on the
// same Execute call), and operators had to keep the two values
// in agreement. Removed in favor of a single source of truth:
// AdminRuntimeDefaults.ExecutionTimeoutSeconds below.
AdminRuntimeDefaults: &plugin_pb.AdminRuntimeDefaults{
// On by default: S3 lifecycle is a standard bucket feature
// (PutBucketLifecycleConfiguration is part of the S3 API),
// and a bucket with rules set but no worker running silently
// retains data past its declared expiration. Operators who
// want the worker off can still disable it in the admin UI;
// the default error is "data lingers" not "worker burns CPU
// on empty rule sets" (the worker fast-exits with no
// configured rules).
Enabled: true,
DetectionIntervalMinutes: 24 * 60, // daily
DetectionTimeoutSeconds: 60,
MaxJobsPerDetection: 1,
// Effectively no per-pass wall-clock cap. Lifecycle is a
// scheduled batch — its natural duration is "as long as it
// takes to process today's events." The scheduler's global
// 90s default would kill every real run, and a numeric
// cap operators have to estimate (1h? 8h?) is a recurring
// footgun: too low truncates a legitimate large-bucket
// pass; too high makes the value meaningless.
//
// Use math.MaxInt32 seconds (~68 years) for both knobs to
// say "no timeout in practice" in code-review-readable form.
// Operators who genuinely want a cap can set one in the
// admin UI; the underlying context.WithTimeout machinery
// is unchanged.
ExecutionTimeoutSeconds: math.MaxInt32,
JobTypeMaxRuntimeSeconds: math.MaxInt32,
},
}
}
func (h *Handler) Detect(ctx context.Context, request *plugin_pb.RunDetectionRequest, sender pluginworker.DetectionSender) error {
if request == nil || sender == nil {
return fmt.Errorf("detect: nil request or sender")
}
if request.JobType != "" && request.JobType != jobType {
return fmt.Errorf("job type %q is not handled by s3 lifecycle handler", request.JobType)
}
s3Endpoints := clusterS3Endpoints(request.ClusterContext)
if len(s3Endpoints) == 0 {
_ = sender.SendActivity(pluginworker.BuildDetectorActivity("skipped", "no s3 servers registered with master", nil))
return sender.SendComplete(&plugin_pb.DetectionComplete{JobType: jobType, Success: true})
}
filerAddresses := []string{}
if request.ClusterContext != nil {
filerAddresses = append(filerAddresses, request.ClusterContext.FilerAddresses...)
}
if len(filerAddresses) == 0 {
_ = sender.SendActivity(pluginworker.BuildDetectorActivity("skipped", "no filer addresses in cluster context", nil))
return sender.SendComplete(&plugin_pb.DetectionComplete{JobType: jobType, Success: true})
}
// FilerAddresses are pb.ServerAddress strings (host:httpPort.grpcPort);
// Execute dials filer_grpc_address verbatim, so resolve it to a gRPC address.
filerGrpcAddress := pb.ServerAddress(filerAddresses[0]).ToGrpcAddress()
proposal := &plugin_pb.JobProposal{
JobType: jobType,
ProposalId: fmt.Sprintf("s3-lifecycle-%d", time.Now().UnixNano()),
Priority: plugin_pb.JobPriority_JOB_PRIORITY_NORMAL,
Parameters: map[string]*plugin_pb.ConfigValue{
"filer_grpc_address": {Kind: &plugin_pb.ConfigValue_StringValue{StringValue: filerGrpcAddress}},
},
}
if err := sender.SendProposals(&plugin_pb.DetectionProposals{
JobType: jobType,
Proposals: []*plugin_pb.JobProposal{proposal},
}); err != nil {
return err
}
return sender.SendComplete(&plugin_pb.DetectionComplete{
JobType: jobType,
Success: true,
TotalProposals: 1,
})
}
func (h *Handler) Execute(ctx context.Context, request *plugin_pb.ExecuteJobRequest, sender pluginworker.ExecutionSender) error {
if request == nil || request.Job == nil || sender == nil {
return fmt.Errorf("execute: nil request/job/sender")
}
if request.Job.JobType != "" && request.Job.JobType != jobType {
return fmt.Errorf("job type %q is not handled by s3 lifecycle handler", request.Job.JobType)
}
cfg := ParseConfig(request.GetAdminConfigValues(), request.GetWorkerConfigValues())
s3Endpoints := clusterS3Endpoints(request.ClusterContext)
if len(s3Endpoints) == 0 {
return fmt.Errorf("execute: no s3 servers registered with master")
}
filerAddress := readString(request.Job.Parameters, "filer_grpc_address", "")
if filerAddress == "" {
return fmt.Errorf("execute: missing filer_grpc_address in job parameters")
}
// Run lifetime is bounded by the scheduler's Execution Timeout
// (admin UI). The handler used to wrap ctx in another
// context.WithTimeout(cfg.MaxRuntime), but that doubled the
// concept — both knobs were wall-clock caps on the same Execute
// call, and the smaller one always won (typically the 90s
// scheduler default would clobber a 60-min worker setting).
// Single source of truth is now AdminRuntimeDefaults.ExecutionTimeoutSeconds.
runCtx := ctx
_ = sender.SendProgress(&plugin_pb.JobProgressUpdate{
JobId: request.Job.JobId, JobType: jobType,
State: plugin_pb.JobState_JOB_STATE_RUNNING, Stage: "starting",
Message: fmt.Sprintf("scheduler workers=%d s3=%v", cfg.Workers, s3Endpoints),
})
dialCtx, dialCancel := context.WithTimeout(runCtx, 30*time.Second)
filerConn, err := pb.GrpcDial(dialCtx, filerAddress, false, h.grpcDialOption)
dialCancel()
if err != nil {
return fmt.Errorf("dial filer %s: %w", filerAddress, err)
}
defer filerConn.Close()
filerClient := filer_pb.NewSeaweedFilerClient(filerConn)
bucketsPath, err := lookupBucketsPath(runCtx, filerClient)
if err != nil {
return fmt.Errorf("buckets path: %w", err)
}
s3Conn, s3Endpoint, skipped := h.selectS3Front(runCtx, s3Endpoints)
if s3Conn == nil {
return noReadyS3Error(skipped)
}
defer s3Conn.Close()
if len(skipped) > 0 {
skippedErr := errors.Join(skipped...)
glog.Warningf("s3 lifecycle: using s3 %s, skipped: %v", s3Endpoint, skippedErr)
_ = sender.SendProgress(&plugin_pb.JobProgressUpdate{
JobId: request.Job.JobId, JobType: jobType,
State: plugin_pb.JobState_JOB_STATE_RUNNING, Stage: "starting",
Message: fmt.Sprintf("using s3 %s, skipped: %v", s3Endpoint, skippedErr),
})
}
rpc := s3_lifecycle_pb.NewSeaweedS3LifecycleInternalClient(s3Conn)
report, err := h.executeDailyReplay(runCtx, request, bucketsPath, filerClient, rpc, cfg, sender)
if err != nil {
return err
}
return sendCompletion(request, sender, report)
}
// JobCompleted must set JobType to the handler's jobType constant;
// routed requests may leave request.Job.JobType empty, and the admin
// ignores completions with empty JobType. Failed passes carry their
// result too, so every bucket failure stays in the job record.
func sendCompletion(request *plugin_pb.ExecuteJobRequest, sender pluginworker.ExecutionSender, report *runReport) error {
return sender.SendCompleted(&plugin_pb.JobCompleted{
JobId: request.Job.JobId,
JobType: jobType,
Success: report.err == nil,
ErrorMessage: report.errorMessage(),
Result: &plugin_pb.JobResult{
Summary: report.summaryLine(),
OutputValues: report.outputValues(),
},
})
}
// executeDailyReplay runs one bounded daily-replay pass via
// dailyrun.Run. The walker fires inside runShard on rule-content edits
// and against the steady-state walk view; all rule kinds are serviced.
// The returned error covers setup only; the pass's own error is in the
// report.
func (h *Handler) executeDailyReplay(ctx context.Context, request *plugin_pb.ExecuteJobRequest, bucketsPath string, filerClient filer_pb.SeaweedFilerClient, rpc s3_lifecycle_pb.SeaweedS3LifecycleInternalClient, cfg Config, sender pluginworker.ExecutionSender) (*runReport, error) {
startedAt := time.Now()
eng := engine.New()
inputs, parseErrors, err := scheduler.LoadCompileInputs(ctx, filerClient, bucketsPath)
if err != nil {
return nil, fmt.Errorf("daily_replay: load lifecycle inputs: %w", err)
}
for _, pe := range parseErrors {
glog.Warningf("daily_replay: invalid lifecycle policy on bucket %s: %v", pe.Bucket, pe.Err)
}
eng.Compile(inputs, engine.CompileOptions{PriorStates: scheduler.AllActivePriorStates(inputs)})
shards := make([]int, 0, s3lifecycle.ShardCount)
for i := 0; i < s3lifecycle.ShardCount; i++ {
shards = append(shards, i)
}
limiter, limiterDesc := buildLimiterFromClusterContext(request.GetClusterContext())
// Reuse one LifecycleClient across the replay drain and the
// walker's per-entry dispatch.
client := lifecycleRPCAdapter{c: rpc}
// Bucket list for the walker — derived from inputs so it matches
// the snapshot the engine compiled.
buckets := make([]string, 0, len(inputs))
for _, in := range inputs {
if in.Bucket != "" {
buckets = append(buckets, in.Bucket)
}
}
persister := &dailyrun.FilerCursorPersister{Store: dispatcher.NewFilerStoreClient(filerClient)}
report := &runReport{bucketsTotal: len(buckets) + len(parseErrors), invalidPolicies: parseErrors}
loadLastFullPass(ctx, persister, report)
runStats := &dailyrun.RunStats{}
// Share the limiter with processMatches so walker + replay can't
// combine to burst past the cluster cap.
walkerDispatch := &dailyrun.WalkerDispatcher{Client: client, Limiter: limiter, Stats: runStats}
walker := dailyrun.WalkerFunc(func(walkCtx context.Context, view *engine.Snapshot, shards []int) (time.Time, error) {
return dailyrun.WalkBuckets(walkCtx, view, shards, buckets,
dailyrun.FilerListFunc(runStats.CountListed(filerClient), bucketsPath, view), walkerDispatch, runStats, persister)
})
_ = sender.SendProgress(&plugin_pb.JobProgressUpdate{
JobId: request.Job.JobId, JobType: jobType,
State: plugin_pb.JobState_JOB_STATE_RUNNING, Stage: "starting",
Message: fmt.Sprintf("daily_replay shards=%d workers=%d rate=%s buckets=%d invalid_policies=%d walker=on",
len(shards), cfg.Workers, limiterDesc, len(buckets), len(parseErrors)),
})
stopProgress := startProgressReporter(request, sender, runStats.Summary, progressInterval)
report.err = dailyrun.Run(ctx, dailyrun.Config{
Shards: shards,
BucketsPath: bucketsPath,
Engine: eng,
FilerClient: filerClient,
Client: client,
Persister: persister,
Lister: dispatcher.NewFilerSiblingLister(filerClient, bucketsPath),
Workers: cfg.Workers,
Limiter: limiter,
RetentionWindow: cfg.MetaLogRetention,
Walker: walker,
WalkerInterval: cfg.WalkerInterval,
ClientName: "worker-s3-lifecycle-daily",
Stats: runStats,
})
stopProgress()
report.canceled = ctx.Err() != nil
report.elapsed = time.Since(startedAt)
report.summary = runStats.Summary()
if report.err != nil {
glog.Warningf("daily_replay: %v", report.err)
}
recordFullPass(ctx, persister, report, startedAt)
return report, nil
}
func loadLastFullPass(ctx context.Context, persister *dailyrun.FilerCursorPersister, report *runReport) {
fp, found, err := persister.LoadFullPass(ctx)
if err != nil {
glog.Warningf("daily_replay: %v", err)
return
}
if found {
report.lastFullPass = fp.CompletedAt
publishFullPass(fp)
}
}
// recordFullPass advances the full-pass record only when the pass
// covered every shard and every policy.
func recordFullPass(ctx context.Context, persister *dailyrun.FilerCursorPersister, report *runReport, startedAt time.Time) {
if !report.fullPass() {
return
}
fp := dailyrun.FullPass{StartedAt: startedAt.UTC(), CompletedAt: time.Now().UTC()}
saveCtx, saveCancel := context.WithTimeout(ctx, 10*time.Second)
defer saveCancel()
if err := persister.SaveFullPass(saveCtx, fp); err != nil {
glog.Warningf("daily_replay: %v", err)
report.fullPassSaveErr = err
return
}
report.lastFullPass = fp.CompletedAt
publishFullPass(fp)
}
func publishFullPass(fp dailyrun.FullPass) {
stats.S3LifecycleDailyRunLastFullPassTimestampSeconds.Set(float64(fp.CompletedAt.UnixNano()) / 1e9)
}
// buildLimiterFromClusterContext parses the per-worker share the admin
// wrote into ClusterContext.Metadata (see weed/admin/plugin/plugin.go's
// s3_lifecycle injection) and returns a rate.Limiter, or nil when no
// rate cap applies. The description string is for the JobProgressUpdate
// so operators can see "rate=unlimited" / "rate=12.5/s burst=25" in
// the activity log.
//
// Tolerant of missing keys, empty strings, malformed numbers, and
// non-positive values — all treated as "no limit" rather than failing
// the run. The admin allocator is the single point that decides whether
// to populate these keys; the worker doesn't second-guess.
func buildLimiterFromClusterContext(cc *plugin_pb.ClusterContext) (*rate.Limiter, string) {
if cc == nil || cc.Metadata == nil {
return nil, "unlimited"
}
rps, ok := parsePositiveFloat(cc.Metadata[MetadataKeyDeletesPerSecond])
if !ok {
return nil, "unlimited"
}
burst, _ := parsePositiveInt(cc.Metadata[MetadataKeyDeletesBurst])
if burst <= 0 {
// Sensible default: enough headroom for one tick's worth of
// throughput. Caller may also supply 0 to opt into this default.
burst = int(rps * 2)
if burst < 1 {
burst = 1
}
}
return rate.NewLimiter(rate.Limit(rps), burst), fmt.Sprintf("%.3g/s burst=%d", rps, burst)
}
func parsePositiveFloat(s string) (float64, bool) {
if s == "" {
return 0, false
}
v, err := strconv.ParseFloat(s, 64)
if err != nil || v <= 0 {
return 0, false
}
return v, true
}
func parsePositiveInt(s string) (int, bool) {
if s == "" {
return 0, false
}
v, err := strconv.Atoi(s)
if err != nil || v <= 0 {
return 0, false
}
return v, true
}
// clusterS3Endpoints returns the master-discovered S3 gRPC addresses for the
// cluster. The handler uses the first one that passes checkLifecycleReady;
// the master refreshes the list on KeepConnected so a stale entry self-heals
// on the next run.
func clusterS3Endpoints(cc *plugin_pb.ClusterContext) []string {
if cc == nil {
return nil
}
out := make([]string, 0, len(cc.S3GrpcAddresses))
for _, addr := range cc.S3GrpcAddresses {
if addr != "" {
out = append(out, addr)
}
}
return out
}
type lifecycleRPCAdapter struct {
c s3_lifecycle_pb.SeaweedS3LifecycleInternalClient
}
func (a lifecycleRPCAdapter) LifecycleDelete(ctx context.Context, req *s3_lifecycle_pb.LifecycleDeleteRequest) (*s3_lifecycle_pb.LifecycleDeleteResponse, error) {
ctx, _ = credential.WithS3InternalAdminAuth(ctx)
return a.c.LifecycleDelete(ctx, req)
}
func lookupBucketsPath(ctx context.Context, client filer_pb.SeaweedFilerClient) (string, error) {
resp, err := client.GetFilerConfiguration(ctx, &filer_pb.GetFilerConfigurationRequest{})
if err != nil {
return "", err
}
if path := resp.GetDirBuckets(); path != "" {
return path, nil
}
return "/buckets", nil
}
func readString(values map[string]*plugin_pb.ConfigValue, field, fallback string) string {
v, ok := values[field]
if !ok || v == nil {
return fallback
}
if k, ok := v.Kind.(*plugin_pb.ConfigValue_StringValue); ok {
return k.StringValue
}
return fallback
}
var _ pluginworker.JobHandler = (*Handler)(nil)