mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-16 00:38:53 +01:00
* s3 lifecycle: bound each LifecycleDelete attempt with its own deadline Replay and walker deletes used the job context with no request deadline, so one stalled unary RPC could hold a shard for the whole job. Each attempt now runs under DeleteTimeout (default 1m), set after the limiter wait. A timeout while the job is still live is retried like a transport error and then halts the shard or walk without advancing; parent cancellation still stops both the attempt and the backoff. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: admin setting for the delete RPC timeout Adds delete_timeout_seconds to the S3 Lifecycle admin form and passes it to both the replay drain and the walker. 0 keeps the 60s default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: describe replay and walker timeouts separately in the help text Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: fail the pass when a replay delete exhausts its attempts A transport error or attempt timeout that outlived the retries halted the shard with a nil error, so the job reported success. processMatches now returns the error (the cursor still holds at the event), and runShard treats a drain error as a clean stop only when its own context ended, since an attempt timeout also wraps DeadlineExceeded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test that a replayed LifecycleDelete resolves after a lost reply Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: skip the expiration write once the attempt deadline passed A worker retries a LifecycleDelete whose attempt timed out, while the first attempt can still be running on the S3 server. An attempt that stalled before its marker write would then stack a second delete marker after the retry wrote one. The gRPC deadline reaches the handler context, so check it right before the expiration writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: recheck the attempt deadline after the suspended null delete The null-version delete can block on the owner filer past the attempt deadline; the worker may have retried and written a marker by then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: finish the suspended marker once the null version is deleted Skipping the marker after a successful null delete left an older version as the latest, and the retry then resolved as a no-op without hiding it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: let a routed delete marker commit only while its latest guard holds createDeleteMarkerIf passes an optional condition, evaluated on the .versions directory entry under the object's lock on the owner filer, and reports a refused commit as errLatestVersionChanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: write the expiration marker as a compare-and-set on the latest pointer A timed-out attempt can still be running on the S3 server while the worker's retry writes the marker. The marker commit is now conditioned on the .versions pointer as read just before it and skipped when the latest is already a delete marker, so whichever attempt commits second writes nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: resolve the marker as stale when another version became latest The marker's compare-and-set only covered changes after its own pointer read, so a version promoted after the expiration checks passed could be hidden by the marker. The enabled path now passes the validated version and a conflict retry that finds a different named latest resolves the request as VERSION_NOT_LATEST. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: resolve a null expiration as stale once a version takes the pointer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: read the marker's latest version the way getLatestObjectVersion does A pointer names a version only with its file name; an id alone falls back to the null object, and the null-is-latest signal makes the null version current. An expiration that validated a named version no longer hides a null written by a suspended PUT, and a current null under an id-only pointer now expires instead of resolving as stale. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: hold the object write lock for an unrouted expiration marker Off the ring createDeleteMarkerIf cannot carry the pointer guard, so a PUT committing between the pointer read and the marker write was hidden. The read and the write now run under the object write lock that unrouted writers take. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: share transient-error retries between walker and replay deletes (#12649) * s3 lifecycle: retry only transient LifecycleDelete failures dispatchWithRetry retried every RPC error except parent cancellation, so an authentication, permission or argument error spent the whole retry budget before surfacing. Retry only attempt timeouts and the failures util.IsTransientError recognizes (Unavailable, ResourceExhausted, network resets); everything else returns on the first attempt. A nil response is now an error instead of a nil dereference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: take a delete-rate token for every LifecycleDelete attempt processMatches waited on the shared limiter once per match, so the retries behind it ran outside the cluster-wide deletes/sec budget. The retry loop now waits on the limiter before each attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: give walker deletes the replay retry policy WalkerDispatcher sent each LifecycleDelete once, so one dropped connection aborted the bucket's daily walk while the same failure on the replay path was retried. The walker now goes through deleteWithRetry and shares its transient-error retries, per-attempt limiter wait and nil response handling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: retry a DeadlineExceeded the server sends back A server or proxy can end the RPC with DeadlineExceeded while the attempt's own deadline is still running. util.IsTransientError rejects that status, so the delete halted with retries left. It is a timed-out attempt like the client-side one, so retry it the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: say walker deletes get 3 attempts in the timeout help Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
526 lines
21 KiB
Go
526 lines
21 KiB
Go
package s3_lifecycle
|
||
|
||
import (
|
||
"context"
|
||
"errors"
|
||
"fmt"
|
||
"math"
|
||
"strconv"
|
||
"time"
|
||
|
||
"github.com/seaweedfs/seaweedfs/weed/credential"
|
||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||
"github.com/seaweedfs/seaweedfs/weed/pb"
|
||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
|
||
"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
|
||
pluginworker "github.com/seaweedfs/seaweedfs/weed/plugin/worker"
|
||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
|
||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/dailyrun"
|
||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/dispatcher"
|
||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine"
|
||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/scheduler"
|
||
"github.com/seaweedfs/seaweedfs/weed/stats"
|
||
"golang.org/x/time/rate"
|
||
"google.golang.org/grpc"
|
||
)
|
||
|
||
func init() {
|
||
pluginworker.RegisterHandler(pluginworker.HandlerFactory{
|
||
JobType: jobType,
|
||
Category: pluginworker.CategoryDefault,
|
||
Aliases: []string{"s3-lifecycle", "s3.lifecycle", "lifecycle"},
|
||
Build: func(opts pluginworker.HandlerBuildOptions) (pluginworker.JobHandler, error) {
|
||
return NewHandler(opts.GrpcDialOption), nil
|
||
},
|
||
})
|
||
}
|
||
|
||
// Handler is the worker-side runner for S3 object lifecycle expiration.
|
||
// One Execute call drives one bounded dailyrun.Run pass against the S3
|
||
// endpoints discovered from the master; admin caps concurrency at one
|
||
// job per worker so a fresh proposal only spawns a new run after the
|
||
// prior one exits.
|
||
type Handler struct {
|
||
grpcDialOption grpc.DialOption
|
||
}
|
||
|
||
func NewHandler(grpcDialOption grpc.DialOption) *Handler {
|
||
return &Handler{grpcDialOption: grpcDialOption}
|
||
}
|
||
|
||
func (h *Handler) Capability() *plugin_pb.JobTypeCapability {
|
||
return &plugin_pb.JobTypeCapability{
|
||
JobType: jobType,
|
||
CanDetect: true,
|
||
CanExecute: true,
|
||
MaxDetectionConcurrency: 1,
|
||
MaxExecutionConcurrency: 1,
|
||
DisplayName: "S3 Lifecycle",
|
||
Description: "Daily batch: scan the filer meta-log and delete objects whose lifecycle rule has fired.",
|
||
Weight: 20,
|
||
}
|
||
}
|
||
|
||
func (h *Handler) Descriptor() *plugin_pb.JobTypeDescriptor {
|
||
return &plugin_pb.JobTypeDescriptor{
|
||
JobType: jobType,
|
||
DisplayName: "S3 Lifecycle",
|
||
Description: "Daily S3 object expiration scan with per-shard cursors and bounded retry.",
|
||
Icon: "fas fa-recycle",
|
||
DescriptorVersion: 1,
|
||
AdminConfigForm: &plugin_pb.ConfigForm{
|
||
FormId: "s3-lifecycle-admin",
|
||
Title: "S3 Lifecycle",
|
||
Description: "Cluster-wide delete-throughput cap.",
|
||
Sections: []*plugin_pb.ConfigSection{
|
||
{
|
||
SectionId: "scope",
|
||
Title: "Scope",
|
||
Description: "Cluster-wide delete-throughput cap.",
|
||
Fields: []*plugin_pb.ConfigField{
|
||
{
|
||
Name: ClusterDeletesPerSecondAdminKey,
|
||
Label: "Cluster Delete Rate (per second)",
|
||
Description: "Cluster-wide ceiling on lifecycle delete RPCs per second, divided evenly across active s3_lifecycle workers at job-dispatch time. 0 = unlimited.",
|
||
FieldType: plugin_pb.ConfigFieldType_CONFIG_FIELD_TYPE_INT64,
|
||
Widget: plugin_pb.ConfigWidget_CONFIG_WIDGET_NUMBER,
|
||
MinValue: &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
|
||
},
|
||
{
|
||
Name: ClusterDeletesBurstAdminKey,
|
||
Label: "Cluster Delete Burst",
|
||
Description: "Token-bucket burst capacity across the cluster (max simultaneous deletes). 0 = 2 × rate. Same allocation rule as the rate.",
|
||
FieldType: plugin_pb.ConfigFieldType_CONFIG_FIELD_TYPE_INT64,
|
||
Widget: plugin_pb.ConfigWidget_CONFIG_WIDGET_NUMBER,
|
||
MinValue: &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
|
||
},
|
||
{
|
||
Name: MetaLogRetentionDaysAdminKey,
|
||
Label: "Meta-Log Retention (days)",
|
||
Description: "How far back the filer's meta-log subscription can reach. Rules whose TTL exceeds this run via the walker; shrinking this value will trigger a one-time recovery walk on the next run for any rule that's now too old to replay. 0 = unbounded (no partition; every rule serviced by replay).",
|
||
FieldType: plugin_pb.ConfigFieldType_CONFIG_FIELD_TYPE_INT64,
|
||
Widget: plugin_pb.ConfigWidget_CONFIG_WIDGET_NUMBER,
|
||
MinValue: &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
|
||
},
|
||
{
|
||
Name: WalkerIntervalMinutesAdminKey,
|
||
Label: "Walker Interval (minutes)",
|
||
Description: "Minimum time between steady-state walker fires per shard. Cold-start and rule-change recovery walks ignore this — they run unconditionally. 0 = fire on every run (use when the worker is scheduled at the desired walk cadence, e.g. hourly). Set to a positive value when the worker runs at a tighter cadence than the desired walk frequency, to avoid hammering filer with a full subtree scan per run.",
|
||
FieldType: plugin_pb.ConfigFieldType_CONFIG_FIELD_TYPE_INT64,
|
||
Widget: plugin_pb.ConfigWidget_CONFIG_WIDGET_NUMBER,
|
||
MinValue: &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
|
||
},
|
||
{
|
||
Name: DeleteTimeoutSecondsAdminKey,
|
||
Label: "Delete RPC Timeout (seconds)",
|
||
Description: "Deadline for each lifecycle delete RPC to an S3 server, counted after the rate-limit wait. Replay and walker deletes get up to 3 attempts. After that a replay shard stops at that event and resumes there on the next run, and a walk fails and runs again from the start on the next run. 0 = 60 seconds.",
|
||
FieldType: plugin_pb.ConfigFieldType_CONFIG_FIELD_TYPE_INT64,
|
||
Widget: plugin_pb.ConfigWidget_CONFIG_WIDGET_NUMBER,
|
||
MinValue: &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
|
||
},
|
||
},
|
||
},
|
||
},
|
||
DefaultValues: map[string]*plugin_pb.ConfigValue{
|
||
ClusterDeletesPerSecondAdminKey: {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
|
||
ClusterDeletesBurstAdminKey: {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
|
||
MetaLogRetentionDaysAdminKey: {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
|
||
WalkerIntervalMinutesAdminKey: {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
|
||
DeleteTimeoutSecondsAdminKey: {Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: 0}},
|
||
},
|
||
},
|
||
// WorkerConfigForm intentionally absent — the prior
|
||
// "Per-Run Time Limit (minutes)" knob duplicated the admin
|
||
// scheduler's Execution Timeout (both wall-clock caps on the
|
||
// same Execute call), and operators had to keep the two values
|
||
// in agreement. Removed in favor of a single source of truth:
|
||
// AdminRuntimeDefaults.ExecutionTimeoutSeconds below.
|
||
AdminRuntimeDefaults: &plugin_pb.AdminRuntimeDefaults{
|
||
// On by default: S3 lifecycle is a standard bucket feature
|
||
// (PutBucketLifecycleConfiguration is part of the S3 API),
|
||
// and a bucket with rules set but no worker running silently
|
||
// retains data past its declared expiration. Operators who
|
||
// want the worker off can still disable it in the admin UI;
|
||
// the default error is "data lingers" not "worker burns CPU
|
||
// on empty rule sets" (the worker fast-exits with no
|
||
// configured rules).
|
||
Enabled: true,
|
||
DetectionIntervalMinutes: 24 * 60, // daily
|
||
DetectionTimeoutSeconds: 60,
|
||
MaxJobsPerDetection: 1,
|
||
// Effectively no per-pass wall-clock cap. Lifecycle is a
|
||
// scheduled batch — its natural duration is "as long as it
|
||
// takes to process today's events." The scheduler's global
|
||
// 90s default would kill every real run, and a numeric
|
||
// cap operators have to estimate (1h? 8h?) is a recurring
|
||
// footgun: too low truncates a legitimate large-bucket
|
||
// pass; too high makes the value meaningless.
|
||
//
|
||
// Use math.MaxInt32 seconds (~68 years) for both knobs to
|
||
// say "no timeout in practice" in code-review-readable form.
|
||
// Operators who genuinely want a cap can set one in the
|
||
// admin UI; the underlying context.WithTimeout machinery
|
||
// is unchanged.
|
||
ExecutionTimeoutSeconds: math.MaxInt32,
|
||
JobTypeMaxRuntimeSeconds: math.MaxInt32,
|
||
},
|
||
}
|
||
}
|
||
|
||
func (h *Handler) Detect(ctx context.Context, request *plugin_pb.RunDetectionRequest, sender pluginworker.DetectionSender) error {
|
||
if request == nil || sender == nil {
|
||
return fmt.Errorf("detect: nil request or sender")
|
||
}
|
||
if request.JobType != "" && request.JobType != jobType {
|
||
return fmt.Errorf("job type %q is not handled by s3 lifecycle handler", request.JobType)
|
||
}
|
||
s3Endpoints := clusterS3Endpoints(request.ClusterContext)
|
||
if len(s3Endpoints) == 0 {
|
||
_ = sender.SendActivity(pluginworker.BuildDetectorActivity("skipped", "no s3 servers registered with master", nil))
|
||
return sender.SendComplete(&plugin_pb.DetectionComplete{JobType: jobType, Success: true})
|
||
}
|
||
filerAddresses := []string{}
|
||
if request.ClusterContext != nil {
|
||
filerAddresses = append(filerAddresses, request.ClusterContext.FilerAddresses...)
|
||
}
|
||
if len(filerAddresses) == 0 {
|
||
_ = sender.SendActivity(pluginworker.BuildDetectorActivity("skipped", "no filer addresses in cluster context", nil))
|
||
return sender.SendComplete(&plugin_pb.DetectionComplete{JobType: jobType, Success: true})
|
||
}
|
||
|
||
// FilerAddresses are pb.ServerAddress strings (host:httpPort.grpcPort);
|
||
// Execute dials filer_grpc_address verbatim, so resolve it to a gRPC address.
|
||
filerGrpcAddress := pb.ServerAddress(filerAddresses[0]).ToGrpcAddress()
|
||
proposal := &plugin_pb.JobProposal{
|
||
JobType: jobType,
|
||
ProposalId: fmt.Sprintf("s3-lifecycle-%d", time.Now().UnixNano()),
|
||
Priority: plugin_pb.JobPriority_JOB_PRIORITY_NORMAL,
|
||
Parameters: map[string]*plugin_pb.ConfigValue{
|
||
"filer_grpc_address": {Kind: &plugin_pb.ConfigValue_StringValue{StringValue: filerGrpcAddress}},
|
||
},
|
||
}
|
||
if err := sender.SendProposals(&plugin_pb.DetectionProposals{
|
||
JobType: jobType,
|
||
Proposals: []*plugin_pb.JobProposal{proposal},
|
||
}); err != nil {
|
||
return err
|
||
}
|
||
return sender.SendComplete(&plugin_pb.DetectionComplete{
|
||
JobType: jobType,
|
||
Success: true,
|
||
TotalProposals: 1,
|
||
})
|
||
}
|
||
|
||
func (h *Handler) Execute(ctx context.Context, request *plugin_pb.ExecuteJobRequest, sender pluginworker.ExecutionSender) error {
|
||
if request == nil || request.Job == nil || sender == nil {
|
||
return fmt.Errorf("execute: nil request/job/sender")
|
||
}
|
||
if request.Job.JobType != "" && request.Job.JobType != jobType {
|
||
return fmt.Errorf("job type %q is not handled by s3 lifecycle handler", request.Job.JobType)
|
||
}
|
||
cfg := ParseConfig(request.GetAdminConfigValues(), request.GetWorkerConfigValues())
|
||
s3Endpoints := clusterS3Endpoints(request.ClusterContext)
|
||
if len(s3Endpoints) == 0 {
|
||
return fmt.Errorf("execute: no s3 servers registered with master")
|
||
}
|
||
filerAddress := readString(request.Job.Parameters, "filer_grpc_address", "")
|
||
if filerAddress == "" {
|
||
return fmt.Errorf("execute: missing filer_grpc_address in job parameters")
|
||
}
|
||
|
||
// Run lifetime is bounded by the scheduler's Execution Timeout
|
||
// (admin UI). The handler used to wrap ctx in another
|
||
// context.WithTimeout(cfg.MaxRuntime), but that doubled the
|
||
// concept — both knobs were wall-clock caps on the same Execute
|
||
// call, and the smaller one always won (typically the 90s
|
||
// scheduler default would clobber a 60-min worker setting).
|
||
// Single source of truth is now AdminRuntimeDefaults.ExecutionTimeoutSeconds.
|
||
runCtx := ctx
|
||
|
||
_ = sender.SendProgress(&plugin_pb.JobProgressUpdate{
|
||
JobId: request.Job.JobId, JobType: jobType,
|
||
State: plugin_pb.JobState_JOB_STATE_RUNNING, Stage: "starting",
|
||
Message: fmt.Sprintf("scheduler workers=%d s3=%v", cfg.Workers, s3Endpoints),
|
||
})
|
||
|
||
dialCtx, dialCancel := context.WithTimeout(runCtx, 30*time.Second)
|
||
filerConn, err := pb.GrpcDial(dialCtx, filerAddress, false, h.grpcDialOption)
|
||
dialCancel()
|
||
if err != nil {
|
||
return fmt.Errorf("dial filer %s: %w", filerAddress, err)
|
||
}
|
||
defer filerConn.Close()
|
||
filerClient := filer_pb.NewSeaweedFilerClient(filerConn)
|
||
|
||
bucketsPath, err := lookupBucketsPath(runCtx, filerClient)
|
||
if err != nil {
|
||
return fmt.Errorf("buckets path: %w", err)
|
||
}
|
||
|
||
s3Conn, s3Endpoint, skipped := h.selectS3Front(runCtx, s3Endpoints)
|
||
if s3Conn == nil {
|
||
return noReadyS3Error(skipped)
|
||
}
|
||
client := h.newFailoverClient(runCtx, s3Conn, s3Endpoint, s3Endpoints, func(msg string) {
|
||
_ = sender.SendProgress(&plugin_pb.JobProgressUpdate{
|
||
JobId: request.Job.JobId, JobType: jobType,
|
||
State: plugin_pb.JobState_JOB_STATE_RUNNING, Stage: "running",
|
||
Message: msg,
|
||
})
|
||
})
|
||
defer client.Close()
|
||
if len(skipped) > 0 {
|
||
skippedErr := errors.Join(skipped...)
|
||
glog.Warningf("s3 lifecycle: using s3 %s, skipped: %v", s3Endpoint, skippedErr)
|
||
_ = sender.SendProgress(&plugin_pb.JobProgressUpdate{
|
||
JobId: request.Job.JobId, JobType: jobType,
|
||
State: plugin_pb.JobState_JOB_STATE_RUNNING, Stage: "starting",
|
||
Message: fmt.Sprintf("using s3 %s, skipped: %v", s3Endpoint, skippedErr),
|
||
})
|
||
}
|
||
|
||
report, err := h.executeDailyReplay(runCtx, request, bucketsPath, filerClient, client, cfg, sender)
|
||
if err != nil {
|
||
return err
|
||
}
|
||
|
||
return sendCompletion(request, sender, report)
|
||
}
|
||
|
||
// JobCompleted must set JobType to the handler's jobType constant;
|
||
// routed requests may leave request.Job.JobType empty, and the admin
|
||
// ignores completions with empty JobType. Failed passes carry their
|
||
// result too, so every bucket failure stays in the job record.
|
||
func sendCompletion(request *plugin_pb.ExecuteJobRequest, sender pluginworker.ExecutionSender, report *runReport) error {
|
||
return sender.SendCompleted(&plugin_pb.JobCompleted{
|
||
JobId: request.Job.JobId,
|
||
JobType: jobType,
|
||
Success: report.err == nil,
|
||
ErrorMessage: report.errorMessage(),
|
||
Result: &plugin_pb.JobResult{
|
||
Summary: report.summaryLine(),
|
||
OutputValues: report.outputValues(),
|
||
},
|
||
})
|
||
}
|
||
|
||
// executeDailyReplay runs one bounded daily-replay pass via
|
||
// dailyrun.Run. The walker fires inside runShard on rule-content edits
|
||
// and against the steady-state walk view; all rule kinds are serviced.
|
||
// The returned error covers setup only; the pass's own error is in the
|
||
// report.
|
||
func (h *Handler) executeDailyReplay(ctx context.Context, request *plugin_pb.ExecuteJobRequest, bucketsPath string, filerClient filer_pb.SeaweedFilerClient, client dailyrun.LifecycleClient, cfg Config, sender pluginworker.ExecutionSender) (*runReport, error) {
|
||
startedAt := time.Now()
|
||
eng := engine.New()
|
||
inputs, parseErrors, err := scheduler.LoadCompileInputs(ctx, filerClient, bucketsPath)
|
||
if err != nil {
|
||
return nil, fmt.Errorf("daily_replay: load lifecycle inputs: %w", err)
|
||
}
|
||
for _, pe := range parseErrors {
|
||
glog.Warningf("daily_replay: invalid lifecycle policy on bucket %s: %v", pe.Bucket, pe.Err)
|
||
}
|
||
eng.Compile(inputs, engine.CompileOptions{PriorStates: scheduler.AllActivePriorStates(inputs)})
|
||
|
||
shards := make([]int, 0, s3lifecycle.ShardCount)
|
||
for i := 0; i < s3lifecycle.ShardCount; i++ {
|
||
shards = append(shards, i)
|
||
}
|
||
|
||
limiter, limiterDesc := buildLimiterFromClusterContext(request.GetClusterContext())
|
||
|
||
// Bucket list for the walker — derived from inputs so it matches
|
||
// the snapshot the engine compiled.
|
||
buckets := make([]string, 0, len(inputs))
|
||
for _, in := range inputs {
|
||
if in.Bucket != "" {
|
||
buckets = append(buckets, in.Bucket)
|
||
}
|
||
}
|
||
persister := &dailyrun.FilerCursorPersister{Store: dispatcher.NewFilerStoreClient(filerClient)}
|
||
report := &runReport{bucketsTotal: len(buckets) + len(parseErrors), invalidPolicies: parseErrors}
|
||
loadLastFullPass(ctx, persister, report)
|
||
|
||
runStats := &dailyrun.RunStats{}
|
||
// Share the limiter with processMatches so walker + replay can't
|
||
// combine to burst past the cluster cap.
|
||
walkerDispatch := &dailyrun.WalkerDispatcher{Client: client, Limiter: limiter, DeleteTimeout: cfg.DeleteTimeout, Stats: runStats}
|
||
walker := dailyrun.WalkerFunc(func(walkCtx context.Context, view *engine.Snapshot, shards []int) (time.Time, error) {
|
||
return dailyrun.WalkBuckets(walkCtx, view, shards, buckets,
|
||
dailyrun.FilerListFunc(runStats.CountListed(filerClient), bucketsPath, view), walkerDispatch, runStats, persister)
|
||
})
|
||
|
||
_ = sender.SendProgress(&plugin_pb.JobProgressUpdate{
|
||
JobId: request.Job.JobId, JobType: jobType,
|
||
State: plugin_pb.JobState_JOB_STATE_RUNNING, Stage: "starting",
|
||
Message: fmt.Sprintf("daily_replay shards=%d workers=%d rate=%s buckets=%d invalid_policies=%d walker=on",
|
||
len(shards), cfg.Workers, limiterDesc, len(buckets), len(parseErrors)),
|
||
})
|
||
|
||
stopProgress := startProgressReporter(request, sender, runStats.Summary, progressInterval)
|
||
report.err = dailyrun.Run(ctx, dailyrun.Config{
|
||
Shards: shards,
|
||
BucketsPath: bucketsPath,
|
||
Engine: eng,
|
||
FilerClient: filerClient,
|
||
Client: client,
|
||
Persister: persister,
|
||
Lister: dispatcher.NewFilerSiblingLister(filerClient, bucketsPath),
|
||
Workers: cfg.Workers,
|
||
Limiter: limiter,
|
||
DeleteTimeout: cfg.DeleteTimeout,
|
||
RetentionWindow: cfg.MetaLogRetention,
|
||
Walker: walker,
|
||
WalkerInterval: cfg.WalkerInterval,
|
||
ClientName: "worker-s3-lifecycle-daily",
|
||
Stats: runStats,
|
||
})
|
||
stopProgress()
|
||
report.canceled = ctx.Err() != nil
|
||
report.elapsed = time.Since(startedAt)
|
||
report.summary = runStats.Summary()
|
||
if report.err != nil {
|
||
glog.Warningf("daily_replay: %v", report.err)
|
||
}
|
||
recordFullPass(ctx, persister, report, startedAt)
|
||
return report, nil
|
||
}
|
||
|
||
func loadLastFullPass(ctx context.Context, persister *dailyrun.FilerCursorPersister, report *runReport) {
|
||
fp, found, err := persister.LoadFullPass(ctx)
|
||
if err != nil {
|
||
glog.Warningf("daily_replay: %v", err)
|
||
return
|
||
}
|
||
if found {
|
||
report.lastFullPass = fp.CompletedAt
|
||
publishFullPass(fp)
|
||
}
|
||
}
|
||
|
||
// recordFullPass advances the full-pass record only when the pass
|
||
// covered every shard and every policy.
|
||
func recordFullPass(ctx context.Context, persister *dailyrun.FilerCursorPersister, report *runReport, startedAt time.Time) {
|
||
if !report.fullPass() {
|
||
return
|
||
}
|
||
fp := dailyrun.FullPass{StartedAt: startedAt.UTC(), CompletedAt: time.Now().UTC()}
|
||
saveCtx, saveCancel := context.WithTimeout(ctx, 10*time.Second)
|
||
defer saveCancel()
|
||
if err := persister.SaveFullPass(saveCtx, fp); err != nil {
|
||
glog.Warningf("daily_replay: %v", err)
|
||
report.fullPassSaveErr = err
|
||
return
|
||
}
|
||
report.lastFullPass = fp.CompletedAt
|
||
publishFullPass(fp)
|
||
}
|
||
|
||
func publishFullPass(fp dailyrun.FullPass) {
|
||
stats.S3LifecycleDailyRunLastFullPassTimestampSeconds.Set(float64(fp.CompletedAt.UnixNano()) / 1e9)
|
||
}
|
||
|
||
// buildLimiterFromClusterContext parses the per-worker share the admin
|
||
// wrote into ClusterContext.Metadata (see weed/admin/plugin/plugin.go's
|
||
// s3_lifecycle injection) and returns a rate.Limiter, or nil when no
|
||
// rate cap applies. The description string is for the JobProgressUpdate
|
||
// so operators can see "rate=unlimited" / "rate=12.5/s burst=25" in
|
||
// the activity log.
|
||
//
|
||
// Tolerant of missing keys, empty strings, malformed numbers, and
|
||
// non-positive values — all treated as "no limit" rather than failing
|
||
// the run. The admin allocator is the single point that decides whether
|
||
// to populate these keys; the worker doesn't second-guess.
|
||
func buildLimiterFromClusterContext(cc *plugin_pb.ClusterContext) (*rate.Limiter, string) {
|
||
if cc == nil || cc.Metadata == nil {
|
||
return nil, "unlimited"
|
||
}
|
||
rps, ok := parsePositiveFloat(cc.Metadata[MetadataKeyDeletesPerSecond])
|
||
if !ok {
|
||
return nil, "unlimited"
|
||
}
|
||
burst, _ := parsePositiveInt(cc.Metadata[MetadataKeyDeletesBurst])
|
||
if burst <= 0 {
|
||
// Sensible default: enough headroom for one tick's worth of
|
||
// throughput. Caller may also supply 0 to opt into this default.
|
||
burst = int(rps * 2)
|
||
if burst < 1 {
|
||
burst = 1
|
||
}
|
||
}
|
||
return rate.NewLimiter(rate.Limit(rps), burst), fmt.Sprintf("%.3g/s burst=%d", rps, burst)
|
||
}
|
||
|
||
func parsePositiveFloat(s string) (float64, bool) {
|
||
if s == "" {
|
||
return 0, false
|
||
}
|
||
v, err := strconv.ParseFloat(s, 64)
|
||
if err != nil || v <= 0 {
|
||
return 0, false
|
||
}
|
||
return v, true
|
||
}
|
||
|
||
func parsePositiveInt(s string) (int, bool) {
|
||
if s == "" {
|
||
return 0, false
|
||
}
|
||
v, err := strconv.Atoi(s)
|
||
if err != nil || v <= 0 {
|
||
return 0, false
|
||
}
|
||
return v, true
|
||
}
|
||
|
||
// clusterS3Endpoints returns the master-discovered S3 gRPC addresses for the
|
||
// cluster. The handler uses the first one that passes checkLifecycleReady and
|
||
// moves to the next ready one if it becomes unavailable during the pass; the
|
||
// master refreshes the list on KeepConnected.
|
||
func clusterS3Endpoints(cc *plugin_pb.ClusterContext) []string {
|
||
if cc == nil {
|
||
return nil
|
||
}
|
||
out := make([]string, 0, len(cc.S3GrpcAddresses))
|
||
for _, addr := range cc.S3GrpcAddresses {
|
||
if addr != "" {
|
||
out = append(out, addr)
|
||
}
|
||
}
|
||
return out
|
||
}
|
||
|
||
type lifecycleRPCAdapter struct {
|
||
c s3_lifecycle_pb.SeaweedS3LifecycleInternalClient
|
||
}
|
||
|
||
func (a lifecycleRPCAdapter) LifecycleDelete(ctx context.Context, req *s3_lifecycle_pb.LifecycleDeleteRequest) (*s3_lifecycle_pb.LifecycleDeleteResponse, error) {
|
||
ctx, _ = credential.WithS3InternalAdminAuth(ctx)
|
||
return a.c.LifecycleDelete(ctx, req)
|
||
}
|
||
|
||
func lookupBucketsPath(ctx context.Context, client filer_pb.SeaweedFilerClient) (string, error) {
|
||
resp, err := client.GetFilerConfiguration(ctx, &filer_pb.GetFilerConfigurationRequest{})
|
||
if err != nil {
|
||
return "", err
|
||
}
|
||
if path := resp.GetDirBuckets(); path != "" {
|
||
return path, nil
|
||
}
|
||
return "/buckets", nil
|
||
}
|
||
|
||
func readString(values map[string]*plugin_pb.ConfigValue, field, fallback string) string {
|
||
v, ok := values[field]
|
||
if !ok || v == nil {
|
||
return fallback
|
||
}
|
||
if k, ok := v.Kind.(*plugin_pb.ConfigValue_StringValue); ok {
|
||
return k.StringValue
|
||
}
|
||
return fallback
|
||
}
|
||
|
||
var _ pluginworker.JobHandler = (*Handler)(nil)
|