mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-16 00:38:53 +01:00
* s3 lifecycle: bound each LifecycleDelete attempt with its own deadline Replay and walker deletes used the job context with no request deadline, so one stalled unary RPC could hold a shard for the whole job. Each attempt now runs under DeleteTimeout (default 1m), set after the limiter wait. A timeout while the job is still live is retried like a transport error and then halts the shard or walk without advancing; parent cancellation still stops both the attempt and the backoff. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: admin setting for the delete RPC timeout Adds delete_timeout_seconds to the S3 Lifecycle admin form and passes it to both the replay drain and the walker. 0 keeps the 60s default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: describe replay and walker timeouts separately in the help text Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: fail the pass when a replay delete exhausts its attempts A transport error or attempt timeout that outlived the retries halted the shard with a nil error, so the job reported success. processMatches now returns the error (the cursor still holds at the event), and runShard treats a drain error as a clean stop only when its own context ended, since an attempt timeout also wraps DeadlineExceeded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test that a replayed LifecycleDelete resolves after a lost reply Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: skip the expiration write once the attempt deadline passed A worker retries a LifecycleDelete whose attempt timed out, while the first attempt can still be running on the S3 server. An attempt that stalled before its marker write would then stack a second delete marker after the retry wrote one. The gRPC deadline reaches the handler context, so check it right before the expiration writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: recheck the attempt deadline after the suspended null delete The null-version delete can block on the owner filer past the attempt deadline; the worker may have retried and written a marker by then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: finish the suspended marker once the null version is deleted Skipping the marker after a successful null delete left an older version as the latest, and the retry then resolved as a no-op without hiding it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: let a routed delete marker commit only while its latest guard holds createDeleteMarkerIf passes an optional condition, evaluated on the .versions directory entry under the object's lock on the owner filer, and reports a refused commit as errLatestVersionChanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: write the expiration marker as a compare-and-set on the latest pointer A timed-out attempt can still be running on the S3 server while the worker's retry writes the marker. The marker commit is now conditioned on the .versions pointer as read just before it and skipped when the latest is already a delete marker, so whichever attempt commits second writes nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: resolve the marker as stale when another version became latest The marker's compare-and-set only covered changes after its own pointer read, so a version promoted after the expiration checks passed could be hidden by the marker. The enabled path now passes the validated version and a conflict retry that finds a different named latest resolves the request as VERSION_NOT_LATEST. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: resolve a null expiration as stale once a version takes the pointer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: read the marker's latest version the way getLatestObjectVersion does A pointer names a version only with its file name; an id alone falls back to the null object, and the null-is-latest signal makes the null version current. An expiration that validated a named version no longer hides a null written by a suspended PUT, and a current null under an id-only pointer now expires instead of resolving as stale. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: hold the object write lock for an unrouted expiration marker Off the ring createDeleteMarkerIf cannot carry the pointer guard, so a PUT committing between the pointer read and the marker write was hidden. The read and the write now run under the object write lock that unrouted writers take. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: share transient-error retries between walker and replay deletes (#12649) * s3 lifecycle: retry only transient LifecycleDelete failures dispatchWithRetry retried every RPC error except parent cancellation, so an authentication, permission or argument error spent the whole retry budget before surfacing. Retry only attempt timeouts and the failures util.IsTransientError recognizes (Unavailable, ResourceExhausted, network resets); everything else returns on the first attempt. A nil response is now an error instead of a nil dereference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: take a delete-rate token for every LifecycleDelete attempt processMatches waited on the shared limiter once per match, so the retries behind it ran outside the cluster-wide deletes/sec budget. The retry loop now waits on the limiter before each attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: give walker deletes the replay retry policy WalkerDispatcher sent each LifecycleDelete once, so one dropped connection aborted the bucket's daily walk while the same failure on the replay path was retried. The walker now goes through deleteWithRetry and shares its transient-error retries, per-attempt limiter wait and nil response handling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: retry a DeadlineExceeded the server sends back A server or proxy can end the RPC with DeadlineExceeded while the attempt's own deadline is still running. util.IsTransientError rejects that status, so the delete halted with retries left. It is a timed-out attempt like the client-side one, so retry it the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: say walker deletes get 3 attempts in the timeout help Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
177 lines
6.3 KiB
Go
177 lines
6.3 KiB
Go
package dailyrun
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"fmt"
|
|
"math/rand"
|
|
"time"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb"
|
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
|
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/router"
|
|
"github.com/seaweedfs/seaweedfs/weed/stats"
|
|
"github.com/seaweedfs/seaweedfs/weed/util"
|
|
"golang.org/x/time/rate"
|
|
"google.golang.org/grpc/codes"
|
|
)
|
|
|
|
// transportRetryAttempts is the in-run retry budget shared by the walker
|
|
// and replay deletes. Only transport failures and attempt timeouts are
|
|
// retried; server outcomes (RETRY_LATER, BLOCKED) wait for the next run.
|
|
const transportRetryAttempts = 3
|
|
|
|
// transportRetryInitial is the first sleep after a failed attempt.
|
|
// Doubled per attempt up to transportRetryMax. Phase 2 keeps these
|
|
// small because the daily run's own MaxRuntime is the larger cap.
|
|
const (
|
|
transportRetryInitial = 200 * time.Millisecond
|
|
transportRetryMax = 5 * time.Second
|
|
)
|
|
|
|
// DefaultDeleteTimeout bounds one LifecycleDelete attempt when the
|
|
// caller sets no timeout.
|
|
const DefaultDeleteTimeout = time.Minute
|
|
|
|
var (
|
|
errDeleteAttemptTimedOut = errors.New("lifecycle delete attempt timed out")
|
|
errNilDeleteResponse = errors.New("nil response")
|
|
)
|
|
|
|
// deleteOnce sends one LifecycleDelete attempt under its own deadline,
|
|
// so a stalled RPC fails that attempt instead of holding the shard for
|
|
// the job's lifetime.
|
|
func deleteOnce(ctx context.Context, client LifecycleClient, req *s3_lifecycle_pb.LifecycleDeleteRequest, timeout time.Duration) (*s3_lifecycle_pb.LifecycleDeleteResponse, error) {
|
|
if timeout <= 0 {
|
|
timeout = DefaultDeleteTimeout
|
|
}
|
|
attemptCtx, cancel := context.WithTimeout(ctx, timeout)
|
|
defer cancel()
|
|
resp, err := client.LifecycleDelete(attemptCtx, req)
|
|
if err != nil && ctx.Err() == nil && attemptCtx.Err() != nil {
|
|
err = fmt.Errorf("%w after %s: %w", errDeleteAttemptTimedOut, timeout, err)
|
|
}
|
|
if err == nil && resp == nil {
|
|
err = errNilDeleteResponse
|
|
}
|
|
return resp, err
|
|
}
|
|
|
|
// isRetryableDeleteError reports whether a failed attempt may be resent.
|
|
// Authentication, permission and argument errors fail the same way
|
|
// again, so they surface at once.
|
|
func isRetryableDeleteError(err error) bool {
|
|
if errors.Is(err, errDeleteAttemptTimedOut) || util.IsTransientError(err) {
|
|
return true
|
|
}
|
|
st, ok := util.ServerStatus(err)
|
|
return ok && st.Code() == codes.DeadlineExceeded
|
|
}
|
|
|
|
// deleteWithRetry sends req, retrying transport failures and attempt
|
|
// timeouts up to transportRetryAttempts times with jittered exponential
|
|
// backoff. Every attempt waits on limiter, so retries stay inside the
|
|
// shared delete budget. The resent request keeps its version and identity
|
|
// witness, so a delete that committed before its reply was lost resolves
|
|
// as NOOP_RESOLVED on the retry.
|
|
func deleteWithRetry(ctx context.Context, client LifecycleClient, limiter *rate.Limiter, req *s3_lifecycle_pb.LifecycleDeleteRequest, timeout time.Duration) (*s3_lifecycle_pb.LifecycleDeleteResponse, error) {
|
|
backoff := transportRetryInitial
|
|
for attempt := 1; ; attempt++ {
|
|
if limiter != nil {
|
|
waitStart := time.Now()
|
|
if err := limiter.Wait(ctx); err != nil {
|
|
return nil, fmt.Errorf("limiter: %w", err)
|
|
}
|
|
stats.S3LifecycleDispatchLimiterWaitSeconds.Observe(time.Since(waitStart).Seconds())
|
|
}
|
|
resp, err := deleteOnce(ctx, client, req, timeout)
|
|
if err == nil {
|
|
return resp, nil
|
|
}
|
|
// Parent cancellation is shutdown, not a transport flake.
|
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
|
return nil, ctxErr
|
|
}
|
|
if attempt >= transportRetryAttempts || !isRetryableDeleteError(err) {
|
|
return nil, err
|
|
}
|
|
select {
|
|
case <-ctx.Done():
|
|
return nil, ctx.Err()
|
|
case <-time.After(jitter(backoff)):
|
|
}
|
|
backoff = min(backoff*2, transportRetryMax)
|
|
}
|
|
}
|
|
|
|
// dispatchWithRetry sends the LifecycleDelete for m under the shared
|
|
// retry policy and returns the server's outcome.
|
|
func dispatchWithRetry(ctx context.Context, client LifecycleClient, limiter *rate.Limiter, m router.Match, timeout time.Duration) (s3_lifecycle_pb.LifecycleDeleteOutcome, error) {
|
|
resp, err := deleteWithRetry(ctx, client, limiter, buildDeleteRequest(m), timeout)
|
|
if err != nil {
|
|
return s3_lifecycle_pb.LifecycleDeleteOutcome_LIFECYCLE_DELETE_OUTCOME_UNSPECIFIED, err
|
|
}
|
|
return resp.Outcome, nil
|
|
}
|
|
|
|
// jitter returns a duration in the range [d/2, d) using equal jitter.
|
|
// Prevents thundering herds when many daily-run workers retry simultaneously.
|
|
func jitter(d time.Duration) time.Duration {
|
|
if d <= 0 {
|
|
return 0
|
|
}
|
|
half := d / 2
|
|
if half <= 0 {
|
|
return d
|
|
}
|
|
return half + time.Duration(rand.Int63n(int64(half)))
|
|
}
|
|
|
|
// buildDeleteRequest constructs the LifecycleDelete RPC payload for a
|
|
// router Match. Mirrors dispatcher.dispatchOne's request shape — both
|
|
// targets the same server-side handler and the proto encoding must
|
|
// match exactly. Duplicated rather than shared because Phase 5
|
|
// deletes the dispatcher's call site and the cross-package dependency
|
|
// would just create a temporary import.
|
|
func buildDeleteRequest(m router.Match) *s3_lifecycle_pb.LifecycleDeleteRequest {
|
|
rh := m.Key.RuleHash
|
|
return &s3_lifecycle_pb.LifecycleDeleteRequest{
|
|
Bucket: m.Bucket,
|
|
ObjectPath: m.ObjectKey,
|
|
VersionId: m.VersionID,
|
|
RuleHash: rh[:],
|
|
ActionKind: toProtoActionKind(m.Key.ActionKind),
|
|
ExpectedIdentity: toProtoIdentity(m.Identity),
|
|
}
|
|
}
|
|
|
|
func toProtoActionKind(k s3lifecycle.ActionKind) s3_lifecycle_pb.ActionKind {
|
|
switch k {
|
|
case s3lifecycle.ActionKindExpirationDays:
|
|
return s3_lifecycle_pb.ActionKind_EXPIRATION_DAYS
|
|
case s3lifecycle.ActionKindExpirationDate:
|
|
return s3_lifecycle_pb.ActionKind_EXPIRATION_DATE
|
|
case s3lifecycle.ActionKindNoncurrentDays:
|
|
return s3_lifecycle_pb.ActionKind_NONCURRENT_DAYS
|
|
case s3lifecycle.ActionKindNewerNoncurrent:
|
|
return s3_lifecycle_pb.ActionKind_NEWER_NONCURRENT
|
|
case s3lifecycle.ActionKindAbortMPU:
|
|
return s3_lifecycle_pb.ActionKind_ABORT_MPU
|
|
case s3lifecycle.ActionKindExpiredDeleteMarker:
|
|
return s3_lifecycle_pb.ActionKind_EXPIRED_DELETE_MARKER
|
|
}
|
|
return s3_lifecycle_pb.ActionKind_ACTION_KIND_UNSPECIFIED
|
|
}
|
|
|
|
func toProtoIdentity(id *router.EntryIdentity) *s3_lifecycle_pb.EntryIdentity {
|
|
if id == nil {
|
|
return nil
|
|
}
|
|
return &s3_lifecycle_pb.EntryIdentity{
|
|
MtimeNs: id.MtimeNs,
|
|
Size: id.Size,
|
|
HeadFid: id.HeadFid,
|
|
ExtendedHash: id.ExtendedHash,
|
|
}
|
|
}
|