s3 lifecycle: bound each LifecycleDelete attempt with its own deadline (#12646)

* s3 lifecycle: bound each LifecycleDelete attempt with its own deadline

Replay and walker deletes used the job context with no request
deadline, so one stalled unary RPC could hold a shard for the whole
job. Each attempt now runs under DeleteTimeout (default 1m), set after
the limiter wait. A timeout while the job is still live is retried like
a transport error and then halts the shard or walk without advancing;
parent cancellation still stops both the attempt and the backoff.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: admin setting for the delete RPC timeout

Adds delete_timeout_seconds to the S3 Lifecycle admin form and passes
it to both the replay drain and the walker. 0 keeps the 60s default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: describe replay and walker timeouts separately in the help text

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: fail the pass when a replay delete exhausts its attempts

A transport error or attempt timeout that outlived the retries halted
the shard with a nil error, so the job reported success. processMatches
now returns the error (the cursor still holds at the event), and
runShard treats a drain error as a clean stop only when its own context
ended, since an attempt timeout also wraps DeadlineExceeded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: test that a replayed LifecycleDelete resolves after a lost reply

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: skip the expiration write once the attempt deadline passed

A worker retries a LifecycleDelete whose attempt timed out, while the
first attempt can still be running on the S3 server. An attempt that
stalled before its marker write would then stack a second delete marker
after the retry wrote one. The gRPC deadline reaches the handler context,
so check it right before the expiration writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: recheck the attempt deadline after the suspended null delete

The null-version delete can block on the owner filer past the attempt
deadline; the worker may have retried and written a marker by then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: finish the suspended marker once the null version is deleted

Skipping the marker after a successful null delete left an older version
as the latest, and the retry then resolved as a no-op without hiding it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: let a routed delete marker commit only while its latest guard holds

createDeleteMarkerIf passes an optional condition, evaluated on the
.versions directory entry under the object's lock on the owner filer,
and reports a refused commit as errLatestVersionChanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: write the expiration marker as a compare-and-set on the latest pointer

A timed-out attempt can still be running on the S3 server while the
worker's retry writes the marker. The marker commit is now conditioned
on the .versions pointer as read just before it and skipped when the
latest is already a delete marker, so whichever attempt commits second
writes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: resolve the marker as stale when another version became latest

The marker's compare-and-set only covered changes after its own pointer
read, so a version promoted after the expiration checks passed could be
hidden by the marker. The enabled path now passes the validated version
and a conflict retry that finds a different named latest resolves the
request as VERSION_NOT_LATEST.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: resolve a null expiration as stale once a version takes the pointer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: read the marker's latest version the way getLatestObjectVersion does

A pointer names a version only with its file name; an id alone falls back
to the null object, and the null-is-latest signal makes the null version
current. An expiration that validated a named version no longer hides a
null written by a suspended PUT, and a current null under an id-only pointer
now expires instead of resolving as stale.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: hold the object write lock for an unrouted expiration marker

Off the ring createDeleteMarkerIf cannot carry the pointer guard, so a PUT
committing between the pointer read and the marker write was hidden. The
read and the write now run under the object write lock that unrouted
writers take.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: share transient-error retries between walker and replay deletes (#12649)

* s3 lifecycle: retry only transient LifecycleDelete failures

dispatchWithRetry retried every RPC error except parent cancellation, so
an authentication, permission or argument error spent the whole retry
budget before surfacing. Retry only attempt timeouts and the failures
util.IsTransientError recognizes (Unavailable, ResourceExhausted, network
resets); everything else returns on the first attempt. A nil response
is now an error instead of a nil dereference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: take a delete-rate token for every LifecycleDelete attempt

processMatches waited on the shared limiter once per match, so the
retries behind it ran outside the cluster-wide deletes/sec budget. The
retry loop now waits on the limiter before each attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: give walker deletes the replay retry policy

WalkerDispatcher sent each LifecycleDelete once, so one dropped
connection aborted the bucket's daily walk while the same failure on the
replay path was retried. The walker now goes through deleteWithRetry and
shares its transient-error retries, per-attempt limiter wait and nil
response handling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: retry a DeadlineExceeded the server sends back

A server or proxy can end the RPC with DeadlineExceeded while the
attempt's own deadline is still running. util.IsTransientError rejects
that status, so the delete halted with retries left. It is a timed-out
attempt like the client-side one, so retry it the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: say walker deletes get 3 attempts in the timeout help

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chris LuandClaude Opus 5.5 authored and GitHub committed 2026-11-15 13:09:22 -08:00
1 parent 504de9b562
commit 63806b7073
16 files changed
+1083 -78

No files matched your search

+100 -15
View File
@@ -88,10 +88,19 @@ func (s3a *S3ApiServer) lifecycleDispatch(ctx context.Context, req *s3_lifecycle
return resp, nil
}
}
// Past the attempt deadline the worker may already be resending
// this request, so a late marker write would stack a second one.
if err := ctx.Err(); err != nil {
return nil, err
}
switch state {
case s3_constants.VersioningEnabled:
if _, err := s3a.createDeleteMarker(req.Bucket, req.ObjectPath); err != nil {
return retryLater("TRANSPORT_ERROR: createDeleteMarker: " + err.Error()), nil
written, resp := s3a.lifecycleDeleteMarker(req.Bucket, req.ObjectPath, lifecycleAddressedVersion(req, entry))
if resp != nil {
return resp, nil
}
if !written {
return noopResolved("LATEST_IS_DELETE_MARKER"), nil
}
return done(), nil
case s3_constants.VersioningSuspended:
@@ -122,8 +131,10 @@ func (s3a *S3ApiServer) lifecycleDispatch(ctx context.Context, req *s3_lifecycle
}
}
}
if _, err := s3a.createDeleteMarker(req.Bucket, req.ObjectPath); err != nil {
return retryLater("TRANSPORT_ERROR: createDeleteMarker: " + err.Error()), nil
// The null delete is done, so the marker must follow even past
// the deadline: without it a retry finds an older version latest.
if _, resp := s3a.lifecycleDeleteMarker(req.Bucket, req.ObjectPath, ""); resp != nil {
return resp, nil
}
return done(), nil
default:
@@ -225,6 +236,76 @@ func (s3a *S3ApiServer) lifecycleDispatch(ctx context.Context, req *s3_lifecycle
}
}
// lifecycleDeleteMarker writes an expiration's delete marker as a
// compare-and-set on the .versions pointer, skipping it when the latest is
// already a delete marker. An attempt the worker gave up on may still be
// running while its retry writes, so whichever commits second sees the marker.
// A non-empty expected is the version that passed the expiration checks; a
// different latest resolves the request as stale instead of hiding it.
func (s3a *S3ApiServer) lifecycleDeleteMarker(bucket, object, expected string) (written bool, resp *s3_lifecycle_pb.LifecycleDeleteResponse) {
normalized := s3_constants.NormalizeObjectKey(object)
if s3a.objectWriteOwner(bucket, normalized) != "" {
return s3a.lifecycleDeleteMarkerIf(bucket, object, normalized, expected)
}
// Off the ring the marker commits without the guard, so the pointer read
// and the write serialize with other writers on the object write lock.
code := s3a.withObjectWriteLock(bucket, normalized, nil, func() s3err.ErrorCode {
written, resp = s3a.lifecycleDeleteMarkerIf(bucket, object, normalized, expected)
return s3err.ErrNone
})
if code != s3err.ErrNone {
return false, retryLater("TRANSPORT_ERROR: object write lock: " + s3err.GetAPIError(code).Code)
}
return written, resp
}
func (s3a *S3ApiServer) lifecycleDeleteMarkerIf(bucket, object, normalized, expected string) (bool, *s3_lifecycle_pb.LifecycleDeleteResponse) {
for range 3 {
pointer, err := s3a.getObjectScopedEntryRoutedByKey(bucket, normalized, s3a.bucketDir(bucket), normalized+s3_constants.VersionsFolder)
if err != nil {
if !isObjectEntryMiss(err) {
return false, retryLater("TRANSPORT_ERROR: latest-pointer lookup: " + err.Error())
}
pointer = nil
}
if pointer != nil && string(pointer.Extended[s3_constants.ExtLatestVersionIsDeleteMarker]) == "true" && !nullVersionIsLatest(pointer) {
return false, nil
}
if expected != "" && pointer != nil {
if latest := pointerLatestVersion(pointer); latest != "" && latest != expected {
return false, noopResolved("VERSION_NOT_LATEST")
}
}
guard := &filer_pb.WriteCondition{Clauses: []*filer_pb.WriteCondition_Clause{{
Kind: filer_pb.WriteCondition_IF_ENTRY_EQUAL,
ExpectedEntry: pointer,
}}}
if _, err := s3a.createDeleteMarkerIf(bucket, object, guard); err != nil {
if errors.Is(err, errLatestVersionChanged) {
continue
}
return false, retryLater("TRANSPORT_ERROR: createDeleteMarker: " + err.Error())
}
return true, nil
}
return false, retryLater("LATEST_CHANGED: createDeleteMarker")
}
// pointerLatestVersion reads the latest version off a .versions pointer the
// way getLatestObjectVersion does: a named pointer with its file outranks the
// null object, the null-is-latest signal names the null version, and anything
// else leaves the latest to the caller's earlier read.
func pointerLatestVersion(pointer *filer_pb.Entry) string {
id := string(pointer.Extended[s3_constants.ExtLatestVersionIdKey])
if id != "" && len(pointer.Extended[s3_constants.ExtLatestVersionFileNameKey]) > 0 {
return id
}
if nullVersionIsLatest(pointer) {
return "null"
}
return ""
}
// lifecycleRequireLatest refuses a current-version expiration whose version
// is no longer the key's latest. A version file never changes, so the identity
// check cannot see a newer write.
@@ -236,22 +317,26 @@ func (s3a *S3ApiServer) lifecycleRequireLatest(req *s3_lifecycle_pb.LifecycleDel
}
return retryLater("TRANSPORT_ERROR: latest-version lookup: " + err.Error())
}
versionIdOrNull := func(e *filer_pb.Entry) string {
if id := string(e.Extended[s3_constants.ExtVersionIdKey]); id != "" {
return id
}
return "null"
}
addressed := req.VersionId
if addressed == "" {
addressed = versionIdOrNull(entry)
}
if versionIdOrNull(latest) != addressed {
if versionIdOrNull(latest) != lifecycleAddressedVersion(req, entry) {
return noopResolved("VERSION_NOT_LATEST")
}
return nil
}
func versionIdOrNull(e *filer_pb.Entry) string {
if id := string(e.Extended[s3_constants.ExtVersionIdKey]); id != "" {
return id
}
return "null"
}
func lifecycleAddressedVersion(req *s3_lifecycle_pb.LifecycleDeleteRequest, entry *filer_pb.Entry) string {
if req.VersionId != "" {
return req.VersionId
}
return versionIdOrNull(entry)
}
func (s3a *S3ApiServer) lifecycleAbortMPU(ctx context.Context, req *s3_lifecycle_pb.LifecycleDeleteRequest) (*s3_lifecycle_pb.LifecycleDeleteResponse, error) {
// req.ObjectPath is `.uploads/<upload_id>` (set by the router from the
// init directory's bucket-relative path); reject anything that isn't