mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-16 08:45:22 +01:00
* s3 lifecycle: bound each LifecycleDelete attempt with its own deadline Replay and walker deletes used the job context with no request deadline, so one stalled unary RPC could hold a shard for the whole job. Each attempt now runs under DeleteTimeout (default 1m), set after the limiter wait. A timeout while the job is still live is retried like a transport error and then halts the shard or walk without advancing; parent cancellation still stops both the attempt and the backoff. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: admin setting for the delete RPC timeout Adds delete_timeout_seconds to the S3 Lifecycle admin form and passes it to both the replay drain and the walker. 0 keeps the 60s default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: describe replay and walker timeouts separately in the help text Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: fail the pass when a replay delete exhausts its attempts A transport error or attempt timeout that outlived the retries halted the shard with a nil error, so the job reported success. processMatches now returns the error (the cursor still holds at the event), and runShard treats a drain error as a clean stop only when its own context ended, since an attempt timeout also wraps DeadlineExceeded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test that a replayed LifecycleDelete resolves after a lost reply Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: skip the expiration write once the attempt deadline passed A worker retries a LifecycleDelete whose attempt timed out, while the first attempt can still be running on the S3 server. An attempt that stalled before its marker write would then stack a second delete marker after the retry wrote one. The gRPC deadline reaches the handler context, so check it right before the expiration writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: recheck the attempt deadline after the suspended null delete The null-version delete can block on the owner filer past the attempt deadline; the worker may have retried and written a marker by then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: finish the suspended marker once the null version is deleted Skipping the marker after a successful null delete left an older version as the latest, and the retry then resolved as a no-op without hiding it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: let a routed delete marker commit only while its latest guard holds createDeleteMarkerIf passes an optional condition, evaluated on the .versions directory entry under the object's lock on the owner filer, and reports a refused commit as errLatestVersionChanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: write the expiration marker as a compare-and-set on the latest pointer A timed-out attempt can still be running on the S3 server while the worker's retry writes the marker. The marker commit is now conditioned on the .versions pointer as read just before it and skipped when the latest is already a delete marker, so whichever attempt commits second writes nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: resolve the marker as stale when another version became latest The marker's compare-and-set only covered changes after its own pointer read, so a version promoted after the expiration checks passed could be hidden by the marker. The enabled path now passes the validated version and a conflict retry that finds a different named latest resolves the request as VERSION_NOT_LATEST. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: resolve a null expiration as stale once a version takes the pointer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: read the marker's latest version the way getLatestObjectVersion does A pointer names a version only with its file name; an id alone falls back to the null object, and the null-is-latest signal makes the null version current. An expiration that validated a named version no longer hides a null written by a suspended PUT, and a current null under an id-only pointer now expires instead of resolving as stale. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: hold the object write lock for an unrouted expiration marker Off the ring createDeleteMarkerIf cannot carry the pointer guard, so a PUT committing between the pointer read and the marker write was hidden. The read and the write now run under the object write lock that unrouted writers take. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: share transient-error retries between walker and replay deletes (#12649) * s3 lifecycle: retry only transient LifecycleDelete failures dispatchWithRetry retried every RPC error except parent cancellation, so an authentication, permission or argument error spent the whole retry budget before surfacing. Retry only attempt timeouts and the failures util.IsTransientError recognizes (Unavailable, ResourceExhausted, network resets); everything else returns on the first attempt. A nil response is now an error instead of a nil dereference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: take a delete-rate token for every LifecycleDelete attempt processMatches waited on the shared limiter once per match, so the retries behind it ran outside the cluster-wide deletes/sec budget. The retry loop now waits on the limiter before each attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: give walker deletes the replay retry policy WalkerDispatcher sent each LifecycleDelete once, so one dropped connection aborted the bucket's daily walk while the same failure on the replay path was retried. The walker now goes through deleteWithRetry and shares its transient-error retries, per-attempt limiter wait and nil response handling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: retry a DeadlineExceeded the server sends back A server or proxy can end the RPC with DeadlineExceeded while the attempt's own deadline is still running. util.IsTransientError rejects that status, so the delete halted with retries left. It is a timed-out attempt like the client-side one, so retry it the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: say walker deletes get 3 attempts in the timeout help Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
443 lines
21 KiB
Go
443 lines
21 KiB
Go
package s3api
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"fmt"
|
|
"slices"
|
|
"strconv"
|
|
"strings"
|
|
"time"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/filer"
|
|
"github.com/seaweedfs/seaweedfs/weed/glog"
|
|
"github.com/seaweedfs/seaweedfs/weed/pb"
|
|
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
|
|
"github.com/seaweedfs/seaweedfs/weed/util"
|
|
"google.golang.org/protobuf/proto"
|
|
)
|
|
|
|
// objectWriteOwner resolves the filer that owns all of an object's writes,
|
|
// regardless of versioning state, or "" when no ring view is available. Normal,
|
|
// suspended, and versioned writes to the same object hash to one owner and
|
|
// serialize on its per-path lock.
|
|
func (s3a *S3ApiServer) objectWriteOwner(bucket, object string) pb.ServerAddress {
|
|
if s3a.objectWriteLockClient == nil {
|
|
return ""
|
|
}
|
|
return s3a.objectWriteLockClient.PrimaryForKey(s3a.objectRouteKey(bucket, object))
|
|
}
|
|
|
|
// latestPointerRecompute builds the RECOMPUTE_LATEST mutation that re-derives an
|
|
// object's .versions pointer. excludeName, when set, omits a version about to be
|
|
// deleted (so the pointer is repointed before the blob is removed); demote, when
|
|
// set, stamps the displaced prior latest with NoncurrentSinceNs.
|
|
func (s3a *S3ApiServer) latestPointerRecompute(bucket, object string, useInvertedFormat bool, excludeName string, demote bool) *filer_pb.ObjectMutation {
|
|
versionsPath := s3a.toFilerPath(bucket, object+s3_constants.VersionsFolder)
|
|
vdir, vname := util.FullPath(versionsPath).DirAndName()
|
|
rc := &filer_pb.Recompute{
|
|
ScanDir: versionsPath,
|
|
// Inverted ids sort newest-first, so the newest is the first ascending
|
|
// entry; legacy ids sort oldest-first (scan to the last).
|
|
Descending: !useInvertedFormat,
|
|
NameToKey: s3_constants.ExtLatestVersionFileNameKey,
|
|
SizeToKey: s3_constants.ExtLatestVersionSizeKey,
|
|
MtimeToKey: s3_constants.ExtLatestVersionMtimeKey,
|
|
CopyExtended: map[string]string{
|
|
s3_constants.ExtLatestVersionIdKey: s3_constants.ExtVersionIdKey,
|
|
s3_constants.ExtLatestVersionETagKey: s3_constants.ExtETagKey,
|
|
s3_constants.ExtLatestVersionOwnerKey: s3_constants.ExtAmzOwnerKey,
|
|
s3_constants.ExtLatestVersionIsDeleteMarker: s3_constants.ExtDeleteMarkerKey,
|
|
s3_constants.ExtLatestVersionStorageClassKey: s3_constants.AmzStorageClass,
|
|
// Version files never carry the null-current signal, so this mapping
|
|
// deletes a stale one from the pointer whenever it recomputes.
|
|
s3_constants.ExtNullVersionIsLatestKey: s3_constants.ExtNullVersionIsLatestKey,
|
|
},
|
|
ExcludeName: excludeName,
|
|
}
|
|
if demote {
|
|
rc.DemoteKey = s3_constants.ExtNoncurrentSinceNsKey
|
|
rc.DemoteValue = []byte(strconv.FormatInt(time.Now().UnixNano(), 10))
|
|
}
|
|
return &filer_pb.ObjectMutation{
|
|
Type: filer_pb.ObjectMutation_RECOMPUTE_LATEST,
|
|
Directory: vdir,
|
|
Name: vname,
|
|
Recompute: rc,
|
|
}
|
|
}
|
|
|
|
// routedVersionedFinalize flips the .versions pointer to the newest version and
|
|
// demotes the prior latest, atomically under the object's per-path lock on the
|
|
// owner filer, via a single RECOMPUTE_LATEST. versionEntry, when set, is PUT
|
|
// into .versions in the same transaction, so the recompute's scan sees it even
|
|
// when the gateway's own filer is not the owner; otherwise the version file is
|
|
// already written on the owner. latestGuard, when set, must hold on the
|
|
// .versions directory entry under the same lock.
|
|
func (s3a *S3ApiServer) routedVersionedFinalize(owner pb.ServerAddress, bucket, object string, useInvertedFormat bool, versionEntry *filer_pb.Entry, latestGuard *filer_pb.WriteCondition) s3err.ErrorCode {
|
|
mutations := []*filer_pb.ObjectMutation{s3a.latestPointerRecompute(bucket, object, useInvertedFormat, "", true)}
|
|
if versionEntry != nil {
|
|
mutations = append([]*filer_pb.ObjectMutation{{
|
|
Type: filer_pb.ObjectMutation_PUT,
|
|
Directory: s3a.toFilerPath(bucket, object+s3_constants.VersionsFolder),
|
|
Entry: versionEntry,
|
|
}}, mutations...)
|
|
}
|
|
req := &filer_pb.ObjectTransactionRequest{
|
|
LockKey: s3a.toFilerPath(bucket, object),
|
|
RouteKey: s3a.objectRouteKey(bucket, object),
|
|
Mutations: mutations,
|
|
}
|
|
if latestGuard != nil {
|
|
req.Condition = latestGuard
|
|
req.ConditionKey = s3a.toFilerPath(bucket, object+s3_constants.VersionsFolder)
|
|
}
|
|
resp, err := s3a.objectTxnOnFiler(owner, req)
|
|
switch {
|
|
case err != nil:
|
|
glog.Errorf("routedVersionedFinalize: %s/%s on %s: %v", bucket, object, owner, err)
|
|
return s3err.ErrInternalError
|
|
case resp.ErrorCode == filer_pb.FilerError_PRECONDITION_FAILED:
|
|
return s3err.ErrPreconditionFailed
|
|
case resp.Error != "":
|
|
glog.Errorf("routedVersionedFinalize: %s/%s: %s", bucket, object, resp.Error)
|
|
return s3err.ErrInternalError
|
|
default:
|
|
return s3err.ErrNone
|
|
}
|
|
}
|
|
|
|
// removeUploadDirMutation deletes a completed upload's directory metadata-only:
|
|
// the finished object's chunks are the part chunks, so freeing their data would
|
|
// destroy the object.
|
|
func (s3a *S3ApiServer) removeUploadDirMutation(bucket, uploadID string) *filer_pb.ObjectMutation {
|
|
return &filer_pb.ObjectMutation{
|
|
Type: filer_pb.ObjectMutation_DELETE,
|
|
Directory: s3a.genUploadsFolder(bucket),
|
|
Name: uploadID,
|
|
IsRecursive: true,
|
|
}
|
|
}
|
|
|
|
// uploadExistsCondition requires the upload directory to still exist when the
|
|
// commit transaction runs, so a delete that does not take the object lock
|
|
// (abort, lifecycle, s3.clean.uploads) fails the commit instead of letting it
|
|
// publish the object over freed chunks.
|
|
func uploadExistsCondition(uploadDirectory string) (string, *filer_pb.WriteCondition) {
|
|
return uploadDirectory, &filer_pb.WriteCondition{
|
|
Clauses: []*filer_pb.WriteCondition_Clause{{Kind: filer_pb.WriteCondition_IF_EXISTS}},
|
|
}
|
|
}
|
|
|
|
// routedMultipartFinalize commits a completed multipart upload in one
|
|
// ObjectTransaction on the owner filer: PUT the version file, remove the upload
|
|
// directory, recompute the latest pointer. The transaction applies mutations in
|
|
// order with no rollback, so the order picks which partial states are
|
|
// reachable: PUT first keeps the chunks referenced at all times, and removing
|
|
// the upload directory before the recompute means a published object never
|
|
// coexists with a leftover .uploads directory that s3.clean.uploads would purge
|
|
// with data. cond is the request's write condition, checked against the object
|
|
// (its latest version when the slot at the object path is empty) under the same
|
|
// lock; the upload directory rides the PUT as a mutation precondition.
|
|
func (s3a *S3ApiServer) routedMultipartFinalize(owner pb.ServerAddress, bucket, object string, useInvertedFormat bool, versionDir, versionFileName string, chunks []*filer_pb.FileChunk, decorate func(*filer_pb.Entry), uploadID string, cond *filer_pb.WriteCondition) s3err.ErrorCode {
|
|
now := time.Now().Unix()
|
|
versionEntry := &filer_pb.Entry{
|
|
Name: versionFileName,
|
|
Attributes: &filer_pb.FuseAttributes{
|
|
Mtime: now,
|
|
Crtime: now,
|
|
FileMode: uint32(0770),
|
|
Uid: filer_pb.OS_UID,
|
|
Gid: filer_pb.OS_GID,
|
|
},
|
|
Chunks: chunks,
|
|
}
|
|
if decorate != nil {
|
|
decorate(versionEntry)
|
|
}
|
|
|
|
routeKey := ""
|
|
if owner != "" {
|
|
routeKey = s3a.objectRouteKey(bucket, object)
|
|
}
|
|
// A filer older than the mutation-condition support silently ignores
|
|
// putCondition, so on upgrade the guard degrades to this gateway-side
|
|
// check — the same window the pre-transaction code had.
|
|
if exists, err := s3a.exists(s3a.genUploadsFolder(bucket), uploadID, true); err == nil && !exists {
|
|
return s3err.ErrNoSuchUpload
|
|
}
|
|
putConditionKey, putCondition := uploadExistsCondition(s3a.genUploadsFolder(bucket) + "/" + uploadID)
|
|
resp, err := s3a.routedPut(owner, routeKey, s3a.toFilerPath(bucket, object), versionDir+"/"+versionFileName, versionEntry, cond, "", versionDir, true, putCondition, putConditionKey, []*filer_pb.ObjectMutation{
|
|
s3a.removeUploadDirMutation(bucket, uploadID),
|
|
s3a.latestPointerRecompute(bucket, object, useInvertedFormat, "", true),
|
|
})
|
|
switch {
|
|
case err != nil:
|
|
glog.Errorf("routedMultipartFinalize: %s/%s upload %s on %s: %v", bucket, object, uploadID, owner, err)
|
|
return s3err.ErrInternalError
|
|
case resp.ErrorCode == filer_pb.FilerError_PRECONDITION_FAILED:
|
|
if resp.FailedMutation == nil {
|
|
return s3err.ErrPreconditionFailed
|
|
}
|
|
return s3err.ErrNoSuchUpload
|
|
case resp.Error != "":
|
|
glog.Errorf("routedMultipartFinalize: %s/%s upload %s: %s", bucket, object, uploadID, resp.Error)
|
|
return s3err.ErrInternalError
|
|
default:
|
|
return s3err.ErrNone
|
|
}
|
|
}
|
|
|
|
// wormDeleteCondition returns the object-lock guards for a delete, or nil when
|
|
// the bucket has no object lock. Governance bypass gates the retention check to
|
|
// COMPLIANCE mode so the filer still protects compliance versions under lock.
|
|
func wormDeleteCondition(worm, bypass bool) *filer_pb.WriteCondition {
|
|
if !worm {
|
|
return nil
|
|
}
|
|
retention := &filer_pb.WriteCondition_Clause{
|
|
Kind: filer_pb.WriteCondition_IF_EXTENDED_TIME_ELAPSED,
|
|
ExtKey: s3_constants.ExtRetentionUntilDateKey,
|
|
}
|
|
if bypass {
|
|
retention.GateKey = s3_constants.ExtObjectLockModeKey
|
|
retention.GateValue = s3_constants.RetentionModeCompliance
|
|
}
|
|
return &filer_pb.WriteCondition{Clauses: []*filer_pb.WriteCondition_Clause{
|
|
{Kind: filer_pb.WriteCondition_IF_EXTENDED_NOT_EQUAL, ExtKey: s3_constants.ExtLegalHoldKey, ExtValue: s3_constants.LegalHoldOn},
|
|
retention,
|
|
}}
|
|
}
|
|
|
|
// routedDeleteSpecificVersion removes one version under the owner filer's
|
|
// object lock. The repoint and the delete ride one transaction so the version's
|
|
// WORM guards gate both; the recompute is itself guarded to fire only while the
|
|
// pointer still names the deleted version — a noncurrent delete must leave it
|
|
// alone, since it can point at an older version than a re-derive would pick, or
|
|
// at nothing, as when a suspended bucket's null write cleared it while the null
|
|
// object stays current. ifMatch, when set, guards the version under the same
|
|
// lock and fails before anything applies.
|
|
func (s3a *S3ApiServer) routedDeleteSpecificVersion(owner pb.ServerAddress, bucket, object, versionId string, worm, bypass, metadataOnly bool, ifMatch *filer_pb.WriteCondition) s3err.ErrorCode {
|
|
if !isValidVersionID(versionId) {
|
|
return s3err.ErrInvalidRequest
|
|
}
|
|
versionFileName := s3a.getVersionFileName(versionId)
|
|
recompute := s3a.latestPointerRecompute(bucket, object, isNewFormatVersionId(versionId), versionFileName, false)
|
|
recompute.GetRecompute().OnlyIfCurrentName = versionFileName
|
|
versionsPath := s3a.toFilerPath(bucket, object+s3_constants.VersionsFolder)
|
|
if ifMatch != nil {
|
|
recompute.Condition, recompute.ConditionKey = ifMatch, versionsPath+"/"+versionFileName
|
|
}
|
|
cond := wormDeleteCondition(worm, bypass)
|
|
req := &filer_pb.ObjectTransactionRequest{
|
|
LockKey: s3a.toFilerPath(bucket, object),
|
|
RouteKey: s3a.objectRouteKey(bucket, object),
|
|
ConditionKey: versionsPath + "/" + versionFileName,
|
|
Condition: cond,
|
|
Mutations: []*filer_pb.ObjectMutation{
|
|
recompute,
|
|
{Type: filer_pb.ObjectMutation_DELETE, Directory: versionsPath, Name: versionFileName, IsDeleteData: !metadataOnly, RemoveEmptyParent: true},
|
|
},
|
|
}
|
|
resp, err := s3a.objectTxnOnFiler(owner, req)
|
|
switch {
|
|
case err != nil:
|
|
glog.Errorf("routedDeleteSpecificVersion: %s/%s %s on %s: %v", bucket, object, versionId, owner, err)
|
|
return s3err.ErrInternalError
|
|
case resp.ErrorCode == filer_pb.FilerError_PRECONDITION_FAILED:
|
|
if resp.FailedMutation != nil {
|
|
return s3err.ErrPreconditionFailed
|
|
}
|
|
// Legal hold or retention in force on the version.
|
|
return s3err.ErrAccessDenied
|
|
case resp.Error != "":
|
|
glog.Errorf("routedDeleteSpecificVersion: %s/%s %s: %s", bucket, object, versionId, resp.Error)
|
|
return s3err.ErrInternalError
|
|
default:
|
|
return s3err.ErrNone
|
|
}
|
|
}
|
|
|
|
// routedDeleteNullVersion deletes the null version (the regular object entry, not
|
|
// a .versions file) off the distributed lock. There is no pointer to recompute;
|
|
// the WORM guards, when present, gate the delete on the object entry itself
|
|
// (condition defaults to lock_key). The second return reports whether the delete
|
|
// was settled here: the raw delete cannot remove an entry other keys are nested
|
|
// under, which the lock path handles by stripping the object off it instead.
|
|
// ifMatch, when set, guards the delete itself and keeps the whole decision on
|
|
// the owner: the lock path does not exclude routed deletes, so a nested entry
|
|
// is stripped there under the same guard, and any other failure settles as
|
|
// one, since the transaction may have committed.
|
|
func (s3a *S3ApiServer) routedDeleteNullVersion(owner pb.ServerAddress, bucket, object string, worm, bypass, metadataOnly bool, ifMatch *filer_pb.WriteCondition) (s3err.ErrorCode, bool) {
|
|
fullpath := util.NewFullPath(s3a.bucketDir(bucket), object)
|
|
dir, name := fullpath.DirAndName()
|
|
guard, star := nullVersionCondition(ifMatch)
|
|
resp, err := s3a.objectTxnOnFiler(owner, &filer_pb.ObjectTransactionRequest{
|
|
LockKey: string(fullpath),
|
|
RouteKey: s3a.objectRouteKey(bucket, object),
|
|
Condition: wormDeleteCondition(worm, bypass),
|
|
Mutations: []*filer_pb.ObjectMutation{
|
|
{Type: filer_pb.ObjectMutation_DELETE, Directory: dir, Name: name, IsDeleteData: !metadataOnly, Condition: guard},
|
|
},
|
|
})
|
|
switch {
|
|
case err != nil && ifMatch != nil:
|
|
glog.Errorf("routedDeleteNullVersion: %s/%s on %s: %v", bucket, object, owner, err)
|
|
return s3err.ErrInternalError, true
|
|
case err != nil:
|
|
glog.Warningf("routedDeleteNullVersion: %s/%s on %s, falling back to lock: %v", bucket, object, owner, err)
|
|
return s3err.ErrNone, false
|
|
case resp.ErrorCode == filer_pb.FilerError_PRECONDITION_FAILED && resp.FailedMutation == nil:
|
|
return s3err.ErrAccessDenied, true
|
|
case resp.ErrorCode == filer_pb.FilerError_PRECONDITION_FAILED && star:
|
|
// A filer that predates IF_OBJECT_EXISTS fails it closed.
|
|
return s3a.routedDemoteNullVersion(owner, bucket, object, worm, bypass, metadataOnly, ifMatch), true
|
|
case resp.ErrorCode == filer_pb.FilerError_PRECONDITION_FAILED:
|
|
return s3err.ErrPreconditionFailed, true
|
|
case resp.Error != "" && ifMatch != nil && ownerRefusedNonEmpty(resp):
|
|
return s3a.routedDemoteNullVersion(owner, bucket, object, worm, bypass, metadataOnly, ifMatch), true
|
|
case resp.Error != "" && ifMatch != nil:
|
|
glog.Errorf("routedDeleteNullVersion: %s/%s on %s returned %q", bucket, object, owner, resp.Error)
|
|
return s3err.ErrInternalError, true
|
|
case resp.Error != "":
|
|
glog.Warningf("routedDeleteNullVersion: %s/%s returned %q, falling back to lock", bucket, object, resp.Error)
|
|
return s3err.ErrNone, false
|
|
default:
|
|
return s3err.ErrNone, true
|
|
}
|
|
}
|
|
|
|
// routedDemoteNullVersion removes the null version from the entry as read on
|
|
// the owner: a directory, which other keys may be nested under, is stripped as
|
|
// the lock path's demote does, and a file is deleted. The write is guarded by
|
|
// ifMatch and by the entry it was derived from, so once the last nested key is
|
|
// gone a concurrent routed delete cannot pass the same check.
|
|
func (s3a *S3ApiServer) routedDemoteNullVersion(owner pb.ServerAddress, bucket, object string, worm, bypass, metadataOnly bool, ifMatch *filer_pb.WriteCondition) s3err.ErrorCode {
|
|
fullpath := util.NewFullPath(s3a.bucketDir(bucket), object)
|
|
dir, name := fullpath.DirAndName()
|
|
entry, err := s3a.lookupEntryOnFiler(context.Background(), owner, dir, name)
|
|
switch {
|
|
case errors.Is(err, filer_pb.ErrNotFound):
|
|
return s3err.ErrPreconditionFailed
|
|
case err != nil:
|
|
glog.Errorf("routedDemoteNullVersion: %s/%s on %s: %v", bucket, object, owner, err)
|
|
return s3err.ErrInternalError
|
|
case entry.IsDirectory && !entry.IsDirectoryKeyObject():
|
|
return s3err.ErrPreconditionFailed
|
|
}
|
|
guard := &filer_pb.WriteCondition{Clauses: append(slices.Clone(ifMatch.GetClauses()),
|
|
&filer_pb.WriteCondition_Clause{Kind: filer_pb.WriteCondition_IF_ENTRY_EQUAL, ExpectedEntry: proto.Clone(entry).(*filer_pb.Entry)})}
|
|
mutation := &filer_pb.ObjectMutation{Type: filer_pb.ObjectMutation_DELETE, Directory: dir, Name: name, IsDeleteData: !metadataOnly, Condition: guard}
|
|
if entry.IsDirectory {
|
|
clearDirectoryMarkerMetadata(entry)
|
|
mutation = &filer_pb.ObjectMutation{Type: filer_pb.ObjectMutation_PUT, Directory: dir, Entry: entry, Condition: guard}
|
|
}
|
|
resp, err := s3a.objectTxnOnFiler(owner, &filer_pb.ObjectTransactionRequest{
|
|
LockKey: string(fullpath),
|
|
RouteKey: s3a.objectRouteKey(bucket, object),
|
|
Condition: wormDeleteCondition(worm, bypass),
|
|
Mutations: []*filer_pb.ObjectMutation{mutation},
|
|
})
|
|
switch {
|
|
case err != nil:
|
|
glog.Errorf("routedDemoteNullVersion: %s/%s on %s: %v", bucket, object, owner, err)
|
|
return s3err.ErrInternalError
|
|
case resp.ErrorCode == filer_pb.FilerError_PRECONDITION_FAILED:
|
|
if resp.FailedMutation != nil {
|
|
return s3err.ErrPreconditionFailed
|
|
}
|
|
return s3err.ErrAccessDenied
|
|
case resp.Error != "":
|
|
glog.Errorf("routedDemoteNullVersion: %s/%s on %s returned %q", bucket, object, owner, resp.Error)
|
|
return s3err.ErrInternalError
|
|
default:
|
|
return s3err.ErrNone
|
|
}
|
|
}
|
|
|
|
// nullVersionCondition narrows If-Match: * to an entry that carries an
|
|
// object, reporting whether it did: a nested entry stripped of its null
|
|
// version stays behind as a plain directory, which IF_EXISTS still matches.
|
|
func nullVersionCondition(ifMatch *filer_pb.WriteCondition) (*filer_pb.WriteCondition, bool) {
|
|
if ifMatch == nil {
|
|
return nil, false
|
|
}
|
|
cond, narrowed := proto.Clone(ifMatch).(*filer_pb.WriteCondition), false
|
|
for _, c := range cond.Clauses {
|
|
if c.Kind == filer_pb.WriteCondition_IF_EXISTS {
|
|
c.Kind, narrowed = filer_pb.WriteCondition_IF_OBJECT_EXISTS, true
|
|
}
|
|
}
|
|
return cond, narrowed
|
|
}
|
|
|
|
// ownerRefusedNonEmpty reports a transaction whose delete mutation the owner
|
|
// refused because other keys are nested under the entry, before applying it.
|
|
func ownerRefusedNonEmpty(resp *filer_pb.ObjectTransactionResponse) bool {
|
|
if resp.FailedMutation == nil {
|
|
return false
|
|
}
|
|
msg := strings.TrimPrefix(resp.Error, fmt.Sprintf("mutation %d: ", *resp.FailedMutation))
|
|
return errors.Is(filer.DeleteEntryError(msg), filer.ErrNonEmptyFolder)
|
|
}
|
|
|
|
// versionedFinalize flips the .versions latest pointer for a versioned PutObject:
|
|
// on the routed path RECOMPUTE_LATEST rides in the version file's PUT transaction,
|
|
// committing atomically under the object's per-path lock; off the ring
|
|
// updateLatestVersionInDirectory does it under the object write lock.
|
|
func (s3a *S3ApiServer) versionedFinalize(bucket, object string, versionId *string, useInvertedFormat bool) *putFinalize {
|
|
return &putFinalize{
|
|
lockKey: s3a.toFilerPath(bucket, object),
|
|
mutations: []*filer_pb.ObjectMutation{s3a.latestPointerRecompute(bucket, object, useInvertedFormat, "", true)},
|
|
newVersion: func(entry *filer_pb.Entry) string {
|
|
*versionId = generateVersionId(useInvertedFormat)
|
|
entry.Name = s3a.getVersionFileName(*versionId)
|
|
entry.Extended[s3_constants.ExtVersionIdKey] = []byte(*versionId)
|
|
return s3a.toFilerPath(bucket, object+s3_constants.VersionsFolder+"/"+entry.Name)
|
|
},
|
|
afterCreate: func(versionEntry *filer_pb.Entry) s3err.ErrorCode {
|
|
if err := s3a.updateLatestVersionInDirectory(bucket, object, *versionId, s3a.getVersionFileName(*versionId), versionEntry); err != nil {
|
|
glog.Errorf("putVersionedObject: failed to update latest version in directory: %v", err)
|
|
return s3err.ErrInternalError
|
|
}
|
|
return s3err.ErrNone
|
|
},
|
|
}
|
|
}
|
|
|
|
// finalizeSuspendedNullWrite retires the null delete marker a suspended DELETE left
|
|
// in .versions, so reads resolve the null version the caller just wrote at the
|
|
// regular path. Pointer first: clearing the marker while the pointer still names it
|
|
// makes reads rescan .versions and promote an older version. Call only once the
|
|
// write has committed — retiring the marker for a write that then fails republishes
|
|
// the deleted key.
|
|
//
|
|
// identityKey/identityValue name the extended attribute that marks the entry as the
|
|
// caller's write (an upload id, an etag). The cleanup rewrites shared .versions state
|
|
// off the object write lock, so it is skipped unless the regular path still holds that
|
|
// write: a DELETE that landed in between owns the null slot, and retiring its marker
|
|
// would resurrect an older version under a key that was deleted. Narrows that race,
|
|
// does not close it. owner, when set, is the filer the write went to, so the check
|
|
// reads its own write back rather than a peer that may be behind.
|
|
func (s3a *S3ApiServer) finalizeSuspendedNullWrite(owner pb.ServerAddress, bucket, object, identityKey, identityValue string) error {
|
|
dir, name := util.FullPath(s3a.toFilerPath(bucket, object)).DirAndName()
|
|
current, err := s3a.lookupEntryPreferringOwner(owner, dir, name)
|
|
if err != nil && !errors.Is(err, filer_pb.ErrNotFound) {
|
|
return fmt.Errorf("re-read %s/%s: %w", bucket, object, err)
|
|
}
|
|
if current == nil || string(current.Extended[identityKey]) != identityValue {
|
|
glog.V(2).Infof("finalizeSuspendedNullWrite: %s/%s superseded by a concurrent write", bucket, object)
|
|
return nil
|
|
}
|
|
|
|
if err := s3a.updateIsLatestFlagsForSuspendedVersioning(bucket, object); err != nil {
|
|
return err
|
|
}
|
|
// Best-effort: with the pointer gone the regular-path object already owns the
|
|
// null slot, so a surviving marker is neither read nor listed.
|
|
s3a.removeNullVersionFile(bucket, object)
|
|
return nil
|
|
}
|