s3: make conditional DeleteObject atomic on versioned buckets (#12624)

* s3: re-check a versioned DeleteObject If-Match under the object lock

A versioned or suspended DeleteObject without a versionId routed to the
owner filer after checking If-Match outside any lock, so concurrent
conditional deletes could all pass the check and each add a delete
marker. Keep conditional deletes on the lock path, which re-checks the
precondition under the object write lock, as conditional versioned PUTs
already do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: guard a conditional version delete on the owner filer

DeleteObject with a versionId checked If-Match before routing the
delete to the owner filer, so concurrent conditional deletes of one
version could all report success. Carry the If-Match clause into the
owner's transaction as a guard on the version (the null entry for
versionId=null), evaluated under the object's path lock before anything
applies, and map that guard failing to 412 instead of the object-lock
403. An If-Match the filer cannot evaluate stays on the lock path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: carry an If-Match ETag list into the routed delete

A DeleteObject whose If-Match listed several ETags fell back to the
gateway's object write lock, while a single-ETag delete of the same key
ran under the owner filer's path lock. The two locks do not exclude each
other, so a list delete and a single-ETag delete could both report
success. Send every candidate to the filer, whose strong comparison
matches checkDeleteIfMatch, so all conditional deletes with an owner are
decided under the owner's lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: fail a conditional null delete on an owner transport error

The owner may have committed the delete before the error, and the gateway
lock path it fell back to does not exclude routed deletes, so a second
If-Match delete could pass there too. Answer 500 as the specific-version
path does; unconditional deletes still fall back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: fail a conditional null delete on any owner error but the nested refusal

Only the owner's non-empty-folder refusal is known to apply nothing. After
any other owner error the delete may have been partly applied, and a lock-path
re-check does not exclude a concurrent routed delete of the same version, so a
conditional null-version delete answers 500 instead of falling back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: strip a nested conditional null delete on the owner

A conditional versionId=null delete that the owner refuses because other
keys are nested under the entry used to fall back to the gateway lock and
re-check If-Match there. That lock does not exclude routed deletes: once
the last nested key is removed, a concurrent routed delete can pass the
same check on the owner, and the fallback then treats the missing entry as
deleted, so both requests answer 204.

Demote the entry on the owner instead, in a transaction guarded by the
If-Match clauses and by the entry it was read as, with the object-lock
guards on the request. A missing or already demoted entry answers 412.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: add an IF_OBJECT_EXISTS write condition

IF_EXISTS also holds for a plain directory, which is what is left of a key
once the object on it is stripped while other keys stay nested under it.
IF_OBJECT_EXISTS holds only for an entry that carries an object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: require an object for a conditional null delete with If-Match: *

Stripping the null version off an entry other keys are nested under leaves a
plain directory. Once the last nested key is gone, a routed delete with
If-Match: * still found an entry there and removed the empty directory, so a
second conditional delete could also answer 204. The owner now checks that
the entry carries an object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: settle a wildcard null delete on the entry when the filer predates IF_OBJECT_EXISTS

A filer older than IF_OBJECT_EXISTS fails it closed, so with the gateway
upgraded first every If-Match: * delete of a null version answered 412. When
the narrowed guard fails, the gateway now reads the entry on the owner: absent
or a plain directory is a 412, otherwise the file is deleted, or the directory
stripped, under IF_EXISTS and IF_ENTRY_EQUAL on the entry it read. The demote
keeps the caller's If-Match clauses, since the entry guard already requires an
object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

weed/s3api/s3api_object_handlers_delete.go

@@ -173,7 +173,7 @@ func (s3a *S3ApiServer) deleteVersionedObject(r *http.Request, bucket, object, v
 		if owner := s3a.routableWriteOwner(bucket, object); owner != "" {
 			worm, lockErr := s3a.isObjectLockEnabled(bucket)
 			if lockErr == nil {
-				code, settled := s3a.routedDeleteNullVersion(owner, bucket, object, worm, worm && governanceBypassAllowed, false)
+				code, settled := s3a.routedDeleteNullVersion(owner, bucket, object, worm, worm && governanceBypassAllowed, false, nil)
 				if settled {
 					if code != s3err.ErrNone {
 						return result, code
@@ -288,29 +288,28 @@ func (s3a *S3ApiServer) DeleteObjectHandler(w http.ResponseWriter, r *http.Reque
 	// Fast path: route the delete to the owner filer under its per-path lock;
 	// routedObjectOwner excludes versioned/object-lock buckets.
 	if !deleteHandled && !versioningConfigured {
-		if cond, condOk := buildDeleteCondition(r); condOk {
-			if owner, ownerOk := s3a.routedObjectOwner(bucket, object); ownerOk {
-				resp, err := s3a.routedDelete(owner, bucket, object, cond)
-				switch {
-				case err != nil:
-					glog.Warningf("DeleteObjectHandler: routed delete to %s failed for %s/%s, falling back to lock: %v", owner, bucket, object, err)
-				case resp.ErrorCode == filer_pb.FilerError_PRECONDITION_FAILED:
-					deleteCode, deleteHandled = s3err.ErrPreconditionFailed, true
-				case resp.Error != "":
-					// Non-precondition error: fall back (the lock path handles cases
-					// the raw delete cannot, e.g. a non-empty directory marker).
-					glog.Warningf("DeleteObjectHandler: routed delete to %s returned %q for %s/%s, falling back to lock", owner, resp.Error, bucket, object)
-				default:
-					deleteCode, deleteHandled = s3err.ErrNone, true
-				}
+		if owner, ownerOk := s3a.routedObjectOwner(bucket, object); ownerOk {
+			resp, err := s3a.routedDelete(owner, bucket, object, buildDeleteCondition(r))
+			switch {
+			case err != nil:
+				glog.Warningf("DeleteObjectHandler: routed delete to %s failed for %s/%s, falling back to lock: %v", owner, bucket, object, err)
+			case resp.ErrorCode == filer_pb.FilerError_PRECONDITION_FAILED:
+				deleteCode, deleteHandled = s3err.ErrPreconditionFailed, true
+			case resp.Error != "":
+				// Non-precondition error: fall back (the lock path handles cases
+				// the raw delete cannot, e.g. a non-empty directory marker).
+				glog.Warningf("DeleteObjectHandler: routed delete to %s returned %q for %s/%s, falling back to lock", owner, resp.Error, bucket, object)
+			default:
+				deleteCode, deleteHandled = s3err.ErrNone, true
 			}
 		}
 	}
 	// Versioned/suspended delete with no specific version: route off the lock when
 	// the bucket has an owner. createDeleteMarker routes its own pointer flip; a
 	// delete marker never removes a locked version, so object-lock buckets route
-	// here too. The If-Match precondition was already checked above.
-	if !deleteHandled && versioningConfigured && versionId == "" {
+	// here too. A conditional delete stays on the lock path, which re-checks
+	// If-Match under the lock, so concurrent ones cannot all pass the check above.
+	if !deleteHandled && versioningConfigured && versionId == "" && r.Header.Get(s3_constants.IfMatch) == "" {
 		if owner := s3a.routableWriteOwner(bucket, object); owner != "" {
 			deleteResult, deleteCode = s3a.deleteVersionedObject(r, bucket, object, versionId, versioningState)
 			deleteHandled = true
@@ -322,10 +321,13 @@ func (s3a *S3ApiServer) DeleteObjectHandler(w http.ResponseWriter, r *http.Reque
 	// gate the delete on the version's WORM guards, evaluated on the owner — for
 	// governance bypass the retention guard is scoped to COMPLIANCE so the filer
 	// allows a governance-mode delete while still denying compliance and legal
-	// hold, without the gateway reading the version.
+	// hold, without the gateway reading the version. If-Match rides the same
+	// transaction, so concurrent conditional deletes of one version cannot both
+	// pass.
 	if !deleteHandled && versionId != "" {
 		worm, lockErr := s3a.isObjectLockEnabled(bucket)
 		bypass := worm && s3a.evaluateGovernanceBypassRequest(r, bucket, object)
+		ifMatch := buildDeleteCondition(r)
 		if lockErr == nil {
 			if owner := s3a.objectWriteOwner(bucket, object); owner != "" {
 				deleteResult.versionId = versionId
@@ -335,11 +337,11 @@ func (s3a *S3ApiServer) DeleteObjectHandler(w http.ResponseWriter, r *http.Reque
 					}
 				}
 				if versionId == "null" {
-					deleteCode, deleteHandled = s3a.routedDeleteNullVersion(owner, bucket, object, worm, bypass, false)
+					deleteCode, deleteHandled = s3a.routedDeleteNullVersion(owner, bucket, object, worm, bypass, false, ifMatch)
 				} else {
 					// A version id never changes kind, so a marker read here is
 					// the one deleted; markers carry no WORM guard.
-					deleteCode, deleteHandled = s3a.routedDeleteSpecificVersion(owner, bucket, object, versionId, worm && !deleteResult.deleteMarker, bypass, false), true
+					deleteCode, deleteHandled = s3a.routedDeleteSpecificVersion(owner, bucket, object, versionId, worm && !deleteResult.deleteMarker, bypass, false, ifMatch), true
 				}
 			}
 		}