3 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandDevin	74f196ceab	s3: route regular copy source reads to the write owner (#12173) ... * s3: read regular copy sources from their write owner * filer: report mutation sentinel codes in ObjectTransaction responses A failed mutation only surfaced resp.Error text, so routed writers could not distinguish an existing-directory conflict from a generic failure and could not apply the prefix-object retry the plain CreateEntry path has. * s3api: retry a routed mkfile as a prefix object over a directory mkFile retries ErrExistingIsDirectory by marking the entry a prefix object; the routed path returned the conflict instead, so a routed write to a key that is also a prefix failed where the lock path succeeds. The filer now reports the code on the transaction response, making the retry recognizable. * s3api: commit copy destinations on the key's write owner An owner-routed source read still left the destination on the gateway's default filer, so a second copy resolved the first copy's destination on the owner, found nothing, and returned NoSuchKey. Write unversioned and suspended destinations through routedMkFile, and write the version file on the owner the finalize recomputes on. * s3api: cache remote objects on the filer that served the entry An owner-routed lookup can return a remote-only entry that has not replicated to the gateway's default filer; the cache RPC then cannot find it and the copy or read degrades to a 503. Ask the entry's write owner to cache it first, falling back to the default filer. * s3: read versioned object state from the key's write owner CopyObject destinations now commit version files and the .versions pointer on the owner's object-scope ring key, so versioned reads through the default filer could miss them until replication. Generalize the routed read to dir/name lookups and paginated lists scoped to the object's owner, and point the versioned read paths at it: getSpecificObjectVersion, doGetLatestObjectVersion, the .versions quick-checks in Get/Head/attributes, the self-heal rescans, tagging, delete conditionals, directory-marker history checks, rename destination, multipart completion checks, and the version-id format probe. Reads fall back to the default filer when no ring owner is resolvable, preserving single-filer deployments. * test(s3api): fake RECOMPUTE_LATEST/DELETE and cover versioned copy reads The ambiguous-PUT fake filer now creates parent directories on PUT (mirroring InsertEntry), applies DELETE mutations, and re-derives the .versions pointer for RECOMPUTE_LATEST the way applyRecomputeLatest does, so owner-vs-replica tests can exercise versioned writes. TestVersionedCopyDestinationReadsOnOwner copies into a versioned bucket whose writes commit on the owner while the gateway's default filer is a replica with no replication, then asserts reads by versionId, latest-version resolution, and a copy chained off the new version all resolve on the owner. * test(s3api): gofmt the versioned-copy coverage * s3: only swap in a cache fill of the resolved remote generation A cache response can come back from a filer holding an older generation of the key. Replacing the authorized source or swapping its chunks under a size and range computed for the resolved entry serves the wrong object or truncates the response. Match RemoteEntry identity and size before using a cached entry, in both the copy and streaming paths. * filer: keep directory attributes when an object lands as prefix object The object-onto-directory retry replaced the existing directory's mode, owner, and extended attributes with the object's. Preserve the directory identity; merge directory-held extended keys only when the old entry never held an object, so a prior generation's object metadata is not carried forward. * s3: keep marker lock checks on the filer that replaces the marker The object-lock precondition read the key's write owner while mkdir still replaces the marker on the default filer, so a locked marker not yet replicated to the owner could be overwritten. Read the entry where the replacement lands, in both PutObjectHandler and zip extraction. Also hash a trailing-slash marker key to the same route key as the object it names, so marker and object share one owner and one write lock. * s3: route tag writes to the owner tag reads resolve GetObjectTagging reads the key's write owner, but PutObjectTagging and DeleteObjectTagging still wrote through the default filer, so a successful change returned the owner's stale tags until replication. setTags, rmTags, and the versioned tag updates now write on the owner first, falling back to the default filer when no owner resolves. * s3: keep rename lookups on the filer that performs the move AtomicRenameEntry and the retry token it carries both execute on the default filer, but the source and destination reads went to the write owner — a completed rename's token was invisible to its own retry, and a source that only exists on the default filer looked absent. Read both ends on the default filer. * s3: send the .versions pointer repair with the object's route key The heal reads the .versions entry on the object's owner, but the conditional UpdateEntry is routed by the .versions path's own ring key, which can hash to a different filer — leaving the pointer the routed reads consult unchanged. Ship the CAS repair as a PATCH_EXTENDED ObjectTransaction on the object's route key instead. * filer: preserve only directory identity on a prefix-object write Merging the old directory's extended attributes could carry a marker's encryption, version, or etag fields onto the new plaintext object, since a directory's own keys cannot be told apart from the object it held. Keep only mode, uid, gid, user name and group names. * s3: also require matching size when a cache fill replaces the source The remote-generation check covers identity; size equality is the cheap guard that keeps the streamed or copied range consistent with what was resolved. * s3: lock the object path for a routed .versions pointer repair conditionalUpdateEntryRouted held the lock on the .versions path while version writes and deletes serialize on the object path, so a repair could pass IF_ENTRY_EQUAL and then apply over a pointer a writer had advanced in between. Lock the object path like the writers and keep the condition on the .versions entry via condition_key. * s3: do not replay an owner-side tag mutation on another filer withWriteOwnerFilerClient re-ran fn on the default filer after any owner error, so an authoritative not-found could update a stale replica entry and still report success, and a post-apply transport error could apply the mutation twice. The failover walk now ends on any error fn returns; only a channel that never came up moves to another filer. * filer: keep a directory's xattrs when a prefix object replaces it The prefix-object branch preserves the directory's identity but dropped its Extended map wholesale; FUSE xattrs live there under the xattr- prefix and belong to the directory, not to the object the marker held. * s3: delete null and specific versions on the filer that serves them On the default filer an absent entry deletes as success, so a version that only reached the write owner survived a reported success while owner-routed reads kept serving it. The suspended-versioning and lifecycle deletes now go through routedDeleteNullVersion and routedDeleteSpecificVersion when an owner resolves; an unsettled attempt keeps the lock path. * s3: route a lifecycle null-version delete to the object entry The "null" version is the regular object entry, not a .versions file, so it goes through routedDeleteNullVersion; only a real version id takes the .versions-file delete. * s3: repoint .versions only while it still names the deleted version routedDeleteSpecificVersion recomputed the latest pointer on every version delete, so removing a noncurrent version re-derived the pointer to the newest remaining file — resurrecting an older version over a cleared pointer (a suspended bucket's null write keeps it clear while the null object stays current) or over a pointer that moved on. The pointer is now read on the owner and repointed under an IF_ENTRY_EQUAL guard only while it still names the deleted version; a contested pointer is left for the self-heal path rather than overwritten by a stale scan. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: let lifecycle expiry keep its metadata-only deletes on routed paths The routed lifecycle deletes hardcoded IsDeleteData, so expiring an entry with a volume TTL queued per-chunk deletions instead of leaving the data for the volume to reclaim — extra work on the owner for deletes recordMetadataOnlyIf still counts as metadata-only. Pass the caller's cleanup choice through routedDeleteNullVersion and routedDeleteSpecificVersion; the interactive DELETE path keeps deleting data. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: let RECOMPUTE_LATEST skip a pointer that moved on A version delete repoints .versions only while the pointer still names the deleted version, but the gateway cannot carry the pointer's current state into the transaction's condition — the condition slot is already spent on the version's WORM guards. only_if_current_name moves the check under the transaction lock: the filer compares the pointer's name_to_key value before scanning and skips the recompute when it moved (or was cleared by a suspended bucket's null write), so a noncurrent delete cannot resurrect an older version. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: repoint inside the version's locked delete repointLatestBeforeVersionDelete ran in its own transaction ahead of the delete, so a version delete denied by retention or legal hold still moved the latest pointer — GET and HEAD went on serving the older version it rolled to. The repoint and the delete now ride one transaction behind the version's WORM guards, atomically gated by the new only_if_current_name recompute guard instead of a stale-read IF_ENTRY_EQUAL. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 14:35:39 -07:00
Chris Lu	f3e4a73696	s3: return NoSuchKey/NoSuchBucket for a missing CopyObject source (#10332) ... * s3: CopyObject returns NoSuchKey for a missing copy source * s3: CopyObject returns NoSuchBucket for a missing source bucket	2026-07-14 11:53:53 -07:00
Chris Lu	77694d3a93	s3: surface transient store errors during multipart resume/copy instead of masking them (#10207) ... * s3: surface multipart-list store errors instead of masking them listMultipartUploads masked a filer/metadata-store list error as an empty 200 response, and listObjectParts masked it as NoSuchUpload. A client resuming a multipart upload (the Docker Registry S3 driver resolves an in-progress upload via ListMultipartUploads, then ListParts) reads either as "upload gone" and fails the upload permanently with a non-retryable error. Return ErrInternalError so a transient store error stays retryable, keeping ErrNotFound as an empty list / NoSuchUpload respectively. * s3: return retryable error for transient CopyObject source lookups Resolving the copy source is reported as InvalidArgument ("Copy Source must mention the source bucket and key") whenever the lookup returns any error, including a transient store error. Clients don't retry a 400, so a resumable blob commit fails permanently. Keep the client error for a missing, invalid, directory, or delete-marker source; map a transient store error to ErrInternalError. * s3: mark versioned lookup miss with the not-found sentinel recoverLatestVersionWithoutPointer's terminal miss (a .versions directory with no pointer, no versions, and no null object) returned a plain error, so errors.Is-based callers could not tell this designed NoSuchKey state from a store failure and reported it as an internal error. * s3: reject a delete-marker copy source with NoSuchKey getLatestObjectVersion returns the delete-marker entry when the latest version is a delete marker, and the copy handlers copied its empty stub into the destination. Every other handler detects ExtDeleteMarkerKey and answers NoSuchKey; do the same for CopyObject and UploadPartCopy. * s3: align copy-source not-found detection with the grpc status idiom The lookup path canonicalizes text-form not-found into the sentinel in filer_pb.LookupEntry and wraps with %w after that, so matching sentinel text here was unreachable -- and risky, since a store error whose message merely mentions the sentinel would be downgraded to a terminal 400. Match the raw grpc NotFound code instead, like the other version-lookup sites, and give transient filer errors the 503 that the upload-entry lookup in this file already returns. * s3: keep source-bucket versioning lookup errors retryable in copy CopyObject and UploadPartCopy mapped any source-bucket versioning-state lookup error to a terminal InvalidCopySource, including transient store errors; getVersioningState signals a missing bucket with the not-found sentinel, so split on that and let real store errors surface as 500, the same mapping the destination-bucket lookup already uses. * s3: match grpc-transported not-found in multipart list error paths The list client path returns raw grpc status errors without the sentinel reconstruction that lookups get in filer_pb.LookupEntry, so errors.Is on the not-found sentinel never matched a store-reported missing directory; match the sentinel text as well via a shared helper. The common missing- directory case still lists as empty with no error and is unaffected. * s3: complete and abort multipart surface store errors prepareMultipartCompletionState mapped any upload-directory list or lookup error to NoSuchUpload, so a transient store error at completion time made the client discard a fully-uploaded object as gone. Split on not-found like the sibling listing paths. abortMultipartUpload did the same on s3a.exists, whose errors are never not-found (filer_pb.Exists reports that as false with no error) -- a store failure there answered NoSuchUpload and silently leaked the uploaded parts. * s3: reject part numbers below 1 in UploadPartCopy Only the upper bound was checked, unlike PutObjectPart; partNumber=0 passed the route regex and validation and wrote an out-of-range 0000_copy.part into the upload directory.	2026-07-02 13:25:04 -07:00