3 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	7bb0a1c127	s3: replay a delete whose reply the transport dropped (#11022) ... * s3: stop retrying a delete the filer refused for a non-empty folder The filer looked and the children are there, so the answer will not change. retryFilerOp spent six attempts and up to 3.1s of backoff on it before the caller could act on the condition it was already holding. Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD * s3: thread the request context through the unversioned delete path doDeleteEntry issued every DeleteEntry on context.Background(), so an S3 client that hung up left the gateway working on its behalf, out of reach of both cancellation and the per-request retry allowance that DeleteMultipleObjectsHandler installs. Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD * s3: treat a cancelled filer RPC as terminal, not transient isRetryableFilerErr matched context.Canceled and DeadlineExceeded by sentinel, which only holds while the error is still local. Once it has crossed gRPC it is a status, so an abandoned request was retried six times on behalf of a caller that had already gone. Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD * s3: replay a delete whose reply the transport dropped A delete is idempotent at the filer, which answers an entry that is already gone with an empty resp.Error, so a reply lost in transit can be reissued rather than surfaced. Surfaced, it becomes a 500 on the bucket delete, which boto3 resends and is then answered NoSuchBucket, or a per-key InternalError inside the 200 of a multi-object delete, which no SDK retries at all. The replay runs through retryFilerOp, so it draws on the allowance the request already installs rather than paying a backoff per key, and stops for a caller that has gone. rm and rmObject re-enter WithFilerClient per attempt, so each one walks the failover list again on a connection the failed attempt had invalidated; the multi-object loop holds one client for the batch, so there the replay reuses it. Classification stays structural. The filer reports its own refusals in resp.Error, which carries no status and has the deleted path - and, for a recursive delete, the children it stopped on - formatted into it, so no key name can steer the decision either way. rm and rmObject now take the caller's context. Cleanup and rollback paths pass context.Background() deliberately: they have to run whether or not the caller is still waiting. Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD * s3: share one retry allowance across multipart completion cleanup The unused-entry loop deletes once per entry, and each delete now retries, so a filer that stays unavailable held the response for 3.1s per entry after the object was already committed. Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD	2026-08-28 14:30:21 -07:00
Chris Lu	863fec6c3f	S3: let a key that is a prefix of other keys be an object (#10912) ... * filer: keep the sentinel when CreateEntry reports an update failure CreateEntry flattened the error UpdateEntry wraps, so errors.Is stopped matching and ErrExistingIsDirectory and ErrExistingIsFile never reached the S3 mapper, which answered a retryable 500 instead. * s3: let a key that is a prefix of other keys be an object S3 keys are flat, so "a/b" and "a/b/c" are independent objects that coexist in either write order. The filer stores a key as a path, so one of them has to live on the directory the other is nested under. Writing the nested key first refused the prefix key outright. Writing it second promoted the file to a directory, which kept its data but lost the key: an empty object left nothing to recognise it by and disappeared, and one with data listed under a trailing slash it never had. Mark the directory that carries such a key, and write the object onto it when the path is already a directory. The mark makes an empty prefix object visible to listings and readable by GET and HEAD, keeps the empty folder cleaner off it, and lists it under the key it was written with. Deleting the key strips the mark back off along with the data. * filer: keep a TTL off a directory that stands for an object An expired entry is deleted a row at a time, so expiring a directory removes it and leaves everything under it unreachable. Promoting a file to a directory carried its TTL across, and a promoted file is exactly the one that has keys nested under it. Drop the TTL on promotion, and leave one an older build wrote alone. The lifecycle worker still expires the object, through the delete that leaves the directory behind. * s3: delete the null version of a key other keys are nested under The routed delete cannot remove an entry that other keys live under, and answered a retryable 500 rather than falling back to the lock path the unversioned delete already falls back to. That path then looked the entry up under the bucket with the whole key as its name, so the demote wrote it back one directory too high and failed as not found. Fall back on any non-precondition error, and split the key before deleting it. Trailing-slash directory markers with children reach the same delete. * filer: keep the sentinel when MkFile and Mkdir report a create failure Same flattening one layer out: every mkFile caller lost the sentinel, so a CopyObject onto a key that other keys are nested under answered a retryable 500 where a PutObject of the same key answers 409. * s3: copy and rename a key that other keys are nested under Such a key is stored on the directory those keys live in, and copy and rename both refused it: the source lookup maps every directory entry to NoSuchKey, so a key a plain GET serves could not be copied or moved, and the destination side refused it as a directory conflict. The source is read through a view of the entry as the object it names. The destination is written the way a PutObject of that key writes it. A rename at either end copies the object's own data across and strips it off the source key rather than going through AtomicRenameEntry, which moves a directory by moving everything under it - the nested keys are not part of what is being renamed.	2026-08-24 15:10:34 -07:00
Chris Lu	2d9ea0285c	s3: add the RenameObject endpoint (#10659) ... * s3: add the RenameObject endpoint PUT /{bucket}/{key}?renameObject with x-amz-rename-source moves an object through the filer's AtomicRenameEntry, so no bytes are read or rewritten and the ETag, tags and SSE keys travel with the entry. Only unversioned buckets: a versioned rename would have to rebuild the .versions chain, and AWS offers RenameObject on directory buckets, which cannot be versioned. The source arrives in a header, so it is authorized separately for read and delete; both keys are locked, in key order, across the precondition checks and the move. * s3: let a matched source ETag precondition settle its date precondition RFC 7232 has an ETag precondition outrank the date precondition on its own side, and AWS documents the same for CopyObject: a matching x-amz-copy-source-if-match with a failing x-amz-copy-source-if-unmodified-since copies rather than returning 412. The source check evaluated all four headers in sequence, so the date header could still veto a decided ETag match. validateConditionalHeadersForReads already applies this precedence; the source path now matches it. * s3: cover a rename source named as a directory without a trailing slash Renaming a directory would move a whole subtree, so it has to stay a missing key whether or not the caller wrote the trailing slash. * s3: accept a bare object key as the RenameObject source AWS spells x-amz-rename-source both ways. Its CLI, Java and Rust examples pass the bare source key, and only a second CLI example and the boto3 conditional example pass bucket/key; the API reference's own example is a bare key too. The header was read as bucket/key only, so the form AWS leads with was rejected with InvalidArgument and the endpoint was unusable as documented. A value is now read as a literal key first — the only reading that can never name the wrong object — and as bucket-qualified second, when it carries the request's own bucket and the literal key does not exist. That costs one extra lookup only for a source that starts with the bucket's own name. Another bucket's name in the source is no longer a distinct error: RenameObject moves within one bucket, so it is simply part of a key this bucket does not hold, and it reports NoSuchKey. * s3: only a proven absence picks the other reading of a rename source A source that resolves to a directory is not a miss to fall through on: the literal path is still what the caller named, so answering for it beats renaming a different object under the bucket-qualified reading. With a directory at bucket/source.txt and an object at source.txt, a rename naming the former moved the latter. A failed lookup is not a proof of absence either, so a blip can no longer redirect a rename to the other reading.	2026-08-08 21:24:30 -07:00