* s3: stop retrying a delete the filer refused for a non-empty folder
The filer looked and the children are there, so the answer will not change.
retryFilerOp spent six attempts and up to 3.1s of backoff on it before the
caller could act on the condition it was already holding.
Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD
* s3: thread the request context through the unversioned delete path
doDeleteEntry issued every DeleteEntry on context.Background(), so an S3
client that hung up left the gateway working on its behalf, out of reach of
both cancellation and the per-request retry allowance that
DeleteMultipleObjectsHandler installs.
Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD
* s3: treat a cancelled filer RPC as terminal, not transient
isRetryableFilerErr matched context.Canceled and DeadlineExceeded by
sentinel, which only holds while the error is still local. Once it has
crossed gRPC it is a status, so an abandoned request was retried six times
on behalf of a caller that had already gone.
Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD
* s3: replay a delete whose reply the transport dropped
A delete is idempotent at the filer, which answers an entry that is already
gone with an empty resp.Error, so a reply lost in transit can be reissued
rather than surfaced. Surfaced, it becomes a 500 on the bucket delete, which
boto3 resends and is then answered NoSuchBucket, or a per-key InternalError
inside the 200 of a multi-object delete, which no SDK retries at all.
The replay runs through retryFilerOp, so it draws on the allowance the
request already installs rather than paying a backoff per key, and stops for
a caller that has gone. rm and rmObject re-enter WithFilerClient per attempt,
so each one walks the failover list again on a connection the failed attempt
had invalidated; the multi-object loop holds one client for the batch, so
there the replay reuses it.
Classification stays structural. The filer reports its own refusals in
resp.Error, which carries no status and has the deleted path - and, for a
recursive delete, the children it stopped on - formatted into it, so no key
name can steer the decision either way.
rm and rmObject now take the caller's context. Cleanup and rollback paths
pass context.Background() deliberately: they have to run whether or not the
caller is still waiting.
Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD
* s3: share one retry allowance across multipart completion cleanup
The unused-entry loop deletes once per entry, and each delete now retries,
so a filer that stays unavailable held the response for 3.1s per entry after
the object was already committed.
Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD
* util, pb: classify a filer error by the status the server sent
DoSeaweedListWithSnapshot wrapped a failed ListEntries with %v, dropping the
gRPC status, so IsTransientError fell back to matching substrings against a
message that now held the caller's path. Keep the status with %w and let it
decide, reading the server's own text rather than the wrapper's.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* s3: keep the bucket and prefix out of the list retry decision
A bucket named transport, or a prefix under logs/unavailable/, made a
PermissionDenied listing look transient and got it retried; a key holding the
not-found sentence suppressed a retry that should have run. Both checks now
read the filer's status, and only fall back to the text when there is none.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* filer, s3: classify a delete failure before the path is wrapped into it
The filer put the non-empty-folder marker behind its own "delete directory %s"
wrapper and the gateway matched it as a substring, so a key named after the
marker turned a real delete failure into the demote-the-marker no-op and the
request answered 204. Keep the marker leading the message that crosses the
wire, turn it back into a sentinel where the response is read, and match that.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* filer: keep the sentinel when CreateEntry reports an update failure
CreateEntry flattened the error UpdateEntry wraps, so errors.Is stopped
matching and ErrExistingIsDirectory and ErrExistingIsFile never reached
the S3 mapper, which answered a retryable 500 instead.
* s3: let a key that is a prefix of other keys be an object
S3 keys are flat, so "a/b" and "a/b/c" are independent objects that
coexist in either write order. The filer stores a key as a path, so one
of them has to live on the directory the other is nested under.
Writing the nested key first refused the prefix key outright. Writing it
second promoted the file to a directory, which kept its data but lost the
key: an empty object left nothing to recognise it by and disappeared, and
one with data listed under a trailing slash it never had.
Mark the directory that carries such a key, and write the object onto it
when the path is already a directory. The mark makes an empty prefix
object visible to listings and readable by GET and HEAD, keeps the empty
folder cleaner off it, and lists it under the key it was written with.
Deleting the key strips the mark back off along with the data.
* filer: keep a TTL off a directory that stands for an object
An expired entry is deleted a row at a time, so expiring a directory
removes it and leaves everything under it unreachable. Promoting a file
to a directory carried its TTL across, and a promoted file is exactly the
one that has keys nested under it.
Drop the TTL on promotion, and leave one an older build wrote alone. The
lifecycle worker still expires the object, through the delete that leaves
the directory behind.
* s3: delete the null version of a key other keys are nested under
The routed delete cannot remove an entry that other keys live under, and
answered a retryable 500 rather than falling back to the lock path the
unversioned delete already falls back to. That path then looked the entry
up under the bucket with the whole key as its name, so the demote wrote it
back one directory too high and failed as not found.
Fall back on any non-precondition error, and split the key before deleting
it. Trailing-slash directory markers with children reach the same delete.
* filer: keep the sentinel when MkFile and Mkdir report a create failure
Same flattening one layer out: every mkFile caller lost the sentinel, so
a CopyObject onto a key that other keys are nested under answered a
retryable 500 where a PutObject of the same key answers 409.
* s3: copy and rename a key that other keys are nested under
Such a key is stored on the directory those keys live in, and copy and
rename both refused it: the source lookup maps every directory entry to
NoSuchKey, so a key a plain GET serves could not be copied or moved, and
the destination side refused it as a directory conflict.
The source is read through a view of the entry as the object it names.
The destination is written the way a PutObject of that key writes it. A
rename at either end copies the object's own data across and strips it off
the source key rather than going through AtomicRenameEntry, which moves a
directory by moving everything under it - the nested keys are not part of
what is being renamed.
Chris Lu