A remote-only entry whose object was deleted from the remote storage
outside the filer answered GET with 500 and stayed in the filer. The
remote's not-found was lost on the way: the backends' ReadFile returned
it as an untyped error, so FetchAndWriteNeedle failed with codes.Unknown
and nothing downstream could tell it from any other failure.
- remote_storage: GCS, S3 (NoSuchKey) and Azure (BlobNotFound) reads
return ErrRemoteObjectNotFound. GCS reports a missing bucket the same
way as a missing object, so it confirms the bucket with a listing.
- volume server and filer: the not-found crosses gRPC as codes.NotFound
carrying the sentinel's text, and the filer's cache RPC returns
codes.NotFound, which the S3 gateway already maps to NoSuchKey.
- s3api: the origin fallback answers NoSuchKey on a confirmed not-found.
- filer: a confirmed not-found removes the stale entry, so the lazy
remote-metadata cache converges on the remote. Only remote-only files
outside .versions and without an active object lock are removed, only
if unchanged since the fetch (checked on the object's write owner,
under the lock S3 object writes take), and with a metadata-only
delete: the filer skips its inline remote delete, and the delete
events' entries carry a marker that makes filer.remote.sync and
filer.remote.gateway skip their remote delete, while filer.sync still
replicates it. The replicated DeleteEntryRequest carries
keep_remote_object, so the destination's delete events are marked too.
The store drops the marker from every write, so clients cannot plant
it.
* util, pb: classify a filer error by the status the server sent
DoSeaweedListWithSnapshot wrapped a failed ListEntries with %v, dropping the
gRPC status, so IsTransientError fell back to matching substrings against a
message that now held the caller's path. Keep the status with %w and let it
decide, reading the server's own text rather than the wrapper's.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* s3: keep the bucket and prefix out of the list retry decision
A bucket named transport, or a prefix under logs/unavailable/, made a
PermissionDenied listing look transient and got it retried; a key holding the
not-found sentence suppressed a retry that should have run. Both checks now
read the filer's status, and only fall back to the text when there is none.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* filer, s3: classify a delete failure before the path is wrapped into it
The filer put the non-empty-folder marker behind its own "delete directory %s"
wrapper and the gateway matched it as a substring, so a key named after the
marker turned a real delete failure into the demote-the-marker no-op and the
request answered 204. Keep the marker leading the message that crosses the
wire, turn it back into a sentinel where the response is read, and match that.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
Peter Dodd