* filer: self-heal fetchWholeChunk on stale volume locations
Upstream #10156/#10800 wired cache invalidation into the buffer-based
read paths, but manifest resolution still goes through fetchWholeChunk,
which returns the raw error on failure. When cached volume locations
are stale (volume tiered to remote storage, server rolled), resolving
a large multipart file fails permanently even though other locations
are healthy.
Thread the ChunkGroup's cacheInvalidator through ResolveChunkManifest /
ResolveOneChunkManifest / fetchWholeChunk, and on failure invalidate,
re-lookup and retry once via the existing retryFetchWithFreshLocations
helper. The streaming bytesBuffer is reset before the retry so partial
bytes from the failed attempt cannot corrupt the manifest
proto.Unmarshal. Non-mount callers pass nil and keep their semantics.
* filer: move the manifest self-heal tests in with the other manifest tests
Also make the stale server stream a prefix and then abort mid-body, which is
what actually leaves partial bytes in the buffer: an HTTP error status returns
before ReadUrlAsStream ever calls the writer, so a 500 never exercised the
Reset the tests claimed to cover.
Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD
* filer: keep the cached volume locations when a manifest read is cancelled
A cancelled or timed-out read says nothing about where the volume lives, so
dropping the location and going back to the master only costs the next reader
a round trip. PrepareStreamContentWithThrottler already guards its self-heal
this way. The guard also goes inside retryFetchWithFreshLocations, since the
caller can be cancelled between its own check and the invalidation, and that
covers the reader cache and prefetch paths too.
fetchWholeChunk returns the context error rather than the stream failure it
provoked, and ResolveOneChunkManifest wraps with %w so errors.Is still sees it.
That matters even where no invalidator is passed: volume.fsck resolves
manifests with nil and tells its own abort from a corrupt manifest that way,
so the cancellation check sits ahead of the nil-invalidator return.
Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD
* filer: self-heal manifest reads on the filer and s3 paths too
Every caller that already holds the location cache backing its lookup function
can hand it over: the filer's read, copy and deletion paths and the log cache
have the MasterClient right there, and s3api has the FilerClient. MinusChunks
takes one for the same reason, since the deletion path resolves manifests
through it. Only the shell tools and the replication sinks, whose lookup
functions cache privately with nothing to invalidate, keep passing nil.
Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD
---------
Co-authored-by: bruce-zzz <bruce.zou@hhy-data.com>
* s3: track manifest blob ownership through multipart completion
Manifest blobs made three orphan paths. A partial fold that failed midway
kept its earlier batches on volume servers while the write fell back to
flat chunks; the fold now records each saved blob and deletes them on
error. A completion that failed after preparing left its fresh manifests
behind on every retry; the completion state now owns them and deletes
them unless a failed rollback left the version entry still holding them.
And a completed upload removes its parts metadata-only, which stranded
the part-manifest blobs superseded by flattening; those are collected
during flattening and deleted once the completion commits.
Two shared-chunk hazards nearby: the version-file rollback deleted its
data, destroying the still-registered parts (worse once manifests
resolve to inner chunks), and the idempotent-replay cleanup data-deleted
leftover parts whose chunks the live object references. Both are
metadata-only now.
* s3: trim chunk manifest comments
* s3: test manifest fold rollback and part range selection
The fold-with-rollback and the boundary-to-byte-range logic were only
exercised by hand against a live server; give both an injectable seam
and cover the fold, the below-threshold and SSE no-ops, the midway
failure deleting the blobs it saved, and offset-vs-legacy-index range
selection including indexes that no longer address the chunk list.
s3: fold large chunk lists into manifest chunks on the direct write path
The S3 gateway uploads chunks itself and hands the filer a fully prepared
entry. On the routed write path (ObjectTransaction) the filer stores that
entry as-is, so a large PutObject or CompleteMultipartUpload persisted its
whole flat chunk list - a 900GB object carries 120k chunk references in one
entry. Manifestize on the gateway before the entry is written, the same way
mount, WebDAV, and filer.copy prepare theirs.
Multipart part boundaries also record byte offsets now: the stored chunk
indexes stop matching the entry once the list is folded, and partNumber
reads plus GetObjectAttributes prefer the offsets. Legacy index-only
records still work, with bounds checks instead of a possible panic.
Copy paths resolve a manifested source into data chunks before their
per-chunk copy loops - copying a manifest chunk raw would store its blob
as object data still pointing at the source - and a large copied list is
folded again on the destination. Completion likewise resolves manifest
chunks a part entry may carry (the filer folds an oversized UploadPartCopy
range) before rebasing part offsets.
