S3: track manifest blob ownership through multipart completion (#10386)

* s3: track manifest blob ownership through multipart completion

Manifest blobs made three orphan paths. A partial fold that failed midway
kept its earlier batches on volume servers while the write fell back to
flat chunks; the fold now records each saved blob and deletes them on
error. A completion that failed after preparing left its fresh manifests
behind on every retry; the completion state now owns them and deletes
them unless a failed rollback left the version entry still holding them.
And a completed upload removes its parts metadata-only, which stranded
the part-manifest blobs superseded by flattening; those are collected
during flattening and deleted once the completion commits.

Two shared-chunk hazards nearby: the version-file rollback deleted its
data, destroying the still-registered parts (worse once manifests
resolve to inner chunks), and the idempotent-replay cleanup data-deleted
leftover parts whose chunks the live object references. Both are
metadata-only now.

* s3: trim chunk manifest comments

* s3: test manifest fold rollback and part range selection

The fold-with-rollback and the boundary-to-byte-range logic were only
exercised by hand against a live server; give both an injectable seam
and cover the fold, the below-threshold and SSE no-ops, the midway
failure deleting the blobs it saved, and offset-vs-legacy-index range
selection including indexes that no longer address the chunk list.
This commit is contained in:
Chris Lu authored and GitHub committed 2026-07-21 08:59:40 -07:00
1 parent 5a54beac80
commit 68a4e3347f
7 files changed
+235 -68

No files matched your search

+3 -5
View File
@@ -423,8 +423,7 @@ func (s3a *S3ApiServer) CopyObjectHandler(w http.ResponseWriter, r *http.Request
return
}
// Re-fold a large copied chunk list into manifest chunks, mirroring the
// PutObject path (no-op below filer.ManifestBatch or for SSE chunks).
// re-fold a large copied chunk list, mirroring the PutObject path
dstEntry.Chunks = s3a.manifestizeChunks(fmt.Sprintf("%s/%s", s3a.bucketDir(dstBucket), dstObject), dstBucket, 0, dstChunks)
// Apply destination-specific metadata (e.g., SSE-C IV and headers)
@@ -919,9 +918,8 @@ func (s3a *S3ApiServer) CopyObjectPartHandler(w http.ResponseWriter, r *http.Req
entry = cachedEntry
}
// The part-copy paths below iterate entry.GetChunks() per chunk, so a
// manifested source must be resolved into its data chunks first.
if err := s3a.flattenManifestChunks(r.Context(), entry); err != nil {
// per-chunk part-copy needs a flat source; resolved manifests stay with the source
if _, err := s3a.flattenManifestChunks(r.Context(), entry); err != nil {
glog.Errorf("CopyObjectPartHandler: resolve source manifest chunks %s/%s: %v", srcBucket, srcObject, err)
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
return