3 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandClaude Opus 5.5	d659228338	shell: s3.clean.uploads frees an open upload under the object write lock (#12558) ... * shell: s3.clean.uploads deletes uploads over filer gRPC Same DeleteEntry the HTTP handler runs, without minting a filer JWT, and the gRPC path is the one a locked, routed delete needs next. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads frees an open upload under the object write lock The cleaner deleted a stale upload recursively with no lock, so a CompleteMultipartUpload could pass its upload-exists check after the part entries and their chunks were gone, and publish an object over freed chunks. Do what AbortMultipartUpload does: hold the object's distributed write lock, and free an open upload in an ObjectTransaction routed to the object's owner filer, conditioned on the object not carrying the upload; if it does, delete the metadata only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads keeps the object write lock for the whole cleanup A large upload's recursive delete can outlast one 15 second lease, and a completion on the distributed-lock path could then take the lock and commit over chunks being freed. Renew the lock in the background and skip the delete once it is lost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads locks the path a completion commits under For a trailing-slash key a non-versioned completion writes and locks the entry inside the directory the key names, not the object path, so the cleaner's transaction did not serialize with it. Lock and check the same path the completion uses for the bucket's versioning state; the route key stays the object's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: abort locks the path a completion commits under A routed abort locked and checked the object path, but for a trailing-slash key a non-versioned completion commits under the entry inside the directory the key names. Use the same path the completion locks for the bucket's versioning state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: end s3.clean.uploads bucket failure line with a newline Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cluster: NewBlockingLongLivedLockWithin bounds the wait on a held lock Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads skips an upload whose object lock stays held Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cluster: share the contended lock filer setup across lock tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads reads the bucket's versioning state per upload The commit path a routed completion locks depends on versioning for a trailing-slash key. Reading it once per bucket before the run left a window as long as the run; read it just before the delete, as the gateway's abort does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 10:16:20 -08:00
Ryan Leung	7ad8419815	shell: preserve completed uploads for bucket-root objects (#12545) ... Signed-off-by: rleungx <rleungx@gmail.com>	2026-11-10 19:39:10 -08:00
Chris Lu	15520f601f	s3: commit multipart upload and remove .uploads atomically; purge completed uploads metadata-only (#11375) ... * s3: commit versioned multipart upload in one transaction CompleteMultipartUpload wrote the version file, flipped the .versions pointer, then removed .uploads/<id> metadata-only as a best-effort post-commit step. A filer error or gateway crash in that window left the upload directory referencing the same chunks as the published object, and the next s3.clean.uploads run purged it with data -- corrupting a committed object. Put the version file, remove the upload directory metadata-only (its chunks are the object's chunks), and recompute the latest pointer in one ObjectTransaction under the object's per-path lock on the owner filer. The mutation order keeps every partial state safe: the chunks stay referenced at all times, and a published object never coexists with the upload directory the cleaner would purge. Unused part entries are freed before the transaction, since the metadata-only directory delete would otherwise leak their chunks. * s3: remove upload directory inside the multipart object PUT The same committed-object/stranded-upload window existed on the suspended and non-versioned paths: writeMultipartObject committed the object, then a best-effort rm dropped .uploads/<id>. Ride the metadata-only removal on the routed PUT itself so the two land in one transaction; the unrouted mkFile fallback keeps post-commit cleanup. * shell: purge completed uploads metadata-only in s3.clean.uploads A leftover .uploads/<id> can outlive a committed object when the completion's metadata-only delete fails or the gateway dies in between; its part entries then share chunks with the live object, and a recursive purge frees them out from under it. Before purging a stale upload, check whether it completed: the object entry or any version file under <key>.versions carrying the upload id. If so, delete with skipChunkDeletion. If the lookup fails, skip the upload for this run rather than risk live chunks. * s3: abort multipart completion when unused part cleanup fails Deleting the upload directory metadata-only erases the only metadata pointing at part entries whose deletion failed, orphaning their chunks. Propagate the error so the completion fails while the upload directory still exists and the request remains retriable. * s3: require the upload directory to exist at multipart commit A delete that does not take the object lock (abort, lifecycle, s3.clean.uploads) can remove .uploads/<id> and its chunks between the prepare step and the commit transaction. The commit now carries an IF_EXISTS precondition on the upload directory so the race fails the request with NoSuchUpload instead of publishing an object over freed chunks. * s3: keep the version file when the upload directory is gone The finalize transaction has no rollback, so a failure at the latest-pointer recompute leaves the version written and .uploads/<id> removed. Deleting the version then destroys the only remaining record of the upload, making a retried CompleteMultipartUpload return NoSuchUpload while the version's chunks leak. Roll back only while the upload directory survives; otherwise keep the version, which a retry resolves through SeaweedFSUploadId and the version reconciler promotes. * s3: keep manifests when a routed object write partially commits For non-versioned and suspended completions the object PUT precedes the upload-directory DELETE, so an error can mean the object entry exists while the response reports failure. Freeing this attempt's manifest chunks then destroys the committed object. Keep them when the object entry survived, and after a failed null-marker finalize which always follows a committed write. * s3: skip the keep-version path on precondition failure A rejected precondition means no mutation ran, so there is no version file to preserve and this attempt's manifests are orphans the error cleanup should free. * s3: keep manifests when the object-existence check itself fails A transient lookup error previously read as absent, letting the error cleanup free manifest chunks a committed object still references. * s3: keep the upload directory when post-commit part cleanup fails Removing it metadata-only after a failed entry delete erases the only reference to the leftover chunks. Leave the directory so the entries keep their chunk references for s3.clean.uploads or manual recovery. * pb: fix filer list entry counting on 32-bit int(limit) wraps to -1 on 386 when limit is math.MaxUint32, so the beyond-limit check discarded every streamed entry. Compare in uint64 instead; the semantics are unchanged on 64-bit platforms. * shell: resolve trailing-slash object keys in s3.clean.uploads Completion stores a key ending in / inside the directory it names (<bucket>/dir/dir), but FullPath+DirAndName on the normalized key looked one level too high. Deriving dir and name with path.Dir and path.Base mirrors getEntryNameAndDir so the completed-upload check finds the entry instead of purging its chunks. * s3: heal a suspended completion hidden behind a delete marker Removing .uploads/<id> inside the commit transaction means a failed finalizeSuspendedNullWrite leaves nothing to retry against: the object entry is committed but the marker still makes the key read as deleted, and a retried CompleteMultipartUpload can only report NoSuchUpload. When the upload directory is gone, check the regular path for an entry carrying the upload id and re-run the marker finalize, so the retry both succeeds and repairs the key. Only suspended buckets can hold this state; anything newer owns the key. * s3: report store errors when resuming a committed multipart object	2026-09-17 21:09:21 -07:00