mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-20 13:30:46 +02:00
4160b92864a44cef07b0e99b63a07af4408df511
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Lu
|
15520f601f |
s3: commit multipart upload and remove .uploads atomically; purge completed uploads metadata-only (#11375)
* s3: commit versioned multipart upload in one transaction CompleteMultipartUpload wrote the version file, flipped the .versions pointer, then removed .uploads/<id> metadata-only as a best-effort post-commit step. A filer error or gateway crash in that window left the upload directory referencing the same chunks as the published object, and the next s3.clean.uploads run purged it with data -- corrupting a committed object. Put the version file, remove the upload directory metadata-only (its chunks are the object's chunks), and recompute the latest pointer in one ObjectTransaction under the object's per-path lock on the owner filer. The mutation order keeps every partial state safe: the chunks stay referenced at all times, and a published object never coexists with the upload directory the cleaner would purge. Unused part entries are freed before the transaction, since the metadata-only directory delete would otherwise leak their chunks. * s3: remove upload directory inside the multipart object PUT The same committed-object/stranded-upload window existed on the suspended and non-versioned paths: writeMultipartObject committed the object, then a best-effort rm dropped .uploads/<id>. Ride the metadata-only removal on the routed PUT itself so the two land in one transaction; the unrouted mkFile fallback keeps post-commit cleanup. * shell: purge completed uploads metadata-only in s3.clean.uploads A leftover .uploads/<id> can outlive a committed object when the completion's metadata-only delete fails or the gateway dies in between; its part entries then share chunks with the live object, and a recursive purge frees them out from under it. Before purging a stale upload, check whether it completed: the object entry or any version file under <key>.versions carrying the upload id. If so, delete with skipChunkDeletion. If the lookup fails, skip the upload for this run rather than risk live chunks. * s3: abort multipart completion when unused part cleanup fails Deleting the upload directory metadata-only erases the only metadata pointing at part entries whose deletion failed, orphaning their chunks. Propagate the error so the completion fails while the upload directory still exists and the request remains retriable. * s3: require the upload directory to exist at multipart commit A delete that does not take the object lock (abort, lifecycle, s3.clean.uploads) can remove .uploads/<id> and its chunks between the prepare step and the commit transaction. The commit now carries an IF_EXISTS precondition on the upload directory so the race fails the request with NoSuchUpload instead of publishing an object over freed chunks. * s3: keep the version file when the upload directory is gone The finalize transaction has no rollback, so a failure at the latest-pointer recompute leaves the version written and .uploads/<id> removed. Deleting the version then destroys the only remaining record of the upload, making a retried CompleteMultipartUpload return NoSuchUpload while the version's chunks leak. Roll back only while the upload directory survives; otherwise keep the version, which a retry resolves through SeaweedFSUploadId and the version reconciler promotes. * s3: keep manifests when a routed object write partially commits For non-versioned and suspended completions the object PUT precedes the upload-directory DELETE, so an error can mean the object entry exists while the response reports failure. Freeing this attempt's manifest chunks then destroys the committed object. Keep them when the object entry survived, and after a failed null-marker finalize which always follows a committed write. * s3: skip the keep-version path on precondition failure A rejected precondition means no mutation ran, so there is no version file to preserve and this attempt's manifests are orphans the error cleanup should free. * s3: keep manifests when the object-existence check itself fails A transient lookup error previously read as absent, letting the error cleanup free manifest chunks a committed object still references. * s3: keep the upload directory when post-commit part cleanup fails Removing it metadata-only after a failed entry delete erases the only reference to the leftover chunks. Leave the directory so the entries keep their chunk references for s3.clean.uploads or manual recovery. * pb: fix filer list entry counting on 32-bit int(limit) wraps to -1 on 386 when limit is math.MaxUint32, so the beyond-limit check discarded every streamed entry. Compare in uint64 instead; the semantics are unchanged on 64-bit platforms. * shell: resolve trailing-slash object keys in s3.clean.uploads Completion stores a key ending in / inside the directory it names (<bucket>/dir/dir), but FullPath+DirAndName on the normalized key looked one level too high. Deriving dir and name with path.Dir and path.Base mirrors getEntryNameAndDir so the completed-upload check finds the entry instead of purging its chunks. * s3: heal a suspended completion hidden behind a delete marker Removing .uploads/<id> inside the commit transaction means a failed finalizeSuspendedNullWrite leaves nothing to retry against: the object entry is committed but the marker still makes the key read as deleted, and a retried CompleteMultipartUpload can only report NoSuchUpload. When the upload directory is gone, check the regular path for an entry carrying the upload id and re-run the marker finalize, so the retry both succeeds and repairs the key. Only suspended buckets can hold this state; anything newer owns the key. * s3: report store errors when resuming a committed multipart object |
||
|
Chris Lu
|
ba5b14b457 |
master, filer, s3api: bound the collection deletes that strand a caller (#11026)
* master: bound each volume server DeleteCollection, and finish the fan-out A collection delete fanned out to every volume server holding it with context.Background(), so a server that accepted the connection and then went quiet held the whole delete open with nothing to end it. Each RPC is bounded now, on the same budget allocateVolumeTimeout gives the other master-to-volume-server admin RPC. The volume server runs the delete to completion regardless of the request context, so giving up costs the confirmation and not the deletion. The walk itself is the caller's, not a per-server one: - It outlives the caller. A cancelled request must not abandon a destructive fan-out part-done, with volumes left behind and no request still running to come back for them. - It no longer stops at the first server that refuses, which left the collection on every server after it in the list. The first failure is still what is reported, and the collection stays in the topology so a later delete comes back for the rest. - It sends one RPC per server rather than one per replica. ListVolumeServers reports a node once for every replica it holds, while DeleteCollection removes the whole collection from the server it reaches, so a collection with thousands of volumes repeated the same whole-collection delete thousands of times over. Both passes run too. Returning after a failed normal pass left the collection's EC shards in place with nothing left to retry them. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * master: delete the EC shards behind /col/delete too The HTTP handler carried its own copy of the volume-server walk and only ever ran the normal pass, so a collection deleted through it kept its EC shards. It shares the gRPC path now, which also gets it the bounded RPCs and the one-per-server fan-out. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * filer: bound the collection delete a bucket delete leaves behind Deleting a bucket entry deletes its collection afterwards, deliberately detached from the request so a client that hangs up cannot strand the bucket's volumes. Detached meant unbounded, though: with the master down or mid-election the wait for a leader has nothing to end it, so the handler parks, and the client retrying behind it parks another. It keeps outliving the request and now carries a deadline of its own. The budget bounds the wait, not the work: the master keeps deleting on its own fan-out once asked, so giving up costs the confirmation. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * s3api: bound the collection RPCs a bucket creation and deletion issue Neither carried a deadline, so a transient failure anywhere down the chain held the S3 request open until the client gave up on it. Both budgets are taken outside the filer failover walk, so one budget covers the whole walk rather than granting each filer a fresh one. The walk itself stops when that budget is spent, and stops without blaming anyone: the caller's own expiry is not evidence against the filer that was answering, and the next filer has no time left to answer in either. Recorded as a filer failure, a slow master upstream would flag every filer in the walk, and the three failures that open the circuit take unrelated object reads down with them. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * s3api: a failed collection listing no longer fails a bucket creation PutBucket lists collections to notice a leftover one it is about to reuse. The result feeds a warning and nothing else -- s3a.exists is what decides whether the bucket already exists -- yet a transient failure of that listing returned 500 and refused the creation. It is advisory now, so a failure is logged and the creation continues, exactly as it does when the listing returns false. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP |
||
|
|
df93d01c06 |
admin: add bucket lifecycle rule editing (#10860)
* admin: add bucket lifecycle rule editing * address greptile's comments * more small fixes * coderabbit's comments * more comment fixes * more fixes * more * maybe last * last ? * 14850 * 14851 * filer: stamp the content MD5 on every SaveInsideFiler write An entry's ETag falls back to Attributes.Md5, so conditional writers key IF_ETAG_MATCH off it. SaveInsideFiler carried the looked-up attributes forward without refreshing the hash, leaving it describing whatever the previous writer stored: a later conditional write matched the stale hash and overwrote content that had already changed. * s3api: give the bucket lifecycle constants and the write route key one definition each The extended-attribute keys, the XML size cap and the object-write ring key prefix were each spelled out in two places, so the admin dashboard's copies could drift from the gateway's. Move them to the packages both sides already import and alias them where the short local name reads better. * admin: patch the bucket entry's lifecycle keys instead of rewriting the entry The save read the bucket entry, edited its extended map and wrote the whole entry back, guarded by IF_UNMODIFIED_SINCE. Nothing that writes a bucket entry advances its mtime - not the S3 gateway's patchBucketEntry, not SetBucketOwner, not SetBucketQuota - so the guard never fired and the stale snapshot reverted whatever else had changed since the lookup. Send the PATCH_EXTENDED mutation the S3 gateway already uses for these keys: the filer re-reads and merges under the bucket path lock, so only the two lifecycle keys move. That removes the reason for the mtime snapshot, the verification retry loop and the compensating restore of the cleared day-TTL rules, which the migration now logs instead. * s3api: run the delete-lifecycle day-TTL migration through the shared helper DeleteBucketLifecycleHandler kept its own copy of the read-strip-write sequence the put handler now shares, including a missing return that let a ToText failure persist a truncated filer.conf and write a second response. It also wrote the whole file back unconditionally, reverting any concurrent edit; the shared helper writes conditionally. * admin: answer 404 when a lifecycle request names a bucket that does not exist Every SetBucketLifecycle failure came back as 500, including the lookup miss for an unknown bucket, so a client or monitor read a caller error as a server fault and retried it. * s3api: emit lifecycle XML a client would recognize Two changes to what MarshalCanonical writes, both visible through GetBucketLifecycleConfiguration, which replays the stored bytes verbatim: stamp the S3 namespace on the root, and put a size range under <And>. A <Filter> carries one predicate, so two size bounds side by side is a shape AWS does not document. Parsing still accepts either. * admin: fix the lifecycle editor's handling of stored status, deletes and empty saves Four things the editor got wrong: A stored <Status> the S3 API never validated, say 'enabled', left both radio buttons unchecked, so reading the form threw on a null querySelector result and Save did nothing. Collapse anything but an exact 'Enabled' to 'Disabled', which is what the engine already does with it. Deleting a rule re-rendered an open edit form from the snapshot taken when editing began, discarding what had been typed; every other transition folds the form in first. The Transition warning only matched a bare <Transition>, missing the form with attributes, self-closed or namespace-prefixed. Saving an emptied rule list clears the configuration through a path with no prompt, next to a Delete-all-rules button that asks. Also collapses the three divergent copies of formatBytes on this page to one. * filer: stop the day-TTL migration from deleting an operator's path rule The migration removed every rule under the bucket's path that carried a day TTL in the bucket's collection. The add path it is retiring used AddLocationConf, which merged its TTL onto whatever already sat at the prefix, so a rule can hold operator settings the lifecycle path never wrote - a disk type, WORM retention, a read-only flag, a placement pin. Deleting the whole rule to retire its TTL took those with it, leaving objects under that prefix on defaults nobody asked for. Delete only rules shaped like ones the add path created from scratch; anything else keeps its settings and loses just the TTL. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
d35c4b3d2d |
s3: fail over routed object writes when the owner filer is unreachable (#10251)
* s3: fail over routed object writes when the owner filer is unreachable A routed object write (multipart completion, PUT, delete, versioned finalize, metadata replace) dialed the ring-selected owner filer directly with no failover. After a filer restarts onto a new address the lock ring can still name the old one, so every routed write hangs on the dead address until the gateway is restarted; CompleteMultipartUpload in particular exceeds client timeouts. Route the transaction through withFilerClientFailover, skipping an owner that recently failed, so a live filer forwards it to the real owner by route_key. Mirrors the read path's getObjectEntryRoutedByKey. * s3: fail over bucket-config writes when the owner filer is unreachable patchBucketEntry dialed the bucket's ring owner directly, so a restarted filer's stale ring address hung every bucket-config write (versioning, lifecycle, object lock, ownership, ACL, policy, CORS) - the same failure as routed object writes. Route it through objectTxnOnFiler so it skips an unreachable owner and a live filer forwards by route_key. |
||
|
Chris Lu
|
6b06fe5ec4 |
s3: commit a versioned PutObject and its latest pointer in one transaction (#9756)
* s3: commit a versioned PutObject and its latest pointer in one transaction A versioned PutObject wrote the version file and flipped the .versions latest pointer in two separate routed transactions. Fold the RECOMPUTE_LATEST into the version file's PUT so both commit atomically under the object's per-path lock: the recompute, applied after the PUT in the same transaction, scans the directory and sees the new version. A crash can no longer leave the version present with a stale pointer. putToFiler now takes a putFinalize describing the finalize step — routed mutations folded into the PUT, or an afterCreate run under the object write lock off the ring. Suspended-versioning keeps its afterCreate-only form; multipart, copy, and delete-marker finalizes are unchanged. * s3: trim verbose finalize comments |
||
|
Chris Lu
|
2a4923e7e8 |
ObjectTransaction: filer-side forwarding via route_key (#9659)
A non-owner filer forwards the whole transaction to the ring owner of route_key, so the owner's per-path lock stays the single serialization point even when the caller's ring view is stale. is_moved bounds forwarding to one hop. The gateway stamps route_key on every routed builder via the shared objectRouteKey helper. Completes taking S3 object mutations off the distributed lock. |
||
|
Chris Lu
|
1f0c366583 |
s3: route metadata-only self-copy off the distributed lock (#9638)
A non-versioned metadata-only self-copy (CopyObject with source == destination and the REPLACE directive) is a read-modify-write of one entry, which is why it held the distributed lock. It now routes to the owner as a serialized PATCH_EXTENDED: the owner merges the new managed metadata (set the replacements, delete the dropped keys) onto a fresh read of the entry under its per-path lock, so a concurrent change to non-managed keys (legal hold, retention, version id) is preserved instead of clobbered, and bumps mtime. PATCH_EXTENDED gains touch_mtime for the mtime bump. Versioned and suspended self-copies create a new version (already routed via the copy finalize) and the no-owner bootstrap keep the lock. |
||
|
Chris Lu
|
eeda7181aa |
s3: route multipart-upload completion off the distributed lock (#9632)
completeMultipartUpload routes its writes to the object's owner filer when an owner is known, off the distributed lock. Idempotent replay is handled gateway-side in prepareMultipartCompletionState (it returns the existing result when the object already carries this UploadId), so the lock is not needed to dedupe retries; with no owner yet, the lock remains as the bootstrap path. Versioned completion flips the .versions pointer via routedVersionedFinalize (RECOMPUTE_LATEST). Non-versioned and suspended completion write the object via routedMkFile (a routed PUT) so the write serializes with concurrent writes to the same key on the owner's per-path lock. The version file itself is a unique path and stays a plain mkFile. |
||
|
Chris Lu
|
5bac8b9281 |
s3: route object-lock object writes off the distributed lock (#9635)
routableWriteOwner no longer excludes object-lock buckets, so a versioned PUT (which creates a new version, never overwriting a locked one) and a non-versioned overwrite (WORM-checked gateway-side before dispatch) route to the owner filer like any other write. routedObjectOwner still excludes object-lock: an unversioned object-lock delete enforces WORM under the lock, so it stays there rather than routing past the check. Version-specific deletes likewise stay on the lock — routing them needs the WORM check (on the version entry) and the latest-pointer recompute (on the object) under one transaction, which the current single condition target cannot express. |
||
|
Chris Lu
|
db954b5503 |
s3: route versioned PutObject finalize off the DLM (#9631)
s3: route versioned PutObject finalize off the distributed lock A versioned write's finalize (flip the .versions pointer to the newest version, demote the prior latest) now runs as a single RECOMPUTE_LATEST ObjectTransaction on the object's owner filer, under its per-path lock, instead of the unserialized updateLatestVersionInDirectory. The version file is written first; the owner re-derives the pointer by scanning the directory. RECOMPUTE_LATEST gains size_to_key / mtime_to_key to cache the chosen version's size and mtime on the pointer, and demote_key / demote_value to stamp the displaced prior latest (NoncurrentSinceNs for lifecycle) when the pointer moves. Falls back to updateLatestVersionInDirectory when no owner is known yet. |
||
|
Chris Lu
|
f9bc6adf98 |
s3: route single-entry object writes to the owner filer, off the DLM (#9629)
s3: route non-versioned object PUT and DELETE off the distributed lock A non-versioned, non-object-lock object write now goes straight to the key's owner filer as a single-mutation ObjectTransaction, which serializes it with the owner's per-path lock and evaluates the precondition, instead of taking a cluster-wide lock. PUT and DELETE use the object's full path as the lock key, so a concurrent create and delete of the same key serialize against each other. The fast path is taken only when the precondition reduces to clauses the filer can evaluate (existence and a single strong-ETag match); time-based conditions, ETag lists, weak ETags, post-create hooks, and an unknown owner fall back to the lock. A routed mutation error other than a failed precondition also falls back, so the lock path stays the authority for the cases it alone covers. PrimaryForKey returns "" until the ring view arrives, keeping writes on the lock until routing is known. |