2 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandDevin	74f196ceab	s3: route regular copy source reads to the write owner (#12173) ... * s3: read regular copy sources from their write owner * filer: report mutation sentinel codes in ObjectTransaction responses A failed mutation only surfaced resp.Error text, so routed writers could not distinguish an existing-directory conflict from a generic failure and could not apply the prefix-object retry the plain CreateEntry path has. * s3api: retry a routed mkfile as a prefix object over a directory mkFile retries ErrExistingIsDirectory by marking the entry a prefix object; the routed path returned the conflict instead, so a routed write to a key that is also a prefix failed where the lock path succeeds. The filer now reports the code on the transaction response, making the retry recognizable. * s3api: commit copy destinations on the key's write owner An owner-routed source read still left the destination on the gateway's default filer, so a second copy resolved the first copy's destination on the owner, found nothing, and returned NoSuchKey. Write unversioned and suspended destinations through routedMkFile, and write the version file on the owner the finalize recomputes on. * s3api: cache remote objects on the filer that served the entry An owner-routed lookup can return a remote-only entry that has not replicated to the gateway's default filer; the cache RPC then cannot find it and the copy or read degrades to a 503. Ask the entry's write owner to cache it first, falling back to the default filer. * s3: read versioned object state from the key's write owner CopyObject destinations now commit version files and the .versions pointer on the owner's object-scope ring key, so versioned reads through the default filer could miss them until replication. Generalize the routed read to dir/name lookups and paginated lists scoped to the object's owner, and point the versioned read paths at it: getSpecificObjectVersion, doGetLatestObjectVersion, the .versions quick-checks in Get/Head/attributes, the self-heal rescans, tagging, delete conditionals, directory-marker history checks, rename destination, multipart completion checks, and the version-id format probe. Reads fall back to the default filer when no ring owner is resolvable, preserving single-filer deployments. * test(s3api): fake RECOMPUTE_LATEST/DELETE and cover versioned copy reads The ambiguous-PUT fake filer now creates parent directories on PUT (mirroring InsertEntry), applies DELETE mutations, and re-derives the .versions pointer for RECOMPUTE_LATEST the way applyRecomputeLatest does, so owner-vs-replica tests can exercise versioned writes. TestVersionedCopyDestinationReadsOnOwner copies into a versioned bucket whose writes commit on the owner while the gateway's default filer is a replica with no replication, then asserts reads by versionId, latest-version resolution, and a copy chained off the new version all resolve on the owner. * test(s3api): gofmt the versioned-copy coverage * s3: only swap in a cache fill of the resolved remote generation A cache response can come back from a filer holding an older generation of the key. Replacing the authorized source or swapping its chunks under a size and range computed for the resolved entry serves the wrong object or truncates the response. Match RemoteEntry identity and size before using a cached entry, in both the copy and streaming paths. * filer: keep directory attributes when an object lands as prefix object The object-onto-directory retry replaced the existing directory's mode, owner, and extended attributes with the object's. Preserve the directory identity; merge directory-held extended keys only when the old entry never held an object, so a prior generation's object metadata is not carried forward. * s3: keep marker lock checks on the filer that replaces the marker The object-lock precondition read the key's write owner while mkdir still replaces the marker on the default filer, so a locked marker not yet replicated to the owner could be overwritten. Read the entry where the replacement lands, in both PutObjectHandler and zip extraction. Also hash a trailing-slash marker key to the same route key as the object it names, so marker and object share one owner and one write lock. * s3: route tag writes to the owner tag reads resolve GetObjectTagging reads the key's write owner, but PutObjectTagging and DeleteObjectTagging still wrote through the default filer, so a successful change returned the owner's stale tags until replication. setTags, rmTags, and the versioned tag updates now write on the owner first, falling back to the default filer when no owner resolves. * s3: keep rename lookups on the filer that performs the move AtomicRenameEntry and the retry token it carries both execute on the default filer, but the source and destination reads went to the write owner — a completed rename's token was invisible to its own retry, and a source that only exists on the default filer looked absent. Read both ends on the default filer. * s3: send the .versions pointer repair with the object's route key The heal reads the .versions entry on the object's owner, but the conditional UpdateEntry is routed by the .versions path's own ring key, which can hash to a different filer — leaving the pointer the routed reads consult unchanged. Ship the CAS repair as a PATCH_EXTENDED ObjectTransaction on the object's route key instead. * filer: preserve only directory identity on a prefix-object write Merging the old directory's extended attributes could carry a marker's encryption, version, or etag fields onto the new plaintext object, since a directory's own keys cannot be told apart from the object it held. Keep only mode, uid, gid, user name and group names. * s3: also require matching size when a cache fill replaces the source The remote-generation check covers identity; size equality is the cheap guard that keeps the streamed or copied range consistent with what was resolved. * s3: lock the object path for a routed .versions pointer repair conditionalUpdateEntryRouted held the lock on the .versions path while version writes and deletes serialize on the object path, so a repair could pass IF_ENTRY_EQUAL and then apply over a pointer a writer had advanced in between. Lock the object path like the writers and keep the condition on the .versions entry via condition_key. * s3: do not replay an owner-side tag mutation on another filer withWriteOwnerFilerClient re-ran fn on the default filer after any owner error, so an authoritative not-found could update a stale replica entry and still report success, and a post-apply transport error could apply the mutation twice. The failover walk now ends on any error fn returns; only a channel that never came up moves to another filer. * filer: keep a directory's xattrs when a prefix object replaces it The prefix-object branch preserves the directory's identity but dropped its Extended map wholesale; FUSE xattrs live there under the xattr- prefix and belong to the directory, not to the object the marker held. * s3: delete null and specific versions on the filer that serves them On the default filer an absent entry deletes as success, so a version that only reached the write owner survived a reported success while owner-routed reads kept serving it. The suspended-versioning and lifecycle deletes now go through routedDeleteNullVersion and routedDeleteSpecificVersion when an owner resolves; an unsettled attempt keeps the lock path. * s3: route a lifecycle null-version delete to the object entry The "null" version is the regular object entry, not a .versions file, so it goes through routedDeleteNullVersion; only a real version id takes the .versions-file delete. * s3: repoint .versions only while it still names the deleted version routedDeleteSpecificVersion recomputed the latest pointer on every version delete, so removing a noncurrent version re-derived the pointer to the newest remaining file — resurrecting an older version over a cleared pointer (a suspended bucket's null write keeps it clear while the null object stays current) or over a pointer that moved on. The pointer is now read on the owner and repointed under an IF_ENTRY_EQUAL guard only while it still names the deleted version; a contested pointer is left for the self-heal path rather than overwritten by a stale scan. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: let lifecycle expiry keep its metadata-only deletes on routed paths The routed lifecycle deletes hardcoded IsDeleteData, so expiring an entry with a volume TTL queued per-chunk deletions instead of leaving the data for the volume to reclaim — extra work on the owner for deletes recordMetadataOnlyIf still counts as metadata-only. Pass the caller's cleanup choice through routedDeleteNullVersion and routedDeleteSpecificVersion; the interactive DELETE path keeps deleting data. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: let RECOMPUTE_LATEST skip a pointer that moved on A version delete repoints .versions only while the pointer still names the deleted version, but the gateway cannot carry the pointer's current state into the transaction's condition — the condition slot is already spent on the version's WORM guards. only_if_current_name moves the check under the transaction lock: the filer compares the pointer's name_to_key value before scanning and skips the recompute when it moved (or was cleared by a suspended bucket's null write), so a noncurrent delete cannot resurrect an older version. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: repoint inside the version's locked delete repointLatestBeforeVersionDelete ran in its own transaction ahead of the delete, so a version delete denied by retention or legal hold still moved the latest pointer — GET and HEAD went on serving the older version it rolled to. The repoint and the delete now ride one transaction behind the version's WORM guards, atomically gated by the new only_if_current_name recompute guard instead of a stale-read IF_ENTRY_EQUAL. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 14:35:39 -07:00
Chris Lu	268424a5e1	s3: extract zip members on PutObject (#11786) ... * filer: add zip archive extraction helpers SpoolZipReader makes an upload body seekable for archive/zip, SanitizeZipMemberName rejects zip-slip and non-regular members, and CheckZipLimits caps member count and total uncompressed size. For seaweedfs/seaweedfs#11722 * filer: extract zip members on ?extract=zip upload POST/PUT to a directory path with ?extract=zip (or X-Seaweedfs-Extract: zip) writes each archive member as its own entry, so a bundle of small files lands in one request instead of one roundtrip each. Members keep their mtime, mode and mime-by-extension; directories are created; unsafe names are skipped and reported. extract.keep=true also stores the archive itself. For seaweedfs/seaweedfs#11722 * filer: bound zip spool memory by actual bytes A chunked upload has no Content-Length, so the old size hint let an unbounded body go through io.ReadAll. Read up to the memory threshold plus one byte first, then spill the prefix and the rest to a temp file, capped at ZipExtractMaxBytes total. * filer: guard zip member size total against overflow Declared member sizes summing past math.MaxUint64 wrapped the running total and slipped past the 1 TiB cap. * server: only take the inline fast path at a short first read uploadReaderToChunks read at most chunkSize bytes, then treated any read below SaveToFilerLimit as the whole file. When SaveToFilerLimit exceeded chunkSize, longer uploads were stored truncated. * server: resolve storage options per extracted zip member A member can land under a storage rule more specific than the request path. Re-running detectStorageOption0 per destination applies that rule's collection, replication, TTL, filename limit, and read-only guard to files and directories alike, while explicit query overrides still win. * server: accept raw request body for POST extract uploads Every POST was forced through MultipartReader, so a raw zip body always got a 400. Only parse multipart when the Content-Type is actually multipart. * server: extract zip directories before files and stop on cancel Processing directory members first keeps their stored mtime/mode from being overwritten by parents auto-created for an earlier file member. Checking ctx.Err() between members keeps a disconnected client from driving thousands more writes. * server: assert stored mtime and mode for extracted members * s3: add zip extraction header constants * s3: extract zip members on PutObject with extract header x-amz-meta-snowball-auto-extract or x-seaweedfs-extract: zip on a PutObject fans out the archive to one object per member under the object's directory, reusing putToFiler so chunking, collections, storage rules, versioning, and bucket-default encryption apply to each member. The archive itself is not stored. Explicit SSE request headers are rejected since one encryption config cannot represent independently created member objects. * s3: bound member fan-out by request cancellation Stop starting new member writes once the request context is done, matching the filer extract loop. * filer: bound compressed zip spool and reject non-local member names The 1 TiB uncompressed limit was also capping the compressed bytes spooled to the host's temp directory, so a handful of concurrent extractions could fill the filer's disk before any ZIP validation ran. SpoolZipReader now takes a separate compressed-size limit (the filer passes its TUS max upload size) with a 5 GiB default. SanitizeZipMemberName gains a filepath.IsLocal check: the segment loop already rejects "..", but CodeQL only treats member names as sanitized through a recognized guard, and the extra check also covers reserved Windows names. * server: honor destination rules for zip members and clamp maxMB An x-seaweedfs-destination header picks the storage rules for a logical destination while members still land under the request path, matching PostHandler. Directory members now also enforce the actual path's read-only rule instead of relying on option resolution to fail. maxMB up to 2048 overflowed the int32 chunk size, so an oversized query value made extraction (and ordinary autoChunk uploads) store members empty yet report success. The parsing is shared and clamped at 2047 MB. * s3: scope zip member writes through the per-key checks PutObject applies The extraction loop bypassed most of what a standalone PutObject would face: members were written without IAM authorization for their key, directory members replaced markers without entry lock checks, and lock headers persisted retention without the s3:PutObjectRetention or s3:PutObjectLegalHold permissions. authorizeMember now runs IAM, table-bucket path rules, bucket policy, lock-header validation, and non-versioned lock enforcement per member key, and directory members take the same guarded mkdir path as an explicit dir marker. The archive's own Content-Md5 and x-amz-checksum-* values are verified while the body spools instead of being silently dropped, members stop inheriting the archive's Content-Encoding, and non-versioned members resolve their lifecycle TTL like normal uploads. * s3: fail malformed lock headers before spooling the archive Per-member re-validation turned a bad object-lock header into one error entry per member inside a 200 response. Validate once up front like PutObjectHandler does, and keep only the dedicated retention/legal-hold permission checks scoped to each member key. * filer: return 413 for oversized archives and honor saveInside in extract Map filer.ErrZipTooLarge to 413 instead of 500 for client-side archive size violations. Extracted members written under saveInside now store as inline entry content instead of unreachable volume chunks. * s3: write extracted members under their own request context Point member writes at the member key via mux.SetURLVars so SSE-KMS builds the encryption context from the member ARN, not the archive's. Map the spool limit error to ErrEntityTooLarge.	2026-10-16 21:07:05 -07:00