mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-01 18:00:46 +01:00
ab355b43b96e49eb8a153e3c01aeee8c4c1956f3
1627
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
ab355b43b9 |
s3tables: keep the vector-namespace fence on a declare of an existing index (#12293)
s3tables: keep the fence on a declare of an existing index Deciding the existing-index exception before the handler ran left a window: an index deleted between that lookup and the handler's own check would be recreated, unfenced, by a caller without s3vectors:CreateIndex. Drop the up-front exception, so every CreateTable runs fenced and a fresh create is refused at its first write. The engine's declare still passes because the idempotent path writes nothing; for a build whose idempotent path re-applies state already in place, the fence answers an UpdateEntry that leaves the entry exactly as stored without writing. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
e2826c48bd |
s3tables: make the vector namespace of a vector bucket read-only (#12292)
* s3tables: make the vector namespace of a vector bucket read-only S3 Vectors authorizes index operations against the vector bucket policy, but an index is also a catalog table in the bucket's "vector" namespace, and the S3 Tables API only checks s3tables grants. A caller the vector policy denies DeleteIndex could still delete the index, and its data, with DeleteTable. Untrusted writes that name the vector namespace of a LANCE bucket now run behind a fence that passes filer reads and refuses filer writes. The operation's own checks still answer everything short of a write, so a caller without the grant gets its usual denial and learns nothing about the bucket, and a CreateTable that finds the index already there, which is what the vector engine's declare does, still succeeds. A refused write answers 403 AccessDenied. Trusted callers (the gateway, the shell) are unaffected, and so is a namespace named vector in an Iceberg bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: refuse register and deregister in a vector bucket's vector namespace Every catalog write the Lance namespace makes goes through the S3 Tables handler and its vector namespace fence, but register over an existing table at the same location and deregister only add or remove a marker file in the dataset directory. Check the vector namespace before those writes, after the table read that already gates them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: refuse deferred creates in a vector bucket's vector namespace Manager.AuthorizeCreateTable gates the Iceberg create paths that write metadata into the table bucket before the catalog registers the table. Those writes never pass the handler's fence, so the gate applies the same read-only rule once the caller's own create permission is confirmed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: let a declare of an existing index past the vector namespace fence The vector engine declares every index the gateway has just created, and that CreateTable only reaches its idempotent path. Upstream that path writes nothing, so the fence never trips, but a build whose idempotent path repairs table state would refuse the engine. Decide the exception up front instead: a CreateTable naming a finished index in the vector namespace runs unfenced, while a new name or an unfinished create stays fenced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
9a09506be6 |
s3vectors: bound ListVectors parameters and lowercase distanceMetric (#12290)
* s3vectors: bound ListVectors maxResults and segment parameters segmentCount had no upper bound, so segmentCount=1000000, segmentIndex=999999 made the segment emulation scan the whole index to return nothing. Apply the AWS ListVectors windows before authorization and any engine call: maxResults 1..1000, segmentCount 1..16, segmentIndex in [0, segmentCount), and segmentCount and segmentIndex given together. The fields become pointers so an explicit 0 is rejected while an omitted one still takes the default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: store and return distanceMetric in lowercase CreateIndex stored distanceMetric exactly as sent, so an index created with "COSINE" came back as "COSINE" from GetIndex and QueryVectors. AWS returns the lowercase enum values. Lowercase the metric when the index is created, and again when the record is read so indexes already stored in mixed case render the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
21fc46692c |
s3vectors: authorize CreateVectorBucket as a vector action, close DeleteIndex race, fold policy service case (#12287)
* s3vectors: authorize CreateVectorBucket as an s3vectors action CreateVectorBucket went through the catalog's CreateTableBucket check, so an identity granted s3vectors:* was denied the create while s3tables:* was allowed it. Check s3vectors:CreateVectorBucket in the gateway, the same way the catalog checks a bucket create, and create the record on the internal manager under the caller's name so the caller stays the owner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: delete only the index record DeleteIndex checked DeleteIndex read the record, checked it is a Lance index, then deleted by name. A table that replaced the index in between was deleted too. Send the checked record's version token, so a replaced record fails the delete with a 409 ConflictException. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: reject s3vectors rules in any letter case validatePolicyDocument kept s3vectors actions and resources out of the shared policy slot with case-sensitive checks, so "S3Vectors:GetIndex" was stored as a rule that never takes effect. IAM service prefixes are case-insensitive; fold case in both checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3a8d756bdd |
s3: drop retired volume servers from the gateway location cache (#12281)
* wdclient: publish applied volume-location deletes to registered listeners The master already tells each connected client which volumes a server stopped hosting; only the MasterClient's own vidMap applied it. Expose the same filtered deletions (moves between a server's disks excluded) through OnVolumeLocationDelete so clients keeping a second location cache can drop the retired server too. * wdclient: drop retired volume servers from the FilerClient location cache The S3 gateway's FilerClient keeps its own vidMap, filled by filer lookups, and only forgot a retired volume server after a read against it failed (a ~15s header-timeout stall when the host still resolves). When built with a MasterClient, subscribe to its applied deletions so the cache forgets the server as soon as the master reports it gone. * wdclient: drop retired EC servers from the regular location map too Filer lookups cache EC locations alongside regular ones in vid2Locations, so an EC removal has to clear the retired server there as well as in the EC map for the delete to reach what reads actually use. * wdclient: tombstone pushed deletions against in-flight provider fills A filer lookup already running when a delete lands can return the retired location afterwards and re-cache it. Keep a short tombstone on each deleted (vid, url) pair so fills neither cache nor return it while the lookup source is still settling. * wdclient: clear the FilerClient location cache on master reconnect Removals the master pushed while the gateway link was down never arrive as deletes, so on reconnect the lookup-fed cache is dropped outright rather than kept generation-wise like the stream-fed one, and relearned through filer lookups as before. * wdclient: invoke push hooks outside their registration locks A callback that registers or replaces a hook would wait on the read lock the invocation already holds and stall the master stream. Copy the hook under the lock and call it after the lock is released. * wdclient: report pushed adds so confirmed re-adds clear the suppression A server coming back at the same URL inside the tombstone window stayed hidden from provider fills. updateVidMap now carries the additions it applied through the same hook (renamed OnVolumeLocationChange), and the registered cache untombstones them without eagerly caching them. * wdclient: amortize tombstone expiry over a delete batch A server leaving takes every volume it hosted with it, and scanning all recentDeletes per removed vid squared the batch. Prune lazily on reads and at most once per half TTL on writes. * wdclient: let every location cache subscribe to pushed volume changes The setters were single-slot, so a second FilerClient built on the same MasterClient would have silently replaced the first one's subscription. Registration now returns an unhook func, delivered to all listeners, and FilerClient.Close releases its own. * wdclient: hold the deletion fence until the master re-confirms a location * wdclient: check the deletion fence and the cache add under one lock * wdclient: drop provider answers that crossed a cache reset * wdclient: bound the deletion fence by evicting its oldest mark * wdclient: refresh a repeated removal instead of evicting another mark * wdclient: evict the oldest deletion-fence mark in constant time * wdclient: fence a server only once it serves the volume in no form ec.encode registers the shards before the regular copy is deleted from the same server, and ec.decode mounts the regular copy before the shards come off it. Both arrive add-first in separate heartbeats, so the later removal tombstoned a (vid, url) pair the server still serves and dropped it from the FilerClient map; the filer's answer naming that server was then fenced out and the volume read as not found until the gateway reconnected. Tombstone and report a removal only when neither map still names the url for that volume. * wdclient: serve a provider answer that crossed a cache reset The answer is as fresh as the provider's view; only the cache it would land in has been re-told. Dropping it failed every read whose lookup was in flight during a master reconnect with 'volume not found'. Return it and leave the new view to be filled by the next lookup. * wdclient: test the moved volume whose old server keeps proxying * wdclient: stop caching inside the master lookup provider The provider wrote every answered location through addLocation, which clears the deletion fence: a lookup still naming a server a pushed delete had just retired untombstoned and re-cached it. Fallback lookups already gate caching through fillLocation, and authoritative lookups bypass the cache by contract, so the provider no longer writes the map at all. * wdclient: count only master-pushed locations when suppressing a removal hasLocation could not tell a confirmed regular copy from a lookup fill: the master answers an EC volume with its shard servers, so the regular map holds shard locations that arrived by lookup, and entries an earlier generation learned linger past a reconnect. An EC removal whose url still showed either was suppressed, leaving the retired server served. The vid map now records the generation each regular location was pushed at, drops an unconfirmed regular entry alongside the shard that fed it, and answers hasServingLocation from pushed marks and current-generation entries only. * wdclient: test removals against lookup-cached and stale locations * wdclient: fence removals only in the gateway cache The fence and the pushed-location marks ran inside every MasterClient, including the filer's, whose map is also filled by lookups and cleared by InvalidateCache. A removal reported while the server still served the volume (e.g. a decoded copy whose mark a failed read had wiped) left the pair fenced until the master pushed it again, with reads failing as volume not found meanwhile. Only the S3 gateway's FilerClient now tombstones, from the removals its push-only MasterClient reports. That client decides a removal from the current generation of both its maps after the whole message is applied, so pushedLocations goes away, and the master lookup provider caches as before. The hooks follow the existing single-slot SetOn...Fn pattern and are wired next to SetOnPeerUpdateFn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * wdclient: let a removal fence expire after a minute The master drops a broadcast when a client's channel is full, so the re-add that would lift a fence can be lost while the gateway stays connected. A mark held until that re-add then hid a live server, and a volume whose only copy it held read as not found until the next master reconnect. Bounding marks by a minute still outlasts a lookup in flight when the removal landed; expiring early only falls back to learning by a failed read, as before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicholas Oh
|
f5989a21fb | s3: honor assignment fsync in streaming copies (#12277) | ||
|
|
a94edebf00 |
fix(s3tables): keep the data under a deleted namespace (#12276)
* fix(s3tables): keep the data under a deleted namespace A rename leaves the source table directory holding data the renamed table still points at, and DeleteNamespace removed it recursively. The delete now keeps the directory, minus its catalog record, when files remain under it; CreateNamespace can take it back. * fix(s3tables): keep hidden-directory data, report name takeover as 409 - DeleteNamespace also looks for files under hidden directories, since a table location may sit under one. - CreateNamespace answers 409 when a file takes the name during the take-back of a kept directory. * s3tables: a vanished leftover holds nothing to keep A bare directory removed between the namespace listing and the walk made dirHoldsFiles answer its ListEntries NotFound, and the delete reported that miss as a 404 or 500 on a live namespace. Skip the gone directory instead: whatever it held is already gone, so it cannot keep the namespace. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: a delete's gone entry is a conflict, not a success The filer answers a precondition delete on an entry that is already gone with FailedPrecondition carrying the not-found text, which DoRemoveRequest reads as a completed delete -- DeleteNamespace answered 200 for a delete that never ran. deleteDirectoryIfMatches now calls DeleteEntry directly, keeping the FailedPrecondition-to-conflict mapping and refusing a response-level error. keepAsLeftover likewise reports a directory removed since the scan as ErrConcurrentUpdate, so the handler answers 409 rather than 500. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Jay HemnaniandChris Lu
|
ecd716e0e2 |
s3: allow POST Object uploads when auth is off (#12273)
* s3: allow POST Object uploads when auth is off With no identities configured every other handler skips authentication, but PostPolicyBucketHandler still verified the form signature, so a browser form upload failed with InvalidAccessKeyId (or a malformed credential error without signature fields). Skip the signature check when auth is off; the policy conditions are still enforced. * s3: update AuthPostPolicy comment for the auth-off path --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
05584c541c |
s3: take list markers as raw keys under encoding-type=url (#12275)
* s3: take ListObjects marker/start-after as raw keys Under encoding-type=url AWS url-encodes the response fields but treats request markers as raw keys: SDKs decode Marker/NextMarker/StartAfter and send the decoded key back. Unescaping the marker server-side read a literal "%XX" key as the key it encodes — StartAfter=a%2Bb resumed after "a+b", skipping it, and a paginator whose marker decoded to an earlier key got the same NextMarker twice and failed. Take the request value as-is; only the server's own base64: continuation token still decodes. The marker-echo dedup now compares contents in the response wire encoding and NextMarker in its raw form, since a raw "%XX" marker equals the escaped emission of a different key. * s3: take ListMultipartUploads key-marker as a raw key Same raw-marker rule as ListObjects: under encoding-type=url the emitted key markers are url-escaped keys, and a client decodes them before sending the marker back, so the request value is already the raw key. Keeping the token decode only for the non-url path, where the response could not carry the raw key and falls back to the base64: token form. * s3: cover ListMultipartUploads url key-marker at the handler The selectMultipartUploads tests bypass the request marker decoding, so a regression that unescapes an encoding-type=url key-marker would pass them. Drive a listing through ListMultipartUploadsHandler with a literal "a%2Bb" marker and assert it resumes after that exact key. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Jay Hemnani
|
26783fbcb3 |
s3: keep an unset CORS MaxAgeSeconds unset (#12272)
A rule without MaxAgeSeconds was stored as 0, so GetBucketCors returned MaxAgeSeconds 0 and preflights sent Access-Control-Max-Age: 0. Store -1 for unset, which corsRuleFromProto already reads back as nil. |
||
|
hsdfat
|
9df2125bb0 |
fix(s3): reject non-integer max-parts, part-number-marker and max-uploads (#12270)
getObjectResources and getBucketMultipartResources dropped the strconv error, so a value that does not parse was read as 0. ListParts with max-parts=abc then reached the filer with limit 0, which the filer reads as its default listing limit, and returned every part with <MaxParts>0</MaxParts>; part-number-marker=abc restarted the listing from part 1; and ListMultipartUploads with max-uploads=abc answered an empty, untruncated list even with uploads in progress. Return the existing InvalidArgument errors (ErrInvalidMaxParts, ErrInvalidPartNumberMarker, ErrInvalidMaxUploads) instead, the way max-keys is validated. max-parts and max-uploads are parsed as 32-bit integers to match the documented 0..2147483647 range. |
||
|
Chris LuandDevin
|
97e1758bcc |
s3: auto-create bucket on POST Object uploads to a missing bucket (#12266)
* s3: auto-create bucket on POST Object uploads to a missing bucket PutObject and CreateMultipartUpload lazily create the bucket, but a signed POST Object upload answered NoSuchBucket. Mirror the PUT path: when the versioning-state lookup reports the bucket missing, run the shared handleAutoCreateBucket gate (bucket-name check, AutoCreateBucket option, admin identity) and then re-read the versioning state. The POST handler already records the verified signer identity in the request context, so the admin check and bucket-owner attribution work unchanged. * test: assert the POST auto-create upload lands, not just non-404 The missing-bucket POST test passed whenever the handler created the bucket, even if the upload then failed. Drive the full path through the existing putToFiler fakes — assigned volume accepts the chunk, object entry and payload land — and assert the 201 success response. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: record verified post-policy signer's account id The POST policy path authenticates in the handler, after the internal account header was already scrubbed, so auto-created buckets and the uploaded objects stored no canonical owner. * test: assert POST auto-create records bucket owner, keep writes applied in denied case --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
b70c6dc6ed |
s3: let bucket-scoped Admin grants auto-create matching buckets (#12268)
* s3: let bucket-scoped Admin grants auto-create matching buckets autoCreateBucket required a global Admin identity, so an identity scoped to a prefix like "Admin:outline*" could explicitly CreateBucket but was denied when an upload auto-created the same bucket. Gate on CanDo(ACTION_ADMIN, bucket) — the same check explicit CreateBucket enforces — so a prefix-scoped key lazily creates only matching buckets. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: run auto-create bucket admin through the policy-aware path A scoped Admin identity with an attached IAM policy that denied s3:CreateBucket could still auto-create a matching bucket, because userCanAdminBucket only consulted legacy Actions. Evaluate the same VerifyActionPermission explicit CreateBucket gets, over a synthesized PUT /bucket request so the attached-policy check resolves to s3:CreateBucket on the bucket ARN regardless of the upload's shape. * s3: record verified identity as owner of auto-created buckets An admin authenticated outside the middleware (IAM integration, post policy signature) left no identity in the request context, so an auto-created bucket stored no owner and never reached the owner index. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Nguyễn Lộc Phúcanduser.email
|
7d8b72c2b4 |
fix(s3tables): answer 404 for a table name that was renamed away (#12258)
* s3tables: answer 404 for a table name that was renamed away A rename leaves the source directory in place to hold the data under it, with its catalog attributes cleared. UpdateTable, DeleteTable, the table policy calls and the table maintenance calls read the metadata attribute of that directory and reported its absence as an internal error, so they answered 500, and GetTablePolicy answered NoSuchPolicy. An absent metadata attribute now reads as a missing entry. tableMetadata reports it as both filer_pb.ErrNotFound and the attribute error, so callers that test for either keep working. The policy and maintenance handlers read the attribute through the new readTableMetadataAttr, which reports filer_pb.ErrNotFound, so each call answers NoSuchTable. * s3tables: a view record without its metadata reads absent too * s3tables: refuse a catalog write that lands on a metadata-less record --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
ad6257ebbc |
filer.backup: add -s3.flattenVersions to mirror the current S3 view of versioned buckets (#12255)
* filer.backup: add -s3FlattenVersions to mirror the current S3 view Versioned buckets are stored as <key>.versions/v_<versionId> files plus a latest-version pointer on the .versions directory, so a backup mirror held internal version files and could never be restored through the S3 API. With -s3FlattenVersions the backup maps events on .versions directories onto the flattened object key: a new latest version uploads to <key>, a latest delete marker or a cleared pointer deletes <key> when -doDeleteFiles is set, and the suspended-versioning null object resolves to the regular path. Version files themselves are never mirrored. The initial snapshot seeds only the version the pointer names. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: serialize flattened writes per object key Same-path non-barrier dir updates run concurrently in the metadata processor, so two pointer updates for one object -- or a pointer update and a regular-path write to it -- could overlap and let the older upload finish last. Lock on the flattened key: versions events lock the object key, every other event its own path, via a striped mutex. * filer.backup: apply exclusions to the flattened object key File-name and path excludes ran against the .versions entry, so an excluded object was still copied flat and its deletion still removed the destination copy. Match the rules against the flattened key on both the apply and the versions-dir-removal paths. * filer.backup: re-resolve the pointer on version-file events A tag or attribute update on the current version touches the v_* file, not the .versions pointer, so dropping every version-file event left the mirror with stale tags until the next pointer flip. Re-resolve the pointer on each version-file event; noncurrent versions still stay out. * filer.backup: tolerate vanished versions dirs in flattened snapshot The flattened seed path returned source errors before the existing missing-source policy ran, so one version deleted mid-walk could force a full re-walk. Apply the same 404/superseded skip as the generic seed branch, and cover the review fixes with tests. * filer.backup: lock both flattened keys an event touches A rename or delete can retire one flattened key while writing another, so serialize on the old- and new-side keys together. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: refresh the flattened key only for the still-current child A v_* event on a non-pointed version cannot move the flattened key, so skip the re-upload; tag and attribute updates on the current version still flow through. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: let resolved-latest writes bypass the source staleness check A filer sink skips a CreateEntry whose source path no longer holds the entry, but a flattened write lands on a key whose source path is not the one it was read from, so the check dropped every flattened upload. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: reclaim destination chunks on flattened deletes Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: route regular-path events for versioned keys through the pointer A null-object write or delete at the regular path can coexist with a .versions pointer that names a live version; the pointer decides the flattened key, so deleting the null object must not remove it. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: make the checkpoint identity cover the flattening mode Resuming a checkpoint taken in the other mode would skip the flattened re-seed and leave mirrored version files behind; a mode change now starts a fresh snapshot. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover flattened regular-path routing and per-mode checkpoints Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.backup: resolve both sides of a flattened rename * test: cover flattened renames with one versioned side * replication: let resolved-latest writes overwrite a newer destination The stale-destination skip trusted a newer destination mtime, but a resolved-latest write can legitimately install an older version over the copy of a since-deleted latest. Skip only on provably identical content (matching chunk provenance, or equal ETag with full byte coverage). * filer.backup: rescan .versions when the latest null object is gone Deleting ?versionId=null removes the regular-path object without touching the null-is-latest pointer, so the flattened key kept its stale copy. Fall back to the newest remaining version (or a flat delete), the same rescan the S3 read path applies. * replication: compare inline and remote content in the destination-holds checks A chunk-less entry keeps its bytes in Content and a remote-only entry in the object RemoteEntry points at, but the identical-content shortcuts only compared chunk lists and chunk-derived ETags. Two different inline files looked identical, so a flattened re-copy or a same-mtime rewrite kept the previous version's bytes. Compare Content bytes and remote object identity wherever the checks claim the destination already holds the reference's content. * filer.backup: paginate the null-latest .versions rescan A limit=0 listing is capped at DirListingLimit server-side, and old-format version names sort oldest-first, so a history longer than one page left the flattened copy on an older version after the null object was removed. Page through the directory like the S3 read path does. * replication: require matching writable metadata for resolved-latest skips A skip keeps the destination's stored metadata, so identical bytes were not enough: stale attributes or extended entries (e.g. a superseded version id) survived under equal content. Compare the fields a rewrite controls — attributes minus filer-assigned inode/crtime, with the store's atime/ctime zero-defaults applied — plus the extended map, in both the provenance and ETag fast paths. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: require the extended key to exist when comparing metadata * replication: ignore a stored rule TTL when the resolved write carries none * filer.backup: rename -s3FlattenVersions to -s3.flattenVersions Match the -s3.* flag naming convention used by the other s3 options. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Nguyễn Lộc Phúcandchrislusf
|
eba2cb926d |
fix(s3tables): refuse a create that a namespace delete can lose (#12239)
* s3tables: refuse a create that a namespace delete can lose DeleteNamespace checks the namespace for children and then deletes its directory recursively in a second filer call. The filer locks each path on its own, so a table created under the namespace between the two is not seen by the check and is removed by the delete, while both requests answer 200. The mirror case loses the table too: a create that passed its namespace check before the delete finished re-created the namespace directory through the filer's parent creation and recorded a table in a namespace that no longer exists. Each side now writes before it reads. DeleteNamespace claims the namespace record with a fresh token before it scans, deletes only while the record still holds that claim, and lifts its own claim when it stops. CreateTable, RegisterTable and CreateView re-read the namespace after writing their record and back out when it is claimed or gone: they remove what they wrote and answer 409 or 404 instead of reporting an entry that does not survive, or 500 when the record could not be removed and may remain. An entry the scan missed was written after the claim, so its create sees the claim; a create and a delete can both fail, but no longer both succeed. A create is also refused up front while a claim is held, but only after its authorization check, so a caller that may not create there does not learn of the delete. A create writes its record marked "create" and lifts the mark once the namespace is confirmed open. Until then GetTable, GetView, UpdateTable and a second create treat the record as absent, so no request is told about a table or view that the first create may still back out. * s3tables: keep a create's unfinished record out of reads and listings * s3tables: refuse catalog writes on a record an operation claimed * s3tables: answer 404 when a create's write path loses the namespace * s3tables: confirm the rename destination's namespace like a create * s3tables: cover unfinished-record reads, claimed-record writes, rename claim * s3tables: lift a leftover claim when a no-token delete clears the record * s3tables: answer 409 when a claimed record turns away a write UpdateView, TagResource, and UntagResource surfaced ErrConcurrentUpdate as a 500, so an expected conflict looked like a server fault to the caller. Map it to Conflict the way UpdateTable and the policy handlers already do. * s3tables: name the lost source when a rename finds it deleted * s3tables: report an entry deleted before its update as not found The filer answers an UpdateEntry on an entry removed since the lookup with codes.NotFound. mutateEntryExtended returned that error unchanged instead of filer_pb.ErrNotFound, so a rename whose source was deleted mid-claim answered 500 instead of NoSuchTable or NoSuchView. MemFiler now answers a missing entry the same way, so the rename tests for tables and views take the real filer's path. * s3tables: unpublish the destination when a rename's source clear fails * s3tables: refuse to delete a record another operation claimed * s3tables: release catalog claims on an uncanceled cleanup context * s3tables: answer 404 when a racing delete already removed the table * s3tables: keep the rename destination marked until the source clears * s3tables: guard the rename rollback against a recreated source and an unfinished destination * s3tables: drop the unfinished destination only once the source is restored * s3tables: re-resolve the purge target on the record a delete claims * s3tables: cover a delete that meets a moved metadata location * s3tables: answer 409 when a create's record changed under it * s3tables: adopt a record still marked by its unfinished create * s3tables: skip unfinished creates in the namespace empty check * s3tables: delete only the record the caller was authorized on DeleteView and DeleteTable check permissions on the record they read first, then read the name again to claim it. A record that took the name in between, such as a table or another owner's view created after a concurrent delete, was claimed and removed on the first record's authorization, and DeleteView never compared the caller's version token with it. The second read now repeats the first one's checks: a record of the other kind or an unfinished create answers 404, and a token that no longer matches answers 409. A delete without a token also compares the record's version token with the authorized one and answers 409 when they differ. * s3tables: re-check the parent after a create lifts its mark A create that passed checkParentOpen and paused before lifting its mark could still confirm while a namespace delete held the claim: the delete scans the still-marked record, skips it, and removes it under a create that already answered success. confirmParent now re-reads the same parent record after the mark clears and backs the confirmed record out when the namespace moved on. * s3tables: mark the rename destination with a claim of its own claimCatalogEntry adopted every create-marked record, including a rename's destination: a create landing after the source cleared could replace the destination before the rename published it, and with the source name reused the restore could leave the table on neither name. The destination now marks with the rename's own claim, which readers still treat as unpublished but a create must refuse, and the commit re-checks the destination namespace after the record goes live. The source keeps its rename claim until the commit lands so a namespace delete scanning the cleared record still refuses. * s3tables: compare the parent's createdAt, not its editable metadata A namespace update rewrites ExtendedKeyMetadata in place, so a create or rename whose second parent read landed across an UpdateNamespace saw a changed blob and reported the still-live namespace as gone: creates backed their record out and answered 404, renames undid their destination. createdAt survives the edit but changes on a delete+recreate, so it still identifies the record. * s3tables: retry lifting the rename's claim off the cleared source The release's only failure mode is a transient filer error — nothing else can write the cleared record — yet a single loss left the claim behind: the old name stayed refused to creates and the namespace counted the record as a child. Retry the release before warning. * s3tables: page the namespace scan before deleting it A page of skippable entries — bare directories or unfinished creates — ended the scan with hasChildren false, so a table on a later page was removed with the namespace. Keep listing with StartFromFileName until a child shows or the listing ends. --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
ee7e4a64d2 |
s3vectors: record request metrics on the dispatch path (#12241)
* s3vectors: record request metrics on the dispatch path The S3 API tracks every operation through S3RequestHistogram, S3RequestCounter, and S3InFlightRequestsGauge; the vectors dispatch rode the same listener with no instrumentation. The op name labels each series under s3vectors.<Operation>. * s3vectors: count auth rejects and keep bucketless metrics alive |
||
|
Chris LuandDevin
|
72ad975803 |
s3vectors: bound GetVectors and DeleteVectors key lists (#12238)
* s3vectors: bound GetVectors and DeleteVectors key lists Both forwarded req.Keys to the engine unbounded; AWS caps these at 500, the same window a PutVectors batch shares. * s3vectors: cap GetVectors at the AWS 100-key limit GetVectors allows at most 100 keys per request while DeleteVectors allows 500, so give vectorKeysBounds the per-call maximum instead of sharing the put-batch window. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: assert out-of-window key lists never reach the engine Record get/delete calls on the fake engine and cover both handlers rejecting empty and oversized lists with 400 ValidationException. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Ping QiuandClaude Opus 5.5
|
2a1afadcd0 |
s3api: return NoSuchUpload when aborting an upload that was already completed (#12227)
s3: AbortMultipartUpload answers NoSuchUpload for an upload that no longer exists
Complete and abort both remove .uploads/<id>. abortMultipartUpload answered
204 whenever that record was missing, so aborting an upload that had already
completed reported success while the object stayed in place, and a client
could read it as the object being discarded. AWS returns 404 NoSuchUpload
("The specified multipart upload does not exist") for that case, and
ListParts here already does.
Answer ErrNoSuchUpload when the record is not found. checkUploadStillOpen
reuses abort for cleanup after the upload is gone; NoSuchUpload is not a
failed cleanup there.
The leftover-directory path from #11385 is unchanged: a .uploads/<id> that
outlived its completed object is still removed metadata-only and answered.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Chris LuandDevin
|
d0441bbdbb |
s3vectors: validate index ARNs and expand live integration coverage (#12222)
* test: align vector bucket policy coverage with vector actions * s3vectors: validate ARN service and resource shape before resolving indexes * test: exercise signed vector bucket policies and catalog lifecycle * ci: require the expanded vector integration matrix and all gateway tests * test: reject index ARNs in bucket-level API requests * s3vectors: scope ARN resolution to the gateway namespace * s3vectors: reject name fields that conflict with resolved ARNs bucketARNs and resolveIndexARNs already reject an indexArn that names a different bucket than vectorBucketArn, but a vectorBucketName or indexName disagreeing with the ARNs was silently dropped. Validate the plain names against the resolved ARN values the same way. * test: cover conflicting vector names and ARNs in live requests --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
cc5912b8b8 |
s3vectors: e2e conformance for vector bucket policy evaluation (#12221)
* s3vectors: reject empty boolean-operator lists and sub-objects in filters An empty $and/$or list or a nested empty object rendered '()'' — invalid SQL surfaced as an engine error instead of a clean ValidationException. * s3vectors: e2e conformance for vector bucket policy evaluation The handler-level test covers statements; this runs the whole stack a client sees — HTTP dispatch, authentication, policy evaluation, catalog, engine — and walks the full lifecycle: a non-owner is denied without a grant, an Allow on the bucket ARN grants every index under it, a Deny vetoes an Allow on the same action, and deleting the policy restores owner-only access. * s3vectors: prove deny precedence and grant removal in policy e2e * s3vectors: reject field operator objects with no operators * test: use an s3vectors action in the vector policy round-trip |
||
|
Chris Lu
|
9d5929a73f |
s3vectors: reject empty boolean-operator lists and sub-objects in filters (#12220)
* s3vectors: reject empty boolean-operator lists and sub-objects in filters An empty $and/$or list or a nested empty object rendered '()'' — invalid SQL surfaced as an engine error instead of a clean ValidationException. * s3vectors: reject field operator objects with no operators |
||
|
Ping QiuandClaude Opus 5.5
|
3f954e34dc |
s3api: label bucket-level requests correctly in the access log (#12224)
* s3api: label bucket-level requests correctly in the access log getResourceType treated a request as bucket-level only when the object key was "/". NormalizeObjectKey strips the leading slash, so bucket routes arrive with an empty key and every bucket operation was logged as an object operation: ListObjectsV2 as REST.GET.OBJECT, DeleteObjects as REST.POST.OBJECT, and the lifecycle, ACL and policy calls likewise. Treat an empty object key as bucket-level too. The new test covers five bucket-level and two object-level request shapes; it fails without the fix. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * s3api: label only POST ?delete as a batch delete in the access log With bucket-level requests now reaching the bucket branch, any method on /bucket?delete was labelled BATCH.DELETE.OBJECT. Only the POST route is DeleteObjects; label other methods as the plain bucket operation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris Lu
|
ac7b4f4cf6 |
s3vectors: bound ListVectorBuckets and ListIndexes maxResults (#12218)
* s3vectors: bound ListVectorBuckets and ListIndexes maxResults Both pass maxResults to the catalog unbounded; AWS caps these list pages at 100, the same way ListVectors is bounded and topK is bounded. * s3vectors: raise ListVectorBuckets/ListIndexes maxResults cap to 500 AWS documents a valid range of 1-500 for both list APIs; the shared limit of 100 rejected legitimate page sizes. * s3vectors: reject explicit maxResults of zero on list calls AWS requires maxResults >= 1, but a plain int field cannot tell an omitted field from an explicit zero. Making MaxResults a pointer keeps the catalog default for omitted fields while a supplied zero fails validation. * s3vectors: test max ListIndexes page against an existing bucket |
||
|
Chris LuandDevin
|
f122f313e1 |
s3tables: keep the admin bypass off an identified caller's empty action list (#12217)
Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
229ad84e2d |
s3tables: reject s3vectors resources in every ARN partition (#12215)
* s3tables: reject s3vectors resources in bucket policies An arn:aws:s3vectors: Resource is as dead as an s3vectors action: the evaluator compares against the generated s3tables ARN, so the rule the writer believes enforced never matches. * s3tables: reject s3vectors resources in every ARN partition * s3tables: test wildcard-partition and mixed-list s3vectors resources |
||
|
Chris LuandDevin
|
98fe70a247 |
s3vectors: keep bucket policies in vector form and evaluate them in the gateway (#12214)
* s3vectors: translate vector bucket policies to the catalog form The document is stored verbatim on the shared table bucket and evaluated against s3tables actions and ARNs, so a policy written for the vector API was a dead rule. Actions now map to the catalog operation the request authorizes against, and an arn:aws:s3vectors: resource becomes the bucket/.../table/vector/... ARN the index's catalog entry answers to. * s3tables: expose PrincipalFor and PolicyContext for façade evaluation A façade authorizing against its own action namespace needs the same identity resolution and evaluation context the catalog uses internally. * s3vectors: keep bucket policies in vector form and evaluate them in the gateway The earlier approach translated the document to catalog actions and ARNs, which lost the caller's original policy for GetVectorBucketPolicy and could not cover data-plane calls that never touch the catalog. The document now stays in s3vectors form under a dedicated bucket attribute and every handler evaluates it through authorizeVector against the action and ARN the request names; a dedicated key also keeps a vector grant from ever authorizing a catalog call. * s3vectors: close policy-mutation races and keep vector grants visible - Authorize Put/DeleteVectorBucketPolicy inside the policy slot's compare-and-swap, so a grant revoked between reads cannot authorize the pending write. - Assert the policy attributes stay absent in the update precondition, so a concurrent policy or tag write trips the retry instead of being silently reverted. - Migrate a raw vector document out of the shared policy slot on replace or delete, keeping its catalog rules. - Pass the unresolved identity to PolicyContext so open-by-default deployments still work. - Filter ListVectorBuckets on the vector policy itself, so buckets a caller was granted under s3vectors actions stay listed. * s3vectors: reuse SnapshotExtended for the vector-policy CAS precondition * s3vectors: skip buckets that vanish mid-list instead of failing the page * s3vectors: reject non-vector bucket formats inside authorizeVector * s3vectors: check bucket format after authorization in authorizeVector An unauthorized caller probing an ICEBERG bucket got NoSuchBucket,\nleaking that the bucket exists. The format check now runs after the\npermission check so denied callers always see AccessDenied. * s3vectors: cover legacy shared-slot policy removal on delete * s3tables: resolve the forwarded identity name like getAccountID in PrincipalFor A synthesized manager request resolves its caller through the identity- name branch of getAccountID, which normalizes the name (ARNs collapse to their trailing segment, whitespace trims) before falling back to the handler's account ID. PrincipalFor returned the raw name, so ARN-shaped identities never matched their own grants. * s3tables: assert the vector policy key in update preconditions s3vectors-policy is stored on table bucket entries but was missing from s3tablesExtendedKeys, so a catalog update that read the entry before the first vector policy was saved could overwrite it without tripping the compare-and-swap. Asserting the key makes such a race fail instead of silently erasing the policy. * s3vectors: reject rules the evaluator cannot honor in vector bucket policies The shared Statement shape has no negated fields, the vector evaluation supplies no condition keys, and a document that fails the typed decode is discarded wholesale at evaluation — including denies the writer believed active. Reject NotPrincipal, Condition, foreign-service actions and resource ARNs, and require the typed PolicyDocument decode. Also cover that an admin-named identity still reaches its own buckets. * s3vectors: read vector policy fields the way the evaluator decodes them Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: match ARN policy principals through caller normalization Caller principals are normalized ids, so a statement naming a full IAM ARN could never match: an ARN-scoped deny was silently dead and an ARN-scoped allow unreachable. Compare arn: specs against the caller under the same normalizePrincipalID the ownership path uses, while an ARN caller keeps verbatim wildcard matching. * s3tables: grant the admin bypass on the caller's actions, not the name A principal that only normalizes to the admin id skipped every policy deny. When the request carries an identity, the bypass now requires the identity's own admin action — the same gate getAccountID applies — while name-only transports with no action list keep it. PolicyContext takes the request so it can carry IdentityActions, and the vector context helpers route through the shared identity keys so the resolved name and actions agree. * s3tables: match ARN policy principals only for deny statements Normalization drops the ARN's account and path, so an allow matched on the trailing name could grant a same-named caller the statement never verified. Denies keep the normalized match so the rule cannot be ducked through the same collapse. * s3tables: scope service wildcards to their own service s3tables:* counted as blanket admin inside the shared permission check, so an identity holding only catalog grants administered s3vectors operations too. hasAdminActionFor now requires the wildcard to name the operation's service; the global admin markers and s3:* stay service-agnostic. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
4ef123d764 |
s3vectors: fix shared API behavior and test Go/Rust volume parity (#12216)
* s3vectors: reuse shared authenticated identity context * s3vectors: encode empty results as arrays * test: cover S3 Vectors with Go and Rust volume servers * ci: require both volume implementations in vector integration tests * s3vectors: keep identityOf as a shim over the shared identity context Concurrent branches add identityOf call sites; deleting the helper would break the build when they merge in either order. Route it through GetIdentityNameFromContext so every caller resolves the same identity. * test: wait for the vector engine to register before running The gateway resolves weed-vector lazily through the master cluster list, so a listening port does not mean the engine is reachable yet; the first vector write or query could race KeepConnected and fail. Restore the cluster.ps registration wait the previous harness had, and ignore the pytest cache directory the suite creates. * test: wait for master registration across OSS and enterprise Use the exact engine registration event after AddClusterNode completes. The current enterprise cluster.ps does not list vector nodes, so waiting on its output prevents the shared fixture from starting there. * s3vectors: keep contextWithIdentity as a shim alongside identityOf In-flight callers still write the caller name through it; route the write through SetIdentityNameInContext so both helpers agree on the shared key whichever lands first. |
||
|
Chris Lu
|
88565df885 |
s3tables: reject s3vectors resources in bucket policies (#12213)
An arn:aws:s3vectors: Resource is as dead as an s3vectors action: the evaluator compares against the generated s3tables ARN, so the rule the writer believes enforced never matches. |
||
|
Chris Lu
|
15066c8a9f |
vector: implement drop_index and have DeleteIndex drop the data (#12204)
* vector: implement drop_index in the engine The RPC was a stub; it now resolves the table's vended location and removes every object under it through the same credential path open uses. A table the catalog can no longer describe is already dropped, so the call stays idempotent. * s3vectors: drop index data after the catalog delete DeleteTable carries the delete authorization, so it runs first; the engine DropIndex then removes the data under the index's location. NotFound and Unimplemented are tolerated so an index whose data went away with the catalog entry, or an older engine, still answers 200. * s3vectors: let the catalog delete carry the index data removal A vector index is colocated: the catalog entry sits at the dataset root, so DeleteTable's recursive delete already removes entry and data. The engine DropIndex call afterwards could only ever describe a missing or recreated record — a no-op in the common case, and a delete of a replacement index's files when the name was reused in between. |
||
|
Chris Lu
|
2dbf9c9108 |
s3: evaluate all conditional write headers under the object lock (#12211)
* filer_pb,filer: per-mutation preconditions on ObjectTransaction A transaction's request-level condition evaluates one entry before any mutation runs, so a commit guarding two paths — the object it publishes and the upload directory it assembles — could not be expressed. Give each ObjectMutation an optional condition (evaluated against condition_key, else the mutation's target, under the same lock just before the mutation applies) and report which mutation failed through failed_mutation on the response, so callers can tell a mutation guard's rejection apart from the request-level precondition's. * s3: evaluate multipart completion conditions under the object lock CompleteMultipartUpload checked conditional headers gateway-side before committing, and the routed commit transaction guarded only the upload directory — nothing made If-None-Match: * atomic with the publish, so concurrent conditional completions of the same absent key could all report success. The request's reducible write condition (If-None-Match/If-Match, star or single strong ETag) now rides the commit as the transaction's request condition on the owner filer: on the object path for unversioned buckets, falling back to the .versions directory for suspended and enabled versioning. The upload-exists guard moves onto the PUT mutation, keeping its check-before-publish ordering. Conditions the reducer cannot express stay on the distributed write lock, where check and commit already serialize. * filer,s3: resolve the latest version first for versioned conditional writes * filer: keep directory entries holding content out of the absent-slot rule * s3: keep a gateway-side upload-exists check for filers without mutation conditions * filer: honor null-version-is-latest resolving prefer-alt conditions * drop scratch multipart-race repro programs * s3: note the write lock does not cover routed writes on non-reducible conditions * filer: mirror the gateway's remote-etag precedence in condition evaluation storedEntryETag only knew the extended attribute and the chunk fallback, so a synced remote entry compared chunk-derived ETags while the gateway compares the remote ETag. Follow the same precedence: extended attribute, remote ETag while the entry still holds the remote bytes, Md5, then the chunk hash — and no ETag for an entry holding none. * s3: reduce all conditional write headers into owner-lock clauses Time-based headers, combined conditions, ETag lists, and weak ETags stayed on the distributed write lock because buildWriteCondition could not express them, leaving their gateway check racing owner-routed writes. Reduce every parseable header form to WriteCondition clauses — multiple clauses compose AND, ETag candidates carry the raw list for the filer's canonicalization, and RFC 7232 precedence keeps If-Unmodified-Since hidden by If-Match and If-Modified-Since hidden by If-None-Match — so the owner filer evaluates them under the same object-path lock as the commit. Only unparseable headers still fall back, where the lock path reports the same parse error. |
||
|
Chris Lu
|
2737315e9d |
s3: evaluate multipart completion conditions under the object lock (#12209)
* filer_pb,filer: per-mutation preconditions on ObjectTransaction A transaction's request-level condition evaluates one entry before any mutation runs, so a commit guarding two paths — the object it publishes and the upload directory it assembles — could not be expressed. Give each ObjectMutation an optional condition (evaluated against condition_key, else the mutation's target, under the same lock just before the mutation applies) and report which mutation failed through failed_mutation on the response, so callers can tell a mutation guard's rejection apart from the request-level precondition's. * s3: evaluate multipart completion conditions under the object lock CompleteMultipartUpload checked conditional headers gateway-side before committing, and the routed commit transaction guarded only the upload directory — nothing made If-None-Match: * atomic with the publish, so concurrent conditional completions of the same absent key could all report success. The request's reducible write condition (If-None-Match/If-Match, star or single strong ETag) now rides the commit as the transaction's request condition on the owner filer: on the object path for unversioned buckets, falling back to the .versions directory for suspended and enabled versioning. The upload-exists guard moves onto the PUT mutation, keeping its check-before-publish ordering. Conditions the reducer cannot express stay on the distributed write lock, where check and commit already serialize. * filer,s3: resolve the latest version first for versioned conditional writes * filer: keep directory entries holding content out of the absent-slot rule * s3: keep a gateway-side upload-exists check for filers without mutation conditions * filer: honor null-version-is-latest resolving prefer-alt conditions * drop scratch multipart-race repro programs * s3: note the write lock does not cover routed writes on non-reducible conditions |
||
|
Chris Lu
|
c7d77eb9a8 |
s3vectors: accept and return encryptionConfiguration on vector buckets (#12203)
* s3tables: carry encryptionConfiguration in table bucket metadata * s3vectors: accept and return encryptionConfiguration on vector buckets The configuration is validated (sseType AES256 or aws:kms) and stored on the bucket record, so the API round-trips what the caller declares. * s3vectors: reject caller-chosen encryption keys on vector buckets * s3tables: reject caller-chosen encryption keys on table buckets * s3tables: reject unknown sseType in bucket encryptionConfiguration The bucket check rejected aws:kms and kmsKeyArn but stored any other sseType — a made-up or mistyped value would round-trip through GetTableBucket as an encryption claim the storage does not keep. The validator now admits only the AES256 declaration, shared so the vector layer can drop its copy. * s3vectors: delegate encryption validation to the shared catalog check * s3tables: accept the sseAlgorithm name in bucket encryptionConfiguration S3 Tables declares the algorithm as sseAlgorithm while S3 Vectors uses sseType; the shared check now accepts either spelling, still admits only AES256, and still refuses any caller-chosen key. * s3vectors: call the shared encryption validator directly * s3vectors: require each API's own encryption field name The shared validator accepted sseAlgorithm on the vector path, so a CreateVectorBucket carrying the s3tables field stored a configuration GetVectorBucket could never echo back in the vectors shape (and vice versa). ValidateEncryptionConfiguration now takes the wire name its caller expects; the vectors create path normalizes to sseAlgorithm for the catalog and GetVectorBucket renames it back to sseType. |
||
|
Chris Lu
|
fc9be77dbb |
s3vectors: bound QueryVectors topK at the AWS limit (#12202)
* s3vectors: bound QueryVectors topK at the AWS limit of 30 topK passed through unbounded; AWS caps a query at 30 results, so a larger request is a ValidationException before it reaches the engine. * s3vectors: raise QueryVectors topK bound to the AWS limit of 10,000 * s3vectors: validate QueryVectors topK before the catalog lookup * s3vectors: cover the QueryVectors topK bounds |
||
|
|
eecc461b39 |
s3vectors: put, get and delete vector bucket policies (#12201)
* util: do not bind outbound dials to a loopback -ip.bind `weed server -ip.bind=127.0.0.1` keeps every listener local, but the same address became the source of outbound connections, so the filer dialled its remote mount from 127.0.0.1 and failed with "connect: invalid argument". A loopback source cannot reach off the host; leave the source to the OS, as for a wildcard bind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: put, get and delete vector bucket policies The AWS S3 Vectors policy operations had no handlers; the policy document is stored on the shared table bucket, so the handlers delegate to the catalog's PutTableBucketPolicy, GetTableBucketPolicy and DeleteTableBucketPolicy and keep its guard. * s3vectors: treat ICEBERG buckets as absent in policy handlers GetVectorBucketPolicy/PutVectorBucketPolicy/DeleteVectorBucketPolicy delegated straight to the table bucket, so a bucket created for ICEBERG tables — invisible to every other vectors operation — could still have its policy read and written through the vectors API. Gate all three on lanceBucket, the same format check DeleteVectorBucket and CreateIndex already apply. * test/vectors: cover vector bucket policy put/get/delete Round-trips a policy document through the new ops and asserts a get on the deleted policy surfaces NotFoundException, so the boto3 conformance run exercises the added handlers end to end. * s3vectors: validate the bucket ARN and policy document Bucket-level operations now reject an index ARN and an ARN minted for another service, and PutVectorBucketPolicy refuses a document carrying s3vectors actions — the evaluator only honors s3tables actions, so such a rule would be a dead grant. * s3vectors: validate the bucket ARN on the remaining bucket-level operations * s3vectors: cover bucket ARN and policy document validation TestBucketARNs pins the accepted forms and the index/foreign-service rejections; TestValidatePolicyDocument covers a valid document, a malformed one, and the dead s3vectors action namespace. * s3vectors test: use an enforceable action in the bucket policy test The catalog refuses s3vectors actions in a stored policy, so the put call rejected the document before the get and delete assertions ran. --------- Co-authored-by: Ólafur Bjarki Bogason <olafur@genkiinstruments.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandDevin
|
8e44178dc6 |
s3vectors: cap PutVectors at 500 items and check data against the index dimension (#12200)
* s3vectors: cap PutVectors at 500 items and check data against the index dimension A batch could carry any number of vectors and a vector of the wrong dimensionality reached the engine: AWS caps a PutVectors call at 500 items and requires each vector to match the index's declared dimension. * s3vectors: cover the PutVectors batch and dimension checks * s3vectors: reject query vectors that do not match the index dimension Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: cover the PutVectors batch size bounds Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: cover the catalog-backed PutVectors handler path Run PutVectors through the real handlers over the in-memory catalog: a matching vector reaches the engine, a mismatching one is rejected with ValidationException before PutVectors is called, and a full vectorsMaxPutBatch batch is accepted. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
9df7d62b4b |
mount, s3: clarify allowOthers requirement and bucket name error (#12197)
* mount, s3: clarify allowOthers requirement and bucket name error
`weed mount -allowOthers` defaults to true. On Linux, a non-root
user's mount then needs `user_allow_other` in /etc/fuse.conf, but the
help text only said "allows other users to access the file system".
Say so in the help, and point at -allowOthers=false.
Also fix the grammar of the bucket name length error ("must between
[3, 63]" -> "must be between 3 and 63").
Help and error text only; no behaviour change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: normalize the remaining VerifyS3BucketName error messages
The length message was clarified; the rest of the function still had
inconsistent phrasing ("name must start", "cannot be ip addresses",
"prefix xn-- ... in bucket prefix"). Align them on "bucket name ..."
wording matching the AWS bucket naming rules.
---------
Co-authored-by: pingqiu <pingqiu999@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
|
||
|
Chris Lu
|
ec43f4ca00 |
s3: accept bare composite checksum on CompleteMultipartUpload (#12194)
* s3: accept bare composite checksum on CompleteMultipartUpload Clients such as minio-go send the composite object checksum without the "-<parts>" suffix. Compare against the base64 part of the computed composite value as well as the full form. * s3: test object checksum matching with and without part suffix |
||
|
Jay Hemnanianduser.email
|
6225e0a705 |
s3: return x-amz-copy-source-version-id from CopyObject and UploadPartCopy (#12181)
* s3: return x-amz-copy-source-version-id from CopyObject and UploadPartCopy * s3: cover x-amz-copy-source-version-id on latest copies and part copies The explicit-version case was already covered; add a .versions pointer fixture so the unversioned request resolves the latest version and asserts the header on CopyObject, plus the same assertion on UploadPartCopy. * s3: carry the resolved id on a version file that lacks it A .versions child written before the version-id attribute resolves to an empty source version even though the pointer or the rescan already knows it. Stamp the id on the returned entry so copy and get readers report it; the version copy tests now drop the attribute to cover that shape. * s3: report the null version in x-amz-copy-source-version-id A regular-path source in a versioning-configured bucket is the null version; echo it like every other version id, matching the GET path and AWS. Unversioned sources still report no header. * s3: report the version a self-copy retry actually read --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
|
48782e2b30 |
s3: allow PutBucketAcl when auth is off (#12180)
* s3: allow PutBucketAcl when auth is off * s3: resolve the PutBucketAcl owner against the recorded owner The unauthenticated path now resolves the account as anonymous, but XML ACLs were still checked against it while GetBucketAcl reported an empty owner for ownerless buckets, so a read-modify-write ACL update failed with AccessDenied. GetBucketAcl and the XML check now both use the recorded owner when the bucket has one and the resolved account otherwise. The canned-ACL grantee moves to the same owner, and a request carrying no account no longer erases a stored owner. * s3: cover PutBucketAcl without auth for XML bodies and stored owners The XML case posts the anonymous owner GetBucketAcl reports for an ownerless bucket; the canned case asserts a recorded owner survives an unauthenticated update and receives the full-control grant. Ambient AWS credentials are cleared so the memory store never turns auth on. * s3: keep the recorded owner through a PutBucketAcl update The grantee and the stored owner were resolved differently: grants named the recorded owner while config.Owner still took the caller's account, so an admin updating another account's bucket rewrote the owner and broke an XML retry naming the original one. An ownerless bucket adopts the ACL's owner; an owned bucket keeps it. * s3: fall back to the admin owner for an ownerless bucket in ACL handlers An unauthenticated PutBucketAcl resolved the owner as anonymous and stored it on the bucket, but the registry reports an ownerless bucket as admin-owned: BucketOwnerEnforced uploads then stamped anonymous instead of admin, and ownership checks after enabling auth compared against the wrong owner. Both ACL handlers now resolve an anonymous fallback to the admin account, so GetBucketAcl reports the same effective owner the registry does and the stored owner matches it. * s3: resolve the admin owner for bucket grants when the caller is anonymous or untrusted Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: skip recording anonymous identity or account as bucket owner Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: report stored bucket ACL grants verbatim Explicitly saved anonymous full-control grants must survive a GET-to-PUT round trip unchanged, so the read path no longer rewrites them to the reported owner. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: expect the stored anonymous grant verbatim in GetBucketAcl Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
74f196ceab |
s3: route regular copy source reads to the write owner (#12173)
* s3: read regular copy sources from their write owner * filer: report mutation sentinel codes in ObjectTransaction responses A failed mutation only surfaced resp.Error text, so routed writers could not distinguish an existing-directory conflict from a generic failure and could not apply the prefix-object retry the plain CreateEntry path has. * s3api: retry a routed mkfile as a prefix object over a directory mkFile retries ErrExistingIsDirectory by marking the entry a prefix object; the routed path returned the conflict instead, so a routed write to a key that is also a prefix failed where the lock path succeeds. The filer now reports the code on the transaction response, making the retry recognizable. * s3api: commit copy destinations on the key's write owner An owner-routed source read still left the destination on the gateway's default filer, so a second copy resolved the first copy's destination on the owner, found nothing, and returned NoSuchKey. Write unversioned and suspended destinations through routedMkFile, and write the version file on the owner the finalize recomputes on. * s3api: cache remote objects on the filer that served the entry An owner-routed lookup can return a remote-only entry that has not replicated to the gateway's default filer; the cache RPC then cannot find it and the copy or read degrades to a 503. Ask the entry's write owner to cache it first, falling back to the default filer. * s3: read versioned object state from the key's write owner CopyObject destinations now commit version files and the .versions pointer on the owner's object-scope ring key, so versioned reads through the default filer could miss them until replication. Generalize the routed read to dir/name lookups and paginated lists scoped to the object's owner, and point the versioned read paths at it: getSpecificObjectVersion, doGetLatestObjectVersion, the .versions quick-checks in Get/Head/attributes, the self-heal rescans, tagging, delete conditionals, directory-marker history checks, rename destination, multipart completion checks, and the version-id format probe. Reads fall back to the default filer when no ring owner is resolvable, preserving single-filer deployments. * test(s3api): fake RECOMPUTE_LATEST/DELETE and cover versioned copy reads The ambiguous-PUT fake filer now creates parent directories on PUT (mirroring InsertEntry), applies DELETE mutations, and re-derives the .versions pointer for RECOMPUTE_LATEST the way applyRecomputeLatest does, so owner-vs-replica tests can exercise versioned writes. TestVersionedCopyDestinationReadsOnOwner copies into a versioned bucket whose writes commit on the owner while the gateway's default filer is a replica with no replication, then asserts reads by versionId, latest-version resolution, and a copy chained off the new version all resolve on the owner. * test(s3api): gofmt the versioned-copy coverage * s3: only swap in a cache fill of the resolved remote generation A cache response can come back from a filer holding an older generation of the key. Replacing the authorized source or swapping its chunks under a size and range computed for the resolved entry serves the wrong object or truncates the response. Match RemoteEntry identity and size before using a cached entry, in both the copy and streaming paths. * filer: keep directory attributes when an object lands as prefix object The object-onto-directory retry replaced the existing directory's mode, owner, and extended attributes with the object's. Preserve the directory identity; merge directory-held extended keys only when the old entry never held an object, so a prior generation's object metadata is not carried forward. * s3: keep marker lock checks on the filer that replaces the marker The object-lock precondition read the key's write owner while mkdir still replaces the marker on the default filer, so a locked marker not yet replicated to the owner could be overwritten. Read the entry where the replacement lands, in both PutObjectHandler and zip extraction. Also hash a trailing-slash marker key to the same route key as the object it names, so marker and object share one owner and one write lock. * s3: route tag writes to the owner tag reads resolve GetObjectTagging reads the key's write owner, but PutObjectTagging and DeleteObjectTagging still wrote through the default filer, so a successful change returned the owner's stale tags until replication. setTags, rmTags, and the versioned tag updates now write on the owner first, falling back to the default filer when no owner resolves. * s3: keep rename lookups on the filer that performs the move AtomicRenameEntry and the retry token it carries both execute on the default filer, but the source and destination reads went to the write owner — a completed rename's token was invisible to its own retry, and a source that only exists on the default filer looked absent. Read both ends on the default filer. * s3: send the .versions pointer repair with the object's route key The heal reads the .versions entry on the object's owner, but the conditional UpdateEntry is routed by the .versions path's own ring key, which can hash to a different filer — leaving the pointer the routed reads consult unchanged. Ship the CAS repair as a PATCH_EXTENDED ObjectTransaction on the object's route key instead. * filer: preserve only directory identity on a prefix-object write Merging the old directory's extended attributes could carry a marker's encryption, version, or etag fields onto the new plaintext object, since a directory's own keys cannot be told apart from the object it held. Keep only mode, uid, gid, user name and group names. * s3: also require matching size when a cache fill replaces the source The remote-generation check covers identity; size equality is the cheap guard that keeps the streamed or copied range consistent with what was resolved. * s3: lock the object path for a routed .versions pointer repair conditionalUpdateEntryRouted held the lock on the .versions path while version writes and deletes serialize on the object path, so a repair could pass IF_ENTRY_EQUAL and then apply over a pointer a writer had advanced in between. Lock the object path like the writers and keep the condition on the .versions entry via condition_key. * s3: do not replay an owner-side tag mutation on another filer withWriteOwnerFilerClient re-ran fn on the default filer after any owner error, so an authoritative not-found could update a stale replica entry and still report success, and a post-apply transport error could apply the mutation twice. The failover walk now ends on any error fn returns; only a channel that never came up moves to another filer. * filer: keep a directory's xattrs when a prefix object replaces it The prefix-object branch preserves the directory's identity but dropped its Extended map wholesale; FUSE xattrs live there under the xattr- prefix and belong to the directory, not to the object the marker held. * s3: delete null and specific versions on the filer that serves them On the default filer an absent entry deletes as success, so a version that only reached the write owner survived a reported success while owner-routed reads kept serving it. The suspended-versioning and lifecycle deletes now go through routedDeleteNullVersion and routedDeleteSpecificVersion when an owner resolves; an unsettled attempt keeps the lock path. * s3: route a lifecycle null-version delete to the object entry The "null" version is the regular object entry, not a .versions file, so it goes through routedDeleteNullVersion; only a real version id takes the .versions-file delete. * s3: repoint .versions only while it still names the deleted version routedDeleteSpecificVersion recomputed the latest pointer on every version delete, so removing a noncurrent version re-derived the pointer to the newest remaining file — resurrecting an older version over a cleared pointer (a suspended bucket's null write keeps it clear while the null object stays current) or over a pointer that moved on. The pointer is now read on the owner and repointed under an IF_ENTRY_EQUAL guard only while it still names the deleted version; a contested pointer is left for the self-heal path rather than overwritten by a stale scan. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: let lifecycle expiry keep its metadata-only deletes on routed paths The routed lifecycle deletes hardcoded IsDeleteData, so expiring an entry with a volume TTL queued per-chunk deletions instead of leaving the data for the volume to reclaim — extra work on the owner for deletes recordMetadataOnlyIf still counts as metadata-only. Pass the caller's cleanup choice through routedDeleteNullVersion and routedDeleteSpecificVersion; the interactive DELETE path keeps deleting data. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: let RECOMPUTE_LATEST skip a pointer that moved on A version delete repoints .versions only while the pointer still names the deleted version, but the gateway cannot carry the pointer's current state into the transaction's condition — the condition slot is already spent on the version's WORM guards. only_if_current_name moves the check under the transaction lock: the filer compares the pointer's name_to_key value before scanning and skips the recompute when it moved (or was cleared by a suspended bucket's null write), so a noncurrent delete cannot resurrect an older version. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: repoint inside the version's locked delete repointLatestBeforeVersionDelete ran in its own transaction ahead of the delete, so a version delete denied by retention or legal hold still moved the latest pointer — GET and HEAD went on serving the older version it rolled to. The repoint and the delete now ride one transaction behind the version's WORM guards, atomically gated by the new only_if_current_name recompute guard instead of a stale-read IF_ENTRY_EQUAL. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
511727cf3a |
vector: DeleteIndex format guard, sticky discovery, advertise normalize (#12171)
* vector: DeleteIndex format guard, sticky discovery, advertise normalize - handleDeleteIndex verifies the table carries the Lance format before DeleteTable — an Iceberg table under the same namespace is not an index and must not be removed by it. - discovery keeps the current worker while it is still registered — membership-order churn can no longer cancel in-flight calls, and the resolver closes each master conn inside a 5s deadline. - advertised_addr strips scheme and .grpcPort so the UDP probe reaches every supported master form. * vector: empty-follower retry, format check without read grant - discoverVectorWorkers keeps looking past an empty master reply — followers list nothing when workers registered with the leader. - the DeleteIndex format check uses the internal catalog read, so DeleteTable-only callers don't gain a GetTable requirement, and the format must be Lance exactly — format-less tables no longer pass. |
||
|
Nguyễn Lộc Phúcandchrislusf
|
b62bbb99fc |
fix(iceberg): serve a registered table's metadata on load and commit (#12159)
* iceberg: serve a registered table's metadata on load and commit RegisterTable saved only the metadata location, and loads and commits never read the file there. They served a placeholder with no schema, no snapshots and a new random table UUID on every load, so a registered table read as empty, an INSERT failed against the empty schema, and every other commit failed its assert-table-uuid requirement. Metadata uploaded through the S3 gateway, as a migrated table's is, is stored in chunks that registration did not read, so it registered an empty table. Read the file following its chunks, reject one that is not Iceberg metadata with a 400, and keep its content and table UUID in the record the way CreateTable does. Records registered before this change hold only the location. Loads and commits now read the file there, and the first commit stores it in the record under the file's table UUID. * iceberg: transactions read a location-only registration's metadata file * iceberg: cover transactions on a location-only registration * iceberg: refresh the default name mapping on transaction commits * iceberg: cover name-mapping refresh in transactions * iceberg: roll a transaction back to a registration's metadata file A transaction that failed after flipping a table registered before the record kept its metadata could not undo the flip. The rollback restored the record's prior metadata, which for such a table is only a location, and UpdateTable keeps the full metadata a flip stored when the restore sends none. Loads went on serving the change the client was told had failed, and a client retrying the transaction could apply it twice. Restore the metadata loads served from the file instead, under the file's table UUID. * iceberg: register oversized metadata by its location Registration sends the metadata file's content to the S3 Tables manager, which refuses a request over its 10MB body limit, so a table whose metadata is larger could no longer be registered at all. Register such a file by its location and table UUID alone; loads read it from that location, as they do for a record that never kept its metadata. Export the limit as s3tables.MaxRequestBodySize so the catalog can check against it. A commit to such a table still sends the whole metadata to UpdateTable and stays over the limit. * iceberg: commit oversized metadata by its location * iceberg: let maintenance follow a location-only table record * admin: read a table's metadata file when the record keeps none * iceberg: bound the metadata file a location-only record reads * iceberg: bound only chunked metadata reads in the worker Inline content is what the catalog itself wrote and the REST load path accepts it at any size, so rejecting it in loadMetadata stranded tables over 64 MiB from every maintenance scan. Cap only the chunked read, where the recorded location can name any object in the bucket. --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> |
||
|
Jay Hemnanianduser.email
|
9059cc2a07 |
s3: allow PutObjectAcl when auth is off (#12155)
* s3: allow PutObjectAcl when auth is off * s3: ownerless objects claim the admin account when auth is off An object written through the filer carries no owner metadata, so objectOwner fell back to an empty account id and ExtractAcl rejected the canned ACL's canonical grantee. Extend TestPutObjectAclWithoutAuth to cover the ownerless case. * s3: GetObjectAcl reports the admin owner for ownerless objects when auth is off The same empty-owner fallback produced an ACL response with a blank Owner ID and a blank canonical grantee. * s3: ignore the client-supplied account id when auth is off Auth does not strip AmzAccountId on the iam-disabled path, so the ownerless fallback could persist a forged owner. requestAccountId substitutes the admin account, matching the idiom already used in the upload ACL path. --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
d5433ae745 |
vector: cover the identity object in s3vectors auth (#12162)
vector: cover the identity object in the s3vectors auth context The auth middleware must carry the resolved identity object — only the name survives otherwise and authorization loses actions, account, and claims. |
||
|
Chris Lu
|
afb8472c5e |
s3vectors: cover non-filterable key round-trip and nested filter rejection (#12160)
* s3vectors: cover non-filterable key round-trip and nested filter rejection The catalog writes the key list as a JSON string while older records may still carry the array form; indexFromRecord must decode both. The rejection path extracts field names through nested $and/$or filters, so a blocked key deep in the tree still fails the query. * s3vectors: factor the non-filterable rejection into a testable gate Pull the blocked-key check out of handleQueryVectors into nonFilterableField so the rejection itself is covered without standing up the catalog stack; the handler now 400s on whatever the gate names. * s3vectors: report the non-filterable match separately from the key An empty metadata key in nonFilterableMetadataKeys collided with the "no match" sentinel, so a filter on that key skipped rejection. Return a match flag and cover the empty-key case in the test. |
||
|
Chris Lu
|
237db29dca |
vector: round-trip and enforce non-filterable metadata keys (#12157)
vector: non-filterable metadata round-trips and filters reject it - GetIndex omitted nonFilterableMetadataKeys — the catalog stores the list as a JSON string but the reader only accepted an array; both forms now decode. - QueryVectors now rejects filters referencing non-filterable keys instead of silently scanning them. |
||
|
Chris Lu
|
10ba1b782f |
vector: carry the full identity through s3vectors auth (#12152)
vector: keep the full identity in the request context Auth discarded the resolved identity object — the s3tables manager request then re-resolved by name and denied authenticated users their actions, policies, account, and claims. Store the object via SetIdentityInContext so downstream authorization sees it. |
||
|
Chris Lu
|
6536fac9bd |
s3: answer NoSuchBucket for encryption and policy writes to a missing bucket (#12133)
* s3: answer NoSuchBucket for encryption and policy writes to a missing bucket PutBucketEncryption, DeleteBucketEncryption, and PutBucketPolicy surfaced the filer's bucket-miss as InternalError; GetBucketEncryption masked it as ServerSideEncryptionConfigurationNotFoundError. Map the miss to NoSuchBucket instead, matching AWS. Internal readers still treat a missing bucket as "no encryption configured" so uploads keep their behavior. * s3: wrap ErrBucketNotFound only for verified misses in bucket policy ops The filer lookup helpers in get/set/deleteBucketPolicy marked every LookupDirectoryEntry failure as bucket-not-found, so a transient filer outage answered NoSuchBucket for a live bucket. Route them through filer_pb.LookupEntry, which normalizes only true misses to ErrNotFound. * s3: answer NoSuchBucket when the policy delete's own write loses the bucket * s3: classify a filer NotFound on the policy UpdateEntry as ErrBucketNotFound * s3: require the bucket entry to exist on config patch transactions * s3: answer NoSuchBucket when tagging and CORS writes lose the bucket mid-request * s3: keep the existence check on no-op bucket config patches An unchanged config write produced an empty patch that early-returned before the transaction, so a bucket deleted between the read and the write still reported success. Ship the condition-only transaction so IF_EXISTS runs on its own. * s3: guard the iam step and cover failure-vs-miss classification Nil-guard s3a.iam in the bucket policy handlers so the handlers do not panic when no IAM server is wired (unit tests, embedded servers), treat a "precondition failed" response string the same as the error code when mapping a failed patch to ErrNotFound, and add handler-level tests that pin the miss/failure distinction: missing buckets and vanished mid-write buckets answer 404 NoSuchBucket while transient filer failures stay 500 InternalError. |
||
|
Chris Lu
|
4c9e62177d |
s3: keep multipart part encryption on the initiate-time decision (#12134)
* s3: keep multipart part encryption on the initiate-time decision putToFiler applied the *current* bucket default encryption when writing a staged part, so a PutBucketEncryption landing between two UploadPart calls produced an object whose parts disagree: some chunks ciphertext, some plaintext. Completion then marks the object by its majority and GetObject fails decrypting the rest. Parts staged under .uploads now skip the bucket-default lookup; the decision snapshotted into the upload entry at CreateMultipartUpload is replayed through request headers instead, which matches AWS (default encryption applies at initiation). * s3: mark multipart part writes via request context, not the staged path Inferring "is a part" from the .uploads file path let an ordinary PutObject named .uploads/x.part skip bucket default encryption entirely. The part decision now travels on the request context from the two callers that stage parts (UploadPart, UploadPartCopy); a client-chosen key cannot reach it. |