mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-07 12:37:41 +01:00
2fe480a6e2a7cf8de199d12a9bfdd1cfdc6caaaf
1663
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Fable 5.1
|
2fe480a6e2 |
S3 Vectors: the gateway resolves the table it sends to weed-vector (#12468)
* lance catalog: in-process DescribeTable for trusted callers DescribeTable and DescribeTableInternal resolve a table's location and storage options without the HTTP layer; the internal form runs the lookup through a trusted manager so the caller's catalog grants do not gate a resolution its own permission already authorized. Same code as the Enterprise repository. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector.proto: Table carries its location and storage options The gateway resolves the table it addresses, so the engine can open it without a describe round trip of its own. Both fields are optional; an empty location means the engine looks the table up as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: test the in-process describe Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: resolve the table before calling the engine The gateway describes the table in process, through the trusted catalog manager, and sends its location and storage options with every engine call. Storage options are vended for the caller's principal, as the Enterprise router does for queries. A catalog failure answers with the code the engine would have given. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: open a table at the location the gateway sent A request whose table carries a location is opened there, with the storage options the gateway vended over the engine's fallback; the catalog is described only for a table that comes without one, as from an older gateway. dataset::open_at and dataset::drop_at take a description the caller already holds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: run the engine without a catalog VECTOR_ENGINE_NO_CATALOG=1 gives the engine an unreachable catalog URL; every CI leg sets it, so the suite proves the gateway resolves each table it sends. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: an internal describe may write a vector index With credential vending on, the options vended to the engine followed the caller's s3tables grants, which never allow writing the vector namespace, so every write the gateway sent would have run with read-only credentials and reads would have lost their version lease. The engine is that namespace's writer and the caller's vector permission authorized the operation, so an internal describe of a vector table vends write-capable options; other namespaces keep the UpdateTable check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: resolve tables for the engine only while the catalog runs With -port.lance=0 the gateway has no catalog to resolve through; the engine then keeps looking tables up itself, as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: a deregistered table describes as NoSuchTable So the gateway answers NotFoundException for it, not InternalFailure. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: trust a sent location only for a colocated vector index A caller that reaches the engine could otherwise name any storage the engine's credentials cover. A location is taken from the request only when the id names a vector index and the location is its colocated path, which the engine writes already; anything else is described through the catalog as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: scope the read cache by storage endpoint The cache was keyed by bucket and path alone, so the same path reached through another endpoint would have shared entries. Each open wraps the store with the cache scoped to the endpoint its options name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: keep a configured endpoint over one a request names A caller that reaches the engine could otherwise send its fallback credentials to an endpoint of the caller's choosing. When the engine's --storage-option names an aws_endpoint, a table whose request names another one is described through the catalog instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: use request options only where credentials cannot leave A request's storage options are taken only when they hold the keys a gateway's describe produces, so a proxy or an endpoint alias cannot route the engine's credentials elsewhere, and an endpoint only when it is the configured one or the engine holds no credentials to send. Anything else sends the table to the catalog. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: give the engine its endpoint with its credentials Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a drop checks the request's options like an open Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: an unpinned endpoint needs the request to sign for itself Without configured credentials the S3 store still signs with the AWS default chain, so an endpoint from a request is taken only when the request skips signing or brings its own credentials. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a trailing slash does not unpin the endpoint, and say so A configured aws_endpoint written with a trailing slash matched nothing the gateway sends, so every table fell back to the catalog in silence. The comparison ignores the slash, and a fallback for a resolved table is logged at debug. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: resolved by the gateway, or refused The engine no longer has a catalog: --namespace and --filer-signing-key are gone, a request whose table names no location is InvalidArgument, and one whose location is not the index's own or whose storage options the engine's credentials must not be used with is PermissionDenied. A drop removes the colocated path it was given. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: the catalog resolves vector tables whether or not its port is served newLanceServer builds the Lance catalog for the S3 Vectors API; the listener on -port.lance is optional. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors tests: start the engine without a catalog Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: the aux listener test builds the catalog before serving it Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: an unpinned endpoint takes only credentials the request signs with alone Lance fills an option a request leaves out from the engine's environment, under any of object_store's names for it (TOKEN, SESSION_TOKEN, ...), and asks the AWS default chain when the key pair is incomplete. A request that named its own endpoint with only an access key id, or a key pair while the engine's environment held a session token, would have sent the engine's credentials there. Such a request now needs a full key pair, and its own session token whenever the environment would supply one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: accept --namespace and --filer-signing-key, ignored Both flags shipped in 4.52 and are documented, so dropping them made an existing command line exit at startup with an unexpected-argument error. They are kept hidden and ignored, with a warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors design: the engine is started without a catalog Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: the trust check takes the environment it reads The options test read the process environment through signs_alone, so a host exporting AWS_SESSION_TOKEN or TOKEN failed it. The caller now passes the variable names in; describe passes the engine's own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: log both endpoints when a request's options are refused The gateway turns the PermissionDenied into a bare AccessDenied and logs nothing, so a pinned aws_endpoint written in another form than the one the gateway sends (host case, a default port, an IP for a name) failed every call with no trace of which value differed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: the Lance catalog sets the credential role whether or not Iceberg runs -iceberg.credentialRole reached the S3 server only while the Iceberg listener started, so with -port.iceberg=0 the Lance catalog installed its vendor but vending stayed off, and the vector engine got no credentials for the tables the gateway resolves. Both catalogs now set the role through one helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: set the credential role once, before either catalog serves Both catalogs wrote the role as they were built; the second write landed while the first was already serving requests that read it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
86db60b669 |
s3api: mint a versioned PUT's id at commit, after the lock wait and the upload (#12454)
* s3api: mint a versioned PUT's id after the object write lock putVersionedObject minted the version id before the upload and the lock wait. A DELETE that committed while the PUT waited got a newer id, so a conditional PUT whose If-None-Match passed against the delete marker was stored behind it: accepted, but GET returned NoSuchKey. The finalize now re-mints the id once the lock is held and the precondition passed, so the version orders after everything the check saw. An ambiguous routed attempt keeps its id, since recovery looks for that entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: mint a routed versioned PUT's id after the upload On the routed path the version id was still the one minted before the upload, and RECOMPUTE_LATEST picks the newest name. A DELETE that committed during a long upload left its marker latest over the accepted PUT. Re-mint right before the routed transaction, as the lock path does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
2be4498c04 |
s3 lifecycle: walker deletes carry the listed identity and must target the latest version (#12452)
* s3 lifecycle: send the listed entry's identity on walker deletes The daily walker dispatched LifecycleDelete with no ExpectedIdentity, so the server skipped its compare-and-swap and an object rewritten between listing and dispatch was deleted. The list adapter now records each listed entry's identity, computed the same way as on the event path, and the walker sends it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: current-version expiration must target the latest version The walker addresses the latest version of a versioned key by id. A version file never changes, so its identity still matches after a newer PUT, and the delete went ahead: a delete marker over the new version with versioning enabled, or the new null version deleted while suspended. LifecycleDelete now resolves the key's latest version for EXPIRATION_DAYS/EXPIRATION_DATE requests that carry a version id and answers NOOP_RESOLVED (VERSION_NOT_LATEST) when it is a different one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: check the latest version for unaddressed suspended expirations With versioning suspended, an expiration without a version id reads the plain entry, which is the null version but not necessarily the latest: a newer real version can sit in .versions. The latest-version check now runs in the expiration branch, by versioning state: for every suspended request (an unaddressed one addresses the null version) and for enabled requests that carry a version id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test that each version sibling carries its own identity Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f71c0fe523 |
fix(s3tables): route REST RenameTable and refuse unrouted S3 Tables requests (#12446)
* s3tables: route REST RenameTable and refuse unrouted S3 Tables requests
PUT /tables/<arn>/<ns>/<name>/rename had no route, so it fell through to the
/{bucket} routers and ran as a PutObject into a bucket named "tables",
answering 2xx while the table kept its name.
- Answer any s3tables-signed request no S3 Tables route takes with 404
UnknownOperationException, before the health, domain and bucket routers.
- Route the AWS RenameTable REST call (newNamespaceName, newName, both
optional; versionToken).
- Honor versionToken on RenameTable, checked against the snapshot the claim
asserts so a commit between the two reads cannot slip a stale token through.
* s3tables: match the credential scope's service, not a region named s3tables
The unrouted-request fallback widens what a false match costs: an S3
client whose region is named s3tables had every request answered as an
unknown S3 Tables operation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: refuse an empty newName or newNamespaceName on REST rename
An empty field was read as omitted, so {"newNamespaceName":"other",
"newName":""} moved the table under its old name instead of failing
validation the way AWS does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: check the rename version token after the snapshot comparison
Once the separately read metadata is known to equal the claimed snapshot,
the token compares against the parsed metadata; no second decode of the
snapshot bytes is needed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables test: cover policy and tag changes in the rename snapshot compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables test: give each REST rename subtest its own source table
The token subtests renamed the table the first subtest produced, so run
alone they failed on a missing table instead of testing the token.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Chris LuandClaude Opus 5.5
|
7b3983f154 |
s3api: speed up recursive listing of branched directory trees (#12438)
* s3api: look up a trailing-slash prefix's directory instead of listing its parent A prefix such as "account-0001/" names one directory, but the walk found it by listing the parent with the name as a prefix. SQL stores answer that with a LIKE that bounds the scan from below only, so in a wide bucket root the query reads every sibling after the name: seconds for a prefix holding a few keys. Look the directory up by name and serve it as the level's only entry. A miss, an error or a non-directory falls back to the prefix listing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: track failover receives atomically The failover wrapper flags a received reply through a plain bool, which a callback reading from the filer on several goroutines would race on. Use an atomic.Bool so the callback may share its filer client. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: read directories ahead of a recursive listing walk A recursive ListObjects walk enters directories one at a time, so a bucket made of many directories holding one or two keys each costs one filer round trip per directory, in series: seconds per 1000-key page on a store whose reads wait on disk. Read the first window of the directories the walk will enter on a few workers, nearest first in walk order, and hand each window over when the walk gets there. Every window the walk reads itself is drained first so its subdirectories are queued before the walk descends into the first one. The number of windows held ahead grows while directories end inside their first window and falls back when one does not, so a page ending inside a large directory leaves few reads wasted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: drop the prefetcher's record of every directory it was handed A page can cross any number of empty directories without spending its key budget, so a set of every announced directory grows with the tree. A directory is already handled when it is at or behind the walk, held or in flight, or queued; check those, which the queue and window caps bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: end a trailing-slash prefix listing at the file its lookup finds A file named by the prefix lists nothing under it, the same as the sibling listing it fell back to, which threw the lookup's answer away and paid the parent scan anyway. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: read a walk's own listing ahead instead of draining it Draining the window before the walk took its first entry made a page wait for the end of the listing, which a SQL store answering a name prefix reaches only after scanning the rest of the directory; a page that fills before then used to return at once. A background reader now hands the window's directories to the prefetcher as they arrive while the walk takes each entry, holding at most 1 MiB of entries ahead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: bound the bytes a prefetched listing window holds Entries with long chunk lists or inline content made the up to 64 held windows of 32 entries an unbounded amount of memory per listing. A window now stops at 128 KiB of entries and the walk lists the rest of that directory from the window's last entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test recursive listings whose streams break partway Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3e64326f2b |
s3: require their own actions to change a bucket's CORS or versioning (#12433)
* s3: require their own actions to change a bucket's CORS or versioning PUT/DELETE ?cors and PUT ?versioning were authorized as Write, so an identity granted only object writes on a bucket could replace its CORS rules or suspend versioning. They now need PutBucketCors, DeleteBucketCors and PutBucketVersioning (or Admin), and inline policies granting s3:PutBucketCors, s3:DeleteBucketCors or s3:PutBucketVersioning map to those actions instead of Write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test that object Write cannot change bucket CORS or versioning Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: offer the bucket CORS and versioning permissions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test each bucket config action with its own identity Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3a579ece3b |
s3: commit the delete marker in the owner's finalize transaction (#12432)
* s3: commit the delete marker in the owner's finalize transaction createDeleteMarker wrote the marker with mkFile to the gateway's current filer and then sent RECOMPUTE_LATEST to the object's ring owner. When the two differ, the owner scans .versions before peer metadata sync delivers the marker, re-derives the pointer to the previous version, and the marker stays non-current on every filer. routedVersionedFinalize now takes an optional version entry and PUTs it in the same ObjectTransaction ahead of the recompute, the way PutObject and CompleteMultipartUpload already do. The copy destination keeps passing nil: its version file is already written on the owner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test a versioned delete when the gateway's filer is not the owner Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: assert the delete marker and the pointer flip share one transaction Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a8766f5177 |
s3api: apply copy bucket-default SSE after the ACL re-auth (#12436)
* s3api: apply copy bucket-default SSE after the ACL re-auth CopyObject without its own x-amz-server-side-encryption header takes the destination bucket's default encryption, which CopyObjectHandler writes into the request headers. It did so before preparePutObjectACL, which re-authenticates a request that carries an ACL (x-amz-acl or a grant header) for s3:PutObjectAcl. verifySignedHeadersCoverage then found the synthesized x-amz-server-side-encryption* headers outside SignedHeaders and rejected the copy with SignatureDoesNotMatch. Move the bucket-default injection after the ACL check, so the re-authentication sees the request as the client signed it. The destination object still gets the bucket default encryption. * s3api: assert the copy's KMS key and stored ciphertext Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Stefan Scharlott <stefans@Stefans-MacBook-Pro.local> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
aab947b65e |
s3api: answer success when a failed completion already committed (#12416)
* s3api: name the check that a completion already committed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: answer success when a failed completion already committed A finalize can fail after its commit: the filer stored the object and the call still reported an error. The client then got a 500 with an unknown outcome. When the failure may have left this attempt's object behind, check the key the way a retry would and answer success with the committed result when it holds this upload's object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: keep part data when the commit found is another attempt's The object carrying this upload ID can be a concurrent attempt's commit of the same upload, and its part selection can include entries this attempt counts as unused. Free those parts' data only when the committed chunks are this attempt's own; otherwise drop their metadata only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test the committed-completion read on versioned buckets Cover the latest-version read (version id and checksum in the answer), a suspended bucket whose delete marker is still current, and that the upload directory is removed without its data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test versioned recovery through the completion handler Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3cbace2471 |
s3: expose ReadAcp/WriteAcp in shell and admin UI, log ACL-upload denials (#12396)
* shell: accept ReadAcp and WriteAcp in per-bucket and service account actions s3.bucket.access, s3.anonymous.set and s3.serviceaccount.create rejected ReadAcp and WriteAcp, so the ACL actions the S3 gateway enforces could not be granted through them. Share one canonical action table across the three commands and list the ACL actions in their help. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: list ReadAcp and WriteAcp in s3.configure and s3.circuitBreaker help The -actions usage of both commands stopped at Admin, so the ACL actions the gateway enforces could only be found in the source. s3.circuitBreaker now prints the actions it actually accepts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: offer ReadAcp and WriteAcp in the object store user permissions The create and edit user forms only offered Read, Write, List, Tagging and Admin among the bucket permissions, so an identity could not be given the ACL actions from the UI even though uploads with x-amz-acl need WriteAcp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: log the WriteAcp denial of an upload that sets an ACL An upload or copy that carries x-amz-acl or an x-amz-grant-* header also needs WriteAcp. When that check denies, the client only sees a bare AccessDenied and nothing reached the log at the default verbosity, so the header was never suspected. Log the identity and the missing action. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: quote the key in the ACL upload denial log An object key can carry a decoded newline, which let a caller add forged lines to the default log. Quote the bucket and key, and word the message so it also fits a policy deny, not only a missing grant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: say ACL uploads need both Write and WriteAcp The s3.configure example granted WriteAcp alone, which still fails the upload's Write check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: parse S3 action lists with one shared helper s3.bucket.access, s3.anonymous.set and s3.serviceaccount.create each had their own validation loop, so a test of the table did not cover the parsing the commands run. They now call canonicalActionList, which skips empty entries and repeats. s3.anonymous.set validates before contacting IAM, and s3.bucket.access still rejects a list with no actions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep bucket-scoped actions the user form cannot show When a user has both global and bucket-scoped actions, the edit form shows only the global ones, and saving replaced the identity's actions with what the form rebuilt, deleting the scoped grants. Carry those actions through the save. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: list hidden bucket grants in the user edit form Restoring the bucket-scoped grants the form cannot show kept them active even when an admin narrowed the user's permissions, with no way to see or drop them. Show them as checked boxes under Other Bucket Permissions; unchecking one removes it on save. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: reject an s3.anonymous.set -access list with no actions A blank list such as ",," parsed to nothing and cleared the bucket's anonymous grants. Use "none" for that, as s3.bucket.access does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: skip the ACL upload denial log for anonymous requests Anonymous uploads with ACL headers can be sent by anyone, so logging each denial at the default level lets unauthenticated clients flood the log. Keep the line for named identities. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: count kept bucket grants when checking an edited user for permissions The empty check ran before the checked Other Bucket Permissions were merged, so a user left with only those grants could not be saved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: list bucket grants of Admin users in the user edit form parseBucketPermissions returned before reading scoped actions when the user had Admin, so removing Admin and saving dropped grants such as WriteAcp:bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep unpaired bucket grants as separate checkboxes in the user edit form The form saves every selected permission on every selected bucket, so grants such as ReadAcp:a with WriteAcp:b came back as all four pairs. Show the grants through the picker only when they cover that full product; otherwise list each grant on its own. Opening the form also resets the bucket scope left over from the previous user. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep grants the edit pickers have no option for A permission x bucket product was shown through the pickers even when a bucket was not in the bucket list (deleted, prefix or wildcard grants) or a permission had no picker option (DeleteBucket, PutBucketQuota). Saving then dropped those grants. Such grants now go to the checkbox list, global ones included, and are submitted unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: name the s3.bucket.access anonymous example by what it grants Read,Write,List leaves out ReadAcp and WriteAcp, so it is not full access. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep unlisted global grants of Admin users in the user edit form parseBucketPermissions returned for Admin before splitting global grants into picker options and the rest, and only raw option values were re-selected. Saving an Admin user dropped s3tables:* and grants with no picker option such as DeleteBucket. Admin users now use the same split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep the /* suffix of bucket grants in the user edit form parseBucketPermissions dropped a trailing /* before matching bucket options, so saving rewrote ReadAcp:bucket/* as ReadAcp:bucket. The bare form also matches bucket-level requests such as GET ?acl, so the save widened the grant. A /* bucket is not a picker option, so the grant now stays in the checkbox list as written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: test that s3.bucket.access saves ReadAcp and WriteAcp grants Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
c437791991 |
s3: reject POST Object header fields with control characters (#12400)
A Content-Type form field such as "\n" was copied verbatim into the multipart part header sent to the volume server, which broke the framing and surfaced as a Content-MD5 mismatch and a 500. Form fields promoted to request headers must be valid header values, as on a PUT; otherwise answer 400 MalformedPOSTRequest. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0742d316aa |
fix(s3): accept fresh admin tokens when context carries stale authorization (#12390) (#12397)
* fix(s3): accept fresh admin tokens when context carries stale authorization (#12390) Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com> * security: share the admin Bearer check between S3 and admin-plane peers Move the authorization scan into security.CheckGrpcAdminToken, built on IsFilerAdminToken, and call it from CheckGrpcAdminPeer and the S3 IAM cache check. Both now accept a valid admin token in any authorization value instead of S3 re-implementing the decode and claim checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: accept a fresh admin token after a stale one on IAM gRPC The S3 IAM cache check is meant to mirror the filer's, so route the filer's checkAdminAuth through security.CheckGrpcAdminToken too. IamGrpcStore appends its token to the caller's context, so a forwarded older token would otherwise still shadow it here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * security: parse at most four authorization values per admin call Scanning every value let one request repeat a junk Bearer token (cheap over HPACK) tens of thousands of times and burn about half a second of JWT parsing on an unauthenticated port. Real callers send their own token plus at most one forwarded from their context. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * credential: match the other WithS3InternalAdminAuth replace fix The plugin worker fix makes the same set-instead-of-append change to WithS3InternalAdminAuth with different code and a different test at the same spot, so whichever lands second would conflict. Use the identical code and test so the two merge cleanly in either order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * security: test an expired admin token ahead of a fresh one Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * security: test that a same-key session token is not an admin token Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test IAM gRPC with only invalid authorization values Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
15ff5c113a |
fix(s3tables): validate policy conditions and fail safe on unusable ones (#12389)
* s3tables: validate policy conditions and fail safe on unusable ones * s3vectors: reject a condition operator with no keys The shared evaluator now treats a keyless operator as unusable, so a stored keyless Allow would never grant while the write succeeded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: reject an Allow whose Condition is null A null Condition decodes as no Condition, so the Allow granted unconditionally. A null Condition on a Deny only widens the veto and stays accepted, as in the S3 Tables validator. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: treat a NaN numeric condition value as unreadable strconv.ParseFloat accepts "NaN", but no comparison with it holds, so a Deny on NumericLessThan never applied and an Allow on NumericNotEquals matched every request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: reject condition keys the evaluator never supplies A condition on a key such as aws:SourceIp reads no request value, so a Deny on it never applied and an Allow never granted. Accept only the keys getConditionContextValues resolves, as the vector validator does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: require Statement and Effect spelled as written The evaluator decodes field names regardless of case, so an "effect" or "statement" key slipped past the null Condition check and the other raw-key checks while the stored Allow still granted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: treat an infinite numeric condition value as unreadable Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: a stored Allow with a null Condition never grants The typed decode reads "Condition": null as no Condition, so an Allow stored before validation refused it still granted unconditionally, for table and vector policies alike. Record the literal null while decoding a Statement and skip such an Allow during evaluation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: validate a null Condition from the decoded Statement validatePolicyDocument re-parsed the document to find a literal null Condition; the Statement decode now records it, so the validator reads the same flag the evaluator does and nullConditionStatements goes away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: keep Condition objects merged across key spellings Decoding Condition through a json.RawMessage kept only the last of "Condition" and "condition", so an empty object under one spelling erased the other's restriction. Decode the statement as before and probe for the literal null separately. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: refuse a Condition split across key spellings Condition objects under differently cased keys decode into one map, and an operator they share keeps only the last spelling's keys, so a restriction under the other spelling was dropped. Validation now refuses a statement whose Condition is under more than one key, and one stored before that never grants on an Allow and vetoes on a Deny. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: refuse a Condition key repeated under the same spelling PolicyDocument re-marshaled each statement through interface{}, and Statement.UnmarshalJSON counted keys from a map, so both kept only the last of a repeated Condition key. "Condition":null,"Condition":{} or two Condition objects sharing an operator passed validation with a restriction dropped. Statements now decode from their original bytes and the keys are counted while walking the object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: refuse a repeated Condition key in a vector bucket policy The validator read Condition from a map, which keeps only the last of a repeated key, so it accepted a statement the evaluator never grants on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: keep the re-marshaled statement decode, count Condition keys apart Decoding statements from their original bytes let document order pick between spellings of every field, where the re-marshal sorted the keys first, so a stored "effect":"Deny","Effect":"Allow" became an Allow. Statements decode as before again; only the Condition keys are counted from the original bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: test that validation refuses a split Condition on a Deny Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: test that validation refuses a repeated Condition on a Deny Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a0f30afe5f |
s3: keep service account keys when a pushed identity replaces its parent (#12394)
* s3: keep service account keys when a pushed identity replaces its parent Service account changes push the parent user, whose proto carries only the user's own keys. The merge replaced the cached parent with that proto and dropped every service account key until a restart. Read the parent's service accounts from the credential store so the push rebuilds them, deleted and disabled ones included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test that a parent identity push keeps its service account keys Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test a sibling service account survives a deletion push Also check the restored secret and status, not just the key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b0c6297041 |
s3api: per-bucket Block Public Access (BlockPublicAcls, IgnorePublicAcls) (#12381)
* s3api: store and serve a bucket's PublicAccessBlock configuration PutPublicAccessBlock, GetPublicAccessBlock and DeletePublicAccessBlock answered NotImplemented. Store the configuration in the bucket entry's extended attributes next to the ownership controls, carry it in the cached BucketConfig, and answer Get on an unset bucket with NoSuchPublicAccessBlockConfiguration like S3 does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: refuse public ACLs when a bucket blocks them With BlockPublicAcls set, a PutObject, CopyObject, CreateMultipartUpload, POST Object, PutObjectAcl or PutBucketAcl whose canned ACL or grants name the AllUsers or AuthenticatedUsers group is refused with AccessDenied. Private and account grants keep working, so clients that send x-amz-acl: private on every upload are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: ignore public ACL grants for anonymous reads under IgnorePublicAcls A bucket with IgnorePublicAcls no longer lets an AllUsers grant on the bucket or on an object answer an anonymous read or listing. Bucket policies and identity permissions are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: reject PublicAccessBlock bodies with unknown elements A misspelled element such as BlockPublicAcl was ignored, so the request succeeded and stored the flag as false. Answer MalformedXML instead, and store the configuration without the request's namespace. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: fail closed when a bucket's public access block cannot be read A stored PublicAccessBlock that does not parse now blocks and ignores public ACLs instead of acting as if none were set. PutBucketAcl and POST Object return the bucket lookup error rather than skipping the BlockPublicAcls check when the bucket config cannot be loaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test BlockPublicAcls on copy, multipart initiate and a PUT-then-GET Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: keep stored public access block bytes on unrelated bucket updates An unreadable stored configuration was replaced with the in-memory all-true fallback whenever another bucket setting was updated, losing the original value. Write the key only when the update changes the public access block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
af45ab8f2a |
s3api: known-answer fixture generator for SSE-S3 and filer-encrypted chunks (#12355)
* s3api: a known-answer fixture generator for SSE-S3 and filer-encrypted chunks TestGenerateDirectReadFixture writes, from this package's own encryption, the pieces a reader outside the gateway must handle: SSE-S3 chunks with their per-chunk metadata, the object-level key, a single-stream ciphertext, the KEK derived from s3.sse.key and wrapped under a passphrase in both on-disk formats, and chunks encrypted by the filer, plain and gzipped. It is skipped unless DIRECT_READ_FIXTURE names the output file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3api: the fixture generator restores the key manager and checks the write The test set the global key manager's KEK and left it; a later test in the same run then found a signing key where it expects none. The previous key is restored with t.Cleanup. The fixture file's close error is checked too, so a delayed write failure fails the test instead of leaving a short fixture. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3api: the fixture generator unwraps both KEK formats before writing The legacy v1 wrap is built by hand in the test; reading v1 and v2 back through unwrapKEK ties both to the format the key manager accepts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: pin a legacy v1 wrapped KEK made outside the package Generating the v1 sample through legacyKEKWrappingSalt let a change to the salt pass on both sides; a vector wrapped elsewhere under the historical salt must still unwrap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
199b2e2dd9 |
fix(s3api): reject an invalid storage class before uploading the body (#12354)
* fix(s3api): reject an invalid storage class before uploading the body * fix(s3api): reject an invalid storage class on CreateMultipartUpload * fix(s3api): validate storage class and tags before reading the PUT body * fix(s3api): validate storage class and tags with the other PUT headers Checking them next to ValidateRequestEncryption keeps the check ahead of any later hook that may answer the PUT without reaching putToFiler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(s3api): reuse validatePutHeaders in putToFiler Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(s3api): store storage class and tags on directory-marker PUTs The marker path validated x-amz-storage-class and x-amz-tagging but its mkdir callback never wrote them, so a valid value was accepted and then dropped and GetObjectTagging on the marker came back empty. Store both the way putToFiler does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(s3api): read directory-marker tags back through GetObjectTagging Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
8125a1549a |
s3vectors: support tag conditions in vector bucket policies (#12351)
* s3tables: resolve s3vectors:VectorBucketTag like s3tables:TableBucketTag Both name a tag on the bucket that holds the target, whichever resource the request acts on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: give vector policy conditions the bucket, index and request tags A vector bucket policy is evaluated with these condition keys: - aws:ResourceTag/<key>: the tags of the bucket, or of the index when the request names one. - s3vectors:VectorBucketTag/<key>: the bucket's tags, for bucket and index requests alike. - aws:RequestTag/<key> and aws:TagKeys: the tags TagResource and CreateIndex set, and the keys UntagResource removes. An index's tags are read only when the policy has an aws:ResourceTag condition. The bucket's come from the lookup that already reads its policy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: accept tag conditions in vector bucket policies PutVectorBucketPolicy rejected every Condition, since no condition key was supplied. A condition is now accepted when its operator is one the evaluator knows and each key is aws:ResourceTag/<key>, aws:RequestTag/<key>, aws:TagKeys or s3vectors:VectorBucketTag/<key>. Any other key would never match, so it is still rejected, as is a condition field not spelled Condition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: cover tag conditions in vector bucket policies Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: reject condition values the evaluator cannot compare A null condition value was accepted and then panicked the request that matched its statement, a Deny reaching even the bucket owner. A value must now be a string, number or boolean, or a non-empty list of them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a074538c8f |
admin: manage S3 vector buckets (#12334)
* s3vectors: run an operation for trusted local tooling Like s3tables.Manager.Execute, Server.Execute runs one S3 Vectors operation through its handler as a given identity, without a signed request. The admin console can then manage vector buckets through the same validation, authorization and catalog steps as the API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * admin: list, inspect, create and delete vector buckets over the API /api/s3vectors runs the S3 Vectors operations through Server.Execute as the admin account: list vector buckets with their encryption, read one bucket with its policy and each index's data type, dimension, distance metric and non-filterable keys, create and delete a bucket, and delete an index. Going through the operations keeps the API's rules, such as ICEBERG buckets not being vector buckets and the implicit index namespace not blocking a delete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: export the rule for which table buckets are vector buckets IsVectorBucketFormat names the check ListVectorBuckets and isVectorBucket already make, so the admin console can tell the same buckets apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * admin: vector bucket pages Object Store > Vector Buckets lists the vector buckets with their ARN, encryption and creation time, creates and deletes them, and opens one bucket with its indexes (data type, dimension, distance metric, non-filterable keys) and its policy, where an index can be deleted. A LANCE row on the Table Buckets page links to the same bucket's vector page, since both APIs serve it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * admin: stop the vector bucket and index lists at the list limit A page cut short by filtered ICEBERG buckets let the next page carry the list past s3TablesAdminListLimit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: say when a vector bucket or index list is cut at the limit The lists stop at s3TablesAdminListLimit while more remain, so mark them truncated and point to an S3 Vectors client for the rest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: indent the vector bucket policy A policy stored as one line ran off the right edge of the details page. Policies that are not valid JSON are shown as stored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: show a deeply nested vector bucket policy as stored Indenting multiplies the size of a deeply nested document, so a policy past 32 levels of nesting is shown without indenting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
ec968aa016 |
s3vectors: support TagResource, UntagResource and ListTagsForResource (#12335)
* s3vectors: tag vector buckets and indexes
TagResource, UntagResource and ListTagsForResource answer on
/tags/{resourceArn} (POST, DELETE, GET) and by name. Each authorizes its
own s3vectors action on the bucket or index ARN, then reads or writes the
catalog record's tags. A resource holds at most 50 tags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: apply the tags passed to CreateVectorBucket and CreateIndex
The tags field was accepted and dropped. It is now stored on the new
bucket or index, and creating with tags needs s3vectors:TagResource in
addition to the create action.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: cover resource tagging end to end
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: keep tag operations to a bound table bucket
TagResource, UntagResource and ListTagsForResource run through Execute
with WithTableBucketCreatedAt now answer not found when the bucket name
holds another bucket, as the bound creates and deletes already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bind tagging to the authorized bucket and to vector indexes
The catalog tag calls carry the creation time of the bucket authorizeVector
approved, so a bucket created again under the name is not touched. An index
ARN must name a Lance table: another table in the vector namespace of a
format-less bucket is refused, as DeleteIndex does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: match a TagResource grant scoped to the bucket being created
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: merge tag writes into the tags the record holds at the write
TagResource and UntagResource merged into the tags they read before the
compare-and-swap and wrote that map on every attempt, so a tag write that
landed in between was undone. The merge now runs inside the mutation.
WithMaxTags lets an in-process caller cap the merged set there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: check the 50-tag limit inside the catalog's tag write
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: check a tag operation's bound bucket after its reads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: validate each tag key UntagResource removes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: validate untag keys one at a time
UntagResource sized a map by the number of submitted tag keys before
validating them, so a large body of repeated keys allocated far more than
the request itself, before authorization ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: authorize UntagResource before validating its keys
Validating a near-32 MiB body of tag keys costs about 13x its JSON
decode, so an unauthorized caller could make the gateway spend that
work on a request it then refuses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Chris LuandClaude Opus 5.5
|
7298b4ed37 |
fix(s3tables,s3vectors): bind DeleteNamespace and the vector bucket cleanup to the authorized records (#12331)
* s3tables: bind DeleteNamespace to the authorized namespace The delete claimed whatever finished namespace held the name at claim time, so one deleted and created again by another owner after the authorization read was removed under the first one's authorization. The claim now requires the creation time that was authorized. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: bind DeleteVectorBucket's namespace cleanup to the authorized bucket After the bound DeleteTableBucket reported the bucket not empty, the index namespace cleanup ran unbound, so a bucket deleted and created again in between lost its empty index namespace under the first bucket's authorization. DeleteNamespace now honors the bucket binding, checking the bucket after reading the namespace, and the cleanup carries it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
66a6598fdd |
fix(s3tables,s3vectors): bind table and vector bucket writes to the bucket they were authorized against (#12329)
* s3tables: bind table bucket policy writes to the authorized bucket PutTableBucketPolicy compared only the policy bytes, and DeleteTableBucketPolicy removed the policy unconditionally. A bucket deleted and created again under the same name between authorization and the write took a policy change nobody authorized for it. Both writes now require the bucket's CreatedAt and the policy they were authorized against, answer 409 otherwise and 404 for a bucket that is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: bind bucket tag writes to the authorized bucket TagResource and UntagResource checked the creation time only for a table ARN. On a bucket ARN the merged tags landed on a bucket deleted and created again since authorization. extractResourceOwnerAndBucket now returns the resource's CreatedAt for both kinds, and the write requires it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: bind bucket maintenance writes to the authorized bucket PutTableBucketMaintenanceConfiguration wrote onto whatever bucket held the name at write time. authorizeMaintenanceBucket now returns the bucket's CreatedAt like authorizeMaintenanceTable, and putMaintenanceConfiguration takes the check and creation time from both callers instead of an optional closure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: bind vector bucket policy writes to the authorized bucket Put/DeleteVectorBucketPolicy authorized inside the compare-and-swap with the owner from an earlier bucket read, so a bucket deleted and created again by another account in between had its policy rewritten by the old owner. The mutation now requires the entry's createdAt to match the record it authorized against. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: bind DeleteVectorBucket to the authorized bucket DeleteVectorBucket authorized one bucket record, then ran the trusted catalog DeleteTableBucket on whatever bucket held the name by then, so an empty bucket deleted and created again by another account in between was deleted by the old owner. The catalog delete now accepts the authorized bucket's createdAt through an in-process context value and treats any other bucket as gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: bind CreateIndex and DeleteIndex to the authorized bucket Both authorized the bucket, then ran trusted CreateNamespace, CreateTable or DeleteTable on whatever bucket held the name by then, so an index could be created in, or deleted from, a bucket another account created again in between. Those catalog operations now honor the bucket binding DeleteTableBucket already takes, and the vector handlers pass the authorized bucket's createdAt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4850def831 |
fix(s3tables): bind table and view writes to the record they were authorized against (#12328)
* s3tables: bind table and view writes to the record they were authorized against * s3tables: bind UpdateNamespace to the namespace it was authorized against UpdateNamespace writes back the whole record it read, owner and creation time included. A namespace deleted and created again before the write took the first one's owner and CreatedAt, which also undoes the CreatedAt binding creates now rely on. Refuse the write with 409 when the namespace at the path has another CreatedAt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
ce292a4e5e |
s3: require Admin to put or delete a bucket's lifecycle configuration (#12327)
PUT and DELETE ?lifecycle were authorized as Write, so an identity granted only object writes on a bucket could add expiration rules or remove the ones the bucket administrator set. They now need Admin on the bucket, like the encryption, public access block and ownership controls writes. Identities authorized by IAM policies are unaffected: they are still checked against s3:PutLifecycleConfiguration. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d431c1b35a |
s3api: let the vectors route, not the credential scope, raise the body limit (#12326)
SigV4 verification picked the 32 MiB S3 Vectors body limit from the credential scope, which is not verified until the body is hashed. A request claiming the s3vectors scope on any other route could make the gateway buffer 32 MiB before authentication instead of 10 MiB. The vectors handler now sets the limit on the request context, and every other route keeps 10 MiB. Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
02d22800aa |
s3vectors: accept bodies up to the 32 MiB limit, report larger ones as a size error, log engine failures (#12320)
* s3api: hash an s3vectors body up to the Vectors API limit SigV4 verification of a non-S3 request hashed at most 10 MiB of the body and silently dropped the rest, so any S3 Vectors request over 10 MiB failed its signature check with 403, although the API accepts 32 MiB. The limit now follows the signing service, and a body over it is reported as ErrEntityTooLarge instead of being truncated; the embedded IAM answers that with InvalidInput, as it already does for a body over its own limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: answer an oversized request with a ValidationException A body over 32 MiB was refused as "403 AccessDenied: authentication failed". A declared Content-Length over the limit is now refused before authentication, and a streamed body over it, whether the signature check or the handler reads it, gets the same "request body exceeds 32 MiB" ValidationException. The limit is the one SigV4 verification uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: log engine and catalog failures at the default level An engine error reached the client as a bare "internal error", and the cause was logged only with -v=1, so a misconfigured engine left nothing in a default log. Engine and catalog failures are now logged as errors, and a reported unavailability as a warning, with the operation, bucket and index. The response still leaves the engine's message out, since it can name internal hosts and paths, and points at the gateway log instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3api: hash the signed body through readRequestBody streamHashRequestBody repeated readRequestBody's read-past-the-limit check; it now reads the body with readRequestBody and hashes the bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
2eb219df61 |
s3vectors: encrypt the objects of a vector bucket declared AES256 (#12324)
* s3api: encrypt the objects of a vector bucket declared AES256 CreateVectorBucket recorded encryptionConfiguration sseType AES256 and returned it, but nothing on the write path read it: the index's data files, manifests and transactions were stored without SSE. A LANCE table bucket's record now supplies the bucket's default encryption whenever the bucket has none of its own, so PutObject, CreateMultipartUpload and CopyObject apply SSE-S3 to every object the engine writes. The S3 bucket metadata itself is unchanged. ICEBERG table buckets keep recording the declaration only: their catalog reads the manifests clients upload straight from the filer, which would see ciphertext. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3tables: refuse an AES256 vector bucket on a gateway without an SSE-S3 key A vector bucket's AES256 is now applied to every index write, and SSE-S3 fails each write when the gateway has no KEK. Creating the bucket there succeeded and the first CreateIndex came back as InternalFailure. CreateTableBucket now refuses a LANCE bucket declaring AES256 when the gateway reports no SSE-S3 key, naming s3.sse.kek and s3.sse.key; CreateVectorBucket returns it as ValidationException. ICEBERG buckets, whose declaration is only recorded, and tooling that sets no checker are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3api: refuse DeleteBucketEncryption on a bucket's declared encryption GetBucketEncryption reports a vector bucket's declared AES256, but DeleteBucketEncryption answered NoSuchBucketEncryptionConfiguration and the declaration kept applying. The declaration is fixed when the bucket is created, so the delete now returns MethodNotAllowed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): keep an inherited WEED_S3_SSE_KEK out of the test cluster The gateway refuses to start SSE-S3 with both s3.sse.kek and s3.sse.key set, so a developer shell exporting WEED_S3_SSE_KEK made the AES256 bucket test fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d72f761d83 |
seaweed-vector: work on its defaults, and without identities or credentials (#12323)
* seaweed-vector: default --namespace to the gateway's Lance port The old default, http://localhost:8888/lance, is a path nothing serves: the Lance Namespace listens on its own port, 9101 by default on weed s3, weed server, weed filer and weed mini. With it, the filer answered the engine's catalog calls with a 500 and every CreateIndex failed with InternalFailure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: default --listen to a port no SeaweedFS server takes 127.0.0.1:8080 is the volume server's default port, so weed-vector next to weed server exited with "Address already in use (os error 98)". 8090 is not among the defaults of master, volume, filer, S3 and its Iceberg and Lance catalogs, IAM, WebDAV, SFTP, the message queue broker and agent, the admin server or weed mini, nor any of their gRPC ports (HTTP + 10000). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance: take any key while the S3 server runs without identities Without identities the S3 port accepts any signature and treats the caller as the admin. The Lance catalog still checked an x-api-key or Bearer token against identities that did not exist, so weed-vector started with --storage-option credentials on such a server got 401 on every catalog call. The catalog now skips both checks when S3 auth is off and leaves the caller to the S3 authenticator, which admits it as the admin, as it already did for a request with no key. With identities configured nothing changes: an unknown key is still 401. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance: tell clients to skip signing while S3 auth is off A client with no credentials of its own could not reach a table's data on a server without identities, though the S3 port takes unsigned requests there: lance fell back to the AWS default credential chain and failed with CredentialsNotLoaded. weed-vector without --storage-option failed every CreateIndex that way. The catalog now adds aws_skip_signature to the storage options it hands out while S3 auth is off, and leaves it out once identities exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-vector: bind the listener before registering with the master The engine spawned its master registration and then bound --listen, so an engine whose port was taken could reach the master before it exited: on weed server with 18080 taken the master logged "vector node 127.0.0.1:18080 added" and removed it in the same millisecond, and the gateway noticed it. Bind first, and name the address in the error instead of "transport error". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * seaweed-worker: keep an unsigned store off the AWS default chain A store the catalog says takes unsigned requests needs no credentials, but lance builds the AWS default credential chain for any store without them, and the chain probes instance metadata on every dataset open. With no credentials of its own, weed-vector spent a second on each query (1030-1066 ms against 32-46 ms with --storage-option ones). The merge of the operator's fallback and the vended options, which the engine and the Lance worker each did in their own copy, moves into table_options, and gives such a store empty credentials. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test(s3vectors): serve an index with no identities and no engine credentials A second cluster runs weed s3 without -config and weed-vector without --storage-option, with no AWS credentials in either environment: weed s3 would turn them into an admin identity, and the engine would find them. On master its CreateIndex fails with InternalFailure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
ff7045a4e5 |
s3vectors: report the Euclidean distance, not its square (#12318)
A euclidean index answered QueryVectors with Lance's L2 distance, which is the squared Euclidean distance: [3, 4] was 25 from the origin, not 5. AWS defines the metric as the straight-line distance between the vectors. The gateway now returns its square root, so the single engine and the enterprise coordinator, which both hand back the raw Lance value, report the same distance. Ordering is unchanged. Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d6c0d728e4 |
s3vectors: resolve the caller like table buckets and honor bucket-scoped identity actions (#12319)
* s3vectors: resolve the caller the way table buckets do The vector checks used the identity name as principal, while the catalog records a bucket's owner through getAccountID: the account id, or the issuer-scoped user id of a federated identity. An identity with an account or a federated identity did not own the vector bucket it created, a Deny naming its account did not bind it, and an Admin identity in the admin account did not get the admin bypass it has on table buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: honor identity actions scoped to a vector bucket An identity action such as s3vectors:QueryVectors:team-a granted nothing, because the vector policy context never named the bucket. Table buckets match <action>:<bucket> through the same context field, so pass the bucket the check is about. Creating a bucket stays unscoped, as for table buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b9bc42c217 |
s3vectors: vend write credentials to index data only to the vector engine and Lance worker (#12316)
* security: share the filer admin token check CheckGrpcAdminPeer verifies an admin Bearer token inline. Move the check into IsFilerAdminToken so an HTTP endpoint can accept the same token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: recognize the cluster's own components by a signed token A request whose X-Seaweedfs-Cluster-Token header carries an admin token signed with jwt.filer_signing.key comes from a component that holds the cluster key: the vector engine or the Lance maintenance worker. Mark such a request as trusted. It still authenticates as an identity, which stays the principal for vended credentials. A dedicated header leaves the Authorization header to Lance OAuth clients. A gateway with no key trusts no one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: vend write credentials to a vector bucket's indexes only to trusted callers An index in a vector bucket's vector namespace is also a Lance table, so a caller granted s3tables:UpdateTable could describe it, or declare it again, and get credentials that write the index data without passing the vector bucket policy. In that namespace both now vend read-only credentials unless the caller is trusted, and a declare from anyone else no longer writes the dataset markers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance worker: prove the cluster key to the Lance namespace The gateway now vends write credentials to a vector bucket's indexes only to callers that send a token signed with jwt.filer_signing.key. The worker already takes that key for the admin stream; its namespace client now mints a token from it for every catalog request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: describe the index the gateway created and prove the cluster key CreateIndex declared a table the gateway had just created, which only worked because declare is idempotent. Describe it instead. The Lance namespace vends write credentials to a vector bucket's indexes only to callers that send a token signed with jwt.filer_signing.key, so take the key with --filer-signing-key, the same flag the Lance worker has. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance worker: keep the cluster key out of Debug output Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: answer a declare of an existing table with its own location CreateTable succeeds for a name that already exists without looking at the location requested, and declare kept using that location for the dataset markers and the vended credentials. Declaring an existing table with another table's location, such as a vector index, got credentials to that table's data. Use the location the catalog holds, which CreateTable now also reports for a new table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: sign cluster tokens with a key derived for the catalog A cluster token was a filer admin token signed with jwt.filer_signing.key and, by default, never expired. Sent to the Lance namespace on every request, often over plain HTTP, a captured one would also authorize the filer's IAM service. Sign it with a key derived from the cluster key for this one use, which the admin services do not accept, and mint it with a five minute lifetime. weed-vector drops its token lifetime flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: follow a reloaded jwt.filer_signing.key The Lance namespace copied the cluster key at startup. A SIGHUP reload refreshes the gateway's filer guard but not that copy, so after a key rotation the restarted vector engine and worker were no longer trusted. Read the key from the guard on every check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: keep a vector bucket's indexes read-only An index in a vector bucket is also a catalog table the Iceberg REST catalog loads, and LoadTable vended credentials that could write it to any caller granted s3tables:UpdateTable. A commit staged its metadata file in the index directory on the same grant. Apply the vector namespace rule to both, after the caller's own UpdateTable check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: confine a created table's location to its namespace before writing Stage-create skips the registration that confines a location to the table's namespace, yet vends write credentials for that location. A caller allowed to create a table in any namespace of a bucket could name another namespace's prefix, a vector index included, and get write credentials to it. The non-staged create also wrote its first metadata file there before the registration refused the location. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: a declare that finds an existing table answers like describe Creating a name that already exists is idempotent, so a caller allowed only to create tables could declare someone else's table and get write credentials to it, and clear its deregistered marker. Describe would have vended that caller read-only credentials. A declare now looks the name up first; when it only finds the table, writing needs UpdateTable, as it does for describe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
06b9d01423 |
s3vectors: match a filter value against the elements of an array (#12321)
* vector: compare json_array_contains elements as typed JSON values Lance's json_array_contains compares each element's text with the value, or with the value wrapped in quotes, so 1 matches "1", true matches "true", and 1.0 misses a stored 1. The filter swap now replaces it like json_extract and json_exists: the value is parsed as JSON once per batch and compared with jsonb's typed equality, and only literal arguments are swapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: match a filter value against the elements of an array A filter never looked inside an array: {"tags": "space"} and {"tags": {"$in": ["space"]}} both missed "tags": ["space", "classic"]. AWS documents that $eq, and so a bare value, matches an array metadata value when any element equals the input. $eq, a bare value and each $in member now also match through json_array_contains; $ne and $nin remain their negations. Range operators, $exists and nested objects are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ab204a03e6 |
fix(s3): verify SigV4 signature with decoded path behind X-Forwarded-Prefix (#12299)
* fix(s3): verify SigV4 signature with decoded path behind X-Forwarded-Prefix When SeaweedFS is hosted behind a reverse proxy that strips a URL prefix (X-Forwarded-Prefix) and forwards the percent-encoded request path, signature verification fails for object keys containing non-ASCII characters. The client signs the canonical URI as the single percent-encoded path including the prefix, but the server verifies against the already-encoded forwarded path joined to the prefix, producing a mismatch (SignatureDoesNotMatch). Decode the forwarded path and rejoin the prefix before verifying, mirroring the existing un-prefixed decoded retry. ASCII keys are unaffected because the decoded path equals the original, so the retry is skipped. Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com> * s3: build the SigV4 candidate paths once and try them per host The forwarded-prefix and plain branches each repeated the decode-and-verify step for every host candidate. List the escaped and decoded paths once, the prefixed forms first, and verify them in the same order as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d6ec6bc4e7 |
fix(s3tables): let DeleteTableBucket remove empty leftover directories (#12307)
* s3tables: let DeleteTableBucket remove empty leftover directories Page through the bucket instead of reading one page, and check each bare directory a deleted namespace left behind for files: empty ones go with the bucket, one holding files refuses with 409 naming it. The delete claims the bucket before scanning, and CreateNamespace refuses a claimed bucket and confirms it again after its write, so a namespace created meanwhile is seen by the scan or turned away instead of deleted with the bucket. The bucket is removed only while it still carries that claim, otherwise 409. A delete that refuses lifts the claim, retrying a failed release, since a claim left behind refuses every create in the bucket. The new namespace stays marked unfinished until that check passes, so no table or view is created in one that gets backed out; a create takes over a namespace still carrying the mark. Both checks accept only the bucket the create was authorized against, by its creation time: one recreated under the same name answers 404 and gets nothing written into it. The removal is not recursive. Each leftover is claimed first, only while it is still the directory the scan saw, and goes only while it holds no file or catalog record; the bucket goes only while nothing is left in it. A file uploaded or a namespace taken back after the scan is refused with 409 instead of deleted. * s3tables: keep directory-backed S3 objects in DeleteTableBucket * s3tables: keep a file that replaced a leftover directory in DeleteTableBucket * s3tables: claim only the authorized bucket in DeleteTableBucket * s3tables: assert the prefix-object mark when claiming a leftover directory * s3tables: hide unfinished namespaces from readers and DeleteNamespace * s3tables: stop CreateNamespace on any failed bucket check * s3tables: retry lifting claims on leftover directories * s3vectors: declare the index namespace again after losing it to a concurrent declare CreateNamespace now writes a namespace marked unfinished and confirms it after checking the bucket. A second CreateIndex that declares the namespace in that window takes the marked record over, and the first one's confirmation answers 409, which failed its CreateIndex. Declare it again, a bounded number of times, when the create answers a conflict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: name an unfinished namespace that keeps a bucket from being deleted A CreateNamespace that dies between its write and its confirmation leaves a namespace marked unfinished. Listings and GetNamespace hide it and DeleteNamespace answers that it does not exist, while DeleteTableBucket counts it, so the bucket could not be deleted and nothing said why. Name it in the BucketNotEmpty answer; creating it again takes it over. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: read a refused empty-directory delete with filer.DeleteEntryError Use the filer's own reading of a non-recursive delete's answer instead of a copy of its message prefix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: release the held create when the concurrent-declare test fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: buffer the held create's result so it can exit after a failure Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: retry lifting a refused namespace delete's claim A claim left on a namespace refuses every create in it, the same harm the bucket's claim does, so its release retries too. With no one-shot caller left, the retrying release takes the releaseDeleteClaim name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: keep the data under a hidden directory on bucket delete The scan filed every hidden entry by name for a recursive delete, so .hidden/data went with the bucket. A hidden directory waits with the leftovers instead: the walk refuses one that holds a file or an object, and removeEmptyTree takes only one that is still bare and empty. A hidden file is removed by name as before. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandClaude Opus 5.5
|
5a60ce04e3 |
vector: stop re-parsing a filter's JSONPath for every scanned row (#12302)
* vector: parse a filter's JSONPath once per batch instead of per row
Lance's json_extract and json_exists parse their JSONPath argument again
for every row they see. Every translated S3 Vectors filter goes through
them, so a flat scan paid about 1 µs per row before the predicate
compared anything, and a 10%-selective {"category": {"$eq": "c3"}} ran
about 50% slower than the same query unfiltered.
The path is always a literal, so the scanner now parses the filter as
before and swaps in copies of the two functions that parse it once per
batch. They call the same jsonb selector and serializer, so results and
errors do not change. Calls whose path is not a literal keep Lance's
functions. The filter SQL is unchanged, so the gateway, existing tables
and the distributed workers (which run query::ann) need no change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: read the field once per extractor in $in and $nin
$in and $nin rendered one IN per member, each with its own json_extract,
so an N-member list parsed the row's metadata N times. Members that share
an extractor (strings and booleans, or numbers) now go into one IN list,
so a list reads the field at most twice. Matching is unchanged: each
group still only sees its own JSON type and still coalesces NULL before
$nin negates it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: rename the $in literal map so codespell passes
Codespell reads "lits" as a misspelling of "list".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Chris LuandClaude Opus 5.5
|
4be191173a |
vector: keep ListVectors pages and segments flat as an index grows; read only keys in GetVectors (#12300)
* vector: scan only the fragments a ListVectors page covers A page pushed its offset into the scan, but the scan still loaded every fragment first, reading each one's deletion file: about 105 S3 GETs per page at 50k rows against 25 at 10k. The page's fragments now come from the live row counts in the manifest, so a page reads only its own. Tokens and row order are unchanged; a manifest without the counts keeps the old scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: list a ListVectors segment in the engine The gateway paged through the whole index for every segment and kept one row in segmentCount, so a segment cost segmentCount engine pages and 16 segments listed slower than one. The engine now takes segment_count and segment_index: it reads keys across the stride, keeps the segment's rows and takes their data and metadata alone, one call per page. A segment is still the rows whose position modulo segmentCount is segmentIndex, and its token still the position to resume from. An engine that predates the fields does not set segmented, and the gateway then pages through it as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: read only the key column across the table in GetVectors The key lookup scanned every column. On S3, lance reads a column during the scan when a row of it is under 1000 bytes, so a 128-dimension vector was read for every row of the table to return 100 of them, and data and metadata were read even when the request did not ask for them. The scan now projects the requested columns and materializes everything but the key late, for the matching rows only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: start the legacy segment fallback from the page already read With an engine that ignores segment_count, the gateway dropped the page it had just read and fetched it again. The fallback now consumes that page first, so a one-row segment page still costs one engine call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: check metadata on segmented ListVectors pages The segment test listed data only. Its rows now carry metadata on two in three keys, and every segmented row, read through take_rows, must equal the same row in a plain listing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: lease the version while a segment takes its data A segmented page with data or metadata reads keys first and then takes the other columns from the same version, so maintenance could reclaim its files in between. That path now holds a lease like GetVectors. A keys-only page is one scan and stays unleased, as plain pages are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: leave GetVectors materialization to lance Forcing every column late made a 100-key GetVectors with data slower at 10k x 128 in the perf cluster: 54.7 ms against 46.3 ms. The take issued a read per row, where lance's heuristic reads a narrow vector column in one pass per fragment. The projection keeps the gain on its own: with neither data nor metadata requested, the call went from 38.6 ms to 25.3 ms and from 150 to 89 S3 GETs. At 50k x 384 the vector is already late under the heuristic, and a full lookup is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
ab355b43b9 |
s3tables: keep the vector-namespace fence on a declare of an existing index (#12293)
s3tables: keep the fence on a declare of an existing index Deciding the existing-index exception before the handler ran left a window: an index deleted between that lookup and the handler's own check would be recreated, unfenced, by a caller without s3vectors:CreateIndex. Drop the up-front exception, so every CreateTable runs fenced and a fresh create is refused at its first write. The engine's declare still passes because the idempotent path writes nothing; for a build whose idempotent path re-applies state already in place, the fence answers an UpdateEntry that leaves the entry exactly as stored without writing. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
e2826c48bd |
s3tables: make the vector namespace of a vector bucket read-only (#12292)
* s3tables: make the vector namespace of a vector bucket read-only S3 Vectors authorizes index operations against the vector bucket policy, but an index is also a catalog table in the bucket's "vector" namespace, and the S3 Tables API only checks s3tables grants. A caller the vector policy denies DeleteIndex could still delete the index, and its data, with DeleteTable. Untrusted writes that name the vector namespace of a LANCE bucket now run behind a fence that passes filer reads and refuses filer writes. The operation's own checks still answer everything short of a write, so a caller without the grant gets its usual denial and learns nothing about the bucket, and a CreateTable that finds the index already there, which is what the vector engine's declare does, still succeeds. A refused write answers 403 AccessDenied. Trusted callers (the gateway, the shell) are unaffected, and so is a namespace named vector in an Iceberg bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: refuse register and deregister in a vector bucket's vector namespace Every catalog write the Lance namespace makes goes through the S3 Tables handler and its vector namespace fence, but register over an existing table at the same location and deregister only add or remove a marker file in the dataset directory. Check the vector namespace before those writes, after the table read that already gates them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: refuse deferred creates in a vector bucket's vector namespace Manager.AuthorizeCreateTable gates the Iceberg create paths that write metadata into the table bucket before the catalog registers the table. Those writes never pass the handler's fence, so the gate applies the same read-only rule once the caller's own create permission is confirmed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: let a declare of an existing index past the vector namespace fence The vector engine declares every index the gateway has just created, and that CreateTable only reaches its idempotent path. Upstream that path writes nothing, so the fence never trips, but a build whose idempotent path repairs table state would refuse the engine. Decide the exception up front instead: a CreateTable naming a finished index in the vector namespace runs unfenced, while a new name or an unfinished create stays fenced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
9a09506be6 |
s3vectors: bound ListVectors parameters and lowercase distanceMetric (#12290)
* s3vectors: bound ListVectors maxResults and segment parameters segmentCount had no upper bound, so segmentCount=1000000, segmentIndex=999999 made the segment emulation scan the whole index to return nothing. Apply the AWS ListVectors windows before authorization and any engine call: maxResults 1..1000, segmentCount 1..16, segmentIndex in [0, segmentCount), and segmentCount and segmentIndex given together. The fields become pointers so an explicit 0 is rejected while an omitted one still takes the default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: store and return distanceMetric in lowercase CreateIndex stored distanceMetric exactly as sent, so an index created with "COSINE" came back as "COSINE" from GetIndex and QueryVectors. AWS returns the lowercase enum values. Lowercase the metric when the index is created, and again when the record is read so indexes already stored in mixed case render the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
21fc46692c |
s3vectors: authorize CreateVectorBucket as a vector action, close DeleteIndex race, fold policy service case (#12287)
* s3vectors: authorize CreateVectorBucket as an s3vectors action CreateVectorBucket went through the catalog's CreateTableBucket check, so an identity granted s3vectors:* was denied the create while s3tables:* was allowed it. Check s3vectors:CreateVectorBucket in the gateway, the same way the catalog checks a bucket create, and create the record on the internal manager under the caller's name so the caller stays the owner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3vectors: delete only the index record DeleteIndex checked DeleteIndex read the record, checked it is a Lance index, then deleted by name. A table that replaced the index in between was deleted too. Send the checked record's version token, so a replaced record fails the delete with a 409 ConflictException. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: reject s3vectors rules in any letter case validatePolicyDocument kept s3vectors actions and resources out of the shared policy slot with case-sensitive checks, so "S3Vectors:GetIndex" was stored as a rule that never takes effect. IAM service prefixes are case-insensitive; fold case in both checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3a8d756bdd |
s3: drop retired volume servers from the gateway location cache (#12281)
* wdclient: publish applied volume-location deletes to registered listeners The master already tells each connected client which volumes a server stopped hosting; only the MasterClient's own vidMap applied it. Expose the same filtered deletions (moves between a server's disks excluded) through OnVolumeLocationDelete so clients keeping a second location cache can drop the retired server too. * wdclient: drop retired volume servers from the FilerClient location cache The S3 gateway's FilerClient keeps its own vidMap, filled by filer lookups, and only forgot a retired volume server after a read against it failed (a ~15s header-timeout stall when the host still resolves). When built with a MasterClient, subscribe to its applied deletions so the cache forgets the server as soon as the master reports it gone. * wdclient: drop retired EC servers from the regular location map too Filer lookups cache EC locations alongside regular ones in vid2Locations, so an EC removal has to clear the retired server there as well as in the EC map for the delete to reach what reads actually use. * wdclient: tombstone pushed deletions against in-flight provider fills A filer lookup already running when a delete lands can return the retired location afterwards and re-cache it. Keep a short tombstone on each deleted (vid, url) pair so fills neither cache nor return it while the lookup source is still settling. * wdclient: clear the FilerClient location cache on master reconnect Removals the master pushed while the gateway link was down never arrive as deletes, so on reconnect the lookup-fed cache is dropped outright rather than kept generation-wise like the stream-fed one, and relearned through filer lookups as before. * wdclient: invoke push hooks outside their registration locks A callback that registers or replaces a hook would wait on the read lock the invocation already holds and stall the master stream. Copy the hook under the lock and call it after the lock is released. * wdclient: report pushed adds so confirmed re-adds clear the suppression A server coming back at the same URL inside the tombstone window stayed hidden from provider fills. updateVidMap now carries the additions it applied through the same hook (renamed OnVolumeLocationChange), and the registered cache untombstones them without eagerly caching them. * wdclient: amortize tombstone expiry over a delete batch A server leaving takes every volume it hosted with it, and scanning all recentDeletes per removed vid squared the batch. Prune lazily on reads and at most once per half TTL on writes. * wdclient: let every location cache subscribe to pushed volume changes The setters were single-slot, so a second FilerClient built on the same MasterClient would have silently replaced the first one's subscription. Registration now returns an unhook func, delivered to all listeners, and FilerClient.Close releases its own. * wdclient: hold the deletion fence until the master re-confirms a location * wdclient: check the deletion fence and the cache add under one lock * wdclient: drop provider answers that crossed a cache reset * wdclient: bound the deletion fence by evicting its oldest mark * wdclient: refresh a repeated removal instead of evicting another mark * wdclient: evict the oldest deletion-fence mark in constant time * wdclient: fence a server only once it serves the volume in no form ec.encode registers the shards before the regular copy is deleted from the same server, and ec.decode mounts the regular copy before the shards come off it. Both arrive add-first in separate heartbeats, so the later removal tombstoned a (vid, url) pair the server still serves and dropped it from the FilerClient map; the filer's answer naming that server was then fenced out and the volume read as not found until the gateway reconnected. Tombstone and report a removal only when neither map still names the url for that volume. * wdclient: serve a provider answer that crossed a cache reset The answer is as fresh as the provider's view; only the cache it would land in has been re-told. Dropping it failed every read whose lookup was in flight during a master reconnect with 'volume not found'. Return it and leave the new view to be filled by the next lookup. * wdclient: test the moved volume whose old server keeps proxying * wdclient: stop caching inside the master lookup provider The provider wrote every answered location through addLocation, which clears the deletion fence: a lookup still naming a server a pushed delete had just retired untombstoned and re-cached it. Fallback lookups already gate caching through fillLocation, and authoritative lookups bypass the cache by contract, so the provider no longer writes the map at all. * wdclient: count only master-pushed locations when suppressing a removal hasLocation could not tell a confirmed regular copy from a lookup fill: the master answers an EC volume with its shard servers, so the regular map holds shard locations that arrived by lookup, and entries an earlier generation learned linger past a reconnect. An EC removal whose url still showed either was suppressed, leaving the retired server served. The vid map now records the generation each regular location was pushed at, drops an unconfirmed regular entry alongside the shard that fed it, and answers hasServingLocation from pushed marks and current-generation entries only. * wdclient: test removals against lookup-cached and stale locations * wdclient: fence removals only in the gateway cache The fence and the pushed-location marks ran inside every MasterClient, including the filer's, whose map is also filled by lookups and cleared by InvalidateCache. A removal reported while the server still served the volume (e.g. a decoded copy whose mark a failed read had wiped) left the pair fenced until the master pushed it again, with reads failing as volume not found meanwhile. Only the S3 gateway's FilerClient now tombstones, from the removals its push-only MasterClient reports. That client decides a removal from the current generation of both its maps after the whole message is applied, so pushedLocations goes away, and the master lookup provider caches as before. The hooks follow the existing single-slot SetOn...Fn pattern and are wired next to SetOnPeerUpdateFn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * wdclient: let a removal fence expire after a minute The master drops a broadcast when a client's channel is full, so the re-add that would lift a fence can be lost while the gateway stays connected. A mark held until that re-add then hid a live server, and a volume whose only copy it held read as not found until the next master reconnect. Bounding marks by a minute still outlasts a lookup in flight when the removal landed; expiring early only falls back to learning by a failed read, as before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicholas Oh
|
f5989a21fb | s3: honor assignment fsync in streaming copies (#12277) | ||
|
|
a94edebf00 |
fix(s3tables): keep the data under a deleted namespace (#12276)
* fix(s3tables): keep the data under a deleted namespace A rename leaves the source table directory holding data the renamed table still points at, and DeleteNamespace removed it recursively. The delete now keeps the directory, minus its catalog record, when files remain under it; CreateNamespace can take it back. * fix(s3tables): keep hidden-directory data, report name takeover as 409 - DeleteNamespace also looks for files under hidden directories, since a table location may sit under one. - CreateNamespace answers 409 when a file takes the name during the take-back of a kept directory. * s3tables: a vanished leftover holds nothing to keep A bare directory removed between the namespace listing and the walk made dirHoldsFiles answer its ListEntries NotFound, and the delete reported that miss as a 404 or 500 on a live namespace. Skip the gone directory instead: whatever it held is already gone, so it cannot keep the namespace. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: a delete's gone entry is a conflict, not a success The filer answers a precondition delete on an entry that is already gone with FailedPrecondition carrying the not-found text, which DoRemoveRequest reads as a completed delete -- DeleteNamespace answered 200 for a delete that never ran. deleteDirectoryIfMatches now calls DeleteEntry directly, keeping the FailedPrecondition-to-conflict mapping and refusing a response-level error. keepAsLeftover likewise reports a directory removed since the scan as ErrConcurrentUpdate, so the handler answers 409 rather than 500. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Jay HemnaniandChris Lu
|
ecd716e0e2 |
s3: allow POST Object uploads when auth is off (#12273)
* s3: allow POST Object uploads when auth is off With no identities configured every other handler skips authentication, but PostPolicyBucketHandler still verified the form signature, so a browser form upload failed with InvalidAccessKeyId (or a malformed credential error without signature fields). Skip the signature check when auth is off; the policy conditions are still enforced. * s3: update AuthPostPolicy comment for the auth-off path --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
05584c541c |
s3: take list markers as raw keys under encoding-type=url (#12275)
* s3: take ListObjects marker/start-after as raw keys Under encoding-type=url AWS url-encodes the response fields but treats request markers as raw keys: SDKs decode Marker/NextMarker/StartAfter and send the decoded key back. Unescaping the marker server-side read a literal "%XX" key as the key it encodes — StartAfter=a%2Bb resumed after "a+b", skipping it, and a paginator whose marker decoded to an earlier key got the same NextMarker twice and failed. Take the request value as-is; only the server's own base64: continuation token still decodes. The marker-echo dedup now compares contents in the response wire encoding and NextMarker in its raw form, since a raw "%XX" marker equals the escaped emission of a different key. * s3: take ListMultipartUploads key-marker as a raw key Same raw-marker rule as ListObjects: under encoding-type=url the emitted key markers are url-escaped keys, and a client decodes them before sending the marker back, so the request value is already the raw key. Keeping the token decode only for the non-url path, where the response could not carry the raw key and falls back to the base64: token form. * s3: cover ListMultipartUploads url key-marker at the handler The selectMultipartUploads tests bypass the request marker decoding, so a regression that unescapes an encoding-type=url key-marker would pass them. Drive a listing through ListMultipartUploadsHandler with a literal "a%2Bb" marker and assert it resumes after that exact key. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Jay Hemnani
|
26783fbcb3 |
s3: keep an unset CORS MaxAgeSeconds unset (#12272)
A rule without MaxAgeSeconds was stored as 0, so GetBucketCors returned MaxAgeSeconds 0 and preflights sent Access-Control-Max-Age: 0. Store -1 for unset, which corsRuleFromProto already reads back as nil. |
||
|
hsdfat
|
9df2125bb0 |
fix(s3): reject non-integer max-parts, part-number-marker and max-uploads (#12270)
getObjectResources and getBucketMultipartResources dropped the strconv error, so a value that does not parse was read as 0. ListParts with max-parts=abc then reached the filer with limit 0, which the filer reads as its default listing limit, and returned every part with <MaxParts>0</MaxParts>; part-number-marker=abc restarted the listing from part 1; and ListMultipartUploads with max-uploads=abc answered an empty, untruncated list even with uploads in progress. Return the existing InvalidArgument errors (ErrInvalidMaxParts, ErrInvalidPartNumberMarker, ErrInvalidMaxUploads) instead, the way max-keys is validated. max-parts and max-uploads are parsed as 32-bit integers to match the documented 0..2147483647 range. |
||
|
Chris LuandDevin
|
97e1758bcc |
s3: auto-create bucket on POST Object uploads to a missing bucket (#12266)
* s3: auto-create bucket on POST Object uploads to a missing bucket PutObject and CreateMultipartUpload lazily create the bucket, but a signed POST Object upload answered NoSuchBucket. Mirror the PUT path: when the versioning-state lookup reports the bucket missing, run the shared handleAutoCreateBucket gate (bucket-name check, AutoCreateBucket option, admin identity) and then re-read the versioning state. The POST handler already records the verified signer identity in the request context, so the admin check and bucket-owner attribution work unchanged. * test: assert the POST auto-create upload lands, not just non-404 The missing-bucket POST test passed whenever the handler created the bucket, even if the upload then failed. Drive the full path through the existing putToFiler fakes — assigned volume accepts the chunk, object entry and payload land — and assert the 201 success response. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: record verified post-policy signer's account id The POST policy path authenticates in the handler, after the internal account header was already scrubbed, so auto-created buckets and the uploaded objects stored no canonical owner. * test: assert POST auto-create records bucket owner, keep writes applied in denied case --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
b70c6dc6ed |
s3: let bucket-scoped Admin grants auto-create matching buckets (#12268)
* s3: let bucket-scoped Admin grants auto-create matching buckets autoCreateBucket required a global Admin identity, so an identity scoped to a prefix like "Admin:outline*" could explicitly CreateBucket but was denied when an upload auto-created the same bucket. Gate on CanDo(ACTION_ADMIN, bucket) — the same check explicit CreateBucket enforces — so a prefix-scoped key lazily creates only matching buckets. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: run auto-create bucket admin through the policy-aware path A scoped Admin identity with an attached IAM policy that denied s3:CreateBucket could still auto-create a matching bucket, because userCanAdminBucket only consulted legacy Actions. Evaluate the same VerifyActionPermission explicit CreateBucket gets, over a synthesized PUT /bucket request so the attached-policy check resolves to s3:CreateBucket on the bucket ARN regardless of the upload's shape. * s3: record verified identity as owner of auto-created buckets An admin authenticated outside the middleware (IAM integration, post policy signature) left no identity in the request context, so an auto-created bucket stored no owner and never reached the owner index. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Nguyễn Lộc Phúcanduser.email
|
7d8b72c2b4 |
fix(s3tables): answer 404 for a table name that was renamed away (#12258)
* s3tables: answer 404 for a table name that was renamed away A rename leaves the source directory in place to hold the data under it, with its catalog attributes cleared. UpdateTable, DeleteTable, the table policy calls and the table maintenance calls read the metadata attribute of that directory and reported its absence as an internal error, so they answered 500, and GetTablePolicy answered NoSuchPolicy. An absent metadata attribute now reads as a missing entry. tableMetadata reports it as both filer_pb.ErrNotFound and the attribute error, so callers that test for either keep working. The policy and maintenance handlers read the attribute through the new readTableMetadataAttr, which reports filer_pb.ErrNotFound, so each call answers NoSuchTable. * s3tables: a view record without its metadata reads absent too * s3tables: refuse a catalog write that lands on a metadata-less record --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |