mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-27 15:30:45 +01:00
d182be6a9e60f796b3d3b847665615c842cd91df
1576
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandDevin
|
d182be6a9e |
s3api: cap SOSAPI capacity by the reported physical disk bytes (#12118)
calculateClusterCapacity multiplies advertised volume slots by the volume size limit. Slot counts are an admission bound rather than a capacity measurement — an explicit -max can be set higher than the volumes a disk actually fills — so a small disk can report more total and available capacity than it physically has. Bound both numbers by the per-disk filesystem counters the volume server already reports in its heartbeat, falling back to the slot estimate only when they are absent. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
ee315527cb |
vector: address remaining s3vectors review findings (#12115)
* vector: address remaining s3vectors review findings Catalog and permission model: - Vector buckets are declared Lance table buckets end to end: create sets format=lance, get/list/delete reject or skip ICEBERG buckets, and the format lookup runs on an internal trusted catalog so callers need no GetTableBucket grant. - The implicit "vector" index namespace is created through the same internal manager, so a CreateTable grant alone authorizes CreateIndex; typed S3TablesError codes replace string matching on err.Error(). - CreateIndex pre-checks existence after the authorize, tags the record with a create nonce, and rolls back on engine failure under context.WithoutCancel with a versionToken CAS, so a retry never deletes a table it did not create. - DeleteIndex drops the catalog record directly; DropIndex is unimplemented on the shared VectorService. - DeleteVectorBucket removes the empty internal namespace on BucketNotEmpty and retries once. - PutVectors/DeleteVectors require the update grant, and engine gRPC errors map to typed API errors instead of generic 500s. API correctness: - CreateVectorBucket/CreateIndex return their resource ARNs. - ARNs reject empty bucket segments; vectorBucketArn must be an ARN. - ListIndexes returns summaries built only from catalog fields. - ListVectors honors maxResults/nextToken inside a segment via "seg:<pageStart>:<nextIndex>" resume tokens and validates segmentCount/segmentIndex. - QueryVectors requires positive topK; PutVectors rejects empty keys. - CreateVectorBucket rejects a supplied encryptionConfiguration rather than silently dropping it. - Metadata filters emit Lance's real JSONB UDFs: json_get_string/ json_get_float/json_get_bool picked by operand type, json_exists with a bracketed JSONPath, and per-member accessors for $in/$nin. Engine: - weed-vector-server stores the metadata column as Lance JSONB (lance.json extension) so the json_* UDFs apply; reads accept legacy Utf8 JSON columns. Vectors port TLS: - The TLS config is built before binding listeners, loads -cacert.file into ClientCAs when client-cert verification is on, fails startup on an empty/invalid CA file, and closes the gRPC conn on every failure path. * s3vectors: authorize index create before writing the implicit namespace A denied caller could still leave a persistent namespace behind because ensureIndexNamespace wrote through the trusted catalog before the caller's create grant ran. AuthorizeCreateTable now tolerates the not-yet-declared implicit namespace (bucket policy alone decides) and runs before any write; real table creates keep requiring the namespace. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: keep engine error details out of the API response Unavailable now maps to 503 ServiceUnavailableException and unclassified codes return a generic message; the real gRPC error stays in the log. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: exact-type metadata filters + NULL-safe $nin The jsonb extractors convert numeric-looking strings through json_get_float and 'true'/'false' through json_get_bool, so a stored '3' could satisfy a filter for the number 3. Numeric and boolean comparisons now also require the stored value not be a string, and each $in/$nin group coalesces NULL so NOT cannot invert NULL into NULL. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: validate the vectors TLS key pair before binding The reloading certificate provider reads the files asynchronously, so a missing or mismatched pair would only surface as failed handshakes on an announced HTTPS port. Load it once up front. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: make a token-checked table delete atomic The version token was read in one filer call and enforced in another, so a delete+recreate slipping between them let the delete erase the replacement. When a token is supplied the name path now carries the observed mtime as an if_not_modified_after precondition (evaluated under the path lock), and a decoupled delete strips the catalog marker under a compare of the captured attributes before touching the data path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: compare filter operands against the stored JSON type jsonb's strict converters make json_get_string return text for numbers and booleans too, so a get_string IS NULL guard is never true for scalar metadata — numeric and boolean comparisons silently matched nothing. Numbers and booleans now compare through TRY_CAST of the serialized json_extract text, which yields NULL on any other stored type; string comparisons stay on json_get_string behind a CASE guard so the extractor only runs on rows storing a JSON string. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: allocate implicit namespace metadata before use A first table create in an undeclared namespace took the allowMissingNamespace branch with a nil record and panicked on the owner assignment. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: read the delete target's metadata and snapshot in one lookup Two separate lookups let a delete+recreate in between validate the caller token against the old record while the conditional delete aimed at the replacement. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: assert the captured record before a conditional table delete An xattr update keeps Mtime, so the IfNotModifiedAfterNs guard alone could not see a metadata write that had already rotated the version token, and a recreate preserving the mtime could slip the mtime check. The delete now first proves the entry still carries the captured attributes under the filer's path lock, then keeps the mtime precondition. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: match only JSON boolean literals in metadata filters * filer,s3tables: delete versioned tables under one attribute precondition * s3vectors: compare serialized JSON for string filter operands json_get_string converts numeric-looking strings and booleans, so it cannot serve exact-typed string comparisons even behind a guard. String operands now compare the serialized json_extract text directly, which is exact-typed and never errors; a Rust test pins the semantics against a mixed-type dataset. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: compare boolean filter operands as serialized text Arrow's boolean cast accepts '1' and '0', so a TRY_CAST column let a stored number satisfy a boolean filter. Booleans now compare the serialized json_extract text like strings do — a JSON boolean only ever serializes as 'true' or 'false'. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: claim a catalog name's markers in one compare-checked write A create wrote its markers attribute-by-attribute after checking the name was free. A delete's restore landing in the check-to-write gap put the old markers back first, and the create then overwrote only the ones it sets — leaving the old tags, policy, and maintenance settings on the new table. CreateTable, RegisterTable, and CreateView now write all their markers in a single update that refuses to run while any catalog marker is present, so the second writer loses with a conflict instead of inheriting them. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: test the attribute-preconditioned table delete and name claim Adds a BeforeDelete hook plus ExpectedExtended evaluation to the in-memory filer so tests can land a competing write in a handler's check-to-delete window, and covers two losing races: a token delete whose record changed since the read, and a create colliding with a delete's marker restore. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: reserve a catalog record under one compare-checked claim claimCatalogRecord writes a deleting marker onto the record while it still holds exactly the caller's captured attributes, so a delete or rename can hold the record through a multi-step operation and a losing writer backs out before any data or destination entry is written. The claim key joins the markers a name-claim asserts absent, so the name cannot be reused while the reservation stands. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: claim the record before purging or publishing past it A token delete of a decoupled table purged the data first and checked the marker afterwards, so a rename that finished in between left the moved record pointing at purged data while the delete reported a conflict. The delete now claims the record under the captured attributes before any data is removed and releases the claim if the purge fails, keeping the record intact for a retry. The rename claims the source before publishing the destination, so a losing rename backs out while no destination entry exists instead of leaving one behind pointing at data a delete went on to remove. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: test the record claim across delete and rename races A rename completing in the token delete's check-to-claim window turns the delete into a conflict with the data intact, and a delete's claim on the source turns the rename into a conflict before the destination entry is published. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
bfb6e40fb4 |
test: assert the resumed segment token in ListVectors paging (#12108)
test: assert the resumed segment token The second listSegmentVectors result was never checked (staticcheck SA4006); assert it resumes scanning at row 19 after matches 10..18. |
||
|
Chris LuandDevin
|
e2d899ac25 |
test/vectors: fix the S3 Vectors e2e harness (#12112)
* test/vectors: give the vector server static storage credentials The e2e harness ran weed-vector-server without credentials, so lance's AWS object store fell back to IMDS and CreateIndex died on the runner's 411 from the token endpoint. Point it at the test credentials the s3 gateway now reads from s3.json. * vector: take the list item field from the schema when batching vectors Lance stores the fixed-size list item as nullable even when the write schema declared it non-null, so PutVectors built batches that no longer matched: expected FixedSizeList(4 x Float32) but found FixedSizeList(4 x non-null Float32). * vector: store index metadata as lance jsonb so filters resolve The metadata field was plain Utf8, but the filter functions lance's planner registers (json_get_string, json_get_float, json_exists) take a LargeBinary jsonb column, so QueryVectors with a metadata filter failed to plan. Declare the field with the arrow.json extension — lance encodes it to jsonb on write and back on read — and encode the batch's strings when the stored column is LargeBinary. Also rename the filter's json_get_str call to json_get_string, the name lance registers. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
a9d6bfc51f |
test: boto3 s3vectors conformance harness (P1) (#12105)
* vector: boto3 s3vectors conformance harness test/vectors spins a mini cluster, an s3 gateway with -port.vectors and -vectors.grpc, and a weed-vector-server, then exercises the AWS client end to end: bucket and index lifecycle, put/get/list/delete vectors, and a filtered query asserting distances and metadata. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: run the s3vectors conformance subset An e2e job in rust-vector-tests: builds weed and weed-vector-server, then runs test/vectors against a mini cluster plus the vectors gateway. The workflow now also watches vector.proto, its stubs, and test/vectors. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: fix e2e protoc and codespell findings The e2e job needed the same vendored-protoc step as the unit-test job — lance's build scripts locate protoc themselves. Codespell flags the AWS `$nin` operator literal and the `lits` variable: exempt `nin` in .codespellrc like the existing word exemptions, rename the variable, and fold the negation into the IN/NOT IN symbol directly. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: classify catalog errors by code, not message text * s3: drop the implicit vector namespace on vector-bucket delete * test: point the vector engine at the gateway lance port * test: bind the vectors gateway listeners to loopback * test: make the filter and distance assertions load-bearing * test: rebuild the harness binaries on every run * ci: fix the vectors end-to-end job * ci: run the vector tests on gateway and command changes --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
c55b7c5a29 |
s3: fix CreateIndex metadata panic and ListVectors segmented paging (#12106)
* s3: nil-check metadataConfiguration before reading its keys * s3: honor maxResults/nextToken in segmented ListVectors * s3: exclude non-Lance buckets from ListVectorBuckets * s3: test segmented ListVectors paging token math * ci: silence codespell on the "$nin" operator and rename lits |
||
|
Chris LuandDevin
|
dcc94f818d |
vector: S3 Vectors API gateway (P1) (#12103)
* vector: S3 Vectors API gateway over VectorService weed/s3api/s3vectors serves the AWS S3 Vectors JSON API: bucket CRUD maps to S3 Tables table buckets, index CRUD to Lance table records plus the VectorService CreateIndex/DropIndex RPCs, and the data plane (Put/Get/List/Delete/QueryVectors) calls the shared service. The AWS filter document translates to a Lance SQL predicate over the metadata JSON column; unsupported operators fail loudly rather than overmatch. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: -port.vectors and -vectors.grpc on weed s3 Mirror the Lance server startup: the S3 Vectors API binds its own port and dials the weed-vector-server gRPC address; both disabled by default. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: resolve index ARNs, authorize index ops, fix catalog fields - vector buckets declare FormatLance so a new bucket accepts index tables instead of defaulting to ICEBERG. - ListIndexes decodes the `tables` response field; `tableSummaries` hid every index. - resolveIndexARNs honors indexArn alone; handlers no longer treat indexName as the only carrier. - data-plane RPCs fetch the index record first, so callers authorize against the index resource and a missing index 404s before touching the engine; QueryVectors reports the index's configured metric. - IndexArn added to the index-level request types. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: serve the vectors port over TLS when configured When -key.file/-cert.file are set, the vectors listener mirrors the main API and serves TLS rather than a second plaintext endpoint. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: harden the s3vectors gateway handlers - $ne no longer shadows the operator: exclusion filters emit <> instead of equality. - a null element in a PutVectors list fails the request instead of panicking. - requests are capped at 32 MiB; distances return via *float32 so an exact match still reports 0 when returnDistance is set. - CreateIndex declares the `vector` namespace first and validates dimension (1..=4096) before the catalog write; a failed VectorService.CreateIndex rolls the table record back. - non-filterable metadata keys persist as JSON rather than a comma-joined string, and index records carry their createdAt. - ListVectors honors segmentIndex/segmentCount by slicing the full listing in the gateway. - catalog errors map to AccessDeniedException/ConflictException instead of a blanket ValidationException, and malformed ARNs 400 instead of slicing. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: /{Operation} routes and vectors-port TLS, conn cleanup Path-dispatched clients (POST /QueryVectors) reach the same handlers as X-Amz-Target calls. A failed listener bind now closes the vector gRPC connection instead of leaving it open with the endpoint disabled. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Jay Hemnanianduser.email
|
1e7afbacf4 |
s3: list every bucket when auth is off (#12077)
* s3: list every bucket when auth is off With no identities configured, every request may create, read and write any bucket, but ListBuckets treated the caller as an unauthenticated user and returned an empty list, so `aws s3 ls` showed nothing. List all buckets in that mode, with prefix, max-buckets and continuation tokens as before. Listing with auth on is unchanged. * s3: cover the auth-on anonymous ListBuckets case The no-auth listing change also moves the nil-identity gate; pin the security-relevant direction that a request without an identity still sees no buckets once auth enforcement is on. * s3: verify ListBuckets continuation token reaches the next page --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
3abf8b2572 |
filer,pb,mount: keep per-chunk cipher keys out of logs (#12085)
* filer: format ChunkView without its cipher key readChunkSliceAt logs a failed chunk fetch with %+v, which printed the per-chunk AES-256-GCM key into error-level logs on every failed read. Give ChunkView a String() that omits CipherKey so %v and %+v stay safe. * pb: scrub cipher keys when an entry must be logged whole ScrubCipherKeys clones an Entry with chunk CipherKey fields cleared, so a warning that needs the full entry still cannot leak the keys. * log entry identifiers instead of whole metadata messages Several log lines formatted whole protobuf messages — SubscribeMetadataResponse, EventNotification, Entry-carrying requests — whose chunks include cipher_key under -encryptVolumeData, so the keys escaped into logs (some at default verbosity). Log the directory, entry name and event timestamp instead; where a chunk list was printed, print file ids via FileIdsString. * log queue: unwrap SubscribeMetadataResponse and never print raw protos The fallback printed any message with %+v, so a SubscribeMetadataResponse carrying entries would still put cipher keys in the log. Unwrap its event notifications (batched events included) and reduce unknown messages to their type name. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * remote sync logs: name the event's real path Rename targets logged under resp.Directory while the entry lives under NewParentPath; route them through MetadataEventTargetDirectory. Sync failure logs printed only the directory, so same-dir failures shared an identifier — log the source and destination paths instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * log queue: log the entry's destination path on moves A cross-directory rename printed "file" -> "file" under the source path, hiding where the entry landed. Use NewParentPath so the log shows the destination directory next to the new name. * admin: install the go.mod-pinned templ, not @latest install-templ fetched templ@latest, which now emits templruntime.NewAttributeExpression - a symbol absent from the runtime pinned in go.mod - so every fresh 'make' regenerated uncompilable _templ.go files. Install the go.mod version and reinstall when the binary on PATH reports a different one. * admin: drop stray Makefile change belonging to the templ pin PR Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * notification: join the move destination path in log queue output A NewParentPath of "/" printed //name for moves into the root; path.Join keeps a single leading slash. * log queue: skip a nil event notification instead of logging an empty move Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * pb: tolerate nil chunk entries while scrubbing cipher keys for logs Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * subscription and backup failure logs name the event's file The process/marshal failure logs printed only the event directory and timestamp, so a failed sync left operators guessing which file failed. Use MetadataEventSourceFullPath/MetadataEventTargetFullPath so each failure names the source and target paths without printing the entries. * replication: identify entry-less events without printing the proto Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: scrub cipher keys from the bucket-metadata debug dump buildBucketMetadata marshals the whole filer entry to JSON for a V(3) log line; a chunked entry would emit each chunk's cipher_key base64. Marshal the scrubbed copy instead. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
c27d7fe5a4 |
s3: keep transient object-entry errors out of ErrObjectNotFound (#12083)
* s3: keep transient object-entry errors out of ErrObjectNotFound getObjectEntry wrapped every fetch error as ErrObjectNotFound, so a transient filer failure (busy filer, ctx deadline, transport blip) was indistinguishable from a real miss. LifecycleDelete resolves ErrObjectNotFound permanently as NOOP_RESOLVED, so one flaky lookup at dispatch time silently skipped the object's expiration - errors=0, no retry, no log. This matches the flaky s3tests test_lifecyclev2_expiration signature: exactly one object of a same-rule pair surviving past its window. Only collapse genuine misses (filer not-found, version/latest-version sentinels); everything else propagates so the worker retries it as RETRY_LATER instead of resolving it. Also seed bucketConfigCache in the retention test server and teach the fake filer to inject per-path lookup errors. * s3: keep invalid version ids on the not-found path in getObjectEntry A malformed versionId used to collapse into ErrObjectNotFound like every other fetch error, so the handlers answered NoSuchKey. With transient errors now propagated, an invalid id would surface as InternalError; fold it back into the miss classification. * filer_pb: keep Unavailable out of LookupEntry's not-found normalization * s3: reject Unavailable in isObjectEntryMiss before message matching * s3: keep transient fallback errors out of latest-version resolution * filer_pb: limit LookupEntry's not-found text match to miss statuses * s3: limit isFilerNotFound text matching to miss statuses |
||
|
Nguyễn Lộc Phúcanduser.email
|
937633a77d |
iceberg: fix CREATE TABLE rejecting a partition on the first column (#12072)
* iceberg: fix CREATE TABLE rejecting a partition on the first column Apache Spark number the columns of a new table from 0 and leave assigning fresh IDs to the catalog, so a table partitioned or sorted by its first column arrives with source-id 0. The create-table request decoded its partition spec and write order into iceberg-go's bound PartitionSpec and SortOrder, which require positive source IDs, so every such CREATE TABLE, and CTAS through stage-create, was refused with "Malformed request: Invalid request body". Decode both as iceberg.UnboundPartitionSpec and table.UnboundSortOrder, as iceberg-go documents for catalog create-table requests; newTableMetadata already resolves the placeholders through the request schema by name and assigns fresh IDs. A spec or order that does not resolve against the schema is now answered with 400 instead of 500, and the decode error message carries the underlying reason. Tests check that Spark's request bodies resolve to the same metadata as 1-based placeholders, that the handler accepts them, that unresolvable source IDs get 400, and add a Spark integration test that partitions by the first column. * iceberg: gofmt the create-table source-id tests --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
|
b85281f0da |
s3: group ListMultipartUploads by delimiter into CommonPrefixes (#12069)
* s3: group ListMultipartUploads by delimiter into CommonPrefixes The delimiter was echoed but ignored, so every upload was listed. Keys that continue past the delimiter after the prefix are now listed once as a common prefix, counted toward max-uploads like an upload. * s3: bound tracked multipart common prefixes to the page Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: omit upload id marker when a page ends on a common prefix Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
6dd6ad512c |
iam/oidc: drop single-use web-identity tokens and the jti replay cache (#12065)
* iam/oidc: drop single-use web-identity tokens and the jti replay cache Reverts the single-use semantics added in #11936 (and the #12017 refactor): AssumeRoleWithWebIdentity consumed each web-identity token's jti into a bounded replay cache, so a second exchange of the same token was rejected until exp + skew. AWS STS accepts a web-identity token repeatedly until it expires, and Kubernetes workload identity depends on that: the kubelet keeps a projected ServiceAccount token until ~80% of its lifetime, so pod restarts, co-located containers sharing the token file, repeated CLI invocations, and pre-rotation SDK refreshes all re-present the same jti and were refused on 4.50. The replay machinery also carried a real maintenance cost — a bounded concurrent cache, fingerprint-keyed entries for jti-less tokens, adopt/merge handoffs so provider rebuilds could not reopen consumed tokens, and provider-reuse bookkeeping in the refresh path that existed only to preserve that state. Removing it restores AWS semantics and deletes the whole apparatus. Refs #12063. Alternative to the per-provider opt-out in #12064. * iam: reuse unchanged OIDC providers on refresh refreshOIDCProvidersFrom rebuilt every managed provider on any store change, so an unrelated write discarded each provider's JWKS cache and an unreachable issuer could then fail exchanges the cached keys would have allowed. Keep a per-record provider cache keyed on the record content fingerprint and only rebuild when the record changes. * iam/oidc: require exp in ValidateToken Dropping the single-use check also removed the implicit bound on expiry-less tokens: a signed web-identity token without exp could mint STS sessions indefinitely. Require the exp claim, matching AWS AssumeRoleWithWebIdentity which rejects tokens without one. * iam/oidc: bound idle connections on the provider transport A provider replaced on refresh drops its http.Transport without a close; give it an IdleConnTimeout so pooled connections and their goroutines cannot linger. * s3api: cover web-identity token reuse over HTTP Sign one token and exchange that exact string twice, directly covering the Kubernetes projected-token reuse this change restores. * test(iam): refresh reuses an unchanged record's provider Covers the provider-reuse cache: an unrelated record's refresh keeps a provider's JWKS state, a changed record rebuilds it. |
||
|
|
515380c349 |
s3: answer GET and HEAD of a delete marker as S3 does (#12051)
* s3: answer GET and HEAD of a delete marker as S3 does A delete marker as the current version now returns 404 NoSuchKey with x-amz-delete-marker: true and the marker's x-amz-version-id. Asking for the marker by versionId returns 405 MethodNotAllowed with the same headers and Last-Modified, instead of 404 NoSuchKey. * s3: check the tag-conditioned read grant before answering a delete marker The marker branch returned before the entry-based policy recheck the rest of the read path performs, so an s3:ExistingObjectTag-conditioned Deny did not apply to the delete-marker response and its version headers. * s3: send Allow: DELETE with the delete-marker 405 RFC 9110 15.5.6 requires a 405 answer to list the methods the resource supports; S3 sends allow: DELETE for a delete marker named by versionId. * s3: keep delete-marker headers on conditional-read misses If-Match or If-Unmodified-Since against a current delete marker short- circuited to a bare NoSuchKey. Carry the resolved marker through the conditional result so the 404 keeps x-amz-delete-marker and x-amz-version-id, matching the unconditional read. * s3: exercise delete-marker responses through GET and HEAD handlers * s3: read marker version id from the entry before the caller's guess A delete marker written outside the normal PUT path can miss the version-id attribute while the v_<id> file name still names it; the new x-amz-version-id header would then report "null". Reuse versionIdFromEntry so both the handler and conditional-miss paths emit the real id. * s3: answer 405 for versionId=null on a suspended-versioning marker The null marker lives in .versions while getSpecificObjectVersion only checks the plain path, so GET/HEAD ?versionId=null answered NoSuchVersion. On that miss, look up the null marker and answer it like any other marker named by versionId. * s3: propagate null-marker lookup failures instead of answering 404 A transient filer error reading .versions/v_null was swallowed as "no marker", so GET/HEAD answered NoSuchVersion for a live read failure. NotFound and absent entries still mean no marker; anything else is InternalError now. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
d15396fc4d |
s3tables: scope the anonymous identity's grant on the table catalogs (#12056)
* s3tables: scope the resolved anonymous identity instead of defaulting to allow Catalog authorization fell back to the open default whenever the request resolved to the configured anonymous identity, ignoring its grant. The fallback now applies only when no principal was resolved (zero-config), so anonymous requests are checked against the identity's configured actions like any other principal. The S3 Tables API and the Iceberg and Lance catalogs share this handler, so all three are covered. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: authenticate the catalog clients in the s3tables suites Anonymous calls into the Lance and Iceberg catalogs are now scoped by the anonymous identity's grant, so the suites that used to rely on the anonymous pass authenticate like their real deployments do: x-api-key for the Lance probes and REST drivers, client-credentials OAuth for pyiceberg. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * worker(lance): authenticate namespace calls with the storage credential Anonymous catalog calls are now scoped by the anonymous grant, so a worker that holds an S3 credential sends it as x-api-key on the namespace requests instead of relying on the anonymous pass. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: authenticate the pushdown catalog connection in catalog_lancedb Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: follow catalog redirects only within the same host reqwest forwards custom headers on a follow, so a redirect to another host would carry the x-api-key credential to a server it was never meant for. Keep same-host hops for proxies that normalize the path or upgrade the scheme. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: warn when the catalog api key travels in cleartext The x-api-key pair is a real storage credential: over plain http to a non-loopback host the network can read it. Warn at client build so the operator sees it; https and loopback stay quiet, and the key still sends since an http catalog is a valid deployment. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lance: require the same origin on catalog redirects A same-host hop can still downgrade https to http and put the api key on the wire; matching scheme, host, and port keeps the credential on the listener it was issued for. * lance: bound catalog redirect hops so loops fail fast The custom redirect policy drops reqwest's default 10-hop limit, so a catalog that redirects back to itself would hold each call open until the request timeout. Keep the bound alongside the origin check. * lance: gofmt the redirect helpers --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
403d51631b |
s3api: stop echoing stored tags and internal attributes on GET/HEAD (#12053)
* s3api: stop echoing stored tags and internal attributes on GET/HEAD setResponseHeaders copied every entry.Extended key into the response, only filtering xattr- and x-seaweedfs-. Stored object tags (X-Amz-Tagging-*) and the gateway's own attributes (Seaweed-X-Amz-Acl, -Owner, -Version-Id, -Object-Lock-Mode, ...) therefore leaked to any caller holding s3:GetObject, and to anonymous readers on public-read objects, without s3:GetObjectTagging or READ_ACP. Tag count, user metadata, storage class and the real S3 headers (x-amz-version-id, x-amz-object-lock-*) are unaffected. * s3api: match stored tag and internal attribute keys case-insensitively |
||
|
Chris LuandDevin
|
e442eda88e |
s3: authorize POST policy uploads through the shared object check (#12047)
* s3: authorize POST policy uploads through the shared object check doesPolicySignatureMatch gated a verified POST Object signature on the identity's legacy Write action, which cannot represent attached IAM policies, STS session policies, or key-scoped grants. Users authorized only by an attached policy (e.g. s3:PutObject on bucket/*) were denied with AccessDenied even though the equivalent PUT succeeds. The handler already runs AuthorizeObjectWrite against the real object key after the form's x-amz-* fields are promoted to headers, so the coarse check only shadowed the full evaluation. Signature verification now just returns the identity; authorization stays in one place. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: POST Object authorizes attached-policy identities, still denies others Cover the signature check and AuthorizeObjectWrite for an identity whose only write grant comes from an attached IAM policy, plus the deny leg for an identity whose attached policy omits s3:PutObject. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
c4d743a69f |
s3: make ListObjectsV2 continuation tokens round-trip XML-unsafe keys (#11996)
* s3: keep ListObjectsV2 continuation tokens round-tripping XML-unsafe keys A page ending on a key XML 1.0 cannot carry (for example one containing \x01) produced a NextContinuationToken from that raw key. The XML encoder replaced the control characters with U+FFFD, so a client resuming at the token restarted after the mangled name and skipped every key sorting between it and the real one. encoding-type=url encoded the keys but not the token, so it hit the same skip. Encode the token as base64url behind a prefix when the key is not valid XML text; keys that survive XML unchanged keep the bare key as the token, so their tokens are byte-identical to before. The handler decodes the prefix back to the raw key before listing. Fixes #11990 * s3: keep undecodable base64: list tokens resuming at the raw key * s3: round-trip ListObjectsV1 markers over XML-unsafe keys * s3: round-trip ListObjectVersions key markers over XML-unsafe keys * s3: round-trip ListMultipartUploads key markers over XML-unsafe keys * s3: decode only tokens whose key could not have stayed bare * s3: document and pin the encoded-wins rule for ambiguous list tokens A token like base64:AQ reads as both a legacy bare key and the encoded form of an XML-unsafe key. The encoded reading already won in code; state why and cover it so the choice is not reverted as a false simplification. * s3: keep request-marker echoes in the response's wire encoding The decoded marker/key-marker is a raw key; echoing it writes XML-unsafe bytes straight into the response, where the encoder turns them into U+FFFD. Re-encode each echo through encodeListContinuationToken so the client reads back the token it sent, and apply urlPathEscape for V1 when encoding-type=url is active, matching the listed keys. * s3: wire-encode the V1 NextMarker echo like the request marker * s3: pin resuming an XML-unsafe key past a sibling directory * s3: wire-encode ListMultipartUploads markers and keys under encoding-type=url Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: unescape url-encoded marker echoes on list resume Under encoding-type=url the V1 and ListMultipartUploads responses emit marker echoes through urlPathEscape, and clients pass them back verbatim. The incoming marker was used as the resume position without undoing that escaping, so resuming after a url-unsafe-but-XML-safe key like "a b" (marker "a%20b") restarted the listing at the literal escaped text. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: assert the full decoded V1 sequence in the marker round-trip test The test only checked the page count, so a regression that skipped one key while repeating another still passed. Requesting encoding-type=url keeps the XML-unsafe keys legible for the decoded comparison and exercises resuming with the escaped marker echo. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: only unescape markers in the canonical urlPathEscape form A url-encoded request marker like a%41 cannot be encoder output, since urlPathEscape never emits %41 for 'A'. Unescaping it anyway resumed at aA and skipped every key sorting between a%41 and aA, including a%42. Require the unescape to re-escape to the received marker so only verbatim echoes decode; anything else stays the raw key the client sent. * s3: pin a raw-key marker that looks like an escape sequence * s3: cover the marker-echo decode guard with unit cases * s3: wire-encode the ListObjectsV2 StartAfter echo under encoding-type=url AWS encodes Delimiter, Prefix, Key, and StartAfter in V2 responses when url encoding is requested, but StartAfter was echoed raw. The echo now takes urlPathEscape and the incoming start-after param runs through decodeMarkerEcho, so a client resuming from the emitted form lands on the raw key. * Merge master and reconcile ListMultipartUploads marker encoding Resolve the overlap with the merged encoding-type=url pass: keys and markers now go through the single post-pass so nothing is url-escaped twice. Markers are still token-encoded when the response is not url-encoded, and the key-marker decode checks the query-escaped wire form the endpoint actually emits. * s3: echo the decoded start-after under encoding-type=url The escaped echo of the raw input is not a fixpoint: a client resubmitting the echoed StartAfter decoded it to the literal escape text and resumed at the wrong position. * s3: keep url-mode multipart key-marker decodes off the token path Under encoding-type=url the emitted KeyMarker/NextKeyMarker are the url-escaped raw key, never a base64-prefixed token, so the request decode must unescape only. Running the token decode afterward misread a literal "base64:..." key as an encoded token and resumed at the decoded bytes, skipping the keys in between. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
52e05ccda4 |
s3: accept a self-copy that changes the storage class (#12031)
* s3: accept a self-copy that changes the storage class or encryption AWS changes an object's storage class or encryption by copying the object onto itself with the new x-amz-storage-class or SSE headers, and only rejects a self-copy that changes none of metadata, storage class, website redirect location or encryption. CopyObject rejected every self-copy without REPLACE metadata or tagging directives as InvalidRequest, so these requests failed. * s3: reject an invalid x-amz-storage-class on CopyObject A self-copy with a bad storage class used to fail as InvalidRequest; once such copies are admitted the value must be validated like PutObject does, or it would be stored as opaque metadata. * test: cover the invalid storage class rejection on self-copy * s3: only a storage class change admits a self-copy Admitting encryption changes on a self-copy is unsafe: the metadata-only path compares only KMS key IDs, so a new encryption context or bucket-key setting is advertised over the old data key, and SSE-KMS key rotation onto the same entry keeps the old data key. Count only a requested storage class that differs from the object's (STANDARD when none is stored) as a change, so a self-copy to the current class or with only encryption headers is rejected as before. Such a self-copy updates the object in place and has to keep its encryption; one that would re-encrypt it or uses SSE-KMS is rejected too. --------- Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
|
b0e6f7a096 |
s3: allow up to 50 tags in a bucket tag set (#12030)
* s3: allow up to 50 tags in a bucket tag set PutBucketTagging validated the tag set with the object tag rules, which allow 10 tags. AWS allows a bucket up to 50 tags, so tag sets of 11 to 50 tags were rejected with InvalidTag. Validate bucket tags against 50 and keep the limit of 10 for objects and S3 Tables. A tag set that repeats a key is rejected with InvalidTag, as AWS does, instead of keeping one of the values, so 51 tags with a repeated key no longer pass as 50. * s3: reject duplicate tag keys in PutObjectTagging The tag set collapses to a map, so a repeated key was silently kept once.\nMatch the bucket tagging check and answer InvalidTag instead. * s3: reject empty tag keys AWS requires a tag key to be at least one character; the shared\nvalidator only checked the 128-char upper bound. * s3: reject duplicate tag keys in the tagging header parser parseTagsHeader (the x-amz-tagging path used by CopyObject with TaggingDirective=REPLACE) silently kept the last value when a key repeated, while the PutObject and multipart header paths and the new XML checks return InvalidTag for the same input. A repeated key after URL decoding now fails the same way. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: validate x-amz-tagging tags on PutObject before storing the entry The header path only rejected duplicate keys, so an upload could store more than ten tags or keys and values that fail S3 tag rules. Run the parsed set through ValidateTags like the XML tagging handlers do. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: validate x-amz-tagging tags on multipart initiate ParseS3Metadata stored header tags with no tag-set validation, the same gap putToFiler had. Multipart uploads now reject oversize or malformed tag sets with InvalidTag. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: count tag length in characters, not bytes * s3: validate object tags before uploading chunks * s3: reuse parseTagsHeader and drop raw header from warnings --------- Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
5700aa5e91 |
s3: evaluate bucket CORS for path-style OPTIONS on the bucket root (#12028)
* s3: evaluate bucket CORS for path-style OPTIONS on the bucket root
A path-style preflight for /bucket (no trailing slash) missed the bucket
OPTIONS route, whose template is /{bucket}/, and was answered by the global
OPTIONS handler from the global AllowedOrigins instead of the bucket CORS
configuration. Register the bucket OPTIONS route without a path matcher, and
keep OPTIONS for the health probe paths on the service-level handler ahead of
each path-style router.
Fixes #12025
* test: drop the issue reference from the bucket-root preflight test comment
---------
Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Chris LuandDevin
|
fd3b437e04 |
s3api: write the null version's object-lock metadata to the plain entry (#12033)
* s3api: write the null version's lock metadata to the plain entry PutObjectRetention and PutObjectLegalHold with versionId=null loaded the pre-versioning entry but wrote the update under <key>.versions/v_null, so the acknowledged lock never bound the object it was read from. Resolve the entry and its write path in one helper that applies the same null-version rule as getSpecificObjectVersion. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: cover the null-version lock write path Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: persist object-lock metadata with an in-place entry update mkFile rebuilds the target entry from chunks alone, so a lock write on a plain (null-version) entry discarded attributes, inline content, and the remote reference. UpdateEntry keeps the whole fetched entry, matching how the tagging and ACL writers persist metadata. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
fb3fb9dcb8 |
s3api: support s3:RequestObjectTagKeys in bucket policies (#11993)
* s3api: support s3:RequestObjectTagKeys in bucket policies Condition operators accept the ForAllValues: and ForAnyValue: qualifiers, and s3:RequestObjectTagKeys holds the keys of the tags taken from X-Amz-Tagging wherever s3:RequestObjectTag/<key> is set. CopyObject no longer stores an empty-key tag for an empty segment in X-Amz-Tagging, so it keeps the key set a policy has checked. Fixes #7679 * s3: parse tagging header like the tag handlers, not url.ParseQuery url.ParseQuery stops at 10,000 parameters, so an X-Amz-Tagging header padded with empty & segments made requestObjectTags return nil: the copy handler still stored the denied tag while every s3:RequestObjectTag* condition saw nothing. Scan non-empty segments the way parseTagsHeader does so the policy sees the same tags the object gets. * test: gofmt s3tables test files after #11997 The format check fails on these files on master as merged, so every PR's S3 Tables Format Check goes red until they are re-formatted. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
3810fb1123 |
s3: url-encode ListMultipartUploads results for encoding-type=url (#11998)
* s3: url-encode ListMultipartUploads results for encoding-type=url The response echoed EncodingType=url but returned raw upload keys, and XML 1.0 cannot carry control characters, so the encoder replaced them with U+FFFD. The listed key then matched no upload: AbortMultipartUpload with it answered 404 NoSuchUpload, leaving such uploads uncleanable. Encode the key-named elements AWS encodes for this API (Key, Prefix, Delimiter, KeyMarker, NextKeyMarker) the way ListObjects encodes its keys when encoding-type=url is requested. Upload ids stay opaque, and responses without encoding-type=url are unchanged. Fixes #11991 * s3: echo EncodingType only when encoding-type=url is honored A request with an unrecognized encoding-type echoed the value back in <EncodingType> while the keys stayed raw, matching neither AWS (which omits the element unless url encoding applies) nor the ListObjects handlers, which gate the echo on the parsed url flag. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
f5e319ae8e |
iam: GetUserPolicy returns NoSuchEntity for a policy name the user does not have (#11944)
* iam: GetUserPolicy returns NoSuchEntity for a policy name the user does not have When the user has stored inline policies, an unknown name fell through to the policy rebuilt from the user's actions, so a deleted policy still read back. The standalone IAM server already returns NoSuchEntity here. * iam: return service failure when listing inline policies fails A ListUserInlinePolicies error in GetUserPolicy fell through to action-based reconstruction, masking a store failure as a policy document. Propagate it as ServiceFailureException instead, matching the standalone iam server and the other callers in this file. * iam: return service failure when the inline policy store read fails --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
f8579bf728 |
s3: gate POST Object acl fields behind PutObjectAcl (#11939)
* s3: gate POST Object acl fields behind PutObjectAcl The POST Object handler promotes acl and x-amz-grant-* form fields onto the upload's ACL headers, and resolveFileMode honors the promoted canned ACL. The PUT path only applies such headers after an s3:PutObjectAcl grant; mirror that on POST by running the shared per-key authorization with the WriteAcp action whenever the request carries ACL headers. * s3: cover the POST Object acl gate Signed POST requests carrying an acl form field are denied for a writer without the WriteAcp action and for a writer a bucket policy denies s3:PutObjectAcl on the target key; the authorization boundary is exercised directly as well. * s3: cover x-amz-grant-* POST fields and the no-ACL case * s3: encode POST policy conditions as JSON in the acl-gate test |
||
|
Chris Lu
|
cd97c93deb |
command: keep filer, s3, and sftp up when the local unix socket cannot bind (#11999)
* pb: share the stale-socket tolerant unix listener ServeGrpcOnLocalSocket already treats the local socket as optional; pull the remove-and-listen block out so the filer, s3, and sftp http sockets can use it too. * command: keep filer, s3, and sftp up when the local socket cannot bind A stale socket file owned by another uid (e.g. left in /tmp by a containerized job) made os.Remove fail with EPERM and the Fatalf took the whole server down. The gRPC local socket already treats the unix listener as optional; do the same for the http/sftp sockets. * filer: clear shared localSocket flag when the unix socket cannot bind s3.localFilerSocket and (in weed server) the sftp local socket share the filer's -localSocket flag pointer. If the filer cannot bind the socket it now keeps running without it, so consumers still holding the configured path would dial a dead or foreign-owned socket. * command: keep sftp on its own localSocket flag under weed server sftpOptions.localSocket was overwritten with the filer's -localSocket flag, so -sftp.localSocket was ignored and an explicit filer socket path made the sftp server try to bind the same socket the filer already held. * s3: fall back to TCP when the filer local socket cannot be dialed The embedded S3 server copies the filer's -localSocket flag before the filer tries to bind it, so a socket that never comes up left every filer HTTP request dialing a dead path. Try the unix socket first and fall back to the request's TCP address on dial failure. * filer: keep the shared localSocket flag read-only after startup Clearing *fo.localSocket raced with the embedded s3 goroutine reading the same flag pointer and could not help a reader that had already copied it. The s3 filer client's TCP fallback covers the unbound-socket case without mutating shared flag state. |
||
|
|
ad89f003a5 |
s3: end a catalog-vended session with the access key it was vended for (#11945)
* s3: end a catalog-vended session with the access key it was vended for A table credential the Iceberg catalog vends is a self-contained STS session: it stays valid until it expires (an hour by default) even after the access key that loaded the table is deleted or deactivated, or its identity is removed or disabled. An operator revoking a leaked key cannot cut off the table credentials that key already obtained. Sessions are stateless JWTs, so there is no revocation list to consult. This change follows the same pattern as role sessions, which are already refused once their role's ID no longer matches (checkSessionRoleBinding): a vended session carries the access key it was vended for (src_ak, recorded by the previous change), and validateSTSSessionToken checks it against the identity table held in memory. A session of provider iceberg-catalog that carries a source key is refused (InvalidAccessKeyId, as for an unknown key) once its identity is gone or disabled, or the key is no longer one of that identity's active, unexpired credentials. Scope: - Only catalog-vended sessions are checked; web-identity, AssumeRole and every other STS session are validated as before. - The vendor binds a session to the caller's key only when the caller is a stored identity holding that key. A caller that signed with another STS session's key, an OIDC user, or a request that recorded no key gets an unbound session, as today; binding it would refuse every request made with it. A session minted before this change carries no key and is unbound. - The check is one map lookup under the read lock and a scan of one identity's keys (about 11 ns, 0 allocations, with 1,000 identities). Tests (s3api_iceberg_vended_session_test.go) mint real sessions through the catalog vendor and validate them the way a data-path request is validated: deactivating, expiring or deleting key A ends A's session while the session vended through key B on the same identity keeps working; disabling or deleting the identity ends it; callers that are not a stored identity holding the key get unbound sessions that keep working; other STS sessions are not checked. The first two tests and the third fail without this change. * s3: keep the vended-session key binding when the key dies before vend vendedSessionKeyLive gated the binding, so a key revoked between the request's authentication and the vend skipped it and minted an unbound session that stayed usable until it expired. Every catalog auth path pairs the identity name in context with the key that authenticated, so a stored-identity principal always means the request signed with that identity's own key. Bind whenever the principal is a stored identity: a session bound to a now-dead key is refused on its first request, which is what the binding is for. * s3: gate the vended-session key binding on the authenticated caller type lookupByIdentityName(principal) named only that some identity exists, not that the request signed with a stored key: an STS session or OIDC principal that collides with an identity name got a session bound to its temporary key, which vendedSessionKeyLive then refused on every request. Verifiers now record the verified key as a stored credential, and the vendor binds only when that marker is present — keeping the binding even when the key was revoked between authentication and the vend. * s3: enforce the vended key binding under a dedicated claim Sessions minted before the binding existed carry the caller's key in the audit-only SourceAccessKeyId — including temporary STS keys that can never resolve in the identity table. Enforcing on that field would end live table credentials at upgrade; the binding now travels under BoundAccessKeyId, leaving legacy sessions unbound as before. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: bind vended sessions to a credential fingerprint The key id alone cannot identify a credential: an operator can delete a key and re-create it under the same id with a different secret, reviving sessions minted for the original. Verifiers now record a fingerprint of the secret at authentication; the session dies unless it still matches. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: keep recording the caller's key on every catalog session SourceAccessKeyId was unconditional audit metadata before the binding existed; gating it on the stored-credential marker dropped the record for callers that authenticated another way. With the binding under its own claim there is no ambiguity left, so the field records the caller's key again regardless of how it authenticated. * s3: key the vended-session credential fingerprint with the signing key The bound-credential fingerprint travels inside the vended session token. As a plain SHA-256 of the secret it gave any token holder an offline guessing oracle for weak secrets, and CodeQL flags it twice. Store the verified secret in the request marker instead and HMAC it with the STS signing key at vend and at validation. The claim can no longer be mined, and the per-request auth path stops hashing secrets. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
bf3081e23a |
s3 lifecycle: route version-file writes to current-version expiration (#11956)
* s3lifecycle: schedule ExpirationDays from version-file writes On a versioned bucket a PUT lands in the meta-log as the version file plus the .versions/ directory update — there is no bare-key event, so the version-folder skip dropped every version-file event for all kinds but ExpiredDeleteMarker. ExpirationDays therefore only fired via the recovery walker's cold-start / rule-change pass: once a shard had a saved cursor, no daily run ever scheduled another delete marker. A version-file write is the latest for its logical key at write time, so route it through the same event-driven evaluation the regular path runs. The match carries the file's identity with an empty VersionID — the dispatcher resolves the current latest at due time, and a fresher PUT meanwhile resolves to STALE_IDENTITY there. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * lifecycle test: versioned Expiration fires on the second daily run Establishes the shard cursor with a first pass, then PUTs and backdates a new versioned object and runs again. The delete marker must appear without a rule change forcing a recovery walk. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: run TestLifecycleVersionedBucketExpiresOnSecondRun in the lifecycle matrix --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
b65832bdc9 |
iam/oidc: reject reused jti claims with a bounded replay cache (#11936)
* iam/oidc: reject reused jti claims with a bounded replay cache * iam/sts: pass jti replay settings through convertToOIDCConfig * iam/oidc: keep consumed jtis and stop the cleanup worker across re-init Re-initializing a provider reset the JTI store, so an already-used token could be presented again after a config refresh; the store is now allocated once and survives Initialize. The cleanup worker is stopped on every re-init (previously only when protection stayed enabled), is started only after the rest of Initialize has succeeded, and Shutdown no longer consumes a permanent sync.Once, so a re-initialized worker can be stopped again. * iam/oidc: enforce the jti replay cache bound for every token Tokens without a usable jti claim skipped the cache entirely and stayed reusable; they are now keyed by the token's own fingerprint so a repeat presentation is still a replay. Tokens without exp previously failed as expired on first use; their entry now gets a bounded retention. The replay test-and-set now runs before any capacity work so replayed tokens cannot each trigger a full-cache scan, the entry count is incremented as part of the insert so concurrent validations cannot overshoot the bound, and a guaranteed serialized sweep runs before a capacity rejection while opportunistic sweeps are rate limited and confined to one at a time. Raw token identifiers are no longer written to logs or errors, a stable fingerprint is logged instead. * iam/sts: shut down providers dropped by the managed-provider swap Each managed OIDC provider runs a background JTI cleanup worker, but SetIAMManagedOIDCProviders replaced the provider map without stopping the old providers, so every refresh leaked goroutines. Providers absent from the new map are now shut down; providers that stay keep running. * iam/integration: reuse providers across refreshes for unchanged records Every refresh rebuilt providers from their records, so a refreshed provider forgot the JTIs it had already consumed and an already-used token could be presented again. Records are now fingerprinted and an unchanged record reuses its provider, keeping the consumed-token cache; changed records are rebuilt, removed records are dropped, and the swap in STS releases the workers of whatever is no longer referenced. * iam/oidc: consume replay entries only on the single-use exchange path Bearer authentication validates the same token on every request, so consuming it in shared ValidateToken/Authenticate made every token single-use and rejected all but the first request. The replay check now runs only for the STS exchange via AuthenticateSingleUse; bearer paths validate without consuming. A token without exp also no longer gets a bounded replay entry that ages out while the token stays valid: its entry is kept until the capacity bound evicts it, so it can never become usable again. * iam/oidc: keep the mock provider on the mock path for single-use calls * iam/oidc: require exp on single-use tokens A cache slot frees only by expiry, so an expiry-less entry would hold it forever; a burst of distinct tokens could exhaust the bound permanently and block every later exchange. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: give each web-identity test token a unique jti Subtests minted tokens with identical claims inside one second, so the exchange saw the same token fingerprint twice and the second call read as a replay. Each exchange now carries its own jti. * iam/oidc: carry the consumed-token replay store across provider rebuilds A record change rebuilds the managed provider with a fresh JTI store, so a token consumed before the rebuild could be presented again until it expired. The new provider adopts the replaced provider's replay store before it is published, and the replaced provider is shut down so only one cleanup worker sweeps the shared store. * iam/oidc: stop the cleanup worker of providers dropped from the store A refresh removed the provider object from the managed cache but left its JTI cleanup goroutine running for the life of the process. * iam/oidc: merge a provider's consumed replay entries into another's store MergeReplayStateFrom folds one provider's consumed-token entries into another's, skipping already-expired ones. Unlike AdoptReplayStateFrom it swaps no pointers, so it is safe on a provider still serving validators. * iam/integration: share one provider and replay store across identical issuer records Records for the same issuer URL with identical validation settings now reuse a single provider, so a token consumed through one account's record is rejected through the other's — previously each record owned a separate jti store and a spent token could be replayed cross-account. A superseded provider's consumed entries merge into the shared store so no rejection is lost, and a removed or rebuilt provider is only shut down once no remaining record references it. * iam/oidc: hand the superseded provider's live replay store to the shared one * iam/oidc: carry merged replay entries without charging the capacity bound * iam/oidc: key the jti-less replay fingerprint on the signed input Hashing the whole token lets ECDSA signature malleability (s -> N-s) mint a second cache key for identical claims, so a consumed token could be replayed for another exchange. Key the fallback on header.payload instead. * iam/oidc: publish the jti cache bound before the enabled flag Initialize stored jtiEnabled ahead of jtiMaxSize, so a provider flipped from disabled to enabled exposed enabled=true with a zero bound for a window; a single-use validation landing there would compute newCount > maxSize and reject with "JTI cache capacity exceeded". Store the bound first so the flag is the last thing published. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
186bc5cf7f |
s3: bound ListMultipartUploads memory to the page size (#11971)
* s3: bound ListMultipartUploads memory to the page size listMultipartUploads loaded every open upload in the bucket into a slice and selectMultipartUploads filtered and sorted a second full copy, so one request cost O(total open uploads) heap regardless of prefix, key-marker, or max-uploads; concurrent calls multiplied it and could OOM the gateway. The response is ordered by the object key kept in extended attributes, not the upload id the filer sorts by, so every entry still has to be seen. Streaming the listing into a bounded top-K collector keeps at most maxUploads (capped at maxUploadsList) candidates, bounding heap by the page instead of the folder. A caller-supplied max-uploads above the server cap now also paginates rather than accumulating unboundedly. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: cover bounded ListMultipartUploads accumulation Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: report the effective max-uploads cap in ListMultipartUploads * s3: cover ListMultipartUploads retry through the collector --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
f2d5c27f65 |
operation: renew the write JWT when a fan-out rollback delete is rejected (#11970)
* operation: renew the cleanup JWT when a fan-out rollback delete is rejected A stalled holder can out-wait the assignment JWT's 10s expiry, so the type=replicate delete that rolls back landed copies comes back 401 and the needle leaks. Retry the delete once with a fresh token minted through the new LookupJwtFunc hook when a holder answers unauthorized. * s3api: wire a cleanup-JWT renewer into chunked uploads Keep the master client on the server and use it to mint fresh volume JWTs for rollback deletes that run past the assignment token's expiry. * admin: wire a cleanup-JWT renewer into file-browser uploads * operation: bound the cleanup-JWT renewal with a per-attempt deadline A rollback delete that hits 401 renewed the token through a background context, so a master outage could park the wait for a connection and the lookup RPC inside the failed upload's cleanup path. Thread a context through LookupJwtFunc and LookupJwt, and bound each renewal with a 10s deadline covering both the master wait and the lookup call. * operation: let every holder's 401 retry with a renewed cleanup JWT A single renewed flag meant only the first rejecting holder refreshed the token; a holder that stalled past the renewed JWT's lifetime left later copies stranded with a 401 they could not resolve. The retry limit is now per holder — each delete is attempted once with the current token and once more after a bounded renewal — so a slow middle holder cannot strand the rest. * operation: log the volume server's error body on a failed cleanup delete Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * operation: count test upload attempts atomically The chunk upload loop fans AssignFunc out to parallel goroutines, so the plain int counter in the partial-results test raced under -race. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mohd Quamar Tyagi
|
94dd96a322 | weed filer.remote.sync: ignore .system directory under buckets (#11966) (#11969) | ||
|
Chris Lu
|
00e64441c6 |
fix: clone protobuf snapshots without copying internal locks (#11968)
* fix: clone protobuf snapshots without copying internal locks * fix: reuse cloned snapshot fields when filtering |
||
|
Chris Lu
|
8f6121694a |
test: fix HEAD bucket assertions and FUSE failover startup timing (#11967)
test: fix HEAD bucket assertions and warm up failover clusters |
||
|
Chris Lu
|
137d29633b | s3: fix prefix filtering and delimiter pagination (#11962) | ||
|
|
7603528aff |
s3api: support s3:RequestObjectTag/<key> in bucket policies (#11910)
* s3api: support s3:RequestObjectTag/<key> in bucket policies The tags come from X-Amz-Tagging only where the handler stores them from there: not on PutObjectTagging, which reads the body, and not on a copy that keeps the source's tags. For a repeated key the last value counts, since that is what CopyObject stores. * s3api: extract RequestObjectTag only where the handler stores it The header was honored on any request carrying it, so a tag attached to a GET, DELETE, part upload, multipart completion, directory marker or a PUT subresource satisfied a tag-conditioned policy the object never receives. Extraction now follows the handlers: PutObject, a copy under X-Amz-Tagging-Directive: REPLACE, and CreateMultipartUpload store the header; a plain object PUT is identified by carrying only X-Amz-* signing parameters. * s3api: exclude multipart part PUTs before the copy-source check A PUT carrying X-Amz-Copy-Source plus a numeric partNumber and uploadId routes to CopyObjectPartHandler, which stores a part and never reads X-Amz-Tagging. Checking the copy source first let such a request satisfy an s3:RequestObjectTag condition under REPLACE with tags that are never stored. Mirror the router: the part handlers match only when both partNumber (digits only) and uploadId are present, so a lone or non-numeric partNumber still falls through to the storing handlers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: gate tag extraction on the routed subresource, not query key shape Rejecting every PUT whose query held a key outside X-Amz- dropped X-Amz-Tagging from ordinary object PUTs that carry arbitrary parameters — Signature V2 presigns (AWSAccessKeyId/Signature/Expires) among them — so a tag-conditioned Deny could be bypassed by appending an unrelated query parameter while putToFiler still stored the tags. Exclude the object PUT subresources the router actually dispatches (tagging, renameObject, acl, retention, legal-hold) ahead of the copy-source check instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: cover the routed-subresource and part-copy tag extraction cases Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
cec96c02ea |
s3: reject requests carrying more than one routed subresource (#11942)
* s3: reject requests carrying more than one routed subresource Routing picks the first matching subresource route while authorization resolves a single action from the query, and the two can disagree. A bucket-level PUT ?tagging&policy is authorized as s3:PutBucketTagging but served by PutBucketPolicyHandler, letting a tagger rewrite the bucket policy; the same shape lets a list grant read the policy via ?list-type=2&policy or ?versions&policy, and lets an object ACL writer retag via PUT ?acl&tagging. Declining ambiguous requests keeps the authorized action and the served handler identical. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: reuse hasAmbiguousSubresource in the routed-subresource guard The standalone key list dropped versioning, so ?versioning&policy still reached route matching and auth with two operations. The bucket-level check already keys off operationSubresources plus bucketQueryActions; running the same predicate keeps the lists from drifting apart. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
7ff77a4f3b |
s3api: skip error response body when http forbids it (#11931)
WriteResponse wrote the payload even for HEAD requests and statuses that disallow a body (1xx, 204, 304), which made net/http return ErrBodyNotAllowed and logged 'write err: http: request method or response status code does not allow body' for ordinary requests. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
74da88fc29 |
s3, iceberg: record the access key a request was signed with (#11901)
* s3, iceberg: record the access key a request was signed with Several access keys can belong to one identity, but nothing downstream of authentication says which one signed a request: the audit log carries the identity name only, and every table credential the Iceberg catalog vends for that identity gets the same session name. When an identity is shared (one key per person or per service), the log cannot tell the holders apart. The verifier that matched a credential now records that credential's access key id in the request's identity holder (verifyV4Signature for header, presigned, streaming and STS requests; both SigV2 verifiers). Nothing re-reads the request for a key, so a second, unverified key elsewhere in the request (a junk X-Amz-Credential on a SigV2 request, a junk SigV4 header on a SigV2 presigned URL) is never recorded. - s3err.AccessLog gains access_key_id (omitempty): a static key's id, or an STS session's temporary key id. Every other field is unchanged. - The Iceberg and Lance catalogs install the holder before authenticating a SigV4 request; a Bearer token records the key it was minted from (the token is verified with a key derived from it); a Lance x-api-key records its key. - VendTableCredentials ends the session name with "-k" and the first 12 hex digits of SHA-256 of the key id (the table part gives way within the 64-character limit; the tag is never cut), and carries the key id in the session token (src_ak, SessionInfo.SourceAccessKeyId). Tests: auth_access_key_attribution_test.go and auth_access_key_shapes_test.go (two keys on one identity, the deny path, SigV2 and presigned SigV2 with a junk key elsewhere, streaming seed, STS session, failed signatures record nothing), s3api_iceberg_session_name_test.go, iceberg and lance Auth tests, sts source_access_key_test.go. * s3: record the access key of a verified POST-policy form and of a retargeted signature Two verified signatures left access_key_id empty in the audit log: - A browser POST upload is verified in PostPolicyBucketHandler from its form fields by doesPolicySignatureV2Match / doesPolicySignatureV4Match, which did not record the key. Both now take the request context and record the credential's key once its signature verifies; a form that fails verification records nothing. - verifyV4Signature refuses a signature that verified for one virtual-hosted bucket but was routed to another. The key was recorded only after that check, so the refusal carried none. It is now recorded as soon as a host candidate verifies. Test: TestPostPolicyUploadRecordsTheSigningKey (SigV4 and SigV2 forms, and a form with a wrong signature) and TestSignedVirtualHostRetargetRecordsTheSigningKey; both fail before this change. * s3: test VendTableCredentials end to end for the caller's access key TestVendTableCredentialsRecordsTheCallersKey mints sessions through VendTableCredentials with a configured STS service, the key recorded the way the catalog's authentication records it, then validates each session token: SourceAccessKeyId and the session name's key tag match the key, and a caller without a key gets an untagged session. It fails if the vendor stops passing either field. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
df33e81915 |
volume: report the needle read error with a failed read (#11934)
* volume: report the needle read error with a failed read A read that fails on the volume server, for example on a CRC mismatch, returned a bare 500, so mount and filer logs only showed the status. The volume server now writes the error into the response body, and the read helpers add it to the error they return. Co-authored-by: Bruce <82158894+half-life666@users.noreply.github.com> * volume: keep the 500 bare on HEAD requests writeJsonError's body write fails with ErrBodyNotAllowed under net/http when the request method is HEAD, adding an error-writing log line on top of the failed read. Send only the status, matching the no-error path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * util/http: decode gzip-encoded error bodies in statusError Full-chunk reads set Accept-Encoding: gzip themselves, so the transport does not transparently decompress; a gzip-encoded error body reached statusError as compressed bytes and produced an unreadable message. statusError now takes the response and decodes Content-Encoding before extracting the bounded message. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: log the read error when the 500 stays bare On HEAD requests the body write is skipped, which also skipped the V(0) response log inside writeJson — a failed HEAD read left no cause in normal volume-server logs. Log the error in the same format before the bare 500. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * util/http: check the status before decoding gzip in GetAuthenticated The gzip reader eagerly consumes the header when it is created, so statusError saw a drained or corrupt body on a gzipped error response and the server message was lost. Match the other read helpers and check the status first. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * util/http: export StatusError Other volume read paths outside this file need the same error body. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: redact jwt query param when logging failed-read URLs The new read-failure logs print the request URL, and volume reads accept ?jwt= as a credential, so a failed authenticated read would write the bearer token to the volume server log. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * util/http: surface the error body on the remaining chunk-read paths * volume: redact duplicate jwt params in failed-read logs * operation: pass the lookup jwt on the chunk needle read --------- Co-authored-by: Bruce <82158894+half-life666@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
1aad4cfe70 |
iam: optionally require an iam: grant for self-service access-key actions (#11937)
* iam: optionally require an iam: grant for self-service access-key actions
AuthorizeIamAction lets every authenticated identity run CreateAccessKey,
DeleteAccessKey, UpdateAccessKey, ListAccessKeys and GetUser on itself with
no grant. On AWS these actions need an IAM policy (the usual one grants them
on arn:aws:iam::*:user/${aws:username}); nothing is allowed implicitly.
Where identities hold data-plane rights only, the implicit carve-out lets a
leaked key mint further keys for its own identity, with no cap, that keep
working after the operator deletes the key that leaked; it can also
deactivate or delete the keys the operator issued.
A new option, off by default, makes these actions need the iam:<Action> grant
like every other IAM action: -s3.iam.selfServiceRequiresGrant on weed
server, filer and mini, -iam.selfServiceRequiresGrant on weed s3. With it
set, the grant is authorized against the target user's ARN, the caller's own
when UserName is empty, so a policy can confine it to the caller's own keys
(Resource arn:aws:iam:::user/${aws:username}), the same resource scoping
CreateServiceAccount already uses. The admin and iam:* grants are unchanged;
with the option off nothing changes.
Tests (s3api_embedded_iam_authz_test.go): with the option set, an identity
holding only s3:* and a bare identity are refused all five actions on an
empty, own and other target (fails without the change); a grant on the
caller's own user ARN, literal or ${aws:username}, allows them on itself and
not on another user; the admin and an iam:* grant keep every action; with the
option off self-service is allowed as before.
* iam: wire selfServiceRequiresGrant on the standalone iam server
Every other command that runs the embedded IAM API accepts the flag; the
deprecated weed iam could not enable the enforcement at all.
* iam: wire selfServiceRequiresGrant on weed server and filer -iam
The IamOptions used by 'weed server -iam' and 'weed filer -iam' never
received the flag, so startIamServer dereferenced a nil pointer at
startup. Share the -s3.iam.selfServiceRequiresGrant pointer with them.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
|
||
|
Chris LuandDevin
|
c0152e6be1 |
s3: keep '\' a literal object-key character behind -s3.allowBackslashInKey (#11911)
* s3: keep '\' a literal object-key character when AllowBackslashInKey is set NormalizeObjectKey rewrites every S3 object key: '\' becomes '/' and duplicate slashes collapse. On AWS S3 both are ordinary key characters, so keys that are distinct there alias one object here — writing probe/back/slash.txt silently overwrites probe/back\slash.txt. The filer only treats '/' as a path separator, so a verbatim '\' stays inside one entry name. Gate the backslash rewrite and the matching '\' handling in IsValidObjectKey behind a process-wide mode the S3 server sets from the new S3ApiServerOption. Duplicate-slash collapsing stays: an entry name cannot contain an empty segment. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * command: add -s3.allowBackslashInKey flag for AWS-compatible key handling Registers the flag on all four S3 entry points (s3, server -s3, filer -s3, mini) and passes it through S3Options into S3ApiServerOption. Default off, so existing normalization is unchanged. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: correct IsValidObjectKey/NormalizeObjectKey docs under allowBackslashInKey Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: normalize DeleteObjects body keys the same as path keys DeleteObjects keys arrive in the XML body, so they bypassed the normalization applied to route-var keys. The lock, the authorization checks, the table-bucket layout check and the filer delete itself all addressed the raw key while the conditional check resolved the normalized one, so a key spelled with '\' or '//' could be reported deleted while the stored entry survived. Echo the client-sent key in the result while operating on the normalized form. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: report a normalized-empty DeleteObjects key as a delete no-op A multi-delete key of only slashes normalizes to the empty string. Left to reach the filer delete, the empty name resolves to the bucket entry itself, so an unauthorized caller could remove a bucket through the body-key path. Such a key can never name a stored entry, so like a missing key the delete is an idempotent no-op reported before any permission check or filer call. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: drop unused AllowBackslashInObjectKey getter * s3: answer conditional delete of a normalized-empty key with NoSuchKey * s3: normalize the probed key in authorizeAnonymousBucketList The masked-miss probe read the raw mux var, so a spelling like folder// or folder\ was authorized against a key that can never exist while the request itself targets the normalized folder/. Probe the normalized key so the answer matches what the read path authorizes. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: cover anonymous authz of a literal-backslash key under allowBackslashInKey A public grant on the slash key must not authorize an anonymous read of the distinct backslash object when the flag keeps '\' literal; grant the backslash key's own ACL and it reads back. Default mode still folds the spelling onto the covered slash key. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: publish the key mode only after the S3 server is constructed A failed NewS3ApiServerWithStore could previously flip the process-wide key mode out from under an already-running gateway. * s3: normalize the copy-source key before validation and authorization pathToBucketObjectAndVersion returned the raw header spelling, so an equivalent spelling (a//b, a\b) was authorized and compared against the normalized destination while resolveCopySourceEntry looked up the normalized key — a deny on the canonical key could be bypassed and self-copy detection missed. Normalize once at parse so the CopyObject and UploadPartCopy paths agree. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
917b6f4afd |
iceberg: authorize a commit before staging its metadata file (#11891)
* iceberg: authorize a commit before staging its metadata file Both commit paths (CommitTable and CommitTransaction) write the new vN.metadata.json into the table's metadata/ directory and only then reach UpdateTable, whose authorization refuses a caller that may read the table but not commit to it. The refused commit then deletes the staged file on the way out, so such a caller could make the server write and delete a file under the table's prefix on every attempt. The same shape was closed for creates in #10991. Both paths now ask Manager.AuthorizeUpdateTable right after loading the table and answer 403 ForbiddenException before anything is staged. UpdateTable still decides under the policies in force when the pointer flips, so a policy changed mid-commit is caught as before. Test: TestCommitDeniedBeforeStagingMetadata records every file the handlers create; on master both the single-table and the transaction commit by a GetTable-only identity write v2.metadata.json before answering 403, with the change nothing is written. TestCommitAllowedStagesMetadata is the positive control (the owner, and an identity whose own actions include UpdateTable). * iceberg: authorize a view commit before staging its metadata handleUpdateView has the same write-before-authorize gap the table commit paths had: it stages the new metadata file and only reaches the UpdateView authorization at the pointer flip, so a caller who may only load the view could still make the server write and delete files under it. Gate it on AuthorizeUpdateView, mirroring authorizeCommit. * iceberg: seed committed view metadata in the denied-commit test Without stored view metadata the request fails on the missing-metadata check before it could stage a file, so the zero-write assertion could not catch a write-before-denial regression. Seed the same metadata the allowed-path test uses so the denied request would otherwise stage. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
106fd31af1 |
s3: resolve versionId ACL and attributes requests to the version-scoped actions (#11916)
* s3: resolve versionId ACL requests to the version ACL actions A request for an object's ACL carrying versionId acted on the historical version but authorized against the current-object action (s3:GetObjectAcl/s3:PutObjectAcl), because the acl block in resolveFromQueryParameters returned before versionId was considered - the same gap closed for tagging. An identity granted only current-object ACL access - or explicitly denied s3:*ObjectVersionAcl - could still read and modify ACLs on noncurrent versions. ?acl&versionId=<id> on an object now resolves to s3:GetObjectVersionAcl/s3:PutObjectVersionAcl; ?acl alone keeps the current-object actions and bucket ACL is unaffected. s3:PutObjectVersionAcl is mapped onto the WriteAcp identity action so policies can name it. * s3: resolve versionId attributes requests to the version attributes action GET /bucket/key?attributes&versionId=<id> fetched the historical version's attributes but authorized against s3:GetObjectAttributes, so a grant limited to the current-object attributes action could reach noncurrent versions. The request now resolves to s3:GetObjectVersionAttributes, and both attributes actions map onto the Read identity action so policies can name them (s3:GetObjectAttributes was previously rejected as an unmapped action). * s3api: recheck the version action on versioned ACL/attributes requests The phase-2 policy recheck inside GetObjectAclHandler, PutObjectAclHandler, and GetObjectAttributesHandler passed the current-object action even when the request carried versionId and phase 1 had resolved the version action. A Deny scoped to s3:*VersionAcl or s3:GetObjectVersionAttributes could never fire in the recheck, and an Allow limited to the version action went unevaluated. Select the action the same way the resolver does — versioned whenever versionId is present. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: recheck the version action on versioned tagging requests The same phase-2 gap exists in the tagging handlers left by the earlier version-tagging resolution change: versioned GetObjectTagging / PutObjectTagging / DeleteObjectTagging rechecks evaluated the current-object tagging action, so a Deny on s3:*VersionTagging could not fire. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: resolve HEAD ?acl as the head-object action, not an ACL action No HEAD acl route exists, so HEAD /bucket/key?acl lands on HeadObjectHandler — which serves object headers, not the ACL. Resolving it to an ACL action denied readers who hold only GetObject(Version) and let ACL grants reach a head-object read. Dropping HEAD from the acl branch lets the request fall through to GetObject/GetObjectVersion, matching how the tagging branch already treats HEAD. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: require the version read grant for versioned attributes AWS asks for s3:GetObjectVersion alongside s3:GetObjectVersionAttributes when a GetObjectAttributes request names a versionId; the resolved version-scoped action on its own left a lone s3:GetObjectVersion Deny unable to cover the attributes read. Check the version read grant too: the identity side through VerifyActionPermission, the bucket-policy side through the same recheck that evaluates the scoped action. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: check the version read grant on a synthesized versioned GET The versioned attributes check passed the literal s3:GetObjectVersion action, but both policy paths re-resolve the action from the request, so the attributes query parameter mapped it back to s3:GetObjectVersionAttributes and a lone version-read Deny never fired. The literal action also reached the legacy CanDo path, where a plain Read grant no longer matched. Evaluate the grant on a request shaped like a plain versioned GET instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: reject a named versionId on buckets without versioning Tagging, ACL, and attributes handlers answered versionId requests on never-versioned buckets as the current object while evaluating the version-scoped action, so a grant limited to the version action could authorize changes to the live entry. Reject the request up front through the existing checkVersionIdConfigured gate. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: carry presign params onto synthesized policy requests objectKeyPolicyRequest rebuilt the query with only versionId and the STS token, so a presigned request's s3:authType condition value silently changed from REST-QUERY-STRING to unset during phase-2 rechecks (version read grant, copy source, delete keys, post-policy writes). A policy conditioned on s3:authType would then deny a request it already allowed. Carry the SigV2/SigV4 presign keys through; none of them steer ResolveS3Action. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Javier de la TorreandJavier de la Torre
|
de2fc03c7d |
s3tables: accept the external-volume verification object at a table bucket's root (#11900)
* s3tables: accept the external-volume verification object at a table bucket's root Snowflake verifies an S3-compatible external volume before it creates or writes Iceberg tables in it (SYSTEM$VERIFY_EXTERNAL_VOLUME, also run implicitly): it writes, reads, lists and deletes one small object, <base>/verify_<epoch milliseconds>_<number>, a 21-byte body. A table bucket refuses every key outside the Iceberg layout, so the check failed and Snowflake could not use a table bucket as an external volume. PUT, GET, HEAD, DELETE and DeleteObjects now accept exactly that key in a table bucket, narrowly enough that it cannot be used to store data: - at the bucket root only, so nothing is encoded in a prefix, written into a hidden tree or placed on a namespace or table name; - one shape: verify_<13 digits of epoch ms within one hour of the server's clock>_<1-10 digits>; - a PUT must declare a Content-Length of at most 1 KiB (no chunked or undeclared body) and carry no user metadata, tagging, Content-Disposition, Cache-Control, Expires or website redirect. Multipart, copy, rename, ACL, tagging and POST policy writes keep the plain layout rule. Buckets that are not table buckets are unaffected. Tests: table_bucket_verify_object_test.go, the key accepted (with the Iceberg layout as the positive control and the plain rule still refusing it), 15 look-alikes refused (prefixed, hidden tree, wrong shape, outside the window, ordinary keys), and seven PUT refusals through PutObjectHandler. * s3tables: keep the verification key off namespaces and narrow its PUT A namespace may be named like a verification key (verify_<ms>_<n>), and the key then names the namespace's directory. A PUT replaced that directory with a prefix object, dropping the namespace's catalog metadata; a DELETE removed an empty namespace. The key now gets the plain layout rule when it names a directory or its entry cannot be read, and a PUT of it never falls back to a prefix object, so a namespace created after the check is not replaced either. The time window applies to PUT only: GET, HEAD and DELETE take a key of any date, so an object a check left behind for more than an hour can still be read and removed. A verification PUT also refuses an extraction request and the stored Content-Language, Response-Content-Disposition and Content-Encoding headers (aws-chunked framing alone is still accepted). The integration test runs the check's PUT, HEAD, GET, list and DELETE against a table bucket, and the PUT and DELETE over a namespace of the key's shape. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
Chris LuandDevin
|
9a03c09396 |
s3api: cover the inline-content UploadPartCopy route at handler level (#11914)
s3api: exercise the inline-content route through CopyObjectPartHandler Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Javier de la TorreandJavier de la Torre
|
f40267db44 |
s3: answer InvalidBucketName to a denied request on a malformed bucket name (#11893)
* s3: answer InvalidBucketName to a denied request on a malformed bucket name The store authorizes a request before it looks at the bucket, so a caller denied on a bucket whose name breaks the S3 naming rules (upper case, an underscore, too short, two adjacent periods, a non-ASCII letter) gets 403 AccessDenied, where AWS S3 answers 400 InvalidBucketName. Clients and compatibility suites that check the error code read the 403 as a permission problem. Auth and AuthPostPolicy now turn an AccessDenied for a resolved identity into InvalidBucketName when the bucket name fails VerifyS3BucketName or uses a character outside a-z, 0-9, '.' and '-' (VerifyS3BucketName checks with unicode.IsLower/IsNumber, which accept letters such as 'ü'). The answer depends on the name alone, so it reveals nothing about which buckets exist. An authentication failure, an allowed request and a denial on a well-formed name answer as before. Test: TestAuthDeniedInvalidBucketName drives iam.Auth with signed and anonymous requests; on master every malformed name answers 403, with the change 400, while the identity's own bucket stays allowed and a well-formed foreign name stays 403. TestRefineDenialForBucketName covers the rule table. * s3: refine POST-policy and deferred anonymous-read denials on a malformed bucket name Two denial paths bypass the refinement in Auth and AuthPostPolicy: - A browser POST upload carries its credentials in the form, so AuthPostPolicy delegates it to PostPolicyBucketHandler, which verifies the form and writes AccessDenied itself when the signer may not write the bucket or the per-key authorization denies the object. Both denials now go through refineDenialForBucketName with the verified identity. The coarse Write check used to run inside doesPolicySignatureV2Match and doesPolicySignatureV4Match before the signature was compared, so a form with a wrong signature was told whether the key may write the bucket. It now runs in doesPolicySignatureMatch after the signature verifies, as on the header-signed path; a form that fails verification answers SignatureDoesNotMatch (or MissingFields) and is never refined. - Unsigned GET/HEAD object reads are deferred by AuthWithPublicRead to the object handlers, where writeAnonymousAwareNotFound masks a miss as AccessDenied. That denial is refined with the anonymous identity, so a missing malformed bucket answers InvalidBucketName when an anonymous identity is configured, as Auth does. A denial on an entry that was found is left alone: the bucket exists. Test: TestPostPolicyDeniedInvalidBucketName drives AuthPostPolicy with signed forms (Write denied, per-key policy deny, bad signature, well-formed name); TestAnonymousReadDeniedInvalidBucketName drives AuthWithPublicRead with GET and HEAD, with and without an anonymous identity. Both fail before this change. * s3: refine a denied anonymous read of an existing key on a malformed bucket name A deferred anonymous GET/HEAD refined only its masked miss to InvalidBucketName, so in a bucket whose name breaks the S3 rules (one VerifyS3BucketName still accepts, such as a non-ASCII name) a missing key answered 400 and an existing private key 403, telling the caller which keys exist. The entry check now refines its denial the same way, so both answer alike. Without an anonymous identity no identity resolves and both stay 403, as in Auth. The deny-precedence case in TestAnonymousObjectACLConfiguredIdentity runs on the fixture bucket "b", shorter than S3's three-character minimum, so its denial now answers InvalidBucketName. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
Chris LuandDevin
|
6ee7c4c5c7 |
s3: no empty-directory marker in a "<dir>/" self-prefix listing (#11906)
An out-of-band empty directory listed under its own trailing-slash prefix answered the probe with a fake entry of itself: a Contents key on 4.48 and, since recently, a CommonPrefix. That listing enumerates children, where S3 returns nothing for an empty prefix, and clients that read every returned key (Pulumi lock listing) choked on the marker. The prefix object and directory key object paths are unchanged: a real "dir/" object still lists as its own key. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Javier de la TorreandJavier de la Torre
|
97f8ca13ff |
s3tables, iceberg: log client refusals at V(1), not ERROR (#11896)
* s3tables, iceberg: log client refusals at V(1), not ERROR Requests the S3 Tables and Iceberg REST APIs correctly refuse with a 4xx (access denied, not authorized, no such namespace, already exists, not empty, a commit conflict) were logged with glog.Errorf, so a client probing for what it may do, or retrying an optimistic commit, filled the ERROR log and hid real faults from anyone alerting on it. - s3tables: HandleRequest wraps the ResponseWriter in a statusRecorder (with Unwrap for http.ResponseController) and logs a handler error by the status it answered: 4xx at V(1) as "<op> refused (<status>)", 5xx or no status at ERROR as before. - s3tables: the "handleCreateNamespace called" trace moves from ERROR to V(3). - iceberg: writeManagerError's mapping is split into classifyManagerError, shared with a new logManagerError that logs 4xx-mapped errors at V(1) and everything else at ERROR; used by CreateNamespace, CommitTable and CommitTransaction. - storage: loading a remote-tier volume logged "readSuperBlock remote volume N: <nil>" at ERROR even when the read succeeded; it is now logged only when there is an error. Tests: TestRequestRefusedWith4xxIsNotLoggedAsError, TestLogRequestErrorByStatus, TestStatusRecorderKeepsTheFirstStatus, TestStatusRecorderUnwrapsForResponseController, TestNoTraceLoggedAtErrorLevel (fails on master on the CreateNamespace trace), TestLogManagerErrorRefusalsAreNotErrors, TestRemoteSuperBlockLogOnlyOnError. * s3tables: do not log a malformed CreateNamespace body at ERROR handleCreateNamespace logged a body it could not decode at ERROR before answering 400; the dispatcher already logs that refusal once at V(1). Also drop statusRecorder.Flush: it made the recorder an http.Flusher even over a writer that cannot flush, so http.NewResponseController reported success instead of ErrNotSupported. Unwrap already lets the controller reach the wrapped writer. * iceberg: require exactly one log line per classified manager error The test accepted a refusal that produced no log, or both logs. Require one V(1) refusal and no ERROR for a 4xx, and the reverse for a fault. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |