mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-06 12:07:51 +01:00
5dfda3645ead2003ff85cd2f236bb296af39fc22
87
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
d431c1b35a |
s3api: let the vectors route, not the credential scope, raise the body limit (#12326)
SigV4 verification picked the 32 MiB S3 Vectors body limit from the credential scope, which is not verified until the body is hashed. A request claiming the s3vectors scope on any other route could make the gateway buffer 32 MiB before authentication instead of 10 MiB. The vectors handler now sets the limit on the request context, and every other route keeps 10 MiB. Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
02d22800aa |
s3vectors: accept bodies up to the 32 MiB limit, report larger ones as a size error, log engine failures (#12320)
* s3api: hash an s3vectors body up to the Vectors API limit SigV4 verification of a non-S3 request hashed at most 10 MiB of the body and silently dropped the rest, so any S3 Vectors request over 10 MiB failed its signature check with 403, although the API accepts 32 MiB. The limit now follows the signing service, and a body over it is reported as ErrEntityTooLarge instead of being truncated; the embedded IAM answers that with InvalidInput, as it already does for a body over its own limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: answer an oversized request with a ValidationException A body over 32 MiB was refused as "403 AccessDenied: authentication failed". A declared Content-Length over the limit is now refused before authentication, and a streamed body over it, whether the signature check or the handler reads it, gets the same "request body exceeds 32 MiB" ValidationException. The limit is the one SigV4 verification uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: log engine and catalog failures at the default level An engine error reached the client as a bare "internal error", and the cause was logged only with -v=1, so a misconfigured engine left nothing in a default log. Engine and catalog failures are now logged as errors, and a reported unavailability as a warning, with the operation, bucket and index. The response still leaves the engine's message out, since it can name internal hosts and paths, and points at the gateway log instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3api: hash the signed body through readRequestBody streamHashRequestBody repeated readRequestBody's read-past-the-limit check; it now reads the body with readRequestBody and hashes the bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ab204a03e6 |
fix(s3): verify SigV4 signature with decoded path behind X-Forwarded-Prefix (#12299)
* fix(s3): verify SigV4 signature with decoded path behind X-Forwarded-Prefix When SeaweedFS is hosted behind a reverse proxy that strips a URL prefix (X-Forwarded-Prefix) and forwards the percent-encoded request path, signature verification fails for object keys containing non-ASCII characters. The client signs the canonical URI as the single percent-encoded path including the prefix, but the server verifies against the already-encoded forwarded path joined to the prefix, producing a mismatch (SignatureDoesNotMatch). Decode the forwarded path and rejoin the prefix before verifying, mirroring the existing un-prefixed decoded retry. ASCII keys are unaffected because the decoded path equals the original, so the retry is skipped. Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com> * s3: build the SigV4 candidate paths once and try them per host The forwarded-prefix and plain branches each repeated the decode-and-verify step for every host candidate. List the escaped and decoded paths once, the prefixed forms first, and verify them in the same order as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ad89f003a5 |
s3: end a catalog-vended session with the access key it was vended for (#11945)
* s3: end a catalog-vended session with the access key it was vended for A table credential the Iceberg catalog vends is a self-contained STS session: it stays valid until it expires (an hour by default) even after the access key that loaded the table is deleted or deactivated, or its identity is removed or disabled. An operator revoking a leaked key cannot cut off the table credentials that key already obtained. Sessions are stateless JWTs, so there is no revocation list to consult. This change follows the same pattern as role sessions, which are already refused once their role's ID no longer matches (checkSessionRoleBinding): a vended session carries the access key it was vended for (src_ak, recorded by the previous change), and validateSTSSessionToken checks it against the identity table held in memory. A session of provider iceberg-catalog that carries a source key is refused (InvalidAccessKeyId, as for an unknown key) once its identity is gone or disabled, or the key is no longer one of that identity's active, unexpired credentials. Scope: - Only catalog-vended sessions are checked; web-identity, AssumeRole and every other STS session are validated as before. - The vendor binds a session to the caller's key only when the caller is a stored identity holding that key. A caller that signed with another STS session's key, an OIDC user, or a request that recorded no key gets an unbound session, as today; binding it would refuse every request made with it. A session minted before this change carries no key and is unbound. - The check is one map lookup under the read lock and a scan of one identity's keys (about 11 ns, 0 allocations, with 1,000 identities). Tests (s3api_iceberg_vended_session_test.go) mint real sessions through the catalog vendor and validate them the way a data-path request is validated: deactivating, expiring or deleting key A ends A's session while the session vended through key B on the same identity keeps working; disabling or deleting the identity ends it; callers that are not a stored identity holding the key get unbound sessions that keep working; other STS sessions are not checked. The first two tests and the third fail without this change. * s3: keep the vended-session key binding when the key dies before vend vendedSessionKeyLive gated the binding, so a key revoked between the request's authentication and the vend skipped it and minted an unbound session that stayed usable until it expired. Every catalog auth path pairs the identity name in context with the key that authenticated, so a stored-identity principal always means the request signed with that identity's own key. Bind whenever the principal is a stored identity: a session bound to a now-dead key is refused on its first request, which is what the binding is for. * s3: gate the vended-session key binding on the authenticated caller type lookupByIdentityName(principal) named only that some identity exists, not that the request signed with a stored key: an STS session or OIDC principal that collides with an identity name got a session bound to its temporary key, which vendedSessionKeyLive then refused on every request. Verifiers now record the verified key as a stored credential, and the vendor binds only when that marker is present — keeping the binding even when the key was revoked between authentication and the vend. * s3: enforce the vended key binding under a dedicated claim Sessions minted before the binding existed carry the caller's key in the audit-only SourceAccessKeyId — including temporary STS keys that can never resolve in the identity table. Enforcing on that field would end live table credentials at upgrade; the binding now travels under BoundAccessKeyId, leaving legacy sessions unbound as before. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: bind vended sessions to a credential fingerprint The key id alone cannot identify a credential: an operator can delete a key and re-create it under the same id with a different secret, reviving sessions minted for the original. Verifiers now record a fingerprint of the secret at authentication; the session dies unless it still matches. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: keep recording the caller's key on every catalog session SourceAccessKeyId was unconditional audit metadata before the binding existed; gating it on the stored-credential marker dropped the record for callers that authenticated another way. With the binding under its own claim there is no ambiguity left, so the field records the caller's key again regardless of how it authenticated. * s3: key the vended-session credential fingerprint with the signing key The bound-credential fingerprint travels inside the vended session token. As a plain SHA-256 of the secret it gave any token holder an offline guessing oracle for weak secrets, and CodeQL flags it twice. Store the verified secret in the request marker instead and HMAC it with the STS signing key at vend and at validation. The claim can no longer be mined, and the per-request auth path stops hashing secrets. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
74da88fc29 |
s3, iceberg: record the access key a request was signed with (#11901)
* s3, iceberg: record the access key a request was signed with Several access keys can belong to one identity, but nothing downstream of authentication says which one signed a request: the audit log carries the identity name only, and every table credential the Iceberg catalog vends for that identity gets the same session name. When an identity is shared (one key per person or per service), the log cannot tell the holders apart. The verifier that matched a credential now records that credential's access key id in the request's identity holder (verifyV4Signature for header, presigned, streaming and STS requests; both SigV2 verifiers). Nothing re-reads the request for a key, so a second, unverified key elsewhere in the request (a junk X-Amz-Credential on a SigV2 request, a junk SigV4 header on a SigV2 presigned URL) is never recorded. - s3err.AccessLog gains access_key_id (omitempty): a static key's id, or an STS session's temporary key id. Every other field is unchanged. - The Iceberg and Lance catalogs install the holder before authenticating a SigV4 request; a Bearer token records the key it was minted from (the token is verified with a key derived from it); a Lance x-api-key records its key. - VendTableCredentials ends the session name with "-k" and the first 12 hex digits of SHA-256 of the key id (the table part gives way within the 64-character limit; the tag is never cut), and carries the key id in the session token (src_ak, SessionInfo.SourceAccessKeyId). Tests: auth_access_key_attribution_test.go and auth_access_key_shapes_test.go (two keys on one identity, the deny path, SigV2 and presigned SigV2 with a junk key elsewhere, streaming seed, STS session, failed signatures record nothing), s3api_iceberg_session_name_test.go, iceberg and lance Auth tests, sts source_access_key_test.go. * s3: record the access key of a verified POST-policy form and of a retargeted signature Two verified signatures left access_key_id empty in the audit log: - A browser POST upload is verified in PostPolicyBucketHandler from its form fields by doesPolicySignatureV2Match / doesPolicySignatureV4Match, which did not record the key. Both now take the request context and record the credential's key once its signature verifies; a form that fails verification records nothing. - verifyV4Signature refuses a signature that verified for one virtual-hosted bucket but was routed to another. The key was recorded only after that check, so the refusal carried none. It is now recorded as soon as a host candidate verifies. Test: TestPostPolicyUploadRecordsTheSigningKey (SigV4 and SigV2 forms, and a form with a wrong signature) and TestSignedVirtualHostRetargetRecordsTheSigningKey; both fail before this change. * s3: test VendTableCredentials end to end for the caller's access key TestVendTableCredentialsRecordsTheCallersKey mints sessions through VendTableCredentials with a configured STS service, the key recorded the way the catalog's authentication records it, then validates each session token: SourceAccessKeyId and the session name's key tag match the key, and a caller without a key gets an untagged session. It fails if the vendor stops passing either field. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Javier de la TorreandJavier de la Torre
|
f40267db44 |
s3: answer InvalidBucketName to a denied request on a malformed bucket name (#11893)
* s3: answer InvalidBucketName to a denied request on a malformed bucket name The store authorizes a request before it looks at the bucket, so a caller denied on a bucket whose name breaks the S3 naming rules (upper case, an underscore, too short, two adjacent periods, a non-ASCII letter) gets 403 AccessDenied, where AWS S3 answers 400 InvalidBucketName. Clients and compatibility suites that check the error code read the 403 as a permission problem. Auth and AuthPostPolicy now turn an AccessDenied for a resolved identity into InvalidBucketName when the bucket name fails VerifyS3BucketName or uses a character outside a-z, 0-9, '.' and '-' (VerifyS3BucketName checks with unicode.IsLower/IsNumber, which accept letters such as 'ü'). The answer depends on the name alone, so it reveals nothing about which buckets exist. An authentication failure, an allowed request and a denial on a well-formed name answer as before. Test: TestAuthDeniedInvalidBucketName drives iam.Auth with signed and anonymous requests; on master every malformed name answers 403, with the change 400, while the identity's own bucket stays allowed and a well-formed foreign name stays 403. TestRefineDenialForBucketName covers the rule table. * s3: refine POST-policy and deferred anonymous-read denials on a malformed bucket name Two denial paths bypass the refinement in Auth and AuthPostPolicy: - A browser POST upload carries its credentials in the form, so AuthPostPolicy delegates it to PostPolicyBucketHandler, which verifies the form and writes AccessDenied itself when the signer may not write the bucket or the per-key authorization denies the object. Both denials now go through refineDenialForBucketName with the verified identity. The coarse Write check used to run inside doesPolicySignatureV2Match and doesPolicySignatureV4Match before the signature was compared, so a form with a wrong signature was told whether the key may write the bucket. It now runs in doesPolicySignatureMatch after the signature verifies, as on the header-signed path; a form that fails verification answers SignatureDoesNotMatch (or MissingFields) and is never refined. - Unsigned GET/HEAD object reads are deferred by AuthWithPublicRead to the object handlers, where writeAnonymousAwareNotFound masks a miss as AccessDenied. That denial is refined with the anonymous identity, so a missing malformed bucket answers InvalidBucketName when an anonymous identity is configured, as Auth does. A denial on an entry that was found is left alone: the bucket exists. Test: TestPostPolicyDeniedInvalidBucketName drives AuthPostPolicy with signed forms (Write denied, per-key policy deny, bad signature, well-formed name); TestAnonymousReadDeniedInvalidBucketName drives AuthWithPublicRead with GET and HEAD, with and without an anonymous identity. Both fail before this change. * s3: refine a denied anonymous read of an existing key on a malformed bucket name A deferred anonymous GET/HEAD refined only its masked miss to InvalidBucketName, so in a bucket whose name breaks the S3 rules (one VerifyS3BucketName still accepts, such as a non-ASCII name) a missing key answered 400 and an existing private key 403, telling the caller which keys exist. The entry check now refines its denial the same way, so both answer alike. Without an anonymous identity no identity resolves and both stay 403, as in Auth. The deny-precedence case in TestAnonymousObjectACLConfiguredIdentity runs on the fixture bucket "b", shorter than S3's three-character minimum, so its denial now answers InvalidBucketName. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
Chris LuandDevin
|
8b09c96cec |
s3: keep a signed content-length in the canonical request (#11849)
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
0a96972a99 |
s3: verify X-Forwarded-Prefix only from trusted proxies (#11843)
* s3api: check the direct peer against the trusted-proxy allowlist * s3: verify X-Forwarded-Prefix only from trusted proxies * s3: note that trusted proxies must sanitize forwarded headers |
||
|
Chris Lu
|
cabaf9c38d |
iceberg: collapse doubled /v1 prefix in catalog requests (#11767)
* s3: verify SigV4 against the request path the client sent
Internal routers that rewrite URL.Path (the Iceberg catalog doubled-/v1
collapse) broke signature verification: verifyV4Signature re-derived the
canonical URI from the rewritten path while the client had signed the
original one, so valid signed catalog requests were rejected. Stash the
client-sent path in the request context at rewrite time and prefer it when
computing pathForSignature.
* iceberg: collapse repeated /v1 prefixes before route matching
Move the doubled-prefix collapse from the catch-all into a middleware so it
also covers paths that match the {prefix} routes (/v1/v1/namespaces would
otherwise resolve "v1" as the table bucket; the name is unreachable anyway
since bucket names require 3-63 chars). Collapse all leading repetitions in
a single pass and re-dispatch once instead of recursing per segment, and
record the original path in context so signed requests still verify.
* iceberg: slice off repeated /v1 prefixes instead of concatenating
Stripping path[len("/v1"):] per iteration reuses the original string
instead of allocating a fresh copy of the remaining path each pass.
|
||
|
Chris Lu
|
c023493cf0 |
s3api: tolerate session tokens on statically configured credentials (#11694)
* s3api: consolidate session token extraction into extractSessionToken Three call sites duplicated the same header/header/query lookup; share one helper named after the existing s3tables equivalent. * s3api: tolerate session tokens on statically configured credentials Credential vendors like Unity Catalog emit a session token with every vended credential, including static ones (UC's StaticAwsCredentialGenerator only engages when s3.sessionToken is set). Requests signed by a configured access key were routed to STS validation and rejected, so static credential vending never worked against SeaweedFS. Resolve the access key first: when it maps to a configured credential the signature alone authenticates the request, and the attached token is marked ignored so authorization does not route it into the STS session-policy path. STS-issued access keys are never in the static map, so temporary credentials still validate their token exactly as before. * s3api: cover static credentials carrying a session token * test: exercise UC static credential vending against SeaweedFS s3.sessionToken.0 selects UC's StaticAwsCredentialGenerator, which vends the configured keys verbatim. The vended session token is foreign to SeaweedFS and previously failed SigV4; now it round-trips through temporary-table-credentials into real S3 I/O. * s3api: reject temporary credentials in GetFederationToken after auth Token presence alone cannot distinguish a temporary credential from a statically configured one carrying a vended token. Move the check behind verifyV4Signature and key it on the operative session token so tolerated tokens keep the caller eligible. * s3api: test GetFederationToken with authenticated temporary credentials Sign the rejection cases with real session credentials so they reach the post-auth check, and cover a vended static credential being accepted. * test: check vended-credential delete error in UC integration test |
||
|
Chris LuandDevin
|
56fbc3deb5 |
s3api: align S3/IAM error responses with AWS (#11632)
* s3err: add InvalidArgument and AuthorizationHeaderMalformed codes Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: answer unrecognized bucket PUT sub-resources with 501 A PUT on a bucket carrying an unrecognized query (logging, metrics, intelligent-tiering, ...) fell through to the bare CreateBucket route and returned BucketAlreadyOwnedByYou or re-created the bucket. AWS answers these with NotImplemented. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: reject malformed copy-source and multipart PUT parameters A malformed X-Amz-Copy-Source or a non-numeric partNumber fell through to the plain PutObject route and stored the body as a regular object. Answer them with InvalidArgument-class errors instead of writing data. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: verify x-amz-content-sha256 against the streamed body A PUT carrying a hex or base64 payload hash now streams through a verifier that reports a mismatch once the stream is exhausted, instead of storing an object that does not match its declared hash. The error is deferred so intermediate reads that drop (n>0, err) results cannot silently swallow it. Sentinel values (unsigned/streaming payloads) remain exempt and malformed values fail fast. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: map truncated SigV4 headers to the error for the missing field AWS answers an Authorization header missing Credential= with InvalidArgument and one missing or malformed Signature= with AuthorizationHeaderMalformed, instead of a generic MissingFields. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: answer IAM/STS failures in the query-protocol envelope Embedded IAM and STS routes now report authentication, form-parse and authorization failures with the IAM ErrorResponse body instead of the S3 Error envelope, so IAM SDK clients can parse them. Requests signed for s3 keep the S3 envelope, keyed off the credential scope. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: return 403 AccessDenied when the request has no Date Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: answer throttling rejections as SlowDown ErrTooManyRequest and ErrRequestBytesExceed reported made-up codes; AWS serves these throttling rejections as SlowDown. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: reject versionId requests on buckets that never had versioning GET, HEAD and DELETE carrying a non-empty versionId on an unversioned bucket now fail with InvalidArgument instead of being answered as a plain object request. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: answer DeleteObjects over 1000 keys with MalformedXML Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: InvalidArgument for non-numeric or out-of-range part numbers partNumber=abc, 0 and >10000 all resolve to InvalidArgument, matching AWS, instead of InvalidPart. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iam: correct LimitExceeded, InvalidAction and ServiceFailure mappings LimitExceeded is a conflict (409), an unknown Action is InvalidAction (404) rather than NotImplemented, and internal failures report the IAM receiver fault type. Applies to both the embedded IAM endpoint and the standalone iamapi server. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iam: refuse DeleteUser while access keys remain Deleting a user with live credentials orphaned its access keys; AWS answers DeleteConflict until they are removed first. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test/s3: add S3/IAM error-response compatibility harness * s3: return InvalidArgument for malformed x-amz-content-sha256 A header value that decodes to neither 32-byte hex nor base64 is a malformed argument, not a hash mismatch. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: stop spawned mini when readiness times out A slow-starting server otherwise survives the failure path and keeps the S3 port occupied for the next run. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: register atexit cleanup before setup A failed setup previously skipped cleanup, leaking the bucket and IAM user on persistent servers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: reject unrouted subresources on the DELETE bucket catch-all PutBucketHandler gained the same guard when the route-level check moved into the handlers; DeleteBucketHandler was missed, so an authorized DELETE /bucket?logging could delete the bucket instead of answering NotImplemented. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: tolerate unset fixture variables in cleanup and cover DELETE ?logging Cleanup now runs its IAM/multipart steps only when setup reached them, so an early setup failure still removes the bucket. Added a DELETE bucket-subresource case asserting NotImplemented and bucket survival. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: fail a case when its side-effect check reports a regression A non-empty check note now fails the case, so a deleted bucket or an object created by a malformed request cannot slip through behind a passing status check. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
793ce06b10 |
s3api: allow unsigned SSE-C customer key headers on presigned requests (#11578)
* s3api: allow unsigned SSE-C customer key headers on presigned requests AWS requires only x-amz-server-side-encryption-customer-algorithm to be signed on presigned URLs; the key and key-MD5 headers are supplied at request time. Since #9121 rejected any x-amz-* header outside SignedHeaders, SDK-generated presigned SSE-C requests (e.g. .NET GetPreSignedUrlRequest) fail with SignatureDoesNotMatch. Exempt the customer key and copy-source key headers for presigned requests only. * s3api: test presigned SSE-C requests carrying unsigned key headers |
||
|
Chris Lu
|
5a0e017457 |
s3: reject virtual-host bucket retargeting via X-Forwarded-Host (#11281)
* s3: reject virtual-host bucket retargeting via X-Forwarded-Host SigV4 verification tries the client-supplied X-Forwarded-Host as a signed host candidate, while routing and IAM select the bucket from the actual Host header. A presigned URL for one virtual-host bucket could therefore be retargeted to another bucket accessible to the same signing identity by changing Host and adding X-Forwarded-Host. After the signature matches a host candidate, extract the bucket that the candidate implies (via the configured virtual-host domains) and compare it with the bucket the router selected. Reject when they differ, before returning success. * test(s3api): cover virtual-host presigned URL retargeting Add unit tests for bucketFromVirtualHost and end-to-end tests that reproduce the X-Forwarded-Host retargeting attack for both presigned and signed requests, plus a negative test confirming the legitimate same-bucket case still verifies. * s3: harden bucketFromVirtualHost for case and overlapping domains Compare host and domain suffixes case-insensitively so a mixed-case X-Forwarded-Host cannot bypass the consistency check. Only treat the exact path-style domain as non-virtual-host; subdomains of a path-style domain still match the virtual-host router pattern and must be checked. |
||
|
Chris Lu
|
a3638e479e |
fix(s3api/audit): surface OIDC identity claim in audit log for STS sessions (#11269)
* Add ResolveIdentityClaim helper for OIDC audit identity ComputeParentUser derives a stable per-identity hash from (sub, iss) for internal keying, but it is opaque and not human-readable. Audit logs for STS-assumed OIDC sessions currently surface that opaque value (or the random session id) as the requester, leaving no authoritative trace of the federated user. Add ResolveIdentityClaim next to ComputeParentUser to recover a human-readable, server-asserted identity attribute from the STS request context populated at federation time. It walks a priority list (preferred_username, email, name, sub) so a federated session always audits against a stable OIDC claim rather than a client-supplied role session name. For #11264 * Surface authoritative OIDC identity claim in S3 audit log For STS-assumed sessions minted from an OIDC web identity, the audit log requester field is the opaque session subject, which cannot be traced back to the federated user who performed the operation. The OIDC identity claims (preferred_username, email, sub) are already carried in the session request context and reach the auth layer as identity.Claims, but they were never surfaced to the audit log. Add a requester_identity field to the S3 access audit log, populated from the authoritative OIDC identity claim resolved via ResolveIdentityClaim. The claim is propagated through the shared identity holder (the same mechanism the requester name and principal ARN already use) so it survives the request-context copy that hides auth-set values from the outer audit middleware. The existing requester field is left unchanged for backward compatibility; requester_identity is empty for non-federated sessions, where requester already carries the real username. For #11264 * Gate OIDC audit identity on federation marker and harden resolver Address review feedback (Devin Review, Greptile) on the initial implementation: - Non-federated STS sessions no longer gain a false requester_identity. ValidateJWTWithClaims merges the JWT registered sub claim (the opaque session id) into RequestContext for sessions without an explicit request context, so the previous ResolveIdentityClaim fallback to sub surfaced that session id as an authoritative identity. Resolution is now gated on SessionInfo.ParentUser, which is set only for OIDC-federated sessions in AssumeRoleWithWebIdentity. The claim is resolved from the original sessionInfo.RequestContext (not the local claims map, whose sub the bearer path overwrites with the session subject) so SigV4 and bearer sessions surface the same identity. - ResolveIdentityClaim now trims whitespace and treats whitespace-only claims as absent, so a blank preferred_username no longer masks a usable email or sub. The resolved claim is carried on Identity.IdentityClaim (and IAMIdentity for the bearer path) rather than re-derived in recordIdentityInContext, making the federation gate explicit at the auth boundary. For #11264 * Resolve OIDC identity claim for external bearer tokens The external OIDC bearer path (a raw OIDC JWT presented directly, not via STS) populates Claims with preferred_username/email/name/sub from the validated token but did not set IdentityClaim, so requester_identity stayed blank for that authentication path. Resolve the claim there too — sub is the real OIDC subject on this path (not an STS session id), so no federation gate is needed. Also drop an ineffectual ctx assignment flagged by ineffassign in the audit test. For #11264 |
||
|
Chris Lu
|
f5f1dcbd8c |
s3: keep verifying the request host when externalUrl is set (#10970)
* s3: keep verifying the request host when externalUrl is set externalUrl was the only host candidate once set, so a client that dialed the gateway directly instead of through the proxy always got SignatureDoesNotMatch. Make it lead the candidate walk instead: every candidate still needs a valid signature, and the request-derived hosts are already trusted when the flag is unset, so a mixed proxy plus in-cluster topology can now advertise a public endpoint and verify both planes. * s3: cover virtual-hosted addressing behind externalUrl The old pin also rejected an external client that signed bucket.api.example.com, since only the bare externalUrl host was ever tried. The candidate walk covers it; pin the case down. |
||
|
Chris Lu
|
2a70532d0d |
s3: log each request at -v=2 (#10931)
* s3: log each request at -v=2 * s3: quote requester and path in the access log line * s3: record the post-policy signing identity as the requester |
||
|
Chris Lu
|
e6b2849381 |
s3: verify SigV4 against each plausible reverse-proxy host (#10284)
* s3: verify SigV4 against each plausible reverse-proxy host A portless X-Forwarded-Host leaves the client's true port ambiguous: a proxy that kept the Host header makes the backend Host port right, one that rewrote it makes X-Forwarded-Port right, and a client on the scheme's default port signed no port at all. The verifier bet on the backend Host port whenever the hostnames matched, so nginx-style $host/$server_port forwarding got SignatureDoesNotMatch whenever the proxy and backend share a hostname. Try each plausible host value in likelihood order instead of guessing one. * s3: unbracket IPv6 X-Forwarded-Host before matching the request host net.SplitHostPort strips brackets from the request host, so a bracketed portless X-Forwarded-Host like [::1] never matched and lost its port candidate. * s3: cover unbracketed IPv6 forwarded-host candidates; compare with slices.Equal |
||
|
jay
|
a16194f5b4 |
refract: reduce mem alloc while building str (#10261)
Signed-off-by: jayl1e <jayl1e@outlook.com> |
||
|
|
f58721b22f |
s3api: optimize encodePath memory allocations (#10252)
* Optimize s3api encodePath to eliminate O(n^2) allocations encodePath built its result via string concatenation in a loop, which is O(n^2) in allocations. For non-ASCII (e.g. Chinese) runes it additionally called make + hex.EncodeToString + strings.ToUpper per byte (~10 allocations per character). Under high QPS with long non-ASCII object keys this produced a very high allocation rate, frequent GC and long GC pauses, causing S3 request latency spikes. Replace with a preallocated strings.Builder, a manual hex lookup table, and a zero-allocation fast path. EncodePath now delegates to encodePath to remove the duplicated implementation. Output is byte-for-byte identical, verified by TestEncodePath and TestEncodePathEqual. Benchmark (long Chinese path): 261 -> 1 allocs/op, 35810 -> 480 B/op, 7.5x faster. Load test (50M calls): 212x fewer allocations, 76x fewer GC cycles. * s3api: drop regexp from encodePath fast path The unreserved-character scan already decides whether any byte needs encoding, so reservedObjectNames.MatchString was a redundant second pass that also ran the RE2 engine on every authenticated request. Rely on the scan alone; output is unchanged. The ASCII fast path drops from ~188ns to ~11ns per call. --------- Co-authored-by: LiuDoge <liudoge@LiuDogedeMacBook-Air.local> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
7y-9andCodex
|
7589429b43 |
fix(s3api): sort repeated SigV4 query values (#10031)
* fix(s3api): sort repeated SigV4 query values Problem SigV4 canonical query strings must sort repeated parameter values. SeaweedFS preserved the incoming value order, so a correctly signed request with repeated query parameters could fail verification when the request order differed from canonical order. Root cause getCanonicalQueryString delegated directly to url.Values.Encode(), which sorts parameter names but preserves each key's value slice order. Fix Sort every query value slice before encoding the canonical query string, while still excluding X-Amz-Signature for presigned URLs. Reproduction go test ./weed/s3api -run TestGetCanonicalQueryStringSortsRepeatedValues -count=1 failed before the fix with partNumber=2 before partNumber=10. Co-authored-by: Codex <noreply@openai.com> * test(s3api): expand canonical query coverage Co-authored-by: Codex <noreply@openai.com> --------- Co-authored-by: Codex <noreply@openai.com> |
||
|
Chris Lu
|
c1636ac41c |
s3: give STS sessions a distinct owner account instead of admin (#9963)
* s3: give STS sessions a distinct owner account, not admin STS sessions were built with Account: &AccountAdmin, so every assumed-role session shared the admin account for ownership and ACL checks. Use the assumed-role user as the account id instead, matching the JWT auth path. Session permissions are unchanged: they come from the session policies, and admin is granted only through Actions. * s3: resolve STS session identity to the OIDC subject Use sessionInfo.Subject (falling back to the assumed-role user when absent) for the session identity name and account id, so the SigV4 and JWT auth paths resolve the same session to the same identity instead of diverging on AssumedRoleUser vs Subject. * s3: trim verbose comments |
||
|
7y-9andCodex
|
689b5b61bf |
fix(s3api): reject empty v4 signed header names (#9910)
Problem: Signature V4 SignedHeaders parsing accepted empty header name segments such as host; or ;host. Malformed Authorization headers could continue into signature verification instead of failing during header parsing. Root cause: parseSignedHeader only checked that the SignedHeaders value was non-empty, then split it on semicolons without validating each element. Fix: reject empty or whitespace-only signed header elements with ErrMissingFields before returning the parsed header list. Reproduction: go test ./weed/s3api -run TestParseSignedHeaderRejectsEmptyHeaderNames -count=1 failed before the fix because SignedHeaders=host; returned ErrNone. Validation: gofmt -w weed/s3api/auth_signature_v4.go weed/s3api/auth_signature_v4_test.go; git diff --check; go test ./weed/s3api -run TestParseSignedHeaderRejectsEmptyHeaderNames -count=1; go test ./weed/s3api -count=1 Co-authored-by: Codex <noreply@openai.com> |
||
|
Chris Lu
|
88ac2d0431 |
security(s3api): reject unsigned x-amz-* headers in SigV4 requests (#9121)
A presigned URL holder could attach arbitrary x-amz-* headers to a PUT request (e.g. x-amz-tagging, x-amz-acl, x-amz-storage-class, x-amz-server-side-encryption*, x-amz-object-lock-*, x-amz-meta-*, x-amz-website-redirect-location, x-amz-grant-*). Because only the headers declared in SignedHeaders participate in signature verification, the added headers bypass authentication; the PUT handler then persists them into the object's Extended metadata. Match the AWS SigV4 rule: every x-amz-* header present in the request must appear in SignedHeaders. Exempt x-amz-content-sha256 (already tamper-protected via the canonical request's payload-hash line) and, for presigned URLs, the SigV4 protocol parameters that live in the query string (X-Amz-Algorithm/Credential/Date/Expires/SignedHeaders/ Signature) in case they are duplicated as headers. Applies to both header-based and presigned SigV4; non-amz headers are unaffected. |
||
|
Chris Lu
|
efc7f3936f |
fix(s3): handle empty URL path in forwarded prefix signature verification (#8973)
fix(s3): handle empty URL path in forwarded prefix signature verification (#8966) When S3 is behind a reverse proxy with a forwarded prefix (e.g. /s3), requests with an empty URL path (like ListBuckets) would incorrectly get a trailing slash appended (e.g. /s3/), causing signature verification to fail because the client signs /s3 without the slash. |
||
|
Chris Lu
|
e8946e59ca |
fix(s3api): correctly extract host header port in extractHostHeader (#8464)
* Prevent concurrent maintenance tasks per volume * fix panic * fix(s3api): correctly extract host header port when X-Forwarded-Port is present * test(s3api): add test cases for misreported X-Forwarded-Port |
||
|
|
3d81d5bef7 |
Fix S3 signature verification behind reverse proxies (#8444)
* Fix S3 signature verification behind reverse proxies When SeaweedFS is deployed behind a reverse proxy (e.g. nginx, Kong, Traefik), AWS S3 Signature V4 verification fails because the Host header the client signed with (e.g. "localhost:9000") differs from the Host header SeaweedFS receives on the backend (e.g. "seaweedfs:8333"). This commit adds a new -s3.externalUrl parameter (and S3_EXTERNAL_URL environment variable) that tells SeaweedFS what public-facing URL clients use to connect. When set, SeaweedFS uses this host value for signature verification instead of the Host header from the incoming request. New parameter: -s3.externalUrl (flag) or S3_EXTERNAL_URL (environment variable) Example: -s3.externalUrl=http://localhost:9000 Example: S3_EXTERNAL_URL=https://s3.example.com The environment variable is particularly useful in Docker/Kubernetes deployments where the external URL is injected via container config. The flag takes precedence over the environment variable when both are set. At startup, the URL is parsed and default ports are stripped to match AWS SDK behavior (port 80 for HTTP, port 443 for HTTPS), so "http://s3.example.com:80" and "http://s3.example.com" are equivalent. Bugs fixed: - Default port stripping was removed by a prior PR, causing signature mismatches when clients connect on standard ports (80/443) - X-Forwarded-Port was ignored when X-Forwarded-Host was not present - Scheme detection now uses proper precedence: X-Forwarded-Proto > TLS connection > URL scheme > "http" - Test expectations for standard port stripping were incorrect - expectedHost field in TestSignatureV4WithForwardedPort was declared but never actually checked (self-referential test) * Add Docker integration test for S3 proxy signature verification Docker Compose setup with nginx reverse proxy to validate that the -s3.externalUrl parameter (or S3_EXTERNAL_URL env var) correctly resolves S3 signature verification when SeaweedFS runs behind a proxy. The test uses nginx proxying port 9000 to SeaweedFS on port 8333, with X-Forwarded-Host/Port/Proto headers set. SeaweedFS is configured with -s3.externalUrl=http://localhost:9000 so it uses "localhost:9000" for signature verification, matching what the AWS CLI signs with. The test can be run with aws CLI on the host or without it by using the amazon/aws-cli Docker image with --network host. Test covers: create-bucket, list-buckets, put-object, head-object, list-objects-v2, get-object, content round-trip integrity, delete-object, and delete-bucket — all through the reverse proxy. * Create s3-proxy-signature-tests.yml * fix CLI * fix CI * Update s3-proxy-signature-tests.yml * address comments * Update Dockerfile * add user * no need for fuse * Update s3-proxy-signature-tests.yml * debug * weed mini * fix health check * health check * fix health checking --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
d5e71eb0d8 |
Revert "s3api: preserve Host header port in signature verification (#8434)"
This reverts commit
|
||
|
Chris Lu
|
98d89ffad7 |
s3api: preserve Host header port in signature verification (#8434)
Avoid stripping default ports (80/443) from the Host header in extractHostHeader. This fixes SignatureDoesNotMatch errors when SeaweedFS is accessed via a proxy (like Kong Ingress) that explicitly includes the port in the Host header or X-Forwarded-Host, which S3 clients sign. Also cleaned up unused variables and logic after refactoring. |
||
|
Chris Lu
|
5ecee9e64d |
s3: fix signature mismatch with non-standard ports and capitalized host (#8386)
* s3: fix signature mismatch with non-standard ports and capitalized host - ensure host header extraction is case-insensitive in SignedHeaders - prioritize non-standard ports in X-Forwarded-Host over default ports in X-Forwarded-Port - add regression tests for both scenarios fixes https://github.com/seaweedfs/seaweedfs/issues/8382 * simplify |
||
|
Chris Lu
|
cf8e383e1e |
STS: Fallback to Caller Identity when RoleArn is missing in AssumeRole (#8345)
* s3api: make RoleArn optional in AssumeRole * s3api: address PR feedback for optional RoleArn * iam: add configurable default role for AssumeRole * S3 STS: Use caller identity when RoleArn is missing - Fallback to PrincipalArn/Context in AssumeRole if RoleArn is empty - Handle User ARNs in prepareSTSCredentials - Fix PrincipalArn generation for env var credentials * Test: Add unit test for AssumeRole caller identity fallback * fix(s3api): propagate admin permissions to assumed role session when using caller identity fallback * STS: Fix is_admin propagation and optimize IAM policy evaluation for assumed roles - Restore is_admin propagation via JWT req_ctx - Optimize IsActionAllowed to skip role lookups for admin sessions - Ensure session policies are still applied for downscoping - Remove debug logging - Fix syntax errors in cleanup * fix(iam): resolve STS policy bypass for admin sessions - Fixed IsActionAllowed in iam_manager.go to correctly identify and validate internal STS tokens, ensuring session policies are enforced. - Refactored VerifyActionPermission in auth_credentials.go to properly handle session tokens and avoid legacy authorization short-circuits. - Added debug logging for better tracing of policy evaluation and session validation. |
||
|
Chris Lu
|
f44e25b422 |
fix(iam): ensure access key status is persisted and defaulted to Active (#8341)
* Fix master leader election startup issue Fixes #error-log-leader-not-selected-yet * not useful test * fix(iam): ensure access key status is persisted and defaulted to Active * make pb * update tests * using constants |
||
|
Chris LuandCopilot
|
796f23f68a |
Fix STS InvalidAccessKeyId and request body consumption issues (#8328)
* Fix STS InvalidAccessKeyId and request body consumption in Lakekeeper integration test * Remove debug prints * Add Lakekeeper integration tests to CI * Fix connection refused in CI by binding to 0.0.0.0 * Add timeout to docker run in Lakekeeper integration test * Update weed/s3api/auth_credentials.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Chris Luandgemini-code-assist[bot]
|
e6ee293c17 |
Add table operations test (#8241)
* Add Trino blog operations test * Update test/s3tables/catalog_trino/trino_blog_operations_test.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * feat: add table bucket path helpers and filer operations - Add table object root and table location mapping directories - Implement ensureDirectory, upsertFile, deleteEntryIfExists helpers - Support table location bucket mapping for S3 access * feat: manage table bucket object roots on creation/deletion - Create .objects directory for table buckets on creation - Clean up table object bucket paths on deletion - Enable S3 operations on table bucket object roots * feat: add table location mapping for Iceberg REST - Track table location bucket mappings when tables are created/updated/deleted - Enable location-based routing for S3 operations on table data * feat: route S3 operations to table bucket object roots - Route table-s3 bucket names to mapped table paths - Route table buckets to object root directories - Support table location bucket mapping lookup * feat: emit table-s3 locations from Iceberg REST - Generate unique table-s3 bucket names with UUID suffix - Store table metadata under table bucket paths - Return table-s3 locations for Trino compatibility * fix: handle missing directories in S3 list operations - Propagate ErrNotFound from ListEntries for non-existent directories - Treat missing directories as empty results for list operations - Fixes Trino non-empty location checks on table creation * test: improve Trino CSV parsing for single-value results - Sanitize Trino output to skip jline warnings - Handle single-value CSV results without header rows - Strip quotes from numeric values in tests * refactor: use bucket path helpers throughout S3 API - Replace direct bucket path operations with helper functions - Leverage centralized table bucket routing logic - Improve maintainability with consistent path resolution * fix: add table bucket cache and improve filer error handling - Cache table bucket lookups to reduce filer overhead on repeated checks - Use filer_pb.CreateEntry and filer_pb.UpdateEntry helpers to check resp.Error - Fix delete order in handler_bucket_get_list_delete: delete table object before directory - Make location mapping errors best-effort: log and continue, don't fail API - Update table location mappings to delete stale prior bucket mappings on update - Add 1-second sleep before timestamp time travel query to ensure timestamps are in past - Fix CSV parsing: examine all lines, not skip first; handle single-value rows * fix: properly handle stale metadata location mapping cleanup - Capture oldMetadataLocation before mutation in handleUpdateTable - Update updateTableLocationMapping to accept both old and new locations - Use passed-in oldMetadataLocation to detect location changes - Delete stale mapping only when location actually changes - Pass empty string for oldLocation in handleCreateTable (new tables have no prior mapping) - Improve logging to show old -> new location transitions * refactor: cleanup imports and cache design - Remove unused 'sync' import from bucket_paths.go - Use filer_pb.UpdateEntry helper in setExtendedAttribute and deleteExtendedAttribute for consistent error handling - Add dedicated tableBucketCache map[string]bool to BucketRegistry instead of mixing concerns with metadataCache - Improve cache separation: table buckets cache is now separate from bucket metadata cache * fix: improve cache invalidation and add transient error handling Cache invalidation (critical fix): - Add tableLocationCache to BucketRegistry for location mapping lookups - Clear tableBucketCache and tableLocationCache in RemoveBucketMetadata - Prevents stale cache entries when buckets are deleted/recreated Transient error handling: - Only cache table bucket lookups when conclusive (found or ErrNotFound) - Skip caching on transient errors (network, permission, etc) - Prevents marking real table buckets as non-table due to transient failures Performance optimization: - Cache tableLocationDir results to avoid repeated filer RPCs on hot paths - tableLocationDir now checks cache before making expensive filer lookups - Cache stores empty string for 'not found' to avoid redundant lookups Code clarity: - Add comment to deleteDirectory explaining DeleteEntry response lacks Error field * go fmt * fix: mirror transient error handling in tableLocationDir and optimize bucketDir Transient error handling: - tableLocationDir now only caches definitive results - Mirrors isTableBucket behavior to prevent treating transient errors as permanent misses - Improves reliability on flaky systems or during recovery Performance optimization: - bucketDir avoids redundant isTableBucket call via bucketRoot - Directly use s3a.option.BucketsPath for regular buckets - Saves one cache lookup for every non-table bucket operation * fix: revert bucketDir optimization to preserve bucketRoot logic The optimization to directly use BucketsPath bypassed bucketRoot's logic and caused issues with S3 list operations on delimiter+prefix cases. Revert to using path.Join(s3a.bucketRoot(bucket), bucket) which properly handles all bucket types and ensures consistent path resolution across the codebase. The slight performance cost of an extra cache lookup is worth the correctness and consistency benefits. * feat: move table buckets under /buckets Add a table-bucket marker attribute, reuse bucket metadata cache for table bucket detection, and update list/validation/UI/test paths to treat table buckets as /buckets entries. * Fix S3 Tables code review issues - handler_bucket_create.go: Fix bucket existence check to properly validate entryResp.Entry before setting s3BucketExists flag (nil Entry should not indicate existing bucket) - bucket_paths.go: Add clarifying comment to bucketRoot() explaining unified buckets root path for all bucket types - file_browser_data.go: Optimize by extracting table bucket check early to avoid redundant WithFilerClient call * Fix list prefix delimiter handling * Handle list errors conservatively * Fix Trino FOR TIMESTAMP query - use past timestamp Iceberg requires the timestamp to be strictly in the past. Use current_timestamp - interval '1' second instead of current_timestamp. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
a3b83f8808 |
test: add Trino Iceberg catalog integration test (#8228)
* test: add Trino Iceberg catalog integration test - Create test/s3/catalog_trino/trino_catalog_test.go with TestTrinoIcebergCatalog - Tests integration between Trino SQL engine and SeaweedFS Iceberg REST catalog - Starts weed mini with all services and Trino in Docker container - Validates Iceberg catalog schema creation and listing operations - Uses native S3 filesystem support in Trino with path-style access - Add workflow job to s3-tables-tests.yml for CI execution * fix: preserve AWS environment credentials when replacing S3 configuration When S3 configuration is loaded from filer/db, it replaces the identities list and inadvertently removes AWS_ACCESS_KEY_ID credentials that were added from environment variables. This caused auth to remain disabled even though valid credentials were present. Fix by preserving environment-based identities when replacing the configuration and re-adding them after the replacement. This ensures environment credentials persist across configuration reloads and properly enable authentication. * fix: use correct ServerAddress format with gRPC port encoding The admin server couldn't connect to master because the master address was missing the gRPC port information. Use pb.NewServerAddress() which properly encodes both HTTP and gRPC ports in the address string. Changes: - weed/command/mini.go: Use pb.NewServerAddress for master address in admin - test/s3/policy/policy_test.go: Store and use gRPC ports for master/filer addresses This fix applies to: 1. Admin server connection to master (mini.go) 2. Test shell commands that need master/filer addresses (policy_test.go) * move * move * fix: always include gRPC port in server address encoding The NewServerAddress() function was omitting the gRPC port from the address string when it matched the port+10000 convention. However, gRPC port allocation doesn't always follow this convention - when the calculated port is busy, an alternative port is allocated. This caused a bug where: 1. Master's gRPC port was allocated as 50661 (sequential, not port+10000) 2. Address was encoded as '192.168.1.66:50660' (gRPC port omitted) 3. Admin client called ToGrpcAddress() which assumed port+10000 offset 4. Admin tried to connect to 60660 but master was on 50661 → connection failed Fix: Always include explicit gRPC port in address format (host:httpPort.grpcPort) unless gRPC port is 0. This makes addresses unambiguous and works regardless of the port allocation strategy used. Impacts: All server-to-server gRPC connections now use properly formatted addresses. * test: fix Iceberg REST API readiness check The Iceberg REST API endpoints require authentication. When checked without credentials, the API returns 403 Forbidden (not 401 Unauthorized). The readiness check now accepts both auth error codes (401/403) as indicators that the service is up and ready, it just needs credentials. This fixes the 'Iceberg REST API did not become ready' test failure. * Fix AWS SigV4 signature verification for base64-encoded payload hashes AWS SigV4 canonical requests must use hex-encoded SHA256 hashes, but the X-Amz-Content-Sha256 header may be transmitted as base64. Changes: - Added normalizePayloadHash() function to convert base64 to hex - Call normalizePayloadHash() in extractV4AuthInfoFromHeader() - Added encoding/base64 import Fixes 403 Forbidden errors on POST requests to Iceberg REST API when clients send base64-encoded content hashes in the header. Impacted services: Iceberg REST API, S3Tables * Fix AWS SigV4 signature verification for base64-encoded payload hashes AWS SigV4 canonical requests must use hex-encoded SHA256 hashes, but the X-Amz-Content-Sha256 header may be transmitted as base64. Changes: - Added normalizePayloadHash() function to convert base64 to hex - Call normalizePayloadHash() in extractV4AuthInfoFromHeader() - Added encoding/base64 import - Removed unused fmt import Fixes 403 Forbidden errors on POST requests to Iceberg REST API when clients send base64-encoded content hashes in the header. Impacted services: Iceberg REST API, S3Tables * pass sigv4 * s3api: fix identity preservation and logging levels - Ensure environment-based identities are preserved during config replacement - Update accessKeyIdent and nameToIdentity maps correctly - Downgrade informational logs to V(2) to reduce noise * test: fix trino integration test and s3 policy test - Pin Trino image version to 479 - Fix port binding to 0.0.0.0 for Docker connectivity - Fix S3 policy test hang by correctly assigning MiniClusterCtx - Improve port finding robustness in policy tests * ci: pre-pull trino image to avoid timeouts - Pull trinodb/trino:479 after Docker setup - Ensure image is ready before integration tests start * iceberg: remove unused checkAuth and improve logging - Remove unused checkAuth method - Downgrade informational logs to V(2) - Ensure loggingMiddleware uses a status writer for accurate reported codes - Narrow catch-all route to avoid interfering with other subsystems * iceberg: fix build failure by removing unused s3api import * Update iceberg.go * use warehouse * Update trino_catalog_test.go |
||
|
Chris Lu
|
1274cf038c |
s3: enforce authentication and JSON error format for Iceberg REST Catalog (#8192)
* s3: enforce authentication and JSON error format for Iceberg REST Catalog * s3/iceberg: align error exception types with OpenAPI spec examples * s3api: refactor AuthenticateRequest to return identity object * s3/iceberg: propagate full identity object to request context * s3/iceberg: differentiate NotAuthorizedException and ForbiddenException * s3/iceberg: reject requests if authenticator is nil to prevent auth bypass * s3/iceberg: refactor Auth middleware to build context incrementally and use switch for error mapping * s3api: update misleading comment for authRequestWithAuthType * s3api: return ErrAccessDenied if IAM is not configured to prevent auth bypass * s3/iceberg: optimize context update in Auth middleware * s3api: export CanDo for external authorization use * s3/iceberg: enforce identity-based authorization in all API handlers * s3api: fix compilation errors by updating internal CanDo references * s3/iceberg: robust identity validation and consistent action usage in handlers * s3api: complete CanDo rename across tests and policy engine integration * s3api: fix integration tests by allowing admin access when auth is disabled and explicit gRPC ports * duckdb * create test bucket |
||
|
Chris Lu
|
6a9e7360df |
s3api: fix S3 Tables auth to allow auto-hashing of body (#8170)
* s3api: allow auto-hashing of request body for s3tables * s3api: add unit test for s3tables body hashing |
||
|
Chris LuandCopilot
|
f1e27b8f30 |
s3: change s3 tables to use RESTful API (#8169)
* s3: refactor s3 tables to use RESTful API * test/s3tables: guard empty namespaces * s3api: document tag parsing and validate get-table * s3api: limit S3Tables REST body size * Update weed/s3api/s3api_tables.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3tables/handler.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * s3api: accept encoded table bucket ARNs * s3api: validate namespaces and close body * s3api: match encoded table bucket ARNs * s3api: scope table bucket ARN routes * s3api: dedupe table bucket request builders * test/s3tables: allow list tables without namespace * s3api: validate table params and tag ARN * s3api: tighten tag handling and get-table params * s3api: loosen tag ARN route matching * Fix S3 Tables REST routing and tests * Adjust S3 Tables request parsing * Gate S3 Tables target routing * Avoid double decoding namespaces --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Chris Lu
|
5472061231 |
Fix: Populate Claims from STS session RequestContext for policy variable substitution (#8082)
* Fix: Populate Claims from STS session RequestContext for policy variable substitution
When using STS temporary credentials (from AssumeRoleWithWebIdentity) with
AWS Signature V4 authentication, JWT claims like preferred_username were
not available for bucket policy variable substitution (e.g., ${jwt:preferred_username}).
Root Cause:
- STS session tokens store user claims in the req_ctx field (added in PR #8079)
- validateSTSSessionToken() created Identity but didn't populate Claims field
- authorizeWithIAM() created IAMIdentity but didn't copy Claims
- Policy engine couldn't resolve ${jwt:*} variables without claims
Changes:
1. auth_signature_v4.go: Extract claims from sessionInfo.RequestContext
and populate Identity.Claims in validateSTSSessionToken()
2. auth_credentials.go: Copy Claims when creating IAMIdentity in
authorizeWithIAM()
3. auth_sts_identity_test.go: Add TestSTSIdentityClaimsPopulation to
verify claims are properly populated from RequestContext
This enables bucket policies with JWT claim variables to work correctly
with STS temporary credentials obtained via AssumeRoleWithWebIdentity.
Fixes #8037
* Refactor: Idiomatic map population for STS claims
|
||
|
Chris Lu
|
4e835a1d81 |
fix(s3api): ensure S3 configuration persistence and refactor authorization tests (#7989)
* fix(s3api): ensure static config file takes precedence over dynamic updates When a static S3 configuration file is provided, avoid overwriting the configuration from dynamic filer updates. This ensures the documented "Highest Priority" for the configuration file is respected. * refactor(s3api): implement merge-based static config with immutable identities Static identities from config file are now immutable and protected from dynamic updates. Dynamic identities (from admin panel) can be added and updated without affecting static entries. - Track identity names loaded from static config file - Implement merge logic that preserves static identities - Allow dynamic identities to be added or updated - Remove blanket block on config file updates * fix: address PR review comments for static config merge logic Critical Bugs: - Fix existingIdx always-false condition causing duplicate identities - Fix race condition in static config initialization (move useStaticConfig inside mutex) Security & Robustness: - Add nil identity check in VerifyActionPermission to fail closed - Mask access keys in STS validation logs to avoid exposing credentials - Add nil guard for s3a.iam in subscription handler Test Improvements: - Add authCalled tracking to MockIAMIntegration for explicit verification - Lower log level for static config messages to reduce noise * fix: prevent duplicates and race conditions in merge logic Data Integrity: - Prevent service account credential duplicates on repeated merges - Clean up stale accessKeyIdent entries when replacing identities - Check existing credentials before appending Concurrency Safety: - Add synchronization to IsStaticConfig method Test Improvements: - Add mux route vars for proper GetBucketAndObject extraction - Add STS session token header to trigger correct auth path |
||
|
Chris Lu
|
abfa64456b |
Fix STS authorization in streaming/chunked uploads (#7988)
* Fix STS authorization in streaming/chunked uploads During streaming/chunked uploads (SigV4 streaming), authorization happens twice: 1. Initial authorization in authRequestWithAuthType() - works correctly 2. Second authorization in verifyV4Signature() - was failing for STS The issue was that verifyV4Signature() only used identity.canDo() for permission checks, which always denies STS identities (they have empty Actions). This bypassed IAM authorization completely. This commit makes verifyV4Signature() IAM-aware by adding the same fallback logic used in authRequestWithAuthType(): - Traditional identities (with Actions) use legacy canDo() check - STS/JWT identities (empty Actions) fall back to IAM authorization Fixes: https://github.com/seaweedfs/seaweedfs/pull/7986#issuecomment-3723196038 * Add comprehensive unit tests for STS authorization in streaming uploads Created test suite to verify that verifyV4Signature properly handles STS identities by falling back to IAM authorization when shouldCheckPermissions is true. Tests cover: - STS identities with IAM integration (allow and deny cases) - STS identities without IAM integration (should deny) - Traditional identities with Actions (canDo check) - Permission check bypass when shouldCheckPermissions=false - Specific streaming upload scenario from bug report - Action determination based on HTTP method All tests pass successfully. * Refactor authorization logic to avoid duplication Centralized the authorization logic into IdentityAccessManagement.VerifyActionPermission. Updated auth_signature_v4.go and auth_credentials.go to use this new helper. Updated tests to clarify that they mirror the centralized logic. * Refactor tests to use VerifyActionPermission directly Introduced IAMIntegration interface to facilitate mocking of internal IAM integration logic. Updated IdentityAccessManagement to use the interface. Updated tests to directy call VerifyActionPermission using a mocked IAM integration, eliminating duplicated logic in tests. * fix(s3api): ensure static config file takes precedence and refactor tests - Track if configuration was loaded from a static file using `useStaticConfig`. - Ignore filer-based IAM updates when a static configuration is in use to respect "Highest Priority" rule. - Refactor `TestVerifyV4SignatureWithSTSIdentity` to use `VerifyActionPermission` directly. - Fix typed nil interface panic in authorization test. |
||
|
Chris Lu
|
6432019d08 |
Fix STS identity authorization by populating PolicyNames (#7985) (#7986)
* Fix STS identity authorization by populating PolicyNames (#7985) This commit fixes GitHub issue #7985 where STS-assumed identities received empty identity.Actions, causing all S3 operations to be denied even when the role had valid IAM policies attached. Changes: 1. Populate PolicyNames field from sessionInfo.Policies in validateSTSSessionToken() to enable IAM-based authorization for STS identities 2. Fix bucket+objectKey path construction in canDo() method to include proper slash separator between bucket and object key 3. Add comprehensive test suite to validate the fix and prevent regression The fix ensures that STS-assumed identities are properly authorized through the IAM path when iamIntegration is available, allowing roles with valid IAM policies to perform S3 operations as expected. * Update STS identity tests to be more rigorous and use actual implementation path * Fix regression in canDo() path concatenation The previous fix blindly added a slash separator, which caused double slashes when objectKey already started with a slash (common in existing tests and some code paths). This broke TestCanDo and TestObjectLevelListPermissions. This commit updates the logic to only add the slash separator if objectKey is not empty and does not already start with a slash. This fixes the regressions while maintaining the fix for issue #7985. * Refactor STS identity tests: extract helpers and simplify redundant logic - Extracted setupTestSTSService and newTestIdentity helper functions - Removed redundant if-else verification blocks that were already covered by assertions - Cleaned up test cases to improve maintainability as suggested in code review. * Add canDo() verification to STS identity tests Address code review suggestion: verify that identities with empty Actions correctly return false for canDo() checks, which confirms the behavior that forces authorization to fall back to the IAM path. * Simplify TestCanDoPathConstruction variable names Rename expectedPath to fullPath and simplify logging/assertion logic based on code review feedback. * Refactor path construction and logging in canDo() - Compute fullPath early and use it for logging to prevent double slashes - Update TestCanDoPathConstruction to use robust path verification - Add test case for objectKey with leading slash to ensure correct handling |
||
|
Chris Lu
|
23fc3f2621 |
Fix AWS SDK Signature V4 with STS credentials (issue #7941) (#7944)
* Add documentation for issue #7941 fix * ensure auth * rm FIX_ISSUE_7941.md * Integrate STS session token validation into V4 signature verification - Check for X-Amz-Security-Token header in verifyV4Signature - Call validateSTSSessionToken for STS requests - Skip regular access key lookup and expiration check for STS sessions * Fix variable scoping in verifyV4Signature for STS session token validation * Add ErrExpiredToken error for better AWS S3 compatibility with STS session tokens * Support STS session token in query parameters for presigned URLs * Fix nil pointer dereference in validateSTSSessionToken * Enhance STS token validation with detailed error diagnostics and logging * Fix missing credentials in STSSessionClaims.ToSessionInfo() * test: Add comprehensive STS session claims validation tests - TestSTSSessionClaimsToSessionInfo: Validates basic claims conversion - TestSTSSessionClaimsToSessionInfoCredentialGeneration: Verifies credential generation - TestSTSSessionClaimsToSessionInfoPreservesAllFields: Ensures all fields are preserved - TestSTSSessionClaimsToSessionInfoEmptyFields: Tests handling of empty/nil fields - TestSTSSessionClaimsToSessionInfoCredentialExpiration: Validates expiration handling All tests pass with proper timing tolerance for credential generation. * perf: Reuse CredentialGenerator instance for STS session claims Optimize ToSessionInfo() to reuse a package-level defaultCredentialGenerator instead of allocating a new CredentialGenerator on every call. This reduces allocation overhead since this method is called frequently during signature verification (potentially once per request). The CredentialGenerator is stateless and deterministic, making it safe to reuse across concurrent calls without synchronization. * refactor: Surface credential generation errors and remove sensitive logging Two improvements to error handling and security: 1. weed/iam/sts/session_claims.go: - Add logging for credential generation failures in ToSessionInfo() - Wrap errors with context (session ID) to aid debugging - Use glog.Warningf() to surface errors instead of silently swallowing them - Add fmt import for error wrapping 2. weed/s3api/auth_signature_v4.go: - Remove debug logging of actual access key IDs (glog.V(2) call) - Security improvement: avoid exposing sensitive access keys even at debug level - Keep warning-level logging that shows only count of available keys This ensures credential generation failures are observable while protecting sensitive authentication material from logs. * test: Verify deterministic credential generation in session claims tests Update TestSTSSessionClaimsToSessionInfoCredentialGeneration to properly verify deterministic credential generation: - Remove misleading comment about 'randomness' - parts of credentials ARE deterministic - Add assertions that AccessKeyId is identical for same SessionId (hash-based, deterministic) - Add assertions that SessionToken is identical for same SessionId (hash-based, deterministic) - Verify Expiration matches when SessionId is identical - Document that SecretAccessKey is NOT deterministic (uses random.Read) - Truncate expiresAt to second precision to avoid timing issues This test now properly verifies that the deterministic components of credential generation work correctly while acknowledging the cryptographic randomness of the secret access key. * test(sts): Assert credentials expiration relative to now in credential expiration tests Replace wallclock assertions comparing tc.expiresAt to time.Now() (which only verified test setup) with assertions that check sessionInfo.Credentials.Expiration relative to time.Now(), thus exercising the code under test. Include clarifying comment for intent. * feat(sts): Add IsExpired helpers and use them in expiration tests - Add Credentials.IsExpired() and SessionInfo.IsExpired() in new file session_helpers.go. - Update TestSTSSessionClaimsToSessionInfoCredentialExpiration to use helpers for clearer intent. * test: revert test-only IsExpired helpers; restore direct expiration assertions Remove session_helpers.go and update TestSTSSessionClaimsToSessionInfoCredentialExpiration to assert against sessionInfo.Credentials.Expiration directly as requested by reviewer., * fix(s3api): restore error return when access key not found Critical fix: The previous cleanup of sensitive logging inadvertently removed the error return statement when access key lookup fails. This caused the code to continue and call isCredentialExpired() on nil pointer, crashing the server. This explains EOF errors in CORS tests - server was panicking on requests with invalid keys. * fix(sts): make secret access key deterministic based on sessionId CRITICAL FIX: The secret access key was being randomly generated, causing signature verification failures when the same session token was used twice: 1. AssumeRoleWithWebIdentity generates random secret key X 2. Client signs request using secret key X 3. Server validates token, regenerates credentials via ToSessionInfo() 4. ToSessionInfo() calls generateSecretAccessKey(), which generates random key Y 5. Server tries to verify signature using key Y, but signature was made with X 6. Signature verification fails (SignatureDoesNotMatch) Solution: Make generateSecretAccessKey() deterministic by using SHA256 hash of 'secret-key:' + sessionId, just like generateAccessKeyId() already does. This ensures: - AssumeRoleWithWebIdentity generates deterministic secret key from sessionId - ToSessionInfo() regenerates the same secret key from the same sessionId - Client signature verification succeeds because keys match Fixes: AWS SDK v2 CORS tests failing with 'ExpiredToken' errors Affected files: - weed/iam/sts/token_utils.go: Updated generateSecretAccessKey() signature and implementation to be deterministic - Updated GenerateTemporaryCredentials() to pass sessionId parameter Tests: All 54 STS tests pass with this fix * test(sts): add comprehensive secret key determinism test coverage Updated tests to verify that secret access keys are now deterministic: 1. Updated TestSTSSessionClaimsToSessionInfoCredentialGeneration: - Changed comment from 'NOT deterministic' to 'NOW deterministic' - Added assertion that same sessionId produces identical secret key - Explains why this is critical for signature verification 2. Added TestSecretAccessKeyDeterminism (new dedicated test): - Verifies secret key is identical across multiple calls with same sessionId - Verifies access key ID and session token are also identical - Verifies different sessionIds produce different credentials - Includes detailed comments explaining why determinism is critical These tests ensure that the STS implementation correctly regenerates deterministic credentials during signature verification. Without determinism, signature verification would always fail because the server would use different secret keys than the client used to sign. * refactor(sts): add explicit zero-time expiration handling Improved defensive programming in IsExpired() methods: 1. Credentials.IsExpired(): - Added explicit check for zero-time expiration (time.Time{}) - Treats uninitialized credentials as expired - Prevents accidentally treating uninitialized creds as valid 2. SessionInfo.IsExpired(): - Added same explicit zero-time check - Treats uninitialized sessions as expired - Protects against bugs where sessions might not be properly initialized This is important because time.Now().After(time.Time{}) returns true, but explicitly checking for zero time makes the intent clear and helps catch initialization bugs during code review and debugging. * refactor(sts): remove unused IsExpired() helper functions The session_helpers.go file contained two unused IsExpired() methods: - Credentials.IsExpired() - SessionInfo.IsExpired() These were never called anywhere in the codebase. The actual expiration checks use: - isCredentialExpired() in weed/s3api/auth_credentials.go (S3 auth) - Direct time.Now().After() checks Removing unused code improves code clarity and reduces maintenance burden. * fix(auth): pass STS session token to IAM authorization for V4 signature auth CRITICAL FIX: Session tokens were not being passed to the authorization check when using AWS Signature V4 authentication with STS credentials. The bug: 1. AWS SDK sends request with X-Amz-Security-Token header (V4 signature) 2. validateSTSSessionToken validates the token, creates Identity with PrincipalArn 3. authorizeWithIAM only checked X-SeaweedFS-Session-Token (JWT auth header) 4. Since it was empty, fell into 'static V4' branch which set SessionToken = '' 5. AuthorizeAction returned ErrAccessDenied because SessionToken was empty The fix (in authorizeWithIAM): - Check X-SeaweedFS-Session-Token first (JWT auth) - If empty, fallback to X-Amz-Security-Token header (V4 STS auth) - If still empty, check X-Amz-Security-Token query param (presigned URLs) - When session token is found with PrincipalArn, use 'STS V4 signature' path - Only use 'static V4' path when there's no session token This ensures: - JWT Bearer auth with session tokens works (existing path) - STS V4 signature auth with session tokens works (new path) - Static V4 signature auth without session tokens works (existing path) Logging updated to distinguish: - 'JWT-based IAM authorization' - 'STS V4 signature IAM authorization' (new) - 'static V4 signature IAM authorization' (clarified) * test(s3api): add comprehensive STS session token authorization test coverage Added new test file auth_sts_v4_test.go with comprehensive tests for the STS session token authorization fix: 1. TestAuthorizeWithIAMSessionTokenExtraction: - Verifies X-SeaweedFS-Session-Token is extracted from JWT auth headers - Verifies X-Amz-Security-Token is extracted from V4 STS auth headers - Verifies X-Amz-Security-Token is extracted from query parameters (presigned URLs) - Verifies JWT tokens take precedence when both are present - Regression test for the bug where V4 STS tokens were not being passed to authorization 2. TestSTSSessionTokenIntoCredentials: - Verifies STS credentials have all required fields (AccessKeyId, SecretAccessKey, SessionToken) - Verifies deterministic generation from sessionId (same sessionId = same credentials) - Verifies different sessionIds produce different credentials - Critical for signature verification: same session must regenerate same secret key 3. TestActionConstantsForV4Auth: - Verifies S3 action constants are available for authorization checks - Ensures ACTION_READ, ACTION_WRITE, etc. are properly defined These tests ensure that: - V4 Signature auth with STS tokens properly extracts and uses session tokens - Session tokens are prioritized correctly (JWT > X-Amz-Security-Token header > query param) - STS credentials are deterministically generated for signature verification - The fix for passing STS session tokens to authorization is properly covered All 3 test functions pass (6 test cases total). * refactor(s3api): improve code quality and performance - Rename authorization path constants to avoid conflict with existing authType enum - Replace nested if/else with clean switch statement in authorizeWithIAM() - Add determineIAMAuthPath() helper for clearer intent and testability - Optimize key counting in auth_signature_v4.go: remove unnecessary slice allocation - Fix timing assertion in session_claims_test.go: use WithinDuration for symmetric tolerance These changes improve code readability, maintainability, and performance while maintaining full backward compatibility and test coverage. * refactor(s3api): use typed iamAuthPath for authorization path constants - Define iamAuthPath as a named string type (similar to existing authType enum) - Update constants to use explicit type: iamAuthPathJWT, iamAuthPathSTS_V4, etc. - Update determineIAMAuthPath() to return typed iamAuthPath - Improves type safety and prevents accidental string value misuse |
||
|
Chris Lu
|
ae9a943ef6 |
IAM: Add Service Account Support (#7744) (#7901)
* iam: add ServiceAccount protobuf schema Add ServiceAccount message type to iam.proto with support for: - Unique ID and parent user linkage - Optional expiration timestamp - Separate credentials (access key/secret) - Action restrictions (subset of parent) - Enable/disable status This is the first step toward implementing issue #7744 (IAM Service Account Support). * iam: add service account response types Add IAM API response types for service account operations: - ServiceAccountInfo struct for marshaling account details - CreateServiceAccountResponse - DeleteServiceAccountResponse - ListServiceAccountsResponse - GetServiceAccountResponse - UpdateServiceAccountResponse Also add type aliases in iamapi package for backwards compatibility. Part of issue #7744 (IAM Service Account Support). * iam: implement service account API handlers Add CRUD operations for service accounts: - CreateServiceAccount: Creates service account with ABIA key prefix - DeleteServiceAccount: Removes service account and parent linkage - ListServiceAccounts: Lists all or filtered by parent user - GetServiceAccount: Retrieves service account details - UpdateServiceAccount: Modifies status, description, expiration Service accounts inherit parent user's actions by default and support optional expiration timestamps. Part of issue #7744 (IAM Service Account Support). * sts: add AssumeRoleWithWebIdentity HTTP endpoint Add STS API HTTP endpoint for AWS SDK compatibility: - Create s3api_sts.go with HTTP handlers matching AWS STS spec - Support AssumeRoleWithWebIdentity action with JWT token - Return XML response with temporary credentials (AccessKeyId, SecretAccessKey, SessionToken) matching AWS format - Register STS route at POST /?Action=AssumeRoleWithWebIdentity This enables AWS SDKs (boto3, AWS CLI, etc.) to obtain temporary S3 credentials using OIDC/JWT tokens. Part of issue #7744 (IAM Service Account Support). * test: add service account and STS integration tests Add integration tests for new IAM features: s3_service_account_test.go: - TestServiceAccountLifecycle: Create, Get, List, Update, Delete - TestServiceAccountValidation: Error handling for missing params s3_sts_test.go: - TestAssumeRoleWithWebIdentityValidation: Parameter validation - TestAssumeRoleWithWebIdentityWithMockJWT: JWT token handling Tests skip gracefully when SeaweedFS is not running or when IAM features are not configured. Part of issue #7744 (IAM Service Account Support). * iam: address code review comments - Add constants for service account ID and key lengths - Use strconv.ParseInt instead of fmt.Sscanf for better error handling - Allow clearing descriptions by checking key existence in url.Values - Replace magic numbers (12, 20, 40) with named constants Addresses review comments from gemini-code-assist[bot] * test: add proper error handling in service account tests Use require.NoError(t, err) for io.ReadAll and xml.Unmarshal to prevent silent failures and ensure test reliability. Addresses review comment from gemini-code-assist[bot] * test: add proper error handling in STS tests Use require.NoError(t, err) for io.ReadAll and xml.Unmarshal to prevent silent failures and ensure test reliability. Repeated this fix throughout the file. Addresses review comment from gemini-code-assist[bot] in PR #7901. * iam: address additional code review comments - Specific error code mapping for STS service errors - Distinguish between Sender and Receiver error types in STS responses - Add nil checks for credentials in List/GetServiceAccount - Validate expiration date is in the future - Improve integration test error messages (include response body) - Add credential verification step in service account tests Addresses remaining review comments from gemini-code-assist[bot] across multiple files. * iam: fix shared slice reference in service account creation Copy parent's actions to create an independent slice for the service account instead of sharing the underlying array. This prevents unexpected mutations when the parent's actions are modified later. Addresses review comment from coderabbitai[bot] in PR #7901. * iam: remove duplicate unused constant Removed redundant iamServiceAccountKeyPrefix as ServiceAccountKeyPrefix is already defined and used. Addresses remaining cleanup task. * sts: document limitation of string-based error mapping Added TODO comment explaining that the current string-based error mapping approach is fragile and should be replaced with typed errors from the STS service in a future refactoring. This addresses the architectural concern raised in code review while deferring the actual implementation to a separate PR to avoid scope creep in the current service account feature addition. * iam: fix remaining review issues - Add future-date validation for expiration in UpdateServiceAccount - Reorder tests so credential verification happens before deletion - Fix compilation error by using correct JWT generation methods Addresses final review comments from coderabbitai[bot]. * iam: fix service account access key length The access key IDs were incorrectly generated with 24 characters instead of the AWS-standard 20 characters. This was caused by generating 20 random characters and then prepending the 4-character ABIA prefix. Fixed by subtracting the prefix length from AccessKeyLength, so the final key is: ABIA (4 chars) + random (16 chars) = 20 chars total. This ensures compatibility with S3 clients that validate key length. * test: add comprehensive service account security tests Added comprehensive integration tests for service account functionality: - TestServiceAccountS3Access: Verify SA credentials work for S3 operations - TestServiceAccountExpiration: Test expiration date validation and enforcement - TestServiceAccountInheritedPermissions: Verify parent-child relationship - TestServiceAccountAccessKeyFormat: Validate AWS-compatible key format (ABIA prefix, 20 char length) These tests ensure SeaweedFS service accounts are compatible with AWS conventions and provide robust security coverage. * iam: remove unused UserAccessKeyPrefix constant Code cleanup to remove unused constants. * iam: remove unused iamCommonResponse type alias Code cleanup to remove unused type aliases. * iam: restore and use UserAccessKeyPrefix constant Restored UserAccessKeyPrefix constant and updated s3api tests to use it instead of hardcoded strings for better maintainability and consistency. * test: improve error handling in service account security tests Added explicit error checking for io.ReadAll and xml.Unmarshal in TestServiceAccountExpiration to ensure failures are reported correctly and cleanup is performed only when appropriate. Also added logging for failed responses. * test: use t.Cleanup for reliable resource cleanup Replaced defer with t.Cleanup to ensure service account cleanup runs even when require.NoError fails. Also switched from manual error checking to require.NoError for more idiomatic testify usage. * iam: add CreatedBy field and optimize identity lookups - Added createdBy parameter to CreateServiceAccount to track who created each service account - Extract creator identity from request context using GetIdentityNameFromContext - Populate created_by field in ServiceAccount protobuf - Added findIdentityByName helper function to optimize identity lookups - Replaced nested loops with O(n) helper function calls in CreateServiceAccount and DeleteServiceAccount This addresses code review feedback for better auditing and performance. * iam: prevent user deletion when service accounts exist Following AWS IAM behavior, prevent deletion of users that have active service accounts. This ensures explicit cleanup and prevents orphaned service account resources with invalid ParentUser references. Users must delete all associated service accounts before deleting the parent user, providing safer resource management. * sts: enhance TODO with typed error implementation guidance Updated TODO comment with detailed implementation approach for replacing string-based error matching with typed errors using errors.Is(). This provides a clear roadmap for a follow-up PR to improve error handling robustness and maintainability. * iam: add operational limits for service account creation Added AWS IAM-compatible safeguards to prevent resource exhaustion: - Maximum 100 service accounts per user (LimitExceededException) - Maximum 1000 character description length (InvalidInputException) These limits prevent accidental or malicious resource exhaustion while not impacting legitimate use cases. * iam: add missing operational limit constants Added MaxServiceAccountsPerUser and MaxDescriptionLength constants that were referenced in the previous commit but not defined. * iam: enforce service account expiration during authentication CRITICAL SECURITY FIX: Expired service account credentials were not being rejected during authentication, allowing continued access after expiration. Changes: - Added Expiration field to Credential struct - Populate expiration when loading service accounts from configuration - Check expiration in all authentication paths (V2 and V4 signatures) - Return ErrExpiredToken for expired credentials This ensures expired service accounts are properly rejected at authentication time, matching AWS IAM behavior and preventing unauthorized access. * iam: fix error code for expired service account credentials Use ErrAccessDenied instead of non-existent ErrExpiredToken for expired service account credentials. This provides appropriate access denial for expired credentials while maintaining AWS-compatible error responses. * iam: fix remaining ErrExpiredToken references Replace all remaining instances of non-existent ErrExpiredToken with ErrAccessDenied for expired service account credentials. * iam: apply AWS-standard key format to user access keys Updated CreateAccessKey to generate AWS-standard 20-character access keys with AKIA prefix for regular users, matching the format used for service accounts. This ensures consistency across all access key types and full AWS compatibility. - Access keys: AKIA + 16 random chars = 20 total (was 21 chars, no prefix) - Secret keys: 40 random chars (was 42, now matches AWS standard) - Uses AccessKeyLength and UserAccessKeyPrefix constants * sts: replace fragile string-based error matching with typed errors Implemented robust error handling using typed errors and errors.Is() instead of fragile strings.Contains() matching. This decouples the HTTP layer from service implementation details and prevents errors from being miscategorized if error messages change. Changes: - Added typed error variables to weed/iam/sts/constants.go: * ErrTypedTokenExpired * ErrTypedInvalidToken * ErrTypedInvalidIssuer * ErrTypedInvalidAudience * ErrTypedMissingClaims - Updated STS service to wrap provider authentication errors with typed errors - Replaced strings.Contains() with errors.Is() in HTTP layer for error checking - Removed TODO comment as the improvement is now implemented This makes error handling more maintainable and reliable. * sts: eliminate all string-based error matching with provider-level typed errors Completed the typed error implementation by adding provider-level typed errors and updating provider implementations to return them. This eliminates ALL fragile string matching throughout the entire error handling stack. Changes: - Added typed error definitions to weed/iam/providers/errors.go: * ErrProviderTokenExpired * ErrProviderInvalidToken * ErrProviderInvalidIssuer * ErrProviderInvalidAudience * ErrProviderMissingClaims - Updated OIDC provider to wrap JWT validation errors with typed provider errors - Replaced strings.Contains() with errors.Is() in STS service for error mapping - Complete error chain: Provider -> STS -> HTTP layer, all using errors.Is() This provides: - Reliable error classification independent of error message content - Type-safe error checking throughout the stack - No order-dependent string matching - Maintainable error handling that won't break with message changes * oidc: use jwt.ErrTokenExpired instead of string matching Replaced the last remaining string-based error check with the JWT library's exported typed error. This makes the error detection independent of error message content and more robust against library updates. Changed from: strings.Contains(errMsg, "expired") To: errors.Is(err, jwt.ErrTokenExpired) This completes the elimination of ALL string-based error matching throughout the entire authentication stack. * iam: add description length validation to UpdateServiceAccount Fixed inconsistency where UpdateServiceAccount didn't validate description length against MaxDescriptionLength, allowing operational limits to be bypassed during updates. Now validates that updated descriptions don't exceed 1000 characters, matching the validation in CreateServiceAccount. * iam: refactor expiration check into helper method Extracted duplicated credential expiration check logic into a helper method to reduce code duplication and improve maintainability. Added Credential.isCredentialExpired() method and replaced 5 instances of inline expiration checks across auth_signature_v2.go and auth_signature_v4.go. * iam: address critical Copilot security and consistency feedback Fixed three critical issues identified by Copilot code review: 1. SECURITY: Prevent loading disabled service account credentials - Added check to skip disabled service accounts during credential loading - Disabled accounts can no longer authenticate 2. Add DurationSeconds validation for STS AssumeRoleWithWebIdentity - Enforce AWS-compatible range: 900-43200 seconds (15 min - 12 hours) - Returns proper error for out-of-range values 3. Fix expiration update consistency in UpdateServiceAccount - Added key existence check like Description field - Allows explicit clearing of expiration by setting to empty string - Distinguishes between "not updating" and "clearing expiration" * sts: remove unused durationSecondsStr variable Fixed build error from unused variable after refactoring duration parsing. * iam: address remaining Copilot feedback and remove dead code Completed remaining Copilot code review items: 1. Remove unused getPermission() method (dead code) - Method was defined but never called anywhere 2. Improve slice modification safety in DeleteServiceAccount - Replaced append-with-slice-operations with filter pattern - Avoids potential issues from mutating slice during iteration 3. Fix route registration order - Moved STS route registration BEFORE IAM route - Prevents IAM route from intercepting STS requests - More specific route (with query parameter) now registered first * iam: improve expiration validation and test cleanup robustness Addressed additional Copilot feedback: 1. Make expiration validation more explicit - Added explicit check for negative values - Added comment clarifying that 0 is allowed to clear expiration - Improves code readability and intent 2. Fix test cleanup order in s3_service_account_test.go - Track created service accounts in a slice - Delete all service accounts before deleting parent user - Prevents DeleteConflictException during cleanup - More robust cleanup even if test fails mid-execution Note: s3_service_account_security_test.go already had correct cleanup order due to LIFO defer execution. * test: remove redundant variable assignments Removed duplicate assignments of createdSAId, createdAccessKeyId, and createdSecretAccessKey on lines 148-150 that were already assigned on lines 132-134. |
||
|
Chris Luandgemini-code-assist[bot]
|
716f21fbd3 |
s3: support STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER for signed chunked uploads with checksums (#7623)
* s3: support STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER for signed chunked uploads with checksums When AWS SDK v2 clients upload with both chunked encoding and checksum validation enabled, they use the x-amz-content-sha256 header value of STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER instead of the simpler STREAMING-AWS4-HMAC-SHA256-PAYLOAD. This caused the chunked reader to not be properly activated, resulting in chunk-signature metadata being stored as part of the file content. Changes: - Add streamingSignedPayloadTrailer constant for the new header value - Update isRequestSignStreamingV4() to recognize this header - Update newChunkedReader() to handle this streaming type - Update calculateSeedSignature() to accept this header - Add unit test for signed streaming upload with trailer Fixes issue where Quarkus/AWS SDK v2 uploads with checksum validation resulted in corrupted file content containing chunk-signature data. * address review comments: add trailer signature to test, fix constant alignment * test: separate canonical trailer text (\n) from on-wire format (\r\n) * test: add negative test for invalid trailer signature * refactor: check HTTP method first in streaming auth checks (fail-fast) * test: handle crc32 Write error return for completeness * refactor: extract createTrailerStreamingRequest helper to reduce test duplication * fmt * docs: clarify test comment about trailer signature validation status * refactor: calculate chunk data length dynamically instead of hardcoding * Update weed/s3api/chunked_reader_v4_test.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * fix: use current time for signatures instead of hardcoded past date --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> |
||
|
chrislu
|
a77dfb1ddd | add debugging for InvalidAccessKeyId | ||
|
Chris Luandgemini-code-assist[bot]
|
9f07bca9cc |
Fix IPv6 host header formatting to match AWS SDK behavior (#7414)
* Add nginx reverse proxy documentation for S3 API Fixes #7407 Add comprehensive documentation and example configuration for using nginx as a reverse proxy with SeaweedFS S3 API while maintaining AWS Signature V4 authentication compatibility. Changes: - Add docker/nginx/README.md with detailed setup guide - Add docker/nginx/s3-example.conf with working configuration - Update docker/nginx/proxy.conf with important S3 notes The documentation covers: - Critical requirements for AWS Signature V4 authentication - Common mistakes and why they break S3 authentication - Complete working nginx configurations - Debugging tips and troubleshooting - Performance tuning recommendations * Fix IPv6 host header formatting to match AWS SDK behavior Follow-up to PR #7403 When a default port (80 for HTTP, 443 for HTTPS) is stripped from an IPv6 address, the square brackets should also be removed to match AWS SDK behavior for S3 signature calculation. Reference: https://github.com/aws/aws-sdk-go-v2/blob/main/aws/signer/internal/v4/host.go The AWS SDK's stripPort function explicitly removes brackets when returning an IPv6 address without a port. Changes: - Update extractHostHeader to strip brackets from IPv6 addresses when no port or default port is used - Update test expectations to match AWS SDK behavior - Add detailed comments explaining the AWS SDK compatibility requirement This ensures S3 signature validation works correctly with IPv6 addresses behind reverse proxies, matching AWS S3 canonical request format. Fixes the issue raised in PR #7403 comment: https://github.com/seaweedfs/seaweedfs/pull/7403#issuecomment-3471105438 * Update docker/nginx/README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Add nginx reverse proxy documentation for S3 API Fixes #7407 Add comprehensive documentation and example configuration for using nginx as a reverse proxy with SeaweedFS S3 API while maintaining AWS Signature V4 authentication compatibility. Changes: - Add docker/nginx/README.md with detailed setup guide - Add docker/nginx/s3-example.conf with working configuration - Update docker/nginx/proxy.conf with important S3 notes The documentation covers: - Critical requirements for AWS Signature V4 authentication - Common mistakes and why they break S3 authentication - Complete working nginx configurations - Debugging tips and troubleshooting - Performance tuning recommendations Fix IPv6 host header formatting to match AWS SDK behavior Follow-up to PR #7403 When a default port (80 for HTTP, 443 for HTTPS) is stripped from an IPv6 address, the square brackets should also be removed to match AWS SDK behavior for S3 signature calculation. Reference: https://github.com/aws/aws-sdk-go-v2/blob/main/aws/signer/internal/v4/host.go The AWS SDK's stripPort function explicitly removes brackets when returning an IPv6 address without a port. Changes: - Update extractHostHeader to strip brackets from IPv6 addresses when no port or default port is used - Update test expectations to match AWS SDK behavior - Add detailed comments explaining the AWS SDK compatibility requirement This ensures S3 signature validation works correctly with IPv6 addresses behind reverse proxies, matching AWS S3 canonical request format. Fixes the issue raised in PR #7403 comment: https://github.com/seaweedfs/seaweedfs/pull/7403#issuecomment-3471105438 * Revert "Merge branch 'fix-ipv6-brackets-default-port' of https://github.com/seaweedfs/seaweedfs into fix-ipv6-brackets-default-port" This reverts commit |
||
|
|
7e624d5355 |
* Fix s3 auth with proxy request (#7403)
* * Fix s3 auth with proxy request * * 6649 Add unit test for signature v4 * address comments * fix for tests * ipv6 * address comments * setting scheme Works for both cases (direct HTTPS and behind proxy) * trim for ipv6 * Corrected Scheme Precedence Order * trim * accurate --------- Co-authored-by: chrislu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
7d26c8838f |
S3: auth supports X-Forwarded-Host and X-Forwarded-Port (#7398)
* add fix and tests * address comments * idiomatic * ipv6 |
||
|
Tom Crasset
|
824dcac3bf | s3: combine all signature verification checks into a single function (#7330) | ||
|
Chris Lu
|
3d25f206c8 |
S3: Signature verification should not check permissions (#7335)
* Signature verification should not check permissions - that's done later in authRequest * test permissions during signature verfication * fix s3 test path * s3tests_boto3 => s3tests * remove extra lines |