mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-12 01:07:35 +02:00
aa4252d4914abef1aa06296c8bf90f9f5636da90
179
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
fac021544e |
s3api: register the azurekms KMS provider under its build tag (#11691)
* s3api: register the azurekms KMS provider under its build tag weed/kms/azure registers itself with weed/kms from an init(), but the blank import that triggers it was commented out in auth_credentials.go (TODO: Fix Azure SDK compatibility issues). Nothing else imports the package, so building with -tags azurekms still ends with "KMS provider 'azure' not registered" at startup for any config using "type": "azure". Move the import into weed/s3api/kms_azure.go behind //go:build azurekms so default builds stay free of the Azure SDK, and drop the dead comment. Closes #11686 * ci: compile and test azurekms-gated code weed/kms/azure is excluded from every default build, so nothing in CI compiled it; that is how the provider shipped unregistered. Build the tree and run the kms tests with -tags azurekms on every Go change. --------- Co-authored-by: Yi-111-a <> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
c023493cf0 |
s3api: tolerate session tokens on statically configured credentials (#11694)
* s3api: consolidate session token extraction into extractSessionToken Three call sites duplicated the same header/header/query lookup; share one helper named after the existing s3tables equivalent. * s3api: tolerate session tokens on statically configured credentials Credential vendors like Unity Catalog emit a session token with every vended credential, including static ones (UC's StaticAwsCredentialGenerator only engages when s3.sessionToken is set). Requests signed by a configured access key were routed to STS validation and rejected, so static credential vending never worked against SeaweedFS. Resolve the access key first: when it maps to a configured credential the signature alone authenticates the request, and the attached token is marked ignored so authorization does not route it into the STS session-policy path. STS-issued access keys are never in the static map, so temporary credentials still validate their token exactly as before. * s3api: cover static credentials carrying a session token * test: exercise UC static credential vending against SeaweedFS s3.sessionToken.0 selects UC's StaticAwsCredentialGenerator, which vends the configured keys verbatim. The vended session token is foreign to SeaweedFS and previously failed SigV4; now it round-trips through temporary-table-credentials into real S3 I/O. * s3api: reject temporary credentials in GetFederationToken after auth Token presence alone cannot distinguish a temporary credential from a statically configured one carrying a vended token. Move the check behind verifyV4Signature and key it on the operative session token so tolerated tokens keep the caller eligible. * s3api: test GetFederationToken with authenticated temporary credentials Sign the rejection cases with real session credentials so they reach the post-auth check, and cover a vended static credential being accepted. * test: check vended-credential delete error in UC integration test |
||
|
Paolo Valletta
|
4d306e1279 |
s3api: revoke the identities a config reload no longer declares (#11679)
* s3api: revoke the identities a config reload no longer declares A static config file reload merges with isFullState=false, so an identity deleted from -s3.config kept authenticating with all of its access keys until the process restarted. An operator revoking a leaked key got no error and no revocation. The file is the source of truth for the identities it declares, so a reload now drops the ones that have left it, together with their access keys and those of their service accounts. staticIdentityNames was only ever added to, which kept a removed name protected as well; the names the file itself declares are tracked separately from the AWS environment credentials, so a reload never revokes what the file never declared. * s3api: keep a file reload from replacing the dynamic store Removing the last identity from a config file emptied staticIdentityNames, so useStaticConfig turned false and the next reload of that file took the replace path: every filer-managed identity and its access keys disappeared until a dynamic reload brought them back. A static config file is authoritative for the identities it declares and never for the dynamic store, so a file load now always merges. The first file load at startup takes the merge path as well, from an empty state. Reported by the Devin and Greptile reviews of this PR. TestReloadStaticConfigWithoutIdentitiesKeepsDynamic reloads an emptied file twice and asserts that a filer-managed identity and its access key survive both; the existing test now also checks that the service account key was loaded before the reload and that the environment identity's key still works after it. * s3api: clear the environment credentials in the emptied-file test The AWS environment identity is static and is re-added after every merge, so on a runner that has AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set it kept hasStaticConfig true once the file was emptied, and the next reload still took the merge path: the regression this test guards passed unnoticed. Clear both variables whatever the runner has set. With the pre-fix condition (hasStaticConfig alone) and the variables present in the environment, the test now fails with "reload 2: a filer-managed identity must survive a reload of an emptied file". Reported by the Greptile review of this PR. |
||
|
zhao-ycandChris Lu
|
483dd4b12e |
s3api: persist ACLs on PutObject uploads (#11592)
* s3api: persist ACLs on PutObject uploads Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com> * s3api: fix PutObject ACL edge cases found in review - Only enforce BucketOwnerEnforced when explicitly configured; buckets without a stored ownership control keep accepting upload ACLs - Ignore ACL query parameters on SigV2 requests, which do not sign them - Mirror signed-query ACL values into headers after authentication so grant parsing and resolveFileMode agree on presigned uploads - Validate only caller-supplied grantees against the account registry; default grants now work for accounts outside the local registry - Reject unknown grantee keys and accept comma-separated grantee lists without spaces in ParseCustomAclHeader - Guard against identities without an account * s3api: harden upload ACL parsing and authorization Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com> * s3api: evaluate upload ACL grantees individually in policies A comma-joined grant header or a signed query parameter reached policy conditions as one value, so a deny on a later grantee did not fire. Split grant headers into per-grantee values for policy evaluation and share the grantee pair parser with ParseCustomAclHeader. * s3api: keep raw grant header values visible to policy conditions Exact-match conditions written against the signed header value stopped matching once grantees were split for evaluation. Preserve the original wire values alongside the per-grantee values so deny policies fire on either granularity. * s3api: evaluate upload ACL grants as one canonical list in policies Conditions on s3:x-amz-grant-* now see a single comma-separated canonical grant list identical for a single line, repeated header lines, or a signed query parameter. This keeps StringEquals allows and exact-list or allowlist (StringNotEquals) denies accurate regardless of wire encoding. * s3api: preserve upload ACL denies and align policy checks Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com> * s3api: retain upload owner grants and literal policy values Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com> --------- Signed-off-by: zhaoyuchen <43179751+zhao-yc@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
f9289f0570 |
s3: do not promote ?prefix into the object for non-List actions (#11494)
* s3: do not promote ?prefix into the object for non-List actions authRequestWithAuthType mapped an empty object to the prefix parameter for every action, so PUT /bucket?versioning&prefix=x authorized as Write:bucket/x. An object-scoped grant (Write:bucket/*) could then change bucket versioning, lifecycle, cors, and object-lock configuration, and the promoted object also made ResolveS3Action report s3:PutObject to attached IAM policies. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: treat GET ?uploads as a bucket listing for authorization Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: resolve the listing action through the bucket-level object resolveS3AuthTarget fed the promoted prefix to ResolveS3Action, so a bucket-level ?uploads request resolved as s3:GetObject on the prefix ARN in the admin explicit-deny check. Resolve both action and resource against the object the bucket listing actually scopes. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: resolve the listing action through the bucket-level object in AuthorizeAction Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: drop the unreachable object-level uploads case from the resolver test Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
4303b3aa4c |
s3: keep a listing's start position inside the requested prefix (#11493)
* s3: a list marker that sorts past the prefix leaves nothing to list AWS scopes a listing to keys under Prefix; StartAfter, Marker and continuation tokens only reposition inside that range. A marker that diverges from the prefix at a larger byte is after every key the prefix can match, so the page is empty. normalizePrefixMarker used to keep such a marker as the walk cutoff at the bucket root, where the walk descends into the marker's own directory and returns keys the prefix never names. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: keep the listing variant's action when a prefix is promoted to object authRequestWithAuthType promotes ?prefix= into the object argument for the legacy CanDo path. ResolveS3Action treats a non-empty object as object-level, so a bucket-level ?versions or ?uploads request carrying a prefix missed its specific action and fell back to the base List action: an s3:ListBucket grant then covered s3:ListBucketVersions, and an explicit Deny on the specific action was skipped on the same path. Resolve the action against the same bucket-level object the resource ARN already uses. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: treat GET ?uploads as a bucket listing for authorization Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Update weed/s3api/auth_credentials.go Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
be29f44d87 |
s3: record requester identity before the authz verdict (#11479)
* s3: record requester identity before the authz verdict for audit Identity was only stored in request context on the success branch, so denied requests reached WriteErrorResponse without requester attribution and audit entries had empty requester/requester_arn/requester_identity. Authentication failures still resolve no identity, so unauthenticated denials stay unattributed. Fixes #11474 Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: keep the resolved identity through authz denial in Auth Review follow-up on #11479 (devin): authRequest discarded the identity on every error, so a request that authenticated fine but failed the action check still reached handleAuthResult with no identity and the deny path could not audit a requester. Auth now calls authRequestWithAuthType directly, the same entry AuthPostPolicy uses, so the resolved identity reaches the error writer; a failed authN still resolves no identity and stays unattributed. The regression test now signs a denied request end to end through iam.Auth. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
0ca1c19821 |
s3api: unify auth error handling across s3tables, iceberg and lance (#11381)
* s3api: fail closed when S3 Tables signature verification fails * s3api: avoid nil Account dereference in S3 Tables auth log * iceberg: return auth error instead of falling back to DefaultAllow * lance: return auth error instead of falling back to DefaultAllow * s3api: stop trusting client-supplied s3-account-id The header is set by the server after successful authentication; scrub inbound values alongside the other internal headers, and apply the same admin guard to the header fallback branch of getAccountID that the identity branch already has. * test: cover table-catalog auth wrappers and principal resolution * test: configure anonymous identity where catalog clients do not sign * s3api: scrub s3-account-id after signature verification |
||
|
Chris Lu
|
02749c1192 |
s3api: configurable trusted-proxy allowlist for aws:SourceIp (#11302) (#11315)
* s3api: add TrustedProxies allowlist helper for aws:SourceIp extraction Introduces a policy_engine.TrustedProxies type that parses a comma-separated list of bare IPs and CIDRs (mirroring Guard.UpdateWhiteList) and extracts the client IP for aws:SourceIp condition evaluation. When the direct TCP peer is in the allowlist, X-Forwarded-For is walked right-to-left skipping trusted hops (then X-Real-Ip); otherwise the direct peer address is returned. This is the building block for restoring configurable forwarded-header trust removed in |
||
|
Chris Lu
|
a3638e479e |
fix(s3api/audit): surface OIDC identity claim in audit log for STS sessions (#11269)
* Add ResolveIdentityClaim helper for OIDC audit identity ComputeParentUser derives a stable per-identity hash from (sub, iss) for internal keying, but it is opaque and not human-readable. Audit logs for STS-assumed OIDC sessions currently surface that opaque value (or the random session id) as the requester, leaving no authoritative trace of the federated user. Add ResolveIdentityClaim next to ComputeParentUser to recover a human-readable, server-asserted identity attribute from the STS request context populated at federation time. It walks a priority list (preferred_username, email, name, sub) so a federated session always audits against a stable OIDC claim rather than a client-supplied role session name. For #11264 * Surface authoritative OIDC identity claim in S3 audit log For STS-assumed sessions minted from an OIDC web identity, the audit log requester field is the opaque session subject, which cannot be traced back to the federated user who performed the operation. The OIDC identity claims (preferred_username, email, sub) are already carried in the session request context and reach the auth layer as identity.Claims, but they were never surfaced to the audit log. Add a requester_identity field to the S3 access audit log, populated from the authoritative OIDC identity claim resolved via ResolveIdentityClaim. The claim is propagated through the shared identity holder (the same mechanism the requester name and principal ARN already use) so it survives the request-context copy that hides auth-set values from the outer audit middleware. The existing requester field is left unchanged for backward compatibility; requester_identity is empty for non-federated sessions, where requester already carries the real username. For #11264 * Gate OIDC audit identity on federation marker and harden resolver Address review feedback (Devin Review, Greptile) on the initial implementation: - Non-federated STS sessions no longer gain a false requester_identity. ValidateJWTWithClaims merges the JWT registered sub claim (the opaque session id) into RequestContext for sessions without an explicit request context, so the previous ResolveIdentityClaim fallback to sub surfaced that session id as an authoritative identity. Resolution is now gated on SessionInfo.ParentUser, which is set only for OIDC-federated sessions in AssumeRoleWithWebIdentity. The claim is resolved from the original sessionInfo.RequestContext (not the local claims map, whose sub the bearer path overwrites with the session subject) so SigV4 and bearer sessions surface the same identity. - ResolveIdentityClaim now trims whitespace and treats whitespace-only claims as absent, so a blank preferred_username no longer masks a usable email or sub. The resolved claim is carried on Identity.IdentityClaim (and IAMIdentity for the bearer path) rather than re-derived in recordIdentityInContext, making the federation gate explicit at the auth boundary. For #11264 * Resolve OIDC identity claim for external bearer tokens The external OIDC bearer path (a raw OIDC JWT presented directly, not via STS) populates Claims with preferred_username/email/name/sub from the validated token but did not set IdentityClaim, so requester_identity stayed blank for that authentication path. Resolve the claim there too — sub is the real OIDC subject on this path (not an STS session id), so no federation gate is needed. Also drop an ineffectual ctx assignment flagged by ineffassign in the audit test. For #11264 |
||
|
Chris Lu
|
5ff49909a0 |
fix(s3api/iam): avoid transient AccessDenied from full reloads on single IAM file changes (#11271)
* fix(s3api/iam): fail config snapshot on empty or malformed IAM files A full IAM reload reads every identity/policy/service-account/group file from the filer. When an external secrets tool rewrites a file, a reload that reads it mid-rewrite sees empty or partially-written content. The identity, policy and service-account loaders silently skipped such files (``continue``), so the snapshot was missing entries that still existed on disk. The atomic swap then installed an incomplete identity set while ``isAuthEnabled`` stayed on, denying unrelated clients mid-reload (#11259). The group loader and the read-error paths already fail the snapshot in this situation (a skipped entry reads as deleted). Apply the same behavior to empty content and unmarshal failures across the identity, policy, service-account and group loaders, so a transient mid-rewrite fails the reload (preserving the last known-good state) instead of silently dropping entries. * fix(s3api/iam): coalesce burst IAM config reloads through the reload queue onIamConfigChange did a full synchronous reload for every identity/policy file change event. When several independently-refreshing credentials rewrite their files within the same second, that produced a burst of dozens of back-to-back full reloads, each reading the whole store and widening the window where a mid-rewrite file is observed (#11259). Route every IAM config change through the existing coalescing reload queue (scheduleReload/reloadRetryLoop) instead. A burst of N events now collapses into a single reload (plus one tail reload for events that arrived while one was in flight). scheduleReload gains a reason argument for the existing log line; the reloadRetryLoop already retries failed reloads, so the per-event failure handoff is no longer needed. Tests that asserted on the synchronous reload now wire up the queue (centralized in newTestS3ApiServerWithMemoryIAM) and poll via waitForIdentity/waitForIdentityGone. Adds TestOnIamConfigChangeCoalescesBurstReloads showing 50 events coalesce into <=3 reloads. * fix(s3api/iam): skip non-JSON auxiliary files before failing IAM snapshot Per review: the multi-file loaders unmarshal every entry in an IAM directory, so a non-JSON auxiliary file (README, .DS_Store, a migration backup such as identity.json.old) would hit the new empty/malformed errors and reject the whole snapshot, blocking all later IAM reloads. Only *.json files are IAM objects (SeaweedFS writes identities, policies, service accounts and groups as <name>.json, and other call sites already gate on the .json suffix). Skip non-.json entries at the top of each loader loop, before reading content, so auxiliary files are ignored while empty/malformed .json files still fail the snapshot. Adds TestLoadConfigurationIgnoresNonJsonAuxiliaryFiles. * fix(s3api/iam): reject IAM files with empty identifiers and skip aux in listing Per review: - ListPolicyNames listed every regular entry in the policies directory as a policy name, including non-JSON auxiliary files, but GetPolicy cannot retrieve them. Apply the same .json suffix filter used by the loader so the list only exposes retrievable policies. - json.Unmarshal accepts `{}` and unknown fields. The identity and group loaders merge by the decoded Name (not the file name), so a `{}` file could install an empty-key record and displace a real one; the service-account loader accepted an empty Id. Validate Identity.Name, Group.Name and ServiceAccount.Id (via validateServiceAccountId) after unmarshal and fail the snapshot on empty identifiers. Adds TestFilerEtcStoreListPolicyNamesSkipsNonJsonAuxiliary and empty-identifier regression tests for identity, group and service-account files. |
||
|
Chris Lu
|
7fa2f75f30 |
s3: bucket-policy Allow must not override an identity explicit Deny (#11256)
* s3: add isActionExplicitlyDeniedByApplicablePolicies helper Add a helper that reports whether any applicable identity-side policy (attached IAM policies, enabled-group policies, or the IAM-integration session policy) explicitly denies an action. It reuses the existing evaluateAttachedIAMPolicies, resolveS3AuthTarget, buildPrincipalARN, and isActionExplicitlyDeniedByIAM helpers, and fails closed on evaluation errors. A nil identity has no identity-side policy plane, so the helper returns false to keep the bucket policy authoritative for anonymous access. No behavior change yet; the next commits apply it to the two bucket-policy Allow short-circuits. * s3: enforce identity explicit Deny before bucket-policy Allow authRequestWithAuthType short-circuits on a matching bucket-policy Allow and skips VerifyActionPermission, so an explicit Deny in an authenticated identity attached, group, or session policy is bypassed. A non-admin principal with s3:PutBucketPolicy can install a bucket-policy Allow for itself and read an object its identity policy explicitly denies. Before honoring a bucket-policy Allow, check the applicable identity-side policies for a matching explicit Deny via the new isActionExplicitlyDeniedByApplicablePolicies helper, and fail closed. The cross-account behavior is preserved: a bucket Allow still supplies the Allow an identity policy omits (implicit denial), and a nil identity keeps the bucket policy authoritative for anonymous access. Regression tests cover the explicit-Deny override, the implicit-deny Allow preservation, and the unmatched-key fall-through control. * s3: enforce identity explicit Deny in secondary object-key auth authorizeObjectKeyAction authorizes keys the request URL does not name (CopySource, DeleteObjects body keys, POST Object form keys) and shares the same bucket-policy Allow short-circuit as the primary path, so an explicit Deny in the identity, group, or session policy is bypassed the same way when a bucket policy allows the secondary key. Apply isActionExplicitlyDeniedByApplicablePolicies before accepting the bucket-policy Allow, mirroring the primary path. A regression test covers AuthorizeCopySource for both the explicit-Deny override and the implicit-deny Allow preservation. |
||
|
Chris Lu
|
557fffa350 |
iam: preserve native Admin when IAM policies are attached (#11226) (#11232)
* iam: expose tri-state result from attached policy evaluation evaluateIAMPolicies returned a bool that collapsed explicit Deny and no-match into a single false, so the authorization path could not tell "policies forbid this" from "policies say nothing". Introduce evaluateAttachedIAMPolicies returning Allow/Deny/NoMatch and keep evaluateIAMPolicies as a bool projection for existing callers. This is preparation for unioning native permissions with attached policies while preserving deny-always-wins. * iam: preserve native Admin when IAM policies are attached Attaching an IAM policy routed authorization exclusively to the attached policies, dropping the identity native permissions. A user with native Admin lost all access after attaching a non-granting policy, and stayed locked out if that policy was deleted without being detached first (#11226). Treat a native bare Admin grant as a permission floor that survives attached policies: when the attached policies do not explicitly allow, fall back to isAdmin() on the attached-policy path, and on the IAM integration path allow unless an attached policy explicitly denies. Explicit Deny still wins on both paths. Only bare Admin is consulted because inline policies flatten lossily into Actions (dropping conditions), so scoped actions are not unambiguously native and must keep flowing through the policy engine. * iam: regression tests for native Admin surviving attached policies Reproduces issue #11226: - TestNativeAdminSurvivesAttachedPolicy: a user with native Admin keeps Write access after attaching a policy that does not grant it. - TestNativeAdminSurvivesDeletedPolicy: the same user keeps Write access after the attached policy is deleted without being detached. - TestAttachedPolicyExplicitDenyOverridesNativeAdmin: an explicit Deny in an attached policy still constrains a native admin (deny-always-wins). * iam: apply native Admin floor before IAM principal validation The native Admin floor in authorizeWithIAM ran after the auth-path switch, which denies when no session principal or PrincipalArn is present. An Admin identity without a PrincipalArn (no session token) was therefore denied before the floor executed. Move the floor ahead of the switch and derive the principal for its explicit-deny check with buildPrincipalARN, which already handles identities without a PrincipalArn. Adds a regression case for an Admin identity with an empty PrincipalArn. Addresses CodeRabbit review feedback on PR #11232. |
||
|
Chris Lu
|
77a9dd4b9e |
s3: route per-key object authorization through a shared helper (#11072)
* s3: share the per-key object authorization across copy and delete AuthorizeCopySource and AuthorizeObjectDelete both authorize a key the request URL does not name by evaluating the bucket policy and IAM against a synthetic per-key request; only the method and action differed. Extract that into authorizeObjectKeyAction and make the two callers thin wrappers. No behavior change. Claude-Session: https://claude.ai/code/session_01Qo7p6VsoWxMo8816ogJFk5 * s3: route POST Object uploads through the shared object authorization POST Object uploads (presigned-POST / HTML form) authorized the write with only the coarse per-identity Write action, unlike the other write paths which also check the resolved object against the bucket policy and IAM. Route POST through authorizeObjectKeyAction via a new AuthorizeObjectWrite so it is authorized like the equivalent PUT. Claude-Session: https://claude.ai/code/session_01Qo7p6VsoWxMo8816ogJFk5 * s3: test POST Object per-key authorization Drives a signed POST upload and checks the per-key authorization decision for a denied, permitted, and admin caller. Claude-Session: https://claude.ai/code/session_01Qo7p6VsoWxMo8816ogJFk5 |
||
|
Chris Lu
|
f5f1dcbd8c |
s3: keep verifying the request host when externalUrl is set (#10970)
* s3: keep verifying the request host when externalUrl is set externalUrl was the only host candidate once set, so a client that dialed the gateway directly instead of through the proxy always got SignatureDoesNotMatch. Make it lead the candidate walk instead: every candidate still needs a valid signature, and the request-derived hosts are already trusted when the flag is unset, so a mixed proxy plus in-cluster topology can now advertise a public endpoint and verify both planes. * s3: cover virtual-hosted addressing behind externalUrl The old pin also rejected an external client that signed bucket.api.example.com, since only the bare externalUrl host was ever tried. The candidate walk covers it; pin the case down. |
||
|
Chris Lu
|
368b2035b2 |
s3: deny anonymous access when the identity config loads no identities (#10954)
* s3: deny anonymous requests when the identity config loads no identities Naming a config file is the operator asking for authentication. A file that yields no identity - an unpopulated secret mount, or a mistyped top-level key the proto parser silently drops - left the gateway open to every anonymous caller: ListBuckets returned 200, and anonymous PUT could create buckets and write objects. * s3: name the unknown top-level keys in an identity config The proto parser discards what it does not recognise, so a mistyped "identites" loads as an empty config. Naming the dropped keys at startup turns the resulting lockout into a one-line diagnosis. * s3: isolate the auth-enforcement tests from AWS environment credentials * s3: use a singular "identity" as the unrecognised-key example Codespell rejects the misspelling the example used. * s3: cover the empty identity config alongside the unrecognised key * s3: cover a config file whose body is an empty object |
||
|
Chris Lu
|
69cc2869ad |
Fixes from the review of the admin bucket policy UI (#10907)
* admin: treat a missing S3 Tables policy as an empty load, not an error
The bucket/table policy GET relayed the backend's 404 NoSuchPolicy to the
dialog, whose loader treats any non-OK response as a load failure and
keeps Save and Delete blocked. A bucket or table without a policy could
never be given one. Return policy null instead, the same contract
ShowBucketPolicy uses for classic buckets.
* admin: reject policy documents the structured editor would misread
A top-level JSON array passed the object guard (typeof [] is 'object')
and loaded as a zero-statement policy, which the next commit would
rewrite to an empty document. Object elements in Action/Resource were
coerced to '[object Object]' and saved that way on the s3tables surface,
which stores policies verbatim. Both now throw, which routes the
document to the JSON tab like other unrepresentable shapes.
* admin: let the JSON tab save documents the structured editor can't model
Save with the JSON tab active required a round-trip through
policyDocToEditorState, so exactly the documents the dialogs shunt to
'JSON tab only' mode (unrepresentable Effect, Resource+NotResource, and
the like) could never be saved - Delete was the only mutation left.
Invalid JSON still blocks; an unrepresentable document now saves and the
editor state stays marked unparsed.
* admin: pin the policy editor to what each consumer's backend supports
The s3tables evaluator has no NotResource/NotPrincipal fields - it
silently drops them, turning Allow+NotResource into allow-everything and
making Deny+NotPrincipal inert - and it only matches s3tables: actions
against s3tables ARNs, while the editor suggested s3: actions and
arn:aws:s3::: resources. New registerPolicyEditor knobs: allowNegation
hides the Not* modes and routes documents using them to the JSON tab;
resourceSuggestions pins the Resource autocomplete to the open
resource's ARN; the S3 Tables dialogs get an s3tables-only action
datalist. requirePrincipal now also hides NotPrincipal, which
policy_engine.ValidateBucketPolicy always rejects, and the client-side
check requires Principal specifically to match that server rule.
* admin: save S3 Tables policies from a button, not form submission
The multi-input structured editor sits inside a form whose Save button
was type=submit, so Enter in any single-line editor input - accepting an
autocomplete suggestion, say - implicitly submitted whatever half-built
statement the editor held, and the backend stores the document verbatim.
A lone statement with no Principal matches nobody, locking out every
non-owner. Save is now an ordinary button and the form ignores
submission.
* admin: block zero-statement policy saves
Committing the active tab before the emptiness check made 'Policy JSON
is required' dead code: an empty editor serializes to {"Statement":[]},
which the s3tables backend stores verbatim - evaluated default-deny for
every non-owner, while the statement-count column keeps showing 'Not
configured'. All three policy dialogs now refuse a save with no
statements and point at Delete instead. The classic bucket modal only
gained a clearer message; the server already rejected the document.
* admin: guard S3 Tables policy mutations against stale and overlapping requests
The save/delete completions ran against whatever resource the shared
modal happened to show by then: a slow PUT for one bucket would hide the
modal mid-edit of another and misattribute its alerts, a late DELETE
cleared the shared textarea over the newly opened resource with its
loaded flag set, and nothing stopped a double-click from firing two
overlapping mutations. Ported the classic modal's pattern: capture the
target on start, flag the mutation in flight with the buttons disabled,
and only touch the UI when the completion still matches the open
resource. Success now reloads the page, which also keeps the Policy
column's statement count honest.
* admin: confirm before deleting an S3 Tables policy
Delete Policy sat next to Save and fired on a single click; with
default-allow enabled one stray click silently dropped the resource
policy and left the bucket open to every principal. Same confirmation
the classic bucket modal already has.
* admin: let a corrupt stored bucket policy be shown, fixed, and deleted
A stored document the decoder rejects made the policy GET 500, and with
the loaded flag never set the modal blocked both Save and Delete - the
one policy an operator most needs to remove was the one they couldn't,
even though the delete path never reads the document. The GET now
returns the raw bytes alongside a null policy; the dialog hands them to
the JSON tab and unblocks the buttons.
* admin: url-encode the bucket name in the policy API calls
The filer lists any directory under the buckets path, names S3 would
never allow included; one carrying '#' or '%' broke the fetch URL or
addressed a different name than the modal shows.
* admin: drop stale edit-policy responses on the IAM policies page
The same race the bucket and S3 Tables dialogs already guard against:
open one policy's editor while its GET stalls, open another, and the
late response populates the editor under the second policy's name -
Update then saves the first policy's statements over the second.
* admin: warn before a bucket policy save drops unsupported fields
The editor tracks unmodeled top-level keys precisely so
confirmPolicyFieldDiscard can warn before the server's Version+Statement
decode discards them, but only the IAM page called it; the bucket modal
saved a pasted document with e.g. a console-generated Id without a word
while the editor kept displaying the field.
* s3: enforce the bucket policy size cap on both surfaces
The 20KB cap lived only in the admin UI, so a larger policy stored via
the S3 API displayed there but could never be re-saved, desyncing the
two writers the cap comment claimed could not desync. The constant now
lives in policy_engine next to the shared validator and PutBucketPolicy
rejects oversized documents with PolicyTooLarge, matching AWS.
* admin: ship the policy editor's fieldset styles with the editor
The .policy-stmt-* rules that undo Bootstrap's full-width legend reset
stayed behind in policies.templ when the editor markup moved to the
shared script, so the bucket and S3 Tables dialogs rendered Actions/
Resource/Principal as full-width jumbo headings. PolicyDatalists is the
component every consumer already renders once; the styles live there
now.
* s3: mirror bucket policy changes into the IAM store from the metadata subscription
The advanced-IAM path appends the bucket-policy:<bucket> document to
every STS/session evaluation, but only this gateway's own PutBucketPolicy
maintained that mirror - a policy tightened or created through the admin
UI (or another gateway) never reached it, so revoked access stayed live
indefinitely, and the delete side was an unimplemented TODO in any case.
The metadata subscription now diffs the stored policy on every bucket
entry change and updates or removes the mirror, covering all writers and
deletion with one mechanism; IAMManager gains the missing
RemoveBucketPolicy.
* admin: deduplicate the bucket policy write path
Set and Delete carried line-for-line identical filer closures;
bucketPolicyMutation already treats nil as clear-the-key. The shared
helper sits below Set's validation, since ValidatePolicy cannot take the
nil document Delete passes.
* s3: drop ValidateBucketPolicy's re-checks of ValidatePolicy rules
Both callers run ValidatePolicy first, which already enforces the
version and at-least-one-statement rules; the duplicates were dead code
with drifted error text.
* admin: seed a new statement's Resource from the pinned suggestions
A fresh statement on the S3 Tables dialogs started with no resource row
at all; seed it with the broadest pinned ARN the same way cfg.bucket
already seeds the classic modal.
* admin: refuse to save Not* fields the backend would silently drop
Hiding the NotResource/NotPrincipal modes was not enough where negation
is disallowed: the JSON tab accepts any valid document (that is its
job), and a statement's Advanced-fields box can reintroduce the keys, so
an s3tables save could still store fields the evaluator drops - turning
Allow+NotResource into allow-everything. commitPolicyActiveTab now runs
a final document-level check over what would actually be saved; Delete
stays available for cleanup.
* s3: move the IAM bucket policy mirror on a bucket rename
A same-directory rename delivers one event carrying both entries, and
the byte-equality short-circuit skipped the new name's mirror when the
policy was unchanged - while the replayed delete for the old name
removed its mirror, leaving the renamed bucket unmirrored. The mirror
decision is now a pure function that removes the old name and writes the
new one regardless of byte equality, with the rename cases unit tested.
* s3: backfill the IAM bucket policy mirror on lazy bucket loads
The metadata subscription only mirrors changes, so a policy that
predates the IAM integration never reached the bucket-policy:<bucket>
mirror and its grants did not bind on the IAM path until the policy was
next modified. The gateway is deliberately lazy at startup (nothing
lists all buckets), so the backfill hooks the same place a bucket's
policy first becomes known: the cold bucket-config load. EnsureBucketPolicy
writes only when no mirror is stored, so repeat loads cost one cached
read.
* s3: reconcile the bucket policy backfill against concurrent changes
The backfill's check-then-write could race an event-driven mirror update
or removal and re-store bytes that were already stale, with no later
event to heal it. EnsureBucketPolicy now reports whether it wrote, and a
write is reconciled against a fresh authoritative entry read: a changed
policy is re-mirrored, a removed one is removed. Anything changing after
that read fires its own event, which finds the backfill's write already
present and supersedes it. The backfill also carries the entry's raw
bytes rather than a re-marshaled document, so the reconcile can
byte-compare.
* s3: prime the bucket policy mirror before advanced-IAM authorization
The backfill ran from the lazy bucket-config load, but IAM authorization
evaluates the bucket-policy:<bucket> mirror before any handler runs - a
grant carried only by a not-yet-mirrored policy denied forever, and the
denied request never reached the code that would have loaded the bucket.
authorizeWithIAM now primes the bucket config first (an in-memory cache
hit once warm), and the backfill runs synchronously on the cold load so
the very first authorization already sees the mirror.
|
||
|
Chris Lu
|
96304b6870 |
S3: source config credentials from the environment, and let the chart point at an existing secret (#10868)
* s3: resolve ${VAR} in static config credentials from the environment
A deployment that keeps its S3 keys in a secret store had no way to hand
them to the gateway: -config takes a file, so the keys had to be written
into that file. Let a key in the static config name an environment
variable instead, and drop any credential whose reference stays unset so
the placeholder never becomes a usable key.
* helm: source the generated s3 identities from an existing secret
The only way to reuse credentials that already live in a Secret was to
hand-author the whole seaweedfs_s3_config JSON, since the literal keys in
values.yaml end up in git and a lookup-based keyRef renders empty under
helm template and Argo CD. Let s3.credentials.admin/read name a Secret and
its keys instead: the generated config references them as ${VAR} and the
gateway resolves them from the environment, so nothing is read from the
cluster at render time.
* s3: treat an empty environment value as an unresolved credential reference
A secret store can hand over a key that exists but is blank. Resolving it
would leave an access key whose signing secret is empty, so count it as
unresolved and drop the credential.
* helm: render the s3 secret when only the all-in-one auth flag is set
The all-in-one deployment mounts the s3 secret whenever any of the three
enableAuth flags is set, but the secret itself only rendered for the s3 and
filer flags, so allInOne.s3.enableAuth on its own left the pod waiting on a
secret nothing creates.
* helm ci: check the credential wiring on every workload that mounts it
The render check only looked at the standalone s3 deployment and only at
one of the four variables, so a helper that bound a variable to the wrong
secret key would still pass.
* helm: create the all-in-one s3 secret for every flag that mounts it
The all-in-one pod mounts the secret on any of the three enableAuth flags,
so keying its creation off allInOne.s3.enableAuth alone still left
filer.s3.enableAuth without filer.s3.enabled pointing at a secret nothing
creates. Mirror the deployment's own condition instead, and check each
flag renders both the mount and the secret.
* s3: reject a malformed credential reference instead of keying on it
A typo such as ${MY-VAR} matches no substitution, so it survived expansion
and the placeholder itself became the access key the gateway accepted.
Require every ${ in a static credential to open a well-formed reference.
|
||
|
Chris Lu
|
2d9ea0285c |
s3: add the RenameObject endpoint (#10659)
* s3: add the RenameObject endpoint
PUT /{bucket}/{key}?renameObject with x-amz-rename-source moves an object
through the filer's AtomicRenameEntry, so no bytes are read or rewritten and
the ETag, tags and SSE keys travel with the entry.
Only unversioned buckets: a versioned rename would have to rebuild the
.versions chain, and AWS offers RenameObject on directory buckets, which
cannot be versioned. The source arrives in a header, so it is authorized
separately for read and delete; both keys are locked, in key order, across the
precondition checks and the move.
* s3: let a matched source ETag precondition settle its date precondition
RFC 7232 has an ETag precondition outrank the date precondition on its own
side, and AWS documents the same for CopyObject: a matching
x-amz-copy-source-if-match with a failing x-amz-copy-source-if-unmodified-since
copies rather than returning 412. The source check evaluated all four headers in
sequence, so the date header could still veto a decided ETag match.
validateConditionalHeadersForReads already applies this precedence; the source
path now matches it.
* s3: cover a rename source named as a directory without a trailing slash
Renaming a directory would move a whole subtree, so it has to stay a missing
key whether or not the caller wrote the trailing slash.
* s3: accept a bare object key as the RenameObject source
AWS spells x-amz-rename-source both ways. Its CLI, Java and Rust examples pass
the bare source key, and only a second CLI example and the boto3 conditional
example pass bucket/key; the API reference's own example is a bare key too. The
header was read as bucket/key only, so the form AWS leads with was rejected with
InvalidArgument and the endpoint was unusable as documented.
A value is now read as a literal key first — the only reading that can never
name the wrong object — and as bucket-qualified second, when it carries the
request's own bucket and the literal key does not exist. That costs one extra
lookup only for a source that starts with the bucket's own name.
Another bucket's name in the source is no longer a distinct error: RenameObject
moves within one bucket, so it is simply part of a key this bucket does not
hold, and it reports NoSuchKey.
* s3: only a proven absence picks the other reading of a rename source
A source that resolves to a directory is not a miss to fall through on: the
literal path is still what the caller named, so answering for it beats renaming
a different object under the bucket-qualified reading. With a directory at
bucket/source.txt and an object at source.txt, a rename naming the former moved
the latter.
A failed lookup is not a proof of absence either, so a blip can no longer
redirect a rename to the other reading.
|
||
|
Chris Lu
|
b452a5e41b |
s3: honor a bucket owner recorded as an identity (#10567)
* s3: resolve a bucket owner recorded as an identity The admin UI and weed shell record a bucket's owner as an identity name in s3-identity-id and never write the account id the S3 API stores alongside it, so such a bucket looked unowned: its ACL owner fell back to the default admin account, and under the default BucketOwnerEnforced ownership every object uploaded to it was stamped with that account instead of the bucket owner. Resolve the identity to its account when no account id is recorded, in the one place both the bucket metadata and the bucket config derive the owner from. * s3: drop the recorded account when the bucket owner is reassigned Changing the owner of a bucket created through the S3 API left its old account id behind, and that outranks the identity when the owner is resolved, so the new owner never took effect for object ownership or the bucket ACL. |
||
|
Chris Lu
|
cc2775d9f2 |
s3: register an identity's inline account instead of collapsing it into admin (#10548)
* s3: register an identity's inline account instead of collapsing it into admin Credential stores persist an account inline on the identity and never emit a top-level accounts list, so every user created through the IAM API or the admin UI with an email hit the "non exist account ID" branch and was given the shared admin account. Distinct users then presented the same owner id, so ownership checks could not tell them apart and each passed for the others' buckets. Treat an id missing from the account map as undeclared rather than invalid: register it, keeping an email another account already claimed. Both load paths now resolve the account through one helper. * s3: refresh an undeclared account from the identity that carries it The merge path starts from the live account cache, so an identity upserted with the same account id but a new email or display name kept the cached copy: the new address never reached the email index and the replaced one still resolved. Changing a user's email through the admin UI takes exactly that path. An account registered from an inline block is only described by the identity carrying it, so refresh it and move its email claim. Accounts from a top-level list and the predefined defaults are marked declared and stay authoritative. * s3: let an account reclaim an email once its holder moves away Two identities can carry the same email, and the second to load leaves the lookup with the first. Returning early when the incoming metadata matches the cached account meant the loser never re-ran the claim, so an address freed by the holder's update resolved to nobody until the loser itself changed. Re-index on the unchanged path, which is a no-op while another account still holds the address. |
||
|
Chris Lu
|
1ce106e69d |
s3: audit the assumed-role principal and the STS caller (#10519)
* s3: log the requester's principal ARN in the audit entry An STS session authenticates as an opaque session subject, so requester alone gave an operator no way back to the assumed role or the session name. Record the principal ARN next to the identity name and emit it as requester_arn. * s3: record the caller identity in the STS handlers AssumeRole, GetFederationToken and GetCallerIdentity verify the caller themselves and are not wrapped by the auth middleware that records the identity, so every audit entry for minting a session had an empty requester. * s3: resolve the audit principal ARN the way policy evaluation does A JWT-authenticated identity carries no PrincipalArn — the auth layer hands the principal over in a request header — so reading the field directly left requester_arn empty for OIDC callers. buildPrincipalARN is the resolver the policy path already uses: header first, then the identity's own ARN, then a synthesized user ARN for legacy identities that have none. |
||
|
Chris Lu
|
0002e5cc7f |
s3api: load document-style policies from the advanced IAM config (#10481)
* s3api: load document-style policies from the advanced IAM config The advanced IAM file doubles as the S3 identity config when only -s3.iam.config is given. protojson drops its "document" field, so every policy landed with empty content and warned "skipping invalid policy" on each reload. Worse, if the same file also declares identities the empty content sticks in the policy map and fails the whole runtime policy sync into the IAM manager, so policies created later never reach it. * iam: skip an unparsable policy instead of failing the whole runtime sync One policy the engine cannot parse aborted SyncRuntimePolicies before it touched anything, so every other policy stayed unsynced and the engine kept serving whatever it last held. * s3api: reject a non-role RoleArn in AssumeRole as a bad request arn:aws:iam:::user/name can never resolve to a role, but the handler ran it through the trust-policy check and answered "not authorized to assume role", pointing the caller at a permission problem they do not have. * s3api: build the policy content before touching the entry Deleting "document" up front meant a marshal failure left the policy with neither field, so a later rewrite would emit it with no definition at all. * iam: pin the fail-closed handling of an unparsable policy Say in the comment that dropping it from the desired set deletes it from the engine on purpose, and cover it with a test. * s3api: widen the non-role RoleArn test to canonical ARN shapes The reported ARN omits the account id; a user ARN that carries one, and a non-principal ARN, must be rejected the same way. |
||
|
Lars Lehtonen
|
2bea4dd610 | chore(weed/s3api): prune dead code (#10462) | ||
|
Chris Lu
|
62c4333074 |
s3: list the buckets an attached IAM policy grants (#10458)
* s3: list the buckets an attached IAM policy grants ListBuckets served an identity authorized by an attached IAM policy only the buckets it had created itself. A user granted s3:ListBucket on a bucket someone else provisioned could GetObject and ListObjectsV2 against it, but the bucket never showed up in the listing any S3 client uses to build its bucket picker. The owner-index fast path is only valid for an identity whose grants name every bucket it can reach, and the routing check assumed a policy could never be enumerated. Read the names out of the policy instead: statements that allow s3:ListBucket on a concrete bucket ARN become candidates, and the per-bucket permission re-check still decides what is listed. A policy that can reach a bucket it does not name -- a wildcard resource, a policy variable, a NotResource, an STS session policy -- falls back to the full scan, which evaluates the policy per bucket. * s3: share the attached policy name lookup authorizeWithIAM and the ListBuckets enumeration both built an identity's policy names the same way, its own plus the ones from its enabled groups. Pull that into one helper so group eligibility is decided in a single place. * s3: read policy actions the way the IAM authorizer matches them The IAM authorizer matches action names case-insensitively, so a policy granting "S3:LISTBUCKET" or "S3:*" authorizes a list. The ListBuckets classifier read those actions with the local case-sensitive matcher and found no grant, so once the owner index was ready the buckets that policy allows dropped out of the listing. Match the action the looser way in the classifier: case-insensitive, and true for any pattern holding a policy variable. Over-matching only costs a candidate the per-bucket permission check then rejects, while under-matching hides a bucket the caller can read. * s3: infer a multipart grant in any case The classifier matches action patterns case-insensitively but looked the requested action up in a canonical-cased set, so "S3:UPLOADPART" missed the s3:PutObject inference that the authorizer makes. Key the set for lookup in lower case, matching how the IAM authorizer holds it. |
||
|
Chris Lu
|
6b6e6d8547 |
s3: apply filer identity changes despite a static config file (#10392)
* s3: apply filer identity changes despite a static config file A -config file with inline identities disabled the metadata-subscription reload entirely, leaving the best-effort filer->s3 push as the only way s3.configure changes could reach a running gateway. Reload on IAM events regardless: the merge keeps the file's identities protected, and a full credential-manager snapshot now also drops dynamic identities the store no longer has, so revocation works without a restart. * s3: log identity propagation failures as warnings * s3: retry failed IAM reloads and reconcile policies and groups An event-driven reload that fails now hands off to a coalescing retry loop, so a transient filer error cannot strand a revoked credential until the next IAM event. Full-state merges also drop dynamic policies the store no longer has, keeping the static file's, and treat the group snapshot as authoritative even when empty. * s3: serialize IAM configuration loads The SIGHUP file reload, subscription reloads, the retry loop, and the postgres poll run on different goroutines. Without an end-to-end lock a load holding an older store snapshot can commit after a newer one and revert it. Hold reloadMu from snapshot through commit in both load entry points; partial merges from pushed updates stay lock-free and self-heal through the next event-driven reload. * s3: keep static-file groups through full-state reconciliation Group names from the static config file are tracked like identities and policies, and a full snapshot that does not carry them keeps the current definition and its memberships instead of dropping them. * credential: include groups in postgres configuration snapshots Full-state reconciliation treats absent groups as deleted, so a snapshot that never carries them would erase every dynamic group. * s3: revoke static-file groups dropped from the config file A file reload is authoritative for the file's group set while keeping dynamic groups, mirroring how full snapshots are authoritative for dynamic groups while keeping the file's. * credential: fail filer snapshots on unreadable entries A skipped identity or policy file made the load report success with an incomplete snapshot, which reconciliation reads as deletion and the retry loop never sees. Unparseable content is still skipped: it is durable, matches boot behavior, and must not block reloads forever. * s3: ignore groups in static config files Groups are managed through the IAM API and the dynamic store; no deployment defines them in a bootstrap config file. Ignoring them with a warning removes the two-directional group merge: full snapshots are plainly authoritative and file reloads never touch groups. |
||
|
Chris Lu
|
17af32f3ff |
s3: paginate ListBuckets and serve it from a bucket owner index (#10214)
* s3: paginate ListBuckets with max-buckets, continuation-token, and prefix ListBuckets buffered every bucket entry into one slice and one XML body, which falls over with very large bucket counts. Page through the filer listing instead, cap each response at 10000 buckets like AWS, and honor max-buckets, prefix, and an opaque keyset continuation-token. * s3: maintain a bucket owner index under /buckets/.system/owners Map each bucket owner to its buckets as zero-length entries at /buckets/.system/owners/<owner>/<bucket>, with Crtime mirroring the bucket's creation time. The bucket handlers write the index synchronously, the /buckets metadata subscription reconciles changes made elsewhere (weed shell, other gateways, direct filer operations), and a startup backfill indexes pre-existing buckets before writing a ready marker. Owner names are path-escaped so no identity name can escape the index directory. * s3: serve ListBuckets from the bucket owner index Once the owner index is ready, non-admin identities list their owned buckets straight from it, merged with any buckets their legacy actions name explicitly, so ListBuckets costs O(own buckets) instead of a scan of the global /buckets directory. Admins, identities with a bare List grant or wildcard action patterns, and policy-authorized identities whose grants cannot be enumerated keep the paged scan; policy-routed identities get their owned buckets, matching AWS ListBuckets returning only the caller's buckets. * s3: keep dot-prefixed names under /buckets out of bucket surfaces Dot-prefixed entries (.system) can never be valid bucket names, so refuse to resolve them as buckets and skip them in the shell bucket listing, matching what ListBuckets and the admin UI already do. * test: cover ListBuckets pagination and the owner index end to end * s3: fail closed on a nil identity when routing ListBuckets * s3: decide the IAM authorization mechanism in one place VerifyActionPermission and the ListBuckets owner-index routing each re-derived the session-token / attached-policy / legacy-actions split; extract the decision so the two cannot drift. * s3: heal the owner index on concurrent bucket recreation too The mkdir-lost-the-race path answers BucketAlreadyOwnedByYou just like the up-front existence check, so give it the same index repair. * s3: drop owner-index records for buckets deleted during backfill A bucket removed between the backfill reading its page and writing the index record became a permanent phantom in its owner's listing: the delete's own cleanup ran before the record existed. After indexing each page, re-list the same name range and remove records whose bucket is gone; deletes landing after the re-list find the record and remove it themselves. * s3: add ContinuationToken and Prefix to the ListBuckets schema Keep AmazonS3.xsd aligned with the generated ListAllMyBucketsResult so a regeneration does not drop the pagination fields. |
||
|
Chris Lu
|
c46526822b |
s3: embedded IAM inline policy honors prefix-scoped resources (#10192)
* s3: embedded IAM inline policy honors prefix-scoped resources getActions stripped the trailing wildcard from a resource like arn:aws:s3:::bucket/prefix/*, producing a non-wildcard action (Write:bucket/prefix) that CanDo only ever matched at bucket level, so PutObject under the prefix was denied. Preserve the object path with its wildcard (Write:bucket/prefix/*) to match objects under the prefix, matching the standalone iamapi behavior. * s3: prune bucket-confined wildcard actions on bucket delete actionScopedToBucket treated any wildcard as multi-bucket, so a prefix-scoped action like Write:bucket/prefix/* survived deletion of its own bucket and could re-grant access if the bucket was recreated. Scope the wildcard check to the bucket segment only: a wildcard in the object path stays scoped to its bucket, while one in the bucket segment does not. |
||
|
Chris Lu
|
3b9e196e5f |
sts: enforce session-policy explicit deny during role chaining (#10103)
* sts: enforce session-policy explicit deny during role chaining A chained AssumeRole caller authenticates with an STS session token whose inline session policy can explicitly deny sts:AssumeRole. The deny check only evaluated the caller's named policies, so such a session could still chain into any role its trust policy admits. Validate the session token in the deny check and honor an explicit Deny in the inline session policy too. * test(sts): integration coverage for AssumeRole authorization Add an end-to-end AssumeRole authorization test (real weed mini + boto3): a non-admin caller assumes a role its trust policy admits, an explicit identity-side deny is blocked, and a session policy's explicit deny blocks role chaining. * sts: skip OIDC tokens and reject revoked sessions in the chaining deny check Review follow-ups on the session-policy deny check: - Guard session validation with !isOIDCToken so a bearer token our STS service cannot validate does not error into a false deny. - Reject a revoked session before evaluating its policy, restoring the revocation enforcement the AssumeRole path lost when it stopped routing through IsActionAllowed. |
||
|
Chris Lu
|
88a4a939aa |
fix(sts): authorize AssumeRole by the role's trust policy (#10097)
* fix(sts): authorize AssumeRole by the role's trust policy The role's trust policy already declares who may assume it, but the caller also had to pass an identity-side sts:AssumeRole check that only the Admin action could satisfy — legacy static identities have no way to express sts:AssumeRole on a role. So assuming any role required a full admin identity. Drop the redundant check and let the trust policy be the authority; scope it to specific principals to restrict who can assume. * sts: resolve caller principal ARN for the trust-policy check A legacy static identity can reach AssumeRole without a PrincipalArn set; passing the empty value would miss a trust policy that names a concrete principal. Resolve it to the canonical user ARN, sharing the logic GetCallerIdentity already used inline. * sts: enforce explicit identity-side deny for AssumeRole Authorizing a named role by its trust policy alone dropped identity-side evaluation entirely, so a caller whose attached policy explicitly denies sts:AssumeRole could still assume any role the trust policy admits. Re-check the caller's policies through the IAM manager for an explicit deny (deny-always-wins) without requiring an allow; the trust policy stays the allow authority. |
||
|
Chris Lu
|
089acfbf36 |
fix(s3api): apply static config file updates on reload (#10096)
A config-file reload (SIGHUP) routed through MergeS3ApiConfiguration, which skips identities marked static so dynamic admin/filer updates can't clobber them. That also blocked the config file itself from updating its own identities, so editing a secretKey and reloading had no effect. Thread a fromStaticFile flag from the file-load path into the merge: the authoritative file overwrites its static identities (and reapplies service accounts under them), while dynamic updates still leave them immutable. Mark the rebuilt identities static in the merge so a concurrent RemoveIdentity never observes them as removable mid-reload. |
||
|
Chris Lu
|
14d247703a |
s3: register account-less identities' synthesized account so ACL/owner ids resolve (#9971)
* s3: register account-less identities' synthesized account in the lookup #9962 gave each account-less identity a distinct account id derived from its name (instead of collapsing into admin), but never registered that account in the id->account map. GetAccountNameById then returned empty for such ids, so ACL grantee validation rejected canonical grants to the caller's own account with InvalidRequest, and bucket/object owner display was dropped as 'owner is invalid'. This broke a canned PutObjectAcl by an account-less identity (e.g. TestVersionedObjectAcl with the default 'some_admin_user' identity): ValidateAndTransferGrants -> GetAccountNameById -> 'account id is not exists' -> 400 InvalidRequest. Register the synthesized account at config load so its id resolves to a display name. Add a regression test. * s3: reuse explicitly-configured account for account-less identity Address review: if an account with the same id as an account-less identity's synthesized account is explicitly configured (custom display name/email), reuse it instead of the synthesized one. Add a test. |
||
|
Chris Lu
|
e64c821139 |
s3: give account-less identities a distinct owner instead of admin (#9962)
* s3: stop collapsing account-less identities into the admin account Identities configured without an account block all defaulted to the shared admin account, so distinct users got the same owner id and ownership checks could not tell them apart. checkAccessByOwnership also treated that id as an admin bypass, so any account-less caller passed ownership for any bucket. Give such identities a distinct account id from their name, and decide the ownership admin bypass by Admin capability rather than by the account id. isUserAdmin is now nil-safe. * s3: use the context identity in isUserAdmin before re-authenticating The Auth middleware already verifies and stores the identity in the request context. Read it there first so the ownership/admin checks don't re-run signature verification, which is redundant and fails once the request body has been consumed. * s3: nil-guard the context identity in isUserAdmin A non-nil interface wrapping a typed-nil *Identity passes the type assertion; guard against it before calling isAdmin(). * s3: trim verbose comments |
||
|
Chris Lu
|
3e8ec879c4 |
s3: keep dynamic IAM live when -iam.config is set (#9817)
* s3: keep dynamic IAM live when -iam.config is set -iam.config was treated like a static -config identity file: it set useStaticConfig, which makes the filer metadata subscription skip reloads. Identities and policies created at runtime (the IAM gRPC API) then never took effect, so advanced IAM (OIDC/STS) and dynamic IAM were mutually exclusive. Gate useStaticConfig on whether inline identities were actually loaded. An OIDC/STS-only config carries none, so it keeps the dynamic credential store live; a -config identity file still freezes its identities as before. * s3: mark static identities on config reload too A -config reload (grace.OnReload) re-reads the file, but only the startup path marked its identities static, so identities added to the file and reloaded were left unprotected from dynamic filer updates. Move the marking into loadS3ApiConfigurationFromFile and make it additive and scoped to the file's identities, so a reload protects newly added ones without freezing dynamic filer-managed identities. * s3: sync reloaded static identities into the credential manager After marking a (re)loaded config file's identities static, push the updated set into the credential manager so reloaded identities still appear in listings and survive later dynamic merges. Centralize the sync in loadS3ApiConfigurationFromFile and drop the now-redundant call in the reload hook. |
||
|
Chris Lu
|
160e68dd65 |
fix(s3api): keep ListBucket resource ARN at bucket level (#9792)
* fix(s3api): keep ListBucket resource ARN at bucket level ListObjects with ?prefix= was denied for IAM users granted s3:ListBucket on the bucket ARN. authRequestWithAuthType promotes the prefix into object so the legacy CanDo path can honor prefix-scoped Action strings, and that promoted object leaked into the policy resource ARN, producing arn:aws:s3:::bucket/<prefix> which never matches a bucket-level statement. Keep the resource bucket-level for List in the bucket-policy and IAM-attached-policy evaluators; prefix scoping stays in the s3:prefix Condition. The CanDo path is untouched. * fix(s3api): resolve List action at bucket level when prefix is promoted The IAM evaluator built a bucket-level resource ARN but still passed the prefix-promoted object to ResolveS3Action, so listing with a prefix made hasObject true and misresolved ListBucketVersions/ListBucketMultipartUploads to ListBucket. Resolve the action against the same zeroed object, and trim the resource-ARN comments. |
||
|
Chris Lu
|
8e4022d5c7 |
fix(s3api): authorize DeleteObjects per key so object-scoped policies match (#9793)
Bulk DeleteObjects carries the keys in the request body, so the route Auth middleware ran a single bucket-level check with object="", building the resource ARN as arn:aws:s3:::<bucket>. That never matches an s3:DeleteObject policy scoped to <bucket>/*, so the entire batch was denied even though the single-key DELETE worked with the same credentials. Defer authorization to the handler and check each key via AuthorizeBatchDeleteKey, mirroring AuthorizeCopySource: a synthetic DELETE /<bucket>/<key> request resolves s3:DeleteObject (or s3:DeleteObjectVersion when a versionId is given) against the object ARN. Denied keys come back as per-key errors while authorized keys still delete, matching AWS semantics. |
||
|
Chris Lu
|
16717b0bf4 |
fix(s3): authenticate JWT unsigned-streaming uploads (#9729)
A bearer-token client whose SDK appends a CRC32 trailer sends an unsigned-streaming PUT (STREAMING-UNSIGNED-PAYLOAD-TRAILER) with no SigV4 signature, so getRequestAuthType classifies it as authTypeStreamingUnsigned. The auth dispatch ignored the bearer token and fell back to anonymous, and newChunkedReader tried to verify the bearer token as a SigV4 seed signature and failed, so the body could not be decoded either. Dispatch the streaming-unsigned auth on whatever credential is present (SigV4 / JWT / anonymous), and skip the SigV4 seed-signature recompute for JWT requests in the chunked reader. |
||
|
Chris Lu
|
685571d93f |
fix(s3): allow anonymous unsigned-streaming PutObject (#9727)
Modern botocore attaches a CRC32 trailer to plain PutObject, turning the payload into STREAMING-UNSIGNED-PAYLOAD-TRAILER. An anonymous upload then carries that header but no Authorization, so it was classified as authTypeStreamingUnsigned and sent straight to SigV4 verification, which rejected it as AccessDenied while explicit credentials kept working. Fall back to the anonymous identity when an unsigned-streaming request carries no signature, mirroring the plain anonymous path. The request stays classified as unsigned-streaming so the chunked body is still decoded. |
||
|
Mmx233andChris Lu
|
9b9fdb5b76 |
fix(s3): sync IAM policies to advanced IAM Manager policy engine (#9577)
* fix(s3): sync IAM policies to advanced IAM Manager policy engine * test(s3): add unit tests for PutPolicy/DeletePolicy IAM Manager sync * fix(s3): flush loaded policies in SetIAMIntegration, drop extra reload Sync the policies already loaded from the credential store into the IAM Manager's engine from SetIAMIntegration itself, instead of re-running a full LoadS3ApiConfigurationFromCredentialManager after setup. This covers both startup orderings without a second filer round-trip or racing the async loader goroutine: if the load won, the policies are in memory to push; if SetIAMIntegration won, the load's own sync runs afterward. Move the runtime PutPolicy/DeletePolicy sync out of the iam.m write lock so the per-request auth RLock path isn't blocked by the policy recompile. * fix(s3): serialize IAM manager policy resync to avoid stale snapshots SyncRuntimePolicies replaces the manager's full policy set, so applying a policy view captured before a later mutation can resurrect a deleted policy or drop a new one. Funnel every path (PutPolicy, DeletePolicy, SetIAMIntegration, and the credential-manager load) through a single resyncIAMManagerPolicies that serializes on a dedicated mutex and reads iam.policies fresh at apply time, so the live map always wins regardless of interleaving. The load now installs the config into iam.policies before resyncing, closing the window where the manager held policies the map didn't yet have. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
285025eb73 |
s3api: support group inline policies + Condition enforcement (#9569)
* test(s3api): cover IAM inline policy aws:SourceIp + group inline gap Unit tests under weed/s3api/ drive PutUserPolicy / PutGroupPolicy → reload → VerifyActionPermission with a synthetic 127.0.0.1 request and assert that the policy's IpAddress condition flips the outcome. The user-policy cases pass on master (hydrateRuntimePolicies already routes inline docs through the policy engine, so Condition blocks are honored end- to-end). The group-policy case fails: PutGroupPolicy still returns NotImplemented, so a group inline doc never lands in the engine. Integration counterparts live under test/s3/iam/ and exercise the same paths against a live SeaweedFS S3+IAM endpoint. * s3api: support group inline policies + Condition enforcement PutGroupPolicy/GetGroupPolicy/DeleteGroupPolicy/ListGroupPolicies used to return NotImplemented in embedded IAM mode, so anything attached to a group as an inline doc — including aws:SourceIp or any other Condition — was simply unreachable. Wire the four endpoints to the credential-store methods that were already in place (memory, postgres, filer_etc all implement GroupInlinePolicyStore). On every config reload, hydrateRuntimePolicies now also walks LoadGroupInlinePolicies, registers each doc in the IAM policy engine under __inline_group_policy__/<group>/<policy>, and appends that key to Group.PolicyNames so evaluateIAMPolicies picks it up through its existing group walk. PutGroupPolicy/DeleteGroupPolicy are added to the ReloadConfiguration trigger list in DoActions. Side fix: MemoryStore.LoadConfiguration now surfaces store.groups too. Without it iam.groups never repopulated on a memory-store reload, so group policy evaluation silently no-op'd whether the policy was inline or attached. The existing tests didn't notice because no test reloaded through cm after creating a group. The NotImplemented unit test is inverted to drive the new round-trip. * s3api: drop redundant refreshIAMConfiguration from Put/DeleteGroupPolicy DoActions already triggers ReloadConfiguration for both actions via the explicit reload list, so calling refreshIAMConfiguration inline runs the load twice per request. Per PR review. * s3api: scope group-policy resource names per test; tighten deny polling - Integration test resource names get a per-test suffix so retried or parallel CI jobs don't trip EntityAlreadyExists / BucketAlreadyExists. - Deny-path Eventually loops gate on AccessDenied via a typed helper rather than any non-nil error; transient setup errors no longer end the wait prematurely. - ListGroupPolicies returns ServiceFailure when the credential manager is nil, matching Put/Get/DeleteGroupPolicy. * test(s3 iam): cover both IPv4 and IPv6 loopback in allow CIDRs CI runners with happy-eyeballs resolve `localhost` to ::1 first, in which case a 127.0.0.0/8-only allow would silently never match and the deny-driven enforcement test would hang for the allow case. Add ::1/128 to every loopback-matching policy so the allow path works regardless of which loopback family the SDK lands on. |
||
|
Chris Lu
|
d57de6dc20 |
fix(s3): keep anonymous access working with EnableIam default (fixes #9557) (#9567)
fix(s3): keep anonymous access working with EnableIam default `docker run seaweedfs` (and `weed mini` with no config) start with EnableIam=true but no IAM config file and no identities. The advanced-IAM init path was failing in 4.25 because of the missing STS signing key, which masked a latent bug: SetIAMIntegration unconditionally flipped isAuthEnabled to true, and isEnabled() also treated a non-nil iamIntegration as auth-on. Once the mini SSE-S3 KEK landed in 4.26 the STS fallback started succeeding, the integration got installed end to end, and every anonymous S3 request bounced as AccessDenied. Separate the two concerns: SetIAMIntegration just plumbs in the OIDC / embedded-IAM machinery, and a new EnableAuthEnforcement opts in to enforcement. The startup path calls it only when -s3.iam.config is actually provided, so operators with explicit IAM configs still get auth (preserves #7726). isEnabled() now reads isAuthEnabled only. |
||
|
Chris Lu
|
c61d227613 |
s3api: verify source permission on CopyObject and UploadPartCopy (#9555)
* s3api: verify source permission on CopyObject and UploadPartCopy The Auth middleware only authorized the destination because routes key on the request URL. The source from X-Amz-Copy-Source was never evaluated, so an STS session token scoped to one prefix could copy from any other prefix in the same bucket. Add AuthorizeCopySource on IdentityAccessManagement to run the full bucket-policy + IAM/identity flow against the source, using a synthetic GetObject request so action resolution lands on s3:GetObject (or s3:GetObjectVersion when a source versionId is supplied). Both CopyObjectHandler and CopyObjectPartHandler now invoke it before reading the source. * s3api: preserve presigned-URL session token on copy-source check Presigned CopyObject / UploadPartCopy requests carry the STS session token in the query string (X-Amz-Security-Token), not in a header. Rebuilding the synthetic source URL from scratch dropped that token, so the source authorization would fall through to non-STS paths and miss session policy enforcement. Forward X-Amz-Security-Token from the original query (alongside versionId), still excluding unrelated params like uploadId/partNumber that would steer ResolveS3Action away from s3:GetObject. |
||
|
Chris Lu
|
6b94701213 |
mini: quieter startup with a docker-compose-style progress board (#9524)
* mini: quieter startup with a docker-compose-style progress board Replaces noisy startup/shutdown logs with a single in-place progress table on a TTY (or one line per state change off-TTY). Each component renders as `pending -> starting -> ready` during startup and `stopping -> stopped` during shutdown, with elapsed time on transition. Also folds in a few cleanups uncovered while making this readable: - route the admin.go startup prints through glog so quietMiniLogs() filters them under mini but standalone weed admin still shows them - generate a dev SSE-S3 KEK + passphrase on first run via WEED_S3_SSE_KEK and WEED_S3_SSE_KEK_PASSPHRASE env vars (viper.Set has a nested-key conflict between s3.sse.kek and s3.sse.kek.passphrase); persisted under the data folder so restarts reuse the same key - demote worker/master gRPC Recv 'context canceled' to V(1); those are the normal shutdown signal, not Errors/Warnings - drop the 'Optimized Settings' block and the 'credentials loaded from environment variables' message from the welcome banner - only show the credentials setup hints when no S3 identities exist (new s3api.HasAnyIdentity accessor backed by an atomic.Bool) - use S3_BUCKET in the credentials hint so it pairs with AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY - reorder running-services list to master / volume / filer / webdav / s3 / iceberg / admin * mini: refuse in-memory-only SSE-S3 dev keys; surface admin serve errors loadOrCreateMiniHexSecret returns "" when os.WriteFile fails, so SSE-S3 won't encrypt data under a KEK that the next restart can't reproduce (which would orphan whatever was written this run). The caller already treats "" as "skip setting WEED_S3_SSE_* env vars", so SSE-S3 and IAM just stay disabled for this run. startAdminServer's serve goroutine used to only log ListenAndServe failures, so a bind error left the caller blocked on ctx.Done() with no listener. Forward the error through a buffered channel and select on it alongside ctx.Done(). * ci(s3-proxy-signature): match weed mini's new progress-board ready line The readiness probe grepped for "S3 (gateway|service).*(started|ready)", which matched weed mini's old "S3 service is ready at ..." line. Mini now emits " S3 ready (Xs)" from its progress board, so the old pattern misses and the test timed out at the 30-second wait. Widen the alternation to also accept "S3\s+ready". The curl HEAD fallback already covers any remaining cases. |
||
|
Yoann KatchourineandChris Lu
|
1127672d10 |
fix(s3api,iamapi): avoid full SaveConfiguration when creating a single IAM user (#9261)
* fix(s3api,iamapi): avoid full SaveConfiguration when creating a single IAM user
When CreateUser was called, both the embedded IAM path (s3api_embedded_iam.go)
and the standalone IAM path (iamapi_management_handlers.go) would:
1. Load the full config (all N users)
2. Append the new user
3. Call SaveConfiguration, which re-writes ALL N user files in the filer_etc
store, triggering N file-change events and N reload cycles.
The fix replaces the full-save path with credentialManager.CreateUser, which
writes only the single new user's file. Set changed=false to skip the redundant
SaveConfiguration call, and add "CreateUser" to the reload-after-targeted-write
block so the in-memory config is refreshed.
Also adds a nil guard around iama.iam.GetCredentialManager() in DoActions to
avoid a nil-pointer panic in legacy test fixtures that leave iam unset.
Add SetCredentialManagerForTest to IdentityAccessManagement so tests can inject
an in-memory credential store without touching production code paths.
* test(s3api,iamapi): regression tests for CreateUser targeted-write fix
Add TestCreateUserDoesNotSaveAllUsers (iamapi) and
TestEmbeddedIamCreateUserDoesNotSaveAllUsers (s3api) to guard against the
regression where CreateUser would call SaveConfiguration and re-write all
N existing user files.
Both tests:
- Pre-populate 3 existing users
- Invoke CreateUser via the HTTP API
- Assert PutS3ApiConfiguration (full-config save) was NOT called
- Assert the new user is visible in the credential store
- Assert all pre-existing users are still intact
Also update TestEmbeddedIamExecuteAction to verify persistence via the
credential manager directly (mockConfig is no longer updated on CreateUser
since we skip the SaveConfiguration path).
* refactor(s3api,iamapi): share credential-error to IAM-code mapping
Move the credentialErrToIamErrCode helper from weed/iamapi to
weed/s3api as exported CredentialErrToIamErrCode and call it from
both the standalone IAM handler (iamapi) and the embedded IAM
handler (s3api). Previously the standalone path used the helper
while the embedded path duplicated the same switch inline; the two
sites could drift out of sync.
Also extend the mapping to cover ErrUserNotFound and
ErrAccessKeyNotFound (404 NoSuchEntity) so non-CreateUser callers
that opt into the helper get the right HTTP status.
* test(s3api): seed credential store explicitly in CreateUser regression
Previously the test relied on getS3ApiConfigurationFunc's syncOnce
side effect to populate the credential store with mockConfig
identities. If that fixture ever stopped seeding, the post-test
"pre-existing users still exist" assertion would silently start
passing for the wrong reason (the users were never created).
Seed cm.SaveConfiguration directly and assert the seed precondition
before the API call so any seeding regression fails loudly.
* fix(s3api): honor skipPersist in embedded CreateUser targeted path
ExecuteAction documents that skipPersist=true means "the changed
configuration is not saved to the persistent store", but the
targeted credentialManager.CreateUser added for the avoid-bulk-save
fix ran unconditionally. Gate it on !skipPersist so no-persist
callers don't silently leak a write to the credential store. Leave
changed=true in the skipPersist branch so the tail's existing
`if changed { if !skipPersist { ... } }` block keeps suppressing
persistence the same way it does for every other action.
Add TestEmbeddedIamCreateUserSkipPersist to pin the contract: a
CreateUser invocation with skipPersist=true must not call
PutS3ApiConfiguration and must not leave the user in the
credential store.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
|
||
|
Chris Lu
|
cce98fcecf |
fix(s3): strip client-supplied X-SeaweedFS-Principal/Session-Token in AuthSignatureOnly (#9120)
* fix(s3): strip client-supplied X-SeaweedFS-Principal/Session-Token in AuthSignatureOnly AuthSignatureOnly is the only auth gate in front of S3Tables routes (incl. CreateTableBucket) and UnifiedPostHandler, but unlike authenticateRequestInternal it did not clear the internal IAM trust headers before running signature verification. S3Tables authorizeIAMAction reads X-SeaweedFS-Principal directly from the request and prefers it over the authenticated identity's PrincipalArn, so a signed low-privilege caller could append that header after signing (unsigned header, SigV4 still verifies) and have IAM policy evaluated against a spoofed principal, bypassing authorization. Clear both X-SeaweedFS-Principal and X-SeaweedFS-Session-Token at the top of AuthSignatureOnly, mirroring the existing guard in authenticateRequestInternal. Add a regression test covering the header-injection path. * refactor(s3): route AuthSignatureOnly through authenticateRequestInternal Addresses review feedback: both entry points were independently maintaining the internal-IAM-header stripping and the auth-type dispatch switch. Collapse AuthSignatureOnly into a thin wrapper around authenticateRequestInternal so the security-critical header scrub and the signature-verify switch live in one place. Post-auth behavior unique to AuthSignatureOnly (AmzAccountId header) stays inline. No functional change beyond two harmless telemetry tweaks that now match authenticateRequestInternal: the per-branch glog verbosity shifts from V(3) to V(4), and the anonymous-found path now sets AmzAuthType. * refactor(s3): centralize X-SeaweedFS-Principal/Session-Token header names Introduce SeaweedFSPrincipalHeader and SeaweedFSSessionTokenHeader in weed/s3api/s3_constants so the trust-header literals are defined once and referenced consistently by the auth scrub, JWT auth path, bucket policy principal resolution, IAM authorization, and S3Tables IAM evaluation. Replace every remaining usage in weed/s3api and weed/s3api/s3tables. This removes the drift risk the reviewer called out: adding another call site with a typo can no longer silently bypass the scrub. Pure rename, no behavior change. No-op integration-test helper in test/s3/iam/s3_iam_framework.go left untouched (separate module, and the server now strips the client-supplied value regardless). |
||
|
Chris Lu
|
40c1797f8e |
fix(s3): allow anonymous ListBuckets with prefix-scoped List action (#9073)
* fix(s3): allow anonymous ListBuckets with prefix-scoped List action An anonymous identity holding a prefix-scoped action such as "List:prefix-*" was denied at the auth middleware before ListBucketsHandler could apply the per-bucket visibility check. The middleware called CanDo with an empty bucket, which never matches a scoped action, so every anonymous ListBuckets request returned 403 even though matching buckets should have been visible. Defer ListBuckets authorization to the handler for the anonymous identity when it actually carries a List action, mirroring the existing behavior for authenticated users. Anonymous identities with no List action continue to be rejected at the global layer, preserving the secure-by-default posture. Fixes #9072 * refactor(s3): make hasListAction a method on Identity Addresses PR review — consistent with existing CanDo/isAdmin methods and also treats Admin identities as implicitly having List permission. |
||
|
Chris Lu
|
7aaa431bb4 |
s3api: prune bucket-scoped IAM actions on DeleteBucket (#9054)
* s3api: prune bucket-scoped IAM actions on DeleteBucket DeleteBucket removed the bucket directory and collection but left behind any identity actions configured via s3.configure that were scoped to that bucket (e.g. Read:bucket, Write:bucket/prefix), leaving stale auth metadata that users expected to be cleaned up along with the bucket. After a successful delete, strip actions whose resource is exactly the bucket or a prefix under it, save via the credential manager, and let the existing filer metadata subscription fan the reload out to every S3 server. Wildcarded resources and global actions are preserved since they may cover other buckets; static identities are left untouched. Fixes #5310 * s3api: address review feedback on bucket IAM prune - Apply per-identity updates via credentialManager.UpdateUser instead of a full LoadConfiguration/SaveConfiguration round-trip, so the prune no longer clobbers concurrent IAM edits made by s3.configure or the IAM API during a DeleteBucket. - Use a 30s bounded background context for the post-delete cleanup so it survives client disconnect — the bucket is already gone by then and this is best-effort bookkeeping. - Skip static identities via IsStaticIdentity, since the credential store never persists them and UpdateUser would return NotFound. |
||
|
Chris Lu
|
d1823d3784 |
fix(s3): include static identities in listing operations (#8903)
* fix(s3): include static identities in listing operations Static identities loaded from -s3.config file were only stored in the S3 API server's in-memory state. Listing operations (s3.configure shell command, aws iam list-users) queried the credential manager which only returned dynamic identities from the backend store. Register static identities with the credential manager after loading so they are included in LoadConfiguration and ListUsers results, and filtered out before SaveConfiguration to avoid persisting them to the dynamic store. Fixes https://github.com/seaweedfs/seaweedfs/discussions/8896 * fix: avoid mutating caller's config and defensive copies - SaveConfiguration: use shallow struct copy instead of mutating the caller's config.Identities field - SetStaticIdentities: skip nil entries to avoid panics - GetStaticIdentities: defensively copy PolicyNames slice to avoid aliasing the original * fix: filter nil static identities and sync on config reload - SetStaticIdentities: filter nil entries from the stored slice (not just from staticNames) to prevent panics in LoadConfiguration/ListUsers - Extract updateCredentialManagerStaticIdentities helper and call it from both startup and the grace.OnReload handler so the credential manager's static snapshot stays current after config file reloads * fix: add mutex for static identity fields and fix ListUsers for store callers - Add sync.RWMutex to protect staticIdentities/staticNames against concurrent reads during config reload - Revert CredentialManager.ListUsers to return only store users, since internal callers (e.g. DeletePolicy) look up each user in the store and fail on non-existent static entries - Merge static usernames in the filer gRPC ListUsers handler instead, via the new GetStaticUsernames method - Fix CI: TestIAMPolicyManagement/managed_policy_crud_lifecycle was failing because DeletePolicy iterated static users that don't exist in the store * fix: show static identities in admin UI and weed shell The admin UI and weed shell s3.configure command query the filer's credential manager via gRPC, which is a separate instance from the S3 server's credential manager. Static identities were only registered on the S3 server's credential manager, so they never appeared in the filer's responses. - Add CredentialManager.LoadS3ConfigFile to parse a static S3 config file and register its identities - Add FilerOptions.s3ConfigFile so the filer can load the same static config that the S3 server uses - Wire s3ConfigFile through in weed mini and weed server modes - Merge static usernames in filer gRPC ListUsers handler - Add CredentialManager.GetStaticUsernames helper - Add sync.RWMutex to protect concurrent access to static identity fields - Avoid importing weed/filer from weed/credential (which pulled in filer store init() registrations and broke test isolation) - Add docker/compose/s3_static_users_example.json * fix(admin): make static users read-only in admin UI Static users loaded from the -s3.config file should not be editable or deletable through the admin UI since they are managed via the config file. - Add IsStatic field to ObjectStoreUser, set from credential manager - Hide edit, delete, and access key buttons for static users in the users table template - Show a "static" badge next to static user names - Return 403 Forbidden from UpdateUser and DeleteUser API handlers when the target user is a static identity * fix(admin): show details for static users GetObjectStoreUserDetails called credentialManager.GetUser which only queries the dynamic store. For static users this returned ErrUserNotFound. Fall back to GetStaticIdentity when the store lookup fails. * fix(admin): load static S3 identities in admin server The admin server has its own credential manager (gRPC store) which is a separate instance from the S3 server's and filer's. It had no static identity data, so IsStaticIdentity returned false (edit/delete buttons shown) and GetStaticIdentity returned nil (details page failed). Pass the -s3.config file path through to the admin server and call LoadS3ConfigFile on its credential manager, matching the approach used for the filer. * fix: use protobuf is_static field instead of passing config file path The previous approach passed -s3.config file path to every component (filer, admin). This is wrong because the admin server should not need to know about S3 config files. Instead, add an is_static field to the Identity protobuf message. The field is set when static identities are serialized (in GetStaticIdentities and LoadS3ConfigFile). Any gRPC client that loads configuration via GetConfiguration automatically sees which identities are static, without needing the config file. - Add is_static field (tag 8) to iam_pb.Identity proto message - Set IsStatic=true in GetStaticIdentities and LoadS3ConfigFile - Admin GetObjectStoreUsers reads identity.IsStatic from proto - Admin IsStaticUser helper loads config via gRPC to check the flag - Filer GetUser gRPC handler falls back to GetStaticIdentity - Remove s3ConfigFile from AdminOptions and NewAdminServer signature |
||
|
Chris Lu
|
995dfc4d5d |
chore: remove ~50k lines of unreachable dead code (#8913)
* chore: remove unreachable dead code across the codebase Remove ~50,000 lines of unreachable code identified by static analysis. Major removals: - weed/filer/redis_lua: entire unused Redis Lua filer store implementation - weed/wdclient/net2, resource_pool: unused connection/resource pool packages - weed/plugin/worker/lifecycle: unused lifecycle plugin worker - weed/s3api: unused S3 policy templates, presigned URL IAM, streaming copy, multipart IAM, key rotation, and various SSE helper functions - weed/mq/kafka: unused partition mapping, compression, schema, and protocol functions - weed/mq/offset: unused SQL storage and migration code - weed/worker: unused registry, task, and monitoring functions - weed/query: unused SQL engine, parquet scanner, and type functions - weed/shell: unused EC proportional rebalance functions - weed/storage/erasure_coding/distribution: unused distribution analysis functions - Individual unreachable functions removed from 150+ files across admin, credential, filer, iam, kms, mount, mq, operation, pb, s3api, server, shell, storage, topology, and util packages * fix(s3): reset shared memory store in IAM test to prevent flaky failure TestLoadIAMManagerFromConfig_EmptyConfigWithFallbackKey was flaky because the MemoryStore credential backend is a singleton registered via init(). Earlier tests that create anonymous identities pollute the shared store, causing LookupAnonymous() to unexpectedly return true. Fix by calling Reset() on the memory store before the test runs. * style: run gofmt on changed files * fix: restore KMS functions used by integration tests * fix(plugin): prevent panic on send to closed worker session channel The Plugin.sendToWorker method could panic with "send on closed channel" when a worker disconnected while a message was being sent. The race was between streamSession.close() closing the outgoing channel and sendToWorker writing to it concurrently. Add a done channel to streamSession that is closed before the outgoing channel, and check it in sendToWorker's select to safely detect closed sessions without panicking. |
||
|
Chris Lu
|
6ccda3e809 |
fix(s3): allow deleting the anonymous user from admin webui (#8706)
Remove the block that prevented deleting the "anonymous" identity and stop auto-creating it when absent. If no anonymous identity exists (or it is disabled), LookupAnonymous returns not-found and both auth paths return ErrAccessDenied for anonymous requests. To enable anonymous access, explicitly create the "anonymous" user. To revoke it, delete the user like any other identity. Closes #8694 |