mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-30 00:40:44 +01:00
codex-rust-volume-server-bootstrap
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
30069f3e45 |
iam: manage OIDC providers and roles over the filer IAM gRPC service (#11523)
* s3/iam: manage roles through the IAM API, with an opt-in persistent role store
Roles could only come from the IAM config file: the S3 server pinned the
role store to memory and the embedded IAM API had no role actions, so a
role could not be created, retrusted or revoked without editing the file
and restarting every gateway.
Role store
- Read the `roleStore` key (the IAMConfig field already existed). With an
IAM config file the default stays memory; with none it is the filer, as
for OIDC providers, so zero-config clusters keep runtime-created roles.
- Roles from the IAM config file never go into a persistent role store,
which outlives the file and may be shared by S3 servers with different
files. They are served from memory beneath the store, as OIDC providers
are: a stored role of the same name takes precedence, and deleting it
restores the file's. A config-file role cannot be changed or deleted
through the API (UnmodifiableEntity), and removing one from the file
removes it at the next start. An in-memory store holds them as records,
as before. They have no creation time, so CreateDate is omitted rather
than reporting when this server started. SetRoleStore installs a store
the same way, so a store set after startup keeps the config-file roles,
as SetOIDCProviderStore does for providers.
- Watch /etc/iam/roles and drop the cached role definitions on change. The
cached filer store otherwise serves a peer's stale role for up to its 5m
TTL, which keeps a revoked trust policy in force on the other gateways.
- Role stores wrap ErrRoleNotFound for a missing role; the filer store
used to report any failed lookup as "role not found". CreateRole proceeds
only on a confirmed absence, so an unreadable store cannot let it write
over an existing role.
IAM actions
- CreateRole, GetRole, ListRoles, DeleteRole, UpdateAssumeRolePolicy,
AttachRolePolicy, DetachRolePolicy, ListAttachedRolePolicies. The reads
are allowed in read-only mode.
- A role defined in the config file is reloaded from it at every start, so
changing or deleting it through the API is refused (UnmodifiableEntity)
rather than silently reverted.
- DeleteRole with policies attached is refused (DeleteConflict), as on AWS.
- Role names follow AWS's rules ([\w+=,.@-]{1,64}); a role is stored as
<name>.json in the filer, so this also keeps a name from leaving the role
store's directory. At most 10 managed policies per role (AWS's default
quota; MaxManagedPoliciesPerUser is 10 too), LimitExceeded beyond.
- DeletePolicy is refused (DeleteConflict) while a role attaches the
policy, as it already is for users and groups: roles attach policies by
name, so a policy created later under the deleted one's name would
otherwise take effect on the role.
- Role paths other than "/" and role tags are not stored, so they are
refused rather than dropped.
Role IDs and sessions
- Roles get a unique RoleId when first stored (random, AWS AROA form),
kept across updates; a config-file role gets a stable ID derived from its
name, since it is created again at every start.
- Sessions issued through AssumeRoleWithWebIdentity, AssumeRoleWithCredentials
and AssumeRole carry the role's ID (claim "rid"), and a request under a role
whose current ID differs is denied. Resolving a session's policies by role
name let a session outlive its role: once a role was deleted, a role later
created under the same name — with a different trust policy and different
policies — revived every unexpired session of the old one with the new
role's permissions. Sessions issued before this change carry no ID and are
unaffected until they expire.
Integration test (test/s3/iam, run with `make start-services`):
TestWebIdentityWithProviderAndRoleManagedThroughIAMAPI configures an OIDC
provider, a managed policy and a role entirely through the IAM API against a
JWKS served by the test, then checks the trusted subject gets credentials
scoped to the attached policy; another subject, a token signed by another
key, an unsigned token and a token for another audience are refused; and UpdateAssumeRolePolicy moves the
trust at once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* iam: manage OIDC providers and roles over the filer IAM gRPC service
The filer's SeaweedIdentityAccessManagement service covers users, access
keys, policies and service accounts, but not the OIDC providers and roles
that STS web-identity federation needs. A controller that already manages
IAM over this service (seaweedfs-operator's S3OIDCProvider) has no
transport for them; its swadmin client returns ErrOIDCNotWired and names
this as the recommended fix.
- PutOIDCProvider / GetOIDCProvider / DeleteOIDCProvider / ListOIDCProviders
and PutRole / GetRole / DeleteRole / ListRoles.
- They write the filer-backed stores at their default paths, which S3
servers read when configured with a filer-typed "oidcProviderStore" and
"roleStore"; the S3 servers' /etc/iam subscription applies changes
without a restart.
- Put is an upsert, so a controller can reconcile to it. Deleting a
provider or role that does not exist returns NotFound, as DeleteUser does
for a user; clients treat that as already deleted. The provider's account
ID travels in the request, since the filer does not know the STS
accountId.
- PutRole applies the IAM API's rules: AWS role names, at most 10 managed
policies.
- An S3 server serves the roles and providers of its own IAM config file
ahead of the store, so a stored entry with the same name has no effect
on that server.
- PutRole keeps a replaced role's RoleId and gives a role created anew a
fresh one, so sessions of a deleted role do not carry over to a later role
of the same name.
- DeletePolicy returns FailedPrecondition while a role attaches the policy
(see the IAM API's DeleteConflict in the previous change). DeletePolicy on
this service still does not check user attachments, which predates this.
- PutOIDCProvider requires an https issuer (http only for a loopback host):
STS fetches the issuer's signing keys from it, so over plain HTTP anyone
on the network path could substitute their own.
- The OIDC provider and role RPCs refuse to run on an unauthenticated
service (FailedPrecondition until jwt.filer_signing.key is set). Users and
policies keep the service's opt-in auth, but these grant STS access
outright: otherwise anyone who can reach the port could register an issuer
they control, create a role trusting it, and exchange a token for S3
credentials. The filer's unauthenticated notice becomes a warning that says
so.
- A store that cannot be read is Unavailable, never "not found", so a Put
never writes over an entry it could not see.
- Validation is shared with the IAM API through PrepareRoleDefinition and
PrepareOIDCProviderRecord.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: bind every role session to its role, and change roles atomically
Review follow-ups.
Session binding
- The role-ID check ran only when a session carried no policy names, and
AssumeRole embeds the role's attached policies, so those sessions kept
their permissions after the role was deleted or recreated. The check
now runs for every session carrying a role ID, before policy selection.
- A named role that cannot be resolved at issuance gets no session,
instead of one with no role ID (which nothing binds).
- A config-file role's ID is derived from its name and trust policy, not
the name alone: a different role put in the file under the same name
gets a new ID, while an unchanged role keeps its sessions across restarts.
Role writes
- RoleStore gains UpdateRole, a read-modify-write that lands only if the
role is unchanged since the read, and otherwise re-reads and retries. The
filer store uses the filer's write conditions (IF_NOT_EXISTS for a new
role, IF_ENTRY_EQUAL otherwise). CreateRole, UpdateAssumeRolePolicy and
Attach/DetachRolePolicy all go through it, so two gateways no longer
overwrite each other's changes, a change racing a delete no longer
writes the role back, and of two concurrent creates one gets
EntityAlreadyExists.
- The filer store's ListRoles pages past 1,000 entries and fails on a
broken stream instead of returning what arrived, so DeletePolicy's
attachment check sees every role. ListRoles skips a role deleted between
listing and reading it.
- CreateRole validates first; a failed write is ServiceFailure, not
InvalidInput. Any Tags.* parameter is refused, not only the first key.
- ExecuteAction's skipPersist covers the S3ApiConfiguration only; the
comment now says so. Role and OIDC provider actions write their own stores.
Each fix has a test that fails without it. Against a real filer with two
gateways, concurrent AttachRolePolicy calls lost 1-4 of 8 attachments per
run before this change and none after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* iam: PutRole changes roles atomically and checks its ARN; https issuers' keys stay on https
Review follow-ups on top of the role-store changes.
- PutRole goes through RoleStore.UpdateRole, so the decision to keep an
existing role's ID or mint a new one is made against the role as it is
when written. A PutRole racing a DeleteRole can no longer write the
deleted role back with its old ID, which would revive its sessions. A
failed store read or write is Unavailable.
- PutRole refuses a role_arn that does not name the role: STS resolves a
role by the name in the ARN it is given.
- PutOIDCProvider requires an https issuer, but discovery could still name
a plain-http jwks_uri, and a key fetch could be redirected to http. For
an https issuer, a non-https jwks_uri from discovery is refused (the
issuer's own /.well-known/jwks.json is used instead), and the client
that fetches discovery and keys refuses any https-to-http redirect. An
operator-set jwksUri is left as configured.
Each has a test that fails without its guard.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: one role snapshot per decision; DeleteRole is atomic; watch a custom role store path
Review follow-ups.
- Authorization evaluates the policies of the role definition the session's
binding was checked against, instead of reading the role again: a role
replaced in between cannot lend a session its policies.
- AssumeRole and AssumeRoleWithLDAPIdentity issue the session from the
definition whose trust admits the caller (IAMManager.ResolveRoleForPrincipal),
and take its ID, duration cap and embedded policies from that same
definition. A role replaced after the caller's trust check by one that does
not trust the caller now yields AccessDenied, not a session bound to the
replacement.
- A RoleUpdate that returns nil deletes the role, on the same condition as a
write: the filer store deletes with ObjectTransaction on IF_ENTRY_EQUAL,
routed and locked like the conditional CreateEntry. DeleteRole decides
against the role it deletes, so a policy attached meanwhile on another
server is a DeleteConflict, and a delete never removes a role written
after its check.
- S3 servers watch the role store's configured basePath, not only
/etc/iam/roles, so a custom path also drops peers' cached roles on change.
Each has a test that fails without it. Live against a real filer: DeleteRole
refuses while a policy is attached and removes the entry once detached; all
test/s3/iam CI stages pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* s3/iam: state which roles DeletePolicy's attachment check can see
RolesAttachingPolicy sees the stored roles and this server's config-file
roles. A role defined only in another server's IAM config file is invisible
to it, so a config-file role that attaches a managed policy is protected
only on the servers whose file defines it. The doc comment now says so and
how to avoid it: keep such roles in every server's file, or attach only
config-file policies to config-file roles.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* iam: note that a role store set after startup is not watched for peer changes
S3 servers build their metadata watch list once, at startup, from the role
store installed then. SetRoleStore's doc now says that a filer-backed store
installed later with a different basePath is not watched, so peers' changes
to it reach this server's cached roles only when the cache expires.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* iam: DeleteRole deletes only the role it saw; issuer URLs are bare
Review follow-ups.
- The filer IAM service's DeleteRole looked the role up, then deleted by
name, so a PutRole landing in between had its new definition deleted. It
now deletes through RoleStore.UpdateRole, conditional on the entry it
read. If the role was replaced meanwhile, it returns Aborted rather than
deleting the replacement, and the caller decides again.
- PutOIDCProvider refuses an issuer URL with userinfo, a query or a
fragment. The provider's ARN comes from host and path alone, while STS
matches a token's iss claim against the stored URL exactly, so such a
provider shared the bare issuer's ARN and matched no token. A loopback
"localhost" is now matched without regard to case.
Both have tests that fail without them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* iam: write OIDC providers atomically over the filer IAM gRPC service
PutOIDCProvider read the record, then stored unconditionally; a racing
DeleteOIDCProvider left the put's stale read merged into the rewritten
record. DeleteOIDCProvider read, then deleted unconditionally; a racing
PutOIDCProvider's newer record could be removed instead. These are the
races the role RPCs closed with UpdateRole.
OIDCProviderStore gains UpdateProvider with the same contract: memory
under its lock, filer as a conditional write (IF_ENTRY_EQUAL /
IF_NOT_EXISTS) or conditional delete retrying a changed entry.
PutOIDCProvider merges the fields the request cannot carry against the
record as it is written; DeleteOIDCProvider aborts rather than delete a
record replaced meanwhile.
isRoleWriteConflict is renamed isEntryWriteConflict — the conditional-
write check is shared by both stores now.
* iam: guard PutRole against a nil credential manager, fix its doc comment
PutRole read attached policies through s.credentialManager without the
nil check its sibling handlers make, so a server built without one
panicked on a PutRole naming a policy. It now fails the call as
FailedPrecondition like the others.
The doc comment also had the store/static precedence backwards: a stored
role shadows a same-named config-file role (as the overlay serves it),
not the other way around.
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
|
||
|
Chris Lu
|
f40687b34e |
s3: tighten STS session token handling (#11383)
* s3api: test that a session token must not reveal its credential * sts: derive secret access key with HMAC keyed on the signing key * s3api: stop accepting STS session tokens as bearer credentials * security: reject STS session tokens on filer and admin gRPC auth * test: sign s3/iam framework requests with the session credential * s3api: exercise the real auth pipeline in the end-to-end harness |
||
|
Chris Lu
|
62821964dd |
filer/iam-grpc: make admin Bearer auth opt-in (fixes #9509) (#9514)
PR #9442 made the filer refuse to register the IAM gRPC service unless jwt.filer_signing.key was set in security.toml, which broke the admin UI Users/Groups/Policies pages for every deployment that ships without a security.toml — weed mini, plain Helm, vanilla weed filer. The Users tab returns Unimplemented and the page is unusable. Issues #9504, #9505 and #9509 all trace to this gap. The rest of the filer's gRPC surface is unauthenticated by default; treat IAM the same way. The service now always registers, and the auth gate is a no-op when no signing key is configured. When the key is set, every RPC still requires an admin-signed Bearer token, matching the post-#9442 behaviour. Operators who expose the filer gRPC port beyond a trusted network should set the key on both filer and admin. The admin client (IamGrpcStore.withIamClient) already skips attaching the authorization metadata when its key is empty, so no changes there. |
||
|
Chris Lu
|
5e8f99f40a |
filer: require admin-signed JWT on the IAM gRPC service (#9442)
Every IAM RPC (CreateUser, PutPolicy, CreateAccessKey, ...) now requires a Bearer token in the authorization metadata, signed with the filer write-signing key. The service refuses to register on a filer that has no jwt.filer_signing.key set, so the unauthenticated default is gone: operators who use these RPCs must configure the key and attach a token on every call. Bearer scheme matching is case-insensitive (RFC 6750), every handler nil-checks req before dereferencing it, and tests now cover the expired-token path. |
||
|
Chris Lu
|
d1823d3784 |
fix(s3): include static identities in listing operations (#8903)
* fix(s3): include static identities in listing operations Static identities loaded from -s3.config file were only stored in the S3 API server's in-memory state. Listing operations (s3.configure shell command, aws iam list-users) queried the credential manager which only returned dynamic identities from the backend store. Register static identities with the credential manager after loading so they are included in LoadConfiguration and ListUsers results, and filtered out before SaveConfiguration to avoid persisting them to the dynamic store. Fixes https://github.com/seaweedfs/seaweedfs/discussions/8896 * fix: avoid mutating caller's config and defensive copies - SaveConfiguration: use shallow struct copy instead of mutating the caller's config.Identities field - SetStaticIdentities: skip nil entries to avoid panics - GetStaticIdentities: defensively copy PolicyNames slice to avoid aliasing the original * fix: filter nil static identities and sync on config reload - SetStaticIdentities: filter nil entries from the stored slice (not just from staticNames) to prevent panics in LoadConfiguration/ListUsers - Extract updateCredentialManagerStaticIdentities helper and call it from both startup and the grace.OnReload handler so the credential manager's static snapshot stays current after config file reloads * fix: add mutex for static identity fields and fix ListUsers for store callers - Add sync.RWMutex to protect staticIdentities/staticNames against concurrent reads during config reload - Revert CredentialManager.ListUsers to return only store users, since internal callers (e.g. DeletePolicy) look up each user in the store and fail on non-existent static entries - Merge static usernames in the filer gRPC ListUsers handler instead, via the new GetStaticUsernames method - Fix CI: TestIAMPolicyManagement/managed_policy_crud_lifecycle was failing because DeletePolicy iterated static users that don't exist in the store * fix: show static identities in admin UI and weed shell The admin UI and weed shell s3.configure command query the filer's credential manager via gRPC, which is a separate instance from the S3 server's credential manager. Static identities were only registered on the S3 server's credential manager, so they never appeared in the filer's responses. - Add CredentialManager.LoadS3ConfigFile to parse a static S3 config file and register its identities - Add FilerOptions.s3ConfigFile so the filer can load the same static config that the S3 server uses - Wire s3ConfigFile through in weed mini and weed server modes - Merge static usernames in filer gRPC ListUsers handler - Add CredentialManager.GetStaticUsernames helper - Add sync.RWMutex to protect concurrent access to static identity fields - Avoid importing weed/filer from weed/credential (which pulled in filer store init() registrations and broke test isolation) - Add docker/compose/s3_static_users_example.json * fix(admin): make static users read-only in admin UI Static users loaded from the -s3.config file should not be editable or deletable through the admin UI since they are managed via the config file. - Add IsStatic field to ObjectStoreUser, set from credential manager - Hide edit, delete, and access key buttons for static users in the users table template - Show a "static" badge next to static user names - Return 403 Forbidden from UpdateUser and DeleteUser API handlers when the target user is a static identity * fix(admin): show details for static users GetObjectStoreUserDetails called credentialManager.GetUser which only queries the dynamic store. For static users this returned ErrUserNotFound. Fall back to GetStaticIdentity when the store lookup fails. * fix(admin): load static S3 identities in admin server The admin server has its own credential manager (gRPC store) which is a separate instance from the S3 server's and filer's. It had no static identity data, so IsStaticIdentity returned false (edit/delete buttons shown) and GetStaticIdentity returned nil (details page failed). Pass the -s3.config file path through to the admin server and call LoadS3ConfigFile on its credential manager, matching the approach used for the filer. * fix: use protobuf is_static field instead of passing config file path The previous approach passed -s3.config file path to every component (filer, admin). This is wrong because the admin server should not need to know about S3 config files. Instead, add an is_static field to the Identity protobuf message. The field is set when static identities are serialized (in GetStaticIdentities and LoadS3ConfigFile). Any gRPC client that loads configuration via GetConfiguration automatically sees which identities are static, without needing the config file. - Add is_static field (tag 8) to iam_pb.Identity proto message - Set IsStatic=true in GetStaticIdentities and LoadS3ConfigFile - Admin GetObjectStoreUsers reads identity.IsStatic from proto - Admin IsStaticUser helper loads config via gRPC to check the flag - Filer GetUser gRPC handler falls back to GetStaticIdentity - Remove s3ConfigFile from AdminOptions and NewAdminServer signature |
||
|
Chris Lu
|
551a31e156 |
Implement IAM propagation to S3 servers (#8130)
* Implement IAM propagation to S3 servers - Add PropagatingCredentialStore to propagate IAM changes to S3 servers via gRPC - Add Policy management RPCs to S3 proto and S3ApiServer - Update CredentialManager to use PropagatingCredentialStore when MasterClient is available - Wire FilerServer to enable propagation * Implement parallel IAM propagation and fix S3 cluster registration - Parallelized IAM change propagation with 10s timeout. - Refined context usage in PropagatingCredentialStore. - Added S3Type support to cluster node management. - Enabled S3 servers to register with gRPC address to the master. - Ensured IAM configuration reload after policy updates via gRPC. * Optimize IAM propagation with direct in-memory cache updates * Secure IAM propagation: Use metadata to skip persistence only on propagation * pb: refactor IAM and S3 services for unidirectional IAM propagation - Move SeaweedS3IamCache service from iam.proto to s3.proto. - Remove legacy IAM management RPCs and empty SeaweedS3 service from s3.proto. - Enforce that S3 servers only use the synchronization interface. * pb: regenerate Go code for IAM and S3 services Updated generated code following the proto refactoring of IAM synchronization services. * s3api: implement read-only mode for Embedded IAM API - Add readOnly flag to EmbeddedIamApi to reject write operations via HTTP. - Enable read-only mode by default in S3ApiServer. - Handle AccessDenied error in writeIamErrorResponse. - Embed SeaweedS3IamCacheServer in S3ApiServer. * credential: refactor PropagatingCredentialStore for unidirectional IAM flow - Update to use s3_pb.SeaweedS3IamCacheClient for propagation to S3 servers. - Propagate full Identity object via PutIdentity for consistency. - Remove redundant propagation of specific user/account/policy management RPCs. - Add timeout context for propagation calls. * s3api: implement SeaweedS3IamCacheServer for unidirectional sync - Update S3ApiServer to implement the cache synchronization gRPC interface. - Methods (PutIdentity, RemoveIdentity, etc.) now perform direct in-memory cache updates. - Register SeaweedS3IamCacheServer in command/s3.go. - Remove registration for the legacy and now empty SeaweedS3 service. * s3api: update tests for read-only IAM and propagation - Added TestEmbeddedIamReadOnly to verify rejection of write operations in read-only mode. - Update test setup to pass readOnly=false to NewEmbeddedIamApi in routing tests. - Updated EmbeddedIamApiForTest helper with read-only checks matching production behavior. * s3api: add back temporary debug logs for IAM updates Log IAM updates received via: - gRPC propagation (PutIdentity, PutPolicy, etc.) - Metadata configuration reloads (LoadS3ApiConfigurationFromCredentialManager) - Core identity management (UpsertIdentity, RemoveIdentity) * IAM: finalize propagation fix with reduced logging and clarified architecture * Allow configuring IAM read-only mode for S3 server integration tests * s3api: add defensive validation to UpsertIdentity * s3api: fix log message to reference correct IAM read-only flag * test/s3/iam: ensure WaitForS3Service checks for IAM write permissions * test: enable writable IAM in Makefile for integration tests * IAM: add GetPolicy/ListPolicies RPCs to s3.proto * S3: add GetBucketPolicy and ListBucketPolicies helpers * S3: support storing generic IAM policies in IdentityAccessManagement * S3: implement IAM policy RPCs using IdentityAccessManagement * IAM: fix stale user identity on rename propagation |
||
|
Chris Lu
|
5a7c74feac |
migrate IAM policies to multi-file storage (#8114)
* Add IAM gRPC service definition - Add GetConfiguration/PutConfiguration for config management - Add CreateUser/GetUser/UpdateUser/DeleteUser/ListUsers for user management - Add CreateAccessKey/DeleteAccessKey/GetUserByAccessKey for access key management - Methods mirror existing IAM HTTP API functionality * Add IAM gRPC handlers on filer server - Implement IamGrpcServer with CredentialManager integration - Handle configuration get/put operations - Handle user CRUD operations - Handle access key create/delete operations - All methods delegate to CredentialManager for actual storage * Wire IAM gRPC service to filer server - Add CredentialManager field to FilerOption and FilerServer - Import credential store implementations in filer command - Initialize CredentialManager from credential.toml if available - Register IAM gRPC service on filer gRPC server - Enable credential management via gRPC alongside existing filer services * Regenerate IAM protobuf with gRPC service methods * fix: compilation error in DeleteUser * fix: address code review comments for IAM migration * feat: migrate policies to multi-file layout and fix identity duplicated content * refactor: remove configuration.json and migrate Service Accounts to multi-file layout * refactor: standardize Service Accounts as distinct store entities and fix Admin Server persistence * config: set ServiceAccountsDirectory to /etc/iam/service_accounts * Fix Chrome dialog auto-dismiss with Bootstrap modals - Add modal-alerts.js library with Bootstrap modal replacements - Replace all 15 confirm() calls with showConfirm/showDeleteConfirm - Auto-override window.alert() for all alert() calls - Fixes Chrome 132+ aggressively blocking native dialogs * Upgrade Bootstrap from 5.3.2 to 5.3.8 * Fix syntax error in object_store_users.templ - remove duplicate closing braces * create policy * display errors * migrate to multi-file policies * address PR feedback: use showDeleteConfirm and showErrorMessage in policies.templ, refine migration check * Update policies_templ.go * add service account to iam grpc * iam: fix potential path traversal in policy names by validating name pattern * iam: add GetServiceAccountByAccessKey to CredentialStore interface * iam: implement service account support for PostgresStore Includes full CRUD operations and efficient lookup by access key. * iam: implement GetServiceAccountByAccessKey for filer_etc, grpc, and memory stores Provides efficient lookup of service accounts by access key where possible, with linear scan fallbacks for file-based stores. * iam: remove filer_multiple support Deleted its implementation and references in imports, scaffold config, and core interface constants. Redundant with filer_etc. * clear comment * dash: robustify service account construction - Guard against nil sa.Credential when constructing responses - Fix Expiration logic to only set if > 0, avoiding Unix epoch 1970 - Ensure consistency across Get, Create, and Update handlers * credential/filer_etc: improve error propagation in configuration handlers - Return error from loadServiceAccountsFromMultiFile to callers - Ensure listEntries errors in SaveConfiguration (cleanup logic) are propagated unless they are "not found" failures. - Fixes potential silent failures during IAM configuration sync. * credential/filer_etc: add existence check to CreateServiceAccount Ensures consistency with other stores by preventing accidental overwrite of existing service accounts during creation. * credential/memory: improve store robustness and Reset logic - Enforce ID immutability in UpdateServiceAccount to prevent orphans - Update Reset() to also clear the policies map, ensuring full state cleanup for tests. * dash: improve service account robustness and policy docs - Wrap parent user lookup errors to preserve context - Strictly validate Status field in UpdateServiceAccount - Add deprecation comments to legacy policy management methods * credential/filer_etc: protect against path traversal in service accounts Implemented ID validation (alphanumeric, underscores, hyphens) and applied it to Get, Save, and Delete operations to ensure no directory traversal via saId.json filenames. * credential/postgres: improve robustness and cleanup comments - Removed brainstorming comments in GetServiceAccountByAccessKey - Added missing rows.Err() check during iteration - Properly propagate Scan and Unmarshal errors instead of swallowing them * admin: unify UI alerts and confirmations using Bootstrap modals - Updated modal-alerts.js with improved automated alert type detection - Replaced native alert() and confirm() with showAlert(), showConfirm(), and showDeleteConfirm() across various Templ components - Improved UX for delete operations by providing better context and styling - Ensured consistent error reporting across IAM and Maintenance views * admin: additional UI consistency fixes for alerts and confirmations - Replaced native alert() and confirm() with Bootstrap modals in: - EC volumes (repair flow) - Collection details (repair flow) - File browser (properties and delete) - Maintenance config schema (save and reset) - Improved delete confirmation in file browser with item context - Ensured consistent success/error/info styling for all feedbacks * make * iam: add GetServiceAccountByAccessKey RPC and update GetConfiguration * iam: implement GetServiceAccountByAccessKey on server and client * iam: centralize policy and service account validation * iam: optimize MemoryStore service account lookups with indexing * iam: fix postgres service_accounts table and optimize lookups * admin: refactor modal alerts and clean up dashboard logic * admin: fix EC shards table layout mismatch * admin: URL-encode IAM path parameters for safety * admin: implement pauseWorker logic in maintenance view * iam: add rows.Err() check to postgres ListServiceAccounts * iam: standardize ErrServiceAccountNotFound across credential stores * iam: map ErrServiceAccountNotFound to codes.NotFound in DeleteServiceAccount * iam: refine service account store logic, errors and schema * iam: add validation to GetServiceAccountByAccessKey * admin: refine modal titles and ensure URL safety * admin: address bot review comments for alerts and async usage * iam: fix syntax error by restoring missing function declaration * [FilerEtcStore] improve error handling in CreateServiceAccount Refine error handling to provide clearer messages when checking for existing service accounts. * [PostgresStore] add nil guards and validation to service account methods Ensure input parameters are not nil and required IDs are present to prevent runtime panics and ensure data integrity. * [JS] add shared IAM utility script Consolidate common IAM operations like deleteUser and deleteAccessKey into a shared utility script for better maintainability. * [View] include shared IAM utilities in layout Include iam-utils.js in the main layout to make IAM functions available across all administrative pages. * [View] refactor IAM logic and restore async in EC Shards view Remove redundant local IAM functions and ensure that delete confirmation callbacks are properly marked as async. * [View] consolidate IAM logic in Object Store Users view Remove redundant local definitions of deleteUser and deleteAccessKey, relying on the shared utilities instead. * [View] update generated templ files for UI consistency * credential/postgres: remove redundant name column from service_accounts table The id is already used as the unique identifier and was being copied to the name column. This removes the name column from the schema and updates the INSERT/UPDATE queries. * credential/filer_etc: improve logging for policy migration failures Added Errorf log if AtomicRenameEntry fails during migration to ensure visibility of common failure points. * credential: allow uppercase characters in service account ID username Updated ServiceAccountIdPattern to allow [A-Za-z0-9_-]+ for the username component, matching the actual service account creation logic which uses the parent user name directly. * Update object_store_users_templ.go * admin: fix ec_shards pagination to handle numeric page arguments Updated goToPage in cluster_ec_shards.templ to accept either an Event or a numeric page argument. This prevents errors when goToPage(1) is called directly. Corrected both the .templ source and generated Go code. * credential/filer_etc: improve service account storage robustness Added nil guard to saveServiceAccount, updated GetServiceAccount to return ErrServiceAccountNotFound for empty data, and improved deleteServiceAccount to handle response-level Filer errors. |
||
|
Chris Lu
|
6bf088cec9 |
IAM Policy Management via gRPC (#8109)
* Add IAM gRPC service definition - Add GetConfiguration/PutConfiguration for config management - Add CreateUser/GetUser/UpdateUser/DeleteUser/ListUsers for user management - Add CreateAccessKey/DeleteAccessKey/GetUserByAccessKey for access key management - Methods mirror existing IAM HTTP API functionality * Add IAM gRPC handlers on filer server - Implement IamGrpcServer with CredentialManager integration - Handle configuration get/put operations - Handle user CRUD operations - Handle access key create/delete operations - All methods delegate to CredentialManager for actual storage * Wire IAM gRPC service to filer server - Add CredentialManager field to FilerOption and FilerServer - Import credential store implementations in filer command - Initialize CredentialManager from credential.toml if available - Register IAM gRPC service on filer gRPC server - Enable credential management via gRPC alongside existing filer services * Regenerate IAM protobuf with gRPC service methods * iam_pb: add Policy Management to protobuf definitions * credential: implement PolicyManager in credential stores * filer: implement IAM Policy Management RPCs * shell: add s3.policy command * test: add integration test for s3.policy * test: fix compilation errors in policy_test * pb * fmt * test * weed shell: add -policies flag to s3.configure This allows linking/unlinking IAM policies to/from identities directly from the s3.configure command. * test: verify s3.configure policy linking and fix port allocation - Added test case for linking policies to users via s3.configure - Implemented findAvailablePortPair to ensure HTTP and gRPC ports are both available, avoiding conflicts with randomized port assignments. - Updated assertion to match jsonpb output (policyNames) * credential: add StoreTypeGrpc constant * credential: add IAM gRPC store boilerplate * credential: implement identity methods in gRPC store * credential: implement policy methods in gRPC store * admin: use gRPC credential store for AdminServer This ensures that all IAM and policy changes made through the Admin UI are persisted via the Filer's IAM gRPC service instead of direct file manipulation. * shell: s3.configure use granular IAM gRPC APIs instead of full config patching * shell: s3.configure use granular IAM gRPC APIs * shell: replace deprecated ioutil with os in s3.policy * filer: use gRPC FailedPrecondition for unconfigured credential manager * test: improve s3.policy integration tests and fix error checks * ci: add s3 policy shell integration tests to github workflow * filer: fix LoadCredentialConfiguration error handling * credential/grpc: propagate unmarshal errors in GetPolicies * filer/grpc: improve error handling and validation * shell: use gRPC status codes in s3.configure * credential: document PutPolicy as create-or-replace * credential/postgres: reuse CreatePolicy in PutPolicy to deduplicate logic * shell: add timeout context and strictly enforce flags in s3.policy * iam: standardize policy content field naming in gRPC and proto * shell: extract slice helper functions in s3.configure * filer: map credential store errors to gRPC status codes * filer: add input validation for UpdateUser and CreateAccessKey * iam: improve validation in policy and config handlers * filer: ensure IAM service registration by defaulting credential manager * credential: add GetStoreName method to manager * test: verify policy deletion in integration test |