iam: honor configured credential store for IAM API policies and propagate to S3 caches (fixes #9518) (#9522)

* iamapi: route managed policies through credential manager (fixes #9518)

CreatePolicy via the IAM API wrote straight to the filer
/etc/iam/policies.json, ignoring any non-filer credential store. When
credential.postgres was configured, policies created via the IAM API
landed only in the filer while the Admin UI wrote to postgres,
producing a split-brain where ListPolicies/GetPolicy never saw the
Admin UI's policies and vice versa.

GetPolicies/PutPolicies on IamS3ApiConfigure now load managed policies
from credentialManager and persist Create/Update/Delete as a delta
against the store. Inline user/group policies still live in the legacy
policies.json file (no credential-store API for them yet). Pre-existing
managed policies in the legacy file are merged on read so deployments
don't lose data, and re-persisted to the store on the next write so
the legacy file is drained over time.

* credential: route IAM API inline policies through credential manager

Extends the #9518 fix to user-inline and group-inline policies so the
IAM API never writes the legacy /etc/iam/policies.json bundle directly.
The previous patch only routed managed policies; this one finishes the
job for the other two policy types.

- Add GroupInlinePolicyStore + GroupInlinePoliciesLoader optional
  interfaces, mirroring the existing user-inline ones, and matching
  Put/Get/Delete/List/LoadAll wrappers on CredentialManager.
- Implement group-inline storage in memory (new map), filer_etc (new
  field on PoliciesCollection, reusing the legacy file under policyMu),
  and postgres (new group_inline_policies table with ON DELETE CASCADE
  off the groups FK).
- Wire the new methods through PropagatingCredentialStore so wrapped
  stores still delegate correctly.
- IamS3ApiConfigure.PutPolicies now applies managed + user-inline +
  group-inline as deltas through the credential manager; the legacy
  /etc/iam/policies.json file is never written when a credential
  manager is wired up. GetPolicies still reads the legacy bundle once
  as a fallback so unmigrated data is picked up and re-persisted into
  the store on the next write.

* credential: propagate SaveConfiguration writes to running S3 caches

Postgres (and any non-filer) credential stores never fired the S3 IAM
cache invalidation path on bulk identity / group updates. The
PropagatingCredentialStore had explicit Put/Remove handlers for
single-entity calls (CreateUser, PutPolicy, etc.) but inherited
SaveConfiguration unchanged from the embedded store, so the bulk path
the IAM API takes at the end of every handler was silent. Inline-policy
changes recompute identity.Actions and persist via SaveConfiguration,
so until restart the cached Actions on each S3 server stayed stale and
authorization decisions used the pre-change view.

Override SaveConfiguration to snapshot the prior user / group lists,
delegate the save, then fan out PutIdentity / PutGroup for what's in
the new config and RemoveIdentity / RemoveGroup for what got pruned.
Reuses the existing SeaweedS3IamCache RPCs, no protobuf changes.

* iamapi: drain legacy policies.json after authoritative credential-store writes

Review pointed out a resurrection bug: GetPolicies still reads
/etc/iam/policies.json as a one-way migration fallback, but PutPolicies
in the credential-manager path never wrote that file, so legacy-only
entries reappeared on the next read even after the IAM API "deleted"
them. PutPolicies now overwrites the bundle with an empty {} after a
successful credential-store write, unless the store is filer_etc
(which owns the bundle as its own inline-policy backing — clearing it
would wipe filer_etc's data). Also wraps the filer read, JSON
unmarshal, and marshal errors with context per the other review
comments.
This commit is contained in:
Chris Lu authored and GitHub committed 2026-05-17 13:15:27 -07:00
1 parent b4289abb0a
commit ff6f9fd90a
11 files changed
+1073 -24

No files matched your search

+222 -20
View File
@@ -8,6 +8,7 @@ import (
"encoding/json"
"fmt"
"net/http"
"reflect"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/credential"
@@ -202,36 +203,237 @@ func (iama *IamS3ApiConfigure) PutS3ApiConfigurationToFiler(s3cfg *iam_pb.S3ApiC
})
}
// GetPolicies returns the merged view of managed, user-inline, and group-inline
// policies from the configured credential store, plus a one-shot fallback read
// of the legacy /etc/iam/policies.json bundle. Routing reads through the
// credential manager keeps the IAM API in sync with the Admin UI; the legacy
// file fallback lets deployments that wrote policies before this change still
// see them until the next PutPolicies migrates them into the store.
func (iama *IamS3ApiConfigure) GetPolicies(policies *Policies) (err error) {
var buf bytes.Buffer
err = pb.WithOneOfGrpcFilerClients(false, iama.option.Filers, iama.option.GrpcDialOption, func(client filer_pb.SeaweedFilerClient) error {
if err = filer.ReadEntry(iama.masterClient, client, filer.IamConfigDirectory, filer.IamPoliciesFile, &buf); err != nil {
return err
}
return nil
})
if err != nil && err != filer_pb.ErrNotFound {
return err
}
if err == filer_pb.ErrNotFound || buf.Len() == 0 {
ctx := context.Background()
if policies.Policies == nil {
policies.Policies = make(map[string]policy_engine.PolicyDocument)
}
if iama.credentialManager != nil {
managed, mErr := iama.credentialManager.GetPolicies(ctx)
if mErr != nil {
return fmt.Errorf("load managed policies from credential store: %w", mErr)
}
for name, doc := range managed {
policies.Policies[name] = doc
}
inline, iErr := iama.credentialManager.LoadAllInlinePolicies(ctx)
if iErr != nil {
return fmt.Errorf("load user inline policies from credential store: %w", iErr)
}
mergeInline(&policies.InlinePolicies, inline)
groupInline, gErr := iama.credentialManager.LoadAllGroupInlinePolicies(ctx)
if gErr != nil {
return fmt.Errorf("load group inline policies from credential store: %w", gErr)
}
mergeInline(&policies.GroupInlinePolicies, groupInline)
}
// Legacy /etc/iam/policies.json fallback. Credential store wins on
// conflict (per name / per (user, name) / per (group, name)) so we never
// produce duplicates.
var buf bytes.Buffer
rErr := pb.WithOneOfGrpcFilerClients(false, iama.option.Filers, iama.option.GrpcDialOption, func(client filer_pb.SeaweedFilerClient) error {
return filer.ReadEntry(iama.masterClient, client, filer.IamConfigDirectory, filer.IamPoliciesFile, &buf)
})
if rErr != nil && rErr != filer_pb.ErrNotFound {
return fmt.Errorf("read legacy policies from filer: %w", rErr)
}
if rErr == filer_pb.ErrNotFound || buf.Len() == 0 {
return nil
}
if err := json.Unmarshal(buf.Bytes(), policies); err != nil {
return err
var legacy Policies
if err := json.Unmarshal(buf.Bytes(), &legacy); err != nil {
return fmt.Errorf("unmarshal legacy policies: %w", err)
}
for name, doc := range legacy.Policies {
if _, exists := policies.Policies[name]; !exists {
policies.Policies[name] = doc
}
}
mergeInlineFallback(&policies.InlinePolicies, legacy.InlinePolicies)
mergeInlineFallback(&policies.GroupInlinePolicies, legacy.GroupInlinePolicies)
return nil
}
// PutPolicies persists managed, user-inline, and group-inline policies through
// the credential store as deltas relative to its current contents. When the
// credential store is something other than filer_etc (postgres, grpc, memory),
// the legacy /etc/iam/policies.json bundle is cleared after a successful write
// so that data merged in by the GetPolicies fallback during migration cannot
// resurface — otherwise a later DeletePolicy would be undone by the next read.
// For filer_etc the bundle is left alone because filer_etc owns it as its own
// inline-policy backing store (see filer_etc.saveLegacyPoliciesCollection).
// With no credential manager available (a test path), everything is written
// back to the bundle, preserving the pre-refactor behavior.
func (iama *IamS3ApiConfigure) PutPolicies(policies *Policies) (err error) {
var b []byte
if b, err = json.Marshal(policies); err != nil {
return err
ctx := context.Background()
if iama.credentialManager == nil {
legacy := Policies{
Policies: policies.Policies,
InlinePolicies: policies.InlinePolicies,
GroupInlinePolicies: policies.GroupInlinePolicies,
}
return iama.writeLegacyPolicies(ctx, &legacy)
}
// Managed delta.
currentManaged, gErr := iama.credentialManager.GetPolicies(ctx)
if gErr != nil {
return fmt.Errorf("read managed policies from credential store: %w", gErr)
}
for name, doc := range policies.Policies {
existing, found := currentManaged[name]
if !found {
if cErr := iama.credentialManager.CreatePolicy(ctx, name, doc); cErr != nil {
return fmt.Errorf("create policy %s in credential store: %w", name, cErr)
}
continue
}
if !reflect.DeepEqual(existing, doc) {
if uErr := iama.credentialManager.UpdatePolicy(ctx, name, doc); uErr != nil {
return fmt.Errorf("update policy %s in credential store: %w", name, uErr)
}
}
}
for name := range currentManaged {
if _, keep := policies.Policies[name]; !keep {
if dErr := iama.credentialManager.DeletePolicy(ctx, name); dErr != nil {
return fmt.Errorf("delete policy %s from credential store: %w", name, dErr)
}
}
}
// User-inline delta.
currentInline, iErr := iama.credentialManager.LoadAllInlinePolicies(ctx)
if iErr != nil {
return fmt.Errorf("read user inline policies from credential store: %w", iErr)
}
for user, desired := range policies.InlinePolicies {
existing := currentInline[user]
for name, doc := range desired {
if prior, found := existing[name]; found && reflect.DeepEqual(prior, doc) {
continue
}
if pErr := iama.credentialManager.PutUserInlinePolicy(ctx, user, name, doc); pErr != nil {
return fmt.Errorf("put inline policy %s/%s in credential store: %w", user, name, pErr)
}
}
}
for user, existing := range currentInline {
desired := policies.InlinePolicies[user]
for name := range existing {
if _, keep := desired[name]; keep {
continue
}
if dErr := iama.credentialManager.DeleteUserInlinePolicy(ctx, user, name); dErr != nil {
return fmt.Errorf("delete inline policy %s/%s from credential store: %w", user, name, dErr)
}
}
}
// Group-inline delta.
currentGroupInline, ggErr := iama.credentialManager.LoadAllGroupInlinePolicies(ctx)
if ggErr != nil {
return fmt.Errorf("read group inline policies from credential store: %w", ggErr)
}
for group, desired := range policies.GroupInlinePolicies {
existing := currentGroupInline[group]
for name, doc := range desired {
if prior, found := existing[name]; found && reflect.DeepEqual(prior, doc) {
continue
}
if pErr := iama.credentialManager.PutGroupInlinePolicy(ctx, group, name, doc); pErr != nil {
return fmt.Errorf("put group inline policy %s/%s in credential store: %w", group, name, pErr)
}
}
}
for group, existing := range currentGroupInline {
desired := policies.GroupInlinePolicies[group]
for name := range existing {
if _, keep := desired[name]; keep {
continue
}
if dErr := iama.credentialManager.DeleteGroupInlinePolicy(ctx, group, name); dErr != nil {
return fmt.Errorf("delete group inline policy %s/%s from credential store: %w", group, name, dErr)
}
}
}
// filer_etc owns the legacy bundle itself (it's how filer_etc persists
// inline policies). For every other store the bundle is at most a one-
// shot migration source — empty it so deleted legacy-only entries don't
// reappear via the GetPolicies fallback on the next read.
if iama.credentialManager.GetStoreName() == string(credential.StoreTypeFilerEtc) {
return nil
}
return iama.writeLegacyPolicies(ctx, &Policies{})
}
// writeLegacyPolicies serializes the given Policies struct to the legacy
// /etc/iam/policies.json bundle. Called both for the no-credential-manager
// fallback (full bundle) and for the credential-manager path (empty bundle,
// to drain legacy data after migration).
func (iama *IamS3ApiConfigure) writeLegacyPolicies(ctx context.Context, p *Policies) error {
b, err := json.Marshal(p)
if err != nil {
return fmt.Errorf("marshal legacy policies: %w", err)
}
return pb.WithOneOfGrpcFilerClients(false, iama.option.Filers, iama.option.GrpcDialOption, func(client filer_pb.SeaweedFilerClient) error {
if err := filer.SaveInsideFiler(context.Background(), client, filer.IamConfigDirectory, filer.IamPoliciesFile, b); err != nil {
return err
}
return nil
return filer.SaveInsideFiler(ctx, client, filer.IamConfigDirectory, filer.IamPoliciesFile, b)
})
}
// mergeInline overwrites entries in dst with src (credential-store data taking
// precedence over anything already in dst).
func mergeInline(dst *map[string]map[string]policy_engine.PolicyDocument, src map[string]map[string]policy_engine.PolicyDocument) {
if len(src) == 0 {
return
}
if *dst == nil {
*dst = make(map[string]map[string]policy_engine.PolicyDocument, len(src))
}
for outer, byName := range src {
bucket := (*dst)[outer]
if bucket == nil {
bucket = make(map[string]policy_engine.PolicyDocument, len(byName))
(*dst)[outer] = bucket
}
for name, doc := range byName {
bucket[name] = doc
}
}
}
// mergeInlineFallback fills only entries that are missing in dst — the legacy
// file never overrides whatever the credential store returned.
func mergeInlineFallback(dst *map[string]map[string]policy_engine.PolicyDocument, src map[string]map[string]policy_engine.PolicyDocument) {
if len(src) == 0 {
return
}
if *dst == nil {
*dst = make(map[string]map[string]policy_engine.PolicyDocument, len(src))
}
for outer, byName := range src {
bucket := (*dst)[outer]
if bucket == nil {
bucket = make(map[string]policy_engine.PolicyDocument, len(byName))
(*dst)[outer] = bucket
}
for name, doc := range byName {
if _, exists := bucket[name]; !exists {
bucket[name] = doc
}
}
}
}
+253
View File
@@ -0,0 +1,253 @@
package iamapi
import (
"context"
"testing"
"github.com/seaweedfs/seaweedfs/weed/credential"
"github.com/seaweedfs/seaweedfs/weed/credential/memory"
"github.com/seaweedfs/seaweedfs/weed/s3api/policy_engine"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// newIamS3ApiConfigureForTest builds an IamS3ApiConfigure backed by a memory
// credential store and no filer. The filer-less option is fine because the
// new code only touches the filer for inline/group policies, which these
// tests don't exercise.
func newIamS3ApiConfigureForTest(t *testing.T) (*IamS3ApiConfigure, *credential.CredentialManager) {
t.Helper()
store := &memory.MemoryStore{}
require.NoError(t, store.Initialize(nil, ""))
cm := &credential.CredentialManager{Store: store}
cfg := &IamS3ApiConfigure{
option: &IamServerOption{},
credentialManager: cm,
}
return cfg, cm
}
func samplePolicyDocument(resource string) policy_engine.PolicyDocument {
return policy_engine.PolicyDocument{
Version: "2012-10-17",
Statement: []policy_engine.PolicyStatement{
{
Effect: policy_engine.PolicyEffectAllow,
Action: policy_engine.NewStringOrStringSlice("s3:Get*"),
Resource: policy_engine.NewStringOrStringSlicePtr(resource),
},
},
}
}
// TestPutPoliciesCreatesInCredentialStore is the regression test for
// https://github.com/seaweedfs/seaweedfs/issues/9518: the IAM API used to
// write managed policies straight to the filer, bypassing any non-filer
// credential store (e.g. postgres). After the fix, a new policy passed to
// PutPolicies must show up in the credential store.
func TestPutPoliciesCreatesInCredentialStore(t *testing.T) {
cfg, cm := newIamS3ApiConfigureForTest(t)
ctx := context.Background()
doc := samplePolicyDocument("arn:aws:s3:::bucket-a")
require.NoError(t, cfg.PutPolicies(&Policies{
Policies: map[string]policy_engine.PolicyDocument{"policy-a": doc},
}))
stored, err := cm.GetPolicies(ctx)
require.NoError(t, err)
require.Contains(t, stored, "policy-a")
assert.Equal(t, doc, stored["policy-a"])
}
// TestGetPoliciesReadsFromCredentialStore mirrors the read side of the bug:
// ListPolicies / GetPolicy via the IAM API used to read only the filer file
// and therefore missed policies that the Admin UI had written to the store.
func TestGetPoliciesReadsFromCredentialStore(t *testing.T) {
cfg, cm := newIamS3ApiConfigureForTest(t)
ctx := context.Background()
doc := samplePolicyDocument("arn:aws:s3:::bucket-b")
require.NoError(t, cm.CreatePolicy(ctx, "policy-b", doc))
loaded := Policies{}
require.NoError(t, cfg.GetPolicies(&loaded))
require.Contains(t, loaded.Policies, "policy-b")
assert.Equal(t, doc, loaded.Policies["policy-b"])
}
// TestPutPoliciesDeletesRemovedFromCredentialStore makes sure the delta path
// translates an in-memory deletion (the pattern the DeletePolicy handler
// uses) into a credential-store DeletePolicy call.
func TestPutPoliciesDeletesRemovedFromCredentialStore(t *testing.T) {
cfg, cm := newIamS3ApiConfigureForTest(t)
ctx := context.Background()
keep := samplePolicyDocument("arn:aws:s3:::keep")
drop := samplePolicyDocument("arn:aws:s3:::drop")
require.NoError(t, cm.CreatePolicy(ctx, "keep", keep))
require.NoError(t, cm.CreatePolicy(ctx, "drop", drop))
// Caller fetched policies, removed "drop", and asked us to persist.
require.NoError(t, cfg.PutPolicies(&Policies{
Policies: map[string]policy_engine.PolicyDocument{"keep": keep},
}))
remaining, err := cm.GetPolicies(ctx)
require.NoError(t, err)
assert.Contains(t, remaining, "keep")
assert.NotContains(t, remaining, "drop")
}
// TestPutPoliciesUpdatesChangedInCredentialStore confirms the delta also
// detects document changes for the same policy name.
func TestPutPoliciesUpdatesChangedInCredentialStore(t *testing.T) {
cfg, cm := newIamS3ApiConfigureForTest(t)
ctx := context.Background()
original := samplePolicyDocument("arn:aws:s3:::v1")
require.NoError(t, cm.CreatePolicy(ctx, "p", original))
updated := samplePolicyDocument("arn:aws:s3:::v2")
require.NoError(t, cfg.PutPolicies(&Policies{
Policies: map[string]policy_engine.PolicyDocument{"p": updated},
}))
stored, err := cm.GetPolicy(ctx, "p")
require.NoError(t, err)
require.NotNil(t, stored)
assert.Equal(t, updated, *stored)
}
// TestPutPoliciesRoutesUserInlineThroughCredentialStore is the second half of
// the #9518 fix: per-user inline policies created via the IAM API must land
// in the credential store (so postgres users have a single source of truth)
// rather than in the filer bundle.
func TestPutPoliciesRoutesUserInlineThroughCredentialStore(t *testing.T) {
cfg, cm := newIamS3ApiConfigureForTest(t)
ctx := context.Background()
doc := samplePolicyDocument("arn:aws:s3:::user-bucket")
require.NoError(t, cfg.PutPolicies(&Policies{
InlinePolicies: map[string]map[string]policy_engine.PolicyDocument{
"alice": {"inline-1": doc},
},
}))
stored, err := cm.GetUserInlinePolicy(ctx, "alice", "inline-1")
require.NoError(t, err)
require.NotNil(t, stored)
assert.Equal(t, doc, *stored)
}
// TestPutPoliciesDeletesRemovedUserInline mirrors the deletion side. The
// DeleteUserPolicy handler removes the entry from the in-memory map and calls
// PutPolicies; the delta must call DeleteUserInlinePolicy.
func TestPutPoliciesDeletesRemovedUserInline(t *testing.T) {
cfg, cm := newIamS3ApiConfigureForTest(t)
ctx := context.Background()
keep := samplePolicyDocument("arn:aws:s3:::keep")
drop := samplePolicyDocument("arn:aws:s3:::drop")
require.NoError(t, cm.PutUserInlinePolicy(ctx, "bob", "keep", keep))
require.NoError(t, cm.PutUserInlinePolicy(ctx, "bob", "drop", drop))
require.NoError(t, cfg.PutPolicies(&Policies{
InlinePolicies: map[string]map[string]policy_engine.PolicyDocument{
"bob": {"keep": keep},
},
}))
remaining, err := cm.ListUserInlinePolicies(ctx, "bob")
require.NoError(t, err)
assert.ElementsMatch(t, []string{"keep"}, remaining)
}
// TestPutPoliciesRoutesGroupInlineThroughCredentialStore confirms group-attached
// inline policies also persist via the credential manager rather than the
// shared filer bundle.
func TestPutPoliciesRoutesGroupInlineThroughCredentialStore(t *testing.T) {
cfg, cm := newIamS3ApiConfigureForTest(t)
ctx := context.Background()
doc := samplePolicyDocument("arn:aws:s3:::group-bucket")
require.NoError(t, cfg.PutPolicies(&Policies{
GroupInlinePolicies: map[string]map[string]policy_engine.PolicyDocument{
"devs": {"inline-g": doc},
},
}))
stored, err := cm.GetGroupInlinePolicy(ctx, "devs", "inline-g")
require.NoError(t, err)
require.NotNil(t, stored)
assert.Equal(t, doc, *stored)
}
// TestPutPoliciesDeletesRemovedGroupInline covers the deletion delta for
// group-attached inline policies.
func TestPutPoliciesDeletesRemovedGroupInline(t *testing.T) {
cfg, cm := newIamS3ApiConfigureForTest(t)
ctx := context.Background()
keep := samplePolicyDocument("arn:aws:s3:::keep")
drop := samplePolicyDocument("arn:aws:s3:::drop")
require.NoError(t, cm.PutGroupInlinePolicy(ctx, "devs", "keep", keep))
require.NoError(t, cm.PutGroupInlinePolicy(ctx, "devs", "drop", drop))
require.NoError(t, cfg.PutPolicies(&Policies{
GroupInlinePolicies: map[string]map[string]policy_engine.PolicyDocument{
"devs": {"keep": keep},
},
}))
remaining, err := cm.ListGroupInlinePolicies(ctx, "devs")
require.NoError(t, err)
assert.ElementsMatch(t, []string{"keep"}, remaining)
}
// TestPutPoliciesIsAuthoritativeWhenStoreIsNotFilerEtc is the regression test
// for the legacy-file resurrection bug: once the credential store is
// authoritative, a delete must stick across the GetPolicies fallback. We
// can't verify the legacy-file rewrite directly without a fake filer in
// the test harness, but we can verify the cm-side delta behavior: a policy
// that was deleted does not come back via cm on a subsequent fetch.
func TestPutPoliciesIsAuthoritativeWhenStoreIsNotFilerEtc(t *testing.T) {
cfg, cm := newIamS3ApiConfigureForTest(t)
ctx := context.Background()
require.Equal(t, "memory", cm.GetStoreName(), "test depends on memory store name carveout")
doc := samplePolicyDocument("arn:aws:s3:::vanishing")
require.NoError(t, cfg.PutPolicies(&Policies{
Policies: map[string]policy_engine.PolicyDocument{"will-vanish": doc},
}))
require.NoError(t, cfg.PutPolicies(&Policies{
Policies: map[string]policy_engine.PolicyDocument{},
}))
remaining, err := cm.GetPolicies(ctx)
require.NoError(t, err)
assert.NotContains(t, remaining, "will-vanish")
}
// TestGetPoliciesReadsInlineAndGroupInlineFromCredentialStore confirms reads
// pull user-inline and group-inline policies from the store too, so handlers
// like ListUserPolicies / GetUserPolicy see Admin-UI writes.
func TestGetPoliciesReadsInlineAndGroupInlineFromCredentialStore(t *testing.T) {
cfg, cm := newIamS3ApiConfigureForTest(t)
ctx := context.Background()
userDoc := samplePolicyDocument("arn:aws:s3:::user-data")
groupDoc := samplePolicyDocument("arn:aws:s3:::group-data")
require.NoError(t, cm.PutUserInlinePolicy(ctx, "carol", "u-policy", userDoc))
require.NoError(t, cm.PutGroupInlinePolicy(ctx, "admins", "g-policy", groupDoc))
loaded := Policies{}
require.NoError(t, cfg.GetPolicies(&loaded))
require.Contains(t, loaded.InlinePolicies, "carol")
assert.Equal(t, userDoc, loaded.InlinePolicies["carol"]["u-policy"])
require.Contains(t, loaded.GroupInlinePolicies, "admins")
assert.Equal(t, groupDoc, loaded.GroupInlinePolicies["admins"]["g-policy"])
}