Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-10-31 17:30:42 +01:00
Code Issues Packages Projects Releases Wiki Activity

feat: Implement distributed IAM role storage for multi-instance deployments

Browse Source 
PROBLEM SOLVED:
- Roles were stored in memory per-instance, causing inconsistencies
- Sessions and policies had filer storage but roles didn't
- Multi-instance deployments had authentication failures

IMPLEMENTATION:
- Add RoleStore interface for pluggable role storage backends
- Implement FilerRoleStore using SeaweedFS filer as distributed backend
- Update IAMManager to use RoleStore instead of in-memory map
- Add role store configuration to IAM config schema
- Support both memory and filer storage for roles

NEW COMPONENTS:
- weed/iam/integration/role_store.go - Role storage interface & implementations
- weed/iam/integration/role_store_test.go - Unit tests for role storage
- test/s3/iam/iam_config_distributed.json - Sample distributed config
- test/s3/iam/DISTRIBUTED.md - Complete deployment guide

CONFIGURATION:
{
  'roleStore': {
    'storeType': 'filer',
    'storeConfig': {
      'filerAddress': 'localhost:8888',
      'basePath': '/seaweedfs/iam/roles'
    }
  }
}

BENEFITS:
- ✅ Consistent role definitions across all S3 gateway instances
- ✅ Persistent role storage survives instance restarts
- ✅ Scales to unlimited number of gateway instances
- ✅ No session affinity required in load balancers
- ✅ Production-ready distributed IAM system

This completes the distributed IAM implementation, making SeaweedFS
S3 Gateway truly scalable for production multi-instance deployments.
This commit is contained in:
chrislu committed 2025-08-24 12:33:00 -07:00
1 parent 4c324fca15
commit ce17743275
5 files changed
+973 -15
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

test/s3/iam/DISTRIBUTED.md
+288
View File
@@ -0,0 +1,288 @@
# Distributed IAM for SeaweedFS S3 Gateway
This document explains how to configure SeaweedFS S3 Gateway for distributed environments with multiple gateway instances.
## Problem Statement
In distributed environments with multiple S3 gateway instances, the default in-memory storage for IAM components causes **serious consistency issues**:
- **Roles**: Each instance maintains separate in-memory role definitions
- **Sessions**: STS sessions created on one instance aren't visible on others
- **Policies**: Policy updates don't propagate across instances
- **Authentication failures**: Users may get different responses from different instances
## Solution: Filer-Based Distributed Storage
SeaweedFS now supports **distributed IAM storage** using the filer as a centralized backend for:
- ✅ **Role definitions** (FilerRoleStore)
- ✅ **STS sessions** (FilerSessionStore)
- ✅ **IAM policies** (FilerPolicyStore)
All S3 gateway instances share the same IAM state through the filer.
## Configuration
### Standard Configuration (Single Instance)
```json
{
"sts": {
"tokenDuration": 3600000000000,
"maxSessionLength": 43200000000000,
"issuer": "seaweedfs-sts",
"signingKey": "base64-encoded-signing-key"
},
"policy": {
"defaultEffect": "Deny",
"storeType": "memory"
}
}
```
### Distributed Configuration (Multiple Instances)
```json
{
"sts": {
"tokenDuration": 3600000000000,
"maxSessionLength": 43200000000000,
"issuer": "seaweedfs-sts",
"signingKey": "base64-encoded-signing-key",
"sessionStoreType": "filer",
"sessionStoreConfig": {
"filerAddress": "localhost:8888",
"basePath": "/seaweedfs/iam/sessions"
}
},
"policy": {
"defaultEffect": "Deny",
"storeType": "filer",
"storeConfig": {
"filerAddress": "localhost:8888",
"basePath": "/seaweedfs/iam/policies"
}
},
"roleStore": {
"storeType": "filer",
"storeConfig": {
"filerAddress": "localhost:8888",
"basePath": "/seaweedfs/iam/roles"
}
}
}
```
## Storage Backends
### Memory Storage (Default)
- **Performance**: Fastest access (sub-millisecond)
- **Persistence**: Lost on restart
- **Distribution**: ❌ Not shared across instances
- **Use case**: Single instance or development
### Filer Storage (Distributed)
- **Performance**: Network latency + filer performance
- **Persistence**: ✅ Survives restarts
- **Distribution**: ✅ Shared across all instances
- **Use case**: Production multi-instance deployments
## Deployment Examples
### Single Instance (Simple)
```bash
# Standard deployment - no special configuration needed
weed s3 -filer=localhost:8888 -port=8333
```
### Multiple Instances (Distributed)
```bash
# Instance 1
weed s3 -filer=localhost:8888 -port=8333 -iam.config=/path/to/distributed_config.json
# Instance 2
weed s3 -filer=localhost:8888 -port=8334 -iam.config=/path/to/distributed_config.json
# Instance N
weed s3 -filer=localhost:8888 -port=833N -iam.config=/path/to/distributed_config.json
```
All instances share the same IAM state through the filer.
### Load Balancer Configuration
```nginx
upstream seaweedfs-s3 {
# No need for session affinity with distributed storage
server gateway-1:8333;
server gateway-2:8334;
server gateway-3:8335;
}
server {
listen 80;
location / {
proxy_pass http://seaweedfs-s3;
}
}
```
## Storage Locations
When using filer storage, IAM data is stored at:
```
/seaweedfs/iam/
├── sessions/ # STS session tokens
│ ├── session_abc123.json
│ └── session_def456.json
├── policies/ # IAM policy documents
│ ├── policy_S3AdminPolicy.json
│ └── policy_S3ReadOnlyPolicy.json
└── roles/ # IAM role definitions
├── S3AdminRole.json
└── S3ReadOnlyRole.json
```
## Performance Considerations
### Memory vs Filer Storage
| Aspect | Memory | Filer |
|--------|--------|-------|
| **Read Latency** | <1ms | ~5-20ms |
| **Write Latency** | <1ms | ~10-50ms |
| **Consistency** | Per-instance | Global |
| **Persistence** | None | Full |
| **Scalability** | Limited | Unlimited |
### Optimization Tips
1. **Use fast storage** for the filer (SSD recommended)
2. **Co-locate filer** with S3 gateways to reduce network latency
3. **Enable filer caching** for frequently accessed IAM data
4. **Monitor filer performance** - IAM operations depend on it
## Migration Guide
### From Single to Multi-Instance
1. **Stop existing S3 gateway**
2. **Update configuration** to use filer storage:
```json
{
"policy": { "storeType": "filer" },
"sts": { "sessionStoreType": "filer" },
"roleStore": { "storeType": "filer" }
}
```
3. **Start first instance** with new config
4. **Verify IAM data** was migrated to filer
5. **Start additional instances** with same config
### From Memory to Filer Storage
IAM data in memory is **not automatically migrated**. You'll need to:
1. **Export existing roles/policies** from configuration files
2. **Update configuration** to use filer storage
3. **Restart with new config** - data will be loaded from config into filer
4. **Remove config-based definitions** (now stored in filer)
## Troubleshooting
### Configuration Issues
```bash
# Check IAM configuration is loaded correctly
grep "advanced IAM" /path/to/s3-gateway.log
# Verify filer connectivity
weed filer.cat /seaweedfs/iam/roles/
```
### Inconsistent Behavior
```bash
# Check if all instances use same filer
curl http://gateway-1:8333/status
curl http://gateway-2:8334/status
# Verify IAM storage locations exist
weed filer.ls /seaweedfs/iam/
```
### Performance Issues
```bash
# Monitor filer latency
curl http://filer:8888/stats
# Check IAM storage path performance
time weed filer.cat /seaweedfs/iam/roles/TestRole.json
```
## Best Practices
### Security
- **Secure filer access** - use TLS and authentication
- **Limit IAM path access** - only S3 gateways should access `/seaweedfs/iam/`
- **Regular backups** of IAM data in filer
- **Monitor access** to IAM storage paths
### High Availability
- **Replicate filer** across multiple nodes
- **Use fast, reliable storage** for filer backend
- **Monitor filer health** - critical for IAM operations
- **Implement alerts** for IAM storage issues
### Capacity Planning
- **Estimate IAM data size**: Roles + Policies + Sessions
- **Plan for growth**: Active sessions scale with user count
- **Monitor filer disk usage**: `/seaweedfs/iam/` path
- **Set up log rotation**: For IAM audit logs
## Architecture Comparison
### Before (Memory-Only)
```
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ S3 Gateway 1│ │ S3 Gateway 2│ │ S3 Gateway 3│
│ │ │ │ │ │
│ ┌─────────┐ │ │ ┌─────────┐ │ │ ┌─────────┐ │
│ │ IAM │ │ │ │ IAM │ │ │ │ IAM │ │
│ │ Memory │ │ │ │ Memory │ │ │ │ Memory │ │
│ └─────────┘ │ │ └─────────┘ │ │ └─────────┘ │
└─────────────┘ └─────────────┘ └─────────────┘
❌ ❌ ❌
Inconsistent Inconsistent Inconsistent
```
### After (Filer-Distributed)
```
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ S3 Gateway 1│ │ S3 Gateway 2│ │ S3 Gateway 3│
│ │ │ │ │ │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │
└──────────────────┼──────────────────┘
│
┌─────────▼─────────┐
│ SeaweedFS │
│ Filer │
│ ┌───────────────┐ │
│ │ /seaweedfs/ │ │
│ │ iam/ │ │
│ │ ├─roles/ │ │
│ │ ├─policies/ │ │
│ │ └─sessions/ │ │
│ └───────────────┘ │
└───────────────────┘
✅
Consistent
```
## Summary
The distributed IAM system solves critical consistency issues in multi-instance deployments by:
- **Centralizing IAM state** in SeaweedFS filer
- **Ensuring consistency** across all gateway instances
- **Maintaining performance** with configurable storage backends
- **Supporting scalability** for any number of gateway instances
For production deployments with multiple S3 gateway instances, **always use filer-based storage** for all IAM components.
test/s3/iam/iam_config_distributed.json
+176
View File
@@ -0,0 +1,176 @@
{
"sts": {
"tokenDuration": 3600000000000,
"maxSessionLength": 43200000000000,
"issuer": "seaweedfs-sts",
"signingKey": "dGVzdC1zaWduaW5nLWtleS0zMi1jaGFyYWN0ZXJzLWxvbmc=",
"sessionStoreType": "filer",
"sessionStoreConfig": {
"filerAddress": "localhost:8888",
"basePath": "/seaweedfs/iam/sessions"
}
},
"policy": {
"defaultEffect": "Deny",
"storeType": "filer",
"storeConfig": {
"filerAddress": "localhost:8888",
"basePath": "/seaweedfs/iam/policies"
}
},
"roleStore": {
"storeType": "filer",
"storeConfig": {
"filerAddress": "localhost:8888",
"basePath": "/seaweedfs/iam/roles"
}
},
"providers": [
{
"name": "keycloak-oidc",
"type": "oidc",
"config": {
"issuer": "http://keycloak:8080/realms/seaweedfs-test",
"clientId": "seaweedfs-s3",
"clientSecret": "seaweedfs-s3-secret",
"redirectUri": "http://localhost:8333/auth/callback",
"scopes": ["openid", "profile", "email", "roles"],
"usernameClaim": "preferred_username",
"groupsClaim": "roles"
}
}
],
"roles": [
{
"roleName": "S3AdminRole",
"roleArn": "arn:seaweed:iam::role/S3AdminRole",
"trustPolicy": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "keycloak-oidc"
},
"Action": ["sts:AssumeRoleWithWebIdentity"],
"Condition": {
"StringEquals": {
"roles": "s3-admin"
}
}
}
]
},
"attachedPolicies": ["S3AdminPolicy"],
"description": "Full S3 administrator access role"
},
{
"roleName": "S3ReadOnlyRole",
"roleArn": "arn:seaweed:iam::role/S3ReadOnlyRole",
"trustPolicy": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "keycloak-oidc"
},
"Action": ["sts:AssumeRoleWithWebIdentity"],
"Condition": {
"StringEquals": {
"roles": "s3-read-only"
}
}
}
]
},
"attachedPolicies": ["S3ReadOnlyPolicy"],
"description": "Read-only access to S3 resources"
},
{
"roleName": "S3ReadWriteRole",
"roleArn": "arn:seaweed:iam::role/S3ReadWriteRole",
"trustPolicy": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "keycloak-oidc"
},
"Action": ["sts:AssumeRoleWithWebIdentity"],
"Condition": {
"StringEquals": {
"roles": "s3-read-write"
}
}
}
]
},
"attachedPolicies": ["S3ReadWritePolicy"],
"description": "Read-write access to S3 resources"
}
],
"policies": [
{
"name": "S3AdminPolicy",
"document": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "*"
}
]
}
},
{
"name": "S3ReadOnlyPolicy",
"document": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectAcl",
"s3:GetObjectVersion",
"s3:ListBucket",
"s3:ListBucketVersions"
],
"Resource": [
"arn:seaweed:s3:::*",
"arn:seaweed:s3:::*/*"
]
}
]
}
},
{
"name": "S3ReadWritePolicy",
"document": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectAcl",
"s3:GetObjectVersion",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:DeleteObject",
"s3:ListBucket",
"s3:ListBucketVersions"
],
"Resource": [
"arn:seaweed:s3:::*",
"arn:seaweed:s3:::*/*"
]
}
]
}
}
]
}