diff --git a/design-cloudstack-provider.md b/design-cloudstack-provider.md new file mode 100644 index 000000000..24e99db45 --- /dev/null +++ b/design-cloudstack-provider.md @@ -0,0 +1,418 @@ +# SeaweedFS as an Apache CloudStack Object Storage Provider + +A CloudStack ObjectStore plugin that makes SeaweedFS a first-class object storage +backend inside Apache CloudStack, alongside the existing MinIO and Ceph RGW +providers. This is a collaboration with proIO (Swen), who builds private clouds on +CloudStack and wants SeaweedFS as a storage option. + +## The request + +> We can only add MinIO and Ceph as object storage [in CloudStack] today. I want +> to get SeaweedFS into this project... What we need is to build a provider which +> does the communication between Cloudstack and SeaweedFS. + +This is **not** a SeaweedFS-side feature. The work lives in the Apache CloudStack +repo (Java): a new plugin under `plugins/storage/object/seaweedfs/` that implements +CloudStack's ObjectStore plugin framework and talks to SeaweedFS over its S3 and +IAM APIs. SeaweedFS itself needs no changes for the core to work — its S3 API +already covers every bucket operation CloudStack requires, and its IAM API covers +user/credential management. + +## How the CloudStack ObjectStore framework works + +CloudStack 4.18+ introduced an Object Storage framework. An admin registers an +object storage pool via `addObjectStoragePool` (URL + provider + credentials); +tenants then create and manage buckets on it through CloudStack APIs. CloudStack +manages pool and bucket lifecycle; the underlying provider handles the actual +object protocol. + +A provider is a plugin module implementing three interfaces: + +### 1. `ObjectStoreProvider` — registration + +`MinIOObjectStoreProviderImpl` is the reference. It is a Spring `@Component` that: +- Returns a provider name (`"MinIO"`) +- Returns `DataStoreProviderType.OBJECT` +- In `configure()`, injects the lifecycle and driver implementations and calls + `storeMgr.registerDriver(name, driver)` + +### 2. `ObjectStoreLifeCycle` — pool add/remove + +`MinIOObjectStoreLifeCycleImpl.initialize()` reads the URL, name, and +`accesskey`/`secretkey` details from the `addObjectStoragePool` call, tests the +connection by listing buckets, and persists an `ObjectStoreVO` via +`ObjectStoreHelper`. The other methods (attachCluster/Host/Zone, maintain, +deleteDataStore) are no-ops for object storage. + +### 3. `ObjectStoreDriver` — bucket + user operations + +`ObjectStoreDriver` (in `engine/storage/.../object/ObjectStoreDriver.java`) extends +`DataStoreDriver` and defines the bucket/user contract. Every provider must +implement: + +| Method | Purpose | +| --- | --- | +| `createBucket(Bucket, boolean objectLock)` | Create a bucket | +| `listBuckets(long storeId)` | List all buckets | +| `deleteBucket(BucketTO, long storeId)` | Delete a bucket | +| `createUser(long accountId, long storeId)` | Provision a user + credentials for a CloudStack account | +| `setBucketPolicy` / `getBucketPolicy` / `deleteBucketPolicy` | Bucket policy CRUD | +| `setBucketEncryption` / `deleteBucketEncryption` | SSE config | +| `setBucketVersioning` / `deleteBucketVersioning` | Versioning enable/suspend | +| `setBucketQuota(BucketTO, long storeId, long size)` | Per-bucket quota | +| `getAllBucketsUsage(long storeId)` | Usage map for billing/accounting | +| `getBucketAcl` / `setBucketAcl` | ACLs (MinIO/Ceph return null / no-op) | + +`BaseObjectStoreDriverImpl` provides no-op defaults for the `DataStoreDriver` +methods (`createAsync`, `deleteAsync`, `copyAsync`, `canCopy`, `resize`, +`getTO`, `getStoreTO`), so object-store providers only implement the bucket/user +methods above. + +## How the four existing providers differ (and where SeaweedFS lands) + +CloudStack ships four object-store providers. Three are relevant; the simulator +is a test stub. + +| Concern | MinIO | Ceph RGW | Cloudian HyperStore | SeaweedFS | +| --- | --- | --- | --- | --- | +| Bucket CRUD | `MinioClient` (S3) | `AmazonS3` (AWS SDK v1) | `AmazonS3` (AWS SDK v1) | `AmazonS3` (AWS SDK v1) | +| Bucket policy | `MinioClient` | `AmazonS3` | `AmazonS3` | `AmazonS3` | +| Versioning | `MinioClient` | `AmazonS3` | `AmazonS3` | `AmazonS3` | +| Encryption | `MinioClient` | not implemented | `AmazonS3` | `AmazonS3` | +| **User creation** | `MinioAdminClient` | `RgwAdmin` | **`AmazonIdentityManagement`** | **`AmazonIdentityManagement`** | +| **Per-bucket quota** | `MinioAdminClient` | `RgwAdmin` | **not supported** (throws) | **S3 extension** (`PUT /{bucket}?seaweedfs-quota`, SigV4, `s3:PutBucketQuota`) | +| **Usage reporting** | `MinioAdminClient` | `RgwAdmin` | Cloudian admin API | S3 `ListObjectsV2` (MVP); Prometheus / SOSAPI `capacity.xml` (recommended) | + +**Cloudian HyperStore is the direct precedent.** It is an S3-compatible store +that, like SeaweedFS, manages users via the **standard AWS IAM API** using the +AWS IAM Java SDK (`com.amazonaws.services.identitymanagement`). Its driver +(`CloudianHyperStoreObjectStoreDriverImpl`) and util +(`CloudianHyperStoreUtil`) are the template this design follows almost line for +line. Cloudian even validates the quota limitation the same way this design +proposes for the MVP: `setBucketQuota` throws for any non-zero size and only +accepts `0` (no quota). + +The SeaweedFS plugin is therefore a **simpler Cloudian** — same AWS S3 + IAM SDK +clients, same store-details keys (`s3Url`, `iamUrl`, `accesskey`, `secretkey`), +same IAM-user-with-restricted-policy pattern, but with no proprietary admin +client at all (Cloudian has its own `CloudianClient` for its admin API; SeaweedFS +needs only S3 + IAM). For quota, the plugin uses a narrow SeaweedFS S3 extension +(see below); for usage reporting, it falls back to S3 `ListObjectsV2` in the MVP +and recommends Prometheus or SOSAPI `capacity.xml` for production scale. + +### Quota via the S3 `?seaweedfs-quota` extension + +SeaweedFS supports bucket quota natively (server-side enforcement via a +read-only flag when usage exceeds the limit). Rather than exposing the broad +admin REST API (which would require a global bearer token and grant cluster-wide +admin access), the integration uses a **narrow, scoped S3 subresource**: + +- `PUT /{bucket}?seaweedfs-quota` — set bucket quota (IAM permission `s3:PutBucketQuota`) +- `GET /{bucket}?seaweedfs-quota` — get bucket quota (IAM permission `s3:GetBucketQuota`) + +**PUT request body** (JSON): +```json +{"quota_size": 100, "quota_unit": "GB", "quota_enabled": true} +``` + +**GET response body** (JSON): +```json +{"quota_size": 107374182400, "quota_unit": "B", "quota_enabled": true} +``` + +Note: GET always returns `quota_unit: "B"` and the absolute byte count, not +the original unit. A disabled-but-retained quota returns a positive +`quota_size` with `quota_enabled: false`. + +Quota is stored on the bucket's filer entry (positive = enabled, negative = +disabled but retained, zero = no quota), matching the existing admin REST API +behavior. When quota is cleared, the bucket's read-only flag is also lifted. + +**Authentication** uses the existing S3 SigV4 flow — no new global secret is +needed. The CloudStack service credential (the `accesskey`/`secretkey` on the +object store) is the admin credential used for all driver operations: bucket +CRUD, IAM user provisioning, and quota management. It must have broad S3 and +IAM permissions. The per-account IAM users created by `createUser` are the +ones with restricted permissions (full S3 access except bucket +creation/deletion). A future hardening could split quota management onto a +separate credential scoped to only `s3:PutBucketQuota`/`s3:GetBucketQuota`, +but the MVP uses the single admin credential for simplicity, matching how +the MinIO and Ceph providers work. + +The plugin's `setBucketQuota` signs and sends the `PUT /{bucket}?seaweedfs-quota` +request using the AWS SDK v1 `AWSS3V4Signer` for SigV4 signing, then sends the +signed request via `java.net.http.HttpClient` (the AWS S3 SDK doesn't natively +support custom subresources, so we sign manually and send the request +ourselves). The `seaweedfs-quota` query parameter is included in the signed +canonical query string. + +### Usage reporting + +`getAllBucketsUsage` must return a `Map` of bucket name → size. +MinIO uses `MinioAdminClient.getDataUsageInfo`; Ceph uses +`RgwAdmin.listBucketInfo`. SeaweedFS has no admin rollup endpoint, so the MVP +plugin computes it by listing buckets and summing object sizes via S3 +`ListObjectsV2` — expensive for large stores. + +For production scale, SeaweedFS already exposes per-bucket size in: +- **Prometheus metrics** (`bucket_size_bytes` gauge, refreshed every minute) +- **SOSAPI `capacity.xml`** (reports capacity, available space, and usage + through the S3 endpoint) + +Operators should consume one of those instead of S3 list-based aggregation for +large deployments. The MVP's list-based approach is correct but slow; flag it as +a known limitation. + +## SeaweedFS API surface (what the plugin relies on) + +SeaweedFS exposes two relevant APIs, both AWS-compatible: + +### S3 API (`weed s3`) +Full S3-compatible surface. Confirmed against the SeaweedFS S3 wiki and code: +- `CreateBucket`, `HeadBucket`, `ListBuckets`, `DeleteBucket` +- `PutBucketPolicy`, `GetBucketPolicy`, `DeleteBucketPolicy` +- `PutBucketVersioning` (Enabled / Suspended), `GetBucketVersioning` +- `PutBucketEncryption`, `GetBucketEncryption`, `DeleteBucketEncryption` +- `PutBucketAcl`, `GetBucketAcl` +- `ListObjectsV2`, `HeadObject`, `GetObject`, `PutObject`, `DeleteObject` +- Bucket quota via extended attributes / `s3.bucket.quota` (enforced server-side, + surfaced as a read-only state when exceeded — see PR #10224) + +### IAM API (`weed iam` / `iamapi`) +AWS IAM-compatible REST endpoints, implemented in `weed/iamapi/`. Confirmed by +the test suite which uses the **AWS IAM SDK** (`aws-sdk-go/service/iam`) against +the same handlers CloudStack would call: +- `CreateUser`, `DeleteUser`, `ListUsers`, `GetUser` +- `CreateAccessKey`, `DeleteAccessKey`, `ListAccessKeys` +- `PutUserPolicy`, `GetUserPolicy`, `DeleteUserPolicy` +- `AttachUserPolicy`, `ListAttachedUserPolicies` + +This means the CloudStack plugin can manage SeaweedFS users with the **AWS IAM +Java SDK** (`com.amazonaws.services.identitymanagement.AmazonIdentityManagement`), +exactly the way the AWS IAM Go SDK is used in SeaweedFS's own tests. No proprietary +admin client is needed. **Cloudian HyperStore already does exactly this** in the +CloudStack tree — the SeaweedFS plugin follows the same pattern. + +## Design + +### Module layout + +New CloudStack plugin module, mirroring `plugins/storage/object/cloudian/` +(the closest precedent — same AWS S3 + IAM SDK approach): + +``` +plugins/storage/object/seaweedfs/ + pom.xml + src/main/java/org/apache/cloudstack/storage/datastore/ + driver/SeaweedFSObjectStoreDriverImpl.java + lifecycle/SeaweedFSObjectStoreLifeCycleImpl.java + provider/SeaweedFSObjectStoreProviderImpl.java + util/SeaweedFSObjectStoreUtil.java + src/test/java/org/apache/cloudstack/storage/datastore/ + driver/SeaweedFSObjectStoreDriverImplTest.java + provider/SeaweedFSObjectStoreProviderImplTest.java + src/main/resources/META-INF/cloudstack/storage-object-seaweedfs/ + module.properties + spring-storage-object-seaweedfs-context.xml +``` + +### `SeaweedFSObjectStoreProviderImpl` + +Direct copy of `MinIOObjectStoreProviderImpl` with `providerName = "SeaweedFS"`, +injecting the SeaweedFS lifecycle and driver. Registers via +`storeMgr.registerDriver`. + +### `SeaweedFSObjectStoreLifeCycleImpl` + +Copy of `MinIOObjectStoreLifeCycleImpl`. `initialize()` reads `url`, `name`, +`accesskey`, `secretkey` from the `addObjectStoragePool` details map, tests the +connection by calling `AmazonS3.listBuckets()` against the SeaweedFS S3 endpoint, +and persists the `ObjectStoreVO`. No proprietary client needed — the AWS S3 SDK +is enough for the health check. + +### `SeaweedFSObjectStoreDriverImpl` + +The substantive class. Uses two AWS SDK v1 clients (same dependency Ceph already +pulls in, so no new CloudStack dependency): + +- `AmazonS3` for bucket operations (path-style, endpoint-pinned, `us-east-1` + region placeholder — same as Ceph's `getS3Client`) +- `AmazonIdentityManagement` for user/credential operations, pointed at the + SeaweedFS IAM endpoint + +#### Bucket operations — straightforward S3 + +| Interface method | Implementation | +| --- | --- | +| `createBucket` | `s3.createBucket(name)`; reject if `doesBucketExistV2`; persist access/secret key + URL on `BucketVO` (same as Ceph) | +| `listBuckets` | `s3.listBuckets()` → wrap as `BucketObject` (same as Ceph) | +| `deleteBucket` | `s3.deleteBucket(name)` (same as Ceph) | +| `setBucketPolicy` | `s3.setBucketPolicy(...)` with the same public/private JSON the MinIO/Ceph drivers build | +| `getBucketPolicy` / `deleteBucketPolicy` | `s3.getBucketPolicy` / `s3.deleteBucketPolicy` | +| `setBucketVersioning` | `s3.setBucketVersioningConfiguration(Enabled)` | +| `deleteBucketVersioning` | `s3.setBucketVersioningConfiguration(Suspended)` | +| `setBucketEncryption` | `s3.setBucketEncryptionConfiguration(SSE-S3 rule)` | +| `deleteBucketEncryption` | `s3.deleteBucketEncryptionConfiguration` | +| `getBucketAcl` / `setBucketAcl` | no-op / null (same as MinIO and Ceph) | + +#### User creation — the key difference + +MinIO calls `MinioAdminClient.addUser`; Ceph calls `RgwAdmin.createUser`. SeaweedFS +exposes the standard AWS IAM API, so the plugin calls: + +```java +AmazonIdentityManagement iam = getIamClient(storeId); +String userName = "acs-" + account.getUuid(); + +// CreateUser (idempotent — check GetUser first, like Ceph does) +iam.createUser(new CreateUserRequest(userName)); + +// CreateAccessKey → returns the access key + secret key to persist +CreateAccessKeyResult result = iam.createAccessKey( + new CreateAccessKeyRequest().withUserName(userName)); +AccessKey key = result.getAccessKey(); + +// Persist per-account, same pattern as Ceph's CEPH_ACCESS_KEY/CEPH_SECRET_KEY +details.put(SEAWEEDFS_ACCESS_KEY, key.getAccessKeyId()); +details.put(SEAWEEDFS_SECRET_KEY, key.getSecretAccessKey()); +_accountDetailsDao.persist(accountId, details); +``` + +This is the cleanest mapping of the three providers: no proprietary admin client, +just the AWS IAM SDK that CloudStack already has access to. The IAM endpoint URL +is provided as `iamUrl` in the store details. If `iamUrl` is omitted, the driver +defaults it to `s3Url` — SeaweedFS registers its embedded IAM API at `POST /` on +the same S3 endpoint (`UnifiedPostHandler` in `s3api_server.go`), so the IAM +endpoint is the same as the S3 endpoint unless the deployment runs a separate +`weed iam` server. + +#### Bucket quota — S3 `?seaweedfs-quota` extension + +This is the one genuine gap. MinIO and Ceph both have an admin API to set a +per-bucket quota that the backend enforces. SeaweedFS enforces bucket quota +server-side, but the configuration path was **not exposed over a standard S3 or +IAM API** — it was only set via the admin REST API or shell commands. + +The integration adds a **narrow S3 subresource** to SeaweedFS: +- `PUT /{bucket}?seaweedfs-quota` — set bucket quota (IAM permission `s3:PutBucketQuota`) +- `GET /{bucket}?seaweedfs-quota` — get bucket quota (IAM permission `s3:GetBucketQuota`) + +This is implemented in SeaweedFS PR #11279. It uses SigV4 authentication and +dedicated IAM permissions, so the CloudStack service credential can be scoped +to quota management only — no global admin token, no cluster-wide admin access. +The enforcement already exists (PR #10224); this PR only adds the HTTP +configuration surface. + +An earlier approach (PR #11278, closed) added bearer-token auth to the broad +admin REST API. After review, that was unnecessary for this integration — +static S3 config plus standard S3 APIs plus one scoped quota mutation API is +sufficient and far safer. + +> **Note on AWS tools compatibility.** `?seaweedfs-quota` is a SeaweedFS-specific +> S3 subresource, not part of the AWS S3 API. Standard AWS tools (`aws s3api`, +> `s3cmd`, `rclone`) cannot call it directly. This is the same limitation MinIO +> and Ceph have — MinIO quota lives behind a separate admin API (`mc admin +> bucket quota`), and Ceph quota lives behind the Admin Ops API +> (`radosgw-admin quota set`). Neither is callable via `aws s3api` either. +> SeaweedFS's approach is the closest to standard S3 because it uses the same +> endpoint and same SigV4 credentials, just with a custom query parameter. +> Interactive quota management remains available via `weed shell`; the S3 +> extension exists for programmatic integration (CloudStack) where the +> integrator can sign SigV4 requests but cannot run shell commands. + +#### Usage reporting + +`getAllBucketsUsage` must return a `Map` of bucket name → size. +MinIO uses `MinioAdminClient.getDataUsageInfo`; Ceph uses +`RgwAdmin.listBucketInfo`. SeaweedFS has no admin rollup endpoint, so the MVP +plugin computes it by listing buckets and summing object sizes via S3 +`ListObjectsV2` — expensive for large stores. Better options exist in +SeaweedFS already: +- **Prometheus metrics** (`bucket_size_bytes` gauge, refreshed every minute) +- **SOSAPI `capacity.xml`** (reports capacity, available space, and usage + through the S3 endpoint — note: the current "return zero on backend error" + behavior should be validated before using it for billing) + +For the MVP, `listBuckets` + per-bucket size via the S3 API is correct but slow; +flag it as a known limitation. Operators should consume Prometheus or SOSAPI +for production-scale usage reporting. + +### Spring wiring + +`spring-storage-object-seaweedfs-context.xml` registers the provider bean, +identical to the MinIO one. `module.properties` sets +`name=storage-object-seaweedfs`, `parent=storage`. + +### `pom.xml` + +Depends on `aws-java-sdk-s3` and `aws-java-sdk-iam` — both already in the +CloudStack dependency tree (Ceph uses the S3 SDK; the IAM SDK is the standard AWS +bundle). No new third-party dependency, unlike MinIO which pulls in the MinIO +Java client. + +## What changes on the SeaweedFS side + +**One narrow S3 extension is required for quota management.** SeaweedFS PR #11279 +adds the `?seaweedfs-quota` S3 subresource: + +- `PUT /{bucket}?seaweedfs-quota` — set bucket quota (IAM permission `s3:PutBucketQuota`) +- `GET /{bucket}?seaweedfs-quota` — get bucket quota (IAM permission `s3:GetBucketQuota`) + +This is authenticated via standard S3 SigV4 and authorized via dedicated IAM +permissions, so no global admin token is needed. The enforcement already exists +(PR #10224); this PR only adds the HTTP configuration surface. + +One follow-up improvement on the SeaweedFS side would close the usage reporting +gap: + +1. **Validate SOSAPI `capacity.xml` usage calculation** — the current "return + zero on backend error" behavior should be validated before using it for + billing. If reliable, CloudStack can consume it directly instead of + list-based aggregation. + +## Open questions for proIO / Swen + +1. **IAM endpoint path.** ~~Where does `weed iam` listen relative to the S3 + endpoint in a typical proIO deployment?~~ **Resolved.** SeaweedFS registers + its embedded IAM API at `POST /` on the same S3 endpoint + (`UnifiedPostHandler`), so the driver defaults `iamUrl` to `s3Url`. A + separate `iamUrl` is only needed if the deployment runs a standalone + `weed iam` server on a different host/port. +2. **Quota requirements.** Do proIO's customers need server-enforced per-bucket + quotas, or is CloudStack-side accounting sufficient for the first release? + The `?seaweedfs-quota` S3 extension (PR #11279) provides server-enforced + quotas via a scoped credential; this is the recommended path. +3. **Object Lock.** `createBucket` takes an `objectLock` boolean. MinIO supports + it; Ceph ignores it. SeaweedFS has Object Lock support. Should the plugin pass + it through? +4. **Contribution model.** Does proIO want to submit the PR to + `apache/cloudstack` themselves (with SeaweedFS maintainers as reviewers), or + the reverse? Apache CloudStack requires an ICLA for non-trivial contributions. + +## Files + +All in the `apache/cloudstack` repo (new module): + +| File | Purpose | +| --- | --- | +| `plugins/storage/object/seaweedfs/pom.xml` | Maven module | +| `.../datastore/util/SeaweedFSObjectStoreUtil.java` | S3 + IAM client builders, constants, URL validators | +| `.../datastore/provider/SeaweedFSObjectStoreProviderImpl.java` | Spring provider registration | +| `.../datastore/lifecycle/SeaweedFSObjectStoreLifeCycleImpl.java` | Pool add/health-check | +| `.../datastore/driver/SeaweedFSObjectStoreDriverImpl.java` | Bucket + user ops via S3 + IAM SDK | +| `.../resources/META-INF/cloudstack/storage-object-seaweedfs/module.properties` | Module name | +| `.../resources/META-INF/cloudstack/storage-object-seaweedfs/spring-storage-object-seaweedfs-context.xml` | Spring bean | +| `plugins/pom.xml` | Register `storage/object/seaweedfs` module | + +No files in `seaweedfs/seaweedfs` for the MVP. + +### SeaweedFS-side changes (PR #11279) + +| File | Purpose | +| --- | --- | +| `weed/s3api/s3_constants/s3_action_strings.go` | Add `S3_ACTION_PUT_BUCKET_QUOTA` and `S3_ACTION_GET_BUCKET_QUOTA` | +| `weed/s3api/s3_constants/s3_actions.go` | Add coarse-grained `ACTION_PUT_BUCKET_QUOTA` and `ACTION_GET_BUCKET_QUOTA` | +| `weed/s3api/s3_action_resolver.go` | Map `seaweedfs-quota` query param to fine-grained s3: actions | +| `weed/s3api/s3api_bucket_quota_handlers.go` | New — `PutBucketQuotaHandler` and `GetBucketQuotaHandler` | +| `weed/s3api/s3api_bucket_quota_handlers_test.go` | New — tests for unit conversion, validation, and error paths | +| `weed/s3api/s3api_server.go` | Register the two routes in the bucket subrouter | diff --git a/weed/s3api/s3_action_resolver.go b/weed/s3api/s3_action_resolver.go index 707f3171f..176d89de8 100644 --- a/weed/s3api/s3_action_resolver.go +++ b/weed/s3api/s3_action_resolver.go @@ -116,6 +116,11 @@ var bucketQueryActions = map[string]map[string]string{ http.MethodPut: s3_constants.S3_ACTION_PUT_BUCKET_OWNERSHIP_CONTROLS, http.MethodDelete: s3_constants.S3_ACTION_PUT_BUCKET_OWNERSHIP_CONTROLS, // DELETE uses same permission as PUT }, + // SeaweedFS extension: bucket quota subresource + "seaweedfs-quota": { + http.MethodGet: s3_constants.S3_ACTION_GET_BUCKET_QUOTA, + http.MethodPut: s3_constants.S3_ACTION_PUT_BUCKET_QUOTA, + }, } // resolveFromQueryParameters checks query parameters to determine specific S3 actions @@ -366,6 +371,10 @@ func mapBaseActionToS3Format(baseAction string) string { return s3_constants.S3_ACTION_PUT_BUCKET_POLICY case s3_constants.ACTION_DELETE_BUCKET_POLICY: return s3_constants.S3_ACTION_DELETE_BUCKET_POLICY + case s3_constants.ACTION_PUT_BUCKET_QUOTA: + return s3_constants.S3_ACTION_PUT_BUCKET_QUOTA + case s3_constants.ACTION_GET_BUCKET_QUOTA: + return s3_constants.S3_ACTION_GET_BUCKET_QUOTA default: // For unknown actions, prefix with s3: to maintain format consistency return "s3:" + baseAction diff --git a/weed/s3api/s3_constants/s3_action_strings.go b/weed/s3api/s3_constants/s3_action_strings.go index 618c4c7fa..af14ba5e9 100644 --- a/weed/s3api/s3_constants/s3_action_strings.go +++ b/weed/s3api/s3_constants/s3_action_strings.go @@ -100,6 +100,12 @@ const ( S3_ACTION_GET_BUCKET_OWNERSHIP_CONTROLS = "s3:GetBucketOwnershipControls" S3_ACTION_PUT_BUCKET_OWNERSHIP_CONTROLS = "s3:PutBucketOwnershipControls" + // SeaweedFS extension: bucket quota operations + // PUT /{bucket}?seaweedfs-quota and GET /{bucket}?seaweedfs-quota + // These are SeaweedFS-specific subresources, not part of the AWS S3 API. + S3_ACTION_PUT_BUCKET_QUOTA = "s3:PutBucketQuota" + S3_ACTION_GET_BUCKET_QUOTA = "s3:GetBucketQuota" + // Wildcard for all S3 actions S3_ACTION_ALL = "s3:*" ) diff --git a/weed/s3api/s3_constants/s3_actions.go b/weed/s3api/s3_constants/s3_actions.go index f5122bb6d..468ae8809 100644 --- a/weed/s3api/s3_constants/s3_actions.go +++ b/weed/s3api/s3_constants/s3_actions.go @@ -18,6 +18,8 @@ const ( ACTION_PUT_BUCKET_OBJECT_LOCK_CONFIG = "PutBucketObjectLockConfiguration" ACTION_PUT_BUCKET_POLICY = "PutBucketPolicy" ACTION_DELETE_BUCKET_POLICY = "DeleteBucketPolicy" + ACTION_PUT_BUCKET_QUOTA = "PutBucketQuota" + ACTION_GET_BUCKET_QUOTA = "GetBucketQuota" SeaweedStorageDestinationHeader = "x-seaweedfs-destination" MultipartUploadsFolder = ".uploads" diff --git a/weed/s3api/s3api_bucket_quota_handlers.go b/weed/s3api/s3api_bucket_quota_handlers.go new file mode 100644 index 000000000..01ef830ef --- /dev/null +++ b/weed/s3api/s3api_bucket_quota_handlers.go @@ -0,0 +1,236 @@ +package s3api + +import ( + "encoding/json" + "errors" + "fmt" + "io" + "math" + "net/http" + + "github.com/seaweedfs/seaweedfs/weed/filer" + "github.com/seaweedfs/seaweedfs/weed/glog" + "github.com/seaweedfs/seaweedfs/weed/pb/filer_pb" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3err" +) + +// putBucketQuotaMaxBodyBytes caps the request body to prevent DoS via large +// payloads. The valid payload is a few hundred bytes. +const putBucketQuotaMaxBodyBytes = 64 * 1024 + +// bucketQuotaRequest is the JSON body for PUT /{bucket}?seaweedfs-quota. +// +// SeaweedFS stores quota on the bucket's filer entry: +// - positive value: quota enabled, enforced server-side +// - negative value: quota disabled but size retained +// - zero: no quota +// +// The quota_unit field accepts B, KB, MB, GB, TB (case-insensitive). +// quota_size is the numeric size in the given unit. +// quota_enabled false with a positive size stores a negative (disabled) quota. +type bucketQuotaRequest struct { + QuotaSize int64 `json:"quota_size"` + QuotaUnit string `json:"quota_unit"` + QuotaEnabled bool `json:"quota_enabled"` +} + +// bucketQuotaResponse is the JSON body for GET /{bucket}?seaweedfs-quota. +type bucketQuotaResponse struct { + QuotaSize int64 `json:"quota_size"` + QuotaUnit string `json:"quota_unit"` + QuotaEnabled bool `json:"quota_enabled"` +} + +// PutBucketQuotaHandler handles PUT /{bucket}?seaweedfs-quota. +// +// This is a SeaweedFS-specific S3 extension that allows setting a bucket's +// storage quota through the S3 API, authenticated via SigV4 and authorized via +// the s3:PutBucketQuota IAM permission. It avoids the need for a separate +// admin API credential for integrations like Apache CloudStack. +func (s3a *S3ApiServer) PutBucketQuotaHandler(w http.ResponseWriter, r *http.Request) { + bucket, _ := s3_constants.GetBucketAndObject(r) + glog.V(3).Infof("PutBucketQuotaHandler %s", bucket) + + if bucket == "" { + s3err.WriteErrorResponse(w, r, s3err.ErrInvalidBucketName) + return + } + + r.Body = http.MaxBytesReader(w, r.Body, putBucketQuotaMaxBodyBytes) + defer r.Body.Close() + + var req bucketQuotaRequest + dec := json.NewDecoder(r.Body) + if err := dec.Decode(&req); err != nil { + s3err.WriteErrorResponse(w, r, s3err.ErrMalformedXML) + return + } + // Reject trailing data after the JSON object to prevent malformed payloads + // from being silently accepted. + if err := dec.Decode(&struct{}{}); err != io.EOF { + writeQuotaError(w, r, http.StatusBadRequest, "unexpected trailing data after JSON object") + return + } + + if req.QuotaEnabled && req.QuotaSize <= 0 { + writeQuotaError(w, r, http.StatusBadRequest, "quota_size must be > 0 when quota_enabled is true") + return + } + + normalizedUnit, err := normalizeQuotaUnit(req.QuotaUnit) + if err != nil { + writeQuotaError(w, r, http.StatusBadRequest, err.Error()) + return + } + req.QuotaUnit = normalizedUnit + + quotaBytes, err := convertQuotaToBytes(req.QuotaSize, normalizedUnit) + if err != nil { + writeQuotaError(w, r, http.StatusBadRequest, err.Error()) + return + } + + var quota int64 + switch { + case req.QuotaEnabled && quotaBytes > 0: + quota = quotaBytes + case !req.QuotaEnabled && quotaBytes > 0: + quota = -quotaBytes + default: + quota = 0 + } + + err = s3a.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error { + lookupResp, err := client.LookupDirectoryEntry(r.Context(), &filer_pb.LookupDirectoryEntryRequest{ + Directory: s3a.option.BucketsPath, + Name: bucket, + }) + if err != nil { + if errors.Is(err, filer_pb.ErrNotFound) { + return filer_pb.ErrNotFound + } + return fmt.Errorf("failed to look up bucket: %w", err) + } + bucketEntry := lookupResp.Entry + bucketEntry.Quota = quota + + _, err = client.UpdateEntry(r.Context(), &filer_pb.UpdateEntryRequest{ + Directory: s3a.option.BucketsPath, + Entry: bucketEntry, + }) + if err != nil { + return fmt.Errorf("failed to update bucket quota: %w", err) + } + + if quota <= 0 { + if _, err := filer.ClearBucketReadOnly(r.Context(), client, s3a.option.BucketsPath, bucket); err != nil { + return fmt.Errorf("failed to clear bucket read-only flag: %w", err) + } + } + return nil + }) + if err != nil { + if errors.Is(err, filer_pb.ErrNotFound) { + s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchBucket) + return + } + glog.Errorf("PutBucketQuotaHandler %s: %v", bucket, err) + s3err.WriteErrorResponse(w, r, s3err.ErrInternalError) + return + } + + w.WriteHeader(http.StatusNoContent) +} + +// GetBucketQuotaHandler handles GET /{bucket}?seaweedfs-quota. +// +// Returns the current quota configuration for the bucket as JSON. +func (s3a *S3ApiServer) GetBucketQuotaHandler(w http.ResponseWriter, r *http.Request) { + bucket, _ := s3_constants.GetBucketAndObject(r) + glog.V(3).Infof("GetBucketQuotaHandler %s", bucket) + + if bucket == "" { + s3err.WriteErrorResponse(w, r, s3err.ErrInvalidBucketName) + return + } + + entry, err := s3a.getBucketEntry(bucket) + if err != nil { + s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchBucket) + return + } + + // Return the absolute quota magnitude as quota_size; the sign only + // encodes enabled/disabled state internally. This makes the response + // round-trippable: a client can send the same JSON back via PUT without + // the negative sentinel being interpreted as zero. + quotaSize := entry.Quota + if quotaSize < 0 { + quotaSize = -quotaSize + } + resp := bucketQuotaResponse{ + QuotaSize: quotaSize, + QuotaUnit: "B", + QuotaEnabled: entry.Quota > 0, + } + + w.Header().Set("Content-Type", "application/json") + if err := json.NewEncoder(w).Encode(resp); err != nil { + glog.Errorf("GetBucketQuotaHandler %s: failed to encode response: %v", bucket, err) + } +} + +// writeQuotaError writes a JSON error response for quota operations. +func writeQuotaError(w http.ResponseWriter, r *http.Request, statusCode int, message string) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(statusCode) + if err := json.NewEncoder(w).Encode(map[string]string{"error": message}); err != nil { + glog.V(1).Infof("failed to write quota error response: %v", err) + } +} + +// normalizeQuotaUnit normalizes the quota unit string to a canonical uppercase form. +func normalizeQuotaUnit(unit string) (string, error) { + switch unit { + case "", "B", "b": + return "B", nil + case "KB", "kb": + return "KB", nil + case "MB", "mb": + return "MB", nil + case "GB", "gb": + return "GB", nil + case "TB", "tb": + return "TB", nil + default: + return "", fmt.Errorf("unsupported quota_unit %q (supported: B, KB, MB, GB, TB)", unit) + } +} + +// convertQuotaToBytes converts a quota size + unit to bytes. +// Returns an error if the result would overflow int64. +func convertQuotaToBytes(size int64, unit string) (int64, error) { + if size <= 0 { + return 0, nil + } + var multiplier int64 + switch unit { + case "TB": + multiplier = 1024 * 1024 * 1024 * 1024 + case "GB": + multiplier = 1024 * 1024 * 1024 + case "MB": + multiplier = 1024 * 1024 + case "KB": + multiplier = 1024 + case "B": + multiplier = 1 + default: + return 0, fmt.Errorf("unsupported quota_unit %q", unit) + } + if multiplier > 0 && size > math.MaxInt64/multiplier { + return 0, fmt.Errorf("quota_size %d %s overflows maximum bytes (%d)", size, unit, int64(math.MaxInt64)) + } + return size * multiplier, nil +} diff --git a/weed/s3api/s3api_bucket_quota_handlers_test.go b/weed/s3api/s3api_bucket_quota_handlers_test.go new file mode 100644 index 000000000..d15c540a5 --- /dev/null +++ b/weed/s3api/s3api_bucket_quota_handlers_test.go @@ -0,0 +1,156 @@ +package s3api + +import ( + "math" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/gorilla/mux" +) + +func TestNormalizeQuotaUnit(t *testing.T) { + tests := []struct { + input string + want string + err bool + }{ + {"", "B", false}, + {"B", "B", false}, + {"b", "B", false}, + {"KB", "KB", false}, + {"kb", "KB", false}, + {"MB", "MB", false}, + {"mb", "MB", false}, + {"GB", "GB", false}, + {"gb", "GB", false}, + {"TB", "TB", false}, + {"tb", "TB", false}, + {"PB", "", true}, + {"invalid", "", true}, + } + for _, tc := range tests { + t.Run(tc.input, func(t *testing.T) { + got, err := normalizeQuotaUnit(tc.input) + if tc.err && err == nil { + t.Errorf("expected error for %q, got nil", tc.input) + } + if !tc.err && err != nil { + t.Errorf("unexpected error for %q: %v", tc.input, err) + } + if got != tc.want { + t.Errorf("normalizeQuotaUnit(%q) = %q, want %q", tc.input, got, tc.want) + } + }) + } +} + +func TestConvertQuotaToBytes(t *testing.T) { + tests := []struct { + size int64 + unit string + want int64 + wantErr bool + }{ + {0, "B", 0, false}, + {0, "GB", 0, false}, + {1024, "B", 1024, false}, + {1, "KB", 1024, false}, + {1, "MB", 1024 * 1024, false}, + {1, "GB", 1024 * 1024 * 1024, false}, + {1, "TB", 1024 * 1024 * 1024 * 1024, false}, + {2, "GB", 2 * 1024 * 1024 * 1024, false}, + {-1, "GB", 0, false}, + // Overflow: 8388608 TB = 2^23 * 2^40 = 2^63 which overflows int64 + {8388608, "TB", 0, true}, + // MaxInt64 KB overflows + {math.MaxInt64, "KB", 0, true}, + // MaxInt64 B does not overflow + {math.MaxInt64, "B", math.MaxInt64, false}, + } + for _, tc := range tests { + t.Run(tc.unit, func(t *testing.T) { + got, err := convertQuotaToBytes(tc.size, tc.unit) + if tc.wantErr && err == nil { + t.Errorf("expected error for %d %s, got %d", tc.size, tc.unit, got) + } + if !tc.wantErr && err != nil { + t.Errorf("unexpected error for %d %s: %v", tc.size, tc.unit, err) + } + if got != tc.want { + t.Errorf("convertQuotaToBytes(%d, %q) = %d, want %d", tc.size, tc.unit, got, tc.want) + } + }) + } +} + +func TestPutBucketQuotaHandler_InvalidBody(t *testing.T) { + s3a := &S3ApiServer{} + req := httptest.NewRequest(http.MethodPut, "/test-bucket?seaweedfs-quota", strings.NewReader("not json")) + req = mux.SetURLVars(req, map[string]string{"bucket": "test-bucket"}) + rr := httptest.NewRecorder() + s3a.PutBucketQuotaHandler(rr, req) + if rr.Code != http.StatusBadRequest { + t.Errorf("expected 400 for malformed body, got %d", rr.Code) + } +} + +func TestPutBucketQuotaHandler_TrailingData(t *testing.T) { + s3a := &S3ApiServer{} + body := `{"quota_size":100,"quota_unit":"GB","quota_enabled":true} garbage` + req := httptest.NewRequest(http.MethodPut, "/test-bucket?seaweedfs-quota", strings.NewReader(body)) + req = mux.SetURLVars(req, map[string]string{"bucket": "test-bucket"}) + rr := httptest.NewRecorder() + s3a.PutBucketQuotaHandler(rr, req) + if rr.Code != http.StatusBadRequest { + t.Errorf("expected 400 for trailing data, got %d: %s", rr.Code, rr.Body.String()) + } +} + +func TestPutBucketQuotaHandler_EnabledWithZeroSize(t *testing.T) { + s3a := &S3ApiServer{} + body := `{"quota_size":0,"quota_unit":"GB","quota_enabled":true}` + req := httptest.NewRequest(http.MethodPut, "/test-bucket?seaweedfs-quota", strings.NewReader(body)) + req = mux.SetURLVars(req, map[string]string{"bucket": "test-bucket"}) + rr := httptest.NewRecorder() + s3a.PutBucketQuotaHandler(rr, req) + if rr.Code != http.StatusBadRequest { + t.Errorf("expected 400 for enabled quota with size=0, got %d", rr.Code) + } +} + +func TestPutBucketQuotaHandler_InvalidUnit(t *testing.T) { + s3a := &S3ApiServer{} + body := `{"quota_size":100,"quota_unit":"PB","quota_enabled":true}` + req := httptest.NewRequest(http.MethodPut, "/test-bucket?seaweedfs-quota", strings.NewReader(body)) + req = mux.SetURLVars(req, map[string]string{"bucket": "test-bucket"}) + rr := httptest.NewRecorder() + s3a.PutBucketQuotaHandler(rr, req) + if rr.Code != http.StatusBadRequest { + t.Errorf("expected 400 for invalid unit, got %d", rr.Code) + } +} + +func TestPutBucketQuotaHandler_NoBucket(t *testing.T) { + s3a := &S3ApiServer{} + body := `{"quota_size":100,"quota_unit":"GB","quota_enabled":true}` + req := httptest.NewRequest(http.MethodPut, "/?seaweedfs-quota", strings.NewReader(body)) + req = mux.SetURLVars(req, map[string]string{"bucket": ""}) + rr := httptest.NewRecorder() + s3a.PutBucketQuotaHandler(rr, req) + if rr.Code != http.StatusBadRequest { + t.Errorf("expected 400 for missing bucket, got %d", rr.Code) + } +} + +func TestGetBucketQuotaHandler_NoBucket(t *testing.T) { + s3a := &S3ApiServer{} + req := httptest.NewRequest(http.MethodGet, "/?seaweedfs-quota", nil) + req = mux.SetURLVars(req, map[string]string{"bucket": ""}) + rr := httptest.NewRecorder() + s3a.GetBucketQuotaHandler(rr, req) + if rr.Code != http.StatusBadRequest { + t.Errorf("expected 400 for missing bucket, got %d", rr.Code) + } +} diff --git a/weed/s3api/s3api_server.go b/weed/s3api/s3api_server.go index 027f3068b..e19210561 100644 --- a/weed/s3api/s3api_server.go +++ b/weed/s3api/s3api_server.go @@ -994,6 +994,13 @@ func (s3a *S3ApiServer) registerRouter(router *mux.Router) { //DeleteBucketOwnershipControls bucket.Methods(http.MethodDelete).HandlerFunc(track(s3a.iam.Auth(s3a.DeleteBucketOwnershipControls, ACTION_ADMIN), "DELETE")).Queries("ownershipControls", "") + // SeaweedFS extension: bucket quota subresource + // PUT /{bucket}?seaweedfs-quota — set bucket quota (s3:PutBucketQuota) + // GET /{bucket}?seaweedfs-quota — get bucket quota (s3:GetBucketQuota) + // Authenticated via SigV4, authorized via dedicated IAM permissions. + bucket.Methods(http.MethodPut).HandlerFunc(track(s3a.iam.Auth(s3a.cb.Limit(s3a.PutBucketQuotaHandler, ACTION_PUT_BUCKET_QUOTA)), "PUT")).Queries("seaweedfs-quota", "") + bucket.Methods(http.MethodGet).HandlerFunc(track(s3a.iam.Auth(s3a.cb.Limit(s3a.GetBucketQuotaHandler, ACTION_GET_BUCKET_QUOTA)), "GET")).Queries("seaweedfs-quota", "") + // raw buckets // PostPolicy