document credential vending, view rename, and ref-aware snapshot expiry

The catalog page gained a credential vending section: the delegation
header, the role flag that turns it on, and what the vended session is
scoped to. The API matrix marks ReportMetrics supported, adds RenameView
and the LoadTable snapshots parameter, and describes what concurrent
commits now do.

Maintenance: a snapshot a branch or tag points at is never expired, and a
branch's own retention governs the ancestors behind its head.

Also corrects the naming charset, which has allowed hyphens since #10093.
Chris Lu
2026-08-16 13:00:07 -07:00
parent 5febe6078f
commit 400fb54280
3 changed files with 65 additions and 9 deletions
+6 -4
@@ -34,14 +34,14 @@ The tables below list the Iceberg REST Catalog operations and their support stat
|---|---|---|---|
| ListTables | `GET /v1/{prefix}/namespaces/{ns}/tables` | Yes | |
| CreateTable | `POST /v1/{prefix}/namespaces/{ns}/tables` | Yes | Includes stage-create |
| LoadTable | `GET /v1/{prefix}/namespaces/{ns}/tables/{table}` | Yes | Vends S3 FileIO config so clients read data files directly |
| LoadTable | `GET /v1/{prefix}/namespaces/{ns}/tables/{table}` | Yes | Vends S3 FileIO config so clients read data files directly; `?snapshots=refs` returns only the snapshots refs point at |
| TableExists | `HEAD /v1/{prefix}/namespaces/{ns}/tables/{table}` | Yes | |
| UpdateTable | `POST /v1/{prefix}/namespaces/{ns}/tables/{table}` | Yes | Commit with assert-* requirements |
| DropTable | `DELETE /v1/{prefix}/namespaces/{ns}/tables/{table}` | Yes | |
| RegisterTable | `POST /v1/{prefix}/namespaces/{ns}/register` | Yes | Registers an existing `metadata.json` location |
| RenameTable | `POST /v1/{prefix}/tables/rename` | Yes | Catalog-only; data and metadata files do not move |
| ReportMetrics | `POST /v1/{prefix}/namespaces/{ns}/tables/{table}/metrics` | No | Scan/commit metrics are not collected |
| LoadCredentials | `GET /v1/{prefix}/namespaces/{ns}/tables/{table}/credentials` | No | Use the vended FileIO config / S3 keys |
| ReportMetrics | `POST /v1/{prefix}/namespaces/{ns}/tables/{table}/metrics` | Yes | Accepted and discarded; the catalog keeps no metrics store |
| LoadCredentials | `GET /v1/{prefix}/namespaces/{ns}/tables/{table}/credentials` | No | Credentials come back on LoadTable instead, see [[S3 Tables Security]] |
| PlanTableScan / FetchScanTasks | `.../tables/{table}/plan` | No | Server-side scan planning |
## View
@@ -54,6 +54,7 @@ The tables below list the Iceberg REST Catalog operations and their support stat
| ViewExists | `HEAD /v1/{prefix}/namespaces/{ns}/views/{view}` | Yes | |
| ReplaceView | `POST /v1/{prefix}/namespaces/{ns}/views/{view}` | Yes | Commit with assert-view-uuid requirements |
| DropView | `DELETE /v1/{prefix}/namespaces/{ns}/views/{view}` | Yes | |
| RenameView | `POST /v1/{prefix}/views/rename` | Yes | Catalog-only, like RenameTable |
## Transaction
@@ -63,9 +64,10 @@ The tables below list the Iceberg REST Catalog operations and their support stat
# Notes
- **Naming**: namespace and table/view names use the S3 Tables charset — lowercase `a-z`, `0-9`, and `_`. Names with other characters (hyphens, uppercase) are rejected with `400 BadRequestException`.
- **Naming**: namespace and table/view names use the S3 Tables charset — lowercase `a-z`, `0-9`, `_` and `-`, starting and ending alphanumeric. Names with other characters (uppercase, dots) are rejected with `400 BadRequestException`.
- **Table and view names share a namespace**: a name is unique across both within a namespace.
- **Metadata storage**: the catalog entry is a pointer; the Iceberg `metadata.json` is an object at `s3://{bucket}/{ns}/{table}/metadata/vN.metadata.json`. A commit writes a new metadata file and flips the pointer.
- **Concurrent commits**: the metadata file is written exclusively and the pointer is flipped conditionally, so two engines committing from the same base cannot lose each other's snapshot. The loser retries against the winner's metadata and, out of attempts, gets `409 CommitFailedException` — which Iceberg clients handle by re-reading and retrying. A commit that finds its `vN.metadata.json` name taken stages under `vN-{uuid}.metadata.json`.
# See Also
+29 -5
@@ -158,11 +158,35 @@ Removes old snapshots from table metadata and deletes files that are no longer r
1. Sort snapshots by timestamp (newest first)
2. The current snapshot is **always kept**
3. Keep the newest `max_snapshots_to_keep` snapshots regardless of age
4. Among the rest, expire those older than `snapshot_retention_hours`
5. Snapshots within the retention window are kept even if they exceed the count
6. Commit new metadata with expired snapshots removed
7. Delete files exclusively referenced by expired snapshots (best-effort)
3. A snapshot a branch or tag points at is **always kept**, however old it is
4. Keep the newest `max_snapshots_to_keep` snapshots regardless of age
5. Among the rest, expire those older than `snapshot_retention_hours`
6. Snapshots within the retention window are kept even if they exceed the count
7. Commit new metadata with expired snapshots removed
8. Delete files exclusively referenced by expired snapshots (best-effort)
#### Branches and Tags
A named ref pins its snapshot: expiring it would drop the ref with it, and then
delete the files it pointed at, so tags used for time travel and write-audit-publish
branches survive maintenance no matter how far past retention they are.
A branch may also carry its own retention, and those override the worker for the
ancestors behind its head:
| Ref property | Effect |
|---|---|
| `min-snapshots-to-keep` | keeps that many snapshots back from the branch head |
| `max-snapshot-age-ms` | keeps ancestors newer than this age |
A table's own `history.expire.max-snapshot-age-ms` and
`history.expire.min-snapshots-to-keep` properties set the table-wide window and
take precedence over the worker config; the per-ref values above refine it for
one branch.
If a tag or branch appears while a maintenance run is planning, the run is
abandoned and re-planned on the next cycle rather than committing a plan made
before the ref existed.
#### Before / After
+30
@@ -83,6 +83,36 @@ SeaweedFS supports two authentication methods for the Iceberg REST Catalog:
**OAuth2 (DuckDB, Doris)** — Clients exchange S3 credentials for a bearer token via `POST /v1/oauth/tokens` using the `client_credentials` grant type. The S3 access key is used as `client_id` and the secret key as `client_secret`.
### Credential Vending
Engines that expect the catalog to hand out storage credentials, rather than
being configured with S3 keys of their own, send:
```
X-Iceberg-Access-Delegation: vended-credentials
```
Vending is off until a role is named:
```
weed server -s3 -s3.iceberg.credentialRole=arn:aws:iam::role/IcebergTableAccess -s3.iceberg.credentialDurationSeconds=3600
```
The catalog then assumes that role for the caller and returns temporary
credentials in the LoadTable response `config` (`s3.access-key-id`,
`s3.secret-access-key`, `s3.session-token`) and in `storage-credentials`. The
session is scoped by policy to that one table's prefix plus the bucket listing
needed to resolve it, so a credential vended for one table cannot read, write,
or enumerate another.
The role's trust policy decides who may assume it — that is the control point,
and the role needs advanced IAM (`-s3.iam.config`) configured. Requested
durations must be between 900 and 43200 seconds.
Without a role configured, the catalog vends nothing and returns no S3 endpoint
to a client that asked for delegation, so it keeps using the credentials it was
configured with rather than falling back to unsigned requests.
### Authorization (IAM)
Permissions are managed via **S3 Bucket Policies** applied to the Table Bucket.
- You can define granular permissions for `CreateNamespace`, `CreateTable`, `WriteTable`, etc.