From 2d8207abe4bfe85543b3cdcc5e4a47720c22d405 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Tue, 21 Apr 2026 15:27:19 -0700 Subject: [PATCH] Security: document certificate rotation without restarts (k8s cert-manager) Covers which surfaces hot-reload (gRPC mTLS, HTTPS master/volume/filer/s3/ webdav/admin after PR #9181) and which do not (HTTPS client). Includes a cert-manager example and the refresh-window / in-flight caveats. --- Security-Configuration.md | 69 +++++++++++++++++++++++++++++++++++++++ Security-Overview.md | 2 ++ 2 files changed, 71 insertions(+) diff --git a/Security-Configuration.md b/Security-Configuration.md index d71a729..074b9dd 100644 --- a/Security-Configuration.md +++ b/Security-Configuration.md @@ -244,6 +244,75 @@ ca = "/etc/seaweedfs/certs/ca.crt" For the full list of environment variable conventions, see [[Environment Variables]]. +## Certificate Rotation Without Restarts (k8s cert-manager) + +SeaweedFS re-reads certificate and key files from disk periodically, so rotating certs (e.g. when cert-manager renews a `Certificate` resource and updates the mounted `Secret`) does **not** require a process restart. + +**Scope:** + +| Surface | Rotates without restart | Refresh window | +|---|---|---| +| gRPC mTLS (`[grpc.*]`) — both server and client | Yes | ~5h | +| HTTPS server (`[https.master]`, `[https.volume]`, `[https.filer]`, `[https.s3]`, `[https.admin]`, WebDav) | Yes | ~5h | +| HTTPS client (`[https.client]`) | Not yet — loaded once at startup | | + +New TLS handshakes pick up the rotated cert. In-flight connections keep the cert they were established with, which is normal TLS behavior. + +**How it works.** A `GetCertificate` callback is wired into every server-side `tls.Config`. Under the hood it's backed by gRPC's `pemfile.NewProvider`, which stats the cert/key files on its refresh tick and re-parses them only when `mtime` or contents change. The refresh interval is `security.CredRefreshingInterval` (currently 5 hours). To shorten it, fork and change that constant — a flag for this is a reasonable future addition. + +**Requirements for hot reload to work:** + +1. **The file paths must stay stable.** cert-manager → k8s Secret → volume mount does this by default: the file paths (e.g. `/etc/seaweedfs/tls/tls.crt`) are unchanged; only the symlinked target under `..data` flips. Point `cert` / `key` in `security.toml` at those stable paths. +2. **Both cert and key must update atomically.** cert-manager's Secret update is atomic from the pod's perspective (kubelet swaps the `..data` symlink), so the pemfile poll never sees a half-rotated pair. If you manage certs manually, write to a temp file and `mv` into place — do not edit in-place. +3. **The refresh window is up to ~5 hours.** If you need tighter bounds (short-lived certs from Vault, etc.), you currently need a rolling restart or a custom build. + +**cert-manager example (HTTPS filer).** Mount the issued Secret at a known path and reference those files in `security.toml`: + +```yaml +# Certificate resource +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: seaweedfs-filer-tls +spec: + secretName: seaweedfs-filer-tls + duration: 2160h # 90d + renewBefore: 360h # 15d + issuerRef: + name: my-issuer + kind: ClusterIssuer + dnsNames: + - filer.seaweedfs.svc.cluster.local +``` + +```yaml +# Pod spec excerpt +volumes: + - name: filer-tls + secret: + secretName: seaweedfs-filer-tls +containers: + - name: filer + volumeMounts: + - name: filer-tls + mountPath: /etc/seaweedfs/tls + readOnly: true +``` + +```toml +# security.toml +[https.filer] +cert = "/etc/seaweedfs/tls/tls.crt" +key = "/etc/seaweedfs/tls/tls.key" +ca = "/etc/seaweedfs/tls/ca.crt" +``` + +When cert-manager renews the Secret, the next pemfile refresh tick (within ~5h) picks up the new material. You do **not** need a reloader sidecar, a `kubectl rollout restart`, or a SIGHUP handler. + +**Logs to watch for.** A successful rotation is silent on the happy path. If the new cert/key pair fails to parse (e.g. mismatched files), you'll see `pemfile.NewProvider` / `KeyMaterial` errors in the server's logs — existing connections continue with the old cert, so there's no outage, but new handshakes will fail until the files are fixed. + +**HTTPS client (`[https.client]`) caveat.** Client-side certs are currently loaded once at startup. If cert-manager rotates a client cert under a long-lived component (e.g. a mount pod, filer.copy job, a SeaweedFS filer acting as an HTTPS client of a volume server), restart that component after renewal. Server-side rotation in the cluster works as above regardless. + --- The following command is what I used to generate the private key and certificate files, using https://github.com/square/certstrap. To compile this tool, you can run `go get github.com/square/certstrap` - or alternatively `brew install certstrap` if you are on Mac OS and use [Homebrew](https://brew.sh). diff --git a/Security-Overview.md b/Security-Overview.md index 932f612..20d2303 100644 --- a/Security-Overview.md +++ b/Security-Overview.md @@ -60,6 +60,8 @@ volume | http read | unprotected, but url is not guessable See [[Security Configuration]] +> **Rotating certificates without restarts?** SeaweedFS re-reads cert/key files from disk and picks up rotated material (e.g. from Kubernetes cert-manager) without a process restart on both gRPC mTLS and HTTPS servers. See [[Security Configuration#Certificate Rotation Without Restarts (k8s cert-manager)]]. + Servers in SeaweedFS usually support 2 kinds of operations: gRPC and REST. # Securing gRPC operations