diff --git a/Filer-Active-Active-cross-cluster-continuous-synchronization.md b/Filer-Active-Active-cross-cluster-continuous-synchronization.md index 9fa7f7f..62ca7d8 100644 --- a/Filer-Active-Active-cross-cluster-continuous-synchronization.md +++ b/Filer-Active-Active-cross-cluster-continuous-synchronization.md @@ -130,6 +130,78 @@ By default, `filer.sync` will upload files directly to volume servers. This is t However, it uses volume server IP addresses configured for the local cluster. `filer.sync` is usually cross network. These IPs may not be accessible to the `filer.sync` because of network configurations (for example cluster1 and cluster2 are on different hosting providers). In this case, it could be useful to use the `filerProxy` option to make `filer.sync` does all the transfers through the filer. In order to enable this option, `-a.filerProxy` or/and `-b.filerProxy` can be added to the `weed filer.sync` process starting command line. +## Cross-cluster TLS + +`filer.sync` opens two channels to each filer: gRPC (port `18888` by default) for the change-event stream, and HTTP (`8888`) for chunk transfer whenever `-a.filerProxy` / `-b.filerProxy` is used. Both have to be reachable, and both can be secured. + +If the two clusters share a CA, one `security.toml` with `[grpc] ca` plus `[grpc.client] cert`/`key` covers everything. When they don't, give each side its own file: + +``` +weed filer.sync \ + -a=hub.example.com:8888.18888 -a.security=/certs/hub-security.toml \ + -b=edge-filer:8888.18888 -b.security=/certs/edge-security.toml +``` + +Each file supplies the `[grpc.client]` certificate used to dial that cluster, and, when `[https.client] enabled = true`, the certificate for its HTTP leg. The `-a.security` / `-b.security` files are read only for that cluster, so one side can use TLS while the other stays plaintext. + +### Verification is not symmetric + +The two channels do not check certificates the same way: + +- **gRPC** verifies only that the server certificate chains up to `[grpc] ca`. The hostname is *not* checked, so a certificate issued for a different name still works. Constrain who may connect with `allowed_commonNames` or `grpc.allowed_wildcard_domain` on the server side instead. +- **HTTP** is an ordinary Go HTTPS client and does full hostname verification, so the certificate's SAN must match the host you passed in `-a` / `-b`. + +Since `[grpc] ca` takes a PEM bundle, a cluster reached through a proxy that presents its own certificate can be trusted by concatenating both CAs. That also means any CA in the bundle can vouch for any SeaweedFS gRPC server the process dials, so keep public CAs out of it. + +### Behind a Kubernetes Ingress + +An L4 route (TCP passthrough, a `LoadBalancer` Service, or a NodePort) is the better option, because it leaves the filer's own mTLS intact end to end. If all you can create is `Ingress` objects, this works with nginx-ingress. + +Every gRPC method path is `/filer_pb.SeaweedFiler/...`, so one host on 443 serves both channels through two path rules. Annotations, all prefixed `nginx.ingress.kubernetes.io/`: + +```yaml +# path /filer_pb.SeaweedFiler -> filer svc 18888 +backend-protocol: "GRPC" +auth-tls-secret: "/seaweed-ca" +auth-tls-verify-client: "on" +proxy-read-timeout: "3600" # 60s default cuts the metadata stream + +# path / -> filer svc 8888 +backend-protocol: "HTTP" +auth-tls-secret: "/seaweed-ca" +auth-tls-verify-client: "on" +proxy-read-timeout: "3600" +proxy-body-size: "0" # 1m default truncates chunk transfers +``` + +`auth-tls` is a per-location annotation, so it belongs on both rules. nginx performs the client certificate check, and the remote `filer.sync` presents its `[grpc.client]` certificate to satisfy it. + +`backend-protocol: "GRPCS"` does not work here: nginx-ingress emits no `grpc_ssl_*` directives, so `proxy-ssl-secret` cannot supply a client certificate to the backend and the filer rejects the connection with `tlsv13 alert certificate required`. Only `GRPC` (h2c) is available, which makes the hop from nginx to the filer plaintext, so the hub filer's `[grpc.filer] cert`/`key` must be unset. Since every in-cluster caller would otherwise dial that filer with TLS credentials, this effectively moves the authentication boundary from the filer to the ingress. + +The remote side then reaches the cluster with: + +``` +weed filer.sync -a=filer.example.com:443.443 -a.filerProxy -a.security=/certs/a-security.toml ... +``` + +```toml +# a-security.toml +[grpc] +ca = "/certs/ca-bundle.crt" # SeaweedFS CA + the ingress certificate's CA + +[grpc.client] +cert = "/certs/client.crt" +key = "/certs/client.key" + +[https.client] +enabled = true +cert = "/certs/client.crt" +key = "/certs/client.key" +ca = "/certs/ca-bundle.crt" +``` + +The ingress certificate's SAN has to match `filer.example.com`, since the HTTP leg verifies it. + ## Debug log To see all detail of transfers executed by `filer.sync`, options `-a.debug` or/and `-b.debug` can be added to the `weed filer.sync` process starting command line. diff --git a/Security-Configuration.md b/Security-Configuration.md index 1159abc..1f4f417 100644 --- a/Security-Configuration.md +++ b/Security-Configuration.md @@ -71,7 +71,10 @@ expires_after_seconds = 10 # seconds # gRPC mTLS configuration # All gRPC TLS authentications are mutual (mTLS) # The values for ca, cert, and key are paths to the certificate/key files -# The host name is not checked, so the certificate files can be shared +# The host name is not checked, only that the peer certificate chains up to the +# ca below, so the certificate files can be shared. Restrict who may connect with +# allowed_wildcard_domain here or allowed_commonNames per component. +# ca accepts a PEM bundle, so peers issued by different CAs can be trusted at once. [grpc] ca = "" # Set wildcard domain for enable TLS authentication by common names @@ -97,6 +100,11 @@ allowed_commonNames = "" # comma-separated SSL certificate common names cert = "" key = "" allowed_commonNames = "" # comma-separated SSL certificate common names +# Server components also dial their peers, reusing cert/key above by default. +# If your CA issues serverAuth-only certificates, set a separate pair for the +# outgoing direction. Available in every [grpc.] section. +client_cert = "" +client_key = "" # S3 server gRPC options (server-side) # Enables mTLS for incoming gRPC connections to S3 server @@ -118,6 +126,8 @@ allowed_commonNames = "" # comma-separated SSL certificate common names # gRPC client configuration for outgoing gRPC connections # Used by clients (S3, mount, backup, benchmark, filer.copy, filer.replicate, upload, etc.) # when connecting to any gRPC server (master, volume, filer) +# filer.sync can load a separate file per cluster with -a.security / -b.security, see +# https://github.com/seaweedfs/seaweedfs/wiki/Filer-Active-Active-cross-cluster-continuous-synchronization#cross-cluster-tls [grpc.client] cert = "" key = ""