mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-19 10:54:08 +01:00
weed-vector: gRPC TLS to the masters and for the gateway (#12368)
* weed-vector: read the cluster's security.toml The engine finds security.toml where weed processes do (., ~/.seaweedfs, /usr/local/etc/seaweedfs, /etc/seaweedfs) with WEED_* environment variables over it, and builds gRPC client and server credentials by the rules of security.LoadClientTLS and LoadServerTLS: a dedicated client certificate is preferred, both halves of a pair must be set, and grpc.ca is required once either is. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: gRPC TLS to the masters and from the gateway The masters are dialed with [grpc.client]'s credentials when the cluster has them, the master's certificate checked against the host dialed or --tls-server-name. The engine's own listener takes its identity from [grpc.vector] and then requires a client certificate chaining to grpc.ca, which is how the gateway, dialing with [grpc.client], reaches it. The rustls provider is pinned before the first TLS use, as the other binaries linking both aws-lc-rs and ring do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * scaffold: document [grpc.vector] Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: put every gRPC hop under mutual TLS in the hardened leg VECTOR_TLS=1 now also writes [grpc.master], [grpc.volume], [grpc.filer], [grpc.s3], [grpc.client] and [grpc.vector], and probes the master's and the engine's gRPC ports: plaintext HTTP/2 and a client without a certificate are refused, the client certificate gets a SETTINGS frame. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: give the test volume server 64 volume slots Every collection grows 7 volumes and test_vector_bucket_encryption opens two buckets while the previous test's bucket is still being torn down, so 16 slots were marginal: the Go legs failed CreateIndex now and then with "failed to find writable volumes" and master.log showing "create 7 volume, created 4: Not enough data nodes found!". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: reload the listener's certificates and enforce allowed_commonNames The listener now runs the way weed servers run theirs. The certificate, key and CA files are re-read every WEED_TLS_CERT_REFRESH_INTERVAL (weed's default of 5h) and on SIGHUP, and new handshakes use the changed files without a restart; a file that no longer parses is logged and the working configuration stays. [grpc.vector] allowed_commonNames and grpc.allowed_wildcard_domain restrict which client certificates are served, checked against the subject common name as security.Authenticator and the Rust volume server's CommonNameVerifier do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * scaffold: allowed_commonNames and the refresh cadence for [grpc.vector] Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3vectors: the engine's allowlist keeps an outsider certificate out The hardened leg sets [grpc.vector] allowed_commonNames = "localhost" and probes the engine with a second certificate the same CA signed for CN=outsider: the handshake must be refused, while the gateway's certificate still gets an HTTP/2 SETTINGS frame. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: take the master's TLS name from the parsed endpoint A master given without a port (https://master.example) had the whole URL checked against its certificate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: bound the master TLS handshake connect_timeout covers only the TCP connect, so a master that accepted the connection and never answered the handshake held up the other masters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: look up security.toml keys the way viper does An empty WEED_* variable no longer hides the file's value, and keys match regardless of case, as for weed processes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: refuse master allowlists it cannot enforce weed checks a dialed server against allowed_server_commonNames and allowed_server_wildcard_domain on top of grpc.ca. The engine's master dial accepted every certificate the CA signed even when one was set; it now fails at startup and names the key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: read the whole frame header in the gRPC TLS probe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3vectors: probe the volume, filer and S3 gRPC listeners too Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read the master dial credentials only when given masters A standalone engine never dials with grpc.client, so its settings no longer stop it from starting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: run each listener TLS handshake on its own task The accept loop awaited every handshake before accepting the next connection, so a peer that connected and sent nothing stopped every gateway connection after it. Handshakes now run concurrently, each under a 10s timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read the master dial's TLS files at every dial The client certificate, key and CA were read once at startup, so after the files rotated a reconnect still presented the old pair. Each dial now reads them, as the listener re-reads its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read every unit Go durations accept in the refresh interval WEED_TLS_CERT_REFRESH_INTERVAL values in ns, us or a leading + fell back to five hours in the engine while weed servers honored them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: fall back on a refresh interval too long to hold An oversized WEED_TLS_CERT_REFRESH_INTERVAL panicked the engine at startup; weed servers reject it and keep the default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: refuse allowed_commonNames without the listener's certificate With [grpc.vector] allowed_commonNames set but no cert and key, the listener served plaintext and checked no client at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: read the master dial's credentials inside its connect timeout The CA, certificate and key were read synchronously before the timeout started, so a stalled credential path held up every other master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: let a set key win over an empty spelling of it With allowed_server_commonNames = "" beside allowed_server_commonnames = "master", the lookup stopped at the empty exact match and the allowlist went unnoticed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
2ec0623073
commit
eeb8aefe70
15 files changed
+1484
-33
No files matched your search
@@ -7,9 +7,10 @@ use std::time::Duration;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use tokio::time::sleep;
|
||||
use tonic::transport::Endpoint;
|
||||
use tonic::transport::{Certificate, Channel, ClientTlsConfig, Endpoint, Identity};
|
||||
|
||||
use crate::master_pb::{KeepConnectedRequest, seaweed_client::SeaweedClient};
|
||||
use crate::security::GrpcTls;
|
||||
|
||||
fn registration(client_addr: &str) -> KeepConnectedRequest {
|
||||
KeepConnectedRequest {
|
||||
@@ -54,6 +55,32 @@ pub fn grpc_addr(master: &str) -> Result<String> {
|
||||
Ok(format!("{scheme}://{rest}"))
|
||||
}
|
||||
|
||||
/// The master's gRPC endpoint, over TLS with `grpc.client`'s credentials when
|
||||
/// the cluster has them; the master's certificate is checked against the host
|
||||
/// dialed unless the credentials name another.
|
||||
pub async fn master_endpoint(master: &str, tls: Option<&GrpcTls>) -> Result<Endpoint> {
|
||||
let addr = grpc_addr(master)?;
|
||||
let Some(tls) = tls else {
|
||||
return Ok(Endpoint::new(addr)?);
|
||||
};
|
||||
weed_lance_worker::tls::install_default_crypto_provider();
|
||||
let endpoint = Endpoint::new(addr.replacen("http://", "https://", 1))?;
|
||||
let host = dialed_host(&endpoint).with_context(|| format!("master {master} has no host"))?;
|
||||
let (ca, cert, key) = tls.pems().await?;
|
||||
let config = ClientTlsConfig::new()
|
||||
.ca_certificate(Certificate::from_pem(ca))
|
||||
.identity(Identity::from_pem(cert, key))
|
||||
.domain_name(tls.server_name.clone().unwrap_or(host));
|
||||
Ok(endpoint
|
||||
.tls_config(config)
|
||||
.with_context(|| format!("TLS for master {master}"))?)
|
||||
}
|
||||
|
||||
fn dialed_host(endpoint: &Endpoint) -> Option<String> {
|
||||
let host = endpoint.uri().host()?;
|
||||
Some(host.trim_matches(|c| c == '[' || c == ']').to_string())
|
||||
}
|
||||
|
||||
// `host:http.grpc` normalizes to `host:http` so a leader hint naming the
|
||||
// dialed master does not read as a redirect.
|
||||
fn http_addr(master: &str) -> String {
|
||||
@@ -68,11 +95,12 @@ fn http_addr(master: &str) -> String {
|
||||
|
||||
/// Retry forever: try each master until a stream opens, hold it until it
|
||||
/// errors, then rediscover — same shape as wdclient.KeepConnectedToMaster.
|
||||
pub async fn keep_connected(masters: Vec<String>, client_addr: String) {
|
||||
pub async fn keep_connected(masters: Vec<String>, client_addr: String, tls: Option<GrpcTls>) {
|
||||
let client_addr = client_addr.as_str();
|
||||
let tls = tls.as_ref();
|
||||
loop {
|
||||
try_all_masters(&masters, |master| async move {
|
||||
stream_to(&master, client_addr).await
|
||||
stream_to(&master, client_addr, tls).await
|
||||
})
|
||||
.await;
|
||||
sleep(Duration::from_secs(3)).await;
|
||||
@@ -108,16 +136,25 @@ where
|
||||
}
|
||||
}
|
||||
|
||||
/// Bounds the TCP connect and then the master's first answer, the 5s wdclient
|
||||
/// gives a master call; a black-holed or silent master otherwise holds up the
|
||||
/// other masters.
|
||||
/// Bounds the credential reads and the connect, TLS handshake included, and
|
||||
/// then the master's first answer, the 5s wdclient gives a master call; a
|
||||
/// black-holed or silent master otherwise holds up the other masters.
|
||||
const CONNECT_TIMEOUT: Duration = Duration::from_secs(5);
|
||||
|
||||
async fn stream_to(master: &str, client_addr: &str) -> Result<Option<String>> {
|
||||
let channel = Endpoint::new(grpc_addr(master)?)?
|
||||
.connect_timeout(CONNECT_TIMEOUT)
|
||||
.connect()
|
||||
.await?;
|
||||
async fn connect(endpoint: impl Future<Output = Result<Endpoint>>) -> Result<Channel> {
|
||||
tokio::time::timeout(CONNECT_TIMEOUT, async {
|
||||
Ok(endpoint.await?.connect().await?)
|
||||
})
|
||||
.await
|
||||
.context("master did not complete the connection")?
|
||||
}
|
||||
|
||||
async fn stream_to(
|
||||
master: &str,
|
||||
client_addr: &str,
|
||||
tls: Option<&GrpcTls>,
|
||||
) -> Result<Option<String>> {
|
||||
let channel = connect(master_endpoint(master, tls)).await?;
|
||||
// A master answers at once, a follower with its leader hint.
|
||||
let answer_by = tokio::time::Instant::now() + CONNECT_TIMEOUT;
|
||||
let mut client = SeaweedClient::new(channel);
|
||||
@@ -215,7 +252,7 @@ mod tests {
|
||||
#[tokio::test]
|
||||
async fn dial_to_a_black_holed_master_is_bounded() {
|
||||
// TEST-NET-1 (RFC 5737) is reserved, so nothing answers the SYN.
|
||||
let dial = stream_to("192.0.2.1:9333", "127.0.0.1:8080");
|
||||
let dial = stream_to("192.0.2.1:9333", "127.0.0.1:8080", None);
|
||||
let result = tokio::time::timeout(CONNECT_TIMEOUT * 3, dial).await;
|
||||
assert!(matches!(result, Ok(Err(_))), "dial was not bounded");
|
||||
}
|
||||
@@ -229,13 +266,45 @@ mod tests {
|
||||
let _ = tokio::io::copy(&mut conn, &mut tokio::io::sink()).await;
|
||||
});
|
||||
let master = format!("127.0.0.1:9333.{port}");
|
||||
let open = stream_to(&master, "127.0.0.1:8080");
|
||||
let open = stream_to(&master, "127.0.0.1:8080", None);
|
||||
match tokio::time::timeout(CONNECT_TIMEOUT * 3, open).await {
|
||||
Ok(Err(e)) => assert!(format!("{e:#}").contains("did not answer"), "{e:#}"),
|
||||
other => panic!("open was not bounded: {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_master_that_stalls_the_tls_handshake_is_bounded() {
|
||||
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let port = listener.local_addr().unwrap().port();
|
||||
tokio::spawn(async move {
|
||||
let (mut conn, _) = listener.accept().await.unwrap();
|
||||
let _ = tokio::io::copy(&mut conn, &mut tokio::io::sink()).await;
|
||||
});
|
||||
weed_lance_worker::tls::install_default_crypto_provider();
|
||||
let endpoint = Endpoint::new(format!("https://127.0.0.1:{port}"))
|
||||
.unwrap()
|
||||
.tls_config(ClientTlsConfig::new().domain_name("localhost"))
|
||||
.unwrap();
|
||||
match tokio::time::timeout(CONNECT_TIMEOUT * 3, connect(async { Ok(endpoint) })).await {
|
||||
Ok(Err(e)) => assert!(format!("{e:#}").contains("did not complete"), "{e:#}"),
|
||||
other => panic!("handshake was not bounded: {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_tls_name_is_the_dialed_host() {
|
||||
for (master, host) in [
|
||||
("m1:9333", "m1"),
|
||||
("https://master.example", "master.example"),
|
||||
("https://master.example:9333", "master.example"),
|
||||
("[::1]:9333", "::1"),
|
||||
] {
|
||||
let endpoint = Endpoint::new(grpc_addr(master).unwrap()).unwrap();
|
||||
assert_eq!(dialed_host(&endpoint).as_deref(), Some(host), "{master}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn grpc_addr_refuses_a_port_past_65535() {
|
||||
assert_eq!(grpc_addr("m1:9333").unwrap(), "http://m1:19333");
|
||||
|
||||
Reference in new issue
Block a user