weed-vector: gRPC TLS to the masters and for the gateway (#12368)

* weed-vector: read the cluster's security.toml

The engine finds security.toml where weed processes do (., ~/.seaweedfs,
/usr/local/etc/seaweedfs, /etc/seaweedfs) with WEED_* environment
variables over it, and builds gRPC client and server credentials by the
rules of security.LoadClientTLS and LoadServerTLS: a dedicated client
certificate is preferred, both halves of a pair must be set, and grpc.ca
is required once either is.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: gRPC TLS to the masters and from the gateway

The masters are dialed with [grpc.client]'s credentials when the cluster
has them, the master's certificate checked against the host dialed or
--tls-server-name. The engine's own listener takes its identity from
[grpc.vector] and then requires a client certificate chaining to grpc.ca,
which is how the gateway, dialing with [grpc.client], reaches it. The
rustls provider is pinned before the first TLS use, as the other binaries
linking both aws-lc-rs and ring do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* scaffold: document [grpc.vector]

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: put every gRPC hop under mutual TLS in the hardened leg

VECTOR_TLS=1 now also writes [grpc.master], [grpc.volume], [grpc.filer],
[grpc.s3], [grpc.client] and [grpc.vector], and probes the master's and
the engine's gRPC ports: plaintext HTTP/2 and a client without a
certificate are refused, the client certificate gets a SETTINGS frame.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: give the test volume server 64 volume slots

Every collection grows 7 volumes and test_vector_bucket_encryption opens
two buckets while the previous test's bucket is still being torn down, so
16 slots were marginal: the Go legs failed CreateIndex now and then with
"failed to find writable volumes" and master.log showing "create 7
volume, created 4: Not enough data nodes found!".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: reload the listener's certificates and enforce allowed_commonNames

The listener now runs the way weed servers run theirs. The certificate, key
and CA files are re-read every WEED_TLS_CERT_REFRESH_INTERVAL (weed's
default of 5h) and on SIGHUP, and new handshakes use the changed files
without a restart; a file that no longer parses is logged and the working
configuration stays. [grpc.vector] allowed_commonNames and
grpc.allowed_wildcard_domain restrict which client certificates are served,
checked against the subject common name as security.Authenticator and the
Rust volume server's CommonNameVerifier do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* scaffold: allowed_commonNames and the refresh cadence for [grpc.vector]

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the engine's allowlist keeps an outsider certificate out

The hardened leg sets [grpc.vector] allowed_commonNames = "localhost" and
probes the engine with a second certificate the same CA signed for
CN=outsider: the handshake must be refused, while the gateway's certificate
still gets an HTTP/2 SETTINGS frame.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: take the master's TLS name from the parsed endpoint

A master given without a port (https://master.example) had the whole URL
checked against its certificate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: bound the master TLS handshake

connect_timeout covers only the TCP connect, so a master that accepted the
connection and never answered the handshake held up the other masters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: look up security.toml keys the way viper does

An empty WEED_* variable no longer hides the file's value, and keys match
regardless of case, as for weed processes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: refuse master allowlists it cannot enforce

weed checks a dialed server against allowed_server_commonNames and
allowed_server_wildcard_domain on top of grpc.ca. The engine's master dial
accepted every certificate the CA signed even when one was set; it now
fails at startup and names the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: read the whole frame header in the gRPC TLS probe

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: probe the volume, filer and S3 gRPC listeners too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read the master dial credentials only when given masters

A standalone engine never dials with grpc.client, so its settings no longer
stop it from starting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: run each listener TLS handshake on its own task

The accept loop awaited every handshake before accepting the next
connection, so a peer that connected and sent nothing stopped every
gateway connection after it. Handshakes now run concurrently, each under a
10s timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read the master dial's TLS files at every dial

The client certificate, key and CA were read once at startup, so after the
files rotated a reconnect still presented the old pair. Each dial now reads
them, as the listener re-reads its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read every unit Go durations accept in the refresh interval

WEED_TLS_CERT_REFRESH_INTERVAL values in ns, us or a leading + fell back
to five hours in the engine while weed servers honored them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: fall back on a refresh interval too long to hold

An oversized WEED_TLS_CERT_REFRESH_INTERVAL panicked the engine at
startup; weed servers reject it and keep the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: refuse allowed_commonNames without the listener's certificate

With [grpc.vector] allowed_commonNames set but no cert and key, the
listener served plaintext and checked no client at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read the master dial's credentials inside its connect timeout

The CA, certificate and key were read synchronously before the timeout
started, so a stalled credential path held up every other master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: let a set key win over an empty spelling of it

With allowed_server_commonNames = "" beside allowed_server_commonnames
= "master", the lookup stopped at the empty exact match and the
allowlist went unnoticed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Chris LuandClaude Opus 5.5 authored and GitHub committed 2026-11-03 11:42:49 -08:00
1 parent 2ec0623073
commit eeb8aefe70
15 files changed
+1484 -33

No files matched your search

@@ -7,9 +7,10 @@ use std::time::Duration;
use anyhow::{Context, Result};
use tokio::time::sleep;
use tonic::transport::Endpoint;
use tonic::transport::{Certificate, Channel, ClientTlsConfig, Endpoint, Identity};
use crate::master_pb::{KeepConnectedRequest, seaweed_client::SeaweedClient};
use crate::security::GrpcTls;
fn registration(client_addr: &str) -> KeepConnectedRequest {
KeepConnectedRequest {
@@ -54,6 +55,32 @@ pub fn grpc_addr(master: &str) -> Result<String> {
Ok(format!("{scheme}://{rest}"))
}
/// The master's gRPC endpoint, over TLS with `grpc.client`'s credentials when
/// the cluster has them; the master's certificate is checked against the host
/// dialed unless the credentials name another.
pub async fn master_endpoint(master: &str, tls: Option<&GrpcTls>) -> Result<Endpoint> {
let addr = grpc_addr(master)?;
let Some(tls) = tls else {
return Ok(Endpoint::new(addr)?);
};
weed_lance_worker::tls::install_default_crypto_provider();
let endpoint = Endpoint::new(addr.replacen("http://", "https://", 1))?;
let host = dialed_host(&endpoint).with_context(|| format!("master {master} has no host"))?;
let (ca, cert, key) = tls.pems().await?;
let config = ClientTlsConfig::new()
.ca_certificate(Certificate::from_pem(ca))
.identity(Identity::from_pem(cert, key))
.domain_name(tls.server_name.clone().unwrap_or(host));
Ok(endpoint
.tls_config(config)
.with_context(|| format!("TLS for master {master}"))?)
}
fn dialed_host(endpoint: &Endpoint) -> Option<String> {
let host = endpoint.uri().host()?;
Some(host.trim_matches(|c| c == '[' || c == ']').to_string())
}
// `host:http.grpc` normalizes to `host:http` so a leader hint naming the
// dialed master does not read as a redirect.
fn http_addr(master: &str) -> String {
@@ -68,11 +95,12 @@ fn http_addr(master: &str) -> String {
/// Retry forever: try each master until a stream opens, hold it until it
/// errors, then rediscover — same shape as wdclient.KeepConnectedToMaster.
pub async fn keep_connected(masters: Vec<String>, client_addr: String) {
pub async fn keep_connected(masters: Vec<String>, client_addr: String, tls: Option<GrpcTls>) {
let client_addr = client_addr.as_str();
let tls = tls.as_ref();
loop {
try_all_masters(&masters, |master| async move {
stream_to(&master, client_addr).await
stream_to(&master, client_addr, tls).await
})
.await;
sleep(Duration::from_secs(3)).await;
@@ -108,16 +136,25 @@ where
}
}
/// Bounds the TCP connect and then the master's first answer, the 5s wdclient
/// gives a master call; a black-holed or silent master otherwise holds up the
/// other masters.
/// Bounds the credential reads and the connect, TLS handshake included, and
/// then the master's first answer, the 5s wdclient gives a master call; a
/// black-holed or silent master otherwise holds up the other masters.
const CONNECT_TIMEOUT: Duration = Duration::from_secs(5);
async fn stream_to(master: &str, client_addr: &str) -> Result<Option<String>> {
let channel = Endpoint::new(grpc_addr(master)?)?
.connect_timeout(CONNECT_TIMEOUT)
.connect()
.await?;
async fn connect(endpoint: impl Future<Output = Result<Endpoint>>) -> Result<Channel> {
tokio::time::timeout(CONNECT_TIMEOUT, async {
Ok(endpoint.await?.connect().await?)
})
.await
.context("master did not complete the connection")?
}
async fn stream_to(
master: &str,
client_addr: &str,
tls: Option<&GrpcTls>,
) -> Result<Option<String>> {
let channel = connect(master_endpoint(master, tls)).await?;
// A master answers at once, a follower with its leader hint.
let answer_by = tokio::time::Instant::now() + CONNECT_TIMEOUT;
let mut client = SeaweedClient::new(channel);
@@ -215,7 +252,7 @@ mod tests {
#[tokio::test]
async fn dial_to_a_black_holed_master_is_bounded() {
// TEST-NET-1 (RFC 5737) is reserved, so nothing answers the SYN.
let dial = stream_to("192.0.2.1:9333", "127.0.0.1:8080");
let dial = stream_to("192.0.2.1:9333", "127.0.0.1:8080", None);
let result = tokio::time::timeout(CONNECT_TIMEOUT * 3, dial).await;
assert!(matches!(result, Ok(Err(_))), "dial was not bounded");
}
@@ -229,13 +266,45 @@ mod tests {
let _ = tokio::io::copy(&mut conn, &mut tokio::io::sink()).await;
});
let master = format!("127.0.0.1:9333.{port}");
let open = stream_to(&master, "127.0.0.1:8080");
let open = stream_to(&master, "127.0.0.1:8080", None);
match tokio::time::timeout(CONNECT_TIMEOUT * 3, open).await {
Ok(Err(e)) => assert!(format!("{e:#}").contains("did not answer"), "{e:#}"),
other => panic!("open was not bounded: {other:?}"),
}
}
#[tokio::test]
async fn a_master_that_stalls_the_tls_handshake_is_bounded() {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let port = listener.local_addr().unwrap().port();
tokio::spawn(async move {
let (mut conn, _) = listener.accept().await.unwrap();
let _ = tokio::io::copy(&mut conn, &mut tokio::io::sink()).await;
});
weed_lance_worker::tls::install_default_crypto_provider();
let endpoint = Endpoint::new(format!("https://127.0.0.1:{port}"))
.unwrap()
.tls_config(ClientTlsConfig::new().domain_name("localhost"))
.unwrap();
match tokio::time::timeout(CONNECT_TIMEOUT * 3, connect(async { Ok(endpoint) })).await {
Ok(Err(e)) => assert!(format!("{e:#}").contains("did not complete"), "{e:#}"),
other => panic!("handshake was not bounded: {other:?}"),
}
}
#[test]
fn the_tls_name_is_the_dialed_host() {
for (master, host) in [
("m1:9333", "m1"),
("https://master.example", "master.example"),
("https://master.example:9333", "master.example"),
("[::1]:9333", "::1"),
] {
let endpoint = Endpoint::new(grpc_addr(master).unwrap()).unwrap();
assert_eq!(dialed_host(&endpoint).as_deref(), Some(host), "{master}");
}
}
#[test]
fn grpc_addr_refuses_a_port_past_65535() {
assert_eq!(grpc_addr("m1:9333").unwrap(), "http://m1:19333");