weed-vector: gRPC TLS to the masters and for the gateway (#12368)

* weed-vector: read the cluster's security.toml

The engine finds security.toml where weed processes do (., ~/.seaweedfs,
/usr/local/etc/seaweedfs, /etc/seaweedfs) with WEED_* environment
variables over it, and builds gRPC client and server credentials by the
rules of security.LoadClientTLS and LoadServerTLS: a dedicated client
certificate is preferred, both halves of a pair must be set, and grpc.ca
is required once either is.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: gRPC TLS to the masters and from the gateway

The masters are dialed with [grpc.client]'s credentials when the cluster
has them, the master's certificate checked against the host dialed or
--tls-server-name. The engine's own listener takes its identity from
[grpc.vector] and then requires a client certificate chaining to grpc.ca,
which is how the gateway, dialing with [grpc.client], reaches it. The
rustls provider is pinned before the first TLS use, as the other binaries
linking both aws-lc-rs and ring do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* scaffold: document [grpc.vector]

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: put every gRPC hop under mutual TLS in the hardened leg

VECTOR_TLS=1 now also writes [grpc.master], [grpc.volume], [grpc.filer],
[grpc.s3], [grpc.client] and [grpc.vector], and probes the master's and
the engine's gRPC ports: plaintext HTTP/2 and a client without a
certificate are refused, the client certificate gets a SETTINGS frame.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: give the test volume server 64 volume slots

Every collection grows 7 volumes and test_vector_bucket_encryption opens
two buckets while the previous test's bucket is still being torn down, so
16 slots were marginal: the Go legs failed CreateIndex now and then with
"failed to find writable volumes" and master.log showing "create 7
volume, created 4: Not enough data nodes found!".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: reload the listener's certificates and enforce allowed_commonNames

The listener now runs the way weed servers run theirs. The certificate, key
and CA files are re-read every WEED_TLS_CERT_REFRESH_INTERVAL (weed's
default of 5h) and on SIGHUP, and new handshakes use the changed files
without a restart; a file that no longer parses is logged and the working
configuration stays. [grpc.vector] allowed_commonNames and
grpc.allowed_wildcard_domain restrict which client certificates are served,
checked against the subject common name as security.Authenticator and the
Rust volume server's CommonNameVerifier do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* scaffold: allowed_commonNames and the refresh cadence for [grpc.vector]

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the engine's allowlist keeps an outsider certificate out

The hardened leg sets [grpc.vector] allowed_commonNames = "localhost" and
probes the engine with a second certificate the same CA signed for
CN=outsider: the handshake must be refused, while the gateway's certificate
still gets an HTTP/2 SETTINGS frame.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: take the master's TLS name from the parsed endpoint

A master given without a port (https://master.example) had the whole URL
checked against its certificate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: bound the master TLS handshake

connect_timeout covers only the TCP connect, so a master that accepted the
connection and never answered the handshake held up the other masters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: look up security.toml keys the way viper does

An empty WEED_* variable no longer hides the file's value, and keys match
regardless of case, as for weed processes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: refuse master allowlists it cannot enforce

weed checks a dialed server against allowed_server_commonNames and
allowed_server_wildcard_domain on top of grpc.ca. The engine's master dial
accepted every certificate the CA signed even when one was set; it now
fails at startup and names the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: read the whole frame header in the gRPC TLS probe

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: probe the volume, filer and S3 gRPC listeners too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read the master dial credentials only when given masters

A standalone engine never dials with grpc.client, so its settings no longer
stop it from starting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: run each listener TLS handshake on its own task

The accept loop awaited every handshake before accepting the next
connection, so a peer that connected and sent nothing stopped every
gateway connection after it. Handshakes now run concurrently, each under a
10s timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read the master dial's TLS files at every dial

The client certificate, key and CA were read once at startup, so after the
files rotated a reconnect still presented the old pair. Each dial now reads
them, as the listener re-reads its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read every unit Go durations accept in the refresh interval

WEED_TLS_CERT_REFRESH_INTERVAL values in ns, us or a leading + fell back
to five hours in the engine while weed servers honored them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: fall back on a refresh interval too long to hold

An oversized WEED_TLS_CERT_REFRESH_INTERVAL panicked the engine at
startup; weed servers reject it and keep the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: refuse allowed_commonNames without the listener's certificate

With [grpc.vector] allowed_commonNames set but no cert and key, the
listener served plaintext and checked no client at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read the master dial's credentials inside its connect timeout

The CA, certificate and key were read synchronously before the timeout
started, so a stalled credential path held up every other master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: let a set key win over an empty spelling of it

With allowed_server_commonNames = "" beside allowed_server_commonnames
= "master", the lookup stopped at the empty exact match and the
allowlist went unnoticed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

test/s3vectors/conftest.py

@@ -89,8 +89,9 @@ def open_cluster(request, tmp_path_factory):
 
 
 def write_tls(directory):
-    """A CA and one certificate for localhost; volume servers serve HTTPS
-    requiring client certificates, and every client presents the same one."""
+    """A CA and one certificate for localhost. Volume servers serve HTTPS and
+    every gRPC listener (master, volume, filer, gateway, engine) serves TLS,
+    all requiring client certificates, and every client presents the same one."""
     def openssl(*args):
         subprocess.run(["openssl", *args], check=True, capture_output=True, cwd=directory)
     openssl("req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", "ca.key", "-out", "ca.pem",
@@ -101,8 +102,56 @@ def write_tls(directory):
         "subjectAltName=DNS:localhost,IP:127.0.0.1\nextendedKeyUsage=serverAuth,clientAuth\nbasicConstraints=CA:FALSE\n")
     openssl("x509", "-req", "-in", "req.csr", "-CA", "ca.pem", "-CAkey", "ca.key", "-CAcreateserial",
             "-out", "cert.pem", "-days", "2", "-extfile", "ext.cnf")
-    pem = "".join(f'{key} = {json.dumps(str(directory / name), ensure_ascii=False)}\n' for key, name in (("cert", "cert.pem"), ("key", "key.pem"), ("ca", "ca.pem")))
-    (directory / "security.toml").write_text(f"[https.volume]\n{pem}\n[https.client]\nenabled = true\n{pem}")
+    # A second identity the same CA signed, which the engine's allowlist
+    # must keep out.
+    openssl("req", "-newkey", "rsa:2048", "-nodes", "-keyout", "outsider.key", "-out", "outsider.csr", "-subj", "/CN=outsider")
+    openssl("x509", "-req", "-in", "outsider.csr", "-CA", "ca.pem", "-CAkey", "ca.key", "-CAcreateserial",
+            "-out", "outsider.pem", "-days", "2", "-extfile", "ext.cnf")
+    cert, key, ca = (f'{key} = {json.dumps(str(directory / name), ensure_ascii=False)}\n' for key, name in (("cert", "cert.pem"), ("key", "key.pem"), ("ca", "ca.pem")))
+    pem = cert + key + ca
+    grpc = "".join(f"[grpc.{component}]\n{cert}{key}\n" for component in ("master", "volume", "filer", "s3", "client"))
+    grpc += f"[grpc.vector]\n{cert}{key}allowed_commonNames = \"localhost\"\n"
+    (directory / "security.toml").write_text(f"[https.volume]\n{pem}\n[https.client]\nenabled = true\n{pem}\n[grpc]\n{ca}\n{grpc}")
+
+
+def probe_grpc_tls(name, port, directory, outsider=False):
+    """A TLS gRPC listener must refuse plaintext HTTP/2 and a connection
+    without the client certificate, and answer the HTTP/2 preface with it.
+    With `outsider`, a certificate the CA signed for a name outside the
+    listener's allowed_commonNames must be refused too."""
+    preface = b"PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n"
+    with socket.create_connection(("127.0.0.1", port), timeout=5) as sock:
+        sock.sendall(preface)
+        with contextlib.suppress(ConnectionError, TimeoutError):
+            answer = sock.recv(16)
+            # Nothing, or a TLS alert record; a plaintext server sends SETTINGS.
+            assert not answer or answer[0] == 0x15, f"{name} gRPC answered plaintext HTTP/2: {answer!r}"
+    context = ssl.create_default_context(cafile=directory / "ca.pem")
+    context.set_alpn_protocols(["h2"])
+    with contextlib.suppress(ConnectionError, TimeoutError, ssl.SSLError):
+        with socket.create_connection(("127.0.0.1", port), timeout=5) as sock:
+            with context.wrap_socket(sock, server_hostname="localhost") as tls:
+                tls.sendall(preface)
+                assert not tls.recv(16), f"{name} gRPC served a client without a certificate"
+    if outsider:
+        other = ssl.create_default_context(cafile=directory / "ca.pem")
+        other.set_alpn_protocols(["h2"])
+        other.load_cert_chain(directory / "outsider.pem", directory / "outsider.key")
+        with contextlib.suppress(ConnectionError, TimeoutError, ssl.SSLError):
+            with socket.create_connection(("127.0.0.1", port), timeout=5) as sock:
+                with other.wrap_socket(sock, server_hostname="localhost") as tls:
+                    tls.sendall(preface)
+                    assert not tls.recv(16), f"{name} gRPC served a certificate outside allowed_commonNames"
+    context.load_cert_chain(directory / "cert.pem", directory / "key.pem")
+    with socket.create_connection(("127.0.0.1", port), timeout=5) as sock:
+        with context.wrap_socket(sock, server_hostname="localhost") as tls:
+            tls.sendall(preface)
+            header = b""
+            while len(header) < 9 and (chunk := tls.recv(9 - len(header))):
+                header += chunk
+            # The first server frame is SETTINGS (type 4).
+            assert len(header) == 9 and header[3] == 4, \
+                f"{name} gRPC did not answer the HTTP/2 preface over TLS: {header!r}"
 
 
 def read_token(key, fid):
@@ -222,7 +271,7 @@ def run_cluster(param, tmp_path_factory, identities):
         start("volume", prefix + [f"{flag}{key}={value}" for key, value in {
             "master": master_addr, "ip": "127.0.0.1", "ip.bind": "127.0.0.1",
             "port": vol, "port.grpc": vol_grpc, "dir": directory / "volume",
-            "max": 16, "preStopSeconds": 0,
+            "max": 64, "preStopSeconds": 0,
         }.items()])
         wait_for(lambda: listening(vol_grpc), processes, "volume not ready")
         start("filer", [weed, "filer", f"-master={master_addr}", "-ip=127.0.0.1",
@@ -250,6 +299,12 @@ def run_cluster(param, tmp_path_factory, identities):
         start("vector", [engine, "--namespace", f"http://127.0.0.1:{lance}",
                          "--masters", master_addr, "--listen", f"127.0.0.1:{vector}", *credentials], unset)
         wait_for(lambda: listening(vector), processes, "vector engine not ready")
+        if os.environ.get("VECTOR_TLS"):
+            wait_for(lambda: listening(s3_grpc), processes, "s3 gRPC not ready")
+            for name, port in (("master", master_grpc), ("volume", vol_grpc), ("filer", filer_grpc),
+                               ("s3", s3_grpc)):
+                probe_grpc_tls(name, port, directory)
+            probe_grpc_tls("engine", vector, directory, outsider=True)
 
         wait_for(
             lambda: f"Cluster: vector node 127.0.0.1:{vector} added to group ''"