test/s3vectors: a hardened cluster with TLS volume servers, read JWTs and encrypted chunks (#12356)

* test/s3vectors: a hardened cluster, with TLS volume servers, read JWTs and encrypted chunks

Three switches on the test cluster: VECTOR_TLS=1 puts the volume servers
behind HTTPS requiring client certificates and enables https.client for
every process; VECTOR_READ_JWT=1 gives every process a jwt.signing.read.key,
so volume reads need a token; VECTOR_ENCRYPT_VOLUMES=1 starts the gateway
with -encryptVolumeData. The integration matrix gains a hardened leg per
volume server with all three on.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the hardened cluster proves its enforcement

Once the gateway is up, the harness checks that the hardening holds, not
only that it is configured: a TLS volume server refuses plain HTTP (Go
with HTTP 400, Rust with a TLS alert) and a connection without the client
certificate, and answers with it; under a read key an unsigned read of a
real chunk, written through the gateway and located through the filer,
gets 401.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the hardened cluster proves its chunks are encrypted

With VECTOR_ENCRYPT_VOLUMES the probe reads the stored chunk straight
from the volume server, signing the read when a read key is set, and
requires a recorded cipher key and bytes that differ from the plaintext.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: the signed probe read must return the chunk

An empty 200 passed both branches; an unencrypted read now has to return
the bytes written, and an encrypted one has to return something else.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: a read key in the caller's environment does not leak into the cluster

Only VECTOR_READ_JWT turns read JWTs on; an inherited
WEED_JWT_SIGNING_READ_KEY made the volume servers refuse the probe's
unsigned read in an encrypt-only run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: quote certificate paths as TOML strings in security.toml

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: keep non-ASCII certificate paths as UTF-8 in security.toml

TOML has no surrogate-pair escapes, and the Rust volume server reads the
quoted value without decoding escapes at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: an encrypted probe chunk must not be a readable gzip stream

The gateway may gzip the body before encrypting it, so a chunk uploaded
compressed but unencrypted would differ from the body and still pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

.github/workflows/rust-vector-tests.yml

@@ -76,19 +76,25 @@ jobs:
 
 
   vectors-e2e:
-    name: S3 Vectors Integration (${{ matrix.volume }} volume)
+    name: S3 Vectors Integration (${{ matrix.volume }} volume${{ matrix.hardened && ', hardened' || '' }})
     runs-on: ubuntu-24.04
     timeout-minutes: 45
     strategy:
       fail-fast: false
       matrix:
         volume: [go, rust]
+        # Hardened: volume servers behind mutual TLS, read JWTs required,
+        # and every chunk encrypted by the gateway.
+        hardened: ["", "1"]
     env:
       CARGO_PROFILE_DEV_DEBUG: "0"
       CARGO_BUILD_JOBS: "2"
       CARGO_TARGET_DIR: ${{ github.workspace }}/seaweed-vector/target
       VOLUME_SERVER_IMPL: ${{ matrix.volume }}
       VECTOR_QUERY_MODE: single
+      VECTOR_TLS: ${{ matrix.hardened }}
+      VECTOR_READ_JWT: ${{ matrix.hardened }}
+      VECTOR_ENCRYPT_VOLUMES: ${{ matrix.hardened }}
     steps:
       - uses: actions/checkout@v7
         with:
@@ -147,7 +153,7 @@ jobs:
         if: always()
         uses: actions/upload-artifact@v7
         with:
-          name: vectors-e2e-diagnostics-${{ matrix.volume }}
+          name: vectors-e2e-diagnostics-${{ matrix.volume }}${{ matrix.hardened && '-hardened' || '' }}
           path: |
             ${{ runner.temp }}/vector-tests
             ${{ runner.temp }}/vector-results.xml