s3vectors: encrypt the objects of a vector bucket declared AES256 (#12324)

* s3api: encrypt the objects of a vector bucket declared AES256

CreateVectorBucket recorded encryptionConfiguration sseType AES256 and
returned it, but nothing on the write path read it: the index's data
files, manifests and transactions were stored without SSE.

A LANCE table bucket's record now supplies the bucket's default
encryption whenever the bucket has none of its own, so PutObject,
CreateMultipartUpload and CopyObject apply SSE-S3 to every object the
engine writes. The S3 bucket metadata itself is unchanged.

ICEBERG table buckets keep recording the declaration only: their
catalog reads the manifests clients upload straight from the filer,
which would see ciphertext.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3tables: refuse an AES256 vector bucket on a gateway without an SSE-S3 key

A vector bucket's AES256 is now applied to every index write, and SSE-S3
fails each write when the gateway has no KEK. Creating the bucket there
succeeded and the first CreateIndex came back as InternalFailure.

CreateTableBucket now refuses a LANCE bucket declaring AES256 when the
gateway reports no SSE-S3 key, naming s3.sse.kek and s3.sse.key;
CreateVectorBucket returns it as ValidationException. ICEBERG buckets,
whose declaration is only recorded, and tooling that sets no checker
are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: refuse DeleteBucketEncryption on a bucket's declared encryption

GetBucketEncryption reports a vector bucket's declared AES256, but
DeleteBucketEncryption answered NoSuchBucketEncryptionConfiguration and
the declaration kept applying. The declaration is fixed when the bucket
is created, so the delete now returns MethodNotAllowed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): keep an inherited WEED_S3_SSE_KEK out of the test cluster

The gateway refuses to start SSE-S3 with both s3.sse.kek and s3.sse.key
set, so a developer shell exporting WEED_S3_SSE_KEK made the AES256
bucket test fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

test/s3vectors/conftest.py

@@ -102,7 +102,9 @@ def run_cluster(param, tmp_path_factory, identities):
         def start(name, args, unset=()):
             log = stack.enter_context((directory / f"{name}.log").open("w"))
             env = dict(os.environ, AWS_ACCESS_KEY_ID="test", AWS_SECRET_ACCESS_KEY="test",
-                       AWS_DEFAULT_REGION="us-east-1", AWS_EC2_METADATA_DISABLED="true")
+                       AWS_DEFAULT_REGION="us-east-1", AWS_EC2_METADATA_DISABLED="true",
+                       WEED_S3_SSE_KEY="s3vectors-test")
+            env.pop("WEED_S3_SSE_KEK", None)
             for key in unset:
                 env.pop(key)
             process = subprocess.Popen(args, cwd=directory, stdout=log, stderr=log, env=env)

test/s3vectors/s3vectors_test.py

@@ -1,5 +1,6 @@
 """Run the S3 Vectors API against Go/Rust volumes and single/distributed queries."""
 
+import contextlib
 import json
 import time
 import uuid
@@ -297,17 +298,44 @@ def test_vector_policy_permissions(vectors, vector_clients, index):
         vectors.delete_index(**other)
 
 
+def index_object_encryption(vectors, bucket):
+    s3 = boto3.client("s3", endpoint_url=vectors.meta.endpoint_url, aws_access_key_id="test",
+                      aws_secret_access_key="test", region_name="us-east-1")
+    keys = [o["Key"] for page in s3.get_paginator("list_objects_v2").paginate(Bucket=bucket, Prefix=f"vector/{INDEX}/")
+            for o in page.get("Contents", []) if o["Size"] > 0]
+    assert any("/data/" in key for key in keys) and any("/_versions/" in key for key in keys)
+    return {key: s3.head_object(Bucket=bucket, Key=key).get("ServerSideEncryption") for key in keys}
+
+
 def test_vector_bucket_encryption(vectors):
     bucket = "encryption-" + uuid.uuid4().hex[:16]
+    plain = bucket + "-plain"
     config = {"sseType": "AES256"}
     vectors.create_vector_bucket(vectorBucketName=bucket, encryptionConfiguration=config)
+    vectors.create_vector_bucket(vectorBucketName=plain)
     try:
         assert vectors.get_vector_bucket(vectorBucketName=bucket)["vectorBucket"]["encryptionConfiguration"] == config
         expect_error("ValidationException", vectors.create_vector_bucket, vectorBucketName=bucket + "-kms",
                      encryptionConfiguration={"sseType": "aws:kms", "kmsKeyArn": "arn:aws:kms:us-east-1:123456789012:key/test"})
         assert vectors.list_vector_buckets(prefix=bucket + "-kms")["vectorBuckets"] == []
+
+        for name in (bucket, plain):
+            vectors.create_index(vectorBucketName=name, indexName=INDEX, dataType="float32", dimension=4,
+                                 distanceMetric="euclidean")
+            vectors.put_vectors(vectorBucketName=name, indexName=INDEX, vectors=[
+                {"key": "a", "data": {"float32": [1, 0, 0, 0]}, "metadata": {"color": "red"}},
+                {"key": "b", "data": {"float32": [0, 1, 0, 0]}, "metadata": {"color": "blue"}},
+            ])
+            hits = vectors.query_vectors(vectorBucketName=name, indexName=INDEX, queryVector={"float32": [1, 0, 0, 0]},
+                                         topK=1, returnMetadata=True)["vectors"]
+            assert [(h["key"], h["metadata"]) for h in hits] == [("a", {"color": "red"})]
+        assert set(index_object_encryption(vectors, bucket).values()) == {"AES256"}
+        assert set(index_object_encryption(vectors, plain).values()) == {None}
     finally:
-        vectors.delete_vector_bucket(vectorBucketName=bucket)
+        for name in (bucket, plain):
+            with contextlib.suppress(ClientError):
+                vectors.delete_index(vectorBucketName=name, indexName=INDEX)
+            vectors.delete_vector_bucket(vectorBucketName=name)
 
 
 def test_catalog_pagination(vectors, index):