Github-Mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-08 15:41:15 +02:00
Code Issues Packages Projects Releases Wiki Activity
14,954 Commits 177 Branches 359 Tags
d3b8030a69de22ac06cd8e90dad0eeddb194fc45
Commit Graph
3 Commits
Author SHA1 Message Date
Chris Lu 1f515f9d02 fix(s3api): cross-SSE copy operations and bring them back into CI (#9281) (#9282) 
* fix(s3api): cross-SSE copy operations and bring them back into CI (#9281)

Four cross-SSE copy tests were broken on master and excluded from CI with
the comment "pre-existing SSE-C issues":

  - TestSSECObjectCopyIntegration/Copy_SSE-C_to_SSE-C_with_different_key
  - TestSSEKMSObjectCopyIntegration/Copy_SSE-KMS_with_different_key
  - TestCrossSSECopy/SSE-S3_to_SSE-C
  - TestSSEMultipartCopy/Copy_SSE-KMS_Multipart_Object

Each surfaced as a different symptom — 500 InternalError, CRC32 mismatch,
"unexpected EOF", MD5 mismatch — but they were all instances of the same
root pattern that #8908 hit on UploadPartCopy: copy paths writing
destination chunks tagged inconsistently with the bytes on disk, so
detectPrimarySSEType / IsSSE*Encrypted disagreed about what the read
path should do.

Five fixes in this PR, each with its own targeted test:

  1. SSE-C IV format: putToFiler stored entry.Extended[SeaweedFSSSEIV]
     as raw bytes (with a comment saying so), but StoreSSECIVInMetadata
     stored it base64-encoded. The two readers (the GET handler reading
     it raw, and GetSSECIVFromMetadata reading it base64-decoded) each
     matched one writer but not the other. Standardise on raw bytes
     everywhere; GetSSECIVFromMetadata accepts the legacy base64 form
     for backward compat.

  2. SSE-C single-part copy chunk tagging: copyChunkWithReencryption
     re-encrypted the bytes for the destination but never set the
     destination chunk's SseType / SseMetadata. With chunks left
     SseType=NONE, detectPrimarySSEType returned "None" and the GET
     served still-encrypted volume bytes raw without decryption.
     Tag the chunk after re-encryption.

  3. SSE-KMS single-part copy chunk tagging: same shape as (2). Also,
     the function discarded the destSSEKey returned from
     CreateSSEKMSEncryptedReaderWithBucketKey (with `_`) — that key
     carries the freshly-minted EncryptedDataKey + IV the read path
     needs, so it must be captured and serialized into the destination
     chunk's per-chunk metadata (and bubbled up to the entry-level
     SeaweedFSSSEKMSKey for single-chunk objects whose read path falls
     back to the entry-level key).

  4. SSE-KMS multipart source decryption: copyChunkWithSSEKMSReencryption
     decrypted every source chunk with the entry-level sourceSSEKey.
     For multipart SSE-KMS objects each chunk has its own EDK + IV in
     per-chunk metadata, so the entry-level key is wrong. Decrypt with
     per-chunk metadata when present.

  5. Same-key copy fast path chunk tagging: copySingleChunk uses
     createDestinationChunk which dropped SseType / SseMetadata. For
     same-key copies (e.g. SSE-KMS source → SSE-KMS dest with the same
     KMS key) the fast path reuses the source ciphertext as-is, so the
     destination chunks must keep the source's SSE tagging. Add a
     createDestinationChunkPreservingSSE helper for the fast path; the
     re-encryption paths still call createDestinationChunk and then
     overwrite the SSE fields after re-encrypting.

CI: extend the comprehensive-test TEST_PATTERN to include the four test
families that were previously excluded (`.*ObjectCopyIntegration`,
`TestCrossSSECopy`, `TestSSEMultipartCopy`) so this category of
regression is caught going forward. The exclusion comment is removed.

Tests:
  - All four originally-failing tests pass.
  - The full pre-existing TestSSE* / TestCrossSSE / TestGitHub7562 /
    TestCopyToBucketDefaultEncryptedRegression / TestSSEMultipart suite
    still passes.
  - go test -race ./weed/s3api/ passes.

Refs #8908, #9280.

* fix(s3api): SSE-KMS copy ChunkOffset must stay 0 (review feedback on #9282)

CreateSSEKMSEncryptedReaderWithBucketKey initialises a fresh CTR stream
at counter 0 with a per-chunk random IV — there is no base-IV-plus-offset
relationship. The previous commit on this branch wrote
`destSSEKey.ChunkOffset = chunk.Offset` onto the per-chunk metadata,
which the read-side CreateSSEKMSDecryptedReader applies as
calculateIVWithOffset(IV, ChunkOffset) — i.e. it advances the
decryption IV by chunk.Offset/16 blocks beyond where the encryption
actually wrote.

The bug only manifests for SSE-KMS-to-SSE-KMS-with-different-key copies
of multipart sources (where source chunks live at non-zero offsets),
which is why the existing TestSSEKMSObjectCopyIntegration (single-chunk
source) and TestSSEMultipartCopy/Copy_SSE-KMS_Multipart_Object (same-key
copy that takes the fast preserving path, not the re-encrypt path)
both happened to pass. Set ChunkOffset to 0 to match the actual
encryption position. Existing tests still pass; the dangerous case is
only reachable with a multipart SSE-KMS source and a different
destination key, which is not currently exercised in CI.

Found by gemini-code-assist review on PR #9282.

* fix(s3api): use first dst chunk's full key for entry-level SSE-KMS metadata in remaining copy paths (review feedback on #9282)

Earlier this branch fixed copyChunksWithSSEKMSReencryption to populate
the entry-level SeaweedFSSSEKMSKey from the first destination chunk's
fully-formed metadata (with EDK + IV) instead of a stub key with only
KeyID + EncryptionContext + BucketKeyEnabled. The same fix needs to
apply to the other two paths that build entry-level SSE-KMS metadata:

  - copyMultipartCrossEncryption() — cross-encryption to SSE-KMS dest.
    Per-chunk metadata comes from copyCrossEncryptionChunk's
    CreateSSEKMSEncryptedReaderWithBucketKey call, so chunks[0] has
    a real EDK + IV. Use it.

  - copyChunksWithSSEKMS() direct (same-key) branch. After
    createDestinationChunkPreservingSSE in copySingleChunk, dst chunks
    carry the source's per-chunk SSE-KMS metadata. Use chunks[0] for
    the entry-level key so single-chunk same-key copies don't fall
    back to a stub key on the read path.

Without this, single-chunk SSE-KMS reads through these two paths
failed at GET with "Invalid ciphertext format" — KMS unwrap was
called on an empty EDK.

Found by coderabbitai review on PR #9282.

* fix(s3api): add 0-byte fallback to SSE-KMS reencryption entry-level metadata (review feedback on #9282)

copyChunksWithSSEKMSReencryption was missing the fallback for 0-byte
objects (where dstChunks is empty), inconsistent with the fallback in
copyChunksWithSSEKMS direct branch and copyMultipartCrossEncryption.
Without it, a 0-byte SSE-KMS copy would land with no entry-level
SeaweedFSSSEKMSKey, so the read path's IsSSEKMSEncryptedInternal check
would not recognise the empty object as SSE-KMS.

Mirror the existing fallback: build a stub SSEKMSKey with KeyID,
context and bucket-key state; serialize it as the entry-level key.

Found by gemini-code-assist review on PR #9282.

* fix(s3api): SSE-KMS direct copy must check encryption context + bucket-key, not just key ID (review feedback on #9282)

DetermineSSEKMSCopyStrategy / CanDirectCopySSEKMS only compares the
source and destination KMS key IDs, but the destination request can
also change the encryption context or the BucketKey flag. Both are
embedded in the source ciphertext's wrapped EDK; preserving the source
metadata verbatim does not satisfy a destination request that asks
for different settings, so the destination object would silently
report the source's context/flag instead of what was requested.

Add srcSSEKMSStateMatchesDest: deserialize the source's stored
SSEKMSKey and compare its EncryptionContext + BucketKeyEnabled to the
destination request. If either differs, force the slow re-encrypt
path (SSEKMSCopyStrategyDecryptEncrypt) so the destination gets a
freshly-wrapped EDK bound to the requested context/flag.

A malformed source key is treated as non-matching (conservative).
nil and empty encryption-context maps are treated as equal to avoid
spurious divergence when the request omits the context header.

Found by coderabbitai review on PR #9282.

* fix(s3api): copyMultipartSSEKMSChunk falls back to entry-level key + entry-level metadata uses first chunk's full key (review feedback on #9282)

Two related issues in copyMultipartSSEKMSChunks / copyMultipartSSEKMSChunk:

1. copyMultipartSSEKMSChunks built the destination's entry-level
   SeaweedFSSSEKMSKey from a stub (KeyID + context + bucket-key only),
   missing the EDK + IV. Single-chunk reads through this path fall
   back to entry-level keyData and would fail at GET because KMS would
   be asked to unwrap an empty EDK. Mirrors the fix in
   copyChunksWithSSEKMS / copyMultipartCrossEncryption /
   copyChunksWithSSEKMSReencryption: prefer the first dst chunk's
   full per-chunk metadata, fall back to the stub only for 0-byte
   objects.

2. copyMultipartSSEKMSChunk hard-failed when chunk.GetSseMetadata()
   was empty. Newer multipart SSE-KMS uploads populate per-chunk
   metadata, but legacy objects may have only entry-level metadata
   and would now be impossible to copy. Add a sourceEntrySSEKey
   fallback parameter (deserialized once by the caller from
   entry.Extended[SeaweedFSSSEKMSKey]); use it when per-chunk
   metadata is absent.

Found by coderabbitai review on PR #9282.

* refactor(s3api): extract entry-level SSE-KMS deserialization and per-chunk fallback into helpers (review feedback on #9282)

Three medium-priority maintainability comments from gemini-code-assist:

  - The same "deserialize entry.Extended[SeaweedFSSSEKMSKey]" pattern
    appeared in srcSSEKMSStateMatchesDest, copyMultipartSSEKMSChunks
    and copyChunksWithSSEKMSReencryption.

  - The "prefer per-chunk metadata, fall back to entry-level key"
    selection logic appeared inline in copyMultipartSSEKMSChunk and
    copyChunkWithSSEKMSReencryption with subtly different shapes.

  - encryptionContextEqual hand-rolled a map comparison.

Pull both patterns out into named helpers:

  - deserializeEntrySSEKMSKey: returns the entry-level SSEKMSKey or nil
    on missing/malformed data, with a single V(2) log line.
  - resolveChunkSSEKMSKey: centralises the chunk-vs-entry-level
    selection so all sites use the same decryption-side selection
    logic (which must mirror the encryption side).

Replace encryptionContextEqual's manual loop with reflect.DeepEqual,
keeping the empty-vs-nil shortcut at the top because DeepEqual treats
those as different.

No behaviour change; existing copy tests still pass.
 2026-04-29 10:06:51 -07:00
Chris LuandCopilot 0abf70061b S3 API: Fix SSE-S3 decryption on object download (#7366) 
* S3 API: Fix SSE-S3 decryption on object download

Fixes #7363

This commit adds missing SSE-S3 decryption support when downloading
objects from SSE-S3 encrypted buckets. Previously, SSE-S3 encrypted
objects were returned in their encrypted form, causing data corruption
and hash mismatches.

Changes:
- Updated detectPrimarySSEType() to detect SSE-S3 encrypted objects
  by examining chunk metadata and distinguishing SSE-S3 from SSE-KMS
- Added SSE-S3 handling in handleSSEResponse() to route to new handler
- Implemented handleSSES3Response() for both single-part and multipart
  SSE-S3 encrypted objects with proper decryption
- Implemented createMultipartSSES3DecryptedReader() for multipart
  objects with per-chunk decryption using stored IVs
- Updated addSSEHeadersToResponse() to include SSE-S3 response headers

The fix follows the existing SSE-C and SSE-KMS patterns, using the
envelope encryption architecture where each object's DEK is encrypted
with the KEK stored in the filer.

* Add comprehensive tests for SSE-S3 decryption

- TestSSES3EncryptionDecryption: basic encryption/decryption
- TestSSES3IsRequestInternal: request detection
- TestSSES3MetadataSerialization: metadata serialization/deserialization
- TestDetectPrimarySSETypeS3: SSE type detection for various scenarios
- TestAddSSES3HeadersToResponse: response header validation
- TestSSES3EncryptionWithBaseIV: multipart encryption with base IV
- TestSSES3WrongKeyDecryption: wrong key error handling
- TestSSES3KeyGeneration: key generation and uniqueness
- TestSSES3VariousSizes: encryption/decryption with various data sizes
- TestSSES3ResponseHeaders: response header correctness
- TestSSES3IsEncryptedInternal: metadata-based encryption detection
- TestSSES3InvalidMetadataDeserialization: error handling for invalid metadata
- TestGetSSES3Headers: header generation
- TestProcessSSES3Request: request processing
- TestGetSSES3KeyFromMetadata: key extraction from metadata
- TestSSES3EnvelopeEncryption: envelope encryption correctness
- TestValidateSSES3Key: key validation

All tests pass successfully, providing comprehensive coverage for the
SSE-S3 decryption fix.

* Address PR review comments

1. Fix resource leak in createMultipartSSES3DecryptedReader:
   - Wrap decrypted reader with closer to properly release resources
   - Ensure underlying chunkReader is closed when done

2. Handle mixed-encryption objects correctly:
   - Check chunk encryption type before attempting decryption
   - Pass through non-SSE-S3 chunks unmodified
   - Log encryption type for debugging

3. Improve SSE type detection logic:
   - Add explicit case for aws:kms algorithm
   - Handle unknown algorithms gracefully
   - Better documentation for tie-breaking precedence

4. Document tie-breaking behavior:
   - Clarify that mixed encryption indicates potential corruption
   - Explicit precedence order: SSE-C > SSE-KMS > SSE-S3

These changes address high-severity resource management issues and
improve robustness when handling edge cases and mixed-encryption
scenarios.

* Fix IV retrieval for small/inline SSE-S3 encrypted files

Critical bug fix: The previous implementation only looked for the IV in
chunk metadata, which would fail for small files stored inline (without
chunks).

Changes:
- Check object-level metadata (sseS3Key.IV) first for inline files
- Fallback to first chunk metadata only if object-level IV not found
- Improved error message to indicate both locations were checked

This ensures small SSE-S3 encrypted files (stored inline in entry.Content)
can be properly decrypted, as their IV is stored in the object-level
SeaweedFSSSES3Key metadata rather than in chunk metadata.

Fixes the high-severity issue identified in PR review.

* Clean up unused SSE metadata helper functions

Remove legacy SSE metadata helper functions that were never fully
implemented or used:

Removed unused functions:
- StoreSSECMetadata() / GetSSECMetadata()
- StoreSSEKMSMetadata() / GetSSEKMSMetadata()
- StoreSSES3Metadata() / GetSSES3Metadata()
- IsSSEEncrypted()
- GetSSEAlgorithm()

Removed unused constants:
- MetaSSEAlgorithm
- MetaSSECKeyMD5
- MetaSSEKMSKeyID
- MetaSSEKMSEncryptedKey
- MetaSSEKMSContext
- MetaSSES3KeyID

These functions were from an earlier design where IV and other metadata
would be stored in common entry.Extended keys. The actual implementations
use type-specific serialization:

- SSE-C: Uses StoreIVInMetadata()/GetIVFromMetadata() directly for IV
- SSE-KMS: Serializes entire SSEKMSKey structure as JSON (includes IV)
- SSE-S3: Serializes entire SSES3Key structure as JSON (includes IV)

This follows Option A: SSE-S3 uses envelope encryption pattern like
SSE-KMS, where IV is stored within the serialized key metadata rather
than in a separate metadata field.

Kept functions still in use:
- StoreIVInMetadata() - Used by SSE-C
- GetIVFromMetadata() - Used by SSE-C and streaming copy
- MetaSSEIV constant - Used by SSE-C

All tests pass after cleanup.

* Rename SSE metadata functions to clarify SSE-C specific usage

Renamed functions and constants to explicitly indicate they are SSE-C
specific, improving code clarity:

Renamed:
- MetaSSEIV → MetaSSECIV
- StoreIVInMetadata() → StoreSSECIVInMetadata()
- GetIVFromMetadata() → GetSSECIVFromMetadata()

Updated all usages across:
- s3api_key_rotation.go
- s3api_streaming_copy.go
- s3api_object_handlers_copy.go
- s3_sse_copy_test.go
- s3_sse_test_utils_test.go

Rationale:
These functions are exclusively used by SSE-C for storing/retrieving
the IV in entry.Extended metadata. SSE-KMS and SSE-S3 use different
approaches (IV stored in serialized key structures), so the generic
names were misleading. The new names make it clear these are part of
the SSE-C implementation.

All tests pass.

* Add integration tests for SSE-S3 end-to-end encryption/decryption

These integration tests cover the complete encrypt->store->decrypt cycle
that was missing from the original test suite. They would have caught
the IV retrieval bug for inline files.

Tests added:
- TestSSES3EndToEndSmallFile: Tests inline files (10, 50, 256 bytes)
  * Specifically tests the critical IV retrieval path for inline files
  * This test explicitly checks the bug we fixed where inline files
    couldn't retrieve their IV from object-level metadata

- TestSSES3EndToEndChunkedFile: Tests multipart encrypted files
  * Verifies per-chunk metadata serialization/deserialization
  * Tests that each chunk can be independently decrypted with its own IV

- TestSSES3EndToEndWithDetectPrimaryType: Tests type detection
  * Verifies inline vs chunked SSE-S3 detection
  * Ensures SSE-S3 is distinguished from SSE-KMS

Note: Full HTTP handler tests (PUT -> GET through actual handlers) would
require a complete mock server with filer connections, which is complex.
These tests focus on the critical decrypt path and data flow.

Why these tests are important:
- Unit tests alone don't catch integration issues
- The IV retrieval bug existed because there was no end-to-end test
- These tests simulate the actual storage/retrieval flow
- They verify the complete encryption architecture works correctly

All tests pass.

* Fix TestValidateSSES3Key expectations to match actual implementation

The ValidateSSES3Key function only validates that the key struct is not
nil, but doesn't validate the Key field contents or size. The test was
expecting validation that doesn't exist.

Updated test cases:
- Nil key struct → should error (correct)
- Valid key → should not error (correct)
- Invalid key size → should not error (validation doesn't check this)
- Nil key bytes → should not error (validation doesn't check this)

Added comments to clarify what the current validation actually checks.
This matches the behavior of ValidateSSEKMSKey and ValidateSSECKey
which also only check for nil struct, not field contents.

All SSE tests now pass.

* Improve ValidateSSES3Key to properly validate key contents

Enhanced the validation function from only checking nil struct to
comprehensive validation of all key fields:

Validations added:
1. Key bytes not nil
2. Key size exactly 32 bytes (SSES3KeySize)
3. Algorithm must be "AES256" (SSES3Algorithm)
4. Key ID must not be empty
5. IV length must be 16 bytes if set (optional - set during encryption)

Test improvements (10 test cases):
- Nil key struct
- Valid key without IV
- Valid key with IV
- Invalid key size (too small)
- Invalid key size (too large)
- Nil key bytes
- Empty key ID
- Invalid algorithm
- Invalid IV length
- Empty IV (allowed - set during encryption)

This matches the robustness of SSE-C and SSE-KMS validation and will
catch configuration errors early rather than failing during
encryption/decryption.

All SSE tests pass.

* Replace custom string helper functions with strings.Contains

Address Gemini Code Assist review feedback:
- Remove custom contains() and findSubstring() helper functions
- Use standard library strings.Contains() instead
- Add strings import

This makes the code more idiomatic and easier to maintain by using
the standard library instead of reimplementing functionality.

Changes:
- Added "strings" to imports
- Replaced contains(err.Error(), tc.errorMsg) with strings.Contains(err.Error(), tc.errorMsg)
- Removed 15 lines of custom helper code

All tests pass.

* filer fix reading and writing SSE-S3 headers

* filter out seaweedfs internal headers

* Update weed/s3api/s3api_object_handlers.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3_validation_utils.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update s3api_streaming_copy.go

* remove fallback

* remove redundant check

* refactor

* remove extra object fetching

* in case object is not found

* Correct Version Entry for SSE Routing

* Proper Error Handling for SSE Entry Fetching

* Eliminated All Redundant Lookups

* Removed brittle “exactly 5 successes/failures” assertions. Added invariant checks

total recorded attempts equals request count,
successes never exceed capacity,
failures cover remaining attempts,
final AvailableSpace matches capacity - successes.

* refactor

* fix test

* Fixed Broken Fallback Logic

* refactor

* Better Error for Encryption Type Mismatch

* refactor

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-10-23 20:10:12 -07:00
Chris Luandgemini-code-assist[bot] b7b73016dd S3 API: Add SSE-KMS (#7144) 
* implement sse-c

* fix Content-Range

* adding tests

* Update s3_sse_c_test.go

* copy sse-c objects

* adding tests

* refactor

* multi reader

* remove extra write header call

* refactor

* SSE-C encrypted objects do not support HTTP Range requests

* robust

* fix server starts

* Update Makefile

* Update Makefile

* ci: remove SSE-C integration tests and workflows; delete test/s3/encryption/

* s3: SSE-C MD5 must be base64 (case-sensitive); fix validation, comparisons, metadata storage; update tests

* minor

* base64

* Update SSE-C_IMPLEMENTATION.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update weed/s3api/s3api_object_handlers.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update SSE-C_IMPLEMENTATION.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* address comments

* fix test

* fix compilation

* Bucket Default Encryption

To complete the SSE-KMS implementation for production use:
Add AWS KMS Provider - Implement weed/kms/aws/aws_kms.go using AWS SDK
Integrate with S3 Handlers - Update PUT/GET object handlers to use SSE-KMS
Add Multipart Upload Support - Extend SSE-KMS to multipart uploads
Configuration Integration - Add KMS configuration to filer.toml
Documentation - Update SeaweedFS wiki with SSE-KMS usage examples

* store bucket sse config in proto

* add more tests

* Update SSE-C_IMPLEMENTATION.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Fix rebase errors and restore structured BucketMetadata API

Merge Conflict Fixes:
- Fixed merge conflicts in header.go (SSE-C and SSE-KMS headers)
- Fixed merge conflicts in s3api_errors.go (SSE-C and SSE-KMS error codes)
- Fixed merge conflicts in s3_sse_c.go (copy strategy constants)
- Fixed merge conflicts in s3api_object_handlers_copy.go (copy strategy usage)

API Restoration:
- Restored BucketMetadata struct with Tags, CORS, and Encryption fields
- Restored structured API functions: GetBucketMetadata, SetBucketMetadata, UpdateBucketMetadata
- Restored helper functions: UpdateBucketTags, UpdateBucketCORS, UpdateBucketEncryption
- Restored clear functions: ClearBucketTags, ClearBucketCORS, ClearBucketEncryption

Handler Updates:
- Updated GetBucketTaggingHandler to use GetBucketMetadata() directly
- Updated PutBucketTaggingHandler to use UpdateBucketTags()
- Updated DeleteBucketTaggingHandler to use ClearBucketTags()
- Updated CORS handlers to use UpdateBucketCORS() and ClearBucketCORS()
- Updated loadCORSFromBucketContent to use GetBucketMetadata()

Internal Function Updates:
- Updated getBucketMetadata() to return *BucketMetadata struct
- Updated setBucketMetadata() to accept *BucketMetadata struct
- Updated getBucketEncryptionMetadata() to use GetBucketMetadata()
- Updated setBucketEncryptionMetadata() to use SetBucketMetadata()

Benefits:
- Resolved all rebase conflicts while preserving both SSE-C and SSE-KMS functionality
- Maintained consistent structured API throughout the codebase
- Eliminated intermediate wrapper functions for cleaner code
- Proper error handling with better granularity
- All tests passing and build successful

The bucket metadata system now uses a unified, type-safe, structured API
that supports tags, CORS, and encryption configuration consistently.

* Fix updateEncryptionConfiguration for first-time bucket encryption setup

- Change getBucketEncryptionMetadata to getBucketMetadata to avoid failures when no encryption config exists
- Change setBucketEncryptionMetadata to setBucketMetadataWithEncryption for consistency
- This fixes the critical issue where bucket encryption configuration failed for buckets without existing encryption

Fixes: https://github.com/seaweedfs/seaweedfs/pull/7144#discussion_r2285669572

* Fix rebase conflicts and maintain structured BucketMetadata API

Resolved Conflicts:
- Fixed merge conflicts in s3api_bucket_config.go between structured API (HEAD) and old intermediate functions
- Kept modern structured API approach: UpdateBucketCORS, ClearBucketCORS, UpdateBucketEncryption
- Removed old intermediate functions: setBucketTags, deleteBucketTags, setBucketMetadataWithEncryption

API Consistency Maintained:
- updateCORSConfiguration: Uses UpdateBucketCORS() directly
- removeCORSConfiguration: Uses ClearBucketCORS() directly
- updateEncryptionConfiguration: Uses UpdateBucketEncryption() directly
- All structured API functions preserved: GetBucketMetadata, SetBucketMetadata, UpdateBucketMetadata

Benefits:
- Maintains clean separation between API layers
- Preserves atomic metadata updates with proper error handling
- Eliminates function indirection for better performance
- Consistent API usage pattern throughout codebase
- All tests passing and build successful

The bucket metadata system continues to use the unified, type-safe, structured API
that properly handles tags, CORS, and encryption configuration without any
intermediate wrapper functions.

* Fix complex rebase conflicts and maintain clean structured BucketMetadata API

Resolved Complex Conflicts:
- Fixed merge conflicts between modern structured API (HEAD) and mixed approach
- Removed duplicate function declarations that caused compilation errors
- Consistently chose structured API approach over intermediate functions

Fixed Functions:
- BucketMetadata struct: Maintained clean field alignment
- loadCORSFromBucketContent: Uses GetBucketMetadata() directly
- updateCORSConfiguration: Uses UpdateBucketCORS() directly
- removeCORSConfiguration: Uses ClearBucketCORS() directly
- getBucketMetadata: Returns *BucketMetadata struct consistently
- setBucketMetadata: Accepts *BucketMetadata struct consistently

Removed Duplicates:
- Eliminated duplicate GetBucketMetadata implementations
- Eliminated duplicate SetBucketMetadata implementations
- Eliminated duplicate UpdateBucketMetadata implementations
- Eliminated duplicate helper functions (UpdateBucketTags, etc.)

API Consistency Achieved:
- Single, unified BucketMetadata struct for all operations
- Atomic updates through UpdateBucketMetadata with function callbacks
- Type-safe operations with proper error handling
- No intermediate wrapper functions cluttering the API

Benefits:
- Clean, maintainable codebase with no function duplication
- Consistent structured API usage throughout all bucket operations
- Proper error handling and type safety
- Build successful and all tests passing

The bucket metadata system now has a completely clean, structured API
without any conflicts, duplicates, or inconsistencies.

* Update remaining functions to use new structured BucketMetadata APIs directly

Updated functions to follow the pattern established in bucket config:
- getEncryptionConfiguration() -> Uses GetBucketMetadata() directly
- removeEncryptionConfiguration() -> Uses ClearBucketEncryption() directly

Benefits:
- Consistent API usage pattern across all bucket metadata operations
- Simpler, more readable code that leverages the structured API
- Eliminates calls to intermediate legacy functions
- Better error handling and logging consistency
- All tests pass with improved functionality

This completes the transition to using the new structured BucketMetadata API
throughout the entire bucket configuration and encryption subsystem.

* Fix GitHub PR #7144 code review comments

Address all code review comments from Gemini Code Assist bot:

1. **High Priority - SSE-KMS Key Validation**: Fixed ValidateSSEKMSKey to allow empty KMS key ID
   - Empty key ID now indicates use of default KMS key (consistent with AWS behavior)
   - Updated ParseSSEKMSHeaders to call validation after parsing
   - Enhanced isValidKMSKeyID to reject keys with spaces and invalid characters

2. **Medium Priority - KMS Registry Error Handling**: Improved error collection in CloseAll
   - Now collects all provider close errors instead of only returning the last one
   - Uses proper error formatting with %w verb for error wrapping
   - Returns single error for one failure, combined message for multiple failures

3. **Medium Priority - Local KMS Aliases Consistency**: Fixed alias handling in CreateKey
   - Now updates the aliases slice in-place to maintain consistency
   - Ensures both p.keys map and key.Aliases slice use the same prefixed format

All changes maintain backward compatibility and improve error handling robustness.
Tests updated and passing for all scenarios including edge cases.

* Use errors.Join for KMS registry error handling

Replace manual string building with the more idiomatic errors.Join function:

- Removed manual error message concatenation with strings.Builder
- Simplified error handling logic by using errors.Join(allErrors...)
- Removed unnecessary string import
- Added errors import for errors.Join

This approach is cleaner, more idiomatic, and automatically handles:
- Returning nil for empty error slice
- Returning single error for one-element slice
- Properly formatting multiple errors with newlines

The errors.Join function was introduced in Go 1.20 and is the
recommended way to combine multiple errors.

* Update registry.go

* Fix GitHub PR #7144 latest review comments

Address all new code review comments from Gemini Code Assist bot:

1. **High Priority - SSE-KMS Detection Logic**: Tightened IsSSEKMSEncrypted function
   - Now relies only on the canonical x-amz-server-side-encryption header
   - Removed redundant check for x-amz-encrypted-data-key metadata
   - Prevents misinterpretation of objects with inconsistent metadata state
   - Updated test case to reflect correct behavior (encrypted data key only = false)

2. **Medium Priority - UUID Validation**: Enhanced KMS key ID validation
   - Replaced simplistic length/hyphen count check with proper regex validation
   - Added regexp import for robust UUID format checking
   - Regex pattern: ^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$
   - Prevents invalid formats like '------------------------------------' from passing

3. **Medium Priority - Alias Mutation Fix**: Avoided input slice modification
   - Changed CreateKey to not mutate the input aliases slice in-place
   - Uses local variable for modified alias to prevent side effects
   - Maintains backward compatibility while being safer for callers

All changes improve code robustness and follow AWS S3 standards more closely.
Tests updated and passing for all scenarios including edge cases.

* Fix failing SSE tests

Address two failing test cases:

1. **TestSSEHeaderConflicts**: Fixed SSE-C and SSE-KMS mutual exclusion
   - Modified IsSSECRequest to return false if SSE-KMS headers are present
   - Modified IsSSEKMSRequest to return false if SSE-C headers are present
   - This prevents both detection functions from returning true simultaneously
   - Aligns with AWS S3 behavior where SSE-C and SSE-KMS are mutually exclusive

2. **TestBucketEncryptionEdgeCases**: Fixed XML namespace validation
   - Added namespace validation in encryptionConfigFromXMLBytes function
   - Now rejects XML with invalid namespaces (only allows empty or AWS standard namespace)
   - Validates XMLName.Space to ensure proper XML structure
   - Prevents acceptance of malformed XML with incorrect namespaces

Both fixes improve compliance with AWS S3 standards and prevent invalid
configurations from being accepted. All SSE and bucket encryption tests
now pass successfully.

* Fix GitHub PR #7144 latest review comments

Address two new code review comments from Gemini Code Assist bot:

1. **High Priority - Race Condition in UpdateBucketMetadata**: Fixed thread safety issue
   - Added per-bucket locking mechanism to prevent race conditions
   - Introduced bucketMetadataLocks map with RWMutex for each bucket
   - Added getBucketMetadataLock helper with double-checked locking pattern
   - UpdateBucketMetadata now uses bucket-specific locks to serialize metadata updates
   - Prevents last-writer-wins scenarios when concurrent requests update different metadata parts

2. **Medium Priority - KMS Key ARN Validation**: Improved robustness of ARN validation
   - Enhanced isValidKMSKeyID function to strictly validate ARN structure
   - Changed from 'len(parts) >= 6' to 'len(parts) != 6' for exact part count
   - Added proper resource validation for key/ and alias/ prefixes
   - Prevents malformed ARNs with incorrect structure from being accepted
   - Now validates: arn:aws:kms:region:account:key/keyid or arn:aws:kms:region:account:alias/aliasname

Both fixes improve system reliability and prevent edge cases that could cause
data corruption or security issues. All existing tests continue to pass.

* format

* address comments

* Configuration Adapter

* Regex Optimization

* Caching Integration

* add negative cache for non-existent buckets

* remove bucketMetadataLocks

* address comments

* address comments

* copying objects with sse-kms

* copying strategy

* store IV in entry metadata

* implement compression reader

* extract json map as sse kms context

* bucket key

* comments

* rotate sse chunks

* KMS Data Keys use AES-GCM + nonce

* add comments

* Update weed/s3api/s3_sse_kms.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update s3api_object_handlers_put.go

* get IV from response header

* set sse headers

* Update s3api_object_handlers.go

* deterministic JSON marshaling

* store iv in entry metadata

* address comments

* not used

* store iv in destination metadata

ensures that SSE-C copy operations with re-encryption (decrypt/re-encrypt scenario) now properly store the destination encryption metadata

* add todo

* address comments

* SSE-S3 Deserialization

* add BucketKMSCache to BucketConfig

* fix test compilation

* already not empty

* use constants

* fix: critical metadata (encrypted data keys, encryption context, etc.) was never stored during PUT/copy operations

* address comments

* fix tests

* Fix SSE-KMS Copy Re-encryption

* Cache now persists across requests

* fix test

* iv in metadata only

* SSE-KMS copy operations should follow the same pattern as SSE-C

* fix size overhead calculation

* Filer-Side SSE Metadata Processing

* SSE Integration Tests

* fix tests

* clean up

* Update s3_sse_multipart_test.go

* add s3 sse tests

* unused

* add logs

* Update Makefile

* Update Makefile

* s3 health check

* The tests were failing because they tried to run both SSE-C and SSE-KMS tests

* Update weed/s3api/s3_sse_c.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update Makefile

* add back

* Update Makefile

* address comments

* fix tests

* Update s3-sse-tests.yml

* Update s3-sse-tests.yml

* fix sse-kms for PUT operation

* IV

* Update auth_credentials.go

* fix multipart with kms

* constants

* multipart sse kms

Modified handleSSEKMSResponse to detect multipart SSE-KMS objects
Added createMultipartSSEKMSDecryptedReader to handle each chunk independently
Each chunk now gets its own decrypted reader before combining into the final stream

* validate key id

* add SSEType

* permissive kms key format

* Update s3_sse_kms_test.go

* format

* assert equal

* uploading SSE-KMS metadata per chunk

* persist sse type and metadata

* avoid re-chunk multipart uploads

* decryption process to use stored PartOffset values

* constants

* sse-c multipart upload

* Unified Multipart SSE Copy

* purge

* fix fatalf

* avoid io.MultiReader which does not close underlying readers

* unified cross-encryption

* fix Single-object SSE-C

* adjust constants

* range read sse files

* remove debug logs

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-08-21 08:28:07 -07:00