* operation: give an encrypted chunk the plaintext ETag
With -encryptVolumeData the volume server stores ciphertext, so it cannot
echo a Content-MD5 back and the chunk lands with an empty ETag. Every ETag
derived from those chunks then comes out empty for a single chunk, or
d41d8cd98f00b204e9800998ecf8427e-N for several.
The caller already hashes the plaintext to send as Content-MD5, so keep that
digest as the chunk ETag instead of dropping it, and compute it for a
WantMd5 caller under cipher too.
* s3: re-encrypt a part copy from a volume-encrypted source
UploadPartCopy raw-copies source chunks when neither side uses SSE, which
also caught -encryptVolumeData sources. Those chunks are ciphertext a
whole-chunk cipher key decrypts, so copying a byte range out of one and
keeping the key leaves a destination that fails authentication on GET, and
the copied chunks carry no ETag for the part result to report.
Route them through the re-encrypting path already used for SSE: it reads the
source as plaintext, hashes the part, and writes the destination under the
gateway's own encryption.
* s3: fetch only the range a part copy asked for
The re-encrypting UploadPartCopy path opened the source at offset 0 and threw
the prefix away, so assembling an object part by part read the source once per
part. Now that volume-encrypted sources take this path too, that is the common
case rather than an SSE corner.
The chunk stream already seeks, so hand it the range.
* s3: reject an unsatisfiable copy-source-range
A part copy has no way to report a short part, so a range reaching past the
source cannot be clamped the way a GET clamps one. The fast path silently
produced a part shorter than asked for, or an empty one; the re-encrypting
path pads with zeros, so a 2 MiB source copied as bytes=1048576-9999999 came
back as 1 MiB of data followed by 7.5 MiB of nothing.
Answer InvalidRange instead, which is what s3-tests'
test_multipart_copy_invalid_range expects.
* S3: Fix encrypted file copy with multiple chunks (#7530)
When copying encrypted files with multiple chunks (encrypted volumes via
-filer.encryptVolumeData), the copied file could not be read. This was
caused by the chunk copy operation not preserving the IsCompressed flag,
which led to improper handling of compressed/encrypted data during upload.
The fix:
1. Modified uploadChunkData to accept an isCompressed parameter
2. Updated copySingleChunk to pass the source chunk's IsCompressed flag
3. Updated copySingleChunkForRange for partial copy operations
4. Updated all other callers to pass the appropriate compression flag
5. Added comprehensive tests for encrypted volume copy scenarios
This ensures that when copying chunks:
- The IsCompressed flag from the source chunk is passed to the upload
- Compressed data is marked as compressed, preventing double-compression
- Already-encrypted data is not re-encrypted (Cipher: false is correct)
- All chunk metadata (CipherKey, IsCompressed, ETag) is preserved
Tests added:
- TestCreateDestinationChunkPreservesEncryption: Verifies metadata preservation
- TestCopySingleChunkWithEncryption: Tests various encryption/compression scenarios
- TestCopyChunksPreservesMetadata: Tests multi-chunk metadata preservation
- TestEncryptedVolumeScenario: Documents and tests the exact issue #7530 scenario
Fixes#7530
* Address PR review feedback: simplify tests and improve clarity
- Removed TestUploadChunkDataCompressionFlag (panic-based test)
- Removed TestCopySingleChunkWithEncryption (duplicate coverage)
- Removed TestCopyChunksPreservesMetadata (duplicate coverage)
- Added ETag verification to TestEncryptedVolumeCopyScenario
- Renamed to TestEncryptedVolumeCopyScenario for better clarity
- All test coverage now in TestCreateDestinationChunkPreservesEncryption
and TestEncryptedVolumeCopyScenario which focus on the actual behavior
Chris Lu