Files
seaweedfs/weed/s3api/s3api_object_handlers_multipart.go
T
Chris LuandClaude Opus 5.5 df30c883fe s3api: keep a multipart upload's state on the object's owner filer (#12668)
* s3api: create a multipart upload's directory on the object's owner filer

CreateMultipartUpload wrote .uploads/<id> to the gateway's own filer while
the upload's completion commits on the object's write owner. With
independent filer stores, a part or completion reaching another gateway
before peer metadata replication caught up could not see the upload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: look up a multipart upload on the object's owner filer

UploadPart, UploadPartCopy, ListParts, CompleteMultipartUpload,
AbortMultipartUpload and the bucket policy SSE lookup read .uploads/<id>
from the gateway's own filer, so a request on another gateway shortly
after CreateMultipartUpload answered NoSuchUpload until replication
caught up. Read it preferring the object's write owner, like object reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: write multipart parts to the upload's owner filer

UploadPart and UploadPartCopy stored part entries on the gateway's own
filer, so ListParts and CompleteMultipartUpload on another gateway could
miss a part that replication had not delivered yet. Store them next to the
upload directory on the object's write owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: list and clean up a multipart upload on its owner filer

ListParts, CompleteMultipartUpload and AbortMultipartUpload listed and
removed part entries on the gateway's own filer. Do it where the upload
and its parts now live, so completion sees every part and cleanup does
not miss entries replication has not delivered yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test a multipart upload against a lagging gateway filer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: check the upload's owner before freeing a failed part's chunks

A part write that the owner stored but answered with an error went to
recovery with no object, so the owner was never among the filers checked.
When the owner is outside the gateway's filer list, the others' not-found
answers freed chunks the stored part still references.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: list an upload's parts on the prior owner after a ring change

The upload record lookup already falls back to the key's prior owner while
a ring change is in its cooling-off window, but the part listing read only
the new owner. ListParts then reported no parts for an open upload. Fall
back the same way when the new owner has nothing yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: find the upload record on the prior owner past a bare directory

A part routed to a new owner that has not received the upload yet makes
that filer create .uploads/<id> without the upload's key. The post-write
check then read it as an aborted upload and removed the part. While the
ring change is in its cooling-off window, a record on the prior owner
now wins over such a directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: merge the prior owner's parts into an upload listing

During a ring change's cooling-off window the new owner may have received
only some of an upload's parts. Listing returned the new owner's parts
alone once it had any, so ListParts dropped the rest and a completion
naming them failed with InvalidPart. Both owners' pages are now merged,
the owner's entry winning a shared name, and cut to the page limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: fail an upload listing the prior owner could not answer

When the new owner has none of an upload's parts yet and the prior
owner's listing fails, the parts may all be on the prior owner. That
listing read as empty, so CompleteMultipartUpload answered NoSuchUpload
for an upload that still exists; it now returns the error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: fail an upload lookup the prior owner could not answer

During a ring change's cooling-off window an upload the new owner has
not received yet, or holds only as a bare directory, read as NoSuchUpload
when the prior owner's lookup failed. That failure is now returned, so
the client sees a retryable error instead of an upload that is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 19:03:41 -08:00

749 lines
27 KiB
Go

package s3api
import (
"crypto/sha1"
"encoding/base64"
"encoding/json"
"encoding/xml"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"strconv"
"strings"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/google/uuid"
"github.com/pquerna/cachecontrol/cacheobject"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
stats_collect "github.com/seaweedfs/seaweedfs/weed/stats"
)
const (
maxObjectListSizeLimit = 1000 // Limit number of objects in a listObjectsResponse.
maxUploadsList = 10000 // Limit number of uploads in a listUploadsResponse.
maxPartsList = 10000 // Limit number of parts in a listPartsResponse.
)
// NewMultipartUploadHandler - New multipart upload.
func (s3a *S3ApiServer) NewMultipartUploadHandler(w http.ResponseWriter, r *http.Request) {
bucket, object := s3_constants.GetBucketAndObject(r)
if len(object) > s3_constants.MaxS3ObjectKeyLength {
s3err.WriteErrorResponse(w, r, s3err.ErrKeyTooLongError)
return
}
// Reject before the auto-create check so a refused upload cannot leave a bucket behind
if errCode := ValidateRequestEncryption(r.Header); errCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, errCode)
return
}
// Check if bucket exists, and create it if it doesn't (auto-create bucket)
if err := s3a.checkBucket(r, bucket); err == s3err.ErrNoSuchBucket {
// Auto-create bucket if it doesn't exist (requires Admin permission)
if !s3a.handleAutoCreateBucket(w, r, bucket, "NewMultipartUploadHandler") {
return
}
} else if err != s3err.ErrNone {
// Other errors (like access denied) should still fail
s3err.WriteErrorResponse(w, r, err)
return
}
if err := s3a.validateTableBucketObjectPath(bucket, object); err != nil {
s3err.WriteErrorResponse(w, r, s3err.ErrAccessDenied)
return
}
if errCode := s3a.checkVersionFileWrite(bucket, object, "NewMultipartUploadHandler"); errCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, errCode)
return
}
// Check if versioning is enabled for the bucket (needed for object lock)
versioningEnabled, err := s3a.isVersioningEnabled(bucket)
if err != nil {
if errors.Is(err, filer_pb.ErrNotFound) {
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchBucket)
return
}
glog.Errorf("Error checking versioning status for bucket %s: %v", bucket, err)
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
return
}
// Validate object lock headers before processing
if err := s3a.validateObjectLockHeaders(r, bucket, object, versioningEnabled); err != nil {
glog.V(2).Infof("NewMultipartUploadHandler: object lock header validation failed for bucket %s, object %s: %v", bucket, object, err)
s3err.WriteErrorResponse(w, r, mapValidationErrorToS3Error(err))
return
}
// Validate Cache-Control header format if present
if cacheControl := r.Header.Get("Cache-Control"); cacheControl != "" {
if _, err := cacheobject.ParseRequestCacheControl(cacheControl); err != nil {
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidRequest)
return
}
}
// Validate Expires header format if present
if expires := r.Header.Get("Expires"); expires != "" {
if _, err := time.Parse(http.TimeFormat, expires); err != nil {
s3err.WriteErrorResponse(w, r, s3err.ErrMalformedDate)
return
}
}
// An explicit ACL on initiate applies to the completed object and requires
// s3:PutObjectAcl, same as PUT. The validated grants ride on the upload
// entry so CompleteMultipartUpload cannot be handed forged internal headers.
r, aclCode := s3a.preparePutObjectACL(r, bucket)
if aclCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, aclCode)
return
}
createMultipartUploadInput := &s3.CreateMultipartUploadInput{
Bucket: aws.String(bucket),
Key: objectKey(aws.String(object)),
Metadata: make(map[string]*string),
}
// Parse S3 metadata from request headers
metadata, errCode := ParseS3Metadata(r, nil, false)
if errCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, errCode)
return
}
for k, v := range metadata {
createMultipartUploadInput.Metadata[k] = aws.String(string(v))
}
contentType := r.Header.Get("Content-Type")
if contentType != "" {
createMultipartUploadInput.ContentType = &contentType
}
response, errCode := s3a.createMultipartUpload(r, createMultipartUploadInput)
glog.V(3).Info("NewMultipartUploadHandler", string(s3err.EncodeXMLResponse(response)), errCode)
if errCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, errCode)
return
}
if response.ChecksumAlgorithm != "" {
w.Header().Set(s3_constants.AmzChecksumAlgorithm, response.ChecksumAlgorithm)
if response.ChecksumType != "" {
w.Header().Set(s3_constants.AmzChecksumType, response.ChecksumType)
}
}
writeSuccessResponseXML(w, r, response)
}
// CompleteMultipartUploadHandler - Completes multipart upload.
func (s3a *S3ApiServer) CompleteMultipartUploadHandler(w http.ResponseWriter, r *http.Request) {
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html
bucket, object := s3_constants.GetBucketAndObject(r)
// Check if bucket exists before completing multipart upload
if err := s3a.checkBucket(r, bucket); err != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, err)
return
}
if err := s3a.validateTableBucketObjectPath(bucket, object); err != nil {
s3err.WriteErrorResponse(w, r, s3err.ErrAccessDenied)
return
}
if errCode := s3a.checkVersionFileWrite(bucket, object, "CompleteMultipartUploadHandler"); errCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, errCode)
return
}
parts := &CompleteMultipartUpload{}
if err := xmlDecoder(r.Body, parts, r.ContentLength); err != nil {
s3err.WriteErrorResponse(w, r, requestBodyErrorCode(err, s3err.ErrMalformedXML))
return
}
// Get upload id.
uploadID, _, _, _, _ := getObjectResources(r.URL.Query())
err := s3a.checkUploadId(object, uploadID)
if err != nil {
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchUpload)
return
}
// Check conditional headers before completing multipart upload
// This implements AWS S3 behavior where conditional headers apply to CompleteMultipartUpload
if errCode := s3a.checkConditionalHeaders(r, bucket, object); errCode != s3err.ErrNone {
glog.V(3).Infof("CompleteMultipartUploadHandler: Conditional header check failed for %s/%s", bucket, object)
s3err.WriteErrorResponse(w, r, errCode)
return
}
response, errCode := s3a.completeMultipartUpload(r, &s3.CompleteMultipartUploadInput{
Bucket: aws.String(bucket),
Key: objectKey(aws.String(object)),
UploadId: aws.String(uploadID),
}, parts)
glog.V(3).Info("CompleteMultipartUploadHandler", string(s3err.EncodeXMLResponse(response)), errCode)
if errCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, errCode)
return
}
// Set version ID in HTTP header if present
if response.VersionId != nil {
w.Header().Set("x-amz-version-id", *response.VersionId)
}
stats_collect.RecordBucketActiveTime(bucket)
stats_collect.S3UploadedObjectsCounter.WithLabelValues(bucket).Inc()
writeSuccessResponseXML(w, r, response)
}
// AbortMultipartUploadHandler - Aborts multipart upload.
func (s3a *S3ApiServer) AbortMultipartUploadHandler(w http.ResponseWriter, r *http.Request) {
bucket, object := s3_constants.GetBucketAndObject(r)
// Check if bucket exists before aborting multipart upload
if err := s3a.checkBucket(r, bucket); err != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, err)
return
}
if err := s3a.validateTableBucketObjectPath(bucket, object); err != nil {
s3err.WriteErrorResponse(w, r, s3err.ErrAccessDenied)
return
}
// Get upload id.
uploadID, _, _, _, _ := getObjectResources(r.URL.Query())
err := s3a.checkUploadId(object, uploadID)
if err != nil {
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchUpload)
return
}
response, errCode := s3a.abortMultipartUpload(&s3.AbortMultipartUploadInput{
Bucket: aws.String(bucket),
Key: objectKey(aws.String(object)),
UploadId: aws.String(uploadID),
})
if errCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, errCode)
return
}
glog.V(3).Info("AbortMultipartUploadHandler", string(s3err.EncodeXMLResponse(response)))
//https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html
s3err.WriteEmptyResponse(w, r, http.StatusNoContent)
s3err.PostLog(r, http.StatusNoContent, s3err.ErrNone)
}
// ListMultipartUploadsHandler - Lists multipart uploads.
func (s3a *S3ApiServer) ListMultipartUploadsHandler(w http.ResponseWriter, r *http.Request) {
bucket, _ := s3_constants.GetBucketAndObject(r)
// Check if bucket exists before listing multipart uploads
if err := s3a.checkBucket(r, bucket); err != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, err)
return
}
prefix, multipartKeyMarker, uploadIDMarker, delimiter, maxUploads, encodingType, errCode := getBucketMultipartResources(r.URL.Query())
if errCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, errCode)
return
}
if maxUploads < 0 {
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidMaxUploads)
return
}
// The marker is a raw key: under encoding-type=url clients decode the
// escaped response fields and send the key itself back. The token form
// only appears when the response could not carry the raw key.
keyMarker := multipartKeyMarker
if encodingType != s3.EncodingTypeUrl {
keyMarker = decodeListContinuationToken(multipartKeyMarker)
}
if keyMarker != "" {
// Marker not common with prefix is not implemented.
if !strings.HasPrefix(keyMarker, prefix) {
s3err.WriteErrorResponse(w, r, s3err.ErrNotImplemented)
return
}
}
response, errCode := s3a.listMultipartUploads(&s3.ListMultipartUploadsInput{
Bucket: aws.String(bucket),
Delimiter: aws.String(delimiter),
EncodingType: aws.String(encodingType),
KeyMarker: aws.String(keyMarker),
MaxUploads: aws.Int64(int64(maxUploads)),
Prefix: aws.String(prefix),
UploadIdMarker: aws.String(uploadIDMarker),
})
glog.V(3).Infof("ListMultipartUploadsHandler %s errCode=%d", string(s3err.EncodeXMLResponse(response)), errCode)
if errCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, errCode)
return
}
writeSuccessResponseXML(w, r, response)
}
// ListObjectPartsHandler - Lists object parts in a multipart upload.
func (s3a *S3ApiServer) ListObjectPartsHandler(w http.ResponseWriter, r *http.Request) {
bucket, object := s3_constants.GetBucketAndObject(r)
// Check if bucket exists before listing object parts
if err := s3a.checkBucket(r, bucket); err != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, err)
return
}
uploadID, partNumberMarker, maxParts, _, errCode := getObjectResources(r.URL.Query())
if errCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, errCode)
return
}
if partNumberMarker < 0 {
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidPartNumberMarker)
return
}
if maxParts < 0 {
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidMaxParts)
return
}
err := s3a.checkUploadId(object, uploadID)
if err != nil {
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchUpload)
return
}
response, errCode := s3a.listObjectParts(&s3.ListPartsInput{
Bucket: aws.String(bucket),
Key: objectKey(aws.String(object)),
MaxParts: aws.Int64(int64(maxParts)),
PartNumberMarker: aws.Int64(int64(partNumberMarker)),
UploadId: aws.String(uploadID),
})
if errCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, errCode)
return
}
glog.V(3).Infof("ListObjectPartsHandler %s count=%d", string(s3err.EncodeXMLResponse(response)), len(response.Part))
writeSuccessResponseXML(w, r, response)
}
// PutObjectPartHandler - Put an object part in a multipart upload.
func (s3a *S3ApiServer) PutObjectPartHandler(w http.ResponseWriter, r *http.Request) {
bucket, object := s3_constants.GetBucketAndObject(r)
if src := r.Header.Get("X-Amz-Copy-Source"); src != "" {
// Well-formed copy sources are claimed by the CopyObjectPart route; what
// is left here has no "bucket/key" form and is not a plain PutObjectPart.
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidArgument)
return
}
_, err := validateContentMd5(r.Header)
if err != nil {
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidDigest)
return
}
// Check if bucket exists before putting object part
if err := s3a.checkBucket(r, bucket); err != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, err)
return
}
uploadID := r.URL.Query().Get("uploadId")
// validateTableBucketObjectPath is enforced at multipart initiation. checkUploadId
// cryptographically binds uploadID to object path, so parts cannot switch paths.
err = s3a.checkUploadId(object, uploadID)
if err != nil {
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchUpload)
return
}
partIDString := r.URL.Query().Get("partNumber")
partID, err := strconv.Atoi(partIDString)
if err != nil || partID < 1 || partID > s3_constants.MaxS3MultipartParts {
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidArgument)
return
}
if _, errCode := validatePutHeaders(r); errCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, errCode)
return
}
dataReader, s3ErrCode := getRequestDataReader(s3a, r)
if s3ErrCode != s3err.ErrNone {
glog.Errorf("PutObjectPartHandler: getRequestDataReader failed with code %v", s3ErrCode)
s3err.WriteErrorResponse(w, r, s3ErrCode)
return
}
defer dataReader.Close()
glog.V(2).Infof("PutObjectPartHandler %s %s %04d", bucket, uploadID, partID)
// Verify the multipart upload exists (rejects parts after abort)
uploadEntry, err := s3a.uploadFiler(bucket, object).getEntry(s3a.genUploadsFolder(bucket), uploadID)
if errors.Is(err, filer_pb.ErrNotFound) {
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchUpload)
return
} else if err != nil {
glog.Errorf("Could not retrieve upload entry for %s/%s: %v", bucket, uploadID, err)
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
return
}
// Apply SSE settings from the upload entry (unless SSE-C headers are already present)
sseCustomerAlgorithm := r.Header.Get(s3_constants.AmzServerSideEncryptionCustomerAlgorithm)
if sseCustomerAlgorithm == "" && uploadEntry.Extended != nil {
if keyIDBytes, exists := uploadEntry.Extended[s3_constants.SeaweedFSSSEKMSKeyID]; exists {
keyID := string(keyIDBytes)
bucketKeyEnabled := false
if bucketKeyBytes, exists := uploadEntry.Extended[s3_constants.SeaweedFSSSEKMSBucketKeyEnabled]; exists && string(bucketKeyBytes) == "true" {
bucketKeyEnabled = true
}
var encryptionContext map[string]string
if contextBytes, exists := uploadEntry.Extended[s3_constants.SeaweedFSSSEKMSEncryptionContext]; exists {
if err := json.Unmarshal(contextBytes, &encryptionContext); err != nil {
glog.Errorf("Failed to parse encryption context for upload %s: %v", uploadID, err)
encryptionContext = BuildEncryptionContext(bucket, object, bucketKeyEnabled)
}
} else {
encryptionContext = BuildEncryptionContext(bucket, object, bucketKeyEnabled)
}
var baseIV []byte
if baseIVBytes, exists := uploadEntry.Extended[s3_constants.SeaweedFSSSEKMSBaseIV]; exists {
decodedIV, decodeErr := base64.StdEncoding.DecodeString(string(baseIVBytes))
if decodeErr != nil {
glog.Errorf("Failed to decode base IV for multipart upload %s: %v", uploadID, decodeErr)
} else if len(decodedIV) != s3_constants.AESBlockSize {
glog.Errorf("Invalid base IV length for multipart upload %s: expected %d bytes, got %d", uploadID, s3_constants.AESBlockSize, len(decodedIV))
} else {
baseIV = decodedIV
glog.V(4).Infof("Using stored base IV %x for multipart upload %s", baseIV[:8], uploadID)
}
}
if len(baseIV) == 0 {
glog.Errorf("No valid base IV found for SSE-KMS multipart upload %s - cannot proceed with encryption", uploadID)
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
return
}
r.Header.Set(s3_constants.AmzServerSideEncryption, "aws:kms")
r.Header.Set(s3_constants.AmzServerSideEncryptionAwsKmsKeyId, keyID)
if bucketKeyEnabled {
r.Header.Set(s3_constants.AmzServerSideEncryptionBucketKeyEnabled, "true")
}
if len(encryptionContext) > 0 {
if contextJSON, err := json.Marshal(encryptionContext); err == nil {
r.Header.Set(s3_constants.AmzServerSideEncryptionContext, base64.StdEncoding.EncodeToString(contextJSON))
}
}
r.Header.Set(s3_constants.SeaweedFSSSEKMSBaseIVHeader, base64.StdEncoding.EncodeToString(baseIV))
} else {
if err := s3a.handleSSES3MultipartHeaders(r, uploadEntry, uploadID); err != nil {
glog.Errorf("Failed to setup SSE-S3 multipart headers: %v", err)
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
return
}
}
}
// Parts inherit the checksum algorithm declared at CreateMultipartUpload
// when the request doesn't specify one; a conflicting one is rejected.
if headerName := string(uploadEntry.Extended[s3_constants.ExtChecksumAlgorithm]); headerName != "" {
if algo, reqHeaderName, code := detectRequestedChecksumAlgorithm(r); code == s3err.ErrNone {
if algo == ChecksumAlgorithmNone {
if name := checksumAlgorithmNameFromHeaderName(headerName); name != "" {
r.Header.Set(s3_constants.AmzChecksumAlgorithm, name)
}
} else if reqHeaderName != headerName {
s3err.WriteErrorResponse(w, r, s3err.ErrInvalidRequest)
return
}
}
}
filePath := s3a.genPartUploadPath(bucket, uploadID, partID)
if partID == 1 && r.Header.Get("Content-Type") == "" {
dataReader = mimeDetect(r, dataReader)
}
glog.V(2).Infof("PutObjectPart: bucket=%s, object=%s, uploadId=%s, partNumber=%d, size=%d",
bucket, object, uploadID, partID, r.ContentLength)
// MPU parts must NOT inherit the bucket's lifecycle Expiration.Days
// volume TTL: the rule targets the user-visible object, not the
// transient .uploads/<id>/<n> path, and a part write would otherwise
// start the TTL clock before CompleteMultipartUpload ever assembled
// the object. filer.conf storage rules are the opposite case: they
// place the bytes the part becomes, so they resolve against the object.
etag, errCode, sseMetadata := s3a.putToFiler(withMultipartPart(r, s3a.uploadFiler(bucket, object)), filePath, dataReader, bucket, "", partID, 0, nil, false, s3a.toFilerPath(bucket, object))
if errCode != s3err.ErrNone {
glog.Errorf("PutObjectPart: putToFiler failed with error code %v for bucket=%s, object=%s, partNumber=%d",
errCode, bucket, object, partID)
s3err.WriteErrorResponse(w, r, errCode)
return
}
// the write above re-creates a directory an abort removed mid-body
if !s3a.checkUploadStillOpen(w, r, bucket, object, uploadID) {
return
}
glog.V(2).Infof("PutObjectPart: SUCCESS - bucket=%s, object=%s, partNumber=%d, etag=%s, sseType=%s",
bucket, object, partID, etag, sseMetadata.SSEType)
setEtag(w, etag)
// Set SSE response headers for multipart uploads
s3a.setSSEResponseHeaders(w, r, sseMetadata)
writeSuccessResponseEmpty(w, r)
}
func (s3a *S3ApiServer) genUploadsFolder(bucket string) string {
return fmt.Sprintf("%s/%s", s3a.bucketDir(bucket), s3_constants.MultipartUploadsFolder)
}
// isMultipartUploadEntry tells the .uploads/<id> directory createMultipartUpload
// made from the one a part write re-created on its way to the filer: only the
// former carries the destination object key.
func isMultipartUploadEntry(entry *filer_pb.Entry) bool {
return entry != nil && len(entry.Extended[s3_constants.ExtMultipartObjectKey]) > 0
}
// checkUploadStillOpen re-checks the upload after a part landed, and removes the
// part along with the directory it resurrected when an abort was answered while
// the part was in flight. It reports whether the caller may answer success.
func (s3a *S3ApiServer) checkUploadStillOpen(w http.ResponseWriter, r *http.Request, bucket, object, uploadID string) bool {
entry, err := s3a.uploadFiler(bucket, object).getEntry(s3a.genUploadsFolder(bucket), uploadID)
if err != nil && !errors.Is(err, filer_pb.ErrNotFound) {
glog.Errorf("checkUploadStillOpen %s/%s: %v", bucket, uploadID, err)
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
return false
}
if isMultipartUploadEntry(entry) {
return true
}
// the upload is gone either way, so the client still hears NoSuchUpload
if _, code := s3a.abortMultipartUpload(&s3.AbortMultipartUploadInput{
Bucket: aws.String(bucket),
Key: objectKey(aws.String(object)),
UploadId: aws.String(uploadID),
}); code != s3err.ErrNone && code != s3err.ErrNoSuchUpload {
glog.Warningf("checkUploadStillOpen %s/%s: part left behind, cleanup failed", bucket, uploadID)
}
s3err.WriteErrorResponse(w, r, s3err.ErrNoSuchUpload)
return false
}
// getMultipartSSEAlgorithm returns the canonical SSE algorithm ("AES256" or
// "aws:kms") that was stored when the multipart upload was initiated, or ""
// if the upload entry is not found or had no SSE. It is used by the bucket
// policy engine to evaluate s3:x-amz-server-side-encryption conditions for
// UploadPart and UploadPartCopy, which do not re-send the SSE header.
func (s3a *S3ApiServer) getMultipartSSEAlgorithm(bucket, object, uploadID string) string {
entry, err := s3a.uploadFiler(bucket, object).getEntry(s3a.genUploadsFolder(bucket), uploadID)
if err != nil || entry == nil || entry.Extended == nil {
return ""
}
if _, ok := entry.Extended[s3_constants.SeaweedFSSSEKMSKeyID]; ok {
return "aws:kms"
}
if _, ok := entry.Extended[s3_constants.SeaweedFSSSES3Encryption]; ok {
return "AES256"
}
return ""
}
func (s3a *S3ApiServer) genPartUploadPath(bucket, uploadID string, partID int) string {
// Returns just the file path - no filer address needed
// Upload traffic goes directly to volume servers, not through filer
return fmt.Sprintf("%s/%s/%04d_%s.part",
s3a.genUploadsFolder(bucket), uploadID, partID, uuid.NewString())
}
// Generate uploadID hash string from object
func (s3a *S3ApiServer) generateUploadID(object string) string {
object = strings.TrimPrefix(object, "/")
h := sha1.New()
h.Write([]byte(object))
return fmt.Sprintf("%x", h.Sum(nil))
}
// Check object name and uploadID when processing multipart uploading
func (s3a *S3ApiServer) checkUploadId(object string, id string) error {
hash := s3a.generateUploadID(object)
// uploadID becomes a filer directory name. Accept the historical hash-only
// form and the exact current hash_UUID form, not arbitrary hash-prefixed paths.
valid := id == hash // legacy upload IDs generated before the UUID suffix was added
if len(id) == len(hash)+1+32 && strings.HasPrefix(id, hash+"_") {
valid = isLowerHex(id[len(hash)+1:])
}
if !valid {
glog.Errorf("object %s and uploadID %s are not matched", object, id)
return fmt.Errorf("object %s and uploadID %s are not matched", object, id)
}
return nil
}
func isLowerHex(value string) bool {
if value == "" {
return false
}
for i := 0; i < len(value); i++ {
c := value[i]
if (c < '0' || c > '9') && (c < 'a' || c > 'f') {
return false
}
}
return true
}
// Parse bucket url queries for ?uploads
func getBucketMultipartResources(values url.Values) (prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int, encodingType string, errCode s3err.ErrorCode) {
prefix = values.Get("prefix")
keyMarker = values.Get("key-marker")
uploadIDMarker = values.Get("upload-id-marker")
delimiter = values.Get("delimiter")
if values.Get("max-uploads") != "" {
parsed, err := strconv.ParseInt(values.Get("max-uploads"), 10, 32)
if err != nil {
errCode = s3err.ErrInvalidMaxUploads
}
maxUploads = int(parsed)
} else {
maxUploads = maxUploadsList
}
encodingType = values.Get("encoding-type")
return
}
// Parse object url queries
func getObjectResources(values url.Values) (uploadID string, partNumberMarker, maxParts int, encodingType string, errCode s3err.ErrorCode) {
uploadID = values.Get("uploadId")
if values.Get("part-number-marker") != "" {
parsed, err := strconv.ParseInt(values.Get("part-number-marker"), 10, 32)
if err != nil {
errCode = s3err.ErrInvalidPartNumberMarker
}
partNumberMarker = int(parsed)
}
if values.Get("max-parts") != "" {
parsed, err := strconv.ParseInt(values.Get("max-parts"), 10, 32)
if err != nil {
errCode = s3err.ErrInvalidMaxParts
}
maxParts = int(parsed)
} else {
maxParts = maxPartsList
}
encodingType = values.Get("encoding-type")
return
}
func xmlDecoder(body io.Reader, v interface{}, size int64) error {
var lbody io.Reader
if size > 0 {
lbody = io.LimitReader(body, size)
} else {
lbody = body
}
d := xml.NewDecoder(lbody)
d.CharsetReader = func(label string, input io.Reader) (io.Reader, error) {
return input, nil
}
return d.Decode(v)
}
type CompleteMultipartUpload struct {
Parts []CompletedPart `xml:"Part"`
}
type CompletedPart struct {
ETag string
PartNumber int
ChecksumResult
}
// handleSSES3MultipartHeaders handles SSE-S3 multipart upload header setup to reduce nesting complexity
func (s3a *S3ApiServer) handleSSES3MultipartHeaders(r *http.Request, uploadEntry *filer_pb.Entry, uploadID string) error {
if encryptionTypeBytes, exists := uploadEntry.Extended[s3_constants.SeaweedFSSSES3Encryption]; exists && string(encryptionTypeBytes) == s3_constants.SSEAlgorithmAES256 {
// Set SSE-S3 headers to indicate server-side encryption
r.Header.Set(s3_constants.AmzServerSideEncryption, s3_constants.SSEAlgorithmAES256)
// Retrieve and set base IV for consistent multipart encryption - REQUIRED for security
var baseIV []byte
if baseIVBytes, exists := uploadEntry.Extended[s3_constants.SeaweedFSSSES3BaseIV]; exists {
// Decode the base64 encoded base IV
decodedIV, decodeErr := base64.StdEncoding.DecodeString(string(baseIVBytes))
if decodeErr != nil {
return fmt.Errorf("failed to decode base IV for SSE-S3 multipart upload %s: %v", uploadID, decodeErr)
}
if len(decodedIV) != s3_constants.AESBlockSize {
return fmt.Errorf("invalid base IV length for SSE-S3 multipart upload %s: expected %d bytes, got %d", uploadID, s3_constants.AESBlockSize, len(decodedIV))
}
baseIV = decodedIV
glog.V(4).Infof("Using stored base IV %x for SSE-S3 multipart upload %s", baseIV[:8], uploadID)
} else {
return fmt.Errorf("no base IV found for SSE-S3 multipart upload %s - required for encryption consistency", uploadID)
}
// Retrieve and set key data for consistent multipart encryption - REQUIRED for decryption
if keyDataBytes, exists := uploadEntry.Extended[s3_constants.SeaweedFSSSES3KeyData]; exists {
// Key data is already base64 encoded, pass it directly
keyDataStr := string(keyDataBytes)
r.Header.Set(s3_constants.SeaweedFSSSES3KeyDataHeader, keyDataStr)
glog.V(4).Infof("Using stored key data for SSE-S3 multipart upload %s", uploadID)
} else {
return fmt.Errorf("no SSE-S3 key data found for multipart upload %s - required for encryption", uploadID)
}
// Pass the base IV to putToFiler via header for offset calculation
r.Header.Set(s3_constants.SeaweedFSSSES3BaseIVHeader, base64.StdEncoding.EncodeToString(baseIV))
}
return nil
}