mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-11 00:37:52 +02:00
Chris Lu
542495f1f1
s3: fold large chunk lists into manifest chunks on the direct write path The S3 gateway uploads chunks itself and hands the filer a fully prepared entry. On the routed write path (ObjectTransaction) the filer stores that entry as-is, so a large PutObject or CompleteMultipartUpload persisted its whole flat chunk list - a 900GB object carries 120k chunk references in one entry. Manifestize on the gateway before the entry is written, the same way mount, WebDAV, and filer.copy prepare theirs. Multipart part boundaries also record byte offsets now: the stored chunk indexes stop matching the entry once the list is folded, and partNumber reads plus GetObjectAttributes prefer the offsets. Legacy index-only records still work, with bounds checks instead of a possible panic. Copy paths resolve a manifested source into data chunks before their per-chunk copy loops - copying a manifest chunk raw would store its blob as object data still pointing at the source - and a large copied list is folded again on the destination. Completion likewise resolves manifest chunks a part entry may carry (the filer folds an oversized UploadPartCopy range) before rebasing part offsets.