* preserve compression state during replication
* explain why ParseUpload skips compression for replica writes
* fix data race on err result in FetchAndWriteNeedle
The local-write and replica-write goroutines all wrote the named err return under an unsynchronized err==nil check. Give each goroutine its own error slot and combine after wg.Wait(): local error wins, then the first replica failure.
* skip redundant decompression of compressed needles during replication
doUploadData decompressed a compressed input only to report the clear-data length on UploadResult.Size, which both replication callers discard. Skip the decompress when IsReplication.
Problem: Multipart uploads where the first part was a form field and a later part contained the file used the first part's Content-Type for the file metadata.
Root cause: After finding a later part with a filename, parseUpload copied data and MD5 from part2 but read Content-Type from the original part variable.
Fix: Read Content-Type from the discovered file part.
Reproduction: go test ./weed/storage/needle -run TestParseUploadUsesDiscoveredFilePartContentType -count=1 failed before the fix because the parsed MIME type was text/plain instead of application/x-seaweed-test.
Validation: go test ./weed/storage/needle -run TestParseUploadUsesDiscoveredFilePartContentType -count=1; go test ./weed/storage/needle -count=1; git diff --check; git diff --cached --check
ParseUpload runs util.DecompressData on every gzipped multipart upload
just to record OriginalDataSize. The decompress materializes the full
uncompressed slice via bytes.Buffer.ReadFrom inside util.GunzipStream;
for a 64 MiB chunk that's a ~128 MiB heap spike per call (geometric
grow). On 6-way concurrent UploadPartCopy the spike dominated the
remaining heap profile after #9420/#9421/#9422/#9424/#9425.
When no Content-MD5 verification is requested the uncompressed bytes
aren't needed — only the length is. Stream the gunzip through
io.Discard and count: the pooled gzip.Reader's working set replaces
the materialized slice.
Unlike the previous attempt in #9426 the size still comes from the
real bytes, not from a client-set header.
TotalAlloc per call, 4 MiB uncompressed body:
materialize (was, still runs when MD5 is set): ~16.8 MiB
stream-count (no MD5): ~28 KiB
Refs #6541, #9426 (reverted in #9432).
Reverts #9426. The header had the volume server record OriginalDataSize
from a value set by the multipart upstream — a client-controlled
metadata field. On a volume server that isn't JWT-protected, a caller
can lie and the needle stores the lie; bounds-checking the value
doesn't change the trust shape, only the magnitude of the lie. Derive
the size from the bytes again.
The optimization only fired on multipart Content-Encoding: gzip parts
(the s3 chunk-copy fast path), a narrow case that doesn't justify the
trust dependency. A future change can attack the same heap profile by
stream-decompressing to count bytes instead of materializing the
uncompressed slice — no client-trust surface.
Refs https://github.com/seaweedfs/seaweedfs/pull/9426#issuecomment-4417862793
* feat(volume): X-Seaweedfs-Original-Size hint skips redundant gunzip
The full-chunk gzip pass-through (#9425) fixed source-volume
decompression but moved the cost to the destination volume:
parseUpload still ran util.DecompressData on the forwarded gzipped
bytes, just to learn the uncompressed length so it could record
OriginalDataSize in the needle metadata. For 6-way concurrent 64 MiB
UploadPartCopy that decompress-and-discard pass dominated the
remaining heap profile after the streaming chain landed (~297 MiB
inuse via bytes.Buffer.ReadFrom inside util.GunzipStream).
Add an X-Seaweedfs-Original-Size header on the multipart part. When
the upstream sets it (the s3 chunk-copy fast path always knows the
uncompressed size — it's the source chunk's logical size) and no
Content-MD5 verification is requested (which would require
decompressed bytes to compute against), parseUpload uses the hint
directly and skips the decompress.
Header is X-* prefixed (not Seaweed-*) so it doesn't get auto-stored
as a needle pair by PairNamePrefix.
Backward compatible:
- old s3 servers don't set the header, parseUpload decompresses as
before
- new s3 servers talking to old volumes: header is ignored, volume
decompresses
- bad header values (non-numeric, negative, garbage) fall back to the
existing decompress path
End-to-end repro impact (512 MiB src, 6 parallel UploadPartCopy,
post-#9420/#9421/#9422/#9424/#9425 baseline):
RSS, round 2: 1149 MiB → 594 MiB
heap inuse_space: 545 MiB → 349 MiB
HeapSys: 1.35 GiB → 777 MiB
TotalAlloc cum: ~9 GiB → 3.5 GiB
Total reduction from pre-#9420 baseline: 3134 → 594 MiB (-81%).
Test exercises the four matrix corners (hint+no-MD5,
hint+part-MD5, hint+req-MD5, no-hint, garbage-hint) and bounds
allocation per case so a regression that re-introduces the
unconditional decompress fails the hint-present-no-MD5 case.
* review: bound X-Seaweedfs-Original-Size by sizeLimit and uint32
CodeQL on PR 9426 traced a new taint flow: the strconv.Atoi(hint)
value flows into pu.OriginalDataSize -> originalSize -> the existing
uint32(originalSize) cast in volume_server_handlers_write.go:73. The
cast was always there but its input was previously bounded by the
ParseUpload read path (capped at sizeLimit). Adding a user-controlled
hint bypassed that bound, so a malicious header could overflow the
uint32 silently.
Bound the hint at parse time by sizeLimit (the largest needle this
volume will accept anyway) and by math.MaxUint32 (belt-and-suspenders
in case sizeLimit is configured > 4 GiB).
* fix(volume): pre-size ParseUpload buffer to request ContentLength
The volume server's PostHandler reads the multipart upload body via
bytes.Buffer.ReadFrom inside parseUpload. The buffer comes from a
sync.Pool and may have cap=0 when the pool dropped the prior entry,
which makes ReadFrom geometric-grow on each chunk: a 64 MiB upload
allocates roughly 1+2+4+...+64 ≈ 128 MiB just to receive the body.
Under concurrent uploads (every s3 chunk-copy lands here on the
destination volume) this is one of the main contributors to the
runaway-RSS pattern in #6541 — pprof shows ~458 MiB cum in
parseUpload's bytes.Buffer.ReadFrom under Harbor-style assemble load.
Grow the buffer once up front, bounded by the existing sizeLimit so
a misreported Content-Length can't over-allocate. The receive then
fills in place.
Add a regression test that drives ParseUpload with a 16 MiB multipart
body and bounds TotalAlloc at 1.5x the chunk size (pre-fix measures
~4x, so the bound trips deterministically).
* fix(volume): guard ParseUpload pre-grow against int overflow on 32-bit
Address PR review feedback: r.ContentLength is int64, and on 32-bit
platforms int is 32 bits wide, so int(r.ContentLength) for a value above
math.MaxInt32 wraps negative and bytes.Buffer.Grow panics with
"bytes.Buffer.Grow: negative count". Skip the pre-grow optimization in
that range; the existing geometric-grow path remains correct, just
slightly more allocator pressure for that one call.
64-bit platforms (math.MaxInt == math.MaxInt64) are unaffected — the
guard only kicks in for 32-bit builds with very large sizeLimit.
* fix(volume): cap ParseUpload pre-grow at 4 MiB to bound DoS surface
Address PR review: pre-growing the receive buffer to r.ContentLength
trusts the header before any body bytes arrive. A bad header or slow /
idle client could declare a large Content-Length up to sizeLimit (256
MiB by default for volume writes) and force per-request preallocation
without sending data, turning many concurrent slow connections into
avoidable memory pressure.
Cap eager pre-grow at maxEagerPreGrow (4 MiB). Larger uploads still
benefit from the higher starting cap and fall back to ReadFrom's grow
path for the remainder. Per-request waste from a misreported
Content-Length is now bounded at 4 MiB regardless of sizeLimit.
Extract the policy as eagerPreGrow so the unit test can exercise the
gates structurally — replaces the prior TotalAlloc bound (which became
uninformative once savings were capped at 4 MiB).
* fix multipart etag
* address comments
* clean up
* clean up
* optimization
* address comments
* unquoted etag
* dedup
* upgrade
* clean
* etag
* return quoted tag
* quoted etag
* debug
* s3api: unify ETag retrieval and quoting across handlers
Refactor newListEntry to take *S3ApiServer and use getObjectETag,
and update setResponseHeaders to use the same logic. This ensures
consistent ETags are returned for both listing and direct access.
* s3api: implement ListObjects deduplication for versioned buckets
Handle duplicate entries between the main path and the .versions
directory by prioritizing the latest version when bucket versioning
is enabled.
* s3api: cleanup stale main file entries during versioned uploads
Add explicit deletion of pre-existing "main" files when creating new
versions in versioned buckets. This prevents stale entries from
appearing in bucket listings and ensures consistency.
* s3api: fix cleanup code placement in versioned uploads
Correct the placement of rm calls in completeMultipartUpload and
putVersionedObject to ensure stale main files are properly deleted
during versioned uploads.
* s3api: improve getObjectETag fallback for empty ExtETagKey
Ensure that when ExtETagKey exists but contains an empty value,
the function falls through to MD5/chunk-based calculation instead
of returning an empty string.
* s3api: fix test files for new newListEntry signature
Update test files to use the new newListEntry signature where the
first parameter is *S3ApiServer. Created mockS3ApiServer to properly
test owner display name lookup functionality.
* s3api: use filer.ETag for consistent Md5 handling in getEtagFromEntry
Change getEtagFromEntry fallback to use filer.ETag(entry) instead of
filer.ETagChunks to ensure legacy entries with Attributes.Md5 are
handled consistently with the rest of the codebase.
* s3api: optimize list logic and fix conditional header logging
- Hoist bucket versioning check out of per-entry callback to avoid
repeated getVersioningState calls
- Extract appendOrDedup helper function to eliminate duplicate
dedup/append logic across multiple code paths
- Change If-Match mismatch logging from glog.Errorf to glog.V(3).Infof
and remove DEBUG prefix for consistency
* s3api: fix test mock to properly initialize IAM accounts
Fixed nil pointer dereference in TestNewListEntryOwnerDisplayName by
directly initializing the IdentityAccessManagement.accounts map in the
test setup. This ensures newListEntry can properly look up account
display names without panicking.
* cleanup
* s3api: remove premature main file cleanup in versioned uploads
Removed incorrect cleanup logic that was deleting main files during
versioned uploads. This was causing test failures because it deleted
objects that should have been preserved as null versions when
versioning was first enabled. The deduplication logic in listing is
sufficient to handle duplicate entries without deleting files during
upload.
* s3api: add empty-value guard to getEtagFromEntry
Added the same empty-value guard used in getObjectETag to prevent
returning quoted empty strings. When ExtETagKey exists but is empty,
the function now falls through to filer.ETag calculation instead of
returning "".
* s3api: fix listing of directory key objects with matching prefix
Revert prefix handling logic to use strings.TrimPrefix instead of
checking HasPrefix with empty string result. This ensures that when a
directory key object exactly matches the prefix (e.g. prefix="dir/",
object="dir/"), it is correctly handled as a regular entry instead of
being skipped or incorrectly processed as a common prefix. Also fixed
missing variable definition.
* s3api: refactor list inline dedup to use appendOrDedup helper
Refactored the inline deduplication logic in listFilerEntries to use the
shared appendOrDedup helper function. This ensures consistent behavior
and reduces code duplication.
* test: fix port allocation race in s3tables integration test
Updated startMiniCluster to find all required ports simultaneously using
findAvailablePorts instead of sequentially. This prevents race conditions
where the OS reallocates a port that was just released, causing multiple
services (e.g. Filer and Volume) to be assigned the same port and fail
to start.
fix: normalize Windows backslash paths in file uploads
When uploading files from a Windows client to a Linux server,
file paths containing backslashes were not being properly interpreted as
directory separators. This caused files intended for subdirectories to be
created in the root directory with backslashes in their filenames.
Changes:
- Add util.CleanWindowsPath and util.CleanWindowsPathBase helper functions
in weed/util/fullpath.go for reusable path normalization
- Use path.Join/path.Clean/path.Base instead of filepath equivalents
for URL path semantics (filepath is OS-specific)
- Apply normalization in weed admin handlers and filer upload parsing
Fixes#7628
* fix: parse filename in PUT + refactor
* fix: master iu public url with http
* fix: better parsing and handle disposition header
* fix: take mime type from file extension if not set
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
The mime type is always the value passed in.
Compress or not depends on the content detection, file name extension, and compression ratio.
If the content is already compressed, need to know the content size.
Chris Lu