* fix: reject overflowing needle ID deltas
Problem: Parsing a file ID with a delta can wrap a valid maximum needle ID back to zero without returning an error.
Root cause: Needle.ParsePath added the parsed uint64 delta without checking whether the sum exceeded the needle ID range.
Fix: Compare the delta with the remaining uint64 capacity before addition and return a contextual overflow error when it does not fit.
Validation: go test ./weed/storage/needle -run ^TestNeedleParsePathRejectsDeltaOverflow$ -count=1; go test ./weed/storage/needle -count=1; git diff --check 10cdaf381875492a2c752d1038797e96ff18208f..HEAD
Co-authored-by: Codex <noreply@openai.com>
* fix: propagate needle ID delta parse errors
Co-authored-by: Codex <noreply@openai.com>
* print the needle id in hex in the delta overflow error
* batch delete: keep processing after a cookie mismatch
* rust volume: reject overflowing needle id deltas
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
LastIndex returns -1 when the separator is missing and can return any
position when both are present. A path like /vol/file.jpg,abc gives
dotSep<commaSep, so path[commaSep+1:dotSep] slices with start>end and
panics. Only treat the dot as an extension boundary when it sits after
the comma.
Reverts #9426. The header had the volume server record OriginalDataSize
from a value set by the multipart upstream — a client-controlled
metadata field. On a volume server that isn't JWT-protected, a caller
can lie and the needle stores the lie; bounds-checking the value
doesn't change the trust shape, only the magnitude of the lie. Derive
the size from the bytes again.
The optimization only fired on multipart Content-Encoding: gzip parts
(the s3 chunk-copy fast path), a narrow case that doesn't justify the
trust dependency. A future change can attack the same heap profile by
stream-decompressing to count bytes instead of materializing the
uncompressed slice — no client-trust surface.
Refs https://github.com/seaweedfs/seaweedfs/pull/9426#issuecomment-4417862793
* feat(volume): X-Seaweedfs-Original-Size hint skips redundant gunzip
The full-chunk gzip pass-through (#9425) fixed source-volume
decompression but moved the cost to the destination volume:
parseUpload still ran util.DecompressData on the forwarded gzipped
bytes, just to learn the uncompressed length so it could record
OriginalDataSize in the needle metadata. For 6-way concurrent 64 MiB
UploadPartCopy that decompress-and-discard pass dominated the
remaining heap profile after the streaming chain landed (~297 MiB
inuse via bytes.Buffer.ReadFrom inside util.GunzipStream).
Add an X-Seaweedfs-Original-Size header on the multipart part. When
the upstream sets it (the s3 chunk-copy fast path always knows the
uncompressed size — it's the source chunk's logical size) and no
Content-MD5 verification is requested (which would require
decompressed bytes to compute against), parseUpload uses the hint
directly and skips the decompress.
Header is X-* prefixed (not Seaweed-*) so it doesn't get auto-stored
as a needle pair by PairNamePrefix.
Backward compatible:
- old s3 servers don't set the header, parseUpload decompresses as
before
- new s3 servers talking to old volumes: header is ignored, volume
decompresses
- bad header values (non-numeric, negative, garbage) fall back to the
existing decompress path
End-to-end repro impact (512 MiB src, 6 parallel UploadPartCopy,
post-#9420/#9421/#9422/#9424/#9425 baseline):
RSS, round 2: 1149 MiB → 594 MiB
heap inuse_space: 545 MiB → 349 MiB
HeapSys: 1.35 GiB → 777 MiB
TotalAlloc cum: ~9 GiB → 3.5 GiB
Total reduction from pre-#9420 baseline: 3134 → 594 MiB (-81%).
Test exercises the four matrix corners (hint+no-MD5,
hint+part-MD5, hint+req-MD5, no-hint, garbage-hint) and bounds
allocation per case so a regression that re-introduces the
unconditional decompress fails the hint-present-no-MD5 case.
* review: bound X-Seaweedfs-Original-Size by sizeLimit and uint32
CodeQL on PR 9426 traced a new taint flow: the strconv.Atoi(hint)
value flows into pu.OriginalDataSize -> originalSize -> the existing
uint32(originalSize) cast in volume_server_handlers_write.go:73. The
cast was always there but its input was previously bounded by the
ParseUpload read path (capped at sizeLimit). Adding a user-controlled
hint bypassed that bound, so a malicious header could overflow the
uint32 silently.
Bound the hint at parse time by sizeLimit (the largest needle this
volume will accept anyway) and by math.MaxUint32 (belt-and-suspenders
in case sizeLimit is configured > 4 GiB).
