Introduce security.BearerPrefix ("Bearer ", RFC 6750) and use it
everywhere an "Authorization: Bearer <token>" header is constructed,
replacing the scattered "BEARER "/"Bearer " string literals. SeaweedFS
matches the scheme case-insensitively when parsing (security.GetJwt), so
behavior is unchanged; this removes the magic string and settles the
casing on the standard form. The parser's upper-case comparison stays as
is on purpose.
Building on the io.Pipe streaming chunk copy: when a copy operation
covers an entire source chunk (the common case for Harbor's
part-size = chunk-size assemble pattern), ask the source volume for
compressed bytes via Accept-Encoding: gzip and forward them to the
destination as-is.
This trades a Range fetch (where the volume decompresses the chunk
internally to satisfy the byte range) for a full-chunk fetch that
returns whatever wire bytes the chunk is stored as. For gzipped
chunks the source volume avoids the decompression entirely; we never
allocate a chunk-sized decompress buffer.
Implementation: build the source GET directly instead of going
through ReadUrlAsStream, because that helper auto-decompresses gzip
responses (which would defeat the point). Trust the response's
Content-Encoding header over caller hints — for partial ranges the
volume always returns raw bytes regardless of how the chunk is
stored, so labeling those as gzip would corrupt subsequent reads.
End-to-end repro impact (512 MiB src, 6 parallel UploadPartCopy):
+ #9420/#9421/#9422 : 2236 MiB
+ io.Pipe streaming : 1521 MiB
+ this commit : 1149 MiB (round 2 RSS, perfectly flat)
Round 3 now completes (was hitting volume-full before, since
chunks took up uncompressed space on disk; we now store the gzipped
chunks the volume gives us, which fit in the test's 8 GiB volume
budget).
Heap inuse_space (after force GC):
before all: ~1.5 GiB
this PR: 266 MiB
Volume-side bytes.Buffer.ReadFrom inuse:
before: 611 MiB
streaming: 571 MiB
this PR: 297 MiB (now in destination-volume parseUpload's
size-hint decompression — separate
optimization opportunity for a hint header)
* fix: cap pool retention so chunk-copy buffers don't hoard memory
Two pool-retention sites kept the runaway-RSS pattern in #6541 visible
even after #9420 and #9421:
* weed/util/buffer_pool: SyncPoolPutBuffer dropped a buffer back into
sync.Pool regardless of how big it had grown. After a 64 MiB chunk
upload through volume.PostHandler -> needle.ParseUpload, the pool
hoarded a 64 MiB byte array per cached entry for the rest of the
process's lifetime. Cap retention at 4 MiB; oversized buffers are
dropped so GC can reclaim the backing array.
* weed/s3api/...copy.go: uploadChunkData left UploadOption.BytesBuffer
unset, so operation.upload_content fell back to the package-global
valyala/bytebufferpool. That pool also retains high-water buffers
forever, and concurrent UploadPartCopy filled it with one chunk-sized
buffer per concurrent upload. Provide a fresh per-call bytes.Buffer
pre-sized to chunk + multipart framing; it's GC'd as soon as the
upload returns.
Tests:
- weed/util/buffer_pool/sync_pool_test.go: pin the cap (oversized
buffers don't round-trip), the inverse (right-sized buffers do), and
nil-safety.
- weed/s3api/...copy_chunk_upload_test.go: extract newChunkUploadOption
and pin that BytesBuffer is always non-nil and pre-sized, and that
each call gets a distinct buffer.
* feat(s3api): stream chunk copy via io.Pipe to cut peak working set
Final piece for #6541. The buffered chunk-copy path holds two
chunk-sized buffers per copy in flight (download buffer + multipart-
encoded upload buffer). Under concurrent UploadPartCopy that put a
floor on RSS at concurrency × 2 × chunk_size — about 768 MiB for the
6-way / 64 MiB Harbor-style assemble repro, even after the previous
pool/retention fixes.
Replace the buffered path with an io.Pipe between the source GET and
the destination POST: ReadUrlAsStream pumps data into the pipe via a
multipart.Writer, the http.Client reads from the pipe end and POSTs
the body. In-flight per copy is now ~32 KiB (pipe hand-off + http
buffers), regardless of chunk size.
The streaming path is gated by canStreamCopyChunk: only used when no
in-transit transformation is needed (no per-chunk CipherKey, no SSE).
SSE-C / SSE-KMS / SSE-S3 paths still go through the buffered path,
which already handles re-encryption correctly.
Benchmarks (Apple M4, httptest source/dest, B/op = bytes per copy):
Buffered 1 MiB: 6.0 MB B/op, 443 MB/s
Streamed 1 MiB: 374 KB B/op, 727 MB/s
Buffered 8 MiB: 56 MB B/op, 559 MB/s
Streamed 8 MiB: 379 KB B/op, 1138 MB/s
Buffered 64 MiB: 455 MB B/op, 718 MB/s
Streamed 64 MiB: 304 KB B/op, 1387 MB/s
End-to-end repro (512 MiB src, 6 parallel UploadPartCopy):
pre-#9420 RSS round 2: 3134 MiB
+ #9420/#9421/#9422 : 2236 MiB
+ this PR : 1521 MiB
heap inuse_space : 350 MiB (was 1422 / 1187 MiB)
HeapSys (MemStats) : 1.74 GiB (was 2.49 GiB)
* review: surface shouldRetry, add int32 guard, drop redundant drains
Address review on PR 9424:
* coderabbit (HIGH, line 122): ReadUrlAsStream can set shouldRetry=true
with readErr=nil. Before this fix, that fell through to mw.Close()
and the destination POST succeeded against a possibly-truncated
multipart body. Mirror downloadChunkData's explicit check and
surface shouldRetry as a producer error so the dst POST aborts.
* gemini (line 98): chunk size is int64 but ReadUrlAsStream takes int.
Reject sizes above MaxInt32 up front so the int(size) cast can't
truncate negative on 32-bit platforms — same guard downloadChunkData
uses.
* gemini (line 151): util_http.CloseResponse already drains the body
(io.Copy(io.Discard, ...) inside the helper) before closing, so the
manual io.Copy drains we added are redundant. Drop them.
* review: cancel source GET when destination POST fails
Address coderabbit review (line 165 / second pass on PR 9424): when
the POST leg fails or returns an error status, closing pipeReader
only fails the producer's *writes*. ReadUrlAsStream's own read loop
runs under the parent ctx, so it keeps draining the source body in
the background until EOF — wasting source-volume bandwidth and CPU
on a copy that's already failed.
Wrap streamCopyChunkRange in a child context cancelled on return.
ReadUrlAsStream checks ctx.Done() per 256 KiB tick, so the in-flight
read aborts on the next iteration once the function returns. The POST
also moves to streamCtx so the in-flight request can be cancelled the
same way if the producer fails first.
Defer-cancel runs after both legs return, so the success path still
sends EOF cleanly through pipeWriter.Close before cancellation.
Chris Lu