Commit Graph
4 Commits
Author SHA1 Message Date
Chris Lu 94357ac6a9 [volume] preserve compression state during replication (#9946)
* preserve compression state during replication

* explain why ParseUpload skips compression for replica writes

* fix data race on err result in FetchAndWriteNeedle

The local-write and replica-write goroutines all wrote the named err return under an unsynchronized err==nil check. Give each goroutine its own error slot and combine after wg.Wait(): local error wins, then the first replica failure.

* skip redundant decompression of compressed needles during replication

doUploadData decompressed a compressed input only to report the clear-data length on UploadResult.Size, which both replication callers discard. Skip the decompress when IsReplication.
2026-06-13 21:52:59 -07:00
Chris Lu 4bf27278fa topology: fail replica writes fast when a replica is unreachable (#9744)
* operation: bound upload retries and honor context cancellation

retriedUploadData hardcoded 3 attempts and an uninterruptible backoff
sleep. A synchronous replica write to a dead host therefore paid the
full dial timeout three times over before failing.

Add UploadOption.MaxAttempts (<=0 keeps the default of 3) so callers can
cap attempts, and make the loop return as soon as the context is
cancelled so an abandoned upload unwinds instead of retrying.

* topology: fail replica writes fast when a replica is unreachable

DistributedOperation already returns on the first error, but a single
dead replica is itself the slow result: its goroutine retries the upload
three times through the dial timeout (~30s) before any error surfaces,
stalling the originating client write the whole time.

Make the replica write a single attempt (MaxAttempts=1) so a dead
replica fails after one dial timeout instead of three, and thread a
context into DistributedOperation that is cancelled once the outcome is
decided, so a healthy replica is no longer held hostage by one stalled
in a dial. The originating client write is what retries.

* topology: keep replica deletes off the client request context

ReplicatedDelete runs after the local needle is already deleted. Driving
the replica deletes off r.Context() means a client disconnect cancels
them and orphans needles on the replicas, so use a background context.

* operation, topology: trim comments on the replica fail-fast path
2026-05-30 10:45:02 -07:00
Chris Lu 0716577ec8 fix(upload): rewind request body when retrying on connection reset (#9139) (#9222)
* fix(upload): rewind request body when retrying on connection reset (#9139)

When httpClient.Do() returned "connection reset by peer" or "use of
closed network connection", upload_content retried with the same
*http.Request. But the body is a *bytes.Reader the first attempt
already consumed, so the retry sent 0 bytes and Go's transport
surfaced "http: ContentLength=N with Body length 0".

http.NewRequestWithContext populates req.GetBody for *bytes.Reader
bodies; use it to attach a fresh body before retrying.

Reproduces the issue with a unit test (asserts both attempts see
the same payload bytes); the test fails without the fix.

* upload: skip inner retry when body cannot be rewound

Per review feedback: if req.GetBody is nil or returns an error, the
inner retry would call Do(req) with an already-consumed body and the
"connection reset" error would be replaced by the misleading
"ContentLength=N with Body length 0" — the very symptom this PR set
out to fix. Skip the inner retry on rewind failure and let the outer
retriedUploadData loop reissue with a fresh request, and log when
GetBody is unavailable for observability.

* upload: log the actual transport error in the inner retry log line

Per review feedback: the diagnostic glog at the top of the inner
retry branch was logging postErr — the request-construction error
from http.NewRequestWithContext, which is necessarily nil there
because the function returns early at line 423 if it isn't.
Operators were seeing "<nil>" instead of the transient transport
error that triggered the rewind. Reference post_err so the
connection-reset / closed-connection cause is actually visible.
2026-04-26 02:17:55 -07:00
Chris Lu 7d426d2a56 Retry uploader on volume full (#8853)
* retry uploader on volume full

* drop unused upload retry helper
2026-03-30 13:32:31 -07:00