With replication 001 and one node down, ~2/3 of volumes have their
replica on the dead node. The primary write lands locally but the
replica forward fails, so the volume returns 500 "failed to write to
replicas". #9744 made that upload fail fast so the client re-assigns,
but the client retry never fired: the reassign gate matched a fixed
list of error substrings that didn't include this one. The mount
surfaced I/O error and dropped the chunk, leaving missing lines and
null-byte gaps in an append workload while a node rebooted.
Decide reassignment by HTTP status instead of matching message text.
On the write path a volume server only 5xxs on a ReplicatedWrite
failure (local disk, replica peer down, under-replication) — all of
which a different volume dodges — so any 5xx reassigns; a no-response
transport failure (the assigned target itself is down) reassigns too;
a 4xx is a genuine client error and is surfaced. doUploadData tags its
errors with the response status via uploadStatusError, and the gate
moves from util.MultiRetry(errList) to util.RetryOnError(predicate).
This drops the fragile substring list (and the message-prefix
constants and guard test it needed); store_replicate.go is untouched.
* refactor: centralize genUploadUrl in UploadOption
Replace inline genFileUrlFn closures with operation.GenUploadUrl field:
- Add GenUploadUrl func(host, fileId) string to UploadOption struct
- Add GenUploadUrlProxy(filerAddress string) utility function
- Remove genFileUrlFn parameter from UploadWithRetry signature
- Update all callers: mount, gateway, mq, filer_copy, filer_sync
This matches the weed mount -filerProxy pattern exactly,
factorizing the URL generation logic across all consumers.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* docker release: run all platform jobs in one wave, cache rocksdb compile
Drop max-parallel so the 13 per-platform builds run together instead of two
waves of 8 (rocksdb was queuing behind the cap and starting ~8 min late).
Keep cache-to mode=max for rocksdb: its RocksDB static_lib compile is
sha-independent, so it caches across releases and stops being the ~16-min
long-pole that gates the merge fan-in. go-build variants stay mode=min.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* refactor: centralize genUploadUrl in UploadOption
Replace inline genFileUrlFn closures with operation.GenUploadUrl field:
- Add GenUploadUrl func(host, fileId) string to UploadOption struct
- Add GenUploadUrlProxy(filerAddress string) utility function
- Remove genFileUrlFn parameter from UploadWithRetry signature
- Update all callers: mount, gateway, mq, filer_copy, filer_sync
This matches the weed mount -filerProxy pattern exactly,
factorizing the URL generation logic across all consumers.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* Remove accidental ROCmFPX submodule reference
* gofmt chunk upload option block
* Preserve broker cipher and re-read proxy filer per upload attempt
Chunk uploads must keep the configured Cipher, and both the mount and broker current filer can change on failover, so build the proxy upload URL inside the closure instead of capturing the address once.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* operation: bound AssignVolume with a deadline
AssignVolume ran on context.Background(), so when the filer is overwhelmed
the RPC could block indefinitely and wedge every caller holding the
connection. Give it a 30s deadline so a stuck assign fails and the caller's
retry/error path runs instead of hanging forever.
* mount: abort flush when the FUSE request is interrupted
On close(), a killed process blocks in fuse_flush waiting for the mount to
answer. doFlush ran its metadata CreateEntry on context.Background() and
ignored the kernel interrupt channel, so against an overwhelmed filer the
flush never completed and the process stayed in uninterruptible sleep --
making the pod un-killable.
Derive a context from the FUSE cancel channel in Flush/Fsync and thread it
through doFlush -> flushMetadataToFiler -> streamCreateEntry; the retry loop
stops as soon as the context is cancelled. Release and the pre-rename flush
keep a non-cancellable context since they must finish regardless.
* operation: harden the AssignVolume timeout test
Make the test double's signal send non-blocking and bound the receive with a
timeout so a regression can't wedge the test instead of failing it.
* preserve compression state during replication
* explain why ParseUpload skips compression for replica writes
* fix data race on err result in FetchAndWriteNeedle
The local-write and replica-write goroutines all wrote the named err return under an unsynchronized err==nil check. Give each goroutine its own error slot and combine after wg.Wait(): local error wins, then the first replica failure.
* skip redundant decompression of compressed needles during replication
doUploadData decompressed a compressed input only to report the clear-data length on UploadResult.Size, which both replication callers discard. Skip the decompress when IsReplication.
* operation: bound upload retries and honor context cancellation
retriedUploadData hardcoded 3 attempts and an uninterruptible backoff
sleep. A synchronous replica write to a dead host therefore paid the
full dial timeout three times over before failing.
Add UploadOption.MaxAttempts (<=0 keeps the default of 3) so callers can
cap attempts, and make the loop return as soon as the context is
cancelled so an abandoned upload unwinds instead of retrying.
* topology: fail replica writes fast when a replica is unreachable
DistributedOperation already returns on the first error, but a single
dead replica is itself the slow result: its goroutine retries the upload
three times through the dial timeout (~30s) before any error surfaces,
stalling the originating client write the whole time.
Make the replica write a single attempt (MaxAttempts=1) so a dead
replica fails after one dial timeout instead of three, and thread a
context into DistributedOperation that is cancelled once the outcome is
decided, so a healthy replica is no longer held hostage by one stalled
in a dial. The originating client write is what retries.
* topology: keep replica deletes off the client request context
ReplicatedDelete runs after the local needle is already deleted. Driving
the replica deletes off r.Context() means a client disconnect cancels
them and orphans needles on the replicas, so use a background context.
* operation, topology: trim comments on the replica fail-fast path
* fix(upload): rewind request body when retrying on connection reset (#9139)
When httpClient.Do() returned "connection reset by peer" or "use of
closed network connection", upload_content retried with the same
*http.Request. But the body is a *bytes.Reader the first attempt
already consumed, so the retry sent 0 bytes and Go's transport
surfaced "http: ContentLength=N with Body length 0".
http.NewRequestWithContext populates req.GetBody for *bytes.Reader
bodies; use it to attach a fresh body before retrying.
Reproduces the issue with a unit test (asserts both attempts see
the same payload bytes); the test fails without the fix.
* upload: skip inner retry when body cannot be rewound
Per review feedback: if req.GetBody is nil or returns an error, the
inner retry would call Do(req) with an already-consumed body and the
"connection reset" error would be replaced by the misleading
"ContentLength=N with Body length 0" — the very symptom this PR set
out to fix. Skip the inner retry on rewind failure and let the outer
retriedUploadData loop reissue with a fresh request, and log when
GetBody is unavailable for observability.
* upload: log the actual transport error in the inner retry log line
Per review feedback: the diagnostic glog at the top of the inner
retry branch was logging postErr — the request-construction error
from http.NewRequestWithContext, which is necessarily nil there
because the function returns early at line 423 if it isn't.
Operators were seeing "<nil>" instead of the transient transport
error that triggered the rewind. Reference post_err so the
connection-reset / closed-connection cause is actually visible.
Chris Lu