mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-24 23:10:41 +02:00
5e01caa52286c8baf0aeff8a205c953a9e80632a
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandDevin
|
673d1a02e0 |
operation: euthanize pooled upload conns that stall far below a fast host (#12003)
* operation: euthanize pooled upload conns that stall far below a fast host A pooled keep-alive conn can wedge and drain the request body at a trickle while fresh conns to the same volume run at full rate; every later upload routed onto it stalls for tens of seconds, indefinitely. Track per-host healthy-rate uploads (>=1MiB/s on a >=256KiB body) and, on a host that was fast within the last minute, cancel an upload still in flight past max(10s, size/256KiB/s): the transport discards the wedged conn and the existing retry dials a fresh one. A watchdog kill also clears the fast mark so the retried attempt runs unbounded, and hosts that never prove fast keep today's unbounded behavior - a chronically slow link is never penalized. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * operation: keep the stall watchdog armed through the response read uploadDone was set as soon as Do() returned, so a conn that accepted the request body then wedged on the reply escaped the watchdog and could park the upload for hours — the same failure this watchdog exists to kill. Deferring the done-store keeps the timer live until the function returns, so a stalled io.ReadAll is cancelled and retried like a stalled write. The redundant post_err recheck goes with it. * operation: pin watchdog coverage of a stalled response read * operation: disarm the stall watchdog for an upload's remaining attempts Once the watchdog has cancelled an attempt, a concurrent healthy upload to the same host can restore the fast mark during the retry backoff. The re-armed watchdog would then cancel the retry too, and a large upload that merely exceeds the floor could burn every attempt. Track the watchdog kill on uploadStatusError and carry a stallWatchdogDisarmed flag through retriedUploadData so the rest of that upload's attempts run unbounded; other uploads to the host keep their watchdog. * operation: pin watchdog disarm on retry, fresh-dial, and slow-upload cases * operation: drop the send-stall watchdog once the request body drains A wedged pooled conn is only blameworthy while it is still trickling the request body; once the body is on the wire the wait is server-side commit, which the transport's response header timeout already bounds. Cancelling then would fail a healthy write whose replication or fsync is simply slow. Track body EOF through a wrapping reader, gate the send-side timer on it, and re-arm the watchdog only around the response-body read, which the header timeout does not cover. * operation: prove a host fast only on a verified upload A quick rejection moves bytes no faster than the wire did, so mark the host from the body-send duration and only after the response checks out. * operation: count an abandoned request body as sent The transport closes a request body it never finished draining when the server answers early, so EOF alone under-reports the send boundary and the stall watchdog could still blame the conn during the reply wait. * operation: scope the body-sent mark to the current attempt A lagging Close or EOF from the first attempt's body could stamp the shared mark after the inner retry reset it, disarming the watchdog for a send that never happened; each attempt now points the watchdog at its own sent timestamp. * operation: drop only the fast mark the stall timer observed A concurrent upload can re-prove the host while a stalled attempt is in flight; deleting the key outright would erase that newer mark, so the watchdog now removes only the value it armed against. * operation: keep the stall disarm on a per-call upload option retriedUploadData marked the caller's UploadOption, so a caller reusing it across UploadData calls would start the next upload with the watchdog off and lose the cancellation this adds. * test: stall watchdog retry must fail if the conn gets cancelled The fake retry slept through any cancellation, so a re-armed watchdog firing mid-retry still let the upload succeed — the guard was untested. Park on the request context like a real wedged conn so a re-armed watchdog cancels the attempt and fails the test. * operation: prove the fast-host mark with elapsed-to-response Body-drain time only measures when the transport consumed the reader: kernel socket buffers can swallow a small body instantly on a slow link, inflating the rate and marking the host fast. The next upload then gets its conn cancelled mid-send despite making progress. Elapsed-to-response bounds remote speed from below — the server cannot answer before the body lands — so the mark can only under-mark, never over-mark. * operation: scale the stall budget off the host's measured rate Fixed bps limits fit no link well: 256KiB/s never fires on a slow link and waits far too long on a fast one. Record the wire rate of each sizeable upload and let a later upload run for slowFactor times the body's span at that rate — a wedged conn shows as an order-of-magnitude drop on any link speed. The recorded rate uses the wire body size so a pre-compressed payload no longer inflates the mark. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Ilia Demianenko
|
c58bd0dfd3 |
s3: honor assignment fsync in UploadWithRetry (#11449)
* fix: honor assignment fsync in UploadWithRetry * Tests feedback |
||
|
Chris Lu
|
3431bdcb74 |
s3: fix UploadPartCopy with volume-data encryption (#10971)
* operation: give an encrypted chunk the plaintext ETag With -encryptVolumeData the volume server stores ciphertext, so it cannot echo a Content-MD5 back and the chunk lands with an empty ETag. Every ETag derived from those chunks then comes out empty for a single chunk, or d41d8cd98f00b204e9800998ecf8427e-N for several. The caller already hashes the plaintext to send as Content-MD5, so keep that digest as the chunk ETag instead of dropping it, and compute it for a WantMd5 caller under cipher too. * s3: re-encrypt a part copy from a volume-encrypted source UploadPartCopy raw-copies source chunks when neither side uses SSE, which also caught -encryptVolumeData sources. Those chunks are ciphertext a whole-chunk cipher key decrypts, so copying a byte range out of one and keeping the key leaves a destination that fails authentication on GET, and the copied chunks carry no ETag for the part result to report. Route them through the re-encrypting path already used for SSE: it reads the source as plaintext, hashes the part, and writes the destination under the gateway's own encryption. * s3: fetch only the range a part copy asked for The re-encrypting UploadPartCopy path opened the source at offset 0 and threw the prefix away, so assembling an object part by part read the source once per part. Now that volume-encrypted sources take this path too, that is the common case rather than an SSE corner. The chunk stream already seeks, so hand it the range. * s3: reject an unsatisfiable copy-source-range A part copy has no way to report a short part, so a range reaching past the source cannot be clamped the way a GET clamps one. The fast path silently produced a part shorter than asked for, or an empty one; the re-encrypting path pads with zeros, so a 2 MiB source copied as bytes=1048576-9999999 came back as 1 MiB of data followed by 7.5 MiB of nothing. Answer InvalidRange instead, which is what s3-tests' test_multipart_copy_invalid_range expects. |
||
|
Chris Lu
|
44115c1051 |
filer: stop TUS uploads from turning into garbage (#10945)
* filer: store TUS sub-chunks through the regular chunk writer A TUS sub-chunk was written with one assigned file id, retried up to three times against that same id, and abandoned on failure: an attempt that had landed on some replicas left a needle no session record and no entry ever references, unreclaimable by vacuum. dataToChunkWithSSE, which the regular write path uses per chunk, assigns a fresh file id per attempt and hands back the file ids of failed attempts, which are now freed the way the regular write path frees them. * filer: retry a chunk write on a fresh volume when the server 5xxs The filer's chunk writer assigns a fresh file id per attempt but only retried transient network errors, so a volume filling up and turning read-only mid-write failed the whole request even though the very next assignment would have landed elsewhere. Every other write client already routes this through ShouldReassignUpload; the filer's own write path now does the same, for regular uploads and TUS sub-chunks alike. * filer: export the chunk deletion queue The filer test harness in weed/server builds filer.Filer as a struct literal, so any code path reaching DeleteChunks dereferenced a nil queue. Exported like the neighboring DeletionRetryQueue so the harness can arm it. * filer: complete a TUS upload whose chunk records overlap A PATCH retried while its predecessor was still storing a sub-chunk - a proxy timeout with an immediate retry is enough - records the same range twice. HEAD computes Upload-Offset as the covered watermark and reported the upload fully received, but completion demanded exactly adjacent records and failed every attempt: the client concluded success from offset == length, no entry was created, and the session eventually expired, turning the entire upload into deleted needles for the vacuum to chew through. Completion now validates gapless coverage with the same watermark HEAD uses. A record extending coverage joins the entry - the read path resolves partial overlaps by ModifiedTsNs, and the raced copies carry identical bytes - while a fully covered duplicate is freed once the entry lands. * filer: allow one mutating TUS request per session at a time Nothing stopped two PATCHes from writing the same range concurrently: both loaded the same offset, both passed the conflict check, and both recorded their sub-chunks. A client whose request timed out in a proxy retries immediately while the server side is still storing the buffered sub-chunk, which is exactly that race. A session now accepts one PATCH or DELETE at a time, the way tusd locks uploads; a concurrent one is refused with 423 Locked, which TUS clients retry, and HEAD keeps answering so progress polling is unaffected. The chunk state is loaded under the claim, so a retried PATCH sees every record its predecessor left and conflicts cleanly instead of duplicating data. * test: cover a TUS PATCH raced by its own retry Stalls a PATCH mid-body over a raw connection, retries the same range while it is in flight, and expects the retry refused with 423 Locked; the upload then resumes from the reported offset and the final content must be intact. * filer: never free a TUS duplicate the entry still references Coverage is computed from ranges, so a record fully covered by another is treated as a duplicate no matter which needle it names. A malformed record naming a file id the entry keeps would have had that needle freed right after the entry landed - the corruption this change set exists to stop. The duplicates are now freed in one batch, skipping any file id the entry references; their records go with the session directory. * test: bound the raw TUS connection reads http.ReadResponse on the stalled PATCH's connection blocked until the whole go test timeout if the filer never answered. * filer: free the needles of chunk write attempts a retry replaced A volume server stores the needle locally and only then fans out to the replicas, so a replication failure 5xxs with the data already written. Each attempt assigns its own file id, so once a later attempt lands elsewhere nothing references the earlier ones: the caller only sees the chunk that succeeded, and the failed ids were dropped. They are now freed the way the caller frees them when the whole write fails. Retrying on a 5xx makes this reachable on every read-only or full volume, which is exactly the condition that filled the reporter's volumes. |
||
|
Chris Lu
|
e521a2a7b9 |
mount: re-assign to a live volume when a write can't land (#10239)
With replication 001 and one node down, ~2/3 of volumes have their replica on the dead node. The primary write lands locally but the replica forward fails, so the volume returns 500 "failed to write to replicas". #9744 made that upload fail fast so the client re-assigns, but the client retry never fired: the reassign gate matched a fixed list of error substrings that didn't include this one. The mount surfaced I/O error and dropped the chunk, leaving missing lines and null-byte gaps in an append workload while a node rebooted. Decide reassignment by HTTP status instead of matching message text. On the write path a volume server only 5xxs on a ReplicatedWrite failure (local disk, replica peer down, under-replication) — all of which a different volume dodges — so any 5xx reassigns; a no-response transport failure (the assigned target itself is down) reassigns too; a 4xx is a genuine client error and is surfaced. doUploadData tags its errors with the response status via uploadStatusError, and the gate moves from util.MultiRetry(errList) to util.RetryOnError(predicate). This drops the fragile substring list (and the message-prefix constants and guard test it needed); store_replicate.go is untouched. |
||
|
|
bdcc3154ed |
refactor: centralize genUploadUrl in UploadOption (#10164)
* refactor: centralize genUploadUrl in UploadOption Replace inline genFileUrlFn closures with operation.GenUploadUrl field: - Add GenUploadUrl func(host, fileId) string to UploadOption struct - Add GenUploadUrlProxy(filerAddress string) utility function - Remove genFileUrlFn parameter from UploadWithRetry signature - Update all callers: mount, gateway, mq, filer_copy, filer_sync This matches the weed mount -filerProxy pattern exactly, factorizing the URL generation logic across all consumers. Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf) * docker release: run all platform jobs in one wave, cache rocksdb compile Drop max-parallel so the 13 per-platform builds run together instead of two waves of 8 (rocksdb was queuing behind the cap and starting ~8 min late). Keep cache-to mode=max for rocksdb: its RocksDB static_lib compile is sha-independent, so it caches across releases and stops being the ~16-min long-pole that gates the merge fan-in. go-build variants stay mode=min. Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf) * refactor: centralize genUploadUrl in UploadOption Replace inline genFileUrlFn closures with operation.GenUploadUrl field: - Add GenUploadUrl func(host, fileId) string to UploadOption struct - Add GenUploadUrlProxy(filerAddress string) utility function - Remove genFileUrlFn parameter from UploadWithRetry signature - Update all callers: mount, gateway, mq, filer_copy, filer_sync This matches the weed mount -filerProxy pattern exactly, factorizing the URL generation logic across all consumers. Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf) * Remove accidental ROCmFPX submodule reference * gofmt chunk upload option block * Preserve broker cipher and re-read proxy filer per upload attempt Chunk uploads must keep the configured Cipher, and both the mount and broker current filer can change on failover, so build the proxy upload URL inside the closure instead of capturing the address once. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
ef109fe9e1 |
mount: don't hang close() when a writer is killed during flush (#10090)
* operation: bound AssignVolume with a deadline AssignVolume ran on context.Background(), so when the filer is overwhelmed the RPC could block indefinitely and wedge every caller holding the connection. Give it a 30s deadline so a stuck assign fails and the caller's retry/error path runs instead of hanging forever. * mount: abort flush when the FUSE request is interrupted On close(), a killed process blocks in fuse_flush waiting for the mount to answer. doFlush ran its metadata CreateEntry on context.Background() and ignored the kernel interrupt channel, so against an overwhelmed filer the flush never completed and the process stayed in uninterruptible sleep -- making the pod un-killable. Derive a context from the FUSE cancel channel in Flush/Fsync and thread it through doFlush -> flushMetadataToFiler -> streamCreateEntry; the retry loop stops as soon as the context is cancelled. Release and the pre-rename flush keep a non-cancellable context since they must finish regardless. * operation: harden the AssignVolume timeout test Make the test double's signal send non-blocking and bound the receive with a timeout so a regression can't wedge the test instead of failing it. |
||
|
Chris Lu
|
94357ac6a9 |
[volume] preserve compression state during replication (#9946)
* preserve compression state during replication * explain why ParseUpload skips compression for replica writes * fix data race on err result in FetchAndWriteNeedle The local-write and replica-write goroutines all wrote the named err return under an unsynchronized err==nil check. Give each goroutine its own error slot and combine after wg.Wait(): local error wins, then the first replica failure. * skip redundant decompression of compressed needles during replication doUploadData decompressed a compressed input only to report the clear-data length on UploadResult.Size, which both replication callers discard. Skip the decompress when IsReplication. |
||
|
Chris Lu
|
4bf27278fa |
topology: fail replica writes fast when a replica is unreachable (#9744)
* operation: bound upload retries and honor context cancellation retriedUploadData hardcoded 3 attempts and an uninterruptible backoff sleep. A synchronous replica write to a dead host therefore paid the full dial timeout three times over before failing. Add UploadOption.MaxAttempts (<=0 keeps the default of 3) so callers can cap attempts, and make the loop return as soon as the context is cancelled so an abandoned upload unwinds instead of retrying. * topology: fail replica writes fast when a replica is unreachable DistributedOperation already returns on the first error, but a single dead replica is itself the slow result: its goroutine retries the upload three times through the dial timeout (~30s) before any error surfaces, stalling the originating client write the whole time. Make the replica write a single attempt (MaxAttempts=1) so a dead replica fails after one dial timeout instead of three, and thread a context into DistributedOperation that is cancelled once the outcome is decided, so a healthy replica is no longer held hostage by one stalled in a dial. The originating client write is what retries. * topology: keep replica deletes off the client request context ReplicatedDelete runs after the local needle is already deleted. Driving the replica deletes off r.Context() means a client disconnect cancels them and orphans needles on the replicas, so use a background context. * operation, topology: trim comments on the replica fail-fast path |
||
|
Chris Lu
|
0716577ec8 |
fix(upload): rewind request body when retrying on connection reset (#9139) (#9222)
* fix(upload): rewind request body when retrying on connection reset (#9139) When httpClient.Do() returned "connection reset by peer" or "use of closed network connection", upload_content retried with the same *http.Request. But the body is a *bytes.Reader the first attempt already consumed, so the retry sent 0 bytes and Go's transport surfaced "http: ContentLength=N with Body length 0". http.NewRequestWithContext populates req.GetBody for *bytes.Reader bodies; use it to attach a fresh body before retrying. Reproduces the issue with a unit test (asserts both attempts see the same payload bytes); the test fails without the fix. * upload: skip inner retry when body cannot be rewound Per review feedback: if req.GetBody is nil or returns an error, the inner retry would call Do(req) with an already-consumed body and the "connection reset" error would be replaced by the misleading "ContentLength=N with Body length 0" — the very symptom this PR set out to fix. Skip the inner retry on rewind failure and let the outer retriedUploadData loop reissue with a fresh request, and log when GetBody is unavailable for observability. * upload: log the actual transport error in the inner retry log line Per review feedback: the diagnostic glog at the top of the inner retry branch was logging postErr — the request-construction error from http.NewRequestWithContext, which is necessarily nil there because the function returns early at line 423 if it isn't. Operators were seeing "<nil>" instead of the transient transport error that triggered the rewind. Reference post_err so the connection-reset / closed-connection cause is actually visible. |
||
|
Chris Lu
|
7d426d2a56 |
Retry uploader on volume full (#8853)
* retry uploader on volume full * drop unused upload retry helper |