Commit Graph
2 Commits
Author SHA1 Message Date
Chris LuandClaude Opus 5.5 972f85159e util/http: probe dead replicas in the background and key no-answer dials by machine (#12776)
* util/http: probe an unanswering server in the background, not on a read

Once the retry interval passed, the first read put the dead server first
and waited out the full dial timeout before trying a live replica. Keep it
last and dial it from a background probe with a short timeout instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: try every server of a machine that does not answer last

Unreachability was kept per host:port, so a powered-off node running one
volume server per disk cost one stalled read per server before each was
known dead. A dial that times out or finds no route now marks the whole
machine; a refused connection still marks only that port and shows the
machine is up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: probe an unanswering server with an HTTP request

A bare TCP connect cleared servers that accept connections but never send
response headers, and could not dial an address without an explicit port.
Probe with a HEAD request through the global client instead, and have the
tests wait for the probe to settle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: keep a timed-out port last after a sibling port answers

A dial timeout stored only the machine verdict, so a success on another
port of that machine cleared it and the port that timed out went back to
the front. Record the port as well as the machine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: classify refused and unroutable dials with Winsock errnos on Windows

Windows reports these as WSAECONNREFUSED, WSAEHOSTUNREACH and friends,
which errors.Is does not match against the syscall constants. Test the
refusal with a real dial to a closed port.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: wait for background probes through a WaitGroup in tests

A refused probe can finish before the test reads the claimed record, so
inferring completion from the record's timestamp could wait out the
deadline on a probe that already settled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: key machine verdicts apart from hosts without a port

A host without an explicit port is also its machine name, so a sibling
port answering cleared the timed-out host along with the machine verdict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: group a bracketed port-less IPv6 host with its machine

net.SplitHostPort rejects "[2001:db8::9]", so the machine key kept the
brackets and missed the verdict stored for "[2001:db8::9]:9501".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:44:10 -08:00
Chris Lu fe98520358 read: try a replica that stopped answering last, and relearn its volume's locations (#11130)
* http: try a volume server that failed to answer last

A cached location list is shuffled on every read, so once a replica dies
half the reads keep dialing it first and pay a connect failure or timeout
before the healthy replica answers. Remember, per host, when a request got
no answer at all and order such hosts last for the next half minute. Once
that passes, one read probes the host in its usual place while the others
keep it last until the probe settles, so a black-holed server costs one
stalled read per interval instead of one per read.

Nothing is ever skipped: a host that failed is still tried when the others
fail too. Any response, including an error status, counts as reachable.

Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs

* filer: refresh a chunk's locations after one of them fails

A mount's location cache is only relearned when every cached location
fails. When one replica dies and the other still answers, every read
succeeds and the dead replica stays in the cache, and in the shuffled
order it keeps being dialed first long after the master has dropped it.

When a read fails on one location and a later one answers, call the
refresh hook so the cached entry is dropped and looked up again. The read
that already paid for the failure returns its data; the reads after it
start from the locations the master knows now.

Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs

* http: claim the probe for every expired host, and try it first

The claim was only checked for the first url, so with two replicas whose
marks expired together the second was probed by every read at once. Claim
each expired host on its own and put the reads that won a claim ahead of
the reachable hosts, so a probe is always a real attempt and a lost claim
always means the host is tried last.

Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs

* filer: refresh a chunk's locations in the streaming read path too

The streaming loop had no refresh hook, so a manifest or streamed chunk
that failed on one cached location and was served by another kept the
stale entry until every location failed. Give it the same hook as the
buffered loop, built by one refreshUrls function shared by the reader
cache and the stream callers.

Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs

* http: probe at most one expired host per read

Claiming every expired host in one ordering left all but the first claim
without an attempt, since a read stops at its first answer, and a host that
had come back waited another interval for nothing. Claim only the first
expired host a read sees and leave the rest last and unclaimed, so each
following read probes one of them.

Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs

* test: start the live server before releasing the dead server's port

Closing the dead server first let the live server come up on the same
port, in which case the dead location answers and the partial failure
under test never happens.

Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs
2026-09-03 11:51:48 -07:00