* feat(s3): serve from remote on local read failure
When a locally-cached chunk of a remote-mounted object becomes unreadable
(volume server down/restarting, or an evicted needle 404ing under
retry-backoff), fall back to serving the object from its mounted remote
instead of erroring. A bounded pre-flight probe makes a stuck volume trip
the timeout rather than stalling the request.
Gated by -localReadFallbackToRemote (default off) with
-localReadFallbackTimeout (2s default), so existing deployments are
unaffected until they opt in.
* fix(s3): register local-read-fallback flags for mini/server/filer
The mini, server and filer launchers build S3Options directly and only
populate the flag pointers they register. Without registering the two new
flags there, startS3Server dereferenced nil pointers and crashed at boot,
failing every integration suite that runs `weed mini`.
* fix(s3): treat a zero-byte probe read as unreadable
A read that returns no byte -- whether it reports io.EOF or no error at all
-- means the offset is not locally readable, so the probe must fall back to
the remote rather than proceeding to stream a truncated response. Only a
returned byte (including the object's final byte with a trailing io.EOF)
counts as readable.
* s3: finish a mid-stream local read failure from the remote mount
The pre-flight probe only proves the byte at the requested offset readable.
A multi-chunk object can still lose a later chunk after the 200/206 and its
Content-Length are committed, which truncated the body with no fallback.
Resume from the mounted remote at the byte the local copy stopped at, so the
response still carries the declared length. A short local read that surfaces
as a clean EOF is treated the same way instead of silently truncating.
* s3: fall back to the remote mount without a CLI switch
Serving a remote-mounted object from its authoritative remote is what the
read should have done all along -- the alternative is a 500 on an object the
cluster can still reach -- so make it the behavior instead of two new flags,
with the probe bounded by a constant.
* s3: trim the comments on the fallback path
* filer: report only the contiguous prefix when a parallel chunk read fails
The parallel branch of doReadAt fans the chunk reads straight into their own
windows of the output buffer, then sums every task's bytesRead. A middle chunk
failing while a later one succeeds therefore returned a length covering a hole
the reader never filled, handing the caller zeros in the middle of otherwise
valid data.
* s3: only splice the remote onto a local prefix while it is the cached generation
Eligibility establishes a size match, not byte identity: a remote key
overwritten with same-size content between the cache fill and the fallback
would have finished the response with bytes from a second generation, under
the first one's ETag. Stat the remote before resuming and keep the local
error when it no longer matches -- a truncated body is a visible failure,
a spliced one is not.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
