* operation: read a manifest chunk from the next replica when one fails
ChunkedFileReader read each chunk from one random location and returned
the first error, so a chunk whose cached replica went down failed the
whole file although another replica had it. Try the chunk's locations in
turn (local-data replicas first) and resume each attempt at the byte the
previous one reached. A failed write to the caller still ends the read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: drop a chunk volume's cached locations after a failed read
A chunk read that failed on a replica left that replica in the
10-minute location cache, so later reads kept trying it first. Drop the
volume's cached locations after a failure, and when every cached
replica has failed, ask the master once more and try the locations it
now lists.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: don't continue a partly sent chunk on another replica
A volume server streams a large needle before it has verified the
checksum and aborts the response on a mismatch, so bytes that already
reached the reader may be corrupt. Continuing such a chunk on another
replica would hand back a mix of bad and good bytes as a complete read.
Move to the next replica only while nothing of the chunk has been
written; a mid-body failure still fails the read, and the dropped cache
entry lets the next read pick a live replica.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: treat a short chunk body as a failed replica read
A replica that ends a chunk's body cleanly before the chunk's size made
the read look successful, so the next replica was never tried and the
following chunks landed at the wrong offsets. Read exactly the chunk's
remaining length and report a short body as an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: report the needle read error with a failed read
A read that fails on the volume server, for example on a CRC mismatch,
returned a bare 500, so mount and filer logs only showed the status.
The volume server now writes the error into the response body, and the
read helpers add it to the error they return.
Co-authored-by: Bruce <82158894+half-life666@users.noreply.github.com>
* volume: keep the 500 bare on HEAD requests
writeJsonError's body write fails with ErrBodyNotAllowed under net/http
when the request method is HEAD, adding an error-writing log line on top
of the failed read. Send only the status, matching the no-error path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util/http: decode gzip-encoded error bodies in statusError
Full-chunk reads set Accept-Encoding: gzip themselves, so the
transport does not transparently decompress; a gzip-encoded error
body reached statusError as compressed bytes and produced an
unreadable message. statusError now takes the response and decodes
Content-Encoding before extracting the bounded message.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: log the read error when the 500 stays bare
On HEAD requests the body write is skipped, which also skipped the
V(0) response log inside writeJson — a failed HEAD read left no
cause in normal volume-server logs. Log the error in the same
format before the bare 500.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util/http: check the status before decoding gzip in GetAuthenticated
The gzip reader eagerly consumes the header when it is created, so
statusError saw a drained or corrupt body on a gzipped error response
and the server message was lost. Match the other read helpers and check
the status first.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util/http: export StatusError
Other volume read paths outside this file need the same error body.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: redact jwt query param when logging failed-read URLs
The new read-failure logs print the request URL, and volume reads accept
?jwt= as a credential, so a failed authenticated read would write the
bearer token to the volume server log.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util/http: surface the error body on the remaining chunk-read paths
* volume: redact duplicate jwt params in failed-read logs
* operation: pass the lookup jwt on the chunk needle read
---------
Co-authored-by: Bruce <82158894+half-life666@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* batch deletion operations to return individual error results
Modify batch deletion operations to return individual error results instead of one aggregated error, enabling better tracking of which specific files failed to delete (helping reduce orphan file issues).
* Simplified logging logic
* Optimized nested loop
* handles the edge case where the RPC succeeds but connection cleanup fails
* simplify
* simplify
* ignore 'not found' errors here
* Added global http client
* Added Do func for global http client
* Changed the code to use the global http client
* Fix http client in volume uploader
* Fixed pkg name
* Fixed http util funcs
* Fixed http client for bench_filer_upload
* Fixed http client for stress_filer_upload
* Fixed http client for filer_server_handlers_proxy
* Fixed http client for command_fs_merge_volumes
* Fixed http client for command_fs_merge_volumes and command_volume_fsck
* Fixed http client for s3api_server
* Added init global client for main funcs
* Rename global_client to client
* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold
* Reduce the visibility of some functions in the util/http/client pkg
* Added the loadSecurityConfig function
* Use util.LoadSecurityConfiguration() in NewHttpClient func
* Added context for the MasterClient's methods to avoid endless loops
* Returned WithClient function. Added WithClientCustomGetMaster function
* Hid unused ctx arguments
* Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions
* Changed the context termination check in the tryConnectToMaster function
* Added a child context to the tryConnectToMaster function
* Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
