* volume: report the needle read error with a failed read
A read that fails on the volume server, for example on a CRC mismatch,
returned a bare 500, so mount and filer logs only showed the status.
The volume server now writes the error into the response body, and the
read helpers add it to the error they return.
Co-authored-by: Bruce <82158894+half-life666@users.noreply.github.com>
* volume: keep the 500 bare on HEAD requests
writeJsonError's body write fails with ErrBodyNotAllowed under net/http
when the request method is HEAD, adding an error-writing log line on top
of the failed read. Send only the status, matching the no-error path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util/http: decode gzip-encoded error bodies in statusError
Full-chunk reads set Accept-Encoding: gzip themselves, so the
transport does not transparently decompress; a gzip-encoded error
body reached statusError as compressed bytes and produced an
unreadable message. statusError now takes the response and decodes
Content-Encoding before extracting the bounded message.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: log the read error when the 500 stays bare
On HEAD requests the body write is skipped, which also skipped the
V(0) response log inside writeJson — a failed HEAD read left no
cause in normal volume-server logs. Log the error in the same
format before the bare 500.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util/http: check the status before decoding gzip in GetAuthenticated
The gzip reader eagerly consumes the header when it is created, so
statusError saw a drained or corrupt body on a gzipped error response
and the server message was lost. Match the other read helpers and check
the status first.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util/http: export StatusError
Other volume read paths outside this file need the same error body.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: redact jwt query param when logging failed-read URLs
The new read-failure logs print the request URL, and volume reads accept
?jwt= as a credential, so a failed authenticated read would write the
bearer token to the volume server log.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util/http: surface the error body on the remaining chunk-read paths
* volume: redact duplicate jwt params in failed-read logs
* operation: pass the lookup jwt on the chunk needle read
---------
Co-authored-by: Bruce <82158894+half-life666@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume server: do not finish a GET when the needle CRC mismatches
A streamed full-needle read compared the CRC only after every page had been written. Once the response buffer flushed, the client already had a completed 200 and the corrupt bytes. Hold the last page until the checksum matches, and if an earlier page has already been flushed, abort the connection instead of calling http.Error.
Fixes#11459
* volume server: abort partial-content bodies on write error too
The non-Range path drops the unflushed tail and aborts on a mid-body
error; the single-range and multi-range paths still flushed it after
WriteHeader(206) was committed, delivering corrupt bytes as a complete
body.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: assert the started 200 is aborted in the write-error test
The test previously returned on any request error, so it passed without
verifying the abort. It now asserts the client got the committed 200
headers and then a failed body read. Also trims comments.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(master): honour -volume.fileSizeLimitMB on the master's /submit - #6748
`weed server -volume.fileSizeLimitMB=2048` still refused anything over
256MB, and the reason is not the one the report assumes: the option does
reach the volume server. The master does not use it. Uploads through the
master's /submit are buffered by submitForClientHandler, which passed a
hardcoded 256MB to needle.ParseUpload, so the master rejected what the
volume server it started would have accepted.
The limit is now passed in. `weed master` gains its own -fileSizeLimitMB
with the same 256 default, so a standalone master behaves exactly as
before, and `weed server` and `weed mini` hand it the value their volume
server already got.
* master.follower: take the same upload limit, and say which flag to match
Review found the follower left behind. It serves /submit like the leader
and buffers uploads under the same limit, but kept the fixed 256MB, so a
cluster raised above that would accept an upload through the leader and
refuse the identical one through a follower.
Two smaller points from the same review: the master's flag description
named only the standalone volume server's spelling, and now names the
weed server and weed mini form too; and the under-limit test asserted on
the error message alone, so it would have passed had the limit rejected
that payload with different wording. It now requires the request to get
past parsing.
* filer: serve "//" paths at the cleaned path instead of redirecting
http.ServeMux redirects a non-canonical path ("//", "..") to its cleaned
form, but since Go 1.22 it builds the Location from the already-escaped
path, so it is percent-encoded twice (golang/go#79897). A client that
follows the redirect re-posts "/负极全景" as "/%25E8%25B4%259F...", and
the filer stores a directory literally named "%E8%B4%9F...".
Wrap the filer muxes in CleanPathHandler, which rewrites the request to
the same cleaned path ServeMux would have redirected to and dispatches
directly. The decoded name reaches the handler, the round trip goes
away, and clients that do not follow redirects work too.
Fixes#11125
* filer: keep RequestURI in step with the cleaned path
PostHandler derives storage rules, the bucket and the read-only check from
r.RequestURI while writing the entry at r.URL.Path. After CleanPathHandler
rewrote only the URL, a "//" or ".." request would be placed by the raw
path and written to the cleaned one. Rewrite RequestURI too, as the
redirect-following client used to.
* filer: match storage rules on the decoded write path
PostHandler resolved the storage rule from r.RequestURI, the raw
request-target. Clients percent-encode non-ASCII segments on the wire, so
a read-only or TTL rule configured on "/data/只读/" never matched a POST
to "/data/%E5%8F%AA%E8%AF%BB/" and the write went through. Use r.URL.Path,
the decoded path the entry is actually written to, as the header-based
destination check already does. The query string no longer reaches the
rule lookup, so the "?" trimming in the read-only error is gone.
* filer, volume server: return 416 when no requested range overlaps the content
* seaweed-volume: return 416 when no requested range overlaps the content
* server: check the range test error, use the request context, fix the no-overlap comment boundary
* fix(shell): correct volume.list -writable filter unit and comparison
* fix(shell): correct volume.list -writable filter unit and comparison
* chore(shell): fix typo in EC shard helper param names
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* refactor(shell): drop nil sentinel in splitCSVSet, use len() in callers
* fix: avoid reading upload body when writing JSON errors
* writeJson: drop unused JSONP branch
No in-tree caller uses ?callback=. Always serve application/json
with X-Content-Type-Options: nosniff.
* seaweed-volume: drop unused JSONP branch
Mirror Go: always serve application/json with
X-Content-Type-Options: nosniff.
* writeJson: drop unreachable StatusNotModified check
bodyAllowedForStatus already returns early for 304.
* test/volume_server: rename and rewrite JSONP test to assert callback is ignored
CI: /status?callback=myFunc now returns plain application/json
with X-Content-Type-Options: nosniff.
