At BestSpeed, Go 1.26's flate writes 4000 random bytes plus a 300-byte
zero run as a stored block, so the gzip output was 28 bytes larger than
the input and the test failed its own precondition on the toolchain
go.mod pins. With 450 zero bytes the saving is 5.7% on Go 1.26 and 8.4%
on Go 1.27, inside the marginal range on both.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): replicate the stored needle instead of the request body
The primary forwarded the original request body and headers, so a
replica re-derived compression from the raw bytes. Send what Go's
ReplicatedWrite sends: a one-part multipart upload of the stored bytes,
gzip-marked when compressed, with the name, mime, pairs and Content-MD5,
and carry ts, ttl, cm and fsync in the query so form-field values reach
the replica too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): keep the primary's compression on replica writes
A type=replicate upload now stores the bytes as received, like Go's
ParseUpload, so a replica no longer compresses data the primary stored
uncompressed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): replicas keep the primary's name and mime
Forward the primary's filename and mime even past the 255-byte needle
fields, so a replica applies the same limit and sets the same flags,
and skip the filename MIME guess on type=replicate, so an untyped
upload stays untyped on every copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): keep the multipart part Content-Type as sent
Parsing the part type as a mime value lowercased it and dropped values
it could not parse, so a replica stored a different mime than the
primary. Read the header verbatim, as Go's part.Header.Get does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: replicas keep the primary's mime as forwarded
A Go replica guessed the mime from the file extension, so an upload the
primary stored as application/octet-stream, or with no mime at all, got
a different type on the replica. On type=replicate keep the part's
Content-Type as sent, and have the replication sender leave an empty
mime empty instead of filling it in from the extension. This matches
what the Rust replica does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): pick the JPEG orientation fix by file name, as Go does
Go checks the mime and the upload's file name, Rust checked the mime and
the URL path, so the same upload could be rotated on one server and not
on the other. With the name and mime forwarded as stored, a replica now
reaches the primary's decision, and a rotated image has no EXIF tag left
to act on, so the fix stays a no-op there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* compression: compress JSON uploads, and match file types by extension
IsCompressableFileType did not recognise JSON media types, so S3 uploads of application/json (which carry no file name) were never compressed. Its callers also passed filepath.Base of the file name where it expects the extension, so the .json/.js/.html/... extension rules never matched a real file name. Both callers now pass filepath.Ext; the function lower-cases the extension and media type, ignores media-type parameters ("application/json; charset=utf-8"), and treats */json and *+json types as compressible.
* compression: treat application/json-seq as JSON
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): compress JSON uploads and match media types without parameters
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: keep uploader gzip only when it saves at least 10%
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Pat White <5015863+patwhite@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix: parse filename in PUT + refactor
* fix: master iu public url with http
* fix: better parsing and handle disposition header
* fix: take mime type from file extension if not set
* Added global http client
* Added Do func for global http client
* Changed the code to use the global http client
* Fix http client in volume uploader
* Fixed pkg name
* Fixed http util funcs
* Fixed http client for bench_filer_upload
* Fixed http client for stress_filer_upload
* Fixed http client for filer_server_handlers_proxy
* Fixed http client for command_fs_merge_volumes
* Fixed http client for command_fs_merge_volumes and command_volume_fsck
* Fixed http client for s3api_server
* Added init global client for main funcs
* Rename global_client to client
* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold
* Reduce the visibility of some functions in the util/http/client pkg
* Added the loadSecurityConfig function
* Use util.LoadSecurityConfiguration() in NewHttpClient func
The mime type is always the value passed in.
Compress or not depends on the content detection, file name extension, and compression ratio.
If the content is already compressed, need to know the content size.
