mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-25 06:20:42 +01:00
s3 backend: optional HTTP client settings for the cloud tier (#11897)
* s3 backend: optional HTTP client settings for the cloud tier The S3 tier backend builds its client from a bare aws.Config, so it always uses Go's default transport: HTTP/2 wherever the endpoint offers it over TLS, no cap on connections per host, and a 90s idle timeout. Against some S3-compatible services that is a poor fit. Over HTTP/2 every upload and range read of a tiered volume shares one connection and its flow-control window; with a small window, tier uploads and concurrent reads of tiered volumes become throughput-bound on that one connection, while HTTP/1.1 opens a connection per concurrent request. Some services also limit parallel connections per client address, or drop idle connections before 90s. Add three optional settings to [storage.backend.s3.<id>], all off by default so existing deployments keep the SDK's default client: - disable_http2: keep the client on HTTP/1.1 (ForceAttemptHTTP2=false and an empty TLSNextProto, the documented opt-out); - max_connections: MaxConnsPerHost, 0 = no cap; - idle_connection_timeout_seconds: IdleConnTimeout, 0 = Go's default. The session cache key includes the options when any is set, so two backends on the same endpoint with different settings do not share a client. ToProperties reports the settings; the master.toml scaffold documents them. Tests: TestS3BackendHTTPOptionsReachTheClient fails without the change (the settings never reach the client); TestS3BackendDefaultHTTPClientUnchanged, TestS3BackendSessionsDifferByHTTPOptions, TestParseNonNegative. * s3 backend: reject an idle timeout too large for time.Duration An idle_connection_timeout_seconds above math.MaxInt64/1e9 overflowed when multiplied by time.Second. The negative result made the transport keep its default timeout and ToProperties export a negative value. Warn and use the default instead. * s3 backend: parse the idle timeout as 64-bit seconds parseIdleConnTimeout went through strconv.Atoi, so on 32-bit platforms any value above 2147483647 seconds was rejected while the same value was accepted on 64-bit, and the overflow test failed under GOARCH=386. Parse with ParseInt(…, 64) and export the effective value with FormatInt so the accepted range is the same on every platform. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
This commit is contained in:
1 parent
9a03c09396
commit
21add33b6e
4 files changed
+224
-2
No files matched your search
@@ -28,6 +28,10 @@ region = "us-east-2"
|
||||
bucket = "your_bucket_name" # an existing bucket
|
||||
endpoint = ""
|
||||
storage_class = "STANDARD_IA"
|
||||
# optional HTTP client tuning for S3-compatible services:
|
||||
# disable_http2 = false # true keeps the client on HTTP/1.1, one connection per concurrent request
|
||||
# max_connections = 0 # cap on connections to the endpoint, 0 = no cap
|
||||
# idle_connection_timeout_seconds = 0 # close idle connections sooner, 0 = Go default (90s)
|
||||
|
||||
# create this number of logical volumes if no more writable volumes
|
||||
# count_x means how many copies of data.
|
||||
|
||||
Reference in new issue
Block a user