Commit Graph
19 Commits
Author SHA1 Message Date
Chris LuandClaude Opus 5.5 d84ae4f2ab volume: don't let an unreachable tier backend hold up volume server startup (#12312)
* s3 tier backend: bound connect and response-header waits

The tier client used the SDK default transport: a 30s dial timeout and no
limit on waiting for response headers. A blackholed endpoint cost about
two minutes per request with the SDK's retries, and an endpoint that
accepts connections but never answers held the request forever.

Dial with a 10s timeout and give up on a response whose headers do not
arrive within a minute.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stop the disk scan from waiting on a stalled tier backend

A volume server registers with the master only after every volume on its
disks has loaded, and loading a tiered volume reads its superblock from the
remote object. With the S3 endpoint unreachable, each read waited out the
client's timeouts and retries on one of the loader workers, so start-up grew
with the number of tiered volumes and the local volumes were not served
until it finished.

A tiered volume already loads without its superblock when the read fails.
Give that read 10s, and once it stalls, let the backend's other volumes skip
it for a minute instead of each waiting out the timeout. The volume reads
from the backend as before once the endpoint answers again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: bound connect and read waits of the S3 tier client

Mirror the Go tier client: a 10s connect timeout instead of the SDK's
3.1s default, and a minute to receive a response, so an endpoint that
accepts connections but never answers cannot hold a read forever.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: stop volume loading from waiting on a stalled tier backend

Mirror the Go loader: the superblock read of a tiered volume gets 10s, and
once it stalls the backend's other volumes skip it for a minute. The
volume loads with its .vif version as before when the read fails, and
reads from the backend once the endpoint answers again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: keep the .vif replication when a remote superblock read fails

Go applies the .vif replication to a tiered volume whether or not its
superblock read succeeded, so a volume loaded while its backend is down
still reports its replica placement. The Rust loader applied it only on
success and reported the default placement instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: skip the .idx head repair while the tier backend is unreachable

The repair reads needle headers from the remote .dat, so after the
superblock read failed it would wait on the same backend again and hold
up the load. It runs on the next load that reaches the backend.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: skip the .idx head repair while the tier backend is unreachable

Mirrors the Go load: after the remote superblock read failed, the
repair's remote header reads would wait on the same backend again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 tier backend: give up a ranged read whose body stops arriving

The response-header timeout ends once headers arrive, so an endpoint
that answers and then sends no body held ReadAt, and a superblock
read's worker, forever. Cancel the request once the body makes no
progress for s3ReadTimeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: test that a tier read whose body stalls gives up

The SDK's stalled-stream protection already fails it, matching the Go
read's no-progress cancel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): keep the S3 read timeout off upload_part

The SDK's read timeout runs from the start of the request, so a 64 MiB
part on a slow link would time out while its body is still being sent.
Go's ResponseHeaderTimeout only starts after the body is written.
upload_part now keeps the connect timeout and drops the read timeout;
stalled-stream protection still fails an upload that stops moving.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): bound the upload_part response wait from when the part is sent

Without a read timeout, an upload_part whose body was accepted but never
answered waited forever. The part body now goes to the connection in 64 KiB
chunks, and once the last chunk is taken the response must arrive within
S3_READ_TIMEOUT, like Go's ResponseHeaderTimeout. A retry rebuilds the body
and restarts that clock. The body keeps its in-memory contents, so signing
and the checksum header are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): keep a retried part's response clock from an earlier attempt

The send state was a single flag, and a retry body could be rebuilt and
fully sent before the waiter saw the flag drop, so the waiter kept the
earlier attempt's deadline. Each body now carries its attempt number, only
the latest attempt marks the part sent, and the deadline holds only while
no later attempt has started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 18:35:37 -08:00
Javier de la TorreandJavier de la Torre 21add33b6e s3 backend: optional HTTP client settings for the cloud tier (#11897)
* s3 backend: optional HTTP client settings for the cloud tier

The S3 tier backend builds its client from a bare aws.Config, so it always
uses Go's default transport: HTTP/2 wherever the endpoint offers it over TLS,
no cap on connections per host, and a 90s idle timeout. Against some
S3-compatible services that is a poor fit. Over HTTP/2 every upload and
range read of a tiered volume shares one connection and its flow-control
window; with a small window, tier uploads and concurrent reads of tiered
volumes become throughput-bound on that one connection, while HTTP/1.1 opens
a connection per concurrent request. Some services also limit parallel
connections per client address, or drop idle connections before 90s.

Add three optional settings to [storage.backend.s3.<id>], all off by
default so existing deployments keep the SDK's default client:
- disable_http2: keep the client on HTTP/1.1 (ForceAttemptHTTP2=false and an
  empty TLSNextProto, the documented opt-out);
- max_connections: MaxConnsPerHost, 0 = no cap;
- idle_connection_timeout_seconds: IdleConnTimeout, 0 = Go's default.

The session cache key includes the options when any is set, so two backends
on the same endpoint with different settings do not share a client.
ToProperties reports the settings; the master.toml scaffold documents them.

Tests: TestS3BackendHTTPOptionsReachTheClient fails without the change (the
settings never reach the client); TestS3BackendDefaultHTTPClientUnchanged,
TestS3BackendSessionsDifferByHTTPOptions, TestParseNonNegative.

* s3 backend: reject an idle timeout too large for time.Duration

An idle_connection_timeout_seconds above math.MaxInt64/1e9 overflowed when
multiplied by time.Second. The negative result made the transport keep its
default timeout and ToProperties export a negative value. Warn and use the
default instead.

* s3 backend: parse the idle timeout as 64-bit seconds

parseIdleConnTimeout went through strconv.Atoi, so on 32-bit platforms
any value above 2147483647 seconds was rejected while the same value was
accepted on 64-bit, and the overflow test failed under GOARCH=386. Parse
with ParseInt(…, 64) and export the effective value with FormatInt so the
accepted range is the same on every platform.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 14:44:39 -07:00
Lars LehtonenandChris Lu 21ab68aa94 chore(weed/storage/backend/s3_backend): remove unused function (#9715)
* chore(weed/storage/backend/s3_backend): remove unused function

* fix(s3_backend): cache session under the composite region|endpoint key

createSession looked up sessions by region|endpoint but stored them by
region alone, so the cache never hit and a new session was built every
call. With getSession gone the lock can also drop to a plain Mutex.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-05-27 22:14:45 -07:00
promalertandChris Lu 9012069bd7 chore: execute goimports to format the code (#7983)
* chore: execute goimports to format the code

Signed-off-by: promalert <promalert@outlook.com>

* goimports -w .

---------

Signed-off-by: promalert <promalert@outlook.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-01-07 13:06:08 -08:00
chalet 877b9b788a update s3 session cache key (#6923) 2025-06-26 03:21:35 -07:00
chrislu bd4891a117 change version directory 2025-06-03 22:46:10 -07:00
ludwigxia 5452405a81 cloud tier: support for Alibaba Cloud OSS (#6466) 2025-01-22 00:48:55 -08:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Chris Lu d04cdcf40d s3 header add user agent 2021-10-29 12:28:24 -07:00
Chris Lu 7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
Chris Lu c08ac536ed cloud drive: add support for Wasabi
* disable md5, sha256 checking to avoid reading one chunk twice
* single threaded upload to avoid chunk swapping (to be enhanced later)
2021-08-25 17:34:29 -07:00
Chris Lu 00c4e06caa cloud drive: s3 configurable force path style 2021-08-23 03:30:41 -07:00
Chris Lu f0cc130849 do not force path style for better compatibility 2021-08-23 03:09:41 -07:00
Chris Lu 2cca07b44c cloud tier to non-AWS s3 gateways 2021-02-28 18:59:09 -08:00
Chris Lu 709f231e23 tiered storage: add s3 endpoint for private s3 implementation
fix https://github.com/chrislusf/seaweedfs/issues/1238
2020-03-19 21:13:56 -07:00
Chris Lu eecad6062a go fmt 2019-12-25 09:56:42 -08:00
Chris Lu d960b3474a tier storage: support downloading the remote dat files 2019-12-25 09:53:13 -08:00
Chris Lu e4c63ff2f8 go fmt 2019-11-22 14:51:09 -08:00
Chris Lu 9711a6ffaa WIP 2019-11-18 19:24:37 -08:00