* remote: reuse one guarded remote storage client per configuration
BuildGuardedRemoteStorageClient (the filer's lazy remote stat and list,
remote deletes and renames, the volume server's cold reads, the S3
gateway's remote reads) built a new client, with its own http.Transport,
on every call. Each transport kept its connection idle for
IdleConnTimeout (60s) and was never reused or closed, so a steady stream
of remote lookups held one open connection per request. An endpoint
that caps connections per source then refuses every new connection.
- remote_storage: GetRemoteStorage's per-name cache becomes a bounded
LRU (WEED_REMOTE_CLIENT_CACHE_MAX, default 64) keyed by kind, name, a
hash of the whole configuration and a credentials version, built
outside the lock once per key (singleflight) from a copy of the
configuration. A client leaving the cache has its idle connections
closed. GetRemoteStorageWith exposes the cache to other builders.
- BuildGuardedRemoteStorageClient caches the guarded client per
configuration. The endpoint is checked against the deny-list when a
client is built; every new connection still dials through the guarded
dialer. A gcs credentials file, and for an S3-compatible remote using
the default credential chain the shared credentials file and AWS_*
variables, are part of the key by content.
- The shared transport keeps 32 idle connections per host, caps
connections per host (WEED_REMOTE_MAX_CONNS_PER_HOST, default 64) and
has dial, TLS handshake and response header deadlines, so a hung
remote frees its slots.
- The filer drops the clients of removed remotes (and all but the newest
client per name) on every /etc/remote reload, also when /etc/remote
does not exist; reloads are serialized.
* remote: drop clients whose build spans a reload or whose remote is gone
* server: hash the shared AWS config file into the guarded credential version
* server: bound a stalled body on the shared remote transport
* remote: waive path-injection on the configured credentials file
* remote: keep caching remotes the local filer does not name
* remote: drop the inline codeql waiver the analyzer ignores
* remote: version the plain gcs client's credentials file
* remote: hash the environment's inline gcs credentials
GOOGLE_APPLICATION_CREDENTIALS can hold the JSON itself, which the gcs
client accepts; it is not in the configuration hash, so a changed value
kept the old client.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote: version the well-known gcs credentials file
With neither the configuration nor GOOGLE_APPLICATION_CREDENTIALS set,
the SDK reads application_default_credentials.json; rotating it left the
cache key unchanged and the old client behind.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote: share the gcs credentials version with the guarded client
The guarded gcs client resolves the same environment and well-known
credentials as the plain one; keying it only on a configured path left
those rotations invisible to the cache.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote: version s3 keys resolved from the environment
A remote whose access and secret keys both arrive through its type's
environment fallback (aliyun, baidu, contabo, filebase, storj, tencent)
never entered the default-chain branch of guardedCredentialsVersion, so
an in-process environment change kept serving the stale cached client.
The version now hashes the resolved keys for every S3-compatible remote.
* volume: reuse one remote storage client per configuration (rust)
fetch_and_write_needle built a fresh aws_sdk_s3::Client per request,
so every cold read kept its connector's idle connections without ever
reusing them — the same leak the Go side fixes. Cache clients keyed by
the configuration's protobuf encoding in an LRU of
WEED_REMOTE_CLIENT_CACHE_MAX entries (default 64), mirroring
weed/remote_storage/client_cache.go. An evicted entry drops the client's
last Arc and with it the connector's idle connections. The per-request
endpoint validation is unchanged: Rust has no dial-time guard yet, so
the up-front DNS check stays on every call.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
