* volume: build the guarded remote storage client through a shared helper
Fold the endpoint validation, credential check and rebinding-safe dialer
that FetchAndWriteNeedle applies before dialing a caller-supplied remote
storage endpoint into a single BuildGuardedRemoteStorageClient helper, so
other callers that dial the same endpoints can reuse it. No behavior
change on this path.
Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN
* filer: build the remote-mount stream client through the guarded helper
streamFromRemote serves a cold remote-only entry straight from its mounted
origin. Build its client through BuildGuardedRemoteStorageClient so the
same endpoint checks the volume server applies cover this read path too.
Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN
* s3: build the remote-mount stream client through the guarded helper
openRemoteStream serves a remote-mounted object straight from its origin
when the local read cannot. Build its client through the same guarded
helper so the endpoint checks apply here as well.
Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN
* volume: accept only static-key gcs credentials on the fetch request
An inline credentials document of a federated type points the SDK at a url,
file or executable of the caller's choosing for the token exchange, so the
request-supplied value is no longer just a key.
* volume: guard the gcs token endpoint like the other remote endpoints
Inline credentials pick where the token request goes, so route the gcs client
through the same deny-list and rebinding-safe dialer used for S3 and azure.
* rust volume: pin that gcs has no credential-driven dial path
* volume: only check gcs credentials on a gcs remote conf
Only the gcs backend reads that field, so another backend carrying a stale
value should not fail the request.
* gcs: load credentials with the type the caller expects
The untyped loader is deprecated because it reads whatever the document
claims to be; callers handling credentials they do not control now name the
types they accept.
* volume: validate replica upload targets in FetchAndWriteNeedle
The replica leg forwarded the fetched needle to a caller-supplied address
without checking it, so a malformed target could redirect the upload to an
unintended host or path. Require each replica target to be a bare host:port
whose host is not loopback / link-local / unspecified, reusing the address
deny-list; cluster peers legitimately sit on private networks, so RFC 1918 /
CGNAT stay allowed and -volume.allowUntrustedRemoteEndpoints still opts out.
Validate every target up front so a bad one fails the request before the local
write, and upload through a client that re-checks the resolved address at
connect time so a replica hostname cannot rebind to a blocked address after
validation. Mirrored in Rust (validation moved ahead of the local write; the
Rust S3 path's connect-time re-check is still a follow-up there).
* volume: only accept inline gcs credentials in FetchAndWriteNeedle
The gcs credentials value on this request could name a local filesystem path,
which the SDK reads from disk. Accept only inline JSON here; the server-side
GOOGLE_APPLICATION_CREDENTIALS env var still supplies a path. The Rust volume
server has no gcs backend, so there is nothing to mirror.
* remote_storage/azure: allow a per-request HTTP client
Thread an optional *http.Client through NewAzBlobClient and add
azure.MakeWithHTTPClient, mirroring the S3 backend. When set, the client
overrides the azblob transport so a caller can pin the dial path. The
existing makers pass nil, so behavior is unchanged.
* volume: extend the remote-endpoint guard to the azure backend
The endpoint validation and rebinding-safe dialer in FetchAndWriteNeedle
covered the S3-SDK backends. The azure backend also dials a caller-supplied
AzureEndpoint, so route both families through a single guardedRemoteClient
helper that returns the endpoint each backend dials and a constructor bound
to the guarded HTTP client. azure is guarded only when AzureEndpoint is set;
an empty endpoint derives the public host from the account.
-volume.allowUntrustedRemoteEndpoints still opts out.
* rust volume: assert the azure endpoint has no remote-client path
The Rust volume server has no azure backend, so make_remote_storage_client
rejects the type before any client is built. Add a regression test pinning
that invariant.
* remote_storage: build S3-compatible clients through one constructor
The eight non-s3 S3-SDK providers each duplicated the AWS session setup
and only the s3 maker could take a custom *http.Client. Route every
S3-compatible type (s3, wasabi, b2, aliyun, tencent, baidu, filebase,
storj, contabo) through MakeWithHTTPClient with a single options table,
and add S3CompatibleEndpoint so callers can resolve the endpoint a given
type dials. No behavior change.
* volume: apply the remote-endpoint check to all S3-compatible providers
FetchAndWriteNeedle validated the endpoint and used the pinned dialer only
for type "s3". Every S3-SDK backend (wasabi, b2, aliyun, tencent, baidu,
filebase, storj, contabo) dials a caller-supplied endpoint through the same
client, so gate on S3CompatibleEndpoint to apply the same check uniformly.
-volume.allowUntrustedRemoteEndpoints still opts out.
* volume: don't route the guarded remote-endpoint client through a proxy
The guarded client exists to dial the validated endpoint directly and
re-check the resolved IP at connect time. With http.ProxyFromEnvironment
set, the dialer only validates the proxy's address while the proxy
re-resolves the endpoint host, which reopens the rebinding window. Drop
the proxy on this path; operators that need one can opt out with
-volume.allowUntrustedRemoteEndpoints.
* volume: decode IPv6 transition addresses in the remote-endpoint guard
checkBlockedIP normalized only ::ffff: mapped IPv4, so NAT64 (64:ff9b::/96),
6to4 (2002::/16), Teredo (2001:0000::/32), and IPv4-compatible (::/96) addresses
that embed an internal IPv4 (loopback, 169.254.169.254, RFC 1918) passed the
endpoint guard even though the plain IPv4 forms are refused. Extract the
embedded IPv4 from those forms and re-check it against the deny list, which
covers both the up-front validation and the dial-time guard. Mirrored in the
Rust volume server.
* volume: require the full NAT64 well-known prefix before decoding
Only 64:ff9b::/96 carries the embedded IPv4 in the low 32 bits, so also require
bytes 4-11 to be zero before treating an address as NAT64; other 64:ff9b:
prefixes place the IPv4 elsewhere and are left untouched. Add public-target
coverage for 6to4, Teredo, and IPv4-compatible so every decoder is exercised on
both a blocked and an allowed destination. Mirrored in the Rust volume server.
volume: require admin auth and refuse loopback endpoints in FetchAndWriteNeedle
Gate the RPC behind checkGrpcAdminAuth for parity with the rest of the
destructive volume-server RPCs, and reject cluster-internal remote S3
endpoints (loopback / link-local / IMDS / RFC 1918 / CGNAT) before
dialing. Pin the validated address against DNS rebinding by routing the
AWS SDK through an HTTP transport whose DialContext re-resolves the host
and re-applies the deny list on every dial, so an endpoint that resolves
to a public IP at validate-time and then flips to 127.0.0.1 at connect
time is refused. Operators that legitimately fetch from private hosts
can opt out with -volume.allowUntrustedRemoteEndpoints.
Chris Lu