Files
seaweedfs/weed/command/scaffold/security.toml
T
c1c95e127f filer: scoped read-only tokens for metadata subscriptions (#12767)
* security: scoped read-only token for filer metadata subscriptions

A token signed with jwt.filer_signing.subscribe.key names the subscribe
methods and the path prefixes its holder may see. mTLS peers and admin
tokens stay unrestricted. Internal subscribers attach an admin token
when both keys are configured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: enforce subscription token scope on metadata streams

With jwt.filer_signing.subscribe.key set, SubscribeMetadata and
SubscribeLocalMetadata authorize the caller first. A scoped caller gets
a stream that sends only events whose paths are all under its prefixes,
filters batches, and never carries log file refs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: send an admin token on internal metadata subscriptions

Peer filers, metadata followers and the lifecycle reader attach an
admin token when the subscribe key is configured, so they keep the full
stream without mTLS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: add fs.meta.token to mint metadata subscription tokens

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: end scoped metadata subscriptions at token expiry

The scope now carries the token's exp. The scoped stream's context ends
at that time and the call returns Unauthenticated, so an open stream no
longer outlives its token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: sign metadata subscriptions with the source cluster's key

MetadataFollowOption takes a FilerJwtProvider; filer.sync passes the
one loaded from -a.security or -b.security, so clusters with different
admin keys both accept the subscription.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: report fs.meta.token flag errors

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: give scoped metadata subscribers their own client id

A scoped caller could reuse another subscriber's ClientId with a higher
epoch and replace that registration, disconnecting it. Scoped streams now
register under a server-chosen id that no other subscriber holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: accept only HS256 metadata subscription tokens

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: test HS384 metadata subscription token rejection

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: never let a scoped subscriber replace another registration

The server-chosen id can be taken by another subscriber between choosing
and registering it. Registering scoped streams with the lowest epoch makes
addClient refuse them rather than replace the other registration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: send position markers for events a scoped token withholds

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test a scoped stream withholding a whole event batch

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: filter SendMsg on scoped metadata streams

The scoped wrappers only overrode Send, so a SendMsg on the embedded
stream would reach the client unfiltered. Route it through the same
filter and refuse any other message type. Ported from #12790.

Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test a scoped subscription through the real subscribe loop

Run SubscribeLocalMetadata with a scoped token over a flushed backlog,
the in-memory tail and live events: only in-scope events arrive, sibling,
"..", encoded and cross-boundary rename paths are dropped, and no log
file references are sent although the client asks for them. Without the
subscribe key every event still arrives. Ported from #12790.

Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test that the HTTP API refuses metadata subscription tokens

GET, HEAD, PUT, POST and DELETE all refuse a subscription token, also
when the subscribe key equals the read or write key. Ported from #12790.

Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
2026-11-19 12:41:16 -08:00

293 lines
12 KiB
TOML

# Put this file to one of the location, with descending priority
# ./security.toml
# $HOME/.seaweedfs/security.toml
# /etc/seaweedfs/security.toml
# this file is read by master, volume server, filer, and worker
# Any value below can also be supplied as an environment variable instead of
# living in this file: prefix the key with WEED_, upper-case it, and replace
# "." with "_". This is the recommended way to inject the secrets in here from
# a Kubernetes Secret (env valueFrom.secretKeyRef) rather than a ConfigMap.
# The JWT signing keys map to:
# WEED_JWT_SIGNING_KEY, WEED_JWT_SIGNING_READ_KEY (master <-> volume)
# WEED_JWT_FILER_SIGNING_KEY, WEED_JWT_FILER_SIGNING_READ_KEY (s3/clients <-> filer)
# WEED_JWT_FILER_SIGNING_SUBSCRIBE_KEY (filer metadata subscriptions)
# A set env var wins over the file, so the file can stay empty (or absent).
# comma separated origins allowed to make requests to the filer and s3 gateway.
# enter in this format: https://domain.com, or http://localhost:port
[cors.allowed_origins]
values = "*"
# this jwt signing key is read by master and volume server, and it is used for write operations:
# - the Master server generates the JWT, which can be used to write a certain file on a volume server
# - the Volume server validates the JWT on writing
# the jwt defaults to expire after 10 seconds.
[jwt.signing]
key = ""
expires_after_seconds = 10 # seconds
# by default, if the signing key above is set, the Volume UI over HTTP is disabled.
# by setting ui.access to true, you can re-enable the Volume UI. Despite
# some information leakage (as the UI is not authenticated), this should not
# pose a security risk.
[access]
ui = false
# by default the filer UI is enabled. This can be a security risk if the filer is exposed to the public
# and the JWT for reads is not set. If you don't want the public to have access to the objects in your
# storage, and you haven't set the JWT for reads it is wise to disable access to directory metadata.
# This disables access to the Filer UI, and will no longer return directory metadata in GET requests.
[filer.expose_directory_metadata]
enabled = true
# this jwt signing key is read by master and volume server, and it is used for read operations:
# - the Master server generates the JWT, which can be used to read a certain file on a volume server
# - the Volume server validates the JWT on reading
# NOTE: jwt for read is only supported with master+volume setup. Filer does not support this mode.
[jwt.signing.read]
key = ""
expires_after_seconds = 10 # seconds
# If this JWT key is configured, Filer only accepts writes over HTTP if they are signed with this JWT:
# - f.e. the S3 API Shim generates the JWT
# - the Filer server validates the JWT on writing
# NOTE: This key is ALSO used as a fallback signing key for S3 STS if s3.iam.config does not specify a signingKey.
# NOTE: This key also gates the filer IAM gRPC service (CreateUser, PutPolicy,
# CreateAccessKey, ...). When set, every IAM RPC must carry a Bearer
# token signed with this key in its "authorization" gRPC metadata; mint
# such a token with security.GenJwtForFilerAdmin. When empty, the IAM
# gRPC service runs unauthenticated and refuses its OIDC provider and
# role RPCs. When set, set it on both filer and admin.
# The key does not protect the other filer gRPC RPCs, which can read and
# write the IAM store under /etc/iam; if the filer gRPC port is reachable
# beyond a trusted network, enable [grpc.filer] mTLS below.
# the jwt defaults to expire after 10 seconds.
[jwt.filer_signing]
key = ""
expires_after_seconds = 10 # seconds
# If this JWT key is configured, Filer only accepts reads over HTTP if they are signed with this JWT:
# - f.e. the S3 API Shim generates the JWT
# - the Filer server validates the JWT on reading
# the jwt defaults to expire after 10 seconds.
[jwt.filer_signing.read]
key = ""
expires_after_seconds = 10 # seconds
# Optional. If this key is set, the filer's SubscribeMetadata and
# SubscribeLocalMetadata gRPC calls require an mTLS-verified peer, an admin
# token signed with jwt.filer_signing.key, or a read-only token signed with this
# key. That token is an HS256 JWT with "exp", "allowed_methods" (a subset of
# SubscribeMetadata and SubscribeLocalMetadata) and "allowed_prefixes"
# (absolute paths); its holder only receives events whose paths are all under
# one of the prefixes. Mint one with "fs.meta.token" in weed shell. Without
# mTLS, set jwt.filer_signing.key and this key on every filer, S3 gateway,
# mount and filer.sync too, so they keep full access.
[jwt.filer_signing.subscribe]
key = ""
# gRPC mTLS configuration
# All gRPC TLS authentications are mutual (mTLS)
# The values for ca, cert, and key are paths to the certificate/key files
# The host name is not checked, so the certificate files can be shared.
# Set verify_server_host = true under [grpc] or a [grpc.<component>] section to
# require that a server certificate is valid for the host or IP that component
# dials, as the Rust volume server always does.
# Each [grpc.<component>] section also accepts optional client_cert/client_key,
# presented when that component dials other servers. Set them when your CA
# issues separate serverAuth-only and clientAuth-only certificates; when unset,
# the component reuses cert/key for both directions.
# If client and server certificates come from different issuing CAs, put both
# CA certificates in the ca PEM file.
# Once a component's cert/key are set, any error loading them or grpc.ca is
# fatal: the component never falls back to plaintext.
[grpc]
ca = ""
verify_server_host = false
# Set wildcard domain for enable TLS authentication by common names
allowed_wildcard_domain = "" # .mycompany.com
# Optional allowlist a client checks the dialed server's certificate against:
# the subject common name, a DNS SAN, or a URI SAN (e.g. a SPIFFE id) must be
# listed, or match allowed_server_wildcard_domain, else the handshake fails.
# With a per-workload CA, list the identities each server may present so one
# workload cannot impersonate another component. Unset accepts any CA-signed
# server certificate, as before. Any [grpc.<component>] section may set its own
# allowed_server_commonNames to override this list. Only the filer scopes it
# further per dialed target: [grpc.filer] master_, volume_ and
# s3_allowed_server_commonNames. Other components dial every server with one
# allowlist and refuse to start when per-target keys are their only allowlist.
allowed_server_commonNames = "" # comma-separated server identities
allowed_server_wildcard_domain = "" # .mycompany.com
# Volume server gRPC options (server-side)
# Enables mTLS for incoming gRPC connections to volume server
[grpc.volume]
cert = ""
key = ""
client_cert = ""
client_key = ""
allowed_commonNames = "" # comma-separated SSL certificate common names
# Master server gRPC options (server-side)
# Enables mTLS for incoming gRPC connections to master server
[grpc.master]
cert = ""
key = ""
client_cert = ""
client_key = ""
allowed_commonNames = "" # comma-separated SSL certificate common names
# Filer server gRPC options (server-side)
# Enables mTLS for incoming gRPC connections to filer server
[grpc.filer]
cert = ""
key = ""
client_cert = ""
client_key = ""
allowed_commonNames = "" # comma-separated SSL certificate common names
# S3 server gRPC options (server-side)
# Enables mTLS for incoming gRPC connections to S3 server
[grpc.s3]
cert = ""
key = ""
client_cert = ""
client_key = ""
allowed_commonNames = "" # comma-separated SSL certificate common names
[grpc.msg_broker]
cert = ""
key = ""
client_cert = ""
client_key = ""
allowed_commonNames = "" # comma-separated SSL certificate common names
[grpc.msg_agent]
cert = ""
key = ""
client_cert = ""
client_key = ""
allowed_commonNames = "" # comma-separated SSL certificate common names
[grpc.admin]
cert = ""
key = ""
client_cert = ""
client_key = ""
allowed_commonNames = "" # comma-separated SSL certificate common names
[grpc.worker]
cert = ""
key = ""
client_cert = ""
client_key = ""
allowed_commonNames = "" # comma-separated SSL certificate common names
[grpc.mq]
cert = ""
key = ""
client_cert = ""
client_key = ""
allowed_commonNames = "" # comma-separated SSL certificate common names
# weed-vector gRPC options (server-side)
# Enables mTLS for incoming gRPC connections to the vector engine; the S3
# gateway dials it with [grpc.client], and the engine dials the masters with
# [grpc.client] too. The files are re-read every WEED_TLS_CERT_REFRESH_INTERVAL
# (default 5h) and on SIGHUP.
[grpc.vector]
cert = ""
key = ""
allowed_commonNames = "" # comma-separated SSL certificate common names
# gRPC client configuration for outgoing gRPC connections
# Used by clients (S3, mount, backup, benchmark, filer.copy, filer.replicate, upload, etc.)
# when connecting to any gRPC server (master, volume, filer)
[grpc.client]
cert = ""
key = ""
# HTTPS client configuration for outgoing HTTP connections
# Used by S3, mount, filer.copy, backup, and other clients when communicating with master/volume/filer
# Set enabled=true to use HTTPS instead of HTTP for data operations (separate from gRPC)
# If [https.filer] or [https.volume] are enabled on servers, clients must have [https.client] enabled=true
[https.client]
enabled = false # Set to true to enable HTTPS for all outgoing HTTP client connections
cert = "" # Client certificate for mTLS (optional if server doesn't require client cert)
key = "" # Client key for mTLS (optional if server doesn't require client cert)
ca = "" # CA certificate to verify server certificates (required when enabled=true)
insecure_skip_verify = false # Skip TLS certificate verification (NOT recommended for production)
# Volume server HTTPS options (server-side)
# Enables HTTPS for incoming HTTP connections to volume server
[https.volume]
cert = ""
key = ""
ca = ""
# Master server HTTPS options (server-side)
# Enables HTTPS for incoming HTTP connections to master server (web UI, HTTP API)
[https.master]
cert = ""
key = ""
ca = ""
# Filer server HTTPS options (server-side)
# Enables HTTPS for incoming HTTP connections to filer server (web UI, HTTP API)
[https.filer]
cert = ""
key = ""
ca = ""
# disable_tls_verify_client_cert = true|false (default: false)
# Admin server HTTPS options (server-side)
# Enables HTTPS for incoming HTTP connections to admin server
[https.admin]
cert = ""
key = ""
ca = ""
# Admin server authentication
# If password is set, users must login to access the admin interface
# These can be overridden by environment variables with WEED_ prefix:
# WEED_ADMIN_USER, WEED_ADMIN_PASSWORD
# WEED_ADMIN_READONLY_USER, WEED_ADMIN_READONLY_PASSWORD
[admin]
user = ""
password = ""
[admin.readonly]
user = ""
password = ""
# SSE-S3 server-side encryption key management
# These settings configure the Key Encryption Key (KEK) for S3 SSE-S3 encryption.
# Set exactly one of kek or key. If neither is set, SSE-S3 is disabled.
# Can also be set via env vars: WEED_S3_SSE_KEK, WEED_S3_SSE_KEY
[s3.sse]
# hex-encoded 256-bit key, same format as the legacy /etc/s3/sse_kek filer file.
# Use this to migrate from a filer-stored KEK: copy the value from /etc/s3/sse_kek.
# Generate a new one with: openssl rand -hex 32
kek = ""
# any secret string; a 256-bit key is derived automatically via HKDF-SHA256.
# Cannot be used while /etc/s3/sse_kek exists on the filer — delete it first.
key = ""
# Trusted reverse proxies allowed to set forwarding headers on behalf of the
# real client: X-Forwarded-For / X-Real-Ip for aws:SourceIp condition
# evaluation in S3 bucket and IAM policies, and X-Forwarded-Prefix for SigV4
# signature verification of prefix-stripped paths. When the direct TCP peer is
# in this list, these headers are honored; otherwise they are ignored. A
# trusted proxy must overwrite or strip client-supplied values of these
# headers, since honoring them means trusting them. Leave empty to always use
# the direct peer, which is the safe default.
# Can also be set via env var: WEED_S3_TRUSTED_PROXIES_WHITE_LIST
[s3.trusted_proxies]
white_list = "" # comma separated; bare IPs or CIDRs, e.g. "10.10.10.0/24"
# white list. It's checking request ip address.
[guard]
white_list = ""