mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 15:41:15 +02:00
Chris Lu
a0b1272cc3
* filer: require a read token for the root listing maybeCheckJwtAuthorization waved through every GET/HEAD on "/", so a filer with jwt.filer_signing.read.key set still served its root directory listing -- entry names, sizes and chunks[].file_id -- to a caller holding no token at all, and served the same listing to a token restricted by allowed_prefixes. The exemption was added for health checks before the filer had /healthz and /readyz. Both are registered on the default and read-only muxes ahead of the "/" handler and answer without a token, so drop it. Point the mTLS harness at /healthz, which is what it was probing for. * filer: keep the jwt query parameter out of a proxied chunk request The proxy stripped "jwt" from the forwarded query on reads only, on the grounds that a writer's own credential travels there. It does not: an uploader carries its AssignVolume token in the Authorization header, and the query parameter on this path holds a filer credential. Strip it for every method. A volume server has no business seeing a filer token, and because security.GetJwt reads the query before the header, relaying one would hide the writer's own token behind it. * filer: dispatch the chunk proxy after the JWT gate The ?proxyChunkId= branch returned before maybeCheckJwtAuthorization ran, so GET, PUT, POST and DELETE against any needle in the cluster were reachable on the filer's HTTP port with no filer credential, on a filer where every other request answered 401. An anonymous caller read a stored object, replaced its bytes, or deleted the needle, which the master's next vacuum makes permanent. #10434 stopped the filer from minting a volume write token for that caller, which closes the write half only where the volume server has a jwt.signing.key of its own -- not the shipped default, and not what scaffold/security.toml recommends for a filer deployment. The read half stayed open in every configuration, because the filer mints the read token itself. Move the dispatch below the gate. A file id carries no path, so a token restricted by allowed_prefixes cannot be scoped against one and is refused here; every consumer of this endpoint holds an unrestricted token. * filer: mint the volume credential for a proxied write too The proxy minted a volume token on reads and forwarded whatever the caller sent on writes. #10434 made it that way because the branch ran ahead of the JWT gate, so a token minted here would have been signed for an unauthenticated caller; the branch now runs behind the gate, and the credential the caller presents there is a filer one, which a volume server cannot validate and has no business seeing. Mint at the access level the request needs, and drop the caller's Authorization when there is no key to mint from. A proxied uploader then needs only the filer credential, instead of holding one for each hop with a single header to put them in. * mount, mq, filer.sync: send the filer credential for a proxied chunk Every in-tree consumer of ?proxyChunkId= reached the filer anonymously: mount and the broker put the AssignVolume token in the Authorization header, which is a volume credential, and filer.sync sent nothing at all. That was enough only while the branch ran ahead of the filer's JWT gate. Build the URL through one helper, and pick the credential from the URL it returns: a chunk proxied through a filer is a request to the filer, which authorizes it and attaches the volume credential itself, so the token there is a filer one at the access level the request needs. * filer: honor -exposeDirectoryData The flag was declared on all three commands that start a filer and read by none of them: FilerOption.ExposeDirectoryData was only ever assigned from filer.expose_directory_metadata in security.toml, so -exposeDirectoryData=false silently left the listing exposed. Only the TOML key had any effect. Plumb the flag through and let either switch turn the listing off. * filer: count a proxied chunk request once Moving the dispatch below the gate put it after the deferred request observation, so every proxied chunk now landed in FilerRequestHistogram twice, once under its HTTP method and once under chunkProxy. Name the deferred one after the proxy instead, the way the unsupported-method branch already does, which also gives the endpoint the status codes FilerRequestCounter records.
SeaweedFS on Terraform
Experimental. This Terraform support is an early scaffold under active development. Interfaces (variables, outputs, module layout) may change without notice, and not every tier is implemented yet. Not recommended for production without your own review and testing.
Self-contained Terraform/OpenTofu modules to deploy SeaweedFS on cloud VMs
running the weed binary directly under systemd. No Helm and no Kubernetes
required.
What works today is verified end-to-end against a real weed cluster (see
"Test it locally" below).
Layout
terraform/
modules/
core/ cloud-agnostic renderer (ZERO cloud resources): turns an
address map + config into per-node weed argv, systemd units,
config files, disk-mount + secret-fetch scripts, and cloud-init.
Both the cloud wrappers and the local harnesses consume `nodes`.
security/ cloud-agnostic CA + per-component mTLS certs (distinct CNs) +
JWT signing keys (tls/random providers). Emits a `core_security`
object ready for the core, plus PEMs for secret-store delivery.
aws/ thin AWS wrapper: reserves stable ENIs (fixed private IPs)
first, feeds them to core, creates instances + protected EBS
data disks + SG; with enable_security it generates certs/JWT,
stores them in SSM SecureString, grants an instance role, and
renders a boot fetch-secrets.sh. Keyed for_each throughout.
examples/
aws-ha-distributed/ 3 masters + 3 volumes (1/AZ) + 2 filers + 1 S3,
secure-by-default (mTLS via SSM-delivered certs)
aws-all-in-one/ single `weed server` instance via core directly
test/
local/ render a cluster with core and run it as real weed processes
on 127.0.0.1 (no cloud, no docker), then assert it works
local-secure/ generate certs/JWT with the security module, run a real
mTLS cluster, and assert it forms + enforces JWT auth
Design in one paragraph
The chart is the structural reference, not a dependency. A cloud-agnostic
core renders everything portable; thin per-cloud wrappers provision infra.
Addressing is an input to the core (wrapper reserves static IPs first), so
the wrapper -> core dependency is one-way with no apply-time cycle. Stateful
tiers (master/volume/filer) are keyed for_each maps, never count, so a
middle node can be replaced without reindexing its peers or reattaching the
wrong disk. Flag names are verified against the real weed binary
(notably: volume uses -mserver for the master list; gRPC is -port.grpc,
auto = http+10000; minFreeSpacePercent is a string).
Test it locally
Requires tofu (or terraform), jq, curl, and a weed binary. Renders a
3-master + volume + filer + S3 cluster from the core module and runs it as real
processes, asserting quorum, volume registration, and filer/S3 round-trips:
cd terraform/test/local
WEED=/path/to/weed ./run_local_cluster.sh
# => 7 passed, 0 failed
The harness uses a high port range (29333/28080/28888/28333) so it does not
collide with a SeaweedFS cluster already running on the machine, and aborts if a
required port is taken. KEEP=1 ./run_local_cluster.sh leaves the cluster up.
mTLS end-to-end
cd terraform/test/local-secure
WEED=/path/to/weed ./run_local_secure.sh
# => generates a CA + component certs + JWT, renders security.toml, runs a real
# mTLS cluster, asserts master/volume/filer form over mutual TLS and that the
# filer enforces JWT signing (unsigned writes get 401). 5 passed.
Plan-level tests (no cloud)
cd terraform/modules/core && tofu test
# => 11 passed: peers list, -mserver vs -master, metrics gating,
# security.toml conditions, all-in-one inheritance, ...
Validate the cloud wrappers
cd terraform/examples/aws-ha-distributed && tofu init && tofu validate
apply needs AWS credentials, a VPC, subnets, and an AMI with weed installed
(bake with Packer, or install at boot).
Status
Implemented and verified:
- Tiers: master / volume / filer / s3 / all-in-one rendered by the core.
- Disk mount: cloud-init runs a
mount-disks.shthat auto-discovers (or takes explicit candidate devices),blkid-guardsmkfs, mounts, and persists to/etc/fstabby UUID. The AWS wrapper wires the protected EBS disk to/data. - mTLS + JWT: the
securitymodule generates the CA + per-component certs (distinct CNs) + JWT keys; the AWS wrapper stores them in SSM SecureString, grants an instance role, and the core renders a boot fetch script so secrets stay OUT of user_data. Proven end-to-end bytest/local-secure.
Not yet implemented:
- sftp / admin / worker tiers (core does not render them yet).
- GCP/Azure wrappers, the data-plane provider, and the K8s-native module.
- CA in Vault PKI (currently TF-generated, so the CA key lives in state). KMS
decrypt in the IAM policy is scoped by
ViaServicebut usesResource="*"; tighten to the SSM CMK in production.
Gotchas
- This
weedbuild starts an Iceberg REST catalog on:8181by default; sets3.iceberg_port = 0to disable, or a port to relocate it. - Disk auto-discovery assumes a single attached data disk per node. For multiple
data disks, pass explicit
devicescandidates indisk_mounts.
Security note
tofu output/state can contain secrets. Generate secrets outside Terraform
(cloud secret manager / Vault) and fetch them at boot; never commit secret
material.