2 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	a0b1272cc3	filer: authorize the chunk proxy and the root listing like the rest of the filer port (#11152) ... * filer: require a read token for the root listing maybeCheckJwtAuthorization waved through every GET/HEAD on "/", so a filer with jwt.filer_signing.read.key set still served its root directory listing -- entry names, sizes and chunks[].file_id -- to a caller holding no token at all, and served the same listing to a token restricted by allowed_prefixes. The exemption was added for health checks before the filer had /healthz and /readyz. Both are registered on the default and read-only muxes ahead of the "/" handler and answer without a token, so drop it. Point the mTLS harness at /healthz, which is what it was probing for. * filer: keep the jwt query parameter out of a proxied chunk request The proxy stripped "jwt" from the forwarded query on reads only, on the grounds that a writer's own credential travels there. It does not: an uploader carries its AssignVolume token in the Authorization header, and the query parameter on this path holds a filer credential. Strip it for every method. A volume server has no business seeing a filer token, and because security.GetJwt reads the query before the header, relaying one would hide the writer's own token behind it. * filer: dispatch the chunk proxy after the JWT gate The ?proxyChunkId= branch returned before maybeCheckJwtAuthorization ran, so GET, PUT, POST and DELETE against any needle in the cluster were reachable on the filer's HTTP port with no filer credential, on a filer where every other request answered 401. An anonymous caller read a stored object, replaced its bytes, or deleted the needle, which the master's next vacuum makes permanent. #10434 stopped the filer from minting a volume write token for that caller, which closes the write half only where the volume server has a jwt.signing.key of its own -- not the shipped default, and not what scaffold/security.toml recommends for a filer deployment. The read half stayed open in every configuration, because the filer mints the read token itself. Move the dispatch below the gate. A file id carries no path, so a token restricted by allowed_prefixes cannot be scoped against one and is refused here; every consumer of this endpoint holds an unrestricted token. * filer: mint the volume credential for a proxied write too The proxy minted a volume token on reads and forwarded whatever the caller sent on writes. #10434 made it that way because the branch ran ahead of the JWT gate, so a token minted here would have been signed for an unauthenticated caller; the branch now runs behind the gate, and the credential the caller presents there is a filer one, which a volume server cannot validate and has no business seeing. Mint at the access level the request needs, and drop the caller's Authorization when there is no key to mint from. A proxied uploader then needs only the filer credential, instead of holding one for each hop with a single header to put them in. * mount, mq, filer.sync: send the filer credential for a proxied chunk Every in-tree consumer of ?proxyChunkId= reached the filer anonymously: mount and the broker put the AssignVolume token in the Authorization header, which is a volume credential, and filer.sync sent nothing at all. That was enough only while the branch ran ahead of the filer's JWT gate. Build the URL through one helper, and pick the credential from the URL it returns: a chunk proxied through a filer is a request to the filer, which authorizes it and attaches the volume credential itself, so the token there is a filer one at the access level the request needs. * filer: honor -exposeDirectoryData The flag was declared on all three commands that start a filer and read by none of them: FilerOption.ExposeDirectoryData was only ever assigned from filer.expose_directory_metadata in security.toml, so -exposeDirectoryData=false silently left the listing exposed. Only the TOML key had any effect. Plumb the flag through and let either switch turn the listing off. * filer: count a proxied chunk request once Moving the dispatch below the gate put it after the deferred request observation, so every proxied chunk now landed in FilerRequestHistogram twice, once under its HTTP method and once under chunkProxy. Name the deferred one after the proxy instead, the way the unsupported-method branch already does, which also gives the endpoint the status codes FilerRequestCounter records.	2026-09-04 16:39:36 -07:00
Chris Lu	a10607f90a	Add Terraform support for VM-based SeaweedFS deployment (#9754) ... * terraform: add cloud-agnostic core renderer module Renders per-node weed argv, systemd units, config files, disk-mount and secret-fetch scripts, and cloud-init from an address map. Creates zero cloud resources. Flags verified against the weed binary: volume uses -mserver for the master list, gRPC is -port.grpc (auto http+10000), minFreeSpacePercent is a string, filer store via -defaultStoreDir. * terraform: add mTLS and JWT security module Generates the CA, per-component certs with distinct CNs, and JWT signing keys via the tls/random providers. Emits a core_security object plus PEMs for secret-store delivery. * terraform: add AWS deployment module and examples Reserves stable ENIs first, renders config via the core, then creates instances, prevent_destroy EBS data disks mounted at /data, and the cluster security group. With enable_security, generates certs/JWT, stores them in SSM SecureString, grants an instance role, and fetches them at boot so secrets stay out of user_data. Keyed for_each on every stateful tier. * terraform: add local cluster test harnesses run_local_cluster.sh and run_local_secure.sh render a cluster with the core and run real weed processes, asserting master quorum, volume registration, filer/s3 round-trips, mutual-TLS formation, and JWT enforcement. Use an isolated high port range with a guard so they never touch a cluster already running on the machine. The weed binary defaults to $(go env GOPATH)/bin/weed. * terraform: add CI workflow and README fmt/validate/tofu-test plus smoke jobs that build weed and run both harnesses. * terraform: guard against empty filesystem UUID in mount script An empty UUID made grep -q match any fstab line, skipping the fstab entry and breaking the mount. Fail fast when blkid returns no UUID. * terraform: sanitize cluster name in WEED_CLUSTER env keys Hyphens or spaces in cluster_name produced invalid systemd/bash env var names; map non-alphanumerics to underscores. * terraform: omit empty jwt.signing block from security.toml With enable_security and no JWT key, the template emitted [jwt.signing] key="". Gate the block on a non-empty key and cover it with a test. * terraform: mark core security input as sensitive The security object carries JWT signing keys; keep them out of plan output and known values. * terraform: enforce jwt_length minimum of 32 * terraform: note region/AZ coupling in HA example * terraform: guard WORKDIR before recursive delete in test harnesses * terraform: fix README fence language and test count * terraform: handle embedded s3 with no filer nodes Indexing sort(keys(var.filers))[0] errored at plan time when embedded S3 was enabled but no filers were defined; fall back to an empty config source. * terraform: scope kms:Decrypt to a configurable key arn Replace the hardcoded Resource="*" with a kms_key_arn variable (default "*") so production can restrict decrypt to a specific CMK. * terraform: encrypt EBS data volumes at rest Set encrypted = true on the volume/filer data disks and the all-in-one example disk. * terraform: protect filer instances from API termination Filers hold the leveldb2 metadata store, so they are stateful and get the same disable_api_termination as masters and volumes. * terraform: stop instance before detaching in all-in-one example * terraform: drop stale references to the removed plan doc * terraform: correct stale mount-step comment in aws module * terraform: mark Terraform support as experimental in README	2026-05-30 23:43:17 -07:00