Add path filters to workflows that fired on every PR/push regardless
of the diff: CodeQL, go build, the e2e/EC/vacuum/TLS/plugin-worker
integration suites, the Kafka and Postgres gateways, the S3 suites
(Ceph s3tests, s3-go, s3-tables, proxy-signature, https, example,
filer-group), TUS, and the dev binary/container builds. Each scopes
to its subsystem under weed/, its test dir, go.mod/go.sum, and the
workflow file, so docs-, helm-, terraform-, rust- or java-only
changes no longer trigger a full compile-and-test fleet.
* ci(s3tables): route Docker Hub pulls through mirror, drop unused buildx
The integration jobs set up docker/setup-buildx-action only to docker
pull/run images; the buildx bootstrap pulls moby/buildkit from
registry-1.docker.io, which times out and fails the whole job before any
test runs. These jobs never docker build with buildx, so the setup is
pure overhead and an extra registry dependency.
Replace it with a daemon registry-mirror pointing at mirror.gcr.io (a
pull-through cache for Docker Hub) and retry the pre-pulls a few times.
That removes the buildkit pull entirely and routes the rest through the
cache, with graceful fallback to Docker Hub on a miss.
* ci: route Docker Hub through mirror in remaining docker test workflows
Same registry-1.docker.io timeout fix for the other integration jobs.
s3-spark only docker pulls/runs an image, so drop the vestigial buildx
setup and pull through the mirror with retries, matching s3-tables.
kafka-quicktest, s3-proxy-signature, e2e and postgres build/compose and
genuinely need buildx (e2e/postgres export a local layer cache, which the
default driver can't), so keep it and just configure the mirror first —
that way even the moby/buildkit bootstrap pull is served from the cache.
Left samba/pjdfstest alone: they build-push to a local registry and pull
from localhost, so buildx is required and there's no Docker Hub runtime
pull to mirror.
* mini: quieter startup with a docker-compose-style progress board
Replaces noisy startup/shutdown logs with a single in-place progress
table on a TTY (or one line per state change off-TTY). Each component
renders as `pending -> starting -> ready` during startup and
`stopping -> stopped` during shutdown, with elapsed time on transition.
Also folds in a few cleanups uncovered while making this readable:
- route the admin.go startup prints through glog so quietMiniLogs()
filters them under mini but standalone weed admin still shows them
- generate a dev SSE-S3 KEK + passphrase on first run via WEED_S3_SSE_KEK
and WEED_S3_SSE_KEK_PASSPHRASE env vars (viper.Set has a nested-key
conflict between s3.sse.kek and s3.sse.kek.passphrase); persisted under
the data folder so restarts reuse the same key
- demote worker/master gRPC Recv 'context canceled' to V(1); those are
the normal shutdown signal, not Errors/Warnings
- drop the 'Optimized Settings' block and the 'credentials loaded from
environment variables' message from the welcome banner
- only show the credentials setup hints when no S3 identities exist
(new s3api.HasAnyIdentity accessor backed by an atomic.Bool)
- use S3_BUCKET in the credentials hint so it pairs with
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY
- reorder running-services list to master / volume / filer / webdav /
s3 / iceberg / admin
* mini: refuse in-memory-only SSE-S3 dev keys; surface admin serve errors
loadOrCreateMiniHexSecret returns "" when os.WriteFile fails, so SSE-S3
won't encrypt data under a KEK that the next restart can't reproduce
(which would orphan whatever was written this run). The caller already
treats "" as "skip setting WEED_S3_SSE_* env vars", so SSE-S3 and IAM
just stay disabled for this run.
startAdminServer's serve goroutine used to only log ListenAndServe
failures, so a bind error left the caller blocked on ctx.Done() with
no listener. Forward the error through a buffered channel and select
on it alongside ctx.Done().
* ci(s3-proxy-signature): match weed mini's new progress-board ready line
The readiness probe grepped for "S3 (gateway|service).*(started|ready)",
which matched weed mini's old "S3 service is ready at ..." line. Mini
now emits " S3 ready (Xs)" from its progress board, so the
old pattern misses and the test timed out at the 30-second wait.
Widen the alternation to also accept "S3\s+ready". The curl HEAD
fallback already covers any remaining cases.
* Fix S3 signature verification behind reverse proxies
When SeaweedFS is deployed behind a reverse proxy (e.g. nginx, Kong,
Traefik), AWS S3 Signature V4 verification fails because the Host header
the client signed with (e.g. "localhost:9000") differs from the Host
header SeaweedFS receives on the backend (e.g. "seaweedfs:8333").
This commit adds a new -s3.externalUrl parameter (and S3_EXTERNAL_URL
environment variable) that tells SeaweedFS what public-facing URL clients
use to connect. When set, SeaweedFS uses this host value for signature
verification instead of the Host header from the incoming request.
New parameter:
-s3.externalUrl (flag) or S3_EXTERNAL_URL (environment variable)
Example: -s3.externalUrl=http://localhost:9000
Example: S3_EXTERNAL_URL=https://s3.example.com
The environment variable is particularly useful in Docker/Kubernetes
deployments where the external URL is injected via container config.
The flag takes precedence over the environment variable when both are set.
At startup, the URL is parsed and default ports are stripped to match
AWS SDK behavior (port 80 for HTTP, port 443 for HTTPS), so
"http://s3.example.com:80" and "http://s3.example.com" are equivalent.
Bugs fixed:
- Default port stripping was removed by a prior PR, causing signature
mismatches when clients connect on standard ports (80/443)
- X-Forwarded-Port was ignored when X-Forwarded-Host was not present
- Scheme detection now uses proper precedence: X-Forwarded-Proto >
TLS connection > URL scheme > "http"
- Test expectations for standard port stripping were incorrect
- expectedHost field in TestSignatureV4WithForwardedPort was declared
but never actually checked (self-referential test)
* Add Docker integration test for S3 proxy signature verification
Docker Compose setup with nginx reverse proxy to validate that the
-s3.externalUrl parameter (or S3_EXTERNAL_URL env var) correctly
resolves S3 signature verification when SeaweedFS runs behind a proxy.
The test uses nginx proxying port 9000 to SeaweedFS on port 8333,
with X-Forwarded-Host/Port/Proto headers set. SeaweedFS is configured
with -s3.externalUrl=http://localhost:9000 so it uses "localhost:9000"
for signature verification, matching what the AWS CLI signs with.
The test can be run with aws CLI on the host or without it by using
the amazon/aws-cli Docker image with --network host.
Test covers: create-bucket, list-buckets, put-object, head-object,
list-objects-v2, get-object, content round-trip integrity,
delete-object, and delete-bucket — all through the reverse proxy.
* Create s3-proxy-signature-tests.yml
* fix CLI
* fix CI
* Update s3-proxy-signature-tests.yml
* address comments
* Update Dockerfile
* add user
* no need for fuse
* Update s3-proxy-signature-tests.yml
* debug
* weed mini
* fix health check
* health check
* fix health checking
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
dependabot[bot]