The store comes apart into the four roles it always had

`weed server -s3` was never one thing. Master, volume, filer and the S3
gateway ran as four goroutines under one process, on one volume, sharing one
fate. Splitting them into four containers changes nothing a client can see —
the same bucket answers on the same port — but it makes three things possible
that were not: the SeaweedFS admin UI, which wants a cluster to look at; a
restart or an upgrade of one role without the others; and, eventually, a second
volume server somewhere else. A fifth container carries the panel itself.

The single-process files stay exactly as they were. These are `.split.` twins
beside them, four in all, one per folder per shape, each with the .env example
of the same name that both READMEs already promise.

Identities are the part that could not simply be copied across. SeaweedFS picks
its credentials from one source, in order: an -s3.config file, the filer's IAM
store, then AWS_ACCESS_KEY_ID and its secret — and a higher source replaces a
lower one rather than adding to it. The existing files use the env pair, which
is fine precisely because nothing else writes identities there. Hand somebody a
panel that can, and the first user they create lands in the filer's store, the
store outranks the environment, and PocketBase's key stops existing — with the
first failed upload as the notification. So the gateway here is started with no
config file and no AWS_* at all, and the init container seeds PocketBase's
identity into the filer's store instead: the same store the panel writes. One
source of truth, PocketBase's key sitting in Object Store → Users beside every
other, keys minted there picked up without a restart, and a rotated
PB_S3_SECRET re-applied in place on the next boot rather than added as a second
identity.

That seeding is now allowed to fail. The bucket-create it grew out of was
best-effort — `|| true`, on the reasoning that the API Server's own S3 check
would report a gateway that was genuinely unreachable. That reasoning does not
survive the change: a gateway whose IAM store is empty does not refuse anyone,
it serves everyone, and the bucket would be wide open rather than unreachable.
So the step ends by grepping the configuration back for the access key, the
gateway waits on it completing successfully, and a seed that did not land stops
the stack instead of opening it.

The prod files publish the gateway and the panel, both on loopback, and nothing
else. Port 8080 on the volume server hands out file content by file id with no
authentication of any kind — the S3 credentials have no bearing on it — so
publishing it would publish every attachment in the stack, and the panel shows
what that port and the master's would. The panel's own password is required
rather than defaulted, because weed serves it with authentication switched off
entirely when it is empty, and a page that mints bucket credentials is the
bucket. It is passed as WEED_ADMIN_PASSWORD rather than a flag so it stays off
the process command line, and SEAWEED_ADMIN_BIND is the knob a remote host
needs, named after PB_BIND and API_BIND for the same reason.

Master, volume and filer share one /data mount rather than taking three of
their own. That is precisely the layout `weed server -dir=/data` writes — the
master's raft state, the volume's .dat and .idx, the filer's filerldb2, no two
of them naming the same file — so a stack can move between the single-process
file and its twin in either direction with nothing to migrate. A second volume
server would need its own, and the files say so where somebody would go looking.

The dev files map the volume server to 8081 on the host: 8080 there is already
the API Server, and in the all-in-one it is the API Server inside the image.

Unexercised: written on a machine without Docker, so none of the four has been
brought up. Every flag, health path and env name was read out of the pinned
4.45 source rather than recalled — -mdir, -volumeSizeLimitMB, -defaultStoreDir,
-max, admin's -master and -dataDir and WEED_ADMIN_*, the filer's and gateway's
/healthz, the panel's unauthenticated /health — and the four files were parsed,
interpolated against their examples, and checked for duplicate host ports. A
`docker compose config` on the target host is still the first thing to run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tajniak81
2026-09-04 20:35:42 +02:00
co-authored by Claude Opus 5
parent 9a2a4ab72e
commit fe1e314df9
10 changed files with 2020 additions and 0 deletions
+36
View File
@@ -105,6 +105,7 @@ remember — with an `.env` example of the same name:
|---|---|---|
| **Local storage** — the default, unchanged | `docker-compose.prod.yml` | `docker-compose.yml` |
| **SeaweedFS beside the image** | `docker-compose.prod.seaweedfs.yml` | `docker-compose.seaweedfs.yml` |
| **SeaweedFS, split into its roles** | `docker-compose.prod.seaweedfs.split.yml` | `docker-compose.seaweedfs.split.yml` |
| **An S3 endpoint elsewhere** | `docker-compose.prod.s3.yml` | `docker-compose.s3.yml` |
So `docker-compose.prod.seaweedfs.yml` is configured from
@@ -127,6 +128,41 @@ bucket, because PocketBase never issues a `CreateBucket` of its own. The
external-S3 ones add no containers at all: set `PB_S3_ENDPOINT`, and create the
bucket yourself.
### Split SeaweedFS
`weed server -s3` runs master, volume, filer and gateway as four goroutines in
one process. The `.split.` files run them as four containers beside the
all-in-one, plus a fifth: the SeaweedFS **admin UI** on port 23646, where the
cluster can be inspected and — under *Object Store → Users* — further S3
identities minted and revoked. Split also gets you per-role restarts and
upgrades, per-role Prometheus metrics, and room to add a second volume server
later. Still none of them inside the image, for the reason above.
Identities work differently there, and it matters. SeaweedFS reads credentials
from, in descending priority: an `-s3.config` file, the filer's IAM store, then
`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` — and a higher source *replaces* a
lower one rather than adding to it. The single-gateway files use the env vars,
which is why nothing else may write identities there: the first user added in a
panel would displace PocketBase's key. So in the split files `seaweedfs-init`
seeds PocketBase's identity into the filer's store instead — the same store the
admin UI writes — and the gateway runs with no config file at all. One source of
truth, PocketBase's key visible in the panel beside every other, and new keys
picked up without a restart. Rotating `PB_S3_SECRET` in `.env` and restarting
updates that identity in place.
Set `SEAWEED_ADMIN_PASSWORD`: `weed admin` serves the panel with no
authentication when it is empty, and a panel that can mint bucket credentials is
the bucket. In the prod file it is bound to loopback like `SEAWEED_S3_BIND` — it
is storage plumbing, not one of the app's own panels — so a remote host needs
`SEAWEED_ADMIN_BIND=0.0.0.0` behind a reverse proxy. That file publishes nothing
for master, volume and filer: the volume server serves file content by id with
no authentication of any kind, and the admin UI already shows what those ports
would.
Switching between `docker-compose.seaweedfs.yml` and its `.split.` twin needs no
migration: master, volume and filer share one `/data` mount, which is exactly
the layout `weed server -dir=/data` writes.
On every boot the API Server's bootstrap writes PocketBase's *Files storage*
settings from those variables, then asks PocketBase to prove it can reach the
bucket. Watch for it in the log: