seaweed up

Chris Lu
2026-04-26 21:29:19 -07:00
parent f139af8f04
commit 1db13f449b
5 changed files with 469 additions and 0 deletions
+137
@@ -0,0 +1,137 @@
Once a cluster is deployed, ongoing operations against an inventory-managed `cluster.yaml` revolve around four `cluster plan` flags: append-merge (the default re-run behavior), `--dry-run`, `--refresh-host=<ip>`, and `--overwrite`. This page walks through each.
For first-time setup see [[Cluster Plan Workflow]]; for the inventory schema see [[Cluster Plan Inventory Reference]].
## Contents
- [Adding hosts (append-merge)](#adding-hosts-append-merge)
- [Previewing changes (--dry-run)](#previewing-changes---dry-run)
- [Hardware drift detection](#hardware-drift-detection)
- [Refreshing one host (--refresh-host)](#refreshing-one-host---refresh-host)
- [Regenerating from scratch (--overwrite)](#regenerating-from-scratch---overwrite)
- [Sidecars](#sidecars)
- [Working with hand-written cluster.yaml files](#working-with-hand-written-clusteryaml-files)
## Adding hosts (append-merge)
The default behavior when `-o cluster.yaml` already exists is **append-merge**: every existing entry stays byte-identical (your hand edits, your inline comments, your key ordering survive); inventory hosts that aren't in the YAML get appended to the right `*_servers` section.
```bash
# Add a new host to inventory.yaml, then:
seaweed-up cluster plan -i inventory.yaml -o cluster.yaml
seaweed-up cluster deploy -f cluster.yaml
```
What you'll see on stderr:
```
appended to volume_servers: 10.0.0.23:8080
```
What survives byte-for-byte:
- Operator hand edits to `max:`, `disk:`, `dataCenter`, custom config blocks.
- Comments (head, line, foot) on existing entries.
- Indent style (2-space vs 4-space) detected from the existing file.
- Section ordering and key ordering within sections.
What's added:
- One new entry per role per inventory host that isn't already keyed at `ip:port` in the YAML.
- A new `*_servers:` section if the role wasn't represented before.
What's reported but not changed:
- **Orphans** — entries in `cluster.yaml` not produced by this plan run (host removed from inventory, probe failed, or the volume role was dropped because no eligible disks were found). Surface as `WARN: orphan in cluster.yaml ...` lines. The row stays in YAML; the operator decides whether to delete, fix the probe, or repair the inventory.
- **Unparseable** — hand-edited entries `keyOfNode` couldn't extract a key from (e.g. an entry missing `port:`). Documented hazard: a fresh inventory entry with the same IP would still get appended on top, so the operator should clean these up.
## Previewing changes (--dry-run)
Before any rewrite, you can preview the diff:
```bash
seaweed-up cluster plan -i inventory.yaml -o cluster.yaml --dry-run
```
Runs the full probe + Marshal/Merge pipeline, then prints a unified diff between the current `-o` file (or empty for greenfield) and the body plan would write. Exits without touching anything on disk. Sidecars are reported as "would write" lines but not actually written.
```diff
--- cluster.yaml (current)
+++ cluster.yaml (proposed)
@@ -23,4 +23,8 @@
port.grpc: 18081
folders:
- folder: /data1
+ - ip: 10.0.0.23
+ port.ssh: 22
+ port: 8080
+ port.grpc: 18080
```
Pairs naturally with append-merge: operators preview before letting plan mutate the file. Pairs with `--overwrite` too — see what regeneration is about to discard before it runs.
`--dry-run` requires `-o` (without it there's no diff target).
## Hardware drift detection
Every plan run that produces a `cluster.facts.json` sidecar lays the foundation for the next run's drift check. On the second run, plan loads the previous facts file before the fresh probe writes its replacement, then compares per-host disk paths and surfaces a warning if anything moved:
```
WARN: drift on 10.0.0.21:22 (since previous facts.json): added /dev/sdc
```
Drift is **informational, not actionable** — the YAML isn't touched. The operator decides whether the change is intentional (added a disk on purpose) and runs `--refresh-host` to bring the entry up to date, or investigates if it's unexpected (drive failure, vendor swap).
Scope is intentionally narrow today: disk path set only. Size shifts and model-string churn aren't flagged because they're noisy on cloud hosts (in-place resizes, vendor revisions). NICs and CPU aren't compared either.
`--dry-run` also surfaces drift warnings, so you can re-run plan against an inventory that's been quiet for weeks and see whether anything moved before deciding to write.
## Refreshing one host (--refresh-host)
When drift detection flags a host, the natural follow-up is to re-emit its entries from fresh facts without disturbing the rest of the file:
```bash
seaweed-up cluster plan -i inventory.yaml -o cluster.yaml --refresh-host 10.0.0.21
seaweed-up cluster deploy -f cluster.yaml
```
The flag is repeatable (`--refresh-host 10.0.0.21 --refresh-host 10.0.0.22`) and refreshes per-IP rather than per-(ip, port). A host that maps to multiple sections — say `master` + `filer` + `volume` — gets all of its rows refreshed in one shot. Misses (an IP that didn't match any entry) surface as `WARN: --refresh-host <ip> did not match any existing entry`.
What survives a refresh:
- **Entry-level head, line, and foot comments** stay paired with the host (`# primary HDD bank` on top of an entry survives).
What gets clobbered:
- Every spec field on the refreshed entries — including ports (incidentally part of the dedup key but treated as just another field for refresh).
- **Field-level inline comments inside the mapping** (e.g. `port: 8080 # custom`). Pairing each fresh field with its old counterpart is out of scope for now; if you care, promote the comment to a head comment above the entry, which IS preserved.
`--refresh-host` requires `-o` pointing at an existing file. Without `-o` (greenfield) or with `--overwrite`, the flag warns and continues — there's no existing file to refresh into.
## Regenerating from scratch (--overwrite)
When you've drifted too far to merge cleanly — typically because someone restructured the spec by hand and the merge no longer makes sense — `--overwrite` regenerates everything:
```bash
seaweed-up cluster plan -i inventory.yaml -o cluster.yaml --overwrite
```
**This discards all hand edits.** Always pair with `--dry-run` first to see what's about to be lost:
```bash
seaweed-up cluster plan -i inventory.yaml -o cluster.yaml --overwrite --dry-run
```
## Sidecars
`cluster plan -o cluster.yaml` writes two JSON sidecars next to the YAML:
| File | Purpose | Consumer |
| --- | --- | --- |
| `cluster.facts.json` | Raw `HostFacts` slice from the probe (disks, CPU, memory, NICs) | Drift detection on the next plan run; operator audit |
| `cluster.deploy-disks.json` | Per-host allowlist of disk paths plan classified as eligible | `cluster deploy` — fail-closed contract: if the sidecar is missing on a plan-generated spec, deploy refuses rather than scanning every disk |
Both files are regenerated on every plan run (including merge runs). They're audit + deploy contracts, not hand-edit surfaces, so byte-stability isn't a goal. Permissions are `0600` because `facts.json` includes hostnames and disk model strings (host enumeration data).
Plan-generated specs carry a header marker comment so `cluster deploy` knows to apply the fail-closed contract; hand-written specs (no marker) take the legacy path.
## Working with hand-written cluster.yaml files
The plan workflow is opt-in. You can keep hand-writing `cluster.yaml` and using `cluster deploy -f cluster.yaml` directly — see [[Deployment with seaweed-up]] for that flow. None of the plan-side fail-closed contracts apply to hand-written specs (no marker in the header → no sidecar enforcement on deploy).
You can also append-merge into a hand-written file: as long as the entries follow the standard `ip:` / `port:` shape, the merge dedupes correctly and preserves your formatting. The plan marker only gets stamped on greenfield writes.
+175
@@ -0,0 +1,175 @@
The inventory file is the single hand-authored input to [[Cluster Plan Workflow]]. It lists hosts, the roles each one runs, and SSH connection details. Everything else (disk layout, ports, max counts) is discovered or defaulted.
## Contents
- [Top-level shape](#top-level-shape)
- [defaults](#defaults)
- [hosts](#hosts)
- [Roles](#roles)
- [SSH overrides](#ssh-overrides)
- [Labels (DC / Rack)](#labels-dc--rack)
- [Disk knobs](#disk-knobs)
- [Tags and the `external` role](#tags-and-the-external-role)
- [Validation rules](#validation-rules)
## Top-level shape
```yaml
defaults:
ssh:
user: root
port: 22
identity: ~/.ssh/id_rsa
disk:
device_globs: ["/dev/sd*", "/dev/nvme*", "/dev/xvd*", "/dev/vd*"]
exclude: ["/dev/sda"] # boot disk
reserve_pct: 5 # leave 5% free per disk
auto_idx_tier: false # see "Disk knobs" below
allow_ephemeral: false # AWS Nitro / GCP local SSD
hosts:
- ip: 10.0.0.11
roles: [master]
# ...
```
Both `defaults` blocks are optional; everything has a sensible default.
## defaults
| Field | Default | Purpose |
| --- | --- | --- |
| `ssh.user` | OS current user | SSH login user |
| `ssh.port` | `22` | SSH port |
| `ssh.identity` | `~/.ssh/id_rsa` | SSH private key path |
| `disk.device_globs` | `[/dev/sd*, /dev/nvme*, /dev/xvd*, /dev/vd*]` | Block-device prefix allowlist scanned by the probe |
| `disk.exclude` | none | Per-host literal device paths or `/dev/<prefix>*` globs to skip (boot disk, swap disk, etc.) |
| `disk.reserve_pct` | `5` | Per-disk reserved headroom for the `max:` calculation (capped at 10 GiB) |
| `disk.disk_type_auto` | `true` | Auto-pick `hdd` vs `ssd` from `/sys/block/.../queue/rotational` |
| `disk.auto_idx_tier` | `false` | When a host has both rotational and small fast disks, carve the smallest fast one out as `weed volume -dir.idx` storage |
| `disk.idx_tier_size_ratio` | `0.333` | Threshold for auto_idx_tier (smallest fast must be ≤ ratio × smallest slow) |
| `disk.allow_ephemeral` | `false` | Include AWS Nitro instance store / GCP local SSD even though they're non-durable |
## hosts
Each `hosts` entry must have an `ip` and at least one `role`. Per-host overrides for `ssh:` and `labels:` shadow the inventory `defaults`.
```yaml
hosts:
- ip: 10.0.0.11
roles: [master]
- ip: 10.0.0.13
roles: [master, filer]
- ip: 10.0.0.21
roles: [volume]
labels: {zone: us-east-1a, rack: r1}
ssh:
user: ubuntu
port: 2222
identity: ~/.ssh/aws_id_ed25519
```
## Roles
| Role | Goes into `cluster.yaml` section | Notes |
| --- | --- | --- |
| `master` | `master_servers:` | Raft quorum; usually 3 |
| `volume` | `volume_servers:` | Disk count + sizes derived from probe |
| `filer` | `filer_servers:` | Metadata store via `--filer-backend` |
| `s3` | `s3_servers:` | S3 gateway; auto-wired to first filer |
| `sftp` | `sftp_servers:` | SFTP gateway; auto-wired to first filer |
| `admin` | `admin_servers:` | Admin UI; password starts as `CHANGE_ME` |
| `worker` | `worker_servers:` | Maintenance worker; auto-wired to first admin |
| `envoy` | `envoy_servers:` | Edge proxy |
| `external` | nothing emitted | Documented-but-unmanaged hosts (e.g. an external Postgres). Used with `tag:` for `--filer-backend` substitution; never SSH-probed |
A host with multiple roles produces one entry per role across the matching sections. Inventory-side validation rejects duplicate `(ip, role)` pairs.
## SSH overrides
Per-host `ssh:` shadows the inventory `defaults.ssh`. Two rows that share an `ip:ssh-port` target must agree on user/identity (the probe deduplicates by SSH endpoint, so disagreeing rows would silently let one override the other):
```yaml
hosts:
- ip: 10.0.0.10
roles: [master, volume] # one SSH session, both roles share ssh config
- ip: 10.0.0.10
roles: [filer]
ssh: { user: ubuntu } # ERROR: conflicts with the master/volume rows
```
## Labels (DC / Rack)
`labels.zone` and `labels.rack` map onto `DataCenter` / `Rack` fields on the volume server spec (and future filer/s3 fields as they grow). Use them when SeaweedFS's volume placement should respect rack/AZ topology.
```yaml
hosts:
- ip: 10.0.2.10
roles: [volume]
labels: {zone: us-east-1a, rack: r1}
- ip: 10.0.2.20
roles: [volume]
labels: {zone: us-east-1b, rack: r2}
```
Other labels are preserved as inventory annotations but not (yet) consumed by `plan`.
## Disk knobs
The default disk discovery scans every `/dev/sd*`, `/dev/nvme*`, `/dev/xvd*`, `/dev/vd*` block device, skips boot/partitioned/already-mounted disks, and emits one `folders:` entry per remaining device with `max:` derived from size.
Common overrides:
```yaml
defaults:
disk:
# Drop /dev/sda from every host (its partitions usually carry the OS).
exclude: ["/dev/sda"]
# Treat `roles: [volume]` as the data tier even on Nitro EC2 (default
# skips the instance store because it's non-durable).
allow_ephemeral: true
# On a host with both HDDs and a small fast SSD, carve the SSD out
# as `weed volume -dir.idx=...` storage instead of using it as data.
auto_idx_tier: true
```
Already-mounted `/data<N>` disks are recognized as cluster-owned and re-emitted using their existing mountpoint; foreign mounts (`/`, `/var/lib/docker`, anything else) are skipped.
## Tags and the `external` role
Mark a host with `roles: [external]` and a `tag:` to reference it symbolically from `--filer-backend`:
```yaml
hosts:
# SSH-probed hosts...
- ip: 10.0.0.13
roles: [master, filer]
# Filer metadata store — not SSH-managed, never probed.
# Declared so --filer-backend can reference it by tag.
- ip: 10.0.0.41
roles: [external]
tag: postgres-metadata
```
Then the operator can write `--filer-backend postgres://user:pw@tag:postgres-metadata:5432/db?sslmode=disable`. Plan substitutes the tagged host's IP before parsing the DSN, so the generated `cluster.yaml` carries the resolved address — `cluster deploy` doesn't need to know about tags. Decouples "where the metadata DB lives" (one inventory edit) from "what its credentials are" (the DSN file or env var stays stable across IP changes).
Tag substitution only runs on the URL authority's host segment, so a literal `tag:` in a password (`user:tag:secret@…`) or query value (`?note=tag:prod`) is left alone.
## Validation rules
`plan` rejects an inventory at load time when:
- A host has no `ip`.
- A host has no roles.
- A role is unknown.
- A host declares the same `(ip, role)` pair twice.
- Two rows share an `ip:ssh-port` target but disagree on SSH user/identity.
- A `disk.device_globs` or `disk.exclude` entry contains anything fancier than an optional trailing `*`.
- Two hosts carry the same `tag:` (would make tag substitution ambiguous).
Anything else is fine; the planner errs on the side of accepting weird-but-unambiguous inventory shapes and warning at run time.
+152
@@ -0,0 +1,152 @@
[seaweed-up](https://github.com/seaweedfs/seaweed-up) ships a `cluster plan` command that turns a host inventory into the `cluster.yaml` that `cluster deploy` consumes. Instead of hand-writing the topology and chasing per-host disk paths and ports, you list hosts and the roles each one runs; `plan` SSHes to each box, discovers its hardware, and synthesizes a reviewable spec.
This page walks through the workflow end to end. For the inventory schema in detail see [[Cluster Plan Inventory Reference]]; for ongoing operations see [[Cluster Plan Day 2 Operations]].
## Contents
- [Mental model](#mental-model)
- [1. Write inventory.yaml](#1-write-inventoryyaml)
- [2. Probe (optional, dry information-gathering)](#2-probe-optional-dry-information-gathering)
- [3. Synthesize cluster.yaml](#3-synthesize-clusteryaml)
- [4. Preview before applying changes](#4-preview-before-applying-changes)
- [5. Deploy](#5-deploy)
- [6. Day-2: grow the cluster](#6-day-2-grow-the-cluster)
- [7. Day-2: a host's hardware changed](#7-day-2-a-hosts-hardware-changed)
- [Escape hatches](#escape-hatches)
## Mental model
| Phase | Verb | Mutates? | Use when |
| --- | --- | --- | --- |
| 1 | `plan --json` | nothing | sanity-checking SSH/probe |
| 2 | `plan -o cluster.yaml` (greenfield) | creates files | first time |
| 3 | `plan -o cluster.yaml` (re-run) | append-merge | adding hosts |
| 4 | `plan --dry-run` | nothing | preview any of the above |
| 4 | `plan --refresh-host=<ip>` | swaps one entry | drift remediation |
| — | `deploy -f cluster.yaml` | the cluster | apply the spec |
The pattern: inventory is your source of truth; `cluster.yaml` is a reviewable intermediate; `deploy` only acts on `cluster.yaml`.
## 1. Write inventory.yaml
The inventory is the only file you author by hand. It lists hosts, the roles each one runs, and SSH connection details. Everything else (disk layout, ports, max counts) is discovered or defaulted.
```yaml
defaults:
ssh:
user: root
port: 22
identity: ~/.ssh/id_rsa
hosts:
- ip: 10.0.0.11
roles: [master]
- ip: 10.0.0.12
roles: [master]
- ip: 10.0.0.13
roles: [master, filer]
- ip: 10.0.0.21
roles: [volume]
labels: {zone: us-east-1a, rack: r1}
- ip: 10.0.0.22
roles: [volume]
```
Recognized roles: `master`, `volume`, `filer`, `s3`, `sftp`, `admin`, `worker`, `envoy`, `external`. A host can carry multiple roles; `plan` emits one entry per role into the corresponding `*_servers` section. Per-host `ssh:` and `labels:` override the inventory `defaults`. See [[Cluster Plan Inventory Reference]] for every field.
## 2. Probe (optional, dry information-gathering)
```bash
seaweed-up cluster plan -i inventory.yaml --json > facts.json
```
Read-only on every target. Useful for sanity-checking that SSH works and seeing what disks the planner sees on each host before you let it write anything.
## 3. Synthesize cluster.yaml
```bash
seaweed-up cluster plan -i inventory.yaml -o cluster.yaml \
--filer-backend-file /etc/seaweed-up/filer.dsn
```
`plan` SSHes to each host, generates a `cluster.yaml` (the file `cluster deploy` consumes), and writes two sidecars next to it:
- `cluster.facts.json` — the raw probe output (audit + drift baseline for the next run)
- `cluster.deploy-disks.json` — the per-host allowlist of disks `deploy` is permitted to mkfs+mount
The generated `cluster.yaml` carries a header comment marking it plan-generated and is safe to hand-edit. You'll want to review it before the first deploy.
### Filer backend
If your filer needs an external metadata store (Postgres, MySQL, Redis…), pass the DSN with one of:
```bash
# 1. file (recommended: avoids leaking the password via `ps`)
seaweed-up cluster plan -i inventory.yaml -o cluster.yaml \
--filer-backend-file /etc/seaweed-up/filer.dsn
# 2. direct CLI flag
seaweed-up cluster plan -i inventory.yaml -o cluster.yaml \
--filer-backend 'postgres://seaweed:CHANGE_ME@10.0.0.41:5432/seaweedfs?sslmode=disable'
# 3. environment variable (good for CI)
SEAWEEDUP_FILER_BACKEND='postgres://seaweed:s3cret@10.0.0.41/seaweedfs' \
seaweed-up cluster plan -i inventory.yaml -o cluster.yaml
```
Precedence is `file > flag > env`. You can also reference a tagged inventory host symbolically — see [[Cluster Plan Inventory Reference]]'s tag section.
## 4. Preview before applying changes
```bash
seaweed-up cluster plan -i inventory.yaml -o cluster.yaml --dry-run
```
Prints a unified diff of what the next `plan` run *would* write, without touching anything on disk. This is the safe review step before any rewrite. Sidecars are summarized but not written.
## 5. Deploy
```bash
seaweed-up cluster deploy -f cluster.yaml
```
Reads `cluster.yaml` plus the `deploy-disks.json` sidecar (fail-closed: if the sidecar is missing on a plan-generated spec, deploy refuses rather than scanning every disk). Brings up systemd units, mounts disks, configures filer/admin/etc.
## 6. Day-2: grow the cluster
Add a host to `inventory.yaml`:
```yaml
- ip: 10.0.0.23
roles: [volume]
```
Re-run plan:
```bash
seaweed-up cluster plan -i inventory.yaml -o cluster.yaml
```
The default behavior when `-o` already exists is **append-merge** — every existing entry stays byte-identical (your hand edits, your inline comments, your key ordering survive), and the new `10.0.0.23` row gets appended to `volume_servers`. Then `cluster deploy -f cluster.yaml` again.
## 7. Day-2: a host's hardware changed
You added a disk to `10.0.0.21`. Run `plan` and you'll see:
```
WARN: drift on 10.0.0.21:22 (since previous facts.json): added /dev/sdc
```
`plan` compared the new probe against the previous run's `facts.json` and noticed the disk count moved. The YAML wasn't touched — drift is informational. To actually update that one host's entry without disturbing the rest of the file:
```bash
seaweed-up cluster plan -i inventory.yaml -o cluster.yaml --refresh-host 10.0.0.21
```
That host's `master_servers` / `filer_servers` / `volume_servers` rows get re-emitted from fresh facts; every other entry stays byte-identical. Then deploy.
See [[Cluster Plan Day 2 Operations]] for the full drift / refresh / dry-run workflow.
## Escape hatches
- `--overwrite` — regenerate `cluster.yaml` from scratch, discarding all hand edits. Use when you've drifted too far to merge cleanly.
- Hand-written `cluster.yaml` files are still fully supported — see [[Deployment with seaweed-up]] for the no-inventory workflow.
+2
@@ -2,6 +2,8 @@
It is complementary to the Kubernetes Helm chart / Operator: use the [Operator](Deployment-to-Kubernetes-and-Minikube) on Kubernetes, and `seaweed-up` on plain VMs or bare-metal.
> **Tip:** if you'd rather not hand-write `cluster.yaml`, the [[Cluster Plan Workflow]] generates it from a much smaller `inventory.yaml` (just hosts and roles) by SSHing to each box and discovering disks/ports/sizes. Both flows are fully supported and `cluster deploy -f cluster.yaml` doesn't care which produced the file.
## Contents
- [Install](#install)
+3
@@ -172,6 +172,9 @@
* [[Data Backup]]
* [[Deployment to Kubernetes and Minikube]]
* [[Deployment with seaweed-up]]
* [[Cluster Plan Workflow]]
* [[Cluster Plan Inventory Reference]]
* [[Cluster Plan Day 2 Operations]]
### Rust Volume Server
* [[Rust Volume Server]]