seaweedfs/sw-block/design/v3-operations-design.md

V3 Operations Design

Date: 2026-04-13 Status: design draft Purpose: define the user-facing operations experience for sw-block

1. Design Principles

1.1 One command to start

A developer should go from zero to a working block volume in under 10 seconds with one command. No config files, no YAML, no separate services.

1.2 Dry-run by default

Every destructive operation previews what it will do. The user must explicitly confirm with -apply. This is already the SeaweedFS shell convention — extend it to all block operations.

1.3 Shell is the operations interface

weed shell is the single entry point for admin operations. No separate CLI tool. The shell already provides tab completion, command history, distributed locking, and 90+ commands. Block operations plug into the same framework.

1.4 Health at a glance

An admin should never need to grep logs to understand cluster state. block.health shows everything in one screen. Green/yellow/red, not log lines.

1.5 Developer sees only PVC

K8s developers interact with storage through PersistentVolumeClaim. They never see iSCSI, NVMe-oF, replication, failover, or rebuild. The CSI driver handles everything.

1.6 Three personas, three interfaces

Persona	Interface	What they see
Developer	K8s PVC / REST API	"Give me 100GB storage"
Operator	weed shell	Volume health, failover, rebuild, snapshots
Platform engineer	Config + monitoring	Cluster sizing, Prometheus, alerting

2. Setup Experience

2.1 Single-node development (10 seconds)

# Download
curl -O https://github.com/seaweedfs/seaweedfs/releases/latest/weed

# Start everything in one process
./weed server -block -block.size=100G -dir=/data

# Output:
# Master started on :9333
# Volume server started on :8480
# Block service started (iSCSI :3260)
# Created default volume "vol-1" (100GB, RF=1)
#
# Connect from this machine:
#   iscsiadm -m discovery -t st -p localhost:3260
#   iscsiadm -m node -T iqn.2024-01.com.seaweedfs:vol.vol-1 -l
#
# Or use NVMe-oF:
#   nvme discover -t tcp -a 127.0.0.1 -s 4420

The output tells the user exactly what to do next. No documentation needed for the first connection.

2.2 Two-node replicated (30 seconds)

# Node 1 (master + volume server):
./weed server -block -block.size=100G -ip=10.0.0.1

# Node 2 (volume server only):
./weed volume -block -block.size=100G -ip=10.0.0.3 \
  -mserver=10.0.0.1:9333

# Create replicated volume:
./weed shell -master=10.0.0.1:9333
> block.create -name=mydb -size=100G -rf=2 -apply
  Created "mydb" (100GB, RF=2, sync_all)
  Primary: 10.0.0.1:8480
  Replica: 10.0.0.3:8480
  iSCSI:   iqn.2024-01.com.seaweedfs:vol.mydb at 10.0.0.1:3260

2.3 Kubernetes (2 minutes)

# Install CSI driver
helm repo add seaweedfs https://seaweedfs.github.io/charts
helm install sw-block seaweedfs/sw-block-csi \
  --set master.address=10.0.0.1:9333

# Create StorageClass
kubectl apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: sw-block
provisioner: seaweedfs.com/block
parameters:
  replicaFactor: "2"
  durabilityMode: sync_all
allowVolumeExpansion: true
EOF

# Developer just uses PVC:
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-database
spec:
  storageClassName: sw-block
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 100Gi
EOF

2.4 Setup comparison

System	Commands to first working volume	Time
sw-block	1 (weed server -block)	10s
MinIO	1 (minio server)	5s (object only, no block)
SeaweedFS (object)	1 (weed server)	5s
Longhorn	3 (helm install + PVC)	5min
Ceph	5-8 (cephadm bootstrap + OSD + RBD)	30min
TrueNAS	Web installer + wizard	15min

3. Shell Commands

3.1 Command naming convention

All block commands start with block.. Subcommands use dots. This matches the existing shell convention (volume.list, fs.ls, ec.encode).

3.2 Volume lifecycle

block.create     Create a new block volume
block.delete     Delete a block volume (dry-run by default)
block.list       List all block volumes with health status
block.status     Show detailed status of one volume
block.expand     Expand volume size (online, no downtime)

3.3 Data protection

block.snapshot          Create a CoW snapshot
block.snapshot.list     List snapshots for a volume
block.snapshot.delete   Delete a snapshot
block.snapshot.export   Export snapshot to S3 (SeaweedFS object store)
block.snapshot.restore  Restore volume from snapshot

3.4 HA operations

block.promote    Manual failover: promote replica to primary
block.rebuild    Trigger manual rebuild for a replica
block.assign     Change replica assignment (add/remove replica)

3.5 Cluster health

block.health     Cluster-wide health summary
block.servers    List block-capable volume servers
block.events     Recent events (failover, rebuild, degraded)

3.6 Benchmarking

block.bench      Run fio benchmark on a volume (creates temp volume if needed)
block.bench.compare  Compare performance of two runs

3.7 Detailed command specs

block.list

> block.list
  NAME          SIZE    RF  MODE             PRIMARY          REPLICA          HEALTH  IOPS
  pvc-data-01   100G    2   publish_healthy  10.0.0.1:8480    10.0.0.3:8480    1.0     45.2K
  pvc-db-01     50G     2   degraded         10.0.0.3:8480    (rebuilding 78%) 0.8     38.1K
  pvc-temp      200G    1   healthy          10.0.0.1:8480    —                1.0     52.0K

> block.list -json
  [{"name":"pvc-data-01","size":107374182400,"rf":2,"mode":"publish_healthy",...}]

Columns:

• NAME: volume name
• SIZE: allocated size
• RF: replica factor
• MODE: engine projection mode
• PRIMARY: current primary server
• REPLICA: replica server(s) with status
• HEALTH: 0.0-1.0 score
• IOPS: current write IOPS (from last heartbeat)

block.status

> block.status mydb
  Volume:     mydb
  Size:       100G (used: 34G, 34%)
  RF:         2 (sync_all)
  Mode:       publish_healthy
  Epoch:      3
  
  Primary:    10.0.0.1:8480
    WAL head: 284756
    Checkpoint: 284700
    Uptime:   3d 14h
  
  Replica:    10.0.0.3:8480
    State:    in_sync
    Flushed:  284756
    Lag:      0 entries
  
  iSCSI:      iqn.2024-01.com.seaweedfs:vol.mydb at 10.0.0.1:3260
  NVMe-oF:    nqn.2024-01.com.seaweedfs:vol.mydb at 10.0.0.1:4420
  
  Snapshots:
    mydb@2026-04-13T10:30  34G  (3 hours ago)
    mydb@2026-04-12T22:00  32G  (15 hours ago)
  
  Recent events:
    2026-04-11 02:12  rebuild completed (2.1s, 1GB transferred)
    2026-04-09 22:54  auto-failover (10.0.0.1 → 10.0.0.3, epoch 1→2)
  
  Health:     1.0 (last scrub: 2026-04-12, 0 errors)

block.health

> block.health
  ╔═══════════════════════════════════════╗
  ║  BLOCK CLUSTER HEALTH: OK            ║
  ╚═══════════════════════════════════════╝
  
  Volumes:    12 total
              11 healthy
               1 degraded (pvc-db-01: rebuilding)
               0 offline
  
  Servers:    3 online, 0 offline
              10.0.0.1:8480  ██████████  6 volumes, 450G used
              10.0.0.3:8480  ████████    5 volumes, 380G used
              10.0.0.5:8480  ██████      4 volumes, 280G used
  
  Capacity:   1.1T used / 3.0T total (37%)
  
  Last 24h:
    Failovers: 0
    Rebuilds:  1 (pvc-db-01, in progress, 78%)
    Scrub errors: 0
  
  Alerts:     none

block.create (dry-run)

> block.create -name=analytics -size=500G -rf=2
  DRY RUN — no changes made
  
  Would create volume "analytics":
    Size:       500G
    RF:         2 (sync_all)
    Primary:    10.0.0.5:8480 (most free space: 720G)
    Replica:    10.0.0.1:8480 (550G free, different rack)
    iSCSI IQN: iqn.2024-01.com.seaweedfs:vol.analytics
  
  Run with -apply to execute.

> block.create -name=analytics -size=500G -rf=2 -apply
  Created "analytics" (500GB, RF=2, sync_all)
  Primary: 10.0.0.5:8480
  Replica: 10.0.0.1:8480
  iSCSI:   iqn.2024-01.com.seaweedfs:vol.analytics at 10.0.0.5:3260

block.delete (always requires -apply -confirm)

> block.delete -name=old-vol
  DRY RUN — no changes made
  
  Would delete volume "old-vol":
    Size:       50G (used: 23G)
    RF:         2 (primary: 10.0.0.1, replica: 10.0.0.3)
    Snapshots:  3 (would also be deleted)
    Age:        45 days
  
  WARNING: This permanently destroys all data and snapshots.
  Run with -apply -confirm to execute.

> block.delete -name=old-vol -apply -confirm
  Deleted "old-vol" (50GB, 3 snapshots removed)

block.promote (manual failover)

> block.promote -name=mydb -target=10.0.0.3:8480
  DRY RUN — no changes made
  
  Would promote 10.0.0.3:8480 to primary for "mydb":
    Current primary: 10.0.0.1:8480 (epoch 2)
    New primary:     10.0.0.3:8480 (would become epoch 3)
    Replica state:   in_sync (lag: 0 entries)
    Impact:          iSCSI connections will reconnect (~2s disruption)
  
  Run with -apply to execute.

> block.promote -name=mydb -target=10.0.0.3:8480 -apply
  Promoted 10.0.0.3:8480 to primary for "mydb" (epoch 2→3)
  Old primary 10.0.0.1:8480 will rejoin as replica after next heartbeat.

4. REST API

4.1 Endpoints

All under /block/ prefix on the master server.

Method	Path	Description
GET	/block/volumes	List all block volumes
POST	/block/volumes	Create block volume
GET	/block/volumes/{name}	Get volume status
DELETE	/block/volumes/{name}	Delete volume
PUT	/block/volumes/{name}/expand	Expand volume
POST	/block/volumes/{name}/promote	Manual failover
POST	/block/volumes/{name}/snapshots	Create snapshot
GET	/block/volumes/{name}/snapshots	List snapshots
DELETE	/block/volumes/{name}/snapshots/{id}	Delete snapshot
GET	/block/health	Cluster health summary
GET	/block/servers	List block-capable servers
GET	/block/events	Recent events

4.2 Examples

# Create volume
curl -X POST http://master:9333/block/volumes \
  -H "Content-Type: application/json" \
  -d '{"name":"mydb","sizeBytes":107374182400,"replicaFactor":2}'

# Response:
{
  "name": "mydb",
  "sizeBytes": 107374182400,
  "replicaFactor": 2,
  "durabilityMode": "sync_all",
  "primary": "10.0.0.1:8480",
  "replica": "10.0.0.3:8480",
  "iscsiIqn": "iqn.2024-01.com.seaweedfs:vol.mydb",
  "iscsiPortal": "10.0.0.1:3260",
  "epoch": 1,
  "mode": "bootstrap_pending"
}

# Get health
curl http://master:9333/block/health

# Response:
{
  "status": "ok",
  "volumes": {"total": 12, "healthy": 11, "degraded": 1},
  "servers": {"total": 3, "online": 3},
  "capacity": {"usedBytes": 1181116006400, "totalBytes": 3221225472000}
}

5. Observability

5.1 Prometheus metrics

Already implemented. Key metrics:

# Volume level
sw_block_volume_mode{name="mydb"} 1                    # 1=healthy
sw_block_volume_iops{name="mydb",op="write"} 45200
sw_block_volume_size_bytes{name="mydb"} 107374182400
sw_block_volume_used_bytes{name="mydb"} 36507222016

# Replica level
sw_block_replica_state{name="mydb",server="10.0.0.3"} 3  # 3=in_sync
sw_block_replica_lag_entries{name="mydb",server="10.0.0.3"} 0

# Cluster level
sw_block_volumes_total 12
sw_block_volumes_healthy 11
sw_block_volumes_degraded 1
sw_block_servers_online 3

# Events
sw_block_failovers_total 2
sw_block_rebuilds_total 5
sw_block_rebuild_duration_seconds{name="mydb"} 2.1

5.2 Alerting rules

Recommended Prometheus alert rules:

groups:
  - name: sw-block
    rules:
      - alert: BlockVolumeDegraded
        expr: sw_block_volume_mode != 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Block volume {{ $labels.name }} is degraded"

      - alert: BlockVolumeOffline
        expr: sw_block_volume_mode == 0
        for: 1m
        labels:
          severity: critical

      - alert: BlockRebuildStalled
        expr: sw_block_rebuild_duration_seconds > 300
        labels:
          severity: warning
          summary: "Rebuild for {{ $labels.name }} taking > 5 minutes"

      - alert: BlockCapacityHigh
        expr: sw_block_cluster_used_bytes / sw_block_cluster_total_bytes > 0.85
        labels:
          severity: warning
          summary: "Block cluster capacity > 85%"

5.3 Grafana dashboard

Provide a pre-built Grafana dashboard JSON that shows:

• Cluster health panel (big green/yellow/red indicator)
• Volume list with mode, health, IOPS
• IOPS over time (per-volume and aggregate)
• Latency percentiles (p50, p99, p999)
• Rebuild/failover event markers
• Capacity trend
• Per-server utilization

6. Error Messages

Every error should tell the user what happened, why, and what to do:

# Bad:
Error: volume not found

# Good:
Error: block volume "mydb" not found.
Available volumes: pvc-data-01, pvc-db-01, pvc-temp
Run 'block.list' to see all volumes.

# Bad:
Error: cannot create volume

# Good:
Error: cannot create volume "analytics" (500GB, RF=2).
Reason: not enough servers. Need 2 servers for RF=2, but only 1 is online.
Run 'block.servers' to check server status.

# Bad:
Error: promote failed

# Good:
Error: cannot promote 10.0.0.3:8480 for "mydb".
Reason: replica is not in_sync (current state: rebuilding, 78% complete).
Wait for rebuild to complete, then retry.
Run 'block.status mydb' for details.

7. Comparison with Competitors

7.1 What we steal

From	What	How
MinIO	One-command startup	weed server -block
SeaweedFS	Shell as operations interface	block.* commands in weed shell
Ceph	Dry-run on all destructive ops	-apply flag required
Ceph	Declarative cluster spec	Future: block.apply -f cluster.yaml
Longhorn	Auto-rebuild on config change	Change RF in shell, system handles rebuild
Longhorn	One-click snapshot	block.snapshot mydb (one command, no params)
TrueNAS	iSCSI setup wizard	block.setup interactive guide
TrueNAS	Actionable error messages	Every error says what to do next

7.2 What we do better

Area	Competitors	sw-block
Setup time	Ceph: 30min, Longhorn: 5min	10 seconds
Write IOPS (RF=2)	Ceph: 4.5K	61K (13.5x)
Rebuild time (1GB)	Ceph: minutes	2 seconds
CLI	Ceph: many tools, Longhorn: kubectl	Single weed shell
Object + Block	Separate systems	Same binary, same cluster

7.3 What we do not provide (yet)

Feature	Who has it	Priority
Web dashboard	Ceph, Longhorn, TrueNAS	P1 (existing foundation)
Declarative cluster spec	Ceph ceph orch	P3
Multi-cluster management	None (all single-cluster)	Not planned

8. Existing UI / Admin Foundation (V1)

V1 already built several admin interfaces that V3 should reuse:

8.1 iSCSI Admin HTTP Server

File: weed/storage/blockvol/iscsi/cmd/iscsi-target/admin.go

Provides HTTP endpoints for the standalone iscsi-target binary:

Endpoint	Method	What it does
/status	GET	Volume status JSON (role, epoch, WAL head, replica state)
/assign	POST	Inject assignment (epoch, role, lease TTL)
/replica	POST	Set WAL shipping target (data/ctrl addr)
/rebuild	POST	Start/stop rebuild server
/snapshot	POST	Create/delete/restore/list snapshots
/resize	POST	Expand volume size
/metrics	GET	Prometheus metrics (with optional auth token)

These endpoints powered the CP4b-4 HA integration tests. The admin server runs alongside the iSCSI target and controls it via HTTP.

V3 action: Move these endpoints into weed server's volume server HTTP handler. The standalone binary keeps its admin server for testing; the production path uses the integrated weed server endpoints.

8.2 Grafana Dashboard

File: weed/storage/blockvol/monitoring/dashboards/block-overview.json

Pre-built Grafana dashboard with panels for:

• Cluster health (stat panel)
• WAL pressure state
• Volumes by role (bar gauge)
• Prometheus metric queries

V3 action: Update metric names if changed, add V2 metrics (rebuild duration, failover count, shipper state). The dashboard is importable as-is into any Grafana instance.

8.3 Master UI HTML

File: weed/server/master_ui/master.html

The master already serves an HTML UI at http://master:9333/ showing:

• Cluster topology (data centers, racks, nodes)
• Volume count and IDs per node
• Raft cluster state

V3 action: Add a "Block Volumes" tab that calls /block/volumes REST API and renders a table. This is the fastest path to a web UI — ~200 lines of HTML/JS added to the existing template.

8.4 Volume Server UI HTML

File: weed/server/volume_server_ui/volume.html

The volume server serves an HTML UI at http://vs:8480/ui/ showing local volume information.

V3 action: Add block volume section showing local BlockVol status, iSCSI target info, and replica state.

8.5 Prometheus Metrics

Already integrated in weed server. Key block metrics are exported via the standard /metrics endpoint. No separate metrics server needed.

9. Implementation Phases

Phase 1: Shell commands + Master UI tab (P0)

Shell commands in weed/shell/:

• block.list — volume list with health
• block.status — detailed volume status
• block.create — create with dry-run
• block.delete — delete with dry-run + confirm
• block.health — cluster health summary

These call existing master gRPC APIs. No new backend code needed.

Master UI block tab in weed/server/master_ui/master.html:

• Add "Block Volumes" tab to existing UI
• Table: name, size, RF, mode, primary, replica, health
• Auto-refresh every 5 seconds
• Calls /block/volumes REST endpoint

Estimated: ~600 lines (400 shell + 200 HTML/JS).

Phase 2: REST API + Grafana update (P1)

REST API in weed/server/master_server_handlers_block.go:

• CRUD endpoints for volumes
• Health endpoint
• Server list endpoint

These wrap existing BlockVolumeRegistry and blockAssignmentQueue methods. JSON request/response.

Grafana dashboard update:

• Update block-overview.json with V2/V3 metrics
• Add rebuild duration, failover count panels
• Add per-volume IOPS panel

Estimated: ~350 lines (300 API + 50 dashboard JSON).

Phase 3: Enhanced operations (P2)

• block.snapshot / block.snapshot.list
• block.expand
• block.promote with dry-run
• block.events (recent failover/rebuild log)
• block.bench (built-in fio wrapper)
• Migrate iSCSI admin endpoints to weed server

Phase 4: Developer experience (P2)

• weed server -block one-click mode
• Actionable error messages for all block operations
• Connection string output on volume creation
• block.setup interactive wizard

Phase 5: Full Web UI (P3)

Two options:

Option A: Extend master UI (lightweight)

• Add volume detail page (click volume name → full status)
• Add snapshot/expand/promote buttons
• Add cluster health dashboard panel
• ~500 lines HTML/JS, no framework

Option B: Standalone dashboard (full)

• Separate SPA (React/Vue) served from master
• Real-time WebSocket updates
• Performance charts (IOPS, latency over time)
• User auth integration
• ~2000+ lines, needs framework

Recommendation: Start with Option A. Move to Option B only when customer demand justifies the investment.

10. Existing vs New Code Map

Feature	Existing Code	New Code Needed
Volume CRUD	BlockVolumeRegistry, CreateBlockVolume gRPC	Shell formatting + REST wrapper
Status/health	BlockVolumeEntry, heartbeat data	Shell formatting
Prometheus metrics	weed/storage/blockvol/metrics.go	Already done
Grafana dashboard	monitoring/dashboards/block-overview.json	Update metrics
Master web UI	master_ui/master.html	Add Block tab
Volume server UI	volume_server_ui/volume.html	Add Block section
Admin HTTP (iSCSI)	iscsi/cmd/iscsi-target/admin.go	Migrate to weed server
Snapshot API	blockvol.CreateSnapshot() etc	Shell + REST wrapper
Failover API	PromoteBestReplica(), ManualPromote()	Shell wrapper
Rebuild API	StartRebuildFromProbe()	Shell wrapper

Most of the backend exists. The work is wiring it to user-facing interfaces (shell, REST, HTML).