seaweedfs/sw-block/design/v2-sync-recovery-protocol.md

# V2 Sync / Recovery Protocol

Date: 2026-04-07
Status: active design

## Purpose

Define the complete replication protocol for sw-block V2. This document
covers sync, keepup, catch-up, and rebuild as one unified protocol so
the sw agent has full context for implementation.

`v2-rebuild-mvp-session-protocol.md` is the narrower first-slice spec.
This document is the surrounding context and long-term design.

## Design Principles

1. **Primary decides everything.** Replica only reports facts and executes
   contracts. Replica never self-escalates to `needs_rebuild`.

2. **One threshold.** `applied_lsn >= primary_wal_tail` → WAL-recoverable.
   Otherwise → rebuild. Matches Ceph's `last_update >= log_tail`.

3. **Deterministic engine.** Event in → state + commands + projection out.
   No side effects inside the engine. Host executes commands.

4. **Three authority layers:**
   - Assignment: master → identity (who is primary/replica, epoch, replica set)
   - Session: primary → per-replica recovery contract (keepup/catchup/rebuild)
   - Projection: primary → derived volume mode/health

5. **Failure never auto-escalates.** A failed session stays `failed`. The
   primary re-decides from fresh `syncAck` facts. Only the primary can
   issue a rebuild.

## Reference Systems

| System | Catch-up | Rebuild | Decision owner | Decision input |
|---|---|---|---|---|
| Ceph | PG log replay | Full backfill | Primary OSD (peering) | `last_update >= log_tail` |
| Mayastor | None | Segment copy | Control plane | Child sync state |
| Longhorn | None | Snapshot file sync | Controller | Revision counters |
| **sw-block V2** | WAL replay | Flushed extent + live WAL (two-line) | **Primary** | `applied_lsn >= wal_tail` |

## Protocol Overview

### Normal Operation (keepup)

```
Primary                          Replica
  │                                │
  ├─ WriteLBA ────────────────────►│ (live WAL shipping via ShipAll)
  │                                │ apply to local WAL
  │                                │
  ├─ sync(target_lsn=N) ─────────►│
  │◄─ syncAck(durable=N, applied=N)│
  │                                │
  │  decision: quorum → keepup     │
  │  derive: publish_healthy       │
```

### Catch-up (replica behind but within retained WAL)

```
Primary                          Replica
  │                                │
  ├─ sync(target_lsn=1000) ──────►│
  │◄─ syncAck(applied=500)         │
  │                                │
  │  decision: 500 >= wal_tail(100)│
  │  → WAL catch-up               │
  │                                │
  ├─ sessionControl(start_catchup  │
  │    start=500, target=1000,     │
  │    pin=500) ─────────────────►│
  │                                │
  │  LINE 1: WAL replay [500..1000]│
  ├─ walReplay(lsn=501...) ──────►│ apply, pin advances
  │                                │
  │  LINE 2: live WAL from 1001+   │
  ├─ walData(lsn=1001...) ───────►│ apply to local WAL
  │                                │
  │◄─ sessionAck(completed,        │
  │    achieved=1050) ─────────────│
  │                                │
  │  replica back in keepup        │
```

### Rebuild (replica beyond retained WAL, or fresh join)

```
Primary                          Replica
  │                                │
  ├─ sync(target_lsn=5000) ──────►│
  │◄─ syncAck(applied=0)           │
  │                                │
  │  decision: 0 < wal_tail(2000)  │
  │  → rebuild                    │
  │                                │
  ├─ sessionControl(start_rebuild  │
  │    base_lsn=flushedLSN,       │
  │    base_kind=flushed_extent) ─►│
  │                                │
  │  LINE 1: flushed extent blocks │
  ├─ sessionData(chunk...) ──────►│ apply if bitmap clear
  │                                │
  │  LINE 2: live WAL from base_lsn+1
  ├─ walData(lsn=5001...) ───────►│ apply, set bitmap bit
  │                                │
  │  Bitmap: WAL-applied LBA wins  │
  │  over later base block         │
  │                                │
  │◄─ sessionAck(base_complete)    │
  │◄─ sessionAck(completed,        │
  │    achieved=5200) ─────────────│
  │                                │
  │  replica back in keepup        │
```

## Primary Decision Logic

```
func decide(ack SyncAck, walTail, walHead uint64) SessionKind {
    replicaPos := max(ack.AppliedLSN, ack.DurableLSN)

    if replicaPos >= walHead && replicaPos > 0:
        return keepup       // fully caught up

    if replicaPos >= walTail && replicaPos > 0:
        return catchup      // behind but within retained WAL

    if replicaPos == 0 && walTail <= 1:
        return catchup      // fresh replica, WAL retained from beginning

    return rebuild          // gap exceeds retained WAL
}
```

This is one function, one threshold. Matches Ceph's `last_update >= log_tail`.

## Normalized Primary-side Sync Facts

The wire protocol still uses `sync` and `syncAck` as the bounded control
exchange. But inside the primary-owned host/runtime layer, raw wire results and
local control-path observations may be normalized into one small fact
vocabulary before the primary re-decides the next step.

This normalization is not a new replica-visible message family. It is the
primary-owned semantic shape for "what kind of sync fact just arrived?"

Current normalized fact kinds:

| Kind | Meaning | Typical sources |
|---|---|---|
| `sync_quorum_acked` | normal sync closure reached | wire `syncAck(ack_kind=quorum)`, accepted barrier |
| `sync_quorum_timed_out` | control-plane sync closure timed out | wire `syncAck(ack_kind=timed_out)`, rejected barrier |
| `sync_replay_required` | replica is behind but replay-recoverable | fresh planner classification after sync facts |
| `sync_rebuild_required` | replica is outside retained replay coverage | fresh planner classification after sync facts |
| `sync_replay_failed` | a replay/catch-up attempt failed | catch-up failure callback, replay execution failure |

Rules:

1. normalized sync facts are still facts only, never session recommendations
2. different producers may map to the same normalized fact kind
3. only the primary may turn these facts into `keepup`, `catchup`, or `rebuild`
4. this is the bridge between raw protocol input and primary-owned session
   authority

## Two-Line Recovery Model

Both catch-up and rebuild use two concurrent data lines:

### Catch-up: WAL replay + live WAL

- **Line 1**: replay retained WAL entries from `pin_lsn` to `target_lsn`
- **Line 2**: forward live WAL entries from `target_lsn+1` onward
- **Pin movement**: as replay cursor advances, pin can advance (releases old WAL)
- **No bitmap needed**: WAL entries are strictly ordered by LSN, no LBA conflict
- **Completion**: replay cursor reaches target → lines merge → keepup

### Rebuild: flushed extent base + live WAL

- **Line 1**: copy the primary's flushed extent image at `base_lsn`
- **Line 2**: forward live WAL entries newer than `base_lsn`
- **Bitmap required**: base blocks and WAL entries may target the same LBA
- **Bitmap rule**: bit set on WAL `applied` (not received). Base block skipped if bit set.
- **Completion**: all base blocks transferred AND `wal_applied_lsn >= target_lsn`
- **Base boundary rule**: `base_lsn` is a flushed/checkpoint boundary, not a merely committed boundary

### Why two lines instead of sequential (base → then catch-up)

Sequential model:
1. Copy entire flushed extent image
2. Then replay WAL from base LSN to current
3. Problem: must pin WAL for duration of snapshot copy (hours for large volumes)
4. Risk: WAL recycled before replay starts → must restart entire rebuild

Two-line model:
1. Copy flushed extent AND receive live WAL simultaneously
2. WAL pin pressure = only gap between current replay and live head (small)
3. If snapshot copy is slow, WAL line keeps replica current
4. Crash recovery is safe at any point (bitmap + local WAL)

## Bitmap Rules (Rebuild Only)

### When to set bit

Set bitmap bit when WAL entry is **applied to replica's local WAL**:
- Entry has been written to local WAL file
- Entry is replayable after crash
- Does NOT require flush to final extent

### When NOT to set bit

Do not set on:
- Network receive (TCP buffer)
- Queue but not yet local WAL write

### Conflict resolution

When base lane sends a chunk for LBA range:
- Bitmap clear → write base data
- Bitmap set → skip (WAL-applied data is newer)

Short form: **WAL always wins over base.**

### Crash safety

At any crash point:
- Bitmap can be volatile (session-local, in memory)
- Local WAL is durable → replay recovers all applied entries
- After crash: fresh sync → primary re-decides → new session if needed
- The bitmap itself need not persist, but its protected coverage must be
  re-hydrated from local durable WAL before a new rebuild session opens the base lane
- No need to persist bitmap across crashes in MVP

Current bounded claim:
- crash during rebuild is handled by `restart rebuild`
- this is not yet a claim of resumable rebuild with durable `base_progress`
- hydration of bitmap coverage protects durable WAL facts during the fresh
  rebuild; it does not by itself resume prior base-copy progress

### Issue #3 restart hydration rule

To close the volatile-bitmap restart hole:

1. a fresh rebuild session must rebuild bitmap coverage from local durable WAL
   newer than `base_lsn` before `accepted` becomes externally visible
2. the base lane must remain closed until this hydration completes
3. if the replica's local durable base is already newer than the claimed
   `base_lsn`, startup must fail closed instead of accepting a stale rebuild
4. this is why rebuild starts from a flushed/checkpoint boundary rather than a
   merely committed boundary: committed data may still live only in WAL, so it
   is not a safe direct-base image

## Replica State Machine

```
idle → accepted → running → base_complete → completed
                    │              │
                    └──► failed ◄──┘
```

- `idle`: no active session
- `accepted`: session contract valid, epoch/session_id accepted
- `running`: both base lane and WAL lane active
- `base_complete`: base transfer done, WAL lane still running
- `completed`: replica reports `achieved_lsn >= target_lsn`
- `failed`: session stopped without completion, reason reported

Failure does NOT auto-escalate. Primary re-decides from next syncAck.

## Mode Derivation (Projection)

Primary derives volume mode from all replica states + boundaries:

```
any replica in rebuild session     → needs_rebuild
any replica session failed         → degraded
any replica in catch-up session    → bootstrap_pending
no replicas assigned               → allocated_only
replica role + receiver ready      → replica_ready
primary + all readiness + durable  → publish_healthy
assigned but not ready             → bootstrap_pending
```

## Failure Handling

### Principle

No failure auto-escalates. All failures go through:
1. Session marked `failed` with reason
2. Primary waits for next `syncAck` from replica
3. Primary re-decides based on fresh facts

Primary-side normalized reading:

1. failure may first surface as `sync_replay_failed`
2. fresh sync/planner facts may then normalize to either:
   - `sync_replay_required`
   - `sync_rebuild_required`
3. only then does the primary issue the next session contract

### Failure scenarios

| Scenario | Replica does | Primary does |
|---|---|---|
| Transport lost during session | Reports `failed(transport_lost)` or goes silent | Marks session failed, waits for reconnect |
| Replica crash | Restarts, recovers local WAL, hydrates bitmap coverage before any new rebuild base lane opens, then reports facts via syncAck | Re-decides: if `applied_lsn >= wal_tail` → catch-up, else → new rebuild; current claim is fresh restart, not resume of prior base-copy offset |
| Primary crash | Nothing (waits for new primary) | New primary elected, fresh epoch, all replicas report via syncAck |
| Slow progress / timeout | Continues trying | Can cancel session via `cancel_session`, then re-decide |
| WAL recycled during outage | Reports `applied_lsn` which is now < `wal_tail` | Decides rebuild (gap exceeds retained WAL) |

### Failure reason vocabulary (stable)

- `epoch_mismatch`
- `transport_lost`
- `progress_stalled`
- `deadline_exceeded`
- `pin_lost`
- `snapshot_unavailable`
- `local_wal_corrupt`
- `local_extent_corrupt`

## Catch-up Details (Post-MVP)

Catch-up uses the same session contract shape as rebuild, but without
base copy:

```
sessionControl {
    op: start_catchup
    start_lsn: <replica_applied_lsn>
    target_lsn: <primary_wal_head>
    pin_lsn: <replica_applied_lsn>
}
```

Replica receives:
- WAL replay entries [start_lsn .. target_lsn] (line 1)
- Live WAL entries [target_lsn+1 ..] (line 2)

No bitmap needed because WAL entries are ordered by LSN — no LBA conflict
between replay and live.

Pin advances as replay cursor moves forward, releasing old WAL entries.

## Future: Range Bitmap / Delta Rebuild

If primary maintains persistent per-checkpoint dirty block tracking:

```
modified_blocks[checkpoint_lsn_range] → set of dirty LBAs
```

Then rebuild can skip copying blocks that haven't changed since the
replica's last known position. Only modified blocks need to be sent.

This turns rebuild from O(volume_size) to O(changed_blocks), similar to
VMware CBTT (Changed Block Tracking) or DRBD activity log.

Implementation: persist DirtyMap snapshot at each flusher checkpoint along
with the checkpoint LSN. On rebuild, union of DirtyMap snapshots from
`replica_applied_lsn` to `current_checkpoint` = minimal copy set.

## Future: Resumable Rebuild

Current V2/MVP does **not** claim resumable rebuild. After crash during rebuild,
the protocol restarts from fresh `sync` and a fresh rebuild session.

A future resumable rebuild path may be added only with explicit durable state:

1. durable `base_progress`
2. durable proof of replica-side applied WAL coverage
3. primary-side proof that historical change coverage remains reconstructible
   through retained WAL or primary-owned CBT / changed-block history
4. a live delta channel for writes that arrive after resume begins

Without those conditions, resume is not a safe current claim.

## Component Test Requirements

All tests use real BlockVol with real WAL and extent data.
No mocks for storage. Network can be in-process (localhost TCP or direct
function call).

### Protocol tests (unit, engine only)

1. `syncAck` returns facts only, never action
2. `sessionControl(start_rebuild)` rejected on epoch mismatch
3. New `session_id` supersedes old session
4. Primary decision: `applied >= wal_tail` → catch-up
5. Primary decision: `applied < wal_tail` → rebuild
6. Primary decision: `applied >= wal_head` → keepup
7. Session failure does not auto-escalate to needs_rebuild
8. Session failure then fresh syncAck → primary re-decides

### Rebuild correctness (component, real BlockVol)

These tests create real primary + replica volumes with actual WAL and
extent data:

1. **Two-line convergence**: primary writes N blocks, records a flushed
   `base_lsn`, starts rebuild session. Base lane copies extent data from that
   flushed boundary. WAL lane
   ships live entries. Verify replica has all N blocks correct at end.

2. **WAL wins over base**: primary writes block A=1, flushes base, then
   writes A=2. Rebuild sends base (A=1) and WAL (A=2). Verify replica
   has A=2 (WAL-applied wins).

3. **Bitmap on applied not received**: ship WAL entry but delay local
   apply. Send base block for same LBA. Base block should land (bitmap
   not set yet). Then apply WAL entry. Bitmap now set. Final state
   should be WAL data.

4. **Large rebuild**: 1000+ blocks, concurrent base + WAL, verify all
   blocks correct at completion.

5. **Writes during rebuild**: primary continues writing new blocks during
   rebuild. Verify new blocks reach replica via WAL lane and are
   preserved at rebuild completion.

### Crash / failure (component, real BlockVol)

1. **Crash after WAL receive before apply**: restart replica, verify
   base can still cover the LBA (bitmap was not set).

2. **Crash after WAL apply**: restart replica, verify local WAL replay
   recovers the applied data correctly.

3. **Transport lost during rebuild**: session reports `failed(transport_lost)`.
   Primary re-decides from fresh syncAck. New rebuild session starts.

4. **Rebuild completion does not restore quorum until primary accepts**:
   verify replica reports `completed` but mode stays `needs_rebuild` until
   primary processes the completion event.

### Session lifecycle (unit, engine only)

1. `idle → accepted → running → completed → keepup`
2. `idle → accepted → running → failed → (syncAck) → re-decide`
3. `running → cancel → idle`
4. Session supersede: new session_id replaces old

### End-to-end (integration, hardware — after component tests pass)

1. Fresh replica join on m01/m02: create RF=2, kill replica VS, restart,
   verify rebuild completes and volume returns to `publish_healthy`.

2. Sustained I/O during rebuild: fio running on primary while rebuild
   progresses. Verify data continuity after rebuild completion.

## Implementation Order

1. Protocol engine (`sw-block/protocol/`) — already started, 7 events, 398 lines
2. Rebuild session on blockvol layer — two-line model, bitmap, completion
3. Session control wiring on volume server — sessionControl/sessionAck
4. Flushed extent base transfer — extent read + chunk send
5. Component tests against real BlockVol
6. Integration test on hardware