seaweedfs/sw-block/design/wal-replication-v2-state-machine.md

# WAL Replication V2 State Machine

Date: 2026-03-26
Status: design proposal
Purpose: define the V2 replication state machine for a moving-head primary where replicas may transition between keep-up, catch-up, and reconstruction while the primary continues accepting writes

## Why This Document Exists

The hard part of V2 is not the existence of three modes:

- keep-up
- catch-up
- reconstruction

The hard part is that the primary head continues advancing while replicas move between those modes.

So V2 must be specified as a real state machine:

- state definitions
- state-owned LSN anchors
- allowed transitions
- retention obligations
- abort rules

This document treats edge cases as state-transition cases.

## Scope

This is a protocol/state-machine design.

It does not yet define:
- exact RPC payloads
- exact snapshot storage format
- exact implementation package boundaries

Those can follow after the state model is stable.

## Core Terms

### `headLSN`

The primary's current highest WAL LSN.

### `replicaFlushedLSN`

The highest LSN durably persisted on the replica.

### `cpLSN`

A checkpoint/snapshot base point. A snapshot at `cpLSN` represents the block state exactly at that LSN.

### `promotionBarrierLSN`

The LSN a replica must durably reach before it can re-enter `InSync`.

### `Recovery Feasibility`

Whether `(startLSN, endLSN]` can be reconstructed completely, in order, under the current epoch.

This is not a static fact. It changes over time as WAL is reclaimed, payload generations are garbage-collected, or snapshots are released.

### `Recovery Reservation`

A bounded primary-side reservation proving a recovery window is recoverable and pinning all dependencies needed to finish the current catch-up or rebuild-tail replay.

A transition into recovery is valid only after the reservation is granted.

## State Set

Replica may be in one of these states:

1. `Bootstrapping`
2. `InSync`
3. `Lagging`
4. `CatchingUp`
5. `PromotionHold`
6. `NeedsRebuild`
7. `Rebuilding`
8. `CatchUpAfterRebuild`
9. `Failed`

Only `InSync` replicas count for sync durability.

## State Semantics

### 1. `Bootstrapping`

Replica has not yet earned sync eligibility and does not yet have trusted reconnect progress.

Properties:
- fresh replica identity or newly assigned replica
- may receive initial baseline/live stream
- not yet eligible for `sync_all`

Counts for:
- `sync_all`: no
- `sync_quorum`: no
- `best_effort`: background/bootstrap only

Owned anchors:
- current assignment epoch

### 2. `InSync`

Replica is eligible for sync durability.

Properties:
- receiving live ordered stream
- `replicaFlushedLSN` is near the primary head
- normal barrier protocol is valid

Counts for:
- `sync_all`: yes
- `sync_quorum`: yes
- `best_effort`: yes, but not required for ACK

Owned anchors:
- `replicaFlushedLSN`

### 3. `Lagging`

Replica has fallen out of the normal live-stream envelope but recovery path is not yet chosen.

Properties:
- primary no longer treats it as sync-eligible
- replica may still be recoverable from WAL or extent-backed recovery records
- or may require rebuild

Counts for:
- `sync_all`: no
- `sync_quorum`: no
- `best_effort`: background recovery only

Owned anchors:
- last known `replicaFlushedLSN`

### 4. `CatchingUp`

Replica is replaying from its own durable point toward a chosen target.

Properties:
- short-gap recovery mode
- primary must reserve and pin the required recovery window
- primary head continues to move

Counts for:
- `sync_all`: no
- `sync_quorum`: no
- `best_effort`: background recovery only

Owned anchors:
- `catchupStartLSN = replicaFlushedLSN`
- `catchupTargetLSN`
- `promotionBarrierLSN`
- `recoveryReservationID`
- `reservationExpiry`

### 5. `PromotionHold`

Replica has reached the chosen promotion point but must demonstrate short stability before re-entering `InSync`.

Properties:
- prevents immediate flapping back into sync eligibility
- replica has already reached `promotionBarrierLSN`
- promotion requires stable barriers or elapsed hold time

Counts for:
- `sync_all`: no
- `sync_quorum`: no
- `best_effort`: stabilization only

Owned anchors:
- `promotionBarrierLSN`
- `promotionHoldUntil` or equivalent hold criterion

### 6. `NeedsRebuild`

Replica cannot recover from retained recovery records alone.

Properties:
- catch-up window is insufficient or no longer provable
- replica must not count toward sync durability
- replica no longer pins old catch-up history

Counts for:
- `sync_all`: no
- `sync_quorum`: no
- `best_effort`: background repair candidate only

Owned anchors:
- last known `replicaFlushedLSN`

### 7. `Rebuilding`

Replica is fetching and installing a checkpoint/snapshot base image.

Properties:
- primary must preserve the chosen snapshot/base
- primary must preserve the required WAL or recovery tail after `cpLSN`

Counts for:
- `sync_all`: no
- `sync_quorum`: no
- `best_effort`: background rebuild only

Owned anchors:
- `snapshotID`
- `snapshotCpLSN`
- `tailReplayStartLSN = snapshotCpLSN + 1`
- `recoveryReservationID`
- `reservationExpiry`

### 8. `CatchUpAfterRebuild`

Replica has installed the base image and is replaying trailing history after it.

Properties:
- semantically similar to `CatchingUp`
- base point is checkpoint/snapshot, not the replica's original own state

Counts for:
- `sync_all`: no
- `sync_quorum`: no
- `best_effort`: background recovery only

Owned anchors:
- `snapshotCpLSN`
- `catchupTargetLSN`
- `promotionBarrierLSN`
- `recoveryReservationID`
- `reservationExpiry`

### 9. `Failed`

Replica recovery failed in a way that needs operator/control-plane action beyond normal retry.

Properties:
- terminal or semi-terminal fault state
- may require delete/recreate/manual intervention

Counts for:
- `sync_all`: no
- `sync_quorum`: no
- `best_effort`: no direct role

## Transition Rules

### `Bootstrapping -> InSync`

Trigger:
- initial bootstrap completes
- barrier confirms durable progress under the current epoch

Action:
- establish trusted `replicaFlushedLSN`
- grant sync eligibility for the first time

### `InSync -> Lagging`

Trigger:
- disconnect
- barrier timeout
- barrier fsync failure
- stream error

Action:
- remove sync eligibility immediately

### `Lagging -> CatchingUp`

Trigger:
- reconnect succeeds
- primary grants a recovery reservation proving `(replicaFlushedLSN, catchupTargetLSN]` is recoverable for a bounded window

Action:
- choose `catchupTargetLSN`
- pin required recovery dependencies for the reservation lifetime

### `Lagging -> NeedsRebuild`

Trigger:
- required recovery window is not recoverable
- impossible progress reported
- epoch mismatch invalidates direct catch-up
- background janitor determines the replica is outside recoverable budget

Action:
- stop treating replica as a catch-up candidate

### `CatchingUp -> PromotionHold`

Trigger:
- replica replays to `catchupTargetLSN`
- barrier confirms `promotionBarrierLSN`

Action:
- start promotion debounce window

### `PromotionHold -> InSync`

Trigger:
- promotion hold criteria satisfied
  - stable barrier successes
  - or elapsed hold time

Action:
- restore sync eligibility
- clear promotion anchors

### `PromotionHold -> Lagging`

Trigger:
- disconnect
- failed barrier
- failed live stream health check

Action:
- cancel promotion attempt
- remove sync eligibility

### `CatchingUp -> NeedsRebuild`

Trigger:
- catch-up cannot converge
- recovery reservation is lost
- catch-up timeout policy exceeded
- epoch changes

Action:
- abandon WAL-only catch-up
- move to reconstruction path

### `NeedsRebuild -> Rebuilding`

Trigger:
- control plane or primary chooses reconstruction base
- snapshot/base image transfer starts
- primary grants a rebuild reservation

Action:
- bind replica to `snapshotID` and `snapshotCpLSN`

### `Rebuilding -> CatchUpAfterRebuild`

Trigger:
- snapshot/base image installed successfully
- trailing recovery reservation is still valid

Action:
- replay trailing history after `snapshotCpLSN`

### `Rebuilding -> NeedsRebuild`

Trigger:
- rebuild copy fails
- rebuild reservation is lost
- rebuild WAL-tail budget is exceeded
- epoch changes

Action:
- abort current rebuild session
- remain excluded from sync durability

### `CatchUpAfterRebuild -> PromotionHold`

Trigger:
- trailing replay reaches target
- barrier confirms durable replay through `promotionBarrierLSN`

Action:
- start promotion debounce

### `CatchUpAfterRebuild -> NeedsRebuild`

Trigger:
- reservation is lost
- replay cannot converge
- epoch changes

Action:
- abandon current attempt
- require a fresh rebuild plan

### Any state -> `Failed`

Trigger examples:
- unrecoverable protocol inconsistency
- repeated rebuild failure beyond retry policy
- snapshot corruption
- local replica storage failure

## Retention Obligations By State

The key V2 rule is:

- recoverability is not a static fact
- it is a bounded promise the primary must honor once it admits a replica into recovery

### `InSync`

Primary must retain:
- recent WAL under normal retention policy

Primary does not need:
- snapshot pin purely for this replica

### `Lagging`

Primary must retain:
- enough recent information to evaluate recoverability or intentionally declare `NeedsRebuild`

This state should be short-lived.

### `CatchingUp`

Primary must retain for the reservation lifetime:
- recovery metadata for `(catchupStartLSN, promotionBarrierLSN]`
- every payload referenced by that recovery window
- current epoch lineage for the session

### `PromotionHold`

Primary must retain:
- whatever live-stream and barrier state is required to validate promotion

This state should be brief and must not pin long-lived history.

### `NeedsRebuild`

Primary retains:
- no special old recovery window for this replica

This state explicitly releases the old catch-up hold.

### `Rebuilding`

Primary must retain for the reservation lifetime:
- chosen `snapshotID`
- any base-image dependencies
- trailing history after `snapshotCpLSN`

### `CatchUpAfterRebuild`

Primary must retain for the reservation lifetime:
- recovery metadata for `(snapshotCpLSN, promotionBarrierLSN]`
- every payload referenced by that trailing window

## Moving-Head Rules

The primary head continues advancing during:
- `CatchingUp`
- `Rebuilding`
- `CatchUpAfterRebuild`

Therefore transitions must never use current head at finish time as an implicit target.

Instead, each transition must select explicit targets.

### Catch-up target

When catch-up starts, choose:
- `catchupTargetLSN = H0`

Replica first chases to `H0`, not to an infinite moving head.

Then:
- either enter `PromotionHold` and promote
- or begin another bounded cycle
- or abort to rebuild

### Rebuild target

When rebuild starts, choose:
- `snapshotCpLSN = C`
- trailing replay target `H0`

Replica installs the snapshot at `C`, then replays `(C, H0]`, then enters `PromotionHold`.

## Tail-Chasing Rule

Replica may fail to converge if:
- catch-up speed < primary ingest speed

V2 must define bounded behavior:

1. bounded catch-up window
2. bounded catch-up time
3. policy after failure to converge:
   - for `sync_all`: bounded retry, then fail requests
   - for `best_effort`: keep serving and continue background recovery or escalate to rebuild

No silent downgrade of `sync_all` is allowed.

## Recovery Feasibility

The primary must not admit a replica into catch-up based on a best-effort guess.

It must prove the requested recovery window is recoverable and then reserve it.

Recommended abstraction:

- `CheckRecoveryFeasibility(startLSN, endLSN) -> fully recoverable | needs rebuild`
- `ReserveRecoveryWindow(startLSN, endLSN) -> reservation`

Only a successful reservation may drive:
- `Lagging -> CatchingUp`
- `NeedsRebuild -> Rebuilding`
- `Rebuilding -> CatchUpAfterRebuild`

## Recovery Classes

V2 must support more than one local record type without leaking that detail into replica state.

### `WALInline`

Properties:
- payload lives directly in WAL
- recoverable while WAL is retained

### `ExtentReferenced`

Properties:
- recovery metadata points at payload outside WAL
- payload must be resolved from extent/snapshot generation state

The FSM does not care how payload is stored.

It only cares whether the requested window is fully recoverable for the lifetime of the reservation.

The engine-level rule is:

- every record in `(startLSN, endLSN]` must be payload-resolvable
- the resolved version must correspond to that record's historical state
- the payload must stay pinned until the reservation ends

If any required payload is not resolvable:
- the window is not recoverable
- the replica must go to `NeedsRebuild`

## Snapshot Rule

Rebuild must use a real checkpoint/snapshot base image.

Valid:
- immutable snapshot at `cpLSN`
- copy-on-write checkpoint image
- frozen base image with exact `cpLSN`

Invalid:
- current extent treated as historical `cpLSN`

## Epoch / Fencing Rule

Every transition is epoch-bound.

If epoch changes during:
- `Bootstrapping`
- `Lagging`
- `CatchingUp`
- `PromotionHold`
- `Rebuilding`
- `CatchUpAfterRebuild`

Then:
- abort current transition
- discard old sender assumptions
- restart negotiation under the new epoch

This prevents stale-primary recovery traffic from being accepted.

## Multi-Replica Volume Rules

Different replicas may be in different states simultaneously.

Example:
- replica A = `InSync`
- replica B = `CatchingUp`
- replica C = `Rebuilding`

Volume-level durability policy is computed per mode.

### `sync_all`
- all required replicas must be `InSync`

### `sync_quorum`
- enough replicas must be `InSync`

### `best_effort`
- primary local durability only
- replicas recover in background

## Illegal or Suspicious Conditions

These should force rejection or abort:

1. replica reports `replicaFlushedLSN > headLSN`
2. replica progress belongs to wrong epoch
3. requested recovery window is not recoverable
4. recovery reservation cannot be granted
5. snapshot base does not match claimed `cpLSN`
6. replay stream shows impossible gap/ordering after reconstruction

## Design Guidance

V2 should be implemented so that:

1. state owns recovery semantics
2. anchors make transitions explicit
3. retention obligations are derived from state
4. catch-up admission requires reservation, not guesswork
5. mode semantics are derived from `InSync` eligibility

This is better than burying recovery behavior across many ad hoc code paths.

## Bottom Line

V2 is fundamentally a state machine problem.

The correct abstraction is not:
- some edge cases around WAL replay

It is:
- replicas move through explicit states while the primary head continues advancing and recovery windows must be provable and reserved

So V2 must be designed around:
- state definitions
- anchor LSNs
- transition rules
- retention obligations
- recoverability checks
- recovery reservations
- abort conditions