seaweedfs/sw-block/design/wal-replication-v2-orchestrator.md

WAL Replication V2 Orchestrator

Date: 2026-03-26 Status: design proposal Purpose: define the volume-level orchestration model that sits above the per-replica WAL V2 FSM

Why This Document Exists

ReplicaFSM alone is not enough.

It can describe one replica relative to the current primary, but it cannot by itself model:

• primary head continuing to advance
• multiple replicas in different states
• durability mode semantics
• primary lease loss and epoch change
• primary failover and replica promotion
• fencing of old recovery sessions

So WAL V2 needs a second layer:

• per-replica ReplicaFSM
• volume-level Orchestrator

Scope

This document defines the volume-level logic only.

It does not define:

• exact network protocol
• exact master RPCs
• exact storage backend internals

It assumes the per-replica state machine from:

• wal-replication-v2-state-machine.md

Core Model

The orchestrator owns:

1. current primary lineage

• epoch
• lease/authority state

2. volume durability mode

• best_effort
• sync_all
• sync_quorum

3. moving primary progress

• headLSN
• checkpoint/snapshot anchors

4. replica set

• one ReplicaFSM per replica
• per-replica role in the current volume topology

5. volume-level admission decision

• can writes proceed?
• can sync requests complete?
• must promotion/failover occur?

Two FSM Layers

Layer A: ReplicaFSM

Owns per-replica state such as:

• Bootstrapping
• InSync
• Lagging
• CatchingUp
• PromotionHold
• NeedsRebuild
• Rebuilding
• CatchUpAfterRebuild
• Failed

Layer B: VolumeOrchestrator

Owns system-wide state such as:

• current epoch
• current primary identity
• durability mode
• set of required replicas
• current headLSN
• whether writes or promotions are allowed

The orchestrator does not replace ReplicaFSM. It drives it.

Volume State

The orchestrator should track at least:

type VolumeMode string

type PrimaryState string

const (
    PrimaryServing  PrimaryState = "Serving"
    PrimaryDraining PrimaryState = "Draining"
    PrimaryLost     PrimaryState = "Lost"
)

type VolumeModel struct {
    Epoch        uint64
    PrimaryID    string
    PrimaryState PrimaryState
    Mode         VolumeMode

    HeadLSN       uint64
    CheckpointLSN uint64

    RequiredReplicaIDs []string
    Replicas           map[string]*ReplicaFSM
}

This is a model shape, not a required production struct.

Orchestrator Responsibilities

1. Advance primary head

When primary commits a new write:

• increment headLSN
• enqueue/send to replica sender loops
• evaluate whether the current mode still allows ACK

2. Evaluate sync eligibility

The orchestrator computes volume-level durability from replica states.

Derived rule:

• only ReplicaFSM.IsSyncEligible() counts

3. Drive recovery entry

When a replica disconnects or falls behind:

• feed disconnect/lag events into that replica FSM
• decide whether to try catch-up or rebuild
• acquire recovery reservation if required

4. Handle primary authority changes

When lease is lost or a new primary is chosen:

• increment epoch
• abort stale recovery sessions
• reevaluate all replica relationships from the new primary's perspective

5. Drive promotion / failover

When current primary is lost:

• choose promotion candidate
• assign new epoch
• move old primary to stale/lost
• convert the promoted replica into the new serving primary
• reclassify remaining replicas relative to the new primary

Required Volume-Level Events

The orchestrator should be able to simulate at least these events.

Write/progress events

• WriteCommitted(lsn)
• CheckpointAdvanced(lsn)
• BarrierCompleted(replicaID, flushedLSN)

Replica health events

• ReplicaDisconnected(replicaID)
• ReplicaReconnect(replicaID, flushedLSN)
• ReplicaReservationLost(replicaID)
• ReplicaCatchupTimeout(replicaID)
• ReplicaRebuildTooSlow(replicaID)

Topology/control events

• PrimaryLeaseLost()
• EpochChanged(newEpoch)
• PromoteReplica(replicaID)
• ReplicaAssigned(replicaID)
• ReplicaRemoved(replicaID)

Mode Semantics

best_effort

Rules:

• ACK after primary local durability
• replicas may be Lagging, CatchingUp, NeedsRebuild, or Rebuilding
• background recovery continues

Volume implication:

• primary can keep serving while replicas recover

sync_all

Rules:

• ACK only when all required replicas are InSync and durable through target LSN
• bounded retry only
• no silent downgrade

Volume implication:

• one lagging required replica can block sync completion
• orchestrator may fail requests, not silently reinterpret policy

sync_quorum

Rules:

• ACK when quorum of required nodes are durable through target LSN
• lagging replicas may recover in background as long as quorum remains

Volume implication:

• orchestrator must count eligible replicas, not just healthy sockets

Primary-Head Simulation Rules

The orchestrator must explicitly model that the primary keeps moving.

Rule 1: head moves independently of replica recovery

A replica entering CatchingUp does not freeze headLSN.

Rule 2: each recovery attempt uses explicit targets

For a replica in recovery, orchestrator chooses:

• catchupTargetLSN = H0
• or snapshotCpLSN = C and replay target H0

Rule 3: promotion is explicit

A replica is not restored to InSync just because it reaches H0.

It must still pass:

• barrier confirmation
• PromotionHold

Failover / Promotion Model

The orchestrator must be able to simulate:

1. old primary loses lease
2. old primary is fenced by epoch change
3. one replica is promoted
4. promoted replica becomes new primary under a higher epoch
5. all old recovery sessions from the old primary are invalidated
6. remaining replicas are reevaluated relative to the new primary's head and retained history

Important consequence:

• failover is not a ReplicaFSM transition only
• it is a volume-level re-rooting of all replica relationships

Suggested Promotion Rules

Promotion candidate should prefer:

1. highest valid durable progress
2. current epoch-consistent history
3. healthiest replica among tied candidates

After promotion:

• PrimaryID changes
• Epoch increments
• all replica reservations from the previous primary are void
• all non-primary replicas must renegotiate recovery against the new primary

Multi-Replica Examples

Example 1: sync_all

• replica A = InSync
• replica B = Lagging
• replica C = InSync

If A and B are required replicas in RF=3 sync_all:

• writes needing sync durability fail or wait
• even though one replica is still healthy

Example 2: sync_quorum

• replica A = InSync
• replica B = CatchingUp
• replica C = InSync

If quorum is 2:

• volume can continue serving sync requests
• B recovers in background

Example 3: failover

• old primary lost
• replica A promoted
• replica B was previously CatchingUp under old epoch

After promotion:

• B's old session is aborted
• B re-enters evaluation against A's history

What The Tiny Prototype Should Simulate

The V2 prototype should be able to drive at least these scenarios:

1. steady state keep-up

• primary head advances
• all required replicas remain InSync

2. short outage

• one replica disconnects
• primary keeps writing
• reconnect succeeds within recoverable window
• replica returns via PromotionHold

3. long outage

• one replica disconnects too long
• recoverability expires
• replica goes NeedsRebuild
• rebuild and trailing replay complete

4. tail chasing

• replica catch-up speed is below primary ingest speed
• orchestrator chooses fail, throttle, or rebuild path depending on mode

5. failover

• primary lease lost
• new epoch assigned
• replica promoted
• old recovery sessions fenced

6. mixed-state quorum

• different replicas in different states
• orchestrator computes correct sync_all / sync_quorum result

Relationship To WAL V1

WAL V1 already contains pieces of this logic, but they are scattered across:

• shipper state
• barrier code
• retention code
• assignment/promotion code
• rebuild code
• heartbeat/master logic

V2 should separate these into:

• per-replica recovery FSM
• volume-level orchestrator

Bottom Line

The next step after ReplicaFSM is not Smart WAL.

The next step is the volume-level orchestrator model.

Why:

• primary keeps moving
• durability mode is volume-scoped
• failover/promotion is volume-scoped
• replica recovery must be evaluated in the context of the whole volume

So V2 needs:

• ReplicaFSM for one replica
• VolumeOrchestrator for the moving multi-replica system