seaweedfs/sw-block/design/wal-replication-v2.md

WAL Replication V2

Date: 2026-03-26 Status: design proposal Purpose: redesign WAL-based block replication around explicit short-gap catch-up and long-gap reconstruction

Goal

Provide a replication architecture that:

• keeps the primary write path fast
• supports correct synchronous durability semantics
• supports short-gap reconnect catch-up using WAL
• avoids paying unbounded WAL retention tax for long-lag replicas
• uses reconstruction from a real checkpoint/snapshot base for larger lag

This design replaces a "WAL does everything" mindset with a 3-tier recovery model.

Core Principle

WAL is excellent for:

• recent ordered delta
• local crash recovery
• short-gap replica catch-up

WAL is not the right long-range recovery mechanism for lagging block replicas.

Long-gap recovery should use:

• a real checkpoint/snapshot base image
• plus WAL tail replay after that base point

Correctness Boundary

Never reconstruct old state from current extent alone.

Example:

1. LSN 100: block A = foo
2. LSN 120: block A = bar

If a replica needs state at LSN 100, current extent contains bar, not foo.

Therefore:

• current extent is latest state
• not historical state

So long-gap recovery must use a base image that is known to represent a real checkpoint/snapshot cpLSN.

3-Tier Replication Model

Tier A: Keep-up

Replica is close enough to the primary that normal ordered streaming keeps it current.

Properties:

• normal steady-state mode
• no special recovery path
• replica stays InSync

Tier B: Lagging Catch-up

Replica fell behind, but the primary still has enough recoverable history covering the missing range.

Properties:

• reconnect handshake determines the replica durable point
• primary proves and reserves a bounded recovery window
• primary replays missing history
• replica returns to InSync only after replay, barrier confirmation, and promotion hold

Tier C: Reconstruction

Replica is too far behind for direct replay.

Properties:

• replica must rebuild from a real checkpoint/snapshot base
• after base image install, primary replays trailing history after cpLSN
• replica only re-enters InSync after durable catch-up completes

Architecture

Primary Artifacts

The primary owns three forms of state:

1. Active WAL

• recent ordered metadata/delta stream
• bounded by retention policy

2. Checkpoint Snapshot

• immutable point-in-time base image at cpLSN
• used for long-gap reconstruction

3. Current Extent

• latest live block state
• not a substitute for historical checkpoint state

Replica Artifacts

Replica maintains:

1. local WAL or equivalent recovery log
2. replica receivedLSN
3. replica flushedLSN
4. local extent state

Sender Model

Do not ship recovery data inline from foreground write goroutines.

Per replica, use:

• one ordered send queue
• one sender loop

The sender loop owns:

• live stream shipping
• reconnect handling
• short-gap catch-up
• reconstruction tail replay

This guarantees:

• strict LSN order per replica
• clean transport state ownership
• no inline shipping races in the primary write path

Write Path

Primary write path:

1. allocate monotonic LSN
2. append recovery metadata to local WAL or journal
3. enqueue the record to each replica sender queue
4. return according to durability mode semantics

Flusher later:

• flushes dirty data to extent
• manages checkpoints
• manages bounded retention of WAL and other recovery dependencies

Recovery Classes

V2 supports more than one local record type.

WALInline

Properties:

• payload lives directly in WAL
• recoverable while WAL is retained

ExtentReferenced

Properties:

• journal entry contains metadata only
• payload is resolved from extent/snapshot generation state
• direct-extent writes and future smart-WAL paths fall into this class

Replica state does not encode these classes.

Instead, the primary must answer a stricter question for reconnect:

• is (startLSN, endLSN] fully recoverable under the current epoch, and can it be reserved for the duration of recovery?

Replica Progress Model

Each replica reports progress explicitly.

receivedLSN

• highest LSN received and appended locally
• not yet a durability guarantee

flushedLSN

• highest LSN durably persisted on the replica
• authoritative sync durability signal

Only flushedLSN counts for:

• sync_all
• sync_quorum

Replica States

Replica state is defined by wal-replication-v2-state-machine.md.

Important highlights:

• Bootstrapping
• InSync
• Lagging
• CatchingUp
• PromotionHold
• NeedsRebuild
• Rebuilding
• CatchUpAfterRebuild
• Failed

Only InSync replicas count toward sync durability.

Protocol

1. Normal Streaming

Primary sender loop:

• sends ordered replicated write records

Replica:

1. validates ordering
2. appends locally
3. advances receivedLSN

2. Barrier / Sync

Primary sends:

• BarrierReq{LSN, Epoch}

Replica:

1. wait until receivedLSN >= LSN
2. flush durable local state
3. set flushedLSN = LSN
4. reply BarrierResp{Status, FlushedLSN}

Primary uses this to evaluate mode policy.

3. Reconnect Handshake

On reconnect, primary obtains:

• current epoch
• primary head
• replica durable flushedLSN

Then primary evaluates recovery feasibility.

Possible outcomes:

1. replica already caught up

• state -> PromotionHold or InSync depending on policy

2. bounded catch-up possible

• reserve recovery window
• state -> CatchingUp

3. direct replay not possible

• state -> NeedsRebuild

Recovery Feasibility and Reservation

The key V2 rule is:

• fully recoverable is not enough
• the primary must also reserve the recovery window

Recommended engine-side flow:

1. CheckRecoveryFeasibility(startLSN, endLSN)
2. if feasible, ReserveRecoveryWindow(startLSN, endLSN)
3. only then start CatchingUp or CatchUpAfterRebuild

A recovery reservation pins:

• recovery metadata
• referenced payload generations
• required snapshots/base images
• current epoch lineage for the session

If the reservation is lost during recovery:

• abort the current attempt
• fall back to NeedsRebuild

Tier B: Lagging Catch-up Algorithm

When a replica is behind but within a recoverable retained window:

1. choose a bounded target H0
2. reserve (ReplicaFlushedLSN, H0]
3. replay the missing range
4. barrier confirms durable flushedLSN >= H0
5. enter PromotionHold
6. only then restore InSync

Tail-chasing problem

If the primary is writing faster than the replica can catch up, the replica may never converge.

To handle this:

1. define a bounded catch-up window
2. if catch-up rate is slower than ingest rate for too long:
   • either temporarily throttle primary admission for strict sync_all
   • or fail sync_all requests and let control-plane policy react
   • or abort to rebuild
3. do not let a replica remain in unbounded perpetual CatchingUp

Important rule

For sync_all, the data path must not silently downgrade to best_effort.

Correct behavior:

• bounded retry
• then fail

Any mode change must be explicit policy, not silent transport behavior.

Tier C: Reconstruction Algorithm

When a replica is too far behind for direct replay:

1. mark replica NeedsRebuild
2. choose a real checkpoint/snapshot base at cpLSN
3. create a rebuild reservation
4. replica enters Rebuilding
5. replica pulls immutable checkpoint/snapshot image
6. replica installs that base image and sets base progress to cpLSN
7. primary replays trailing history (cpLSN, H0]
8. barrier confirms durable replay
9. replica enters PromotionHold
10. replica returns to InSync

Why snapshot/base image must be real

If the replica needs state at cpLSN, the base image must represent exactly that checkpoint.

Invalid:

• current extent copied at some later time and treated as historical cpLSN

Valid:

• immutable snapshot
• copy-on-write checkpoint image
• frozen base image

Retention and Budget

V2 retention is bounded.

WAL / recovery metadata retention

Primary keeps only a bounded recent recovery window:

• max_retained_wal_bytes
• optionally max_retained_wal_time

Recovery reservation budget

Reservations are also bounded:

• timeout
• bytes pinned
• snapshot dependency lifetime

If a catch-up or rebuild session exceeds its reservation budget:

• primary aborts the session
• replica falls back to NeedsRebuild
• a newer rebuild plan may be chosen later

Sync Modes

best_effort

• ACK after primary local durability
• replicas may lag
• background catch-up or rebuild allowed

sync_all

• ACK only when all required replicas are InSync and durably at target LSN
• bounded retry only
• no silent downgrade

sync_quorum

• ACK when enough replicas are InSync and durably at target LSN

Why This Direction

V2 separates three different concerns cleanly:

1. fast steady-state replication
2. short-gap replay
3. long-gap reconstruction

This avoids forcing WAL alone to solve all recovery cases.

Implementation Order

Recommended order:

1. pure FSM
2. ordered sender loop
3. bounded direct replay
4. checkpoint/snapshot reconstruction
5. smarter local write path and recovery classes
6. policy and control-plane integration

Phase 13 current direction

Current Phase 13 / WAL V1 is still:

• fixing correctness of WAL-centered sync replication
• still focused mainly on bounded WAL replay and rebuild fallback

That is the right bridge.

V2 should follow after WAL V1 closes.

Bottom Line

V2 is not "more WAL features."

It is:

• explicit recovery feasibility
• explicit recovery reservations
• ordered sender loops
• short-gap replay for recent lag
• checkpoint/snapshot reconstruction for long lag
• promotion back to InSync only after durable proof