seaweedfs/sw-block/design/v2-protocol-truths.md

# V2 Protocol Truths

Date: 2026-03-30
Status: active
Purpose: record the compact, stable truths that later phases must preserve, and provide a conformance reference for implementation reviews

## Why This Document Exists

`FSM`, `simulator`, `prototype`, and `engine` are not a code-production pipeline.
They are an evidence ladder.

So the most important output to carry forward is not only code, but:

1. accepted semantics
2. must-hold boundaries
3. failure classes that must stay closed
4. explicit places where later phases may improve or drift

This document is the compact truth table for the V2 line.

## How To Use It

For each later phase or slice, ask:

1. does the new implementation remain aligned with these truths?
2. if not, is the deviation constructive or risky?
3. which truth is newly strengthened by this phase?

Deviation labels:

- `Aligned`: implementation preserves the truth
- `Constructive deviation`: implementation changes shape but strengthens the truth
- `Risky deviation`: implementation weakens or blurs the truth

## Core Truths

### T1. `CommittedLSN` is the external truth boundary

Short form:

- external promises are anchored at `CommittedLSN`, not `HeadLSN`

Meaning:

- recovery targets
- promotion safety
- flush/visibility reasoning

must all be phrased against `CommittedLSN`.

Prevents:

- using optimistic WAL head as committed truth
- acknowledging lineage that failover cannot preserve

Evidence anchor:

- strong in design
- strong in simulator
- strong in prototype
- strong in engine

### T2. `ZeroGap <=> ReplicaFlushedLSN == CommittedLSN`

Short form:

- zero-gap requires exact equality with committed truth

Meaning:

- replica ahead is not zero-gap
- replica behind is not zero-gap

Prevents:

- unsafe fast-path completion
- replica-ahead being mistaken for in-sync

Evidence anchor:

- strong in prototype
- strong in engine

### T3. `CatchUp` is bounded replay on a still-trusted base

Short form:

- `CatchUp = KeepUp with bounded debt`

Meaning:

- catch-up is a short-gap, low-cost, bounded replay path
- it only makes sense while the replica base is still trustworthy enough to continue from

Prevents:

- turning catch-up into indefinite moving-head chase
- hiding broad recovery complexity in replay logic

Evidence anchor:

- strong in design
- strong in simulator
- strong in prototype
- strong in engine

### T4. `NeedsRebuild` is explicit when replay is not the right answer

Short form:

- `NeedsRebuild <=> replay is unrecoverable, unstable, or no longer worth bounded replay`

Meaning:

- long-gap
- lost recoverability
- no trusted base
- budget violation

must escalate explicitly.

Prevents:

- pretending catch-up will eventually succeed
- carrying V1/V1.5-style unbounded degraded chase forward

Evidence anchor:

- strong in simulator
- strong in prototype
- strong in engine

### T5. `Rebuild` is the formal primary recovery path

Short form:

- `Rebuild = frozen TargetLSN + trusted base + optional tail + barrier`

Meaning:

- rebuild is not a shameful fallback
- it is the general recovery framework

Prevents:

- overloading catch-up with broad recovery semantics
- treating full/partial rebuild as unrelated protocols

Evidence anchor:

- strong in design
- strong in prototype
- strong in engine

### T6. Full and partial rebuild share one correctness contract

Short form:

- `full rebuild` and `partial rebuild` differ in transfer choice, not in truth model

Meaning:

- both require frozen `TargetLSN`
- both require trusted pinned base
- both require explicit durable completion

Prevents:

- optimization layers redefining protocol truth
- bitmap/range paths bypassing trusted-base rules

Evidence anchor:

- strong in design
- partial in engine
- stronger real-system proof still deferred

### T7. No recovery result may outlive its authority

Short form:

- `ValidMutation <=> sender exists && sessionID matches && epoch current && endpoint current`

Meaning:

- stale session
- stale epoch
- stale endpoint
- stale sender

must all fail closed.

Prevents:

- late results mutating current lineage
- changed-address stale completion bugs

Evidence anchor:

- strong in simulator
- strong in prototype
- strong in engine

### T8. `ReplicaID` is stable identity; `Endpoint` is mutable location

Short form:

- `ReplicaID != address`

Meaning:

- address changes may invalidate sessions
- address changes must not destroy sender identity

Prevents:

- reintroducing address-shaped identity
- changed-address restarting as logical removal + add

Evidence anchor:

- strong in prototype
- strong in engine
- strong in bridge P0

### T9. Truncation is a protocol boundary, not cleanup

Short form:

- replica-ahead cannot complete until divergent tail is explicitly truncated

Meaning:

- truncation is part of recovery contract
- not a side-effect or best-effort cleanup

Prevents:

- completing recovery while replica still contains newer divergent writes

Evidence anchor:

- strong in design
- strong in engine

### T10. Recoverability must be proven from real retained history

Short form:

- `CatchUp allowed <=> required replay range is recoverable from retained history`

Meaning:

- the engine should consume storage truth
- not test-reconstructed optimism

Prevents:

- replay on missing WAL
- fake recoverability based only on watermarks

Evidence anchor:

- strong in simulator
- strong in engine
- strengthened in driver/adapter phases

### T11. Trusted-base choice must be explicit and causal

Short form:

- `snapshot_tail` requires both trusted checkpoint and replayable tail

Meaning:

- snapshot existence alone is insufficient
- fallback to full-base must be explainable

Prevents:

- over-trusting old checkpoints
- silently choosing an invalid rebuild source

Evidence anchor:

- strong in simulator
- strong in engine
- strengthened by Phase 06

### T12. Current extent cannot fake old history

Short form:

- historical correctness requires reconstructable history, not current-state approximation

Meaning:

- live extent state is not sufficient proof of an old target point
- historical reconstruction must be justified by checkpoint + retained history

Prevents:

- using current extent as fake proof of older state

Evidence anchor:

- strongest in simulator
- engine currently proves prerequisites, not full reconstruction proof

### T13. Promotion requires recoverable committed prefix

Short form:

- promoted replica must be able to recover committed truth, not merely advertise a high watermark

Meaning:

- candidate selection is about recoverable lineage, not optimistic flush visibility

Prevents:

- promoting a replica that cannot reconstruct committed prefix after crash/restart

Evidence anchor:

- strong in simulator
- partially carried into engine semantics
- real-system validation still needed

### T14. `blockvol` executes I/O; engine owns recovery policy

Short form:

- adapters may translate engine decisions into concrete work
- they must not silently re-decide recovery classification or source choice

Meaning:

- master remains control authority
- engine remains recovery authority
- storage remains truth source

Prevents:

- V1/V1.5 policy leakage back into service glue

Evidence anchor:

- strong in Phase 07 service-slice planning
- initial bridge P0 aligns
- real-system proof still pending

### T15. Reuse reality, not inherited semantics

Short form:

- V2 may reuse existing Seaweed control/runtime/storage paths
- it must not inherit old semantics as protocol truth

Meaning:

- reuse existing heartbeat, assignment, `blockvol`, receiver, shipper, retention, and runtime machinery when useful
- keep `ReplicaID`, epoch authority, recovery classification, committed truth, and rebuild boundaries anchored in accepted V2 semantics

Prevents:

- V1/V1.5 structure silently redefining V2 behavior
- convenience reuse turning old runtime assumptions into new protocol truth

Evidence anchor:

- strong in Phase 07/08 direction
- should remain active in later implementation phases

### T16. Full-base rebuild completes against an explicit achieved boundary

Short form:

- `full_base` rebuild requires `AchievedLSN >= TargetLSN`
- engine and local runtime must converge to the same achieved boundary

Meaning:

- the engine plans a frozen minimum target `TargetLSN`
- the backend may produce an actual rebuilt boundary `AchievedLSN`
- exact-target extent equality is not required on a mutable-extent backend
- after install, `checkpoint`, `nextLSN`, receiver progress, flusher checkpoint, and engine-visible rebuild completion must all align to the same `AchievedLSN`

Prevents:

- local runtime truth advancing beyond engine/accounting truth
- rebuild completion at one boundary while storage/runtime state reflects another
- treating "at least target" as safe without making the newer achieved boundary explicit

Evidence anchor:

- strengthened by `Phase 09` backend execution closure work
- phase-level decision exists
- real-system proof still depends on executor/runtime alignment

### T17. Extent/WAL recovery split must be fixed before replay begins

Short form:

- `extent copy + WAL replay` is only correct if the split boundary is explicit and gap-free

Meaning:

- extent data must represent a known recovery boundary
- WAL replay must start from the matching next boundary
- if unflushed writes could otherwise fall between extent and replay, the backend must first fix the split boundary
- snapshot/CoW/bitmap export paths follow the same rule: transport must read from a stable recovery view, not a mutating mixed state

Prevents:

- missing writes between extent copy and replay
- point-in-time export that mixes two different recovery views
- rebuild correctness depending on timing accidents instead of an explicit boundary contract

Evidence anchor:

- present in rebuild correctness reasoning from V1
- strengthened by `Phase 09` full-base execution work
- future snapshot/bitmap-based paths must preserve it explicitly

## Current Strongest Evidence By Layer

| Layer | Main value |
|------|------------|
| `FSM` / design | define truth and non-goals |
| simulator | prove protocol truth and failure-class closure cheaply |
| prototype | prove implementation-shape and authority semantics cheaply |
| engine | prove the accepted contracts survive real implementation structure |
| service slice / runner | prove truth survives real control/storage/system reality |

## Phase Conformance Notes

### Phase 04

- `Aligned`: T7, T8
- strengthened sender/session ownership and stale rejection

### Phase 4.5

- `Aligned`: T3, T4, T5, T10, T12
- major tightening:
  - bounded catch-up
  - first-class rebuild
  - crash-consistency and recoverability proof style

### Phase 05

- `Aligned`: T1, T2, T3, T4, T5, T7, T8, T9, T10, T11
- engine core slices closed:
  - ownership
  - execution
  - recoverability gating
  - orchestrated entry path

### Phase 06

- `Aligned`: T10, T11, T14
- `Constructive deviation`: planner/executor split replaced convenience wrappers without changing protocol truth
- strengthened:
  - real storage/resource contracts
  - explicit release symmetry
  - failure-class validation against engine path

### Phase 07 P0

- `Aligned`: T8, T10, T14
- bridge now makes stable `ReplicaID` explicit at service boundary
- bridge states the hard rule that engine decides policy and `blockvol` executes I/O
- real `weed/storage/blockvol/` integration still pending

## Current Carry-Forward Truths For Later Phases

Later phases must not regress these:

1. `CommittedLSN` remains the external truth boundary
2. `CatchUp` stays narrow and bounded
3. `Rebuild` remains the formal primary recovery path
4. stale authority must fail closed
5. stable identity must remain separate from mutable endpoint
6. trusted-base choice must remain explicit and causal
7. service glue must not silently re-decide recovery policy
8. reuse reality, but do not inherit old semantics as V2 truth
9. full-base rebuild must converge to one explicit achieved boundary
10. extent/WAL recovery split must be fixed before replay

## Review Rule

Every later phase or slice should explicitly answer:

1. which truths are exercised?
2. which truths are strengthened?
3. does this phase introduce any constructive or risky deviation?
4. which evidence layer now carries the truth most strongly?

## Phase Alignment Rule

From `Phase 05` onward, every phase or slice should align explicitly against this document.

Minimum phase-alignment questions:

1. which truths are in scope?
2. which truths are strengthened?
3. which truths are merely carried forward?
4. does the phase introduce any constructive deviation?
5. does the phase introduce any risky deviation?
6. which evidence layer currently carries each in-scope truth most strongly?

Expected output shape for each later phase:

- `In-scope truths`
- `Strengthened truths`
- `Carry-forward truths`
- `Constructive deviations`
- `Risky deviations`
- `Evidence shift`

## Phase 05-07 Alignment

### Phase 05

Primary alignment focus:

- T1 `CommittedLSN` as external truth boundary
- T2 zero-gap exactness
- T3 bounded `CatchUp`
- T4 explicit `NeedsRebuild`
- T5/T6 rebuild correctness contract
- T7 stale authority must fail closed
- T8 stable `ReplicaID`
- T9 truncation as protocol boundary
- T10/T11 recoverability and trusted-base gating

Main strengthening:

- engine core adopted accepted protocol truths as real implementation structure

Main review risk:

- engine structure accidentally collapsing back to address identity or unfenced execution

### Phase 06

Primary alignment focus:

- T10 recoverability from real retained history
- T11 trusted-base choice remains explicit and causal
- T14 engine owns policy, adapters carry truth and execution contracts

Main strengthening:

- planner/executor/resource contracts
- fail-closed cleanup symmetry
- cross-layer proof path through engine execution

Main review risk:

- executor or adapters recomputing policy from convenience inputs
- storage/resource contracts becoming approximate instead of real

### Phase 07+

Primary alignment focus:

- T8 stable identity at the real service boundary
- T10 real storage truth into engine decisions
- T11 trusted-base proof remains explicit through service glue
- T14 `blockvol` executes I/O but does not own recovery policy
- T16 full-base rebuild converges to one explicit achieved boundary
- T17 extent/WAL split boundary remains explicit and gap-free

Main strengthening:

- real-system service-slice conformance
- real control-plane and storage-plane integration
- diagnosable failure replay through the integrated path

Main review risk:

- V1/V1.5 semantics leaking back in through service glue
- address-shaped identity reappearing at the boundary
- blockvol-side code silently re-deciding recovery policy

## Future Feature Rule

When a later feature expands the protocol surface (for example `SmartWAL` or a new rebuild optimization), the order should be:

1. `FSM / design`
- define the new semantics and non-goals

2. `Truth update`
- either attach the feature to an existing truth
- or add a new protocol truth if the feature creates a new long-lived invariant

3. `Phase alignment`
- define which later phases strengthen or validate that truth

4. `Evidence ladder`
- simulator, prototype, engine, service slice as needed

Do not start feature implementation by editing engine or service glue first and only later trying to explain what truth changed.

## Feature Review Rule

For any future feature, later reviews should ask:

1. did the feature create a new truth or just strengthen an existing one?
2. which phase first validates it?
3. which evidence layer proves it most strongly today?
4. does the feature weaken any existing truth?

This keeps feature growth aligned with protocol truth instead of letting implementation convenience define semantics.