Files
seaweedfs/sw-block/design/v2_scenarios.md
T
pingqiuandClaude Opus 4.6 edec7098e8 feat: add V2 protocol simulator and enginev2 sender/session prototype
Adds sw-block/ directory with:

- distsim: protocol correctness simulator (96 tests)
  - cluster model with epoch fencing, barrier semantics, commit modes
  - endpoint identity, control-plane flow, candidate eligibility
  - timeout events, timer races, same-tick ordering
  - session ownership tracking with ID-based stale fencing

- enginev2: standalone V2 sender/session implementation (63 tests)
  - per-replica Sender with identity-preserving reconciliation
  - RecoverySession with FSM phase transitions and session ID
  - execution APIs: BeginConnect, RecordHandshake, BeginCatchUp,
    RecordCatchUpProgress, CompleteSessionByID — all sender-authority-gated
  - recovery outcome branching: zero-gap, catch-up, needs-rebuild
  - assignment-intent orchestration with epoch fencing

- design docs: acceptance criteria, open questions, first-slice spec,
  protocol development process

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 10:38:27 -07:00

639 lines
20 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# WAL V2 Scenarios
Date: 2026-03-26
Status: working scenario backlog
Purpose: define the scenario set that proves why WAL V2 exists, what it must do better than WAL V1, and what it should handle better than rebuild-heavy systems
Execution note:
- active implementation planning for these scenarios lives under `../.private/phase/`
- `design/` is the design/source-of-truth view
- `.private/phase/` is the execution/checklist view for `sw`
## Why This File Exists
V2 should not grow by adding random simulations.
Each new scenario should prove one of these claims:
1. committed data is never lost
2. uncommitted data is never falsely revived
3. epoch and promotion lineage are safe
4. short-gap recovery is cheaper and cleaner than rebuild
5. catch-up vs rebuild boundary is explicit and correct
6. historical data correctness is preserved
## Scenario Sources
The backlog draws scenarios from three sources:
1. **V1 / V1.5 real failures**
- real bugs and real-hardware gaps observed during Phase 12 / Phase 13
- these are the highest-value scenarios because they came from actual system behavior
2. **V2 design obligations**
- scenarios required by the intended V2 protocol shape
- examples:
- reservations
- lineage-first promotion
- explicit catch-up vs rebuild boundary
3. **Distributed-systems adversarial cases**
- scenarios not yet seen in production, but known to be dangerous
- examples:
- zombie primary
- partitions
- message reordering
- multi-promotion lineage chains
This file is the shared backlog for anyone extending:
- `sw-block/prototype/fsmv2/`
- `sw-block/prototype/volumefsm/`
- `sw-block/prototype/distsim/`
For active development sequencing, see:
- `sw-block/.private/phase/phase-01.md`
- `sw-block/.private/phase/phase-02.md`
- `sw-block/design/v2-scenario-sources-from-v1.md`
Current simulator note:
- current `distsim` coverage already includes:
- changed-address restart comparison across `V1` / `V1.5` / `V2`
- same-address transient outage comparison
- slow control-plane recovery comparison
- stale-endpoint rejection
- committed-prefix-aware promotion eligibility
## V2 Goals
Compared with WAL V1, V2 should improve:
1. state clarity
2. recovery boundary clarity
3. fencing and promotion correctness
4. testability of distributed behavior
5. proof of data correctness at a target `LSN`
Compared with rebuild-heavy systems, V2 should improve:
1. short-gap recovery cost
2. explicit progress semantics
3. catch-up vs rebuild decision quality
## Scenario Format
Each scenario should eventually define:
1. setup
2. event sequence
3. expected commit/ack behavior
4. expected promotion/fencing behavior
5. expected final data state at target `LSN`
Where possible, use synthetic 4K writes with:
- `value = LSN`
That makes correctness assertions trivial.
## Priority 1: Commit Safety
These scenarios prove the most important distributed claim:
- if the system ACKed a write under the configured policy, that write is not lost
### S1. ACK Then Primary Crash
Goal:
- prove a quorum-acknowledged write survives failover
Sequence:
1. primary commits a write
2. replicas durable-ACK enough nodes for policy
3. primary crashes immediately
4. coordinator promotes a valid replica
Expect:
- promoted node contains the committed `LSN`
- final state matches reference model at committed `LSN`
### S2. Non-Quorum Write Then Primary Crash
Goal:
- prove uncommitted data is not revived after failover
Sequence:
1. primary accepts a write locally
2. quorum durability is not reached
3. primary crashes
4. coordinator promotes another node
Expect:
- promoted node does not expose the uncommitted write
- committed `LSN` stays at previous value
### S3. Zombie Old Primary Is Fenced
Goal:
- prove old-epoch traffic cannot corrupt new lineage
Sequence:
1. primary loses lease
2. coordinator bumps epoch and promotes new primary
3. old primary continues trying to send writes / barriers
Expect:
- all old-epoch traffic is rejected
- no stale write becomes committed under the new epoch
## Priority 2: Short-Gap Recovery
These scenarios justify V2 over rebuild-heavy designs.
### S4. Brief Disconnect, WAL Catch-Up Only
Goal:
- prove a short outage recovers via WAL catch-up, not rebuild
Sequence:
1. replica disconnects briefly
2. primary continues writing
3. gap stays inside recoverable window
4. replica reconnects and catches up
Expect:
- `CatchingUp -> PromotionHold -> InSync`
- no rebuild required
- final state matches reference at target `LSN`
### S5. Flapping Replica Stays Recoverable
Goal:
- prove transient disconnects do not force unnecessary rebuild
Sequence:
1. replica disconnects and reconnects repeatedly
2. gaps stay within reserved recoverable windows
Expect:
- replica may move between `Lagging`, `CatchingUp`, and `PromotionHold`
- replica does not enter `NeedsRebuild` unless recoverability is actually lost
### S6. Tail-Chasing Under Load
Goal:
- prove behavior when primary writes faster than catch-up rate
Sequence:
1. replica reconnects behind
2. primary continues writing quickly
3. catch-up target may be reached or may fall behind again
Expect:
- explicit result:
- converge and promote
- or abort to `NeedsRebuild`
- never silently pretend the replica is current
## Priority 3: Catch-Up vs Rebuild Boundary
These scenarios justify the V2 recoverability model.
### S7. Recovery Initially Possible, Then Reservation Expires
Goal:
- prove `check -> reserve -> recover` is enforced
Sequence:
1. primary grants a recoverability reservation
2. catch-up starts
3. reservation expires or is revoked before completion
Expect:
- catch-up aborts
- replica transitions to `NeedsRebuild`
- no partial recovery is treated as success
### S8. Current Extent Cannot Recover Old LSN
Goal:
- prove the historical correctness trap
Sequence:
1. write block `B = 10` at `LSN 10`
2. later write block `B = 12` at `LSN 12`
3. attempt to recover state at `LSN 10` from current extent
Expect:
- mismatch detected
- scenario must fail correctness check
### S9. Snapshot + Tail Rebuild Works
Goal:
- prove correct long-gap reconstruction
Sequence:
1. take snapshot at `cpLSN`
2. later writes extend head
3. lagging replica rebuilds from snapshot
4. replay trailing WAL tail
Expect:
- final state matches reference at target `LSN`
## Priority 4: Quorum and Mixed Replica States
These scenarios justify V2 mode clarity.
### S10. Mixed States Under `sync_quorum`
Goal:
- prove `sync_quorum` remains available with mixed replica states
Sequence:
1. one replica `InSync`
2. one replica `CatchingUp`
3. one replica `Rebuilding`
Expect:
- writes may continue if durable quorum exists
- ACK gating follows quorum rules exactly
### S11. Mixed States Under `sync_all`
Goal:
- prove `sync_all` remains strict
Sequence:
1. same mixed-state setup as above
Expect:
- writes/acks block or fail according to `sync_all`
- no silent downgrade to quorum or best effort
### S12. Promotion Chooses Best Valid Lineage
Goal:
- prove promotion is correctness-first, not “highest apparent LSN wins”
Sequence:
1. candidate nodes have different:
- flushed LSN
- rebuild state
- epoch lineage
2. coordinator chooses a new primary
Expect:
- only a valid-lineage node is promotable
- stale or inconsistent node is rejected
## Priority 5: Smart WAL / Recovery Classes
These scenarios justify V2’s future adaptive write path.
### S13. `WALInline` Window Is Recoverable
Goal:
- prove inline WAL payload replay works directly
Sequence:
1. missing range consists of `WALInline` records
2. planner grants reservation
Expect:
- catch-up allowed
- final state correct
### S14. `ExtentReferenced` Payload Still Resolvable
Goal:
- prove direct-extent records can still support catch-up when pinned
Sequence:
1. missing range includes `ExtentReferenced` records
2. payload objects / generations are still resolvable
3. reservation pins those dependencies
Expect:
- catch-up allowed
- final state correct
### S15. `ExtentReferenced` Payload Lost
Goal:
- prove metadata alone is not enough
Sequence:
1. missing range includes `ExtentReferenced` records
2. metadata still exists
3. payload object / version is no longer resolvable
Expect:
- planner returns `NeedsRebuild`
- catch-up is forbidden
## Priority 6: Restart and Rebuild Robustness
These scenarios justify operational resilience.
### S16. Replica Restarts During Catch-Up
Goal:
- prove restart does not corrupt catch-up state
Sequence:
1. replica is catching up
2. replica restarts
3. reconnect and recover again
Expect:
- no false promotion
- resume or restart recovery cleanly
### S17. Replica Restarts During Rebuild
Goal:
- prove rebuild interruption is safe
Sequence:
1. replica is rebuilding from snapshot
2. replica restarts mid-copy
Expect:
- rebuild aborts or restarts safely
- no partial base image is treated as valid
### S18. Primary Restarts Without Failover
Goal:
- prove restart with same lineage is handled explicitly
Sequence:
1. primary stops and restarts
2. coordinator either preserves or changes epoch depending on policy
Expect:
- replicas react consistently
- no stale assumptions about previous sender sessions
### S19. Chain Of Custody Across Multiple Promotions
Goal:
- prove committed data survives more than one failover lineage step
Sequence:
1. primary `A` commits writes
2. fail over to `B`
3. `B` commits additional writes
4. fail over to `C`
Expect:
- `C` contains all writes committed by `A` and `B`
- no committed data disappears across multiple promotions
- final state matches reference model at committed `LSN`
### S20. Network Partition With Concurrent Write Attempts
Goal:
- prove epoch fencing prevents split-brain writes during partition
Sequence:
1. cluster partitions into two live sides
2. old primary side continues trying to write
3. coordinator promotes a new primary on the surviving side
4. both sides attempt to send control/data traffic
Expect:
- only the current-epoch side can advance committed state
- stale-side writes are rejected or ignored
- no conflicting committed lineage appears
## Suggested Implementation Order
Implement in this order:
1. `S1` ACK then primary crash
2. `S2` non-quorum write then primary crash
3. `S3` zombie old primary fenced
4. `S4` brief disconnect with WAL catch-up
5. `S7` reservation expiry aborts catch-up
6. `S10` mixed-state quorum policy
7. `S9` long-lag rebuild from snapshot + tail
8. `S13-S15` Smart WAL recoverability
## Coverage Matrix
Status values:
- `covered`
- `partial`
- `not_started`
- `needs_richer_model`
| Scenario | Package | Test / Artifact | Status | Notes |
|---|---|---|---|---|
| `S1` ACK then primary crash | `distsim` | `TestQuorumCommitSurvivesPrimaryFailover` | `covered` | quorum commit survives failover |
| `S2` non-quorum write then primary crash | `distsim` | `TestUncommittedWriteNotPreservedAfterPrimaryLoss` | `covered` | no false revival |
| `S3` zombie old primary fenced | `distsim` | `TestZombieOldPrimaryWritesAreFenced` | `covered` | stale epoch traffic ignored |
| `S4` brief disconnect, WAL catch-up only | `distsim` | `TestReplicaCatchupFromPrimaryWAL` | `covered` | short-gap recovery |
| `S5` flapping replica stays recoverable | `distsim` | `TestS5_FlappingReplica_NoUnnecessaryRebuild`, `TestS5_FlappingWithStateTracking`, `TestS5_FlappingExceedsBudget_EscalatesToNeedsRebuild` | `covered` | both recoverable flapping and explicit budget-exceeded escalation are now asserted |
| `S6` tail-chasing under load | `distsim` | `TestS6_TailChasing_ConvergesOrAborts`, `TestS6_TailChasing_NonConvergent_Aborts`, `TestS6_TailChasing_NonConvergent_EscalatesToNeedsRebuild`, `TestP02_S6_NonConvergent_ExplicitStateTransition` | `covered` | explicit non-convergent `CatchingUp -> NeedsRebuild` path now asserted |
| `S7` reservation expiry aborts catch-up | `fsmv2`, `volumefsm`, `distsim` | `TestFSMReservationLostNeedsRebuild`, `TestModelReservationLostDuringCatchupAfterRebuild`, `TestReservationExpiryAbortsCatchup` | `covered` | present at 3 layers |
| `S8` current extent cannot recover old LSN | `distsim` | `TestCurrentExtentCannotRecoverOldLSN` | `covered` | historical correctness trap |
| `S9` snapshot + tail rebuild works | `distsim` | `TestReplicaRebuildFromSnapshotAndTail`, `TestSnapshotPlusTrailingReplayReachesTargetLSN` | `covered` | long-gap reconstruction |
| `S10` mixed states under `sync_quorum` | `volumefsm`, `distsim` | `TestModelSyncQuorumWithThreeReplicasMixedStates`, `TestSyncQuorumWithMixedReplicaStates` | `covered` | quorum stays available |
| `S11` mixed states under `sync_all` | `distsim` | `TestSyncAllBlocksWithMixedReplicaStates` | `covered` | strict sync_all behavior |
| `S12` promotion chooses best valid lineage | `distsim` | `TestPromotionUsesValidLineageNode`, `TestS12_PromotionChoosesBestLineage_NotHighestLSN`, `TestS12_PromotionRejectsRebuildingCandidate` | `covered` | lineage-first promotion now exercised beyond simple LSN comparison |
| `S13` `WALInline` window recoverable | `distsim` | `TestWALInlineRecordsAreRecoverable` | `covered` | inline payload recoverability |
| `S14` `ExtentReferenced` payload resolvable | `distsim` | `TestExtentReferencedResolvableRecordsAreRecoverable`, `TestMixedClassRecovery_FullSuccess` | `covered` | recoverable direct-extent and mixed-class recovery case |
| `S15` `ExtentReferenced` payload lost | `distsim` | `TestExtentReferencedUnresolvableForcesRebuild`, `TestRecoverableThenUnrecoverable`, `TestTimeVaryingAvailability` | `covered` | metadata alone not enough; active recovery can transition from recoverable to unrecoverable |
| `S16` replica restarts during catch-up | `distsim` | `TestReplicaRestartDuringCatchupRestartsSafely` | `covered` | safe recovery restart |
| `S17` replica restarts during rebuild | `distsim` | `TestReplicaRestartDuringRebuildRestartsSafely` | `covered` | rebuild interruption safe |
| `S18` primary restarts without failover | `distsim` | `TestS18_PrimaryRestart_SameLineage`, `TestS18_PrimaryRestart_ReplicasRejectOldEpoch`, `TestS18_PrimaryRestart_DelayedOldAck_DoesNotAdvancePrefix`, `TestS18_PrimaryRestart_InFlightBarrierDropped`, `TestP02_S18_DelayedAck_ExplicitRejection` | `covered` | delayed stale ack rejection and committed-prefix stability are now asserted directly |
| `S19` chain of custody across promotions | `distsim` | `TestS19_ChainOfCustody_MultiplePromotions`, `TestS19_ChainOfCustody_ThreePromotions` | `covered` | multi-promotion lineage continuity covered |
| `S20` live partition with competing writes | `distsim` | `TestS20_LivePartition_StaleWritesNotCommitted`, `TestS20_LivePartition_HealRecovers`, `TestS20_StalePartition_ProtocolRejectsStaleWrites`, `TestP02_S20_StaleTraffic_CommittedPrefixUnchanged` | `covered` | stale-side protocol traffic is explicitly rejected and committed prefix remains unchanged |
## Ownership Notes
When adding a scenario:
1. add or extend the relevant prototype test:
- `fsmv2`
- `volumefsm`
- `distsim`
2. update this file with:
- status
- package location
3. keep correctness checks tied to:
- committed `LSN`
- reference model state
## Current Coverage Snapshot
Already covered in some form:
- quorum commit survives primary failover
- uncommitted write not preserved after primary loss
- zombie old primary fenced by epoch
- lagging replica catch-up from primary WAL
- reservation expiry aborts catch-up in distributed sim
- `sync_quorum` continues with one lagging replica
- `sync_all` blocks with one lagging replica
- `sync_quorum` with mixed replica states
- `sync_all` with mixed replica states
- rebuild from snapshot + tail
- promotion uses valid lineage node
- flapping recoverable vs budget-exceeded rebuild path
- tail-chasing explicit escalation to rebuild
- restart during catch-up recovers safely
- restart during rebuild recovers safely
- primary restart delayed stale ack rejection
- `WALInline` recoverability
- `ExtentReferenced` resolvable vs unresolvable boundary
- mixed-class Smart WAL recovery and time-varying payload availability
- delayed stale messages and selective drop behavior
- multi-node reservation expiry and rebuild-timeout behavior
- current extent cannot reconstruct old `LSN`
Still important to add:
- explicit coordinator-driven candidate selection among competing valid/invalid lineages
- control-plane latency scenarios derived from `CP13-8 T4b`
- explicit V1 / V1.5 / V2 comparison scenarios for:
- changed-address restart
- same-address transient outage
- slow reassignment recovery
## V1.5 Lessons To Add Or Strengthen
These come directly from WAL V1.5 / Phase 13 behavior and should be treated as high-priority scenario drivers.
### L1. Replica Restart With New Receiver Port
Observed:
- replica VS restarts
- receiver comes back on a new random port
- primary background reconnect retries old address and fails
Implication:
- direct reconnect only works if replica address is stable
Backlog impact:
- strengthen `S18`
- add a restart/address-change sub-scenario under `S20` or a future network/control-plane recovery scenario
### L2. Slow Control-Plane Reassignment Dominates Recovery
Observed:
- sync correctness preserved
- write availability recovery waits for heartbeat/reassignment cycle
Implication:
- "recoverable in theory" is not enough
- recovery latency is part of protocol quality
Backlog impact:
- `S5` is now covered at current simulator level
- strengthen `S18`
- add long-running restart/rejoin timing scenarios
### L3. Background Reconnect Helps Only Same-Address Recovery
Observed:
- background reconnect is useful for transient network failure
- not sufficient for process restart with address change
Implication:
- scenarios must distinguish:
- transient disconnect
- process restart
- address change
Backlog impact:
- keep `S4` as transient disconnect
- strengthen `S18` with restart/address-stability cases
### L4. Tail-Chasing And Retention Pressure Are Structural Risks
Observed:
- Phase 13 reasoning repeatedly exposed:
- lagging replica may pin WAL
- catch-up may not converge while primary keeps advancing
Implication:
- V2 must explicitly model convergence, abort, and rebuild boundaries
Backlog impact:
- strengthen `S6`
- add multi-node retention / timeout variants
### L5. Current Extent Is Not Historical State
Observed:
- using current extent to reconstruct old `LSN` can return later values
Implication:
- V2 must require version-correct base images or resolvable historical payloads
Backlog impact:
- already covered by `S8`
- should remain a permanent regression scenario
## Randomized Simulation
In addition to fixed scenarios, V2 should keep a randomized simulator suite.
Purpose:
1. discover paths that were not explicitly written as named scenarios
2. stress promotion, restart, and recovery ordering
3. check invariants after each random step
Current prototype:
- `sw-block/prototype/distsim/random.go`
- `sw-block/prototype/distsim/random_test.go`
Current invariants checked:
1. current committed `LSN` remains a committed prefix
2. promotable nodes match reference state at committed `LSN`
3. current primary, if valid/running, matches reference state at committed `LSN`
This does not replace named scenarios.
It complements them.
## Scenario Summary
When reviewing or adding scenarios, always record the source:
1. from real V1/V1.5 behavior
2. from explicit V2 design obligation
3. from adversarial distributed-systems reasoning
The best scenarios are the ones that come from real failures first, then are generalized into V2 requirements.
## Development Phases
Execution detail is tracked in:
- `sw-block/.private/phase/phase-01.md`
- `sw-block/.private/phase/phase-02.md`
High-level phase order:
1. close explicit scenario backlog
- `S19`
- `S20`
2. strengthen missing lifecycle scenarios
- `S5`
- `S6`
- `S18`
- stronger `S12`
3. extend protocol-state simulation and version comparison
- `V1`
- `V1.5`
- `V2`
- stronger closure of current `partial` scenarios
4. strengthen random/adversarial simulation
5. add timeout-based scenarios only when the execution path is modeled