mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-18 17:55:26 +01:00
Adds sw-block/ directory with:
- distsim: protocol correctness simulator (96 tests)
- cluster model with epoch fencing, barrier semantics, commit modes
- endpoint identity, control-plane flow, candidate eligibility
- timeout events, timer races, same-tick ordering
- session ownership tracking with ID-based stale fencing
- enginev2: standalone V2 sender/session implementation (63 tests)
- per-replica Sender with identity-preserving reconciliation
- RecoverySession with FSM phase transitions and session ID
- execution APIs: BeginConnect, RecordHandshake, BeginCatchUp,
RecordCatchUpProgress, CompleteSessionByID — all sender-authority-gated
- recovery outcome branching: zero-gap, catch-up, needs-rebuild
- assignment-intent orchestration with epoch fencing
- design docs: acceptance criteria, open questions, first-slice spec,
protocol development process
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
639 lines
20 KiB
Markdown
639 lines
20 KiB
Markdown
# WAL V2 Scenarios
|
||
|
||
Date: 2026-03-26
|
||
Status: working scenario backlog
|
||
Purpose: define the scenario set that proves why WAL V2 exists, what it must do better than WAL V1, and what it should handle better than rebuild-heavy systems
|
||
|
||
Execution note:
|
||
- active implementation planning for these scenarios lives under `../.private/phase/`
|
||
- `design/` is the design/source-of-truth view
|
||
- `.private/phase/` is the execution/checklist view for `sw`
|
||
|
||
## Why This File Exists
|
||
|
||
V2 should not grow by adding random simulations.
|
||
|
||
Each new scenario should prove one of these claims:
|
||
|
||
1. committed data is never lost
|
||
2. uncommitted data is never falsely revived
|
||
3. epoch and promotion lineage are safe
|
||
4. short-gap recovery is cheaper and cleaner than rebuild
|
||
5. catch-up vs rebuild boundary is explicit and correct
|
||
6. historical data correctness is preserved
|
||
|
||
## Scenario Sources
|
||
|
||
The backlog draws scenarios from three sources:
|
||
|
||
1. **V1 / V1.5 real failures**
|
||
- real bugs and real-hardware gaps observed during Phase 12 / Phase 13
|
||
- these are the highest-value scenarios because they came from actual system behavior
|
||
|
||
2. **V2 design obligations**
|
||
- scenarios required by the intended V2 protocol shape
|
||
- examples:
|
||
- reservations
|
||
- lineage-first promotion
|
||
- explicit catch-up vs rebuild boundary
|
||
|
||
3. **Distributed-systems adversarial cases**
|
||
- scenarios not yet seen in production, but known to be dangerous
|
||
- examples:
|
||
- zombie primary
|
||
- partitions
|
||
- message reordering
|
||
- multi-promotion lineage chains
|
||
|
||
This file is the shared backlog for anyone extending:
|
||
|
||
- `sw-block/prototype/fsmv2/`
|
||
- `sw-block/prototype/volumefsm/`
|
||
- `sw-block/prototype/distsim/`
|
||
|
||
For active development sequencing, see:
|
||
- `sw-block/.private/phase/phase-01.md`
|
||
- `sw-block/.private/phase/phase-02.md`
|
||
- `sw-block/design/v2-scenario-sources-from-v1.md`
|
||
|
||
Current simulator note:
|
||
- current `distsim` coverage already includes:
|
||
- changed-address restart comparison across `V1` / `V1.5` / `V2`
|
||
- same-address transient outage comparison
|
||
- slow control-plane recovery comparison
|
||
- stale-endpoint rejection
|
||
- committed-prefix-aware promotion eligibility
|
||
|
||
## V2 Goals
|
||
|
||
Compared with WAL V1, V2 should improve:
|
||
|
||
1. state clarity
|
||
2. recovery boundary clarity
|
||
3. fencing and promotion correctness
|
||
4. testability of distributed behavior
|
||
5. proof of data correctness at a target `LSN`
|
||
|
||
Compared with rebuild-heavy systems, V2 should improve:
|
||
|
||
1. short-gap recovery cost
|
||
2. explicit progress semantics
|
||
3. catch-up vs rebuild decision quality
|
||
|
||
## Scenario Format
|
||
|
||
Each scenario should eventually define:
|
||
|
||
1. setup
|
||
2. event sequence
|
||
3. expected commit/ack behavior
|
||
4. expected promotion/fencing behavior
|
||
5. expected final data state at target `LSN`
|
||
|
||
Where possible, use synthetic 4K writes with:
|
||
|
||
- `value = LSN`
|
||
|
||
That makes correctness assertions trivial.
|
||
|
||
## Priority 1: Commit Safety
|
||
|
||
These scenarios prove the most important distributed claim:
|
||
|
||
- if the system ACKed a write under the configured policy, that write is not lost
|
||
|
||
### S1. ACK Then Primary Crash
|
||
|
||
Goal:
|
||
- prove a quorum-acknowledged write survives failover
|
||
|
||
Sequence:
|
||
1. primary commits a write
|
||
2. replicas durable-ACK enough nodes for policy
|
||
3. primary crashes immediately
|
||
4. coordinator promotes a valid replica
|
||
|
||
Expect:
|
||
- promoted node contains the committed `LSN`
|
||
- final state matches reference model at committed `LSN`
|
||
|
||
### S2. Non-Quorum Write Then Primary Crash
|
||
|
||
Goal:
|
||
- prove uncommitted data is not revived after failover
|
||
|
||
Sequence:
|
||
1. primary accepts a write locally
|
||
2. quorum durability is not reached
|
||
3. primary crashes
|
||
4. coordinator promotes another node
|
||
|
||
Expect:
|
||
- promoted node does not expose the uncommitted write
|
||
- committed `LSN` stays at previous value
|
||
|
||
### S3. Zombie Old Primary Is Fenced
|
||
|
||
Goal:
|
||
- prove old-epoch traffic cannot corrupt new lineage
|
||
|
||
Sequence:
|
||
1. primary loses lease
|
||
2. coordinator bumps epoch and promotes new primary
|
||
3. old primary continues trying to send writes / barriers
|
||
|
||
Expect:
|
||
- all old-epoch traffic is rejected
|
||
- no stale write becomes committed under the new epoch
|
||
|
||
## Priority 2: Short-Gap Recovery
|
||
|
||
These scenarios justify V2 over rebuild-heavy designs.
|
||
|
||
### S4. Brief Disconnect, WAL Catch-Up Only
|
||
|
||
Goal:
|
||
- prove a short outage recovers via WAL catch-up, not rebuild
|
||
|
||
Sequence:
|
||
1. replica disconnects briefly
|
||
2. primary continues writing
|
||
3. gap stays inside recoverable window
|
||
4. replica reconnects and catches up
|
||
|
||
Expect:
|
||
- `CatchingUp -> PromotionHold -> InSync`
|
||
- no rebuild required
|
||
- final state matches reference at target `LSN`
|
||
|
||
### S5. Flapping Replica Stays Recoverable
|
||
|
||
Goal:
|
||
- prove transient disconnects do not force unnecessary rebuild
|
||
|
||
Sequence:
|
||
1. replica disconnects and reconnects repeatedly
|
||
2. gaps stay within reserved recoverable windows
|
||
|
||
Expect:
|
||
- replica may move between `Lagging`, `CatchingUp`, and `PromotionHold`
|
||
- replica does not enter `NeedsRebuild` unless recoverability is actually lost
|
||
|
||
### S6. Tail-Chasing Under Load
|
||
|
||
Goal:
|
||
- prove behavior when primary writes faster than catch-up rate
|
||
|
||
Sequence:
|
||
1. replica reconnects behind
|
||
2. primary continues writing quickly
|
||
3. catch-up target may be reached or may fall behind again
|
||
|
||
Expect:
|
||
- explicit result:
|
||
- converge and promote
|
||
- or abort to `NeedsRebuild`
|
||
- never silently pretend the replica is current
|
||
|
||
## Priority 3: Catch-Up vs Rebuild Boundary
|
||
|
||
These scenarios justify the V2 recoverability model.
|
||
|
||
### S7. Recovery Initially Possible, Then Reservation Expires
|
||
|
||
Goal:
|
||
- prove `check -> reserve -> recover` is enforced
|
||
|
||
Sequence:
|
||
1. primary grants a recoverability reservation
|
||
2. catch-up starts
|
||
3. reservation expires or is revoked before completion
|
||
|
||
Expect:
|
||
- catch-up aborts
|
||
- replica transitions to `NeedsRebuild`
|
||
- no partial recovery is treated as success
|
||
|
||
### S8. Current Extent Cannot Recover Old LSN
|
||
|
||
Goal:
|
||
- prove the historical correctness trap
|
||
|
||
Sequence:
|
||
1. write block `B = 10` at `LSN 10`
|
||
2. later write block `B = 12` at `LSN 12`
|
||
3. attempt to recover state at `LSN 10` from current extent
|
||
|
||
Expect:
|
||
- mismatch detected
|
||
- scenario must fail correctness check
|
||
|
||
### S9. Snapshot + Tail Rebuild Works
|
||
|
||
Goal:
|
||
- prove correct long-gap reconstruction
|
||
|
||
Sequence:
|
||
1. take snapshot at `cpLSN`
|
||
2. later writes extend head
|
||
3. lagging replica rebuilds from snapshot
|
||
4. replay trailing WAL tail
|
||
|
||
Expect:
|
||
- final state matches reference at target `LSN`
|
||
|
||
## Priority 4: Quorum and Mixed Replica States
|
||
|
||
These scenarios justify V2 mode clarity.
|
||
|
||
### S10. Mixed States Under `sync_quorum`
|
||
|
||
Goal:
|
||
- prove `sync_quorum` remains available with mixed replica states
|
||
|
||
Sequence:
|
||
1. one replica `InSync`
|
||
2. one replica `CatchingUp`
|
||
3. one replica `Rebuilding`
|
||
|
||
Expect:
|
||
- writes may continue if durable quorum exists
|
||
- ACK gating follows quorum rules exactly
|
||
|
||
### S11. Mixed States Under `sync_all`
|
||
|
||
Goal:
|
||
- prove `sync_all` remains strict
|
||
|
||
Sequence:
|
||
1. same mixed-state setup as above
|
||
|
||
Expect:
|
||
- writes/acks block or fail according to `sync_all`
|
||
- no silent downgrade to quorum or best effort
|
||
|
||
### S12. Promotion Chooses Best Valid Lineage
|
||
|
||
Goal:
|
||
- prove promotion is correctness-first, not “highest apparent LSN wins”
|
||
|
||
Sequence:
|
||
1. candidate nodes have different:
|
||
- flushed LSN
|
||
- rebuild state
|
||
- epoch lineage
|
||
2. coordinator chooses a new primary
|
||
|
||
Expect:
|
||
- only a valid-lineage node is promotable
|
||
- stale or inconsistent node is rejected
|
||
|
||
## Priority 5: Smart WAL / Recovery Classes
|
||
|
||
These scenarios justify V2’s future adaptive write path.
|
||
|
||
### S13. `WALInline` Window Is Recoverable
|
||
|
||
Goal:
|
||
- prove inline WAL payload replay works directly
|
||
|
||
Sequence:
|
||
1. missing range consists of `WALInline` records
|
||
2. planner grants reservation
|
||
|
||
Expect:
|
||
- catch-up allowed
|
||
- final state correct
|
||
|
||
### S14. `ExtentReferenced` Payload Still Resolvable
|
||
|
||
Goal:
|
||
- prove direct-extent records can still support catch-up when pinned
|
||
|
||
Sequence:
|
||
1. missing range includes `ExtentReferenced` records
|
||
2. payload objects / generations are still resolvable
|
||
3. reservation pins those dependencies
|
||
|
||
Expect:
|
||
- catch-up allowed
|
||
- final state correct
|
||
|
||
### S15. `ExtentReferenced` Payload Lost
|
||
|
||
Goal:
|
||
- prove metadata alone is not enough
|
||
|
||
Sequence:
|
||
1. missing range includes `ExtentReferenced` records
|
||
2. metadata still exists
|
||
3. payload object / version is no longer resolvable
|
||
|
||
Expect:
|
||
- planner returns `NeedsRebuild`
|
||
- catch-up is forbidden
|
||
|
||
## Priority 6: Restart and Rebuild Robustness
|
||
|
||
These scenarios justify operational resilience.
|
||
|
||
### S16. Replica Restarts During Catch-Up
|
||
|
||
Goal:
|
||
- prove restart does not corrupt catch-up state
|
||
|
||
Sequence:
|
||
1. replica is catching up
|
||
2. replica restarts
|
||
3. reconnect and recover again
|
||
|
||
Expect:
|
||
- no false promotion
|
||
- resume or restart recovery cleanly
|
||
|
||
### S17. Replica Restarts During Rebuild
|
||
|
||
Goal:
|
||
- prove rebuild interruption is safe
|
||
|
||
Sequence:
|
||
1. replica is rebuilding from snapshot
|
||
2. replica restarts mid-copy
|
||
|
||
Expect:
|
||
- rebuild aborts or restarts safely
|
||
- no partial base image is treated as valid
|
||
|
||
### S18. Primary Restarts Without Failover
|
||
|
||
Goal:
|
||
- prove restart with same lineage is handled explicitly
|
||
|
||
Sequence:
|
||
1. primary stops and restarts
|
||
2. coordinator either preserves or changes epoch depending on policy
|
||
|
||
Expect:
|
||
- replicas react consistently
|
||
- no stale assumptions about previous sender sessions
|
||
|
||
### S19. Chain Of Custody Across Multiple Promotions
|
||
|
||
Goal:
|
||
- prove committed data survives more than one failover lineage step
|
||
|
||
Sequence:
|
||
1. primary `A` commits writes
|
||
2. fail over to `B`
|
||
3. `B` commits additional writes
|
||
4. fail over to `C`
|
||
|
||
Expect:
|
||
- `C` contains all writes committed by `A` and `B`
|
||
- no committed data disappears across multiple promotions
|
||
- final state matches reference model at committed `LSN`
|
||
|
||
### S20. Network Partition With Concurrent Write Attempts
|
||
|
||
Goal:
|
||
- prove epoch fencing prevents split-brain writes during partition
|
||
|
||
Sequence:
|
||
1. cluster partitions into two live sides
|
||
2. old primary side continues trying to write
|
||
3. coordinator promotes a new primary on the surviving side
|
||
4. both sides attempt to send control/data traffic
|
||
|
||
Expect:
|
||
- only the current-epoch side can advance committed state
|
||
- stale-side writes are rejected or ignored
|
||
- no conflicting committed lineage appears
|
||
|
||
## Suggested Implementation Order
|
||
|
||
Implement in this order:
|
||
|
||
1. `S1` ACK then primary crash
|
||
2. `S2` non-quorum write then primary crash
|
||
3. `S3` zombie old primary fenced
|
||
4. `S4` brief disconnect with WAL catch-up
|
||
5. `S7` reservation expiry aborts catch-up
|
||
6. `S10` mixed-state quorum policy
|
||
7. `S9` long-lag rebuild from snapshot + tail
|
||
8. `S13-S15` Smart WAL recoverability
|
||
|
||
## Coverage Matrix
|
||
|
||
Status values:
|
||
- `covered`
|
||
- `partial`
|
||
- `not_started`
|
||
- `needs_richer_model`
|
||
|
||
| Scenario | Package | Test / Artifact | Status | Notes |
|
||
|---|---|---|---|---|
|
||
| `S1` ACK then primary crash | `distsim` | `TestQuorumCommitSurvivesPrimaryFailover` | `covered` | quorum commit survives failover |
|
||
| `S2` non-quorum write then primary crash | `distsim` | `TestUncommittedWriteNotPreservedAfterPrimaryLoss` | `covered` | no false revival |
|
||
| `S3` zombie old primary fenced | `distsim` | `TestZombieOldPrimaryWritesAreFenced` | `covered` | stale epoch traffic ignored |
|
||
| `S4` brief disconnect, WAL catch-up only | `distsim` | `TestReplicaCatchupFromPrimaryWAL` | `covered` | short-gap recovery |
|
||
| `S5` flapping replica stays recoverable | `distsim` | `TestS5_FlappingReplica_NoUnnecessaryRebuild`, `TestS5_FlappingWithStateTracking`, `TestS5_FlappingExceedsBudget_EscalatesToNeedsRebuild` | `covered` | both recoverable flapping and explicit budget-exceeded escalation are now asserted |
|
||
| `S6` tail-chasing under load | `distsim` | `TestS6_TailChasing_ConvergesOrAborts`, `TestS6_TailChasing_NonConvergent_Aborts`, `TestS6_TailChasing_NonConvergent_EscalatesToNeedsRebuild`, `TestP02_S6_NonConvergent_ExplicitStateTransition` | `covered` | explicit non-convergent `CatchingUp -> NeedsRebuild` path now asserted |
|
||
| `S7` reservation expiry aborts catch-up | `fsmv2`, `volumefsm`, `distsim` | `TestFSMReservationLostNeedsRebuild`, `TestModelReservationLostDuringCatchupAfterRebuild`, `TestReservationExpiryAbortsCatchup` | `covered` | present at 3 layers |
|
||
| `S8` current extent cannot recover old LSN | `distsim` | `TestCurrentExtentCannotRecoverOldLSN` | `covered` | historical correctness trap |
|
||
| `S9` snapshot + tail rebuild works | `distsim` | `TestReplicaRebuildFromSnapshotAndTail`, `TestSnapshotPlusTrailingReplayReachesTargetLSN` | `covered` | long-gap reconstruction |
|
||
| `S10` mixed states under `sync_quorum` | `volumefsm`, `distsim` | `TestModelSyncQuorumWithThreeReplicasMixedStates`, `TestSyncQuorumWithMixedReplicaStates` | `covered` | quorum stays available |
|
||
| `S11` mixed states under `sync_all` | `distsim` | `TestSyncAllBlocksWithMixedReplicaStates` | `covered` | strict sync_all behavior |
|
||
| `S12` promotion chooses best valid lineage | `distsim` | `TestPromotionUsesValidLineageNode`, `TestS12_PromotionChoosesBestLineage_NotHighestLSN`, `TestS12_PromotionRejectsRebuildingCandidate` | `covered` | lineage-first promotion now exercised beyond simple LSN comparison |
|
||
| `S13` `WALInline` window recoverable | `distsim` | `TestWALInlineRecordsAreRecoverable` | `covered` | inline payload recoverability |
|
||
| `S14` `ExtentReferenced` payload resolvable | `distsim` | `TestExtentReferencedResolvableRecordsAreRecoverable`, `TestMixedClassRecovery_FullSuccess` | `covered` | recoverable direct-extent and mixed-class recovery case |
|
||
| `S15` `ExtentReferenced` payload lost | `distsim` | `TestExtentReferencedUnresolvableForcesRebuild`, `TestRecoverableThenUnrecoverable`, `TestTimeVaryingAvailability` | `covered` | metadata alone not enough; active recovery can transition from recoverable to unrecoverable |
|
||
| `S16` replica restarts during catch-up | `distsim` | `TestReplicaRestartDuringCatchupRestartsSafely` | `covered` | safe recovery restart |
|
||
| `S17` replica restarts during rebuild | `distsim` | `TestReplicaRestartDuringRebuildRestartsSafely` | `covered` | rebuild interruption safe |
|
||
| `S18` primary restarts without failover | `distsim` | `TestS18_PrimaryRestart_SameLineage`, `TestS18_PrimaryRestart_ReplicasRejectOldEpoch`, `TestS18_PrimaryRestart_DelayedOldAck_DoesNotAdvancePrefix`, `TestS18_PrimaryRestart_InFlightBarrierDropped`, `TestP02_S18_DelayedAck_ExplicitRejection` | `covered` | delayed stale ack rejection and committed-prefix stability are now asserted directly |
|
||
| `S19` chain of custody across promotions | `distsim` | `TestS19_ChainOfCustody_MultiplePromotions`, `TestS19_ChainOfCustody_ThreePromotions` | `covered` | multi-promotion lineage continuity covered |
|
||
| `S20` live partition with competing writes | `distsim` | `TestS20_LivePartition_StaleWritesNotCommitted`, `TestS20_LivePartition_HealRecovers`, `TestS20_StalePartition_ProtocolRejectsStaleWrites`, `TestP02_S20_StaleTraffic_CommittedPrefixUnchanged` | `covered` | stale-side protocol traffic is explicitly rejected and committed prefix remains unchanged |
|
||
|
||
## Ownership Notes
|
||
|
||
When adding a scenario:
|
||
|
||
1. add or extend the relevant prototype test:
|
||
- `fsmv2`
|
||
- `volumefsm`
|
||
- `distsim`
|
||
2. update this file with:
|
||
- status
|
||
- package location
|
||
3. keep correctness checks tied to:
|
||
- committed `LSN`
|
||
- reference model state
|
||
|
||
## Current Coverage Snapshot
|
||
|
||
Already covered in some form:
|
||
|
||
- quorum commit survives primary failover
|
||
- uncommitted write not preserved after primary loss
|
||
- zombie old primary fenced by epoch
|
||
- lagging replica catch-up from primary WAL
|
||
- reservation expiry aborts catch-up in distributed sim
|
||
- `sync_quorum` continues with one lagging replica
|
||
- `sync_all` blocks with one lagging replica
|
||
- `sync_quorum` with mixed replica states
|
||
- `sync_all` with mixed replica states
|
||
- rebuild from snapshot + tail
|
||
- promotion uses valid lineage node
|
||
- flapping recoverable vs budget-exceeded rebuild path
|
||
- tail-chasing explicit escalation to rebuild
|
||
- restart during catch-up recovers safely
|
||
- restart during rebuild recovers safely
|
||
- primary restart delayed stale ack rejection
|
||
- `WALInline` recoverability
|
||
- `ExtentReferenced` resolvable vs unresolvable boundary
|
||
- mixed-class Smart WAL recovery and time-varying payload availability
|
||
- delayed stale messages and selective drop behavior
|
||
- multi-node reservation expiry and rebuild-timeout behavior
|
||
- current extent cannot reconstruct old `LSN`
|
||
|
||
Still important to add:
|
||
|
||
- explicit coordinator-driven candidate selection among competing valid/invalid lineages
|
||
- control-plane latency scenarios derived from `CP13-8 T4b`
|
||
- explicit V1 / V1.5 / V2 comparison scenarios for:
|
||
- changed-address restart
|
||
- same-address transient outage
|
||
- slow reassignment recovery
|
||
|
||
## V1.5 Lessons To Add Or Strengthen
|
||
|
||
These come directly from WAL V1.5 / Phase 13 behavior and should be treated as high-priority scenario drivers.
|
||
|
||
### L1. Replica Restart With New Receiver Port
|
||
|
||
Observed:
|
||
- replica VS restarts
|
||
- receiver comes back on a new random port
|
||
- primary background reconnect retries old address and fails
|
||
|
||
Implication:
|
||
- direct reconnect only works if replica address is stable
|
||
|
||
Backlog impact:
|
||
- strengthen `S18`
|
||
- add a restart/address-change sub-scenario under `S20` or a future network/control-plane recovery scenario
|
||
|
||
### L2. Slow Control-Plane Reassignment Dominates Recovery
|
||
|
||
Observed:
|
||
- sync correctness preserved
|
||
- write availability recovery waits for heartbeat/reassignment cycle
|
||
|
||
Implication:
|
||
- "recoverable in theory" is not enough
|
||
- recovery latency is part of protocol quality
|
||
|
||
Backlog impact:
|
||
- `S5` is now covered at current simulator level
|
||
- strengthen `S18`
|
||
- add long-running restart/rejoin timing scenarios
|
||
|
||
### L3. Background Reconnect Helps Only Same-Address Recovery
|
||
|
||
Observed:
|
||
- background reconnect is useful for transient network failure
|
||
- not sufficient for process restart with address change
|
||
|
||
Implication:
|
||
- scenarios must distinguish:
|
||
- transient disconnect
|
||
- process restart
|
||
- address change
|
||
|
||
Backlog impact:
|
||
- keep `S4` as transient disconnect
|
||
- strengthen `S18` with restart/address-stability cases
|
||
|
||
### L4. Tail-Chasing And Retention Pressure Are Structural Risks
|
||
|
||
Observed:
|
||
- Phase 13 reasoning repeatedly exposed:
|
||
- lagging replica may pin WAL
|
||
- catch-up may not converge while primary keeps advancing
|
||
|
||
Implication:
|
||
- V2 must explicitly model convergence, abort, and rebuild boundaries
|
||
|
||
Backlog impact:
|
||
- strengthen `S6`
|
||
- add multi-node retention / timeout variants
|
||
|
||
### L5. Current Extent Is Not Historical State
|
||
|
||
Observed:
|
||
- using current extent to reconstruct old `LSN` can return later values
|
||
|
||
Implication:
|
||
- V2 must require version-correct base images or resolvable historical payloads
|
||
|
||
Backlog impact:
|
||
- already covered by `S8`
|
||
- should remain a permanent regression scenario
|
||
|
||
## Randomized Simulation
|
||
|
||
In addition to fixed scenarios, V2 should keep a randomized simulator suite.
|
||
|
||
Purpose:
|
||
|
||
1. discover paths that were not explicitly written as named scenarios
|
||
2. stress promotion, restart, and recovery ordering
|
||
3. check invariants after each random step
|
||
|
||
Current prototype:
|
||
|
||
- `sw-block/prototype/distsim/random.go`
|
||
- `sw-block/prototype/distsim/random_test.go`
|
||
|
||
Current invariants checked:
|
||
|
||
1. current committed `LSN` remains a committed prefix
|
||
2. promotable nodes match reference state at committed `LSN`
|
||
3. current primary, if valid/running, matches reference state at committed `LSN`
|
||
|
||
This does not replace named scenarios.
|
||
It complements them.
|
||
|
||
## Scenario Summary
|
||
|
||
When reviewing or adding scenarios, always record the source:
|
||
|
||
1. from real V1/V1.5 behavior
|
||
2. from explicit V2 design obligation
|
||
3. from adversarial distributed-systems reasoning
|
||
|
||
The best scenarios are the ones that come from real failures first, then are generalized into V2 requirements.
|
||
|
||
## Development Phases
|
||
|
||
Execution detail is tracked in:
|
||
- `sw-block/.private/phase/phase-01.md`
|
||
- `sw-block/.private/phase/phase-02.md`
|
||
|
||
High-level phase order:
|
||
|
||
1. close explicit scenario backlog
|
||
- `S19`
|
||
- `S20`
|
||
2. strengthen missing lifecycle scenarios
|
||
- `S5`
|
||
- `S6`
|
||
- `S18`
|
||
- stronger `S12`
|
||
3. extend protocol-state simulation and version comparison
|
||
- `V1`
|
||
- `V1.5`
|
||
- `V2`
|
||
- stronger closure of current `partial` scenarios
|
||
4. strengthen random/adversarial simulation
|
||
5. add timeout-based scenarios only when the execution path is modeled
|