seaweedfs/sw-block/design/v2-validation-matrix.md

V2 Validation Matrix

Date: 2026-04-08 Status: active

Purpose

This document defines the concrete validation matrix for the V2 protocol and runtime.

It answers four practical questions:

1. what must be green before a stage can be called ready
2. which existing V1 tests should be reused
3. which new V2-only tests are required because truth ownership changed
4. what the final validation signal must be for each scenario

This document is the stage-oriented companion to:

1. sw-block/design/v2-proof-and-retest-pyramid.md
2. sw-block/design/v2-reuse-replacement-boundary.md
3. sw-block/design/v2-sync-recovery-protocol.md
4. sw-block/design/v2-rebuild-mvp-session-protocol.md

Validation Stages

V2 validation is gated in three layers:

1. Rebuild Ready: primary-driven rebuild is correct, convergent, and fail-closed
2. Restore Ready: exact snapshot/export/import and snapshot-tail recovery are correct
3. V2 Ready: assignment, sync facts, keep/catch/rebuild, failover/rejoin, and publication semantics close end-to-end

The intent is to avoid claiming overall V2 readiness from rebuild-only proof.

Reuse Policy From V1

Reuse unchanged

Reuse existing V1 tests unchanged when the observable contract is still the same in V2:

1. WAL append/replay correctness
2. flusher and checkpoint correctness
3. dirty-map and extent correctness
4. receiver data-plane correctness
5. barrier/fsync data integrity when semantic meaning did not change
6. snapshot export/import integrity tests
7. fail-closed mid-transfer and corruption detection tests

Reuse with adapter

Reuse the intent, but update the trigger path when V2 moved the decision point to primary-owned sync facts:

1. rebuild tests that formerly used direct local install helpers
2. catch-up tests that formerly bypassed sync-driven entry
3. reconnect/rejoin tests that now must enter via syncAck -> primary decision
4. failover tests that now must prove assignment/session/projection layering

Retire or replace

Do not reuse tests that encode V1.5 semantics that V2 intentionally removed:

1. shipper or replica self-escalation to rebuild
2. CP13-6 style max-bytes retention-triggered rebuild decisions
3. direct local shortcut paths used as if they were protocol truth

Stage Gate Summary

Stage	Closure meaning	Must-have scope
Rebuild Ready	V2 rebuild is safe and correct on the real session-controlled path	rebuild kernel, runtime, trigger scenarios, data identity
Restore Ready	exact base restore and snapshot-tail recovery are safe and exact	snapshot boundary, integrity, partial-failure safety, tail convergence
V2 Ready	primary-owned assignment/session/projection semantics close on real flows	bootstrap, keepup, catchup, rebuild, failover, rejoin, publish gating

Matrix Linkage

Read this matrix together with:

1. v2-capability-map.md for ownership of the capability tier
2. v2-integration-matrix.md for real scenario coverage

This matrix answers "is the capability closed at all?" It does not by itself guarantee that enough real topology/workload/failure scenarios have been exercised. That second question belongs to the integration matrix.

Validation rows	Capability tier	Primary protocol refs	Main integration refs
R1-R12	Tier 3: RF=2 Recovery And Failover	v2-sync-recovery-protocol.md, v2-rebuild-mvp-session-protocol.md	I-R1-I-R8
S1-S10	Tier 5: Lifecycle Capability	v2-rebuild-mvp-session-protocol.md and snapshot/restore execution rules	I-S1-I-S4
V1-V2	Tier 2: RF=2 Replication Base	v2-sync-recovery-protocol.md	I-V1, I-V2
V3-V8	Tier 3: RF=2 Recovery And Failover	v2-sync-recovery-protocol.md, v2-rebuild-mvp-session-protocol.md	I-V3, I-V4, I-V5
V9-V10	Tier 4: Multi-Replica Runtime (RF>=3)	v2-sync-recovery-protocol.md	future RF>=3 integrated rows; currently bounded engine/component proof only
V11	Tier 3 / Tier 8 boundary	recovery protocol plus launch-envelope disturbance claims	I-V5
V12-V13	Tier 6: Control Plane And Operations	ownership / observability docs and control-plane protocol surfaces	I-V4, I-V6
V14	Tier 0: Semantic Foundation	v2-protocol-truths.md, v2-sync-recovery-protocol.md	exercised across I-V* and negative component packs

Matrix A: Rebuild Ready

ID	Priority	Scenario	Trigger / entry	Reuse	Main proof	Final validation	Coverage	File	Evidence
R1	P0	Fresh replica join rebuild	replica reports applied_lsn=0, primary decides rebuild	New	canonical primary-decided rebuild entry exists	session completes and replica returns to steady state	Covered	weed/storage/blockvol/test/component/rebuild_matrix_gaps_test.go	TestRebuild_R1_SyncAckDrivenDecision — (1) protocol engine decides rebuild from syncAck(applied=0, wal_tail=N), (2) SendSessionControl over real TCP to receiver ctrl port, (3) accepted ack read from TCP, (4) base lane over real TCP via RebuildTransportServer/Client, (5) data verified block-by-block
R2	P0	Primary-initiated 1GB rebuild with live writes	explicit primary rebuild decision	New	two-line rebuild under realistic size, 4KB blocks, live WAL, flusher active	stop writes, flush both, full extent SHA-256 match	Covered	weed/storage/blockvol/test/component/rebuild_primary_initiated_test.go	TestRebuild_PrimaryInitiated_1GB_WithLiveWrites
R3	P0	Stale replica restart beyond WAL window	reconnect with applied_lsn < wal_tail	New	stale restart naturally enters rebuild	final extent digest match	Covered	weed/storage/blockvol/test/component/rebuild_matrix_gaps_test.go	TestRebuild_R3_StaleReplicaRestartBeyondWAL
R4	P0	Rebuild completion dual gate	base finishes before WAL or WAL before base	Reuse with adapter	no premature completion	complete only after base_complete && wal_applied_lsn >= target_lsn	Covered	weed/storage/blockvol/test/component/rebuild_crash_test.go	TestRebuild_CompletionRequiresBothLanes
R5	P0	Mid-transfer failure is fail-closed	connection drop or server death mid-base	Reuse	partial rebuild does not commit mixed state	replica remains logically unchanged after failure	Covered	weed/storage/blockvol/test/component/rebuild_matrix_gaps_test.go	TestRebuild_R5_ConnectionDropMidBase
R6	P0	Wrong session / epoch rejected	stale control/data/ack frames	Reuse with adapter	stale traffic cannot mutate active rebuild	explicit reject or ignore; active session remains valid	Covered	weed/storage/blockvol/test/component/rebuild_crash_test.go	TestRebuild_EpochMismatch_WALEntryRejected, TestRebuild_ControlSurface_StartSupersedeAndComplete
R7	P0	Overlap correctness	same LBA hit by base and WAL in different orders	Reuse	bitmap and WAL-wins semantics are correct	block-by-block compare on overlap set	Covered	weed/storage/blockvol/test/component/rebuild_mvp_test.go	TestRebuild_WALApplied_NeverOverwrittenByBase, TestRebuild_BitmapSetOnApplied_NotReceived, TestRebuild_BasePlusWAL_ConvergesToTarget
R8	P1	Rebuild timeout fails closed	no progress ack	New	watchdog and cancel path are real	failed session, no silent success, pin cleared	Covered	weed/server/volume_server_block_test.go	TestBlockService_WireLocalReplicaRebuildSessionAcks_TimeoutFailsClosedAndClearsPin
R9	P1	Progress pin tracks rebuild truth	rebuild emits progress	New	retention floor follows wal_applied_lsn, not barrier-only closure	observed floor moves and later clears	Covered	weed/storage/blockvol/test/component/rebuild_retention_pin_test.go, weed/server/volume_server_block_test.go	TestRebuild_RetentionPin_FlusherRespectsRebuildPin, TestBlockService_WireLocalReplicaRebuildSessionAcks_ProgressUpdatesCoreAndPin
R10	P1	Failover-rejoin rebuild	old primary comes back as replica	New	rebuild survives real topology change	final extent digest match	Covered	weed/storage/blockvol/test/component/rebuild_failover_rejoin_test.go	TestRebuild_R10_FailoverRejoinRebuild — forced WAL recycling past nodeA position, engine strictly asserts rebuild (not catchup/keepup), CRC validated
R11	P1	Non-empty stale replica full overwrite	replica has old dirty/WAL state	New	full-base rebuild discards stale local runtime correctly	final extent digest match	Covered	weed/storage/blockvol/test/component/rebuild_r11_r12_test.go	TestRebuild_R11_DivergentReplicaFullOverwrite
R12	P1	Restart rebuild, not resume rebuild	crash mid-session	New	current MVP restart semantics are explicit and safe	fresh rebuild converges without durable base-progress	Covered	weed/storage/blockvol/test/component/rebuild_r11_r12_test.go	TestRebuild_R12_CrashMidRebuild_FreshSessionConverges

Rebuild Ready minimum gate

Rebuild Ready requires all of:

1. R1
2. R2
3. R3
4. R4
5. R5
6. R6
7. R7

Matrix B: Restore Ready

ID	Priority	Scenario	Trigger / entry	Reuse	Main proof	Final validation	Coverage	File	Evidence
S1	P0	Exact snapshot export at requested boundary	explicit snapshot request at BaseLSN	Reuse	snapshot base is exact, not approximate	manifest boundary and export boundary match exactly	Covered (V1)	weed/storage/blockvol/v2bridge/transfer_test.go	V1 snapshot export tests
S2	P0	Snapshot checksum mismatch fails	corrupt payload or wrong digest	Reuse + New	no silent bad restore	restore rejected before commit	Covered	weed/storage/blockvol/v2bridge/transfer_test.go, weed/storage/blockvol/test/component/restore_s2_corruption_test.go	Epoch mismatch: TestP1_TransferFullBase_EpochMismatch; Payload corruption: TestRestore_S2_CorruptWALEntryRejected (truncated + empty payloads rejected), TestRestore_S2_CorruptBaseBlockDetected (short block documented as transport-layer responsibility)
S3	P0	Partial snapshot transfer does not commit	disconnect mid-stream	Reuse	no half-installed snapshot state	original state preserved	Covered (V1)	weed/storage/blockvol/v2bridge/transfer_test.go	V1 partial-transfer fail-closed tests
S4	P0	Snapshot import exactness	import known snapshot image	Reuse	imported extent equals exported snapshot image	full extent digest match	Covered (V1)	weed/storage/blockvol/v2bridge/transfer_test.go	V1 import exactness tests
S5	P0	Snapshot-tail rebuild	exact snapshot install plus WAL tail replay	New	restore and live convergence close together	final extent digest match	Covered	weed/storage/blockvol/test/component/restore_ready_test.go	TestRestore_S5_SnapshotTailRebuild
S6	P0	Boundary mismatch rejected	server returns wrong base boundary	Reuse	exact restore contract is enforced	explicit failure, no commit	Covered (V1)	weed/storage/blockvol/v2bridge/transfer_test.go	V1 boundary mismatch tests
S7	P1	Restart after snapshot install before tail replay	crash between base and tail phases	New	snapshot state is durable and fresh decision can continue recovery	final extent digest match	Covered	weed/storage/blockvol/test/component/restore_ready_test.go	TestRestore_S7_CrashBetweenBaseAndTail
S8	P1	Snapshot under concurrent writes	writes continue after snapshot boundary	Reuse with adapter	exact base is preserved and later writes arrive through WAL tail	final extent digest match	Covered	weed/storage/blockvol/test/component/restore_ready_test.go	TestRestore_S8_SnapshotUnderConcurrentWrites
S9	P1	Stale snapshot request rejected	requested historical boundary no longer satisfiable	New	protocol refuses unverifiable restore request	explicit failure	Covered (unit)	sw-block/engine/replication/restore_ready_test.go	TestRestore_S9_StaleSnapshotRequestRejected + TestRestore_S9_BoundaryJustOutsideRetention — gap_beyond_retention when replica LSN < WAL tail, edge case at tail boundary
S10	P1	Snapshot-tail chosen from trusted checkpoint	primary selects exact restore path	New	planner/runtime choose snapshot-tail only when allowed	reaches steady state without semantic drift	Covered (unit)	sw-block/engine/replication/restore_ready_test.go	TestRestore_S10_SnapshotTailChosenFromTrustedCheckpoint — 5 cases: trusted+covered→snapshot-tail, untrusted→full-base, WAL gap→full-base, no checkpoint→full-base, checkpoint>committed edge

Restore Ready minimum gate

Restore Ready requires all of:

1. S1
2. S2
3. S3
4. S4
5. S5
6. S6

Matrix C: V2 Ready

ID	Priority	Scenario	Trigger / entry	Reuse	Main proof	Final validation	Coverage	File	Evidence
V1	P0	Bootstrap to healthy primary	fresh assignment/bootstrap	Reuse	stage-0 bootstrap closure still holds	healthy publish with correct projection	Covered	weed/storage/blockvol/test/component/publish_healthy_test.go	TestPublishHealthy_WholeChain_FreshRF2; Stage 0 hardware PASS (phase20-t6-stage0)
V2	P0	Sustained write plus barrier closure	fio/dd/fsync style load	Reuse	healthy data-plane remains correct under workload	data checksum and barrier success	Covered	weed/storage/blockvol/test/component/bootstrap_shipping_test.go	TestBootstrap_SyncCacheIsDurabilityFence_NotWriteLBA; Stage 1 hardware 32/33
V3	P0	Sync timeout to rebuild	timeout fact enters primary decision	New	fact-driven recovery entry is real	rebuild completes, projection returns to steady state	Covered (server)	weed/server/block_recovery_test.go	TestP16B_FactTriggeredRebuildCycle_AutoInstallsRebuildAndReachesInSync
V4	P0	Sync facts choose keepup vs catchup vs rebuild	different replica reported positions	New	primary owns recovery classification	expected path is chosen from facts	Covered (engine)	sw-block/protocol/engine_test.go	TestSyncAck_ReplicaCaughtUp_KeepUp, TestSyncAck_ReplicaBehindWithinWAL_CatchUp, TestSyncAck_ReplicaBeyondWAL_Rebuild
V5	P0	Rebuild complete requires durability proof for publish	rebuild finishes but publish requires DurableLSN > 0	New	DurableLSN=0 after rebuild blocks publish; DurableLSN>0 (from rebuild completion or barrier) enables publish	no publish without durability evidence	Covered (engine)	sw-block/protocol/v2_ready_test.go	TestV2Ready_V5_RebuildCompleteNotPublishReady — 3 cases: (1) rebuild+DurableLSN>0 → publish (engine design choice: rebuild completion is durability proof), (2) DurableLSN=0 → hard-asserts no publish, (3) barrier after DurableLSN=0 rebuild → hard-asserts publish
V6	P0	Only one live session per replica	repeated triggers or supersede	New	no dual-contract ambiguity	at most one active session per replica	Covered	weed/storage/blockvol/test/component/rebuild_mvp_test.go	TestRebuild_ControlSurface_StartSupersedeAndComplete
V7	P0	Session failure re-enters facts path	catchup/rebuild fail or timeout	New	no local self-escalation survives in V2	new action always comes from fresh primary decision	Covered (server)	weed/server/block_recovery_test.go	TestP16B_OnCatchUpFailed_ReentersFactDecisionForRebuild
V8	P0	Primary failover and old-primary rejoin	failover then rejoin	New	assignment/session/projection layering closes end-to-end	rejoined node converges and health surfaces are correct	Partial	weed/storage/blockvol/test/component/rebuild_failover_rejoin_test.go	TestRebuild_R10_FailoverRejoinRebuild — proves role swap + engine rebuild decision + CRC convergence, but does NOT verify weed/server projection or master health surface
V9	P1	Mixed health aggregate projection	one in-sync, one rebuilding, one stale	Reuse with adapter	volume health is derived from aggregate replica state	projection matches expected degraded or healthy mode	Covered (engine)	sw-block/protocol/v2_ready_test.go	TestV2Ready_V9_MixedHealthAggregateProjection — hard-asserts: one rebuilding → needs_rebuild; all converged + barrier → publish_healthy
V10	P1	Retention floor under active rebuild	rebuild with live WAL pressure	New	active recovery truth is reflected into WAL retention	no premature WAL loss while rebuild is active	Covered	weed/storage/blockvol/test/component/rebuild_retention_pin_test.go	TestRebuild_RetentionPin_FlusherRespectsRebuildPin
V11	P1	Long-haul write through recovery	workload continues during fault and recovery	New	no hidden divergence across long runtime	final extent digest match	Partial	weed/storage/blockvol/test/component/rebuild_primary_initiated_test.go	TestRebuild_PrimaryInitiated_1GB_WithLiveWrites — rebuild-only, not full fault+recovery cycle
V12	P1	Operator-triggered rebuild hint	admin or explicit rebuild assignment	New	alternate entry still converges into same session protocol	rebuild completes on same execution path	Missing	—	—
V13	P1	Observability coherence	running V2 recovery	Reuse with adapter	logs, projections, diagnostics, and state snapshots agree	diagnostic surfaces remain aligned	Missing	—	—
V14	P1	Negative fail-closed matrix	wrong epoch, wrong session, stale ack, wrong kind	Reuse with adapter	ambiguity always biases toward reject or degrade	explicit failure or ignore path only	Covered (engine)	sw-block/protocol/v2_ready_test.go, weed/storage/blockvol/test/component/rebuild_crash_test.go	TestV2Ready_V14_NegativeFailClosedMatrix, TestRebuild_EpochMismatch_WALEntryRejected

V2 Ready minimum gate

V2 Ready requires:

1. all Rebuild Ready minimum rows
2. all Restore Ready minimum rows
3. V1
4. V2
5. V3
6. V4
7. V5
8. V6
9. V7
10. V8

Current Anchor Tests

The following tests are already strong anchors for the matrix and should be treated as seed evidence instead of being replaced:

Matrix row	Current anchor
R2	weed/storage/blockvol/test/component/rebuild_primary_initiated_test.go
R4, R5, R6, R7	rebuild session, transport, and executor tests under weed/storage/blockvol/ and weed/storage/blockvol/v2bridge/
V3, V6, V7	focused recovery/runtime tests under weed/server/
V1, V2	existing bootstrap/workload acceptance and component packs reused from current runner and weed/storage/blockvol/test/component/
S1-S4	existing snapshot export/import tests under weed/storage/blockvol/v2bridge/

Recommended Next Pass

When turning this matrix into execution work, use this order:

1. mark existing tests against R*, S*, and V*
2. classify each row as covered, partial, or missing
3. fill Rebuild Ready gaps first
4. fill Restore Ready next
5. keep V2 Ready small and stage-gated, not as one giant acceptance bucket

The working rule is:

• reuse V1 execution-muscle tests whenever the contract is unchanged
• add new V2 tests only where primary-owned truth changed the entry path or the closure meaning