- phase-13.md: CP13-1 through CP13-6 accepted, CP13-7 active
- phase-13-log.md: full technical + delivery packs for CP13-2..CP13-7
- phase-13-cp4-state-eligibility.md: refined barrier behavior table
(Disconnected/Degraded as recovery entry points, not eligibility)
- phase-12.md: minor cross-reference updates
- Older phase docs: minor wording alignment
- Design docs: V2 development plan and completion overview updated
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two fixes:
1. TestReconnect_GapBeyondRetainedWal_NeedsRebuild: rewritten to test the
real reconnect handshake gap detection path (R < S in
reconnectWithHandshake). Sequence: establish sync → disconnect →
release retention hold via timeout → write + flush to advance WAL past
replica position → reconnect → handshake detects R=0 < S=9 → NeedsRebuild.
Log proves: "reconnect: gap too large R=0 H=8 S=9"
2. TestReplicaState_RebuildComplete_ReentersInSync: reclassified from
primary proof to support evidence (does not start from live NeedsRebuild
shipper state, but proves rebuild mechanics work end-to-end).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. TestWalRetention_TimeoutTriggersNeedsRebuild: add hard assertion that
checkpoint advances past replicaFlushedLSN after NeedsRebuild (proves
hold is actually released, not just state transition)
2. TestWalRetention_RequiredReplicaBlocksReclaim: remove stale "EXPECTED
TO FAIL" / duplicate comment block
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three fixes:
1. TestWalRetention_RequiredReplicaBlocksReclaim: rewritten from log-only
placeholder to hard assertion (checkpointLSN <= replicaFlushedLSN)
2. TestWalRetention_TimeoutTriggersNeedsRebuild: rewritten from log-only
to hard assertion (State() == NeedsRebuild after 1ns timeout)
3. EvaluateRetentionBudgets: uses RetentionBudgetParams struct with
actual BlockSize from volume config instead of hardcoded 4096
All 3 retention tests now have real state/progress assertions.
No placeholder or log-only evidence remains in CP13-6 proof package.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add max-bytes retention budget alongside existing timeout budget:
- shipper_group.go: EvaluateRetentionBudgets now checks both timeout
(last contact time) and max-bytes (entry lag * 4KB > maxBytes).
Either exceeding budget → NeedsRebuild state transition.
- blockvol.go: add walRetentionMaxBytes (64MB default), pass to
EvaluateRetentionBudgets with primaryHeadLSN.
TestWalRetention_MaxBytesTriggersNeedsRebuild upgraded from PASS*
(log-only placeholder) to real PASS: asserts State()==NeedsRebuild
after lag exceeds configured max-bytes budget.
Retention contract: hold-back blocks reclaim for recoverable replicas,
timeout and max-bytes budgets escalate to NeedsRebuild and release hold.
Full rebuild lifecycle remains CP13-7 scope.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace "observable CatchingUp state transition" with the actual 3
signals the test asserts: seeded hasFlushedProgress, receivedLSN
advance, non-zero replicaFlushedLSN.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Findings fixed:
1. TestAdversarial_ReconnectUsesHandshakeNotBootstrap now has 3 observable
proof points instead of just "SyncCache succeeded":
- new shipper HasFlushedProgress=true (seeded from old group)
- replica receivedLSN advances during SyncCache (catch-up delivered entries)
- shipper replicaFlushedLSN > 0 after barrier (durable progress established)
Bootstrap alone would not advance receivedLSN — it only sends the barrier.
2. TestBug2 stale comment removed: "must NOT call SetReplicaAddr" replaced
with accurate CP13-5 explanation that SetReplicaAddrs now preserves
hasFlushedProgress across shipper replacement.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bug: SetReplicaAddrs created fresh shippers (hasFlushedProgress=false),
so after disconnect, the new shipper used bootstrap instead of reconnect
handshake. Bootstrap doesn't replay missed WAL entries — barrier hung.
Fix:
- blockvol.go: SetReplicaAddrs checks if old shipper group had durable
progress (AnyHasFlushedProgress). If so, seeds new shippers with
hasFlushedProgress=true → they use reconnect handshake + catch-up.
- shipper_group.go: add AnyHasFlushedProgress() helper.
3 baseline FAILs now PASS:
- ReconnectUsesHandshakeNotBootstrap: reconnect path used, not bootstrap
- CatchupMultipleDisconnects: repeated disconnect/reconnect recovers
- CatchupDoesNotOverwriteNewerData: catch-up completes, safety exercised
7 tests promoted to CP13-5 primary proof.
TestAdversarial_NeedsRebuildBlocksAllPaths still FAIL (CP13-7 scope).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contract review: 6-state set (Disconnected, Connecting, CatchingUp,
InSync, Degraded, NeedsRebuild). Only InSync proceeds to barrier
request path. All other states either fail immediately or attempt
reconnect (must succeed before reaching barrier).
New test: TestBarrier_NonEligibleStates_FailClosed — systematically
verifies each non-eligible state (Connecting, CatchingUp, NeedsRebuild,
Disconnected) is rejected by Barrier(), and InSync is the only state
that enters the barrier request path.
5 baseline tests promoted to CP13-4 primary proof.
No production code changed — contract review + new focused test only.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bug: BarrierOK with FlushedLSN == 0 (legacy 1-byte response) was counted
as successful sync_all durability even though no authoritative durable
progress was established. This allowed a legacy replica to silently pass
through the sync_all barrier without proving any LSN was fsynced.
Fix (wal_shipper.go): BarrierOK with FlushedLSN == 0 now returns an
error instead of nil. Barrier success requires the replica to report a
non-zero FlushedLSN proving which LSN was durably persisted. This makes
the code match the CP13-3 contract: replicaFlushedLSN is the sole
authority for sync_all durability.
New test: TestBarrier_LegacyResponseRejectedBySyncAll — proves legacy
1-byte responses don't establish durable authority.
Contract review doc updated to reflect the code fix.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contract review (no code changed):
- replicaFlushedLSN is the sole authority for replica durability
- flushedLSN advanced only after fd.Sync() on replica (not on receive)
- shippedLSN/sentLSN are explicitly diagnostic (comment at line 268)
- barrier response carries flushedLSN; shipper updates via monotonic CAS
- sync_all gates on ALL barriers succeeding (fail-closed)
8 baseline tests promoted to CP13-3 primary proof:
- BarrierUsesFlushedLSN, FlushedLSNMonotonicWithinEpoch
- FlushedLSN_OnlyAfterSync, FlushedLSN_NotOnReceive
- ShipperReplicaFlushedLSN_UpdatedOnBarrier, _Monotonic
- BarrierResp_FlushedLSN_Roundtrip, BackwardCompat_1Byte
6 tests classified as support evidence (not primary proof).
Reconnect/retention/rebuild tests explicitly out of scope (CP13-4+).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Change "CP13-3/4/5/6 behavior already implemented in earlier phases" to
"current code already passes tests associated with later checkpoint themes"
— baseline evidence only, not implementation closure.
No .go files changed in CP13-1. All 44 baseline tests already existed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- phase-13-log.md: mark pre-baseline inventory table as superseded,
point to phase-13-cp1-baseline.md for authoritative results
- phase-13-cp1-baseline.md: replace "CP13-X done" language with neutral
"current code passes this test; suggests behavior may already exist"
— checkpoint closure still requires dedicated review
- Expand remaining-open-checkpoints section: CP13-2/5/6/7 all still
require review, main fails cluster around CP13-5 but CP13-7 and
part of CP13-6 also remain open
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copies of design docs removed in Phase 09, preserved in sw-block/docs/archive/
for historical reference.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Engine executors now have IO interfaces for real bridge I/O:
- CatchUpExecutor.IO (CatchUpIO): StreamWALEntries
- RebuildExecutor.IO (RebuildIO): TransferFullBase, TransferSnapshot,
StreamWALEntries (for tail replay)
When IO is set, executor calls real bridge I/O during execution.
When IO is nil, executor uses caller-supplied progress (test mode).
RecoveryPlan.CatchUpStartLSN: bound at plan time for IO bridge.
v2bridge.Executor now implements both interfaces:
- StreamWALEntries: real ScanFrom
- TransferFullBase: validates extent accessible
- TransferSnapshot: validates checkpoint accessible
Chain tests wire IO:
- CatchUpClosure: exec.IO = executor → real WAL scan through engine
- RebuildClosure: exec.IO = executor → real transfer through engine
This closes the engine → executor → v2bridge → blockvol chain.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Finding 1: Identity no longer address-derived
- ReplicaAddr.ServerID field added (stable server identity from registry)
- BlockVolumeAssignment.ReplicaServerID field added (scalar RF=2 path)
- ControlBridge uses ServerID, NOT address, for ReplicaID
- Missing ServerID → replica skipped (fail closed), logged
Finding 2: Wired into real ProcessAssignments
- BlockService.v2Bridge field initialized in StartBlockService
- ProcessAssignments converts each assignment via v2Bridge.ConvertAssignment
BEFORE existing V1 processing (parallel, not replacing yet)
- Logged at glog V(1)
Finding 3: Fail-closed on missing identity
- Empty ServerID in ReplicaAddrs → replica skipped with log
- Empty ReplicaServerID in scalar path → no replica created
- Test: MissingServerID_FailsClosed verifies both paths
7 tests: StableServerID, AddressChange_IdentityPreserved,
MultiReplica_StableServerIDs, MissingServerID_FailsClosed,
EpochFencing_IntegratedPath, RebuildAssignment, ReplicaAssignment
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CatchUpExecutor.OnStep: optional callback fired between executor-managed
progress steps. Enables deterministic fault injection (epoch bump)
between steps without racing or manual sender calls.
E2_EpochBump_MidExecutorLoop:
- Executor runs 5 progress steps
- OnStep hook bumps epoch after step 1 (after 2 successful steps)
- Executor's own loop detects invalidation at step 2's check
- Resources released by executor's release path (not manual cancel)
- Log shows session_invalidated + exec_resources_released
This closes the remaining FC2 gap: invalidation is now detected
and cleaned up by the executor itself, not by external code.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Planner/executor contract:
- RebuildExecutor.Execute() takes no arguments — consumes plan-bound
RebuildSource, RebuildSnapshotLSN, RebuildTargetLSN
- RecoveryPlan binds all rebuild targets at plan time
- Executor cannot re-derive policy from caller-supplied history
Catch-up timing:
- Removed unused completeTick parameter from CatchUpExecutor.Execute
- Per-step ticks synthesized as startTick + stepIndex + 1
- API shape matches implementation
New test: PlanExecuteConsistency_RebuildCannotSwitchSource
- Plans snapshot+tail, then mutates storage history
- Executor succeeds using plan-bound values (not re-derived)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Full-base rebuild resource:
- StorageAdapter.PinFullBase/ReleaseFullBase for full-extent base image
- PlanRebuild full_base branch now acquires FullBasePin
- RecoveryPlan.FullBasePin field, released by ReleasePlan
Session cleanup on resource failure:
- PlanRecovery invalidates session when WAL pin fails
(no dangling live session after failed resource acquisition)
3 new tests:
- PlanRebuild_FullBase_PinsBaseImage: pin acquired + released
- PlanRebuild_FullBase_PinFailure: logged + error
- PlanRecovery_WALPinFailure_CleansUpSession: session invalidated,
sender disconnected (no dangling state)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ProcessAssignment now compares pre/post endpoint state before
logging session_invalidated with "endpoint_changed" reason.
Normal session supersede (same endpoint, assignment_intent) no
longer mislabeled as endpoint change.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Zero-gap completion:
- ExecuteRecovery auto-completes zero-gap sessions (no sender call needed)
- RecoveryResult.FinalState = StateInSync for zero-gap
Epoch transition:
- UpdateSenderEpoch: orchestrator-owned epoch advancement with auto-log
- InvalidateEpoch: per-replica session_invalidated events (not aggregate)
Endpoint-change invalidation:
- ProcessAssignment detects session ID change from endpoint update
- Logs per-replica session_invalidated with "endpoint_changed" reason
All integration tests now use orchestrator exclusively for core lifecycle.
No direct sender API calls for recovery execution in integration tests.
1 new test: EndpointChange_LogsInvalidation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
RecordHandshakeFromHistory and SelectRebuildFromHistory now
return an error instead of panicking on nil history input.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New file: history.go — RetainedHistory connects recovery decisions
to actual WAL retention state:
- IsRecoverable: checks gap against tail/head boundaries
- MakeHandshakeResult: generates HandshakeResult from retention state
- RebuildSourceDecision: chooses snapshot+tail vs full base from
checkpoint state (trusted vs untrusted)
- ProveRecoverability: generates explicit proof explaining why
recovery is or is not allowed
14 new tests (recoverability_test.go):
- Recoverable/unrecoverable gap (exact boundary, beyond head)
- Trusted/untrusted/no checkpoint → rebuild source selection
- Handshake from retained history → outcome classification
- Recoverability proofs (zero-gap, ahead, within retention, beyond)
- E2E: two replicas driven by retained history (catch-up + rebuild)
- Truncation required for replica ahead of committed
Engine module at 44 tests (12 + 18 + 14).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Entry counting:
- Session.setRange now initializes recoveredTo = startLSN
- RecordCatchUpProgress delta counts only actual catch-up work
(recoveredTo - startLSN), not the replica's pre-existing prefix
Rebuild transfer gate:
- BeginTailReplay requires TransferredTo >= SnapshotLSN
- Prevents tail replay on incomplete base transfer
3 new regression tests:
- BudgetEntries_NonZeroStart_CountsOnlyDelta (30 entries within 50 budget)
- BudgetEntries_NonZeroStart_ExceedsBudget (30 entries exceeds 20 budget)
- Rebuild_PartialTransfer_BlocksTailReplay
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Registry is now keyed by stable ReplicaID, not by address.
DataAddr changes preserve sender identity — the core V2 invariant.
Changes:
- ReplicaAssignment{ReplicaID, Endpoint} replaces map[string]Endpoint
- AssignmentIntent.Replicas uses []ReplicaAssignment
- Registry.Reconcile takes []ReplicaAssignment
- Tests use stable IDs ("replica-1", "r1") independent of addresses
New test: ChangedDataAddr_PreservesSenderIdentity
- Same ReplicaID, different DataAddr (10.0.0.1 → 10.0.0.2)
- Sender pointer preserved, session invalidated, new session attached
- This is the exact V1/V1.5 regression that V2 must fix
doc.go: clarified Slice 1 core vs carried-forward files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All mutable state on Sender and Session is now unexported:
- Sender.state, .epoch, .endpoint, .session, .stopped → accessors
- Session.id, .phase, .kind, etc. → read-only accessors
- Session() replaced by SessionSnapshot() (returns disconnected copy)
- SessionID() and HasActiveSession() for common queries
- AttachSession returns (sessionID, error) not (*Session, error)
- SupersedeSession returns sessionID not *Session
Budget configuration via SessionOption:
- WithBudget(CatchUpBudget) passed to AttachSession
- No direct field mutation on session from external code
New test: Encapsulation_SnapshotIsReadOnly proves snapshot
mutation does not leak back to sender state.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
