Commit Graph
2 Commits
Author SHA1 Message Date
pingqiuandClaude Opus 4.6 e0116fc631 fix: three hardware blockers — WAL retention + registry race + shutdown beat
All 43 actions pass on m01/m02 hardware. Auto-failover PASS.
dd_write: 30s → 123ms. Post-failover write: 33,621 IOPS.

1. WAL retention: remove keepup retention floor (MinShippedLSN).
   WAL cannot be pinned during sustained async writes — any pin
   strategy either fills WAL (blocking writes) or over-recycles
   (breaking catch-up). Flusher recycles freely. Future LBA map
   will provide catch-up without WAL retention.
   MinShippedLSN on ShipperGroup retained as diagnostic surface.

2. Registry stale-cleanup race: add RegisteredAt grace period.
   Race: master registers volume → next VS heartbeat arrives before
   VS discovers the volume → stale cleanup deletes the entry →
   failover finds 0 entries. Fix: skip stale cleanup for entries
   registered within 30s (> 2 heartbeat intervals).
   2 new tests: grace protects new entry, old entry still cleaned.

3. Shutdown heartbeat: VS disconnect heartbeat no longer claims
   block inventory authority. Previously, the shutdown beat's
   empty inventory triggered stale cleanup, deleting the entry
   before failover could use it.

Scenario fix: recovery-baseline-failover.yaml now kills the
correct node (discovered primary, not hardcoded), connects to
the correct new primary for post-failover verification.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 22:59:46 -07:00
pingqiuandClaude Opus 4.6 39f1232fe2 feat: validation matrix closure — Rebuild Ready 12/12, Restore Ready 10/10
Close all Rebuild Ready and Restore Ready matrix gaps. V2 Ready at 10/14
(2 partial, 2 missing — honest assessment).

New tests (tester-written):
- R1: syncAck-driven trigger via protocol engine decision
- R3: stale replica restart beyond WAL → rebuild converges
- R5: connection drop mid-base → cancel → fresh rebuild converges
- R10: failover-rejoin with forced WAL recycling, strict rebuild assert
- R11: divergent replica full overwrite convergence
- R12: crash mid-rebuild → fresh session converges (not resume)
- S2: corrupt WAL entry + corrupt base block both rejected
- S5: snapshot-tail rebuild (base + WAL tail replay)
- S7: crash between base install and tail replay
- S8: snapshot under concurrent writes
- V5: rebuild complete without DurableLSN blocks publish_healthy
- V9: mixed replica health aggregate projection
- V14: negative fail-closed matrix (epoch, kind, stale)

Bug fix: StartRebuildSession now clears stale dirty map + resets WAL +
updates checkpoint AFTER safety check but BEFORE session.Start(). Fixes
stale extent data shadowing rebuild base blocks on reopened replicas.

Cleanup: remove 14 obsolete design docs (migration batches, old WAL-v2
specs, simulator goals) — all superseded by current protocol docs.

34 component tests + 8 protocol engine tests + server tests all pass.
1GB CRC validation passes in 19s.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 16:31:55 -07:00