Commit Graph
5 Commits
Author SHA1 Message Date
pingqiuandClaude Opus 4.6 55013e103b feat: Phase 20 Stage 0+1 closure — bootstrap + sustained workload on hardware
Stage 0 (bootstrap closure): PASS on m01/M02
  - create RF=2 sync_all → 10s shipper wait → 4k fsync → publish_healthy
  - Proves: BarrierAccepted observation, ShipperConnected, DurableLSN > 0

Stage 1 (sustained workload): 32/33 actions PASS
  - bootstrap → fio 10s randwrite → dd_write 1M×2 fsync → data checksum
  - Remaining: auto-failover promotion (separate issue)

Key fixes:
  - BarrierAccepted callback: SyncCache success → core DurableLSN update
  - BarrierRejected callback: barrier failures surface to core with reason
  - Shipper state callback for new volumes (not just startup volumes)
  - CatchUpTo ctrl conn reset: prevents stale control channel after recovery
  - CP13-6 max-bytes budget suspended: uses replicaFlushedLSN which can't
    advance without barrier; kills healthy shippers during async writes.
    Will be replaced by v2 negotiated sync/recovery protocol.
  - Barrier diagnostic logging: start/fail/success with reason and LSN
  - Scenario restructured: Stage 0 (bootstrap-closure) + Stage 1 (failover)
  - dd_write: sync_mode param + real stderr capture
  - sw-test-runner suite command: deploy once, run N scenarios
  - WAL size plumbing: proto + API + handler (forward-compatible)

Known: 6 blockvol/server test failures from Barrier() path change
(bounded catch-up in Barrier). Need test updates to match new semantics.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 19:55:12 -07:00
pingqiuandClaude Opus 4.6 4681df6b56 fix: CP13-5 — tighten reconnect proof with observable handshake evidence
Findings fixed:
1. TestAdversarial_ReconnectUsesHandshakeNotBootstrap now has 3 observable
   proof points instead of just "SyncCache succeeded":
   - new shipper HasFlushedProgress=true (seeded from old group)
   - replica receivedLSN advances during SyncCache (catch-up delivered entries)
   - shipper replicaFlushedLSN > 0 after barrier (durable progress established)
   Bootstrap alone would not advance receivedLSN — it only sends the barrier.

2. TestBug2 stale comment removed: "must NOT call SetReplicaAddr" replaced
   with accurate CP13-5 explanation that SetReplicaAddrs now preserves
   hasFlushedProgress across shipper replacement.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 22:56:20 -07:00
pingqiuandClaude Opus 4.6 4bdf6c604e fix: CP13-2 — use advertisedIP (routable), not localServerID (opaque)
Bug: setupReplicaReceiver derived the advertised host from localServerID,
which can be an opaque string (from -id flag, e.g., "my-custom-server-id").
This would publish unusable endpoints like "my-custom-server-id:14260".

Fix:
- volume_server_block.go: add advertisedIP field (always a real IP from
  -ip flag), use it instead of localServerID for replica canonicalization
- volume.go: wire *v.ip → blockService.SetAdvertisedIP() at startup
- blockvol.go: StartReplicaReceiver variadic advertisedHost unchanged

Proof (sync_all_bug_test.go TestBug3, 4 sub-cases):
- fallback: wildcard bind without advertisedHost → outbound-IP
- advertisedHost: explicit IP appears in exported addresses
- StartReplicaReceiver_API: public API forwards host correctly
- opaque_identity_not_routable: proves opaque string produces
  non-routable address, confirming production must use advertisedIP

Identity vs transport separation preserved:
- localServerID: stable identity for V2 control (may be opaque)
- advertisedIP: routable IP for transport endpoints (always real IP)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 20:51:47 -07:00
pingqiuandClaude Opus 4.6 2d47383df7 feat: CP13-2 — canonical replica addressing on production truth surface
Problem: StartReplicaReceiver didn't forward advertisedHost to
NewReplicaReceiver, so wildcard-bind listeners relied on outbound-IP
fallback for canonicalization. On multi-NIC hosts this could select
the wrong interface, leaking non-routable addresses into replication
truth.

Fix:
- blockvol.go: StartReplicaReceiver now accepts optional advertisedHost
  variadic param and forwards it to NewReplicaReceiver
- volume_server_block.go: setupReplicaReceiver extracts host from
  localServerID (the canonical VS identity) and passes it as
  advertisedHost — wildcard-bind addresses now resolve to the
  authoritative server IP, not outbound-IP fallback

Proof (sync_all_bug_test.go TestBug3, upgraded from PASS* to PASS):
- fallback: wildcard bind without advertisedHost still produces ip:port
- advertisedHost: explicit host appears in exported DataAddr/CtrlAddr
- StartReplicaReceiver_API: public API forwards advertisedHost correctly

What CP13-2 does NOT change:
- No reconnect handshake changes (CP13-5)
- No retention policy changes (CP13-6)
- No rebuild behavior changes (CP13-7)
- No barrier protocol changes (CP13-3)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 17:45:42 -07:00
Ping QiuandClaude Opus 4.6 548e47e482 feat: reconnect handshake + WAL catch-up protocol (CP13-5)
Adds the sync_all reconnect protocol: when a degraded shipper
reconnects, it performs a handshake (ResumeShipReq/Resp) to
determine the replica's durable progress, then streams missed
WAL entries to close the gap before resuming live shipping.

New wire messages:
- MsgResumeShipReq (0x03): primary sends epoch, headLSN, retainStart
- MsgResumeShipResp (0x04): replica returns status + flushedLSN
- MsgCatchupDone (0x05): marks end of catch-up stream

Decision matrix after handshake:
- R == H: already caught up → InSync
- S <= R+1 <= H: recoverable gap → CatchingUp → stream → InSync
- R+1 < S: gap exceeds retained WAL → NeedsRebuild
- R > H: impossible progress → NeedsRebuild

WALAccess interface: narrow abstraction (RetainedRange + StreamEntries)
avoids coupling shipper to raw WAL internals.

Bootstrap vs reconnect split: fresh shippers (HasFlushedProgress=false)
use CP13-4 bootstrap path. Previously-synced shippers use handshake.

Catch-up retry budget: maxCatchupRetries=3 before NeedsRebuild.

ReplicaReceiver now initializes receivedLSN/flushedLSN from volume's
nextLSN on construction (handles receiver restart on existing volume).

TestBug2_SyncAll_SyncCache_AfterDegradedShipperRecovers flips FAIL→PASS.
All previously-passing baseline tests remain green.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 15:38:06 -07:00