fix: proactive shipper reconnect on rejoin (Bug 5)

After rejoin, the shipper is configured but no I/O triggers Ship(),
so the shipper stays Disconnected and the core stays at
awaiting_shipper_connected indefinitely.

Fix: observePrimaryShipperConnectivity now calls TryReconnectShippers
when ShipperConfigured=true but ShipperConnected=false. This triggers
the full reconnect protocol (dial + handshake + bounded catch-up)
proactively, bringing the replica current without waiting for I/O.

Option B approach: uses the same reconnect path as Barrier() — not a
fake write or bare dial probe. CatchUpTo(headLSN) replays any retained
WAL entries, bringing the replica fully current.

New methods:
- WALShipper.TryReconnect(): full reconnect without foreground I/O
- ShipperGroup.TryReconnectAll(): probes all disconnected shippers
- BlockVol.TryReconnectShippers(): volume-level entry point

Also fix pre-existing test expectation: engine now emits
start_recovery_task on primary assignment with replicas.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
pingqiuandClaude Opus 4.6 committed 2026-04-09 00:14:46 -07:00
1 parent 53246d2780
commit f90ccf5bfd
5 files changed
+103 -3

No files matched your search

+16
View File
@@ -1332,6 +1332,22 @@ func (bs *BlockService) observePrimaryShipperConnectivity(path string) {
return
}
connected := bs.isPrimaryShipperConnected(path)
if !connected {
// Proactive reconnect: the shipper is configured but not connected,
// and no I/O is happening to trigger Ship(). This occurs on rejoin
// paths where the primary gets a fresh assignment with replica
// addresses but no writes are pending. Without this, the shipper
// sits at Disconnected and the core stays at
// awaiting_shipper_connected indefinitely.
//
// Uses the full reconnect protocol (handshake + bounded catch-up),
// not just a dial probe. This brings the replica current if WAL
// entries are available.
_ = bs.blockStore.WithVolume(path, func(vol *blockvol.BlockVol) error {
connected = vol.TryReconnectShippers()
return nil
})
}
glog.V(0).Infof("block service: recheck shipper connectivity %s connected=%v mode=%s reason=%q",
path, connected, proj.Mode.Name, proj.Publication.Reason)
bs.observePrimaryShipperConnectivityStatus(path, connected)