Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-11-11 14:28:33 +01:00
Code Issues Packages Projects Releases Wiki Activity

feat: reconnect handshake + WAL catch-up protocol (CP13-5)

Browse Source 
Adds the sync_all reconnect protocol: when a degraded shipper
reconnects, it performs a handshake (ResumeShipReq/Resp) to
determine the replica's durable progress, then streams missed
WAL entries to close the gap before resuming live shipping.

New wire messages:
- MsgResumeShipReq (0x03): primary sends epoch, headLSN, retainStart
- MsgResumeShipResp (0x04): replica returns status + flushedLSN
- MsgCatchupDone (0x05): marks end of catch-up stream

Decision matrix after handshake:
- R == H: already caught up → InSync
- S <= R+1 <= H: recoverable gap → CatchingUp → stream → InSync
- R+1 < S: gap exceeds retained WAL → NeedsRebuild
- R > H: impossible progress → NeedsRebuild

WALAccess interface: narrow abstraction (RetainedRange + StreamEntries)
avoids coupling shipper to raw WAL internals.

Bootstrap vs reconnect split: fresh shippers (HasFlushedProgress=false)
use CP13-4 bootstrap path. Previously-synced shippers use handshake.

Catch-up retry budget: maxCatchupRetries=3 before NeedsRebuild.

ReplicaReceiver now initializes receivedLSN/flushedLSN from volume's
nextLSN on construction (handles receiver restart on existing volume).

TestBug2_SyncAll_SyncCache_AfterDegradedShipperRecovers flips FAIL→PASS.
All previously-passing baseline tests remain green.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ping QiuandClaude Opus 4.6 committed 2026-03-25 15:38:06 -07:00
1 parent 8d6379f841
commit 548e47e482
10 files changed
+774 -66
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

weed/storage/blockvol/dist_group_commit_test.go
+5 -5
View File
@@ -69,7 +69,7 @@ func TestDistSync_SyncAll_AllDegraded_Fails(t *testing.T) {
// Create a shipper group with one degraded shipper.
shipper := NewWALShipper("127.0.0.1:99999", "127.0.0.1:99998", func() uint64 {
return vol.epoch.Load()
}, vol.Metrics)
}, nil, vol.Metrics)
shipper.state.Store(uint32(ReplicaDegraded))
group := NewShipperGroup([]*WALShipper{shipper})
@@ -84,8 +84,8 @@ func TestDistSync_SyncQuorum_AllDegraded_RF3_Fails(t *testing.T) {
vol, _ := newTestVolWithMode(t, DurabilitySyncQuorum)
defer vol.Close()
s1 := NewWALShipper("127.0.0.1:99999", "127.0.0.1:99998", func() uint64 { return 0 }, vol.Metrics)
s2 := NewWALShipper("127.0.0.1:99997", "127.0.0.1:99996", func() uint64 { return 0 }, vol.Metrics)
s1 := NewWALShipper("127.0.0.1:99999", "127.0.0.1:99998", func() uint64 { return 0 }, nil, vol.Metrics)
s2 := NewWALShipper("127.0.0.1:99997", "127.0.0.1:99996", func() uint64 { return 0 }, nil, vol.Metrics)
s1.state.Store(uint32(ReplicaDegraded))
s2.state.Store(uint32(ReplicaDegraded))
group := NewShipperGroup([]*WALShipper{s1, s2})
@@ -102,7 +102,7 @@ func TestDistSync_BestEffort_BackwardCompat(t *testing.T) {
vol, _ := newTestVolWithMode(t, DurabilityBestEffort)
defer vol.Close()
s1 := NewWALShipper("127.0.0.1:99999", "127.0.0.1:99998", func() uint64 { return 0 }, vol.Metrics)
s1 := NewWALShipper("127.0.0.1:99999", "127.0.0.1:99998", func() uint64 { return 0 }, nil, vol.Metrics)
s1.state.Store(uint32(ReplicaDegraded))
group := NewShipperGroup([]*WALShipper{s1})
@@ -116,7 +116,7 @@ func TestDistSync_Metrics_IncrementOnFailure(t *testing.T) {
vol, _ := newTestVolWithMode(t, DurabilitySyncAll)
defer vol.Close()
s1 := NewWALShipper("127.0.0.1:99999", "127.0.0.1:99998", func() uint64 { return 0 }, vol.Metrics)
s1 := NewWALShipper("127.0.0.1:99999", "127.0.0.1:99998", func() uint64 { return 0 }, nil, vol.Metrics)
s1.state.Store(uint32(ReplicaDegraded))
group := NewShipperGroup([]*WALShipper{s1})