mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-17 09:17:39 +01:00
Engine execution layer for V2 replication protocol: - RebuildInstaller: full state handoff (dirty map, WAL, superblock, flusher) - TruncateToLSN: exact safety predicate (checkpointLSN == truncateLSN), ErrTruncationUnsafe escalation to NeedsRebuild - SyncReceiverProgress: unconditional Store for post-rebuild alignment - V2StatusSnapshot: CommittedLSN = nextLSN-1 for sync_all V2 bridge real I/O executors: - TransferFullBase: TCP streaming + RebuildInstaller + second catch-up - TransferSnapshot: SHA-256 verified streaming to disk - TruncateWAL: ErrTruncationUnsafe detection + escalation - StreamWALEntries: rebuild-mode TCP apply Engine executor interfaces: - CatchUpIO.TruncateWAL, RebuildIO.TransferFullBase returns achievedLSN - CatchUpExecutor truncation-only skip, NeedsRebuild escalation - RebuildExecutor uses achievedLSN for progress tracking Design docs reorganized: superseded planning docs removed, protocol truths and closure map added. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
6.1 KiB
6.1 KiB
Phase 04
Date: 2026-03-27
Status: complete
Purpose: start the first standalone V2 implementation slice under sw-block/, centered on per-replica sender ownership and explicit recovery-session ownership
Goal
Build the first real V2 implementation slice without destabilizing V1.
This slice should prove:
- per-replica sender identity
- explicit one-session-per-replica recovery ownership
- endpoint/assignment-driven recovery updates
- clean handoff between normal sender and recovery session
Why This Phase Exists
The simulator and design work are now strong enough to support a narrow implementation slice.
We should not start with:
- Smart WAL
- new storage engine
- frontend integration
We should start with the ownership problem that most clearly separates V2 from V1.5.
Source Of Truth
Design:
sw-block/docs/archive/design/v2-first-slice-session-ownership.mdsw-block/design/v2-acceptance-criteria.mdsw-block/design/v2-open-questions.md
Simulator reference:
sw-block/prototype/distsim/
Scope
In scope
- per-replica sender owner object
- explicit recovery session object
- session lifecycle rules
- endpoint update handling
- basic tests for sender/session ownership
Out of scope
- Smart WAL in production code
- real block backend redesign
- V1 integration
- frontend publication
Assigned Tasks For sw
P0
- create standalone V2 implementation area under
sw-block/
- recommended:
sw-block/prototype/enginev2/
- define sender/session types
- sender owner per replica
- recovery session per replica per epoch
- implement basic lifecycle
- create sender
- attach session
- supersede stale session
- close session on success / invalidation
Current Progress
Delivered in this phase so far:
- standalone V2 area created under:
sw-block/prototype/enginev2/
- core types added:
SenderRecoverySessionSenderGroup
- sender/session lifecycle shell implemented
- per-replica ownership implemented
- endpoint-change invalidation implemented
- sender epoch coherence implemented
- session epoch attach validation implemented
- session phase transitions now enforce a real transition map
- session identity fencing implemented
- stale completion rejected by session ID
- execution APIs implemented:
BeginConnectRecordHandshakeRecordHandshakeWithOutcomeBeginCatchUpRecordCatchUpProgressCompleteSessionByID
- completion authority tightened:
- catch-up must converge
- zero-gap handshake fast path allowed
- attach/supersede now establish ownership only
- sender-group orchestration tests added
- recovery outcome branching implemented:
OutcomeZeroGapOutcomeCatchUpOutcomeNeedsRebuild
- assignment-intent orchestration implemented:
- reconcile + recovery target session creation
- stale assignment epoch rejected
- created/superseded/failed outcomes distinguished
- P2 data-boundary correction accepted:
- zero-gap now requires exact equality to committed boundary
- replica-ahead is not zero-gap
- minimal historical-data prototype implemented:
WALHistory- retained-prefix / recycled-range semantics
- executable recoverability proof
- base snapshot for historical state after tail advance
- explicit safe-boundary handling implemented:
- divergent tail requires truncation before
InSync - truncation recorded via sender-owned execution API
- divergent tail requires truncation before
- WAL-backed prototype tests added:
- catch-up recovery with data verification
- rebuild fallback with proof of unrecoverability
- truncate-then-
InSyncwith committed-boundary verification
- current
enginev2test state at latest review: -
- 95 tests passing
- prototype scenario closure completed:
- acceptance criteria mapped to prototype evidence
- V2-boundary scenarios expressed against
enginev2 - small end-to-end prototype harness added
Next phase:
Phase 4.5- bounded
CatchUp - first-class
Rebuild - crash-consistency / recoverability / liveness proof hardening
- bounded
- do not integrate into V1 production tree yet
P1
- implement endpoint update handling
- changed-address update must refresh the right sender owner
- implement epoch invalidation
- stale session must stop after epoch bump
- add tests matching the slice acceptance
P2
- add recovery outcome branching
- distinguish:
- zero-gap fast completion
- positive-gap catch-up completion
- unrecoverable gap /
NeedsRebuild
- add assignment-intent driven orchestration
- move beyond raw reconcile-only tests
- make sender-group react to explicit recovery intent
- add prototype-level end-to-end flow tests
- assignment/update
- session creation
- execution
- completion / invalidation
- rebuild escalation
P3
- add minimal historical-data prototype
- retained prefix/window
- minimal recoverability state
- explicit "why catch-up is allowed" proof
- make safe-boundary data handling explicit
- divergent tail cleanup / truncate rule
- or equivalent explicit boundary handling before
InSync
- strengthen recoverability/rebuild tests
- executable proof of:
- recoverable gap
- unrecoverable gap
- rebuild fallback boundary
P4
- close prototype scenario coverage
- map key acceptance criteria onto
enginev2scenarios/tests - make prototype evidence reviewable scenario-by-scenario
- express the 4 V2-boundary cases against the prototype
- changed-address identity-preserving recovery
NeedsRebuildpersistence- catch-up without overwriting safe data
- repeated disconnect/reconnect cycles
- add one small prototype harness if needed
- enough to show assignment -> recovery -> outcome flow end-to-end
- no product/backend integration yet
Exit Criteria
Phase 04 is done when:
- standalone V2 sender/session slice exists under
sw-block/ - sender ownership is per replica, not set-global
- one active recovery session per replica per epoch is enforced
- endpoint update and epoch invalidation are tested
- sender-owned execution flow is validated
- recovery outcome branching exists at prototype level
- minimal historical-data / recoverability model exists at prototype level
- prototype scenario closure is achieved for key V2 acceptance cases