mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-20 13:30:46 +02:00
pingqiuandClaude Opus 4.6
d2d57851b0
Rebuild session protocol implementation for v2-rebuild-mvp-session-protocol.md. New files: - rebuild_bitmap.go: RebuildBitmap — session-scoped dense bitset for WAL-applied LBA tracking. MarkApplied on local WAL write (not receive). ShouldApplyBase returns false for WAL-covered LBAs (WAL always wins). - rebuild_session.go: RebuildSession — replica-side two-line rebuild. WAL lane (ApplyWALEntry) + base lane (ApplyBaseBlock) with bitmap conflict resolution. TryComplete requires BOTH base_complete AND wal_applied_lsn >= target_lsn. Volume-level control surface: StartRebuildSession, ApplyRebuildSessionWALEntry/BaseBlock, MarkRebuildSessionBaseComplete, TryCompleteRebuildSession, CancelRebuildSession, ActiveRebuildSession. - rebuild_mvp_test.go: 4 correctness tests — base+WAL converge, WAL-applied never overwritten by base, bitmap set on applied not received, control surface start/supersede/complete. - rebuild_transport_test.go: 2 transport-level tests — two-line with real WAL shipping, live writes during base copy with bitmap conflict. Design docs: - v2-rebuild-mvp-session-protocol.md: MVP spec with message set, apply rules, completion/failure/crash rules, test matrix - v2-sync-recovery-protocol.md: full protocol context (keepup/catchup/ rebuild unified design, primary decision logic, two-line model) - v2-session-protocol-shape.md: protocol shape overview Protocol engine (reference, not production): - sw-block/protocol/: 7-event engine with ~300 lines, 13 tests 6 rebuild tests pass, all existing component tests pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
299 lines
6.9 KiB
Markdown
299 lines
6.9 KiB
Markdown
# V2 Session Protocol Shape
|
|
|
|
Date: 2026-04-07
|
|
Status: active draft
|
|
|
|
Implementation-oriented companion:
|
|
|
|
- `v2-rebuild-mvp-session-protocol.md` — concrete rebuild MVP protocol target
|
|
|
|
## Purpose
|
|
|
|
This note fixes the current protocol direction for VS-to-VS recovery control so
|
|
the engine can eventually shrink its semantic surface instead of re-explaining
|
|
transport/runtime details through many events.
|
|
|
|
The goal is to keep one clear split:
|
|
|
|
1. `sync` asks for facts
|
|
2. the primary decides the session
|
|
3. the replica executes and reports progress
|
|
4. data transport stays separate from semantic ack
|
|
|
|
## Message Families
|
|
|
|
The preferred bounded surface is:
|
|
|
|
1. `walData`
|
|
- primary -> replica
|
|
- steady-state live WAL lane
|
|
2. `sync`
|
|
- primary -> replica
|
|
- bounded fact query
|
|
3. `syncAck`
|
|
- replica -> primary
|
|
- bounded facts only
|
|
4. `sessionControl`
|
|
- primary -> replica
|
|
- start/cancel/supersede one session contract
|
|
5. `sessionAck`
|
|
- replica -> primary
|
|
- accepted/progress/completed/failed
|
|
6. `sessionData`
|
|
- primary -> replica
|
|
- historical repair lane
|
|
7. `sessionDataAck` (optional)
|
|
- replica -> primary
|
|
- transport/window control only
|
|
|
|
## Ack Separation Rule
|
|
|
|
These meanings must stay separate:
|
|
|
|
1. transport ack
|
|
2. session ack
|
|
3. sync ack
|
|
|
|
`sessionDataAck` must never imply:
|
|
|
|
1. quorum eligibility
|
|
2. recovery completion
|
|
3. return to `keepup`
|
|
|
|
## Session Decision Rule
|
|
|
|
The primary should decide from fresh sync facts:
|
|
|
|
1. `keepup` if normal sync closure is still true
|
|
2. `catchup` if the replica is still within recoverable WAL history
|
|
3. `rebuild` if the replica is below recoverable retained history
|
|
|
|
The replica does not choose the next session kind.
|
|
|
|
## Recovery Paths
|
|
|
|
### 1. Catch-up
|
|
|
|
`catchup` is the narrow WAL-only recovery path.
|
|
|
|
Expected role:
|
|
|
|
1. network delay
|
|
2. short temporary gap
|
|
3. recoverable WAL-only replay
|
|
|
|
It should not be treated as the main recovery framework.
|
|
|
|
Catch-up uses two WAL lanes:
|
|
|
|
1. replay lane from `pin_lsn` to frozen `current_lsn1`
|
|
2. live lane beyond `current_lsn1`
|
|
|
|
No bitmap is needed because WAL is ordered by LSN.
|
|
|
|
### 2. Rebuild
|
|
|
|
`rebuild` is the formal primary recovery path.
|
|
|
|
It should behave as one integrated contract with two concurrent lanes:
|
|
|
|
1. base lane
|
|
- primary exposes a trusted snapshot/CoW view at `base_lsn`
|
|
- replica receives extent/base data from that frozen view
|
|
2. WAL lane
|
|
- replica accepts WAL from `base_lsn`
|
|
- replica applies WAL into its local recovery state while base transfer
|
|
continues
|
|
|
|
This avoids a large delayed post-snapshot catch-up that would pin old WAL too
|
|
long.
|
|
|
|
### Rebuild Variants
|
|
|
|
All rebuild variants share the same semantic contract:
|
|
|
|
1. trusted base
|
|
2. explicit target
|
|
3. live WAL lane
|
|
4. single completion boundary accepted by the primary
|
|
|
|
The data source may vary:
|
|
|
|
1. `full_copy`
|
|
- copy the full base image
|
|
2. `snapshot_or_cow`
|
|
- copy a trusted frozen snapshot/CoW view
|
|
3. `delta_blocks_since_base`
|
|
- copy only blocks known to have changed since a trusted base boundary
|
|
|
|
This is an optimization choice, not a different session truth model.
|
|
|
|
## Bitmap Rule For Rebuild
|
|
|
|
The replica maintains a bitmap of LBAs already covered by applied WAL.
|
|
|
|
The rule is:
|
|
|
|
1. WAL-applied LBA => later base-copy data for that LBA must be skipped
|
|
2. WAL-received-but-not-applied LBA => not protected by bitmap
|
|
|
|
So the bit is set on `applied`, not on `received`.
|
|
|
|
### Meaning of Applied
|
|
|
|
For this protocol, `applied` means:
|
|
|
|
1. accepted into the replica's local WAL/recovery truth
|
|
2. replayable after replica restart
|
|
|
|
It does not require the update to be flushed into the final extent image before
|
|
the bitmap may protect the LBA.
|
|
|
|
## Range Bitmap Optimization
|
|
|
|
### Purpose
|
|
|
|
A persistent range bitmap can turn some rebuilds from "copy the full base" into
|
|
"copy only blocks changed since a trusted base boundary."
|
|
|
|
This is a rebuild optimization, not a new engine-level recovery kind.
|
|
|
|
### Trusted-Base Rule
|
|
|
|
Range-bitmap optimization is only valid relative to a trusted base boundary.
|
|
|
|
Valid anchors include:
|
|
|
|
1. checkpoint/snapshot at `base_lsn`
|
|
2. previously accepted rebuild/session completion at `base_lsn`
|
|
|
|
Invalid anchor:
|
|
|
|
1. arbitrary replica-reported old `applied_lsn` with no trusted-base proof
|
|
|
|
So the optimization rule is:
|
|
|
|
1. choose trusted `base_lsn`
|
|
2. compute changed blocks for `(base_lsn, target_lsn]`
|
|
3. copy only that changed-block set as the base lane
|
|
4. keep live WAL lane running in parallel
|
|
|
|
### Data Shape
|
|
|
|
Conceptually:
|
|
|
|
1. `rangeBitmap[lsn_range] -> changed_blocks`
|
|
2. planner computes `union(changed_blocks over requested range)`
|
|
3. rebuild sends only those blocks from the trusted base image
|
|
|
|
This is similar in spirit to changed-block tracking or activity-log-assisted
|
|
resync, but it must remain anchored to one explicit trusted base point.
|
|
|
|
### Layering Rule
|
|
|
|
`rangeBitmap` belongs to:
|
|
|
|
1. rebuild planner
|
|
2. storage/checkpoint metadata
|
|
3. execution optimization
|
|
|
|
It does not belong to:
|
|
|
|
1. engine projection truth
|
|
2. session semantic ownership
|
|
3. sync-decision semantics
|
|
|
|
The engine still only needs to know:
|
|
|
|
1. session kind
|
|
2. base boundary
|
|
3. target boundary
|
|
4. progress/completion/failure
|
|
|
|
## Failure Rule
|
|
|
|
Session failure must not silently decide the next semantic state.
|
|
|
|
`SessionFailed` means only:
|
|
|
|
1. this primary-issued contract did not complete
|
|
|
|
After failure:
|
|
|
|
1. the replica reports fresh facts again
|
|
2. the primary re-decides `keepup` / `catchup` / `rebuild`
|
|
|
|
No local component may self-escalate to semantic `needs_rebuild`.
|
|
|
|
## Rebuild-Time Ack Rule
|
|
|
|
During rebuild:
|
|
|
|
1. the replica may continue applying new WAL
|
|
2. the replica must continue reporting session progress
|
|
3. the replica must not be treated as normal quorum-eligible sync success until
|
|
the rebuild contract closes
|
|
|
|
So `syncAck` during rebuild should carry:
|
|
|
|
1. current facts
|
|
2. active session state
|
|
3. not-ready-for-quorum meaning
|
|
|
|
Only after the primary accepts `SessionCompleted` may later `syncAck` regain
|
|
normal quorum semantics.
|
|
|
|
## Minimal Session Shapes
|
|
|
|
### `sessionControl`
|
|
|
|
The minimum contract should carry:
|
|
|
|
1. `session_id`
|
|
2. `epoch`
|
|
3. `replica_id`
|
|
4. `kind`
|
|
5. `base_lsn`
|
|
6. `target_lsn`
|
|
7. `deadline_ms`
|
|
|
|
For rebuild it may also carry:
|
|
|
|
1. `base_kind`
|
|
2. `snapshot_id` or `cow_view_id`
|
|
3. `reservation`
|
|
|
|
### `sessionAck`
|
|
|
|
The minimum replica response should carry:
|
|
|
|
1. `session_id`
|
|
2. `epoch`
|
|
3. `kind`
|
|
4. `phase`
|
|
5. `accepted | progress | completed | failed`
|
|
|
|
For progress reporting, the important facts are:
|
|
|
|
1. `wal_applied_lsn`
|
|
2. `base_progress`
|
|
3. `base_complete`
|
|
4. `achieved_lsn` on completion
|
|
|
|
`bitmap_coverage` may be added later if needed, but it is not required as the
|
|
first semantic surface.
|
|
|
|
## Engine Consequence
|
|
|
|
If this shape is preserved, the engine can eventually reduce its semantic
|
|
surface to a smaller set of facts:
|
|
|
|
1. assignment truth
|
|
2. sync facts and session decision
|
|
3. session progress
|
|
4. session completion
|
|
5. session failure
|
|
|
|
That reduction is only safe because transport ack, session ack, and sync ack are
|
|
kept separate at the protocol boundary.
|