Github-Mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-20 13:30:46 +02:00
Code Issues Packages Projects Releases Wiki Activity

G5-5C mini-plan v0.3 → v0.4: retire Option A, bind Option B per layering correction

Browse Source 
Architect re-ruling 2026-04-27: control-plane / data-plane layering.
Master must own identity / topology / address / RF-health; it must NOT
own runtime recovery scheduling. v0.2/v0.3's Option A (master
observation-driven re-emission) forces master into recovery-scheduling
territory and forces PeerSetGeneration to carry two distinct semantics
(authority version + peer-set-view version). That's the wrong shape:
master gets heavier; control-plane heartbeat cadence couples to
data-plane recovery cadence; protocol cleanliness erodes.

Bind Option B (primary-side degraded-peer probe loop) with explicit
constraints. No master protocol change.

Changes:
- §1.A rewritten: Option B bound shape (only-on-degraded, 5s interval,
  per-peer cooldown, in-flight guard, max-concurrent-probes=1, CP4B-2
  lifecycle discipline). Why-A-retired + Why-C-rejected sections.
- §1.B replaced: master protocol explicitly unchanged. v0.3's
  PeerSetRevision proto field, ObservationStore.obsRev counter, and
  UpdateReplicaSet lex-compare upgrade — all three retired.
- §1.C replaced: truth-domain matrix shows zero master-side write;
  one truth domain (primary data-control) writes; all others untouched.
- §1 Files retired master-side rows; replaced with primary-side probe
  loop infrastructure (peer.go probe entry + replication probe loop +
  flags + lifecycle/cooldown/dispatch tests + component test). Total
  ~225 prod + ~310 test, all primary-side. Zero LOC master / proto.
- §2 acceptance criteria rewritten: lifecycle correctness, cooldown +
  in-flight TOCTOU, dispatch branches, hardware GREEN. New criterion
  #7: zero diff under core/host/master/, core/authority/, proto/.
- §3 INVs replaced: drop INV-MASTER-PEER-SET-GEN-REV-MONOTONIC; add
  INV-REPL-PEER-RECOVERY-PROBE-LOOP-001, retain
  INV-REPL-PEER-RECOVERY-NO-RETRIGGER-LOOP, add
  INV-G5-5C-NO-MASTER-PROTOCOL-CHANGE (anti-creep guard).
- §6 risks rewritten around probe loop concerns: lifecycle bugs
  (CP4B-2 lessons), cooldown tuning, in-flight TOCTOU, scope-creep
  prevention via §3 INV + §2 #7 diff inspection.
- §5 forward-carry: trigger source disposition updated to Option B.
- §7 sign table records full ruling history v0.1 → v0.2 → v0.3 → v0.4
  with retire/keep markings; awaiting single-sign of v0.4.

Standing by for architect single-sign of v0.4.
This commit is contained in:
pingqiu
2026-04-27 14:59:30 -07:00
parent 900e4d0cb3
commit 4f6e5d3e6a
1 changed files with 92 additions and 122 deletions
Download Patch File Download Diff File Expand all files Collapse all files
sw-block/design/v3-phase-15-g5-5c-mini-plan.md
+92 -122
View File
@@ -1,11 +1,17 @@
# V3 Phase 15 — G5-5C (Peer Recovery Trigger After Replica Restart) Mini-Plan
**Date**: 2026-04-27 (v0.3 — revised per architect REVISE ruling on v0.2: V3 path correction + peer-set generation design + truth-domain wording)
**Status**: §1-§6 awaiting architect single-sign per `v3-batch-process.md §5` (Option A trigger source already bound; v0.3 absorbs three more REVISE items from architect's v0.2 review)
**Date**: 2026-04-27 (v0.4 — architect re-binding to Option B after deeper layering analysis; v0.2/v0.3's Option A path retired)
**Status**: §1-§6 awaiting architect single-sign per `v3-batch-process.md §5`
**Repo**: `seaweed_block` (V3) — **not** `seaweedfs` (V2)
**Owner**: sw (master observation-driven re-emit + primary-side recovery dispatch + tests); QA (m01 hardware re-run + scenario authoring)
**Owner**: sw (primary-side probe loop + recovery dispatch + tests); QA (m01 hardware re-run + scenario authoring)
**Process**: `v3-batch-process.md` compressed flow (one mini-plan, one PR, one §close)
**Predecessors**: G5-5 closed at `seaweedfs@c78116fd2` (L3 Replicated IO on hardware; #4 carried to this batch); architect bindings 2026-04-27 (G5-5 round 14 close ruling + G5-5C v0.1 + v0.2 REVISE rulings)
**Predecessors**: G5-5 closed at `seaweedfs@c78116fd2` (L3 Replicated IO on hardware; #4 carried to this batch)
**Architect bindings 2026-04-27**:
- G5-5 round 14 close ruling: #4 carries forward as real recovery-path finding
- G5-5C v0.1 ruling: define trigger source first
- G5-5C v0.2 ruling: bind Option A with A1+A2 in same batch (later retired — see below)
- G5-5C v0.3 ruling: V3 paths + PeerSetGeneration design + truth-domain wording (the design items become moot under v0.4)
- **G5-5C v0.4 ruling (control-plane / data-plane layering correction)**: retire Option A. Master must NOT take on observation-driven re-emission as a recovery scheduling mechanism — that overloads the control plane with runtime recovery cadence and forces `PeerSetGeneration` to carry two semantics (authority version + peer-set-view version). Bind **Option B (primary-side degraded-peer probe loop)** with explicit constraints (see §1.A). Master remains identity / topology / address / RF-health source; primary owns runtime recovery decisions. No master protocol change in this batch.
---
@@ -30,105 +36,70 @@ G5-5C closes that gap: define and implement the **trigger source** that re-arms
| 3 | **Component-scope test** — primary observes peer-up event after restart → shipper re-arms → engine plans catch-up → replica receives missed LSNs → barrier ack at current frontier. Tests the trigger path without the iSCSI/iptables harness. | New unit/component test in `weed/server/` (exact name + file determined at impl) |
| 4 | **Failure-mode test** — peer-up event fires but engine catch-up cannot complete (e.g., gap exceeds retention → rebuild required). Verifies the trigger correctly hands off to the rebuild path (`StartRebuildFromProbe`) instead of looping. | New unit/component test |
### §1.A Trigger source — bound
### §1.A Trigger source — bound (v0.4)
**Architect ruling 2026-04-27 (REVISE before code, v0.1 → v0.2)**: trigger source is **Option A — master observation-driven re-emission**, with **both halves of the causal chain in this batch's scope** (no split into a separate precursor batch). Options B and C are explicitly rejected for this batch.
**Architect re-ruling 2026-04-27 (v0.3 → v0.4)**: bind trigger source to **Option B — primary-side degraded-peer probe loop**, with the explicit constraints below. Options A and C are now rejected.
Rationale (architect): "Master already owns peer-set/observation truth in V3. Periodic probe (Option B) is viable but less disciplined. Transport reconnect (Option C) is rejected for this batch. Splitting Option A into a separate G5-5B precursor would add process overhead for one causal chain — keep both halves together."
**Rationale (architect, v0.4)**: control-plane / data-plane layering. Master must own *identity / topology / address / RF-health*; it must NOT own *runtime recovery scheduling*. v0.2/v0.3's Option A path forced master into recovery-scheduling territory and forced `PeerSetGeneration` to carry two distinct semantics (authority version + peer-set-view version). That is the wrong shape: master gets heavier, control-plane heartbeat cadence couples to data-plane recovery cadence, and the protocol cleanliness of "Epoch / EndpointVersion = authority line, full stop" erodes. Option B keeps master clean and puts runtime recovery decisions where they belong: on primary, next to the engine-driven primitives that already exist (T4d-4).
Option A has two parts, both required:
#### Option B — bound shape
| Part | Side | What it does |
|---|---|---|
| **A1 — Master-side observation-driven re-emission** | master | Today, master's `SubscribeAssignments` only re-emits when the publisher mints a *new* fact (e.g., `IntentRefreshEndpoint`). Observation-freshness changes alone do not trigger re-emission (sw's G5-5 round-7 finding). A1 wires master so that when a previously-stale observation for an existing slot becomes fresh again (replica restart against same `--durable-root` keeps `server_id`/slot identity, only the observation reanimates), the publisher mints a re-emission of the assignment fact for that slot. |
| **A2 — Primary-side recovery dispatch** | primary | Primary's host loop already consumes assignment-fact updates from master via `SubscribeAssignments` (G5-5A code path). On consuming a re-emitted fact for a slot whose shipper is in `ReplicaDegraded`, the host emits a "peer-reappeared" event into the recovery manager, which calls `ProbeReplica`. On `ProbeCatchUpRequired` → engine-driven catch-up via T4d-4 primitives. On `ProbeRebuildRequired` → hand off to existing `StartRebuildFromProbe`. |
| Aspect | Binding |
|---|---|
| **Loop owner** | Primary's per-volume runtime (host- or replication-layer goroutine; exact placement bound at code-start within §1 Files). One loop per primary serving the volume; no master-side participation. |
| **Scope** | Iterates ONLY over peers in `ReplicaDegraded` state. `ReplicaHealthy`, `ReplicaCatchingUp`, `ReplicaNeedsRebuild` are skipped (catching-up has its own recovery in flight; needs-rebuild dispatches via the existing rebuild path, not this loop). |
| **Cadence** | Low frequency. Initial value: **5 s** between iterations, configurable via a new flag (default 5 s). Each iteration is a single pass over the degraded set; no per-peer parallelism within the loop iteration (cap on concurrent probes prevents stampedes). |
| **Per-peer cooldown** | After a probe completes (success or fail), a per-peer cooldown of **5 s** (= one loop tick) is enforced before the same peer is probed again. Prevents retrigger storms on flapping peers. |
| **In-flight guard** | A peer with an active probe / catch-up / rebuild session is NOT re-probed by the loop. The loop reads the peer's state under its existing lock; if the state is no longer `ReplicaDegraded` at probe-dispatch time, the dispatch is skipped (TOCTOU-safe). |
| **Probe action** | Calls existing `ProbeReplica` (or V3 equivalent on the peer / executor). On `ProbeCatchUpRequired` → engine-driven catch-up via T4d-4 primitives (already wired). On `ProbeRebuildRequired` → hand off to the existing rebuild path. On any other outcome (peer still unreachable, transport error) → log + leave peer in `ReplicaDegraded`, next loop iteration retries after cooldown. |
| **Lifecycle** | Loop starts when the volume's primary role is admitted; stops on volume close / role change. Start/Stop discipline mirrors CP4B-2 lessons (BUG-CP4B2-1 stop-before-run deadlock; BUG-CP4B2-2 zero-interval panic; BUG-CP4B2-3 callback panic isolation). All three patterns explicitly covered by tests. |
| **Master interaction** | Loop does NOT consult master, NOT mutate master state, NOT re-trigger master publication. The peer's address came from the most recent `AssignmentFact` already in primary memory (existing `ReplicationVolume.peers` map); the loop just probes the existing peer handle. |
**Pros (recap)**: Authoritative source (master is the single source of peer-set truth in V3); aligns with G5-5A peer-set publication path; no new transport paths; no polling goroutine; observation freshness gating already exists in `ObservationStore.SlotFact` from G5-5A round 54.
#### What this batch deliberately does NOT change
**Cons accepted**: Recovery latency couples to master heartbeat cadence (~seconds, dominated by observation freshness window). For G5-5C #5 hardware deadline this is well within the 30s budget.
- **No master protocol change.** `AssignmentFact`, `PeerSetGeneration`, `SubscribeAssignments`, `Publisher`, `ObservationStore` all unchanged. Master continues to mint assignment facts on lineage events only.
- **No proto field add.** v0.3's proposed `PeerSetRevision` is dropped. `PeerSetGeneration = (epoch<<32)|ev` semantics preserved.
- **No `UpdateReplicaSet` semantic change.** The existing `(generation, lastAppliedGeneration)` stale-replay rule stays exactly as today.
- **No `ObservationStore` change.** `SlotFact` freshness gate (G5-5A round 54) stays as-is.
- **No engine primitive change.** T4d-4 catch-up + rebuild primitives reused unchanged.
**Options B and C — recorded for future reference**:
#### Layering note
- **Option B (periodic probe loop)** — rejected: introduces a polling goroutine and an independent timing source when the same event already flows through the master subscription. Less disciplined per V3 truth-domain split.
- **Option C (transport reconnect signal)** — rejected for this batch: `Ship()` short-circuits in `ReplicaDegraded`, so the shipper does not attempt a reconnect that could carry the signal. C in practice folds back into A or B.
| Layer | Owns in V3 (after this batch) |
|---|---|
| Master / control plane | Mint epoch; publish assignment + declared peer set + addresses; observe slot freshness; (future) report RF health (e.g., desired RF=2, current effective RF=1) — observability only, NOT recovery scheduling |
| Primary / data-control plane | Live ship; mark peers degraded; **periodically probe degraded peers (this batch)**; dispatch engine-driven catch-up / rebuild on probe outcome; durability mode policy (current write / degraded write / fail closed) |
| Replica | Receive ship + barrier; reopen durable storage on restart; resubscribe to master |
| Engine | Recovery primitives (T4d-4) — invoked by primary's recovery dispatcher |
### §1.B Peer-set generation scheme — design (architect REVISE v0.2 #2)
The `gate-degraded` symptom from G5-5 #4 closes by giving primary a runtime mechanism to probe degraded peers and dispatch the existing engine recovery — without dragging master into recovery scheduling.
**Architect REVISE ruling on v0.2**: "v0.2 says master re-emits assignment facts, but does not define how `PeerSetGeneration` changes. A re-emitted fact with the same generation can be dropped by `ReplicationVolume.UpdateReplicaSet`. The mini-plan needs an explicit design: master-maintained peer-set revision, observation revision folded into generation, or another monotonic scheme."
#### Why Option A is retired
**Today's encoding** (`core/host/master/services.go:215`):
```go
fact.PeerSetGeneration = (info.Epoch << 32) | info.EndpointVersion
```
With epoch and EV each constrained to fit in uint32 (asserted just above the encoding line). Generation is purely a function of the authority line `(epoch, ev)`. The publisher comment frames this as "monotonic across process lifetimes because Epoch/EV are durable authority facts preserved across master restart."
- **Layering**: master would have to take on observation-driven re-emission, which is recovery-scheduling work in disguise. Master's heartbeat cadence becomes the primary's recovery cadence — control plane and data plane no longer separable.
- **Protocol cost**: forces `PeerSetGeneration` to carry two semantics OR forces a new `PeerSetRevision` proto field (v0.3 Option γ). Either way, master assignment publication semantics expand.
- **Failure modes multiply**: master restart now resets recovery cadence; observation-store rev counter becomes a critical control-plane fact requiring durability or careful bootstrap. None of this is needed for runtime peer recovery.
**Stale-drop hazard** (`core/replication/volume.go:194-209`):
```go
if generation > 0 && generation <= v.lastAppliedGeneration {
v.replayedGens.Add(1)
// ...stale-replay log + return nil...
}
```
A re-emit with unchanged `(epoch, ev)` produces an identical generation → silent stale-drop in `UpdateReplicaSet`. The peer set is NOT mutated, peer state is NOT recomputed, and any "peer reappeared" signal A1 wants to deliver is lost.
#### Why Option C is rejected
**Design space — three options the architect named, mapped to V3 mechanics:**
`Ship()` short-circuits in `ReplicaDegraded`, so the shipper does not attempt a reconnect that could carry a transport-reconnect signal. C requires an out-of-band reconnect attempt, which is what Option B's probe loop already does — C reduces to B.
| Option | Wire shape | Master-side state | Cross-restart monotonicity | Aliasing risk |
|---|---|---|---|---|
| **α — Master-maintained per-volume monotonic counter** | Unchanged (single `PeerSetGeneration` uint64) | New durable (or quasi-durable) per-volume rev counter; bumps on every emission cause | Requires master to persist the counter (new authority-store entry), or accept reset on master restart with a forced subscriber bootstrap | None |
| **β — Observation revision folded into existing generation packing** | Unchanged (single uint64) | `obsRev` per (volume, replica) maintained by `ObservationStore`; packed as `(epoch << 48) \| (ev << 32) \| obsRev`, narrowing epoch + ev to 16 bits each | Preserved if epoch/ev fit in 16 bits | High — current code asserts epoch + ev fit in uint32; narrowing to uint16 is an invariant change |
| **γ — Add a separate `PeerSetRevision` field alongside `PeerSetGeneration`** | Wire change (additive proto field) — `(generation, revision)` compared lexicographically | `obsRev` per slot in `ObservationStore` (G5-5A round 54 already gates freshness; extend to count fresh-stale-fresh transitions) | Preserved — `PeerSetGeneration` keeps `(epoch<<32)\|ev` semantics; `PeerSetRevision` orders re-emits within a fixed lineage | None within the wire shape; lex compare in `UpdateReplicaSet` |
### §1.B Master protocol — explicitly unchanged
**sw recommendation: Option γ** — for these reasons:
1. **Discipline-coherent** — `PeerSetGeneration` continues to mean "the lineage stamp" (semantically aligned with `(epoch, ev)`); `PeerSetRevision` newly means "the re-emit ordinal within this lineage". Two distinct trust signals carried by two distinct fields.
2. **No bit-arithmetic gymnastics** — Option β narrows epoch/ev which violates an existing uint32 invariant; not worth the wire saving.
3. **No new master-side durable state** — unlike α, no need to add an authority-store entry just for the counter; `obsRev` lives where freshness already lives (`ObservationStore`).
4. **Backward-additive proto change** — V3-only, single field, default-zero is a safe "no revision yet" sentinel; existing tests with `PeerSetGeneration: N` continue to compile and pass with `PeerSetRevision: 0`.
5. **`UpdateReplicaSet` change is mechanical** — replace the `generation <= lastAppliedGeneration` stale-drop check with `(generation, revision) <= (lastAppliedGeneration, lastAppliedRevision)` lex compare. Existing replay-counter forensics extend cleanly.
v0.3 proposed a `PeerSetRevision` proto field, an `ObservationStore.obsRev` counter, and a lex-compare upgrade to `UpdateReplicaSet`. **All three are retired in v0.4.** Master assignment publication semantics, `PeerSetGeneration = (epoch<<32)|ev`, and the existing stale-replay rule in `UpdateReplicaSet` (`core/replication/volume.go:194-209`) are preserved exactly as-is.
**Reject α**: master restart problem and new durable state both add scope outside G5-5C's stated minimal-change posture.
**Reject β**: narrowing epoch + ev to 16 bits silently weakens an existing invariant; one uint32 epoch overflow in any future cluster year erases the option's benefit. Do not retire a stronger invariant for a weaker one.
The stale-drop hazard that motivated v0.3's design is no longer relevant: under Option B, master is **not** asked to re-emit on observation freshness changes. There is no in-flight assignment fact that needs to bypass the existing stale-replay guard.
**Wire change (γ) — proto field**:
```proto
// core/rpc/proto/control.proto — AssignmentFact message
message AssignmentFact {
// ... existing fields ...
uint64 peer_set_generation = N; // existing — (epoch<<32)|ev lineage stamp
uint64 peer_set_revision = N+1; // NEW — re-emit ordinal within lineage; 0 = unrevised
}
```
### §1.C Truth-domain check (v0.4)
**Generation rule update** (`UpdateReplicaSet`):
```
Compare (incoming_gen, incoming_rev) vs (lastApplied_gen, lastApplied_rev):
- lex-greater → apply + advance both
- lex-equal-or-less → stale-replay (existing log + replayedGens counter)
- generation == 0 → unversioned apply (existing semantics, unchanged;
rev is ignored when gen==0)
```
| Truth domain | This batch (v0.4) |
|---|---|
| Master / control plane (epoch, EV, declared peer set, addresses, observation freshness) | **No change.** Master code and protocol untouched. |
| Primary / data-control plane (shipper state, recovery decisions, probe scheduling) | **Write** — new probe loop, peer-state-driven dispatch into existing engine recovery primitives. |
| Replica durable storage + acks | **No change.** Replica already reopens correctly post-restart (G5-5 #4 confirmed). |
| Engine recovery primitives (T4d-4) | **No change.** Reused as-is. |
**A1 emission trigger**:
- On lineage change (`(epoch, ev)` bump) → `PeerSetRevision = 0` (new lineage; rev resets)
- On observation freshness transition (stale → fresh) within unchanged lineage → `PeerSetRevision = obsRev_for_slot++` (or, for multi-slot facts, max across the slots that transitioned)
- `obsRev` lives in `ObservationStore` per (volume, replica), incremented on the fresh-stale-fresh transition (gated by `FreshnessConfig` from G5-5A round 54).
**Open architect choice within γ**: per-slot vs per-volume rev. sw proposes **per-volume max** — simpler, and re-emission is a volume-fact event regardless of which slot transitioned. Architect may bind otherwise at single-sign.
### §1.C Truth-domain wording correction (architect REVISE v0.2 #3)
**Architect REVISE ruling on v0.2**: "Adjust the truth-domain line: A1 is not just a 'read' of master truth if it causes subscription re-emission and peer-set revision movement. It can remain authority-safe, but document it as master observation-driven publication/re-emission, not pure read."
**Corrected truth-domain check** (replaces v0.2's §1 closing line):
| Truth domain | A1 (master) | A2 (primary) |
|---|---|---|
| Master peer-set + observation freshness | **Publication / re-emission** — A1 reads observation freshness from `ObservationStore` AND publishes a fact re-emission via the publisher; `PeerSetRevision` advances; existing `(epoch, ev)`-derived `PeerSetGeneration` does not change unless lineage itself changed | (no master writes from primary side) |
| Primary shipper state + recovery decisions | (no primary writes from master side) | **Write** — recovery manager dispatches, shipper state advances via existing transitions |
| Replica durable storage + acks | (no change) | (no change — replica already reopens correctly post-restart; G5-5 #4 confirmed restart side works) |
| Engine | (no change) | (no change — T4d-4 primitives reused as-is) |
A1 is **authority-safe**: it does not invent a new lineage, it does not advance `(epoch, ev)`, it does not change which replica is authoritative. It re-emits a fact that already represents authoritative truth, with a fresh `PeerSetRevision` so subscribers re-apply.
This is the cleanest possible truth-domain shape: one truth domain (primary data-control) writes, all others are read-only or untouched.
### What G5-5C does NOT deliver (explicit non-claims)
@@ -140,23 +111,19 @@ A1 is **authority-safe**: it does not invent a new lineage, it does not advance
### Files (preliminary — exact set bound at code-start)
All paths are in **`seaweed_block` (V3)**. The v0.2 file map mistakenly listed V2 `weed/server/...` and `weed/storage/blockvol/...` paths; v0.3 corrects this per architect REVISE v0.2 #1.
All paths are in **`seaweed_block` (V3)**. v0.4 retires v0.3's master-side rows entirely (no master code change) and replaces them with primary-side probe loop infrastructure.
| File | Side | Likely change | LOC est |
|---|---|---|---|
| `core/rpc/proto/control.proto` | wire (A1) | Add `uint64 peer_set_revision` to `AssignmentFact` (§1.B Option γ). Regenerate `core/rpc/control/control.pb.go` + `control_grpc.pb.go`. | ~5 (proto) + generated |
| `core/host/master/services.go` | master (A1) | In `SubscribeAssignments`, on observation freshness transition (stale → fresh) for a slot in the volume's topology, mint a re-emission of the current assignment fact with bumped `PeerSetRevision`. Reuse `Publisher.LastPublished` + `ObservationStore.SlotFact` (G5-5A round 54). | ~80 |
| `core/authority/observation_store.go` | master (A1) | Track per-slot `obsRev` (uint64) — increment on fresh-stale-fresh transition, expose `SlotObsRev(volumeID, replicaID)` and a freshness-transition callback (or polling comparison helper used by services.go). | ~40 |
| `core/replication/volume.go` | primary (A2) | `UpdateReplicaSet` signature: replace `generation uint64` with `(generation, revision uint64)` OR add a sibling method that accepts both. Stale-drop guard becomes lex compare on `(generation, revision)`. Existing `lastAppliedGeneration` joined by `lastAppliedRevision`. Existing `replayedGens` counter still applies; consider `replayedRevs` counter for forensics. | ~30 |
| `core/host/volume/host.go` | primary (A2) | In `applyFact`, decode `PeerSetRevision` from the fact (via `decodeReplicaTargets` or a parallel helper), pass to `UpdateReplicaSet`. After `UpdateReplicaSet` returns nil-with-mutation, detect peers that transitioned out of `ReplicaDegraded` (or were degraded at entry and now have a fresh reapply) and dispatch the peer-reappeared event into the recovery dispatcher. | ~60 |
| `core/replication/peer.go` | primary (A2) | Add an entry point on `ReplicaPeer` that the host's peer-reappeared handler calls — e.g., `OnReappeared(...)` — which probes the peer and dispatches to engine-driven catch-up via existing T4d-4 primitives, or hands off to rebuild on `ProbeRebuildRequired`. Includes per-peer cooldown (§6 risk #1). | ~70 |
| `core/host/volume/apply_fact_test.go` | primary tests | New test cases for revision-driven reapply + peer-reappeared dispatch. | ~70 |
| `core/host/master/services_test.go` (existing or new) | master tests | New test cases for observation-freshness-driven re-emission with bumped `PeerSetRevision`. | ~60 |
| `core/replication/volume_test.go` | replication tests | New cases for `(generation, revision)` lex compare in `UpdateReplicaSet`. | ~40 |
| `core/replication/component/...` (one of the existing component-test files) | end-to-end tests | Component test #4 — failure-mode rebuild hand-off without re-trigger loop. | ~50 |
| `sw-block/design/v3-phase-15-g5-5c-mini-plan.md` (this doc) | — | §close appended at batch close. | + §close |
| File | Likely change | LOC est |
|---|---|---|
| `core/replication/peer.go` | New per-peer probe entry point (e.g., `ProbeIfDegraded()`) — checks state, returns early if not `ReplicaDegraded` or in cooldown, otherwise probes via existing transport / executor path; on `ProbeCatchUpRequired` dispatches to engine catch-up (T4d-4); on `ProbeRebuildRequired` hands off to existing rebuild path. Per-peer cooldown timestamp + in-flight guard. | ~90 |
| `core/replication/volume.go` (or a new sibling, e.g., `core/replication/probe_loop.go`) | New `degradedProbeLoop` goroutine started by `ReplicationVolume` lifecycle. Iterates over `peers` map under existing lock discipline, calls each peer's `ProbeIfDegraded()`. Configurable interval (default 5 s) and max-concurrent-probes (default 1). Start/Stop methods integrated with `ReplicationVolume.Close`. | ~120 |
| `cmd/blockvolume/main.go` | New flags: `--degraded-probe-interval` (default 5 s; 0 disables), `--degraded-probe-max-concurrent` (default 1). Threaded into `ReplicationVolume` construction. | ~15 |
| `core/replication/probe_loop_test.go` (new) | Loop lifecycle tests — start before run, stop while running, zero-interval guard (CP4B-2 lessons); cooldown enforcement; in-flight guard; only-degraded-peers iteration. | ~150 |
| `core/replication/peer_state_machine_test.go` (existing) or `peer_test.go` (new) | `ProbeIfDegraded` unit tests — non-degraded skip, degraded-with-cooldown skip, dispatch to catch-up vs rebuild branches. | ~80 |
| `core/replication/component/...` (extend an existing component test file or add `g5_5c_probe_loop_test.go`) | Component test for restart-catch-up: kill peer (substrate-level), write while down, restart, observe convergence via primary's probe loop. | ~80 |
| `sw-block/design/v3-phase-15-g5-5c-mini-plan.md` (this doc) | §close appended at batch close. | + §close |
Total estimate: ~285 LOC production + ~220 LOC tests + ~5 LOC proto + generated, split master-side ~125 / replication ~30 / primary-side ~130 / tests ~220.
Total estimate: ~225 LOC production + ~310 LOC tests, all primary-side. Zero LOC master / proto / observation_store changes (deliberately).
### Architecture truth-domain check (`v3-architecture.md §4`)
@@ -172,12 +139,13 @@ Numbered, verifier-named, single source of truth.
| # | Criterion | Verifier |
|---|---|---|
| 1 | Trigger source = Option A (master observation-driven re-emission), with both A1 (master-side re-emit on observation freshness transition) and A2 (primary-side recovery dispatch on consuming the re-emitted fact) implemented in this batch. | Architect §7 sign + §1.A binding |
| 2 | A1 — Master mints a re-emission of the existing assignment fact for a slot when its observation transitions stale → fresh (slot identity unchanged). The re-emission carries the unchanged `PeerSetGeneration` and a bumped `PeerSetRevision` per §1.B Option γ. | Component test #3a (master-side; `core/host/master/services_test.go` — exact test name pinned at code-start) |
| 3 | A2 — On consuming a re-emitted assignment fact whose `(PeerSetGeneration, PeerSetRevision)` is lex-greater than `(lastAppliedGeneration, lastAppliedRevision)`, `UpdateReplicaSet` re-applies, the host detects peers that were in `ReplicaDegraded`, and the recovery dispatcher calls `ProbeReplica` and dispatches to engine-driven catch-up (T4d-4 primitives) when outcome is `ProbeCatchUpRequired`. | Component test #3b (primary-side; `core/host/volume/apply_fact_test.go` + `core/replication/volume_test.go` — exact test names pinned at code-start) |
| 4 | When the gap exceeds retention and probe outcome is `ProbeRebuildRequired`, recovery manager hands off to `StartRebuildFromProbe` (already wired) without entering a re-trigger loop. Per-peer cooldown prevents flapping. | Component test #4 |
| 5 | `iterate-m01-replicated-write.sh verify_restart_catchup` step turns GREEN on m01/M02 hardware: `LBA[2]=0xef` byte-equal under `m01verify` within 30s deadline (same as G5-5 #3 network-catchup; budget = 1× master heartbeat for re-emit + catch-up time + safety margin, comfortably inside 30s). | Hardware re-run; artifacts archived under `g5-test/logs/artifacts-<timestamp>/` |
| 1 | Trigger source = Option B (primary-side degraded-peer probe loop) with the constraints in §1.A: only-on-degraded, 5 s interval, per-peer cooldown, in-flight guard, max-concurrent-probes=1. No master protocol change. | Architect §7 sign + §1.A binding |
| 2 | Probe loop lifecycle is correct: starts on volume primary admit, stops on volume close / role change, no goroutine leak, no zero-interval panic, no callback panic propagation (CP4B-2 lessons). | `core/replication/probe_loop_test.go` lifecycle tests |
| 3 | Per-peer cooldown + in-flight guard hold under concurrent state changes: a peer that transitions out of `ReplicaDegraded` mid-loop is not double-dispatched; a peer in cooldown is skipped. | `peer_test.go` / `peer_state_machine_test.go` cooldown + TOCTOU tests |
| 4 | On `ProbeCatchUpRequired` the loop dispatches to engine-driven catch-up (T4d-4 primitives, unchanged). On `ProbeRebuildRequired` the loop hands off to the existing rebuild path. The loop does NOT re-fire on a peer with an active catch-up or rebuild session. | `peer_test.go` dispatch-branch tests |
| 5 | `iterate-m01-replicated-write.sh verify_restart_catchup` step turns GREEN on m01/M02 hardware: `LBA[2]=0xef` byte-equal under `m01verify` within 30 s deadline (same as G5-5 #3 network-catchup; budget = ≤1× probe interval (5 s) + catch-up time, comfortably inside 30 s). | Hardware re-run; artifacts archived under `g5-test/logs/artifacts-<timestamp>/` |
| 6 | No regression on G5-5 #1/#2/#3 — all three remain GREEN in the same hardware run. | Same script, same run |
| 7 | No master code touched. `git diff --stat` for the close PR shows zero changes under `core/host/master/`, `core/authority/`, `core/rpc/proto/`, `core/rpc/control/`. | Diff inspection at PR review |
**File + test names**: §2 #3a/#3b/#4 list (file: TBD at impl) is acceptable for v0.2 ratification per QA review; sw concretizes file path + Go test method name at code-start so QA can grep them in CI later. To be appended to this §2 as a code-start addendum (not requiring re-ratification — it's the same tests, just named).
@@ -196,9 +164,9 @@ Numbered, verifier-named, single source of truth.
| INV ID (proposed) | What it claims | Test pointer (proposed) |
|---|---|---|
| `INV-REPL-PEER-RECOVERY-TRIGGER-001` | When a peer in `ReplicaDegraded` becomes observable again (per Option A trigger), the primary's recovery dispatcher re-probes and dispatches to engine-driven catch-up or rebuild without operator intervention. | Component test #3a/#3b + hardware step #5 |
| `INV-REPL-PEER-RECOVERY-NO-RETRIGGER-LOOP` | Hand-off from catch-up trigger to rebuild path is one-way; the trigger does not re-fire while a rebuild session is active. | Component test #4 |
| `INV-MASTER-PEER-SET-GEN-REV-MONOTONIC` | `(PeerSetGeneration, PeerSetRevision)` advances lex-monotonically across all assignment-fact emissions for a (volume, replica). Re-emission on observation freshness transition bumps `PeerSetRevision` while leaving `PeerSetGeneration` (= `(epoch<<32)\|ev`) unchanged. Lineage change resets `PeerSetRevision` to 0 and bumps `PeerSetGeneration`. | `core/host/master/services_test.go` (revision bump + lineage reset) + `core/replication/volume_test.go` (lex-compare stale-drop) |
| `INV-REPL-PEER-RECOVERY-PROBE-LOOP-001` | Primary runs a per-volume background loop that iterates only over peers in `ReplicaDegraded`, at a bounded interval, with per-peer cooldown and in-flight guard. The loop dispatches probe outcomes into existing engine recovery primitives without inventing new recovery substrate. | `core/replication/probe_loop_test.go` lifecycle + cooldown + only-degraded + dispatch tests |
| `INV-REPL-PEER-RECOVERY-NO-RETRIGGER-LOOP` | The probe loop does not re-fire on a peer with an active catch-up or rebuild session. Hand-off from catch-up to rebuild is one-way (loop ignores `NeedsRebuild` peers because the existing rebuild path owns them). | `peer_test.go` dispatch-branch tests + component test |
| `INV-G5-5C-NO-MASTER-PROTOCOL-CHANGE` | G5-5C closes without modifying master assignment publication, `PeerSetGeneration` semantics, `ObservationStore`, `UpdateReplicaSet` stale-replay rule, or any control-plane protocol surface. Runtime peer recovery is a primary-only concern. | §2 #7 diff-inspection at PR review |
**Forward-carry from G5-5 §close (deferred ledger pointers)**:
- `INV-REPL-CATCHUP-FROMLSN-IS-REPLICA-FLUSHED-PLUS-1` — G5-5C hardware re-run exercises this path; ledger pointer added at G5-5C §close (per G5-5 §close binding).
@@ -219,9 +187,9 @@ INVs **rejected / deferred**:
| Carry item | Disposition in G5-5C |
|---|---|
| **Peer recovery trigger after replica restart** (architect-bound 2026-04-27) | **Primary scope of this batch** — §1 + §1.A + §2 #1-#5 directly address. |
| Reuse engine-driven primitives (T4d-4) | Binding adopted in §1 + §3; no new engine primitives introduced. |
| Define trigger source first | §1.A enumerates three options with tradeoffs; architect picks at §7 ratification. |
| **Peer recovery trigger after replica restart** (architect-bound 2026-04-27) | **Primary scope of this batch** — §1 + §1.A + §2 #1-#5 directly address via Option B (primary-side probe loop). |
| Reuse engine-driven primitives (T4d-4) | Binding adopted in §1 + §3; no new engine primitives introduced; probe loop only dispatches to existing primitives. |
| Define trigger source first | Bound at v0.4 to **Option B (primary-side probe loop)** per architect's control-plane / data-plane layering ruling 2026-04-27 — runtime peer recovery is a primary concern, not a master concern. |
| Pass criterion = G5-5 #4 hardware case | §2 #5 uses the exact existing `verify_restart_catchup` step with no harness changes. |
| Seed evidence: `seaweed_block@5c4718f` primary-fail.log | **Pinned**: `V:\share\g5-test\logs\artifacts-20260427T092858Z\primary-fail.log` (G5-5 §close evidence run, surfaces the `gate-degraded + stale-barrier-ack` pattern). No new collection required. |
| Two ledger-pointer additions (`INV-REPL-CATCHUP-FROMLSN-IS-REPLICA-FLUSHED-PLUS-1`, `INV-REPL-LSN-ORDER-FANOUT-001`) | Inscribed at G5-5C §close per §3. |
@@ -236,13 +204,15 @@ Opportunistic carry items from G5-5 §close (no specific gate, not in G5-5C scop
| Risk | Mitigation |
|---|---|
| Chosen trigger source flaps (e.g., Option A: master observation freshness margin too tight → repeated peer-reappeared events) | Add a "last-trigger-time" gate per peer (cooldown of 1 master heartbeat or 5s, whichever larger); covered by component test #4 |
| Trigger fires but probe sees the peer hasn't actually finished durable reopen → false `ProbeCatchUpRequired` | Replica already exposes Healthy via `/status`; primary's probe path uses the same lineage check that G5-5 #2 proved correct. No new mitigation needed beyond reusing the existing probe contract. |
| Gap exceeds retention mid-trigger (race between ship retention pressure and trigger arming) | Probe outcome `ProbeRebuildRequired` already handles this; #4 component test pins the hand-off |
| Probe loop polls when there's nothing to do (no degraded peers) → wasted goroutine wakeups | Bounded by 5 s interval; one cheap state check per peer; per-volume cost flat regardless of cluster size |
| Probe loop goroutine lifecycle bugs (CP4B-2 lessons: stop-before-run deadlock, zero-interval panic, callback panic propagation) | Explicit lifecycle tests as §2 #2 acceptance criterion; cover all three CP4B-2 patterns |
| Per-peer cooldown too short → retrigger storm; too long → slow convergence | Default 5 s = one loop interval; test §2 #3 covers cooldown enforcement; configurable via flag for ops tuning |
| In-flight guard race: peer transitions out of `ReplicaDegraded` between loop's state-read and dispatch | Loop re-checks state under peer's lock at dispatch site (TOCTOU-safe); test §2 #3 covers concurrent state change |
| Probe sees peer hasn't finished durable reopen → false `ProbeCatchUpRequired` | Replica `/status` exposes Healthy; primary's probe path uses the same lineage check that G5-5 #2 proved correct. No new mitigation needed. |
| Gap exceeds retention while loop is mid-iteration (race between ship retention pressure and probe arming) | Probe outcome `ProbeRebuildRequired` handles this; loop hands off to existing rebuild path; §2 #4 covers hand-off |
| Hardware re-run on m01 reveals secondary issues (residual iptables, leftover sessions, etc.) | `iterate-m01-replicated-write.sh start_cluster` already has pre-flight cleanup from G5-5 round 14; no new infra work |
| Component tests pass but hardware doesn't converge (timing/state-of-world differences) | Same closure pattern as G5-5: component tests are necessary but not sufficient; hardware GREEN is the §close gate |
| `obsRev` overflow over long-lived clusters (uint64 — practically infinite, but listed for completeness) | uint64 with rare bumps (only on freshness transitions) is effectively unbounded; no mitigation required |
| Subscriber on master restart sees `PeerSetRevision` decrease (master's `obsRev` resets to 0) | Existing master-restart semantics already invalidate subscriber state model; first post-restart fact carries `(generation = current (epoch, ev), revision = 0)`. Subscriber's `lastAppliedRevision` may be > 0 from before restart → fact appears stale in lex compare. Mitigation: subscriber bootstrap path (host's first attach) explicitly clears `lastAppliedGeneration`/`lastAppliedRevision` so first post-restart fact applies. To be made explicit in `applyFact` first-attach handling; covered by component test #3a master-restart sub-case. |
| Component tests pass but hardware doesn't converge | Same closure pattern as G5-5: component tests are necessary but not sufficient; hardware GREEN is the §close gate |
| Future scope creep: someone wants to add observation re-emission "while we're at it" | Explicit `INV-G5-5C-NO-MASTER-PROTOCOL-CHANGE` (§3) + §2 #7 diff inspection make this hard to merge silently. Future master-side observability of RF health (desired RF=N, current effective RF=M) is a different batch with a different rationale (operator-facing observability, not recovery scheduling). |
---
@@ -250,12 +220,12 @@ Opportunistic carry items from G5-5 §close (no specific gate, not in G5-5C scop
| Item | Owner | When | State |
|---|---|---|---|
| §1.A trigger source binding (Option A, A1+A2 in same batch, B/C rejected) | architect | v0.1 → v0.2 REVISE ruling | ✅ done 2026-04-27 |
| §1.B peer-set generation scheme (γ recommended; per-slot vs per-volume rev open at single-sign) | architect | v0.2 → v0.3 REVISE ruling | ⏳ awaiting single-sign |
| §1.C truth-domain wording (publication / re-emission, not pure read) | architect | v0.2 → v0.3 REVISE ruling | ✅ absorbed in v0.3 |
| §1 Files V3 path correction (`core/...` not `weed/...`) | architect | v0.2 → v0.3 REVISE ruling | ✅ absorbed in v0.3 |
| §1-§6 architect single-sign of v0.3 | architect | Before code start | ⏳ pending |
| Code (proto field add + master-side A1 re-emit + replication lex-compare + primary-side A2 dispatch + tests) | sw | After §1-§6 single-sign | ⏳ blocked on single-sign |
| §1.A v0.1 trigger options enumerated (A/B/C) | sw | v0.1 | ✅ done 2026-04-27 |
| §1.A v0.2 binding: Option A with A1+A2 | architect | v0.1 → v0.2 REVISE | ✅ then **retired** in v0.4 |
| §1 V3 path correction + §1.B PeerSetGeneration design + §1.C truth-domain wording | architect | v0.2 → v0.3 REVISE | ✅ absorbed in v0.3, **then PeerSetGeneration scope retired in v0.4** (V3 path correction kept) |
| §1.A v0.4 re-binding: **Option B** (primary-side probe loop) per layering correction; master protocol unchanged | architect | v0.3 → v0.4 REVISE | ✅ ruled 2026-04-27 |
| §1-§6 architect single-sign of **v0.4** | architect | Before code start | ⏳ pending |
| Code (primary-side probe loop + peer probe entry + lifecycle tests + component test) | sw | After §1-§6 single-sign | ⏳ blocked on single-sign |
| m01 hardware re-run of `verify_restart_catchup` (+ #1/#2/#3 regression check) | QA | After sw lands trigger + component tests | ⏳ blocked |
| §close append + close sign | sw drafts §close; QA verifies evidence; architect single-sign per `v3-batch-process.md §5` | After m01 verification | ⏳ blocked |