Per repository policy: dev/design docs live in
seaweedfs/sw-block/design/, not in seaweed_block/docs/. Formal
product docs come later. This commit relocates the 7 recovery
design markdown docs (4 trunk-merged in seaweed_block phase-15;
3 in-flight on g7-redo branches) plus the 1 hardware canonical
YAML to sw-block/design/ with v3-recovery-* prefix to match the
existing naming pattern (v3-recovery-live-line-backlog-spec.md).
Companion cleanup: a follow-on PR on seaweed_block removes the
docs from docs/ (and the YAML from testrunner/scenarios/) — that
PR is the seaweed_block side of the relocation.
Files added:
v3-recovery-pin-floor-wire.md — was docs/recovery-pin-floor-wire.md
on seaweed_block phase-15 (PR #11+#16)
v3-recovery-wiring-plan.md — was docs/recovery-wiring-plan.md
(PR #13)
v3-recovery-execution-institution.md — was docs/recovery-execution-institution.md
v3-recovery-inv-test-map.md — was docs/recovery-inv-test-map.md
(PR #11/#14/#15)
v3-recovery-unified-wal-stream-kickoff.md — was docs/recovery-unified-wal-stream-kickoff.md
g7-redo/unified-wal-kickoff (v0.3)
v3-recovery-unified-wal-stream-mini-plan.md — was docs/recovery-unified-wal-stream-mini-plan.md
g7-redo/unified-wal-mini-plan (v0.2)
v3-recovery-dual-lane-canonical-runbook.md — was docs/recovery-dual-lane-canonical-runbook.md
g7-redo/hardware-canonical-paper
v3-recovery-dual-lane-canonical.yaml — was testrunner/scenarios/recovery-dual-lane-canonical.yaml
g7-redo/hardware-canonical-paper
Internal cross-references updated in-place via sed:
- docs/recovery-inv-test-map.md → v3-recovery-inv-test-map.md
- docs/recovery-pin-floor-wire.md → v3-recovery-pin-floor-wire.md
- docs/recovery-wiring-plan.md → v3-recovery-wiring-plan.md
- testrunner/scenarios/recovery-dual-lane-canonical.yaml →
v3-recovery-dual-lane-canonical.yaml
Hand-edits:
- runbook §1 companion-YAML link: was
"../v3-recovery-dual-lane-canonical.yaml" (parent dir from
seaweed_block/docs); now same-directory link in design/.
- runbook §8 §3.2 #3 reference: was relative to seaweed_block
memory file (../../.claude/...); rewritten to point to
v3-recovery-unified-wal-stream-kickoff.md §4 directly.
- mini-plan Q15: docs/archive/ wording updated to
sw-block/design/archive/.
Stages-of-evidence still readable from the docs themselves
(kickoff §11, mini-plan §10 resolution logs, inv-test-map row
versions). Original seaweed_block branches preserve git
history for the in-flight content; the cleanup PR closes them
once this lands.
NOTE: this commit does NOT include the user's unrelated
ongoing edits in feature/sw-block (M v3-batch-process.md,
M v3-dev-roadmap.md, M v3-phase-15-g6-mini-plan.md, etc.).
Those stay uncommitted for the user to handle separately.
QA's G7 pre-work surfaced a discrepancy: the v0.1 §harness-notes
pointed at `exec_rebuild_started` / `exec_rebuild_completed` as
the harness markers. Those are RecoveryLog event names (internal
Orchestrator.Log ring buffer, process-local) — NOT visible in
primary.log on hardware. Hardware harnesses can't scrape them
without a /recovery-log HTTP surface (G5-3 forward-carry).
Hardware-visible markers (corrected):
- START: `executor: rebuild start replica=<id> sessionID=<n>
epoch=<n> EV=<n> targetLSN=<n>`
from core/transport/rebuild_sender.go:41
(added at G6 #1, seaweed_block@85475cd)
- COMPLETE: `executor: rebuild complete, sent <n> blocks
(targetLSN=<n>)`
from core/transport/rebuild_sender.go:120
(pre-existing T4d-4 part B / earlier)
Both produced via log.Printf in rebuild_sender.go and routed to
the daemon's stdout/stderr stream (which iterate harness captures
to ${REMOTE_RUN_DIR}/logs/primary.log). Both are sessionID-
correlatable for chained-scenario filtering. The G6 hardware run
already proved the START marker pattern; COMPLETE follows the
same shape.
Files corrected:
- §2 #7 acceptance row (harness helper text)
- §2 entry-marker table row
- §3 risks "Ambiguous rebuild done vs peer healthy" row
- §harness-notes (full rewrite with v0.1 correction note +
marker table + RecoveryLog clarification + recommended helper
shape with sessionID filter)
Negative-references to RecoveryLog event names retained in
explanatory context (so future readers don't re-introduce the
mistake by reading the engine code in isolation).
QA pre-work artifact V:\share\g5-test\scenarios\g7-helpers.sh is
already written against the corrected literals; this commit
brings the §harness-notes source-of-truth into alignment.
Standing by for architect §1.A ratification (Q1 topology / Q2
fold-G6 / Q3 deadline / etc.) before §1.H code-start audit.
Per architect ruling 2026-04-28 + sw §close.appendix: D's WALRecycled
boundary finding is G6 territory, not a G5-5C reopener. Adding the
backlog ticket here so it doesn't get lost between G5-5C close and
G6 kickoff.
Ticket text + evidence pointer + cross-references all preserved
from the §close.appendix; this is the dev-roadmap-side mirror so
the ticket surfaces when planning G6 scope.
Standing by for architect final §close single-sign on G5-5C.
Per architect ruling 2026-04-28 on QA's expanded scenario report:
- A (capacity): 🐛 → ✅ already-fixed at seaweed_block@a250b52, INV inscribed.
- B (500 random LBAs over 65536-LBA volume): ✅ GREEN. Confidence
bump on dirty-map skew + ship order under random write pattern.
- C (kill replica mid-write-storm + restart + 200 LBAs converge):
✅ GREEN. Highest-signal recovery scenario in the expansion;
validates G5-5C peer-recovery trigger under load.
- D (5000-LBA sustained write → WALRecycled past replica LSN):
🐛 boundary finding. Architect: G6 territory, not G5-5C reopener.
Catch-up requires WAL retention; rebuild path is for gap-beyond-
WAL. Engine has dispatch-branch tests (Batch 4); runtime
escalation path under sustained pressure is G6 acceptance scope.
Doc updates:
- New §close.appendix table with all 4 scenario rows + dispositions.
- Semantic clarification on D — catch-up vs WAL recycle vs rebuild.
- §close.forward-carries gets a NEW G6 entry with backlog ticket
text, evidence pointer, cross-reference to INV-G5-5C-PROBE-BEFORE-
CATCHUP, and explicit non-reopener rationale.
- Logs + scenario script paths recorded for QA continuity.
§close substance unchanged: G5-5C gate (verify_restart_catchup
GREEN within 30 s) was met on the canonical case at
seaweed_block@712cbc47 + capacity addendum at a250b52. B/C are
strengthening, not gating; D is forward-carry.
Awaiting architect final §close single-sign on this tree.
Per architect ruling 2026-04-28 + sw addendum landing at
seaweed_block@a250b52: inscribe new INV in the ledger.
Statement: iSCSI/NVMe externally-visible volume capacity and block
size MUST derive from --durable-blocks × --durable-blocksize when
--durable-root is set, not silently fall back to frontend defaults
(DefaultVolumeBlocks=2048 × DefaultBlockSize=512 = 1 MiB). Without
this plumb-through, a daemon configured for N MiB durable storage
advertises a 1 MiB iSCSI/NVMe LUN and any workload above LBA 256
fails.
Test pointers: cmd/blockvolume/frontend_capacity_test.go (6 tests:
ProductOfBlocksAndBlockSize, RejectsZero, OverflowGuard,
IscsiHandlerCapacity, NvmeHandlerCapacity, FrontendDefaults_
StillReturn1MiB). Source-side: cmd/blockvolume/main.go::
computeFrontendVolumeSize flows into both iscsi.TargetConfig and
nvme.TargetConfig handler.
First introduced: P15 G5-5C addendum (P0 product fix).
Owner layer: host (binary, frontend wiring).
Last verified: 2026-04-28 (G5-5C addendum P0; m01 hardware re-
verification pending QA).
Status: ACTIVE.
Awaiting m01 hardware re-run for full §close ledger update.
Architect approved Option B 2026-04-27: absorb the hardware-revealed
gap into G5-5C as Batch #7 instead of carrying to G5-5D.
§1.I scope:
- core/host/volume/peer_command_executor.go (NEW, ~120 LOC)
- core/host/volume/peer_adapter_registry.go (NEW, ~100 LOC)
- core/replication/volume.go ConfigurePeerLifecycleHook (~30 LOC)
- core/host/volume/probe_loop_wiring.go router signature (~20 net)
- cmd/blockvolume/main.go registry wire-up (~20 net)
- ~10 new tests, ~250 LOC test code
INV INV-G5-5C-PER-PEER-ADAPTER-PER-PEER-ENGINE absorbed back
in-batch (was previously deferred to G5-5D in pre-architect-ruling
draft).
Pass criterion unchanged: m01 verify_restart_catchup GREEN within
30s deadline; #1-#3 regression GREEN in the same run.
§close updated: ceremony waits for Batch #7 land + hardware re-run;
G5-5C closes at full L4 in one shot.
m01 hardware run 3 at seaweed_block@ac9392d:
- #1 verify_cluster_ready ✅ GREEN
- #2 verify_byte_equal ✅ GREEN
- #3 verify_network_catchup ✅ GREEN (9s)
- #4 verify_restart_catchup ❌ RED (30s timeout)
Root cause (verified in code + log):
Primary log shows probe loop fired correctly post-restart and the
wire probe SUCCEEDED twice (R=2 S=1 H=3), but no StartCatchUp ever
dispatched. Engine apply.go:117-128 checkReplicaID drops events
whose ReplicaID doesn't match the adapter's tracked Identity —
cmd/blockvolume's host adapter tracks the PRIMARY'S OWN slot
(ReplicaID=r1), not peer r2. Probe results for r2 are correctly
dropped as wrong_replica.
Component test (Batch #6) passed because cluster.go's
WithEngineDrivenRecovery constructs c.primary.adapters[] — one
per peer. cmd/blockvolume only constructs ONE adapter for the
host's own slot. The component test exercised a different
(architecturally-correct) wiring than production has.
§1.H audit verdict was correct on engine SEMANTICS; it did not
extend to whether the production binary CONSTRUCTS per-peer engine
state. That layer was assumed; hardware revealed the assumption.
§close decision:
- G5-5C software pieces all sound, stay landed (50 unit + integ
tests PASS; full ./... regression PASS).
- Hardware finding carries to G5-5D — Per-peer adapter wiring for
primary-side recovery dispatch.
- G5-5D pass criterion = exact verify_restart_catchup case from
this run; seed evidence = sw-block/design/g5-artifacts/primary-fail.log.
- New INV to inscribe at G5-5D close:
INV-G5-5D-PER-PEER-ADAPTER-PER-PEER-ENGINE.
Doc updates:
- §close.evidence: hardware-pin row table filled with run 3 results.
- §close.deltas: 3 implicit assumptions surfaced.
- §close.findings: 2 findings (#1 per-peer adapter gap; #2 script
port-release race already fixed).
- §close.forward-carries: G5-5D added as named carry.
- architect-review-checklist: scope/audit/engine-impact/product
level all updated to reflect actual reached state (L3+, not L4).
Awaiting architect ratification of G5-5D scope at single-sign or
earlier; sw drafts G5-5D mini-plan once architect rules.
Per v3-batch-process.md §2: §close drafted as soon as software is
ready. Hardware row table left as TBD; sw fills evidence pointers
once iterate-m01-replicated-write.sh completes. Forward-carries +
deferred ledger pointers + architect-review-checklist all populated
based on G5-5C scope already in-batch.
Awaiting:
1. m01 hardware run completion → fill #1-#4 evidence rows
2. QA evidence verification → §close.deltas / findings if needed
3. architect single-sign per v3-batch-process.md §5 + §8C.2
Architect single-signed §1-§6 at seaweedfs@ba7bd0ba4 2026-04-27 with:
- Option B trigger source (primary-side degraded-peer probe loop)
- Probe loop placement = core/replication/ owned by ReplicationVolume
- Master protocol unchanged
- §1.H code-start audit gate before code
This commit:
1. Records the single-sign in the doc header.
2. Adds a §1 scope-rule one-liner near the top so future readers find
the architect-bound boundary without re-reading the v0.1→v0.5 trail:
"master owns identity/topology; primary+engine own data recovery;
the protocol aligns the two via (PeerSetGeneration, epoch,
EndpointVersion) fences."
§1.A already bound Option B in v0.4; no flip needed there. No design
change. §1.H audit is the next sw step before any production code.
Architect framing 2026-04-27: enumerate ten protocol boundary rules
and address engine-evolution question.
Engine vs primary runtime vs master split:
- Engine owns: recovery FSM, single in-flight per peer,
generation/epoch fence, probe→decision, backoff/cooldown policy,
stale-ack-cannot-promote-health rule, recovery reason / projection
- Primary runtime/adapter owns: timer / degraded-peer loop, transport
probe execution, feeding probe result into engine, executing
engine-emitted commands, ReplicationVolume / ReplicaPeer connection
lifecycle
- Master owns: identity / topology / assignment / health observation
ONLY. No runtime recovery. No epoch bumps for short up/down.
Six in-scope boundary rules (#1, #2, #3, #4, #7, #8):
- #1 Admitted Peer Rule — already INV-G5-5C-PRIMARY-RECOVERY-AUTHORITY-BOUNDED
- #2 Generation Fence — NEW INV-G5-5C-GENERATION-FENCE
- #3 Single In-Flight Per Peer — NEW INV-G5-5C-SINGLE-INFLIGHT-PER-PEER
- #4 Probe Before Catch-Up — NEW INV-G5-5C-PROBE-BEFORE-CATCHUP
- #7 Backoff/Cooldown — NEW INV-G5-5C-RECOVERY-BACKOFF (extends v0.4
fixed-5s into 5s→10s→20s→40s→60s cap, reset on success)
- #8 Stale Ack Guard — NEW INV-G5-5C-STALE-ACK-NO-HEALTH-PROMOTION
(cross-refs G5-5 round-14 gate-degraded artifact)
Three forward-carries OUT of G5-5C (per §5):
- #5 Durability Mode Explicit → G5-2 / G5-6
- #6 RF Health Reporting Separate From Recovery → future master
observability batch
- #10 Status Surface (recovery reason, effective RF, last probe) →
G5-3 metrics/backpressure
One citation (#9 Replica-side lineage check): already enforced by T4
acceptMutationLineage gate; G5-5C cites, no new code.
§1.H code-start audit gate: sw audits per-INV current owner location
BEFORE writing any code. Halt-condition: if recovery FSM is embedded
in ReplicationVolume, fence is re-derived per call site, in-flight is
implicit, or stale-ack guard is missing — sw stops and re-scopes as
engine-evolution batch instead of layering ifs in core/replication/.
Audit findings published as commit note pre-code; PR includes
audit-summary.
§2 acceptance criteria: add #13 (stale-ack guard), #14 (backoff
progression), #15 (code-start audit). Acceptance count now 15
covering 7 INVs (6 new + reconnect orthogonality from v0.4.3).
Standing by for architect single-sign of v0.4.4.
Architect framing 2026-04-27 (sharpening v0.4.2): reconnect splits
along two orthogonal dimensions — connection recovery vs identity /
lineage change. Each axis has different protocol semantics; G5-5C
must handle both correctly.
Architect's protocol judgment points:
1. PeerSetGeneration only changes for identity / address / lineage
change. Brief disconnects / restarts / freshness flapping do NOT
bump generation.
2. Primary's degraded-peer loop only acts on currently-admitted peers
(§1.E reaffirmed).
3. After reconnect, primary still probes R/S/H — reconnect alone is
not assumed sufficient.
4. If a higher PeerSetGeneration arrives during reconnect / probe,
the in-flight recovery must stop or invalidate.
Changes:
- New §1.F with two cases:
Case 1 (identity unchanged): primary retries existing peer
descriptor; new sessionID minted (sessions are session-scoped, not
peer-scoped); probe R/S/H; catch-up / rebuild as needed; no master
re-emit needed. This is G5-5C's core path.
Case 2 (identity changed): existing UpdateReplicaSet T4a-5 path
(volume.go:229-246) tears down + recreates; in-flight aborts via
Close(); new peer with new lineage takes over.
- Misread guards documented: "primary keeps retrying old address
forever" rejected by Case 2 + §1.E (c); "master must bump on every
blip" rejected by Case 1 + §1.D.
- New INV-G5-5C-RECONNECT-ORTHOGONAL-AXES in §3.
- New §2 #11 (reconnect Case 1 — identity unchanged, no re-emit) and
§2 #12 (reconnect Case 2 — lineage bump mid-flight).
This is structural reaffirmation: the V3 code already does Case 2
correctly (T4a-5 teardown). Case 1 is what the probe loop adds. The
new tests pin both axes against future drift.
Standing by for architect single-sign of v0.4.3.
Architect framing 2026-04-27 (sharpening v0.4.1): §1.D ordering-
independence must NOT be misread as "primary may self-discover and
connect to any replica it sees on the network." Tighten with a
second protocol invariant.
Rule (architect verbatim): "Primary recovery loop may retry only peers
that were previously admitted by a master-issued assignment fact for
the current authority lineage."
Layering: master establishes identity once; primary owns retry /
recovery for that admitted peer until master revokes or changes the
assignment.
This is structurally true in V3 today (probe loop reads
ReplicationVolume.peers, which UpdateReplicaSet populates from master
facts) but v0.4.2 promotes it from implementation detail to protocol
invariant so future contributors don't widen the probe surface.
Changes:
- New §1.E with three scenarios:
(a) first-time replica join — disallowed without master fact
(b) brief outage + recovery (G5-5C core case) — allowed without
master re-emit
(c) epoch / assignment change — probe must stop; in-flight aborts
- Implementation requirement made explicit: ReplicaPeer.Close() must
abort in-flight probe synchronously.
- Authority alignment surface table: replicaID/epoch/EV → identity;
AssignmentFact.Peers → only legal probe targets;
PeerSetGeneration → existing lastAppliedGeneration guard preserved.
- New INV-G5-5C-PRIMARY-RECOVERY-AUTHORITY-BOUNDED in §3.
- New §2 #9 (authority-bounded targets test) and §2 #10 (lineage-
change-during-probe test).
- §1.A bound-shape Master-interaction row references §1.E.
- §1 Files peer.go row notes Close() must abort in-flight probe.
Standing by for architect single-sign of v0.4.2.
Architect framing 2026-04-27: when a replica goes down or recovers,
both the control-plane identity/health loop and the data-plane
governance loop receive feedback. Protocol must treat them as two
independent loops with no ordering dependency, alignment via durable
identity facts (replicaID/epoch/EV/peer address), and idempotency on
primary-side dispatch absorbing duplicate triggers.
This is a sharpening of v0.4, not a re-bind. Design unchanged:
Option B primary-side probe loop, no master protocol change.
Changes:
- New §1.D: explicit two-loop table, five ordering scenarios all
ending safe, anti-requirements (master re-emit NOT prerequisite,
primary recovery NOT blocked on master), idempotency guarantees,
future RF-health observability noted as different-batch scope.
- New INV-G5-5C-TWO-LOOPS-ORDERING-INDEPENDENT in §3 with test
pointer (peer_test.go simultaneous-fire test).
- New §2 #8 acceptance criterion: unit test exercising the
"simultaneous-fire" case (concurrent fact replay + concurrent
ProbeIfDegraded on same degraded peer; idempotent absorption).
Standing by for architect single-sign of v0.4.1.
Architect REVISE ruling 2026-04-27: bind trigger source to Option A
with both halves in scope (no split into G5-5B). Reject B and C.
QA review v0.1 flagged: master-side scope must be explicit; pin §5
evidence path.
Changes:
- §1.A: collapse three-option proposal to bound Option A. Make A1
(master-side observation-driven re-emission) and A2 (primary-side
recovery dispatch) explicit as two halves of one causal chain.
Record B/C rejection rationale for future reference.
- §1 Files: revise table with Side column (master/primary). Add
master-side rows (A1 re-emit logic + ObservationStore freshness
helper). Total estimate ~360 prod + ~150 test, split master ~90 /
primary ~120 / tests ~150.
- §2: rewrite criteria #1-#5 around bound Option A (drop per-Option
deadline language). Split #2/#3 into A1 master-side + A2
primary-side criteria. Hardware deadline at #5 stays 30s.
- §2 verifier note: file paths + test names pinned at code-start
(acceptable for v0.2 per QA review).
- §5: pin G5-5 seed evidence to actual artifact path
V:\share\g5-test\logs\artifacts-20260427T092858Z\primary-fail.log
(no future task — fact-pointer).
- §7: trigger-source binding row marked done (architect REVISE);
single-sign of v0.2 still pending.
- Header: v0.1 → v0.2 status note updated.
Standing by for architect single-sign of v0.2.
No code starts until single-sign.
Per architect single-sign of G5-5 §close (`seaweedfs@c78116fd2`):
(a) v3-dev-roadmap.md
- §3: G5 line note now mentions G5-5 closed at L3 + G5-5C carry-forward
- §4: G5-5 row → CLOSED (link to seaweedfs@c78116fd2); G5-5C row added
as next active gate with bound pass criterion
- §7: G5-5 close commit appended to recently-closed table
(seaweed_block@5c4718f + seaweedfs@c78116fd2, L3 reached, #4 carry)
(b) v3-phase-15-g5-5c-mini-plan.md (new) v0.1 kickoff
- §1 scope: peer recovery trigger after replica restart; reuse T4d-4
primitives (architect binding); no engine logic change
- §1.A: three trigger source options (A master observation, B periodic
probe, C transport reconnect) with tradeoffs; sw recommends A; final
pick deferred to architect ratification
- §2: 6 acceptance criteria, hardware step is exactly G5-5 #4
(verify_restart_catchup → GREEN with no harness changes)
- §3: 2 new INVs proposed (REPL-PEER-RECOVERY-TRIGGER-001 +
-NO-RETRIGGER-LOOP) + 2 deferred ledger pointers from G5-5 close
- §4: G-1 N/A (new build, no V2 PORT)
- §5: forward-carries from G5-5 §close all addressed
- §6: 5 risks tabled
- §7: sign table awaiting architect §1-§6 ratification including
trigger source pick
Standing by for architect ratification of trigger source binding.
No code starts until §1-§6 signed.
Architect's round-15 hygiene callout: §7 sign table still had
three pre-code 'blocked' rows after the real close-state rows
landed in the prior doc-fix commit. Pure leftover from before
the close-state update overwrote earlier rows but didn't delete
the trailing pre-code rows.
Removed:
- 'Code start (script + Go helper) ... blocked on ratification'
- 'm01 hardware verification run ... blocked'
- '§close append + close sign ... blocked'
Sign table now ends cleanly at the §close architect single-sign
pending row. Ready for sign.
§close summary per v3-batch-process.md §12 template:
Done:
- #1 verify_cluster_ready
- #2 verify_byte_equal — live iSCSI replicated write, byte-equal
verified via storage-aware m01verify (LBA[0]=0xab on cross-host
hardware)
- #3 verify_network_catchup — iptables disconnect+heal, replica
converges to LBA[1]=0xcd byte-equal in 8s via engine-driven
catch-up
- 14 bugs surfaced+fixed across 14 m01 self-iteration rounds
- 5 INV-BIN-WIRING-* invariants in v3-invariant-ledger.md from
G5-4 still load-bearing; G5-5 hardware run is Integration backstop
Not done:
- #4 verify_restart_catchup — kill replica + write while down +
restart: replica's LBA[2]=0xef does NOT converge in 30s. Per
architect ruling 2 (round 13): real recovery-path finding,
surface as G5-5C carry-forward.
- #5 verify_race_stress + #6 verify_full_suite — gated on #4 fix
or test sequencing rework.
Product level reached: L3 (Replicated IO) per v3-architecture.md §13.
Falls short of full L4 (Failure/recovery under IO) — process-restart
recovery is the gap, scoped as G5-5C.
Next gate that makes it usable: G5-5C Peer Recovery Trigger After
Replica Restart — fix engine-driven catch-up re-trigger when a
degraded peer becomes reachable again. After G5-5C: re-run #4#5#6
in this same harness; full L4 reached.
Forward-carries to G5-5C (architect-bound 2026-04-27):
- Reuse existing engine-driven recovery primitives (T4d-4); no
ad-hoc re-ship from replication layer.
- Define trigger source first: observation reappearance, periodic
probe loop, or stream/transport reconnect signal.
- Pass criterion: exactly the failed hardware case from G5-5 #4.
- Seed evidence: seaweed_block@5c4718f primary-fail.log shows the
gate-degraded + stale-barrier-ack pattern.
Forward-carries to opportunistic future hardening:
- Unit test for EnsureStorage→assignment-arrives→first-Open
Identity-latch path (would have caught round-10/11 bug pre-m01).
- Generalize start_cluster() pre-flight stale-state cleanup pattern
for future hardware harnesses.
Forward-carries to G5-6:
- G5-DECISION-001 (Path A vs Path B) — engine-state serializability
pinned in T4d still holds; G5-5 doesn't change posture.
Pending: architect single-sign on §close per v3-batch-process.md §5
+ §8C.2.
Refs: 24 commits in seaweed_block@phase-15 spanning rounds 1-14
(documented in §close.evidence.commits table).
Architect's v0.2 review caught that §1 absorbed the 3 binding revisions
but §2 (the close contract per v3-batch-process.md §2) stayed stale:
- §2 #2 still said "byte-equal on replica's walstore extent"
- §2 had old #4 (race stress) instead of new #4 (process restart)
- §2 #3 didn't name /status/recovery as the R/H source
v0.3 rewrites §2 to match §1, with explicit verifier names:
#1 verify_cluster_ready
#2 verify_byte_equal — m01verify Go helper using walstore.OpenReadOnly
+ storage.LogicalStorage.Read(lba) + SHA-256 (NO raw extent peek)
#3 verify_network_catchup — iptables disconnect + polls
/status/recovery?volume=v1 for R/H; asserts RecoveryDecision="catch_up"
#4 verify_restart_catchup — SIGTERM replica + restart same binary +
same --durable-root; polls /status/recovery same as #3#5 verify_race_stress — 10x -race on G5-4.5 integration test
#6 verify_full_suite — go test ./... clean from m01
#7 v3-dev-roadmap.md updated at gate-close per v3-batch-process.md §8
§1 file map and §5 forward-carry table already match v0.3 numbering
(grep confirmed no stale references). Implementation scope unchanged
from v0.2 (~310 prod LOC + ~30 unit tests).
v3-batch-process.md §2 single-source-of-truth discipline preserved:
§2 acceptance criteria IS the close contract; §1 scope description
stays in sync but is not load-bearing for close evidence.
Addresses 3 architect revision requirements (round 51):
REVISION 1 — process restart distinct from network disconnect:
Split G5-4 #4 forward-carry into TWO scenarios:
§2 #3 network disconnect (iptables) — proves live TCP interrupt
+ recovery without process restart
§2 #4 replica process stop/restart — proves durable reopen +
master resubscribe + recovery reconstruction
G5-4 #4 is now FULLY consumed (was: only network proxy in v0.1).
REVISION 2 — storage-aware byte verifier:
Replace raw walstore .extent peek with storage-abstraction Read(lba):
helper opens replica's walstore via core/storage/walstore (or
equivalent OpenReadOnly path), invokes Read(lba) per LBA in the
range, SHA-256 vs primary's known payload. Raw extent peek
REJECTED — walstore on-disk includes WAL frames + checkpoints
+ sparse regions + potentially-stale-but-valid blocks; only
Read(lba) returns the authoritative current value.
Risk added: if walstore.OpenReadOnly is missing, sw adds it as
part of this batch (small scope expansion contained in
core/storage/walstore; read-only opener for verification only,
NOT a substrate semantic change).
REVISION 3 — named R/H observation source:
/status?volume=v1 returns frontend.Projection (no R/S/H). G5-5
adds /status/recovery?volume=v1 returning engine.ReplicaProjection
(Mode, R, S, H, RecoveryDecision); gated by new --status-recovery
daemon flag (default off; production binaries don't enable).
Loopback-only via existing isLoopbackRemote guard. ~30 prod LOC
+ ~30 unit tests. Engine/adapter logic unchanged — surfaces
already-computed projection through HTTP.
Updated §1 file map, §1.4 truth-domain check, §5 forward-carry
table, §6 risks (3 new rows), §close template unchanged.
Re-submitted for architect §1-§6 ratification. After ratify, sw
codes per §1 file map; estimate ~310 prod LOC + ~30 unit tests.
Codifies the lessons from T4 + G5-4 retrospective:
KEEP — earned its keep on T4:
- G-1 V2 PORT read (saved 5 hidden invariants on T4b-4, probe non-
mutation pin on T4c-2, 3 placement decisions on T4d-3)
- Mini-plan acceptance criteria (single source of truth for close)
- Invariant ledger discipline ("claim without test = wish")
- m01 -race verification (caught 2 engine bugs at T4d-4 part C)
- Architect single-sign at close (caught 4 stale refs at G5-4 close)
DROP — overhead without payoff:
- Separate kickoff PROPOSAL doc (mini-plan §1-§6 = same thing)
- Separate G-1 doc (inline §4 of mini-plan)
- Separate closure report doc (§close section of mini-plan)
- Separate forward-carry checklist (§5 of next-batch mini-plan)
- Separate QA scenario catalogue (write tests directly when ready)
- Multi-version doc churn (v0.1→v0.5)
- Cross-doc invariant restatement (ledger is sole source)
- Mixed T-track + G-N naming for same gate
Compressed sign cycles: was 4-5 architect signs per batch; now 2
(scope ratify + close sign).
Per-batch artifact count: was 5+ (kickoff + mini-plan + G-1 +
closure + checklist + scenario catalogue); now 1 (mini-plan with
§close appended).
Decision rules codified:
§6.1 G-1 yes/no (V2 PORT yes; V3-native no)
§6.2 T-track vs G-N naming (architect picks at kickoff)
§6.3 When to skip mini-plan (1-line hotfix-class)
§8 names the 6 first-order control docs to keep current
(v3-dev-roadmap, v3-phase-15-mvp-scope-gates, v3-invariant-ledger,
v3-block-behavior-contract-index, v3-product-placement-authority-
rationale, v2-v3-contract-bridge-catalogue).
§9 first trial: G5-5 m01 hardware first-light.
§11 honesty principle: documentation that catches bugs is
discipline; documentation that doesn't is ceremony. Drop ceremony,
keep discipline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR-atomic with seaweed_block@c820e17 per architect binding round 50
(mini-plan v0.4 §4 #7): ledger inscription required at G5-4 close.
- INV-BIN-WIRING-ROLE-FROM-ASSIGNMENT
- INV-BIN-WIRING-PEER-SET-FROM-ASSIGNMENT-FACT
- INV-BIN-WIRING-LISTENER-LIFECYCLE-LIFO
- INV-BIN-WIRING-ASSIGNMENT-DRIVES-MEMBERPRESENT
- INV-BIN-WIRING-SESSIONID-VIA-ADAPTER
All 5 are ACTIVE with test pointers to
cmd/blockvolume/g5_4_l2_replication_test.go (subprocess integration)
+ source-side checks in cmd/blockvolume/main.go and core/host/volume.
Last verified 2026-04-26 (G5-4 close).
Architect ratification 2026-04-26:
"Role inference, in-process acceptance, G5-DECISION-001 seam, and
sessionID discipline are architecturally correct. G-1 must clarify
replica readiness semantics and confirm ctrl-addr reuse or introduce
repl-addr before code."
2 binding clarifications baked into v0.3:
#1 — §4 #2 acceptance criterion split by role:
- Primary: Healthy=true per existing frontend/write-ready projection
- Replica: replication-ready / listener-bound + ApplyEntry byte-equal
verified — MUST NOT report Healthy=true if existing field implies
frontend-primary-write-ready
- If existing status field is too coarse, G5-4.5 uses precise
assertion names (assertReplicaReplicationReady,
assertPrimaryFrontendReady) instead of unified assertHealthy
#2 — §4 #7 acceptance criterion strengthened:
- Catalogue inscription ALONE insufficient at G5-4 close
- 5 INV-BIN-WIRING-* invariants MUST land in v3-invariant-ledger.md
- Per v3-quality-system.md §6 "an invariant without a test is a wish"
- Ledger updated as PR atomic with code (not after-the-fact)
§7.1 G-1 deliverable extended (G-1-blocking subitems):
- Replica readiness semantics — what existing volume.Status /
ProjectionView field expresses replication-ready (vs Healthy)?
G-1 either proposes new field OR specifies precise assertion names
- --ctrl-addr reuse confirmation — verify NO conflict with NVMe/iSCSI
control-plane traffic on same port. If conflict, G-1 introduces
--repl-addr flag (small scope expansion, contained in this batch)
§3 #4 predicate flipped to ✅ DONE (architect round 50).
§8 sign table updated with explicit ledger requirement at close.
Architect-pre-baked: ratification stays valid; no further mini-plan
revisions needed before G-1.
Sw next: produce G-1 V3-native PORT read deliverable per §7.1
(includes the 2 binding subitems). Code stays blocked until
architect ratifies G-1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses QA's 3 notes + 1 clarification ask:
Note 1 (role inference):
§1.3 rewritten — fact.ReplicaID is master-minted (proto verified
at control.proto:128-148 + mint site at services.go:198-205).
Binary reads `fact.ReplicaID == self.ReplicaID` directly. No
lex-smallest fallback (master always names exactly one bound
replica per volume per line). Removes the binary-side authority
inference that violated the master-authority rule.
Note 2 (acceptance circular):
§4 #2 verifier reframed to G5-4.5 in-process test. m01 hardware
verification belongs to G5-5; G5-4 closes on the in-process pin.
Note 3 (G5-DECISION-001 contradiction):
§5 rewritten — G5-4 ships Path B runtime AND keeps Path A
serializability seam open. T4d-4 part B's RoundTripJSON test
already pins serializability; G5-4 preserves it. G5-6 architect
ratification can promote to Path A by adding persistence on top
of the existing struct, with no engine-state-shape change.
Clarification ask (sessionID minting):
§6 added INV-BIN-WIRING-SESSIONID-VIA-ADAPTER. Adapter mints
unique sessionIDs via process-wide atomic counter at
adapter.go:70; binary inherits for free as long as it dispatches
via the adapter (never via framework shortcuts that hardcode
sessionID=1, which is the known T4c §I + QA G5-1 round 1 SKIP
gap). Pinning this invariant keeps the gap test-side.
Re-submitted for QA re-review per parent kickoff §7 governance loop.
Mirror cmd/blockvolume to T4d-4 part B's WithEngineDrivenRecovery()
framework binding. Single batch (~250 prod + ~150 tests), 5 ordered
subtasks. Design decisions (a-d per kickoff §3 G5-4 row):
(a) Role inference: assignment-driven, no new CLI flag
(b) Peer discovery: AssignmentFact.Peers per T4a-5 P-refined
(c) Listener lifecycle: --ctrl-addr reuse + LIFO Stop in host.Close()
(d) Engine instantiation: one engine per volume, single --volume-id
Pre-merge gates require G-1 V3-native PORT read of cluster.go:357-369
+ V2 lesson check on weed/storage/blockvol/blockvol.go before code.
4 new invariants to inscribe at close (INV-BIN-WIRING-*).
Submitted for QA + architect ratification per parent kickoff §7
governance loop. No code until ratify.
Hand-off doc v0.3 + G5 kickoff v0.2: m01+M02 bring-up smoke surfaced
that cmd/blockvolume binary lacks T4 replication wiring entirely.
Sw-confirmed root cause:
- --t1-readiness HealthyPathExecutor is primary-only by design
- volume.Config.ReplicationVolume slot exists (host.go:73) with godoc
"T4a-5 production wiring sets this" — but T4a-5 only added the
field; the wiring NEVER landed
- T4d-4 part B wired WithEngineDrivenRecovery() for component test
framework (cluster.go:357-369), NOT for the binary
- Result: V3 components compose end-to-end (proven by T4d HARD GATE
#3); the production binary still constructs a primary-only data
plane
Sw confirmed this is real implementation work (150-300 LOC + design),
not a 50-LOC quick patch. Four design decisions needed:
1. Role inference (assignment vs CLI flag vs topology)
2. Peer discovery (from AssignmentFact.Peers)
3. Listener lifecycle (--data-addr reuse + Stop)
4. Engine instantiation (one engine per volume)
G5 kickoff revised to v0.2:
- 5 batches → 6 batches (binary wiring promoted to G5-4)
- G5-1/2/3 are NOT blocked by G5-4 (component framework already
binds T4d-4 part B; QA scenarios + walstore cadence at
component/primary-only scope can run in parallel)
- G5-4 binary wiring: needs full governance loop (kickoff →
architect ratify → mini-plan → architect ratify → G-1 → code).
G-1 source: T4d-4 part B component framework as V3-native PORT
- G5-5 m01 hardware first-light DEPENDS on G5-4 (script can't
drive replica scenarios until binary supports replicas)
- G5-6 G5-DECISION-001 resolution at G5 close (was G5-5 in v0.1)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause for "volume not ready" gate: missing
--expected-slots-per-volume 2 flag on blockmaster.
Default is 3; QA's 2-node topology had 2 slots; controller
silently rejected observation snapshot (cmd/blockmaster/main.go:39).
Fix verified locally on Windows (single-node, no m01/M02 needed):
- Add --expected-slots-per-volume 2 to blockmaster command
- Primary reaches Healthy=true with epoch=1
- assignment-received fires; durable storage opens; status
endpoint serves {"Healthy":true}
Lesson learned (process improvement): for V3-internal bring-up
debug, try single-node local reproduction FIRST. The cluster
bring-up gate is V3 logic, not network topology. Reproduces in
seconds locally with full source-code access; m01/M02 only needed
for cross-node-specific scenarios (real network conditions,
iptables, multi-host wire).
Secondary finding: replica r2 sees primary r1's assignment but
records "supersede, not applying to adapter" because T1
HealthyPathExecutor only handles primary case. For G5-4 replica
bring-up, sw needs to wire T4a-T4d ReplicationVolume + ReplicaPeer
+ ReplicaListener stack (not just --t1-readiness flag). This is
the actual next gap for G5-4.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause: cmd/blockmaster/main.go hardcoded ExpectedSlotsPerVolume=3.
QA's 2-slot topology silently failed validateVolumeTopology in the
controller, so no assignments were minted, no master-log lines,
and volumes timed out at durable open.
Fix landed in seaweed_block@f5de7c5: --expected-slots-per-volume
CLI flag, default 3, set 2 for the 2-node smoke.
QA next: rebuild blockmaster, pass --expected-slots-per-volume 2
in §3.4 of the handoff command sequence; rest unchanged.
Records QA's cross-node smoke attempt 2026-04-26: infrastructure
fully verified READY (m01+M02 reachability, SMB share for binary
distribution, master cross-node listen, network OK), but cluster
bring-up blocked at V3-internal gate.
Symptom: blockvolume on both nodes connects to master but logs
"durable open: frontend: volume not ready" — never reaches steady
state, status endpoint never binds, master log shows no heartbeat
or assignment-mint events.
Hand-off contents:
- §1 specific questions for sw (5 gaps to fill)
- §2 infrastructure verified READY (no action needed)
- §3 copy-pasteable commands sw can run/debug
(build → topology → master → primary → replica → cleanup)
- §4 QA's hypothesis on the gap (assignment-from-master flow)
- §5 debug suggestions for sw (log levels, integration test
references)
- §6 G5-4 script skeleton current state
- §7 QA's next steps once sw answers
Working dirs reproducible:
- Binaries: /mnt/smb/work/share/g5-binaries/{blockmaster,blockvolume}
- Run state: /tmp/g5sm/ on both nodes
- Logs: /tmp/g5sm/logs/{master,primary,replica}.log
Blocks: G5-4 implementation work (script scenario bodies, hardware
first-light scenarios). Does NOT block QA scenario authoring at
component scope (Cluster framework already covers that).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per QA infra-check round 2026-04-26, surfaces real readiness gaps
before architect ratifies G5-4 schedule:
m01 (192.168.1.181 — primary node):
✅ 32-day uptime; sudo password-less; 16 cores; 19 GiB RAM
✅ 177 GiB free disk; Go 1.26.2 installed
✅ iptables / netns / multi-process tools all available
✅ T2 m01 NVMe script template available as pattern reference
M02 (192.168.1.184 — replica node):
✅ Reachable from m01 (0.92ms); same kernel; 178 GiB free disk
❌ Go NOT installed — must scp binaries from m01
Implication for G5-4:
Build binaries on m01, scp to M02. Same cross-node binary pattern
T2 already uses for its iSCSI target deployment. G5-4 skeleton at
seaweed_block/scripts/iterate-m01-replicated-write.sh implements
this build-then-scp flow.
No infrastructure blockers. Architecture ready as soon as G5 mini-plan
ratifies scenario list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two artifacts landing together to close T4 batch series:
1. v3-phase-15-t4d-closure-report.md (NEW)
QA single-sign artifact for T4d batch close per §8C.2; architect
T-end three-sign per §8C.1 (T4d IS final T4 batch — confirmed at
round-48 review). Round-48 + round-49 corrections incorporated:
- Part C commit hash bound to e642ae8 throughout
- CARRY-T4D-LANE-CONTEXT-001 bind point = post-G5 hardening
backlog (not T4e — consistent with "T-end at this close")
- §H Finding #1 reworded — walstore HAS background flusher
(walstore.go:189-190); QA's earlier "caller-driven" was wrong
- §H Finding #3 RESOLVED at a0be6d5 (T2A NVMe race fixed +
m01 -race ×50 PASS)
- 16 invariants pinned (added 2 named for part C bug fixes:
INV-REPL-FAILED-SESSION-KIND-DRIVES-ESCALATION +
INV-REPL-REBUILD-ESCALATION-STICKY-UNTIL-TERMINAL)
- 22/22 packages green under -race on m01 (post-a0be6d5)
2. v3-phase-15-t4d-mini-plan.md (NEW — was uncommitted across
v0.1 → v0.5 evolution)
Final v0.5 incorporates: architect Path B fold; round-47
rebuild path engine-driven HARD GATE expansion; G5-DECISION-001
named decision record; 4-batch shape ratified; T4d-3 G-1 binding.
Active forward-carries (post-G5 hardening backlog):
- CARRY-T4D-LANE-CONTEXT-001 — replace TargetLSN==1 caller shim
with true handler/session-context lane signal
- G5-DECISION-001 — engine recovery state behavior across
primary restart (Path A persist vs Path B rebuild-from-probe)
G5 collective close items (NOT post-G5):
- m01 hardware first-light for replicated write path
- Multi-replica concurrent live + recovery scenarios
- walstore flusher cadence verification + tuning policy
- Minimal metrics/backpressure assessment
- G5-DECISION-001 architect resolution
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pingqiu