Commit Graph
18 Commits
Author SHA1 Message Date
pingqiuandClaude Opus 4.6 59a36013d4 feat: rebuild hardening A1-A5 + session-controlled execution path
A1 Engine kind-routing fix:
  SessionProgressObserved/Completed/Failed now respect active session
  Kind. Rebuild progress no longer leaks into catch-up aggregate.
  sessionKindMismatch guard + observeRebuildProgress helper.
  2 regression tests lock kind isolation.

A2 Retention pin:
  Rebuild session ack drives progress-based WAL retention floor.
  Pin installed at base_lsn on accepted, advances with wal_applied_lsn,
  released on completed/failed/cancelled. rebuildProgressPinFloor
  returns min across all active replicas.
  Retention pin test: 100 blocks fill WAL, 5 flusher cycles with
  20 pinned rebuild entries — all verified correct.

A3 Progress ack emission:
  Automatic sessionAck(running/base_complete/completed/failed) emitted
  from rebuild session lifecycle transitions. sessionAckLocked builds
  ack under session lock. emitRebuildSessionAck callback wired through
  SetOnRebuildSessionAck on BlockVol.
  ObserveReplicaRebuildSessionAck maps acks to core engine events.
  WireLocalReplicaRebuildSessionAcks bridges local callback to server.
  5 server tests proving ack→core, pin advance, pin cleanup.

A4 Deadline/timeout:
  rebuildAckWatch watchdog: armed on accepted/running/base_complete,
  refreshed on each ack, cleared on completed/failed. Timeout
  cancels local session + clears pin + fail-closes.
  2 tests: timeout→fail-close, progress→refresh.

A5 Session-controlled execution path:
  v2bridge.Executor.TransferFullBase now uses session-controlled loop:
  beginControlledFullBase → real sessionControl over TCP →
  transferExtentToSession via RebuildTransportClient →
  PrepareFullBaseRebuild → TryCompleteRebuildSession.
  ReplicaReceiver control channel handles MsgSessionControl alongside
  MsgBarrierReq. Session acks written back on same TCP connection.
  RebuildSessionBase request type separates new per-block stream from
  legacy raw extent stream. Full-base cleanup deferred until success.
  Deadlock fix: ApplyBaseBlock releases session lock before ioMu.
  Hydration skip for full-base sessions.

23 rebuild component tests (all pass):
  11 kernel correctness, 8 transport/runtime, 3 scenario-scale,
  including 1GB primary-initiated with CRC validation.

29 files changed, ~2500 insertions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 14:39:11 -07:00
pingqiuandClaude Opus 4.6 d2d57851b0 feat: rebuild MVP — dual-lane session with bitmap protection
Rebuild session protocol implementation for v2-rebuild-mvp-session-protocol.md.

New files:
- rebuild_bitmap.go: RebuildBitmap — session-scoped dense bitset for
  WAL-applied LBA tracking. MarkApplied on local WAL write (not receive).
  ShouldApplyBase returns false for WAL-covered LBAs (WAL always wins).

- rebuild_session.go: RebuildSession — replica-side two-line rebuild.
  WAL lane (ApplyWALEntry) + base lane (ApplyBaseBlock) with bitmap
  conflict resolution. TryComplete requires BOTH base_complete AND
  wal_applied_lsn >= target_lsn. Volume-level control surface:
  StartRebuildSession, ApplyRebuildSessionWALEntry/BaseBlock,
  MarkRebuildSessionBaseComplete, TryCompleteRebuildSession,
  CancelRebuildSession, ActiveRebuildSession.

- rebuild_mvp_test.go: 4 correctness tests — base+WAL converge,
  WAL-applied never overwritten by base, bitmap set on applied not
  received, control surface start/supersede/complete.

- rebuild_transport_test.go: 2 transport-level tests — two-line with
  real WAL shipping, live writes during base copy with bitmap conflict.

Design docs:
- v2-rebuild-mvp-session-protocol.md: MVP spec with message set, apply
  rules, completion/failure/crash rules, test matrix
- v2-sync-recovery-protocol.md: full protocol context (keepup/catchup/
  rebuild unified design, primary decision logic, two-line model)
- v2-session-protocol-shape.md: protocol shape overview

Protocol engine (reference, not production):
- sw-block/protocol/: 7-event engine with ~300 lines, 13 tests

6 rebuild tests pass, all existing component tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 14:30:34 -07:00
pingqiuandClaude Opus 4.6 44103a1bd7 feat: Phase 20 acceptance fixes + sw-test-runner suite mode
Acceptance rows closed:
- WriteLBA/SyncCache contract: code comments document write-back vs
  durability fence semantics
- RF=2 stable identity: v2bridge always uses SetReplicaAddrs (preserves
  ServerID); blockcmd dispatcher also fixed to use setupPrimaryReplicationMulti;
  test asserts exact expected ReplicaID="vs-2" (not just non-empty)
- Tests treating WriteLBA as commit: replica_read_test rewritten with
  SyncCache as durability fence
- publish_healthy contract: 3 gate tests with hard assertions including
  gate 3 (PrimaryShipperConnected)
- SetReplicaAddr deprecation warning added
- WALShipper.ReplicaID() getter added for identity verification

Test runner enhancements:
- sw-test-runner suite command: build → deploy → run N scenarios in one
  invocation with --skip-deploy support
- Suite YAML definitions for T6 Stage 0 and Stage 1
- deploy action: kill stale processes, clean dirs, cross-compile, upload
- run-phase20-t6.ps1 PowerShell script (deprecated by suite command)

Engine/runtime fixes:
- Recovery executor nil-safety improvements
- Recovery bundle BuildRecoveryBundle defensive checks
- ShipperGroup MinReplicaFlushedLSNAll surface

Docs: acceptance checklist refined, test matrix updated, T6 runbook,
engine maintainer tutorial, design README updated.

26 files changed, ~1600 insertions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 11:30:54 -07:00
pingqiu 43dbebfa04 refactor: close bounded recovery drain and invalidation seams
Move removed-replica drain and replica-scoped invalidation onto explicit core-command paths so the widened multi-replica runtime no longer depends on coarse host-side recovery handling.

Made-with: Cursor
2026-04-04 11:01:12 -07:00
pingqiu 16ba70f856 refactor: make bounded recovery observation events replica-scoped
Carry replica-scoped addressing through bounded recovery planning and completion events so the core no longer depends on a volume-only observation seam. This preserves the current single-replica catch-up and rebuilding behavior while aligning the observation side with the replica-scoped command path.

Made-with: Cursor
2026-04-04 09:18:07 -07:00
pingqiuandClaude Opus 4.6 b304b8e212 refactor: make bounded recovery command addressing replica-scoped
Replace the remaining volume-scoped recovery command and pending slot
with replica-scoped addressing on the bounded core-present path. This
preserves the current single-replica catch-up and rebuilding behavior
while removing the structural blocker for later multi-replica startup
ownership.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 09:05:36 -07:00
pingqiuandClaude Opus 4.6 41082bf92c fix: Batch 6 completion — rebuildAddr folded into resolveRecoveryContext
resolveRecoveryContext now also derives rebuildAddr from assignments,
so the full host-side recovery context is resolved in one call:
- volPath (from replicaID)
- rebuildAddr (from assignments via deriveRebuildAddr)
- recovery bindings (driver + executor via BuildRecoveryBundle)
- replicaFlushedLSN (from sender session)

startTask/runRecovery/runCatchUp/runRebuild now pass assignments
instead of rebuildAddr. No separate rebuildAddr resolution remains
outside the resolver.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 01:52:35 -07:00
pingqiuandClaude Opus 4.6 a48da0f674 refactor: Batch 6 — recovery context resolver extracted
New recoveryContext type + resolveRecoveryContext method consolidates:
- volumePathForReplica (volPath from replicaID)
- v2bridge.BuildRecoveryBundle (driver + executor from BlockVol)
- sender/session lookup (replicaFlushedLSN for catch-up start)

runCatchUp and runRebuild now read as:
  resolve → plan → branch (legacy or core-present)

Removed buildRecoveryBundle (inlined into resolveRecoveryContext).
block_recovery.go no longer has any inline context assembly —
it is now a pure orchestration shell.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 01:46:06 -07:00
pingqiuandClaude Opus 4.6 263611004e refactor: Batch 5 — recovery binding factory moved to v2bridge
New v2bridge.BuildRecoveryBundle(vol, rebuildAddr) assembles all
recovery bindings (Reader + Pinner + StorageAdapter + Executor) from
a real BlockVol instance in one call.

block_recovery.go changes:
- Removed local recoveryBundle type
- buildRecoveryBundle now delegates to v2bridge.BuildRecoveryBundle
  inside WithVolume, returns (driver, executor, err)
- Removed direct v2bridge.NewReader/NewPinner/NewExecutor construction
- Removed bridge import (no longer needed)
- runCatchUp/runRebuild use (driver, executor, err) directly

block_recovery.go no longer knows how to construct Reader, Pinner,
StorageAdapter, or Executor. It only knows: resolve volPath, ask the
factory for bindings, plan, branch to legacy or core-present path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 01:39:40 -07:00
pingqiuandClaude Opus 4.6 ded84b25e6 refactor: Batch 4 steps 2+3 — rebuild status port + recovery bundle factory
Step 2: Rebuild completion status port
- New runtime.RebuildCompletionStatus + DeriveRebuildCommitted:
  reusable shaping logic for post-rebuild snapshot → RebuildCommitted event
- block_recovery.go OnRebuildCompleted: delegates to DeriveRebuildCommitted,
  host only reads raw snapshot via readRebuildStatus (thin binding)
- Removed 15 lines of inline flushedLSN/checkpointLSN/achievedLSN computation

Step 3: Recovery bundle factory
- New buildRecoveryBundle: shared host-side setup for both catch-up and rebuild
  (creates Reader + Pinner + StorageAdapter + Executor + RecoveryDriver)
- runCatchUp and runRebuild both use buildRecoveryBundle instead of
  duplicating the WithVolume → NewReader → NewPinner → NewStorageAdapter →
  NewExecutor → RecoveryDriver chain
- runCatchUp/runRebuild are now thin host-shell methods

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 01:32:34 -07:00
pingqiuandClaude Opus 4.6 0bcfc678d0 refactor: Batch 4 step 1 — typed PendingExecution, zero type assertions
Replace interface{} fields in runtime.PendingExecution with typed handles:
- Driver: *engine.RecoveryDriver (was interface{})
- Plan: *engine.RecoveryPlan (was interface{})
- CatchUpIO: engine.CatchUpIO (was interface{})
- RebuildIO: engine.RebuildIO (was interface{})

block_recovery.go:
- ExecutePendingCatchUp/Rebuild: direct field access (pe.Driver, pe.Plan)
  instead of type assertions (pe.Driver.(*engine.RecoveryDriver))
- CancelFunc: pe.Driver.CancelPlan(pe.Plan, reason) — no casts
- 6 type assertions removed from production path

Test files: remove Plan type assertions — fields are typed end-to-end.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 01:27:29 -07:00
pingqiuandClaude Opus 4.6 3a5fbbfded fix: Batch 3 wiring — production path uses runtime helpers, legacy isolated
H wiring: block_recovery.go now uses runtime.PendingCoordinator
- Removed local pendingRecoveryExecution type + store/take/peek/has/cancel
- ExecutePendingCatchUp/Rebuild delegate to coord.TakeCatchUp/TakeRebuild
- Shutdown uses coord.CancelAll
- Added CancelAll to PendingCoordinator

I wiring: executeCatchUpPlan/executeRebuildPlan replaced
- ExecutePendingCatchUp now calls rt.ExecuteCatchUpPlan with RecoveryManager
  as RecoveryCallbacks (OnCatchUpCompleted/OnRebuildCompleted)
- ExecutePendingRebuild follows same pattern
- Local executeCatchUpPlan/executeRebuildPlan methods removed

J structural: legacy no-core branches extracted
- executeLegacyCatchUp: wraps rt.ExecuteCatchUpPlan for v2Core==nil path
- executeLegacyRebuild: wraps rt.ExecuteRebuildPlan for v2Core==nil path
- Clear "LEGACY NO-CORE COMPATIBILITY" section with structural separation
- runCatchUp/runRebuild now branch cleanly: legacy helper vs core coordinator

Test updates: pendingRecoveryExecution → rt.PendingExecution, field casing,
Plan type assertions.

Validation: all P4, P16B, and ApplyAssignments tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 01:20:41 -07:00
pingqiuandClaude Opus 4.6 e075d77619 refactor: Task J — legacy no-core paths explicitly labeled
Add explicit "LEGACY NO-CORE COMPATIBILITY" section header in
block_recovery.go marking HandleAssignmentResult and
HandleRemovedAssignments as compatibility-only entry points.

The comment block explicitly states:
- These are for pre-Phase-16 no-core paths and older tests
- Core-present paths use StartRecoveryTask + ExecutePending*
- These should NOT be strengthened into semantic-authority proofs

No behavioral change — structural labeling only. All validation passes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 01:05:16 -07:00
pingqiuandClaude Opus 4.6 519c849946 refactor: Task F+G — remove pinner shim, executor already clean
Task F (Pinner):
- block_recovery.go: removed pinnerShimForRecovery (11 lines of pure
  pass-through). v2bridge.Pinner structurally satisfies bridge.BlockVolPinner
  (same method signatures), so it's passed directly.

Task G (Executor):
- Already clean. v2bridge.Executor is used directly without any shim —
  structurally satisfies engine.CatchUpIO and engine.RebuildIO.
  No code changes needed.

After Task E+F+G: zero shim types remain in block_recovery.go.
v2bridge Reader/Pinner/Executor all satisfy sw-block contracts directly.

Validation:
- go test ./weed/storage/blockvol/v2bridge/ -run "TestPinner_|TestExecutor_|TestBridge_" → PASS
- go test ./weed/server/ -run "TestP4_|TestP16B_" → PASS (8 tests)
- go test ./sw-block/bridge/blockvol/... → PASS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 00:45:43 -07:00
pingqiuandClaude Opus 4.6 680b530314 refactor: Task E — reader returns bridge.BlockVolState directly
Reader backend-binding extraction:
- v2bridge/reader.go: Reader.ReadState() now returns bridge.BlockVolState
  directly instead of a local v2bridge.BlockVolState mirror type.
  Removed the local BlockVolState type entirely.
- block_recovery.go: removed readerShimForRecovery (12 lines of 1:1
  field copying). Reader is now passed directly as bridge.BlockVolReader.

Before: v2bridge.Reader → v2bridge.BlockVolState → readerShim → bridge.BlockVolState
After:  v2bridge.Reader → bridge.BlockVolState (direct)

v2bridge now imports sw-block/bridge/blockvol for the contract type
(control.go already did this, reader.go now follows the same pattern).

Validation:
- go test ./sw-block/bridge/blockvol/... → PASS
- go test ./weed/storage/blockvol/v2bridge/ -run "TestReader_" → PASS
- go test ./weed/server/ -run "TestP4_|TestP16B_" → PASS (8 tests)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 00:43:30 -07:00
pingqiuandClaude Opus 4.6 13680c9aa6 feat: Phase 16B rev3 — bounded rebuild execution ownership + review
16B widened from catch-up-only to catch-up + rebuild:
- StartRebuildCommand: core emits rebuild command, adapter executes
- Fail-closed: pending rebuild does not run without fresh command
- Recovery observations close back into core projection

New proofs:
- StartRebuildCommand_ConsumesPendingPlanAndUpdatesProjection
- RunRebuild_FailClosedWithoutFreshStartRebuildCommand

Review docs:
- phase-16-rev3-review.md: widened 16B review object
- phase-16-rev3-manager-rereview.md: manager challenge response
- phase-16-checkpoint-review.md: updated

Non-claims: not full recovery-loop closure, not end-to-end
failover/publication, not launch readiness.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 21:38:44 -07:00
pingqiuandClaude Opus 4.6 8c2485e0e9 feat: Phase 15 + Phase 16A/B — V2 core integration + checkpoint review
Phase 15: V2 core wired into BlockService
- volume_server_block.go: v2Core field, applyCoreAssignmentEvent,
  core command executors (ApplyRole, StartReceiver, ConfigureShipper,
  InvalidateSession, StartCatchUp, StartRebuild, PublishProjection)
- Assignment processing now goes through core engine → command emission
  → bounded execution, replacing direct V1 replication setup
- master_block_registry.go: ClusterHealthSummary, VolumeMode in entries
- master_server_handlers_block.go: blockStatusHandler, entryToVolumeInfo
  refactored with entryReplicaSurface

Phase 16A: Core projection surfaces
Phase 16B: Bounded closure (checkpoint review ready)

Test fixes: add v2Core to manually-constructed BlockService in
idempotence, convergence, soak, and CP13-8A tests (required because
V1 replication setup paths now delegate to core engine).

All tests pass (21s regression).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 20:58:12 -07:00
pingqiuandClaude Opus 4.6 643a5a1074 feat: Phase 12 P3+P4 — diagnosability surfaces, perf floor, rollout gates
P3: Add explicit bounded read-only diagnosis surfaces for all symptom classes:
- FailoverDiagnostic: volume-oriented failover state with per-volume
  DeferredPromotion/PendingRebuild entries and proper timer lifecycle
- PublicationDiagnostic: two-read coherence check (LookupBlockVolume vs
  registry authority) with computed Coherent verdict
- RecoveryDiagnostic: minimal ActiveTasks surface (Path A)
- Blocker ledger: 3 diagnosed + 3 unresolved, finite, from actual file
- Runbook references only exposed surfaces, no internal state

P4: Add bounded performance floor + rollout-gate package:
- Engine-local floor measurement with explicit IOPS gates per workload
- Cost characterization: WAL 2x write amp, -56% replication tax
- Rollout gates with semantic cross-checks against cited evidence
  (baseline numbers, transport/network matrix, blocker counts)
- Launch envelope tightened to actually measured combinations only

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 16:20:22 -07:00