Commit Graph
14 Commits
Author SHA1 Message Date
pingqiu 10833c8b68 refactor: preserve bounded volume mode reason heartbeat truth
Carry explicit volume_mode_reason across the heartbeat/master/API seam so outward surfaces retain the bounded core-owned explanation behind mode transitions.

Made-with: Cursor
2026-04-04 14:21:31 -07:00
pingqiu 6cad5bb8e1 refactor: rebind bounded volume mode heartbeat truth
Make the heartbeat/master boundary preserve explicit volume_mode truth so master consume no longer reconstructs outward mode only from secondary heartbeat signals. Keep backward compatibility by falling back to the previous reconstruction when older heartbeats do not send the field.

Made-with: Cursor
2026-04-04 13:56:41 -07:00
pingqiu 6794f79df9 refactor: preserve bounded publish healthy heartbeat truth
Make the heartbeat/master boundary preserve explicit publish_healthy truth so master consume no longer reconstructs healthy publication only from secondary readiness and degraded heuristics. Keep backward compatibility by falling back to the previous reconstruction when older heartbeats do not send the field.

Made-with: Cursor
2026-04-04 13:43:19 -07:00
pingqiu eb610deb92 refactor: preserve bounded needs_rebuild heartbeat truth
Make the heartbeat/master boundary preserve explicit needs_rebuild truth so primary heartbeat consume no longer collapses that stronger mode into a generic degraded signal. Keep backward compatibility by falling back to the previous heuristic when older heartbeats do not send the field.

Made-with: Cursor
2026-04-04 13:11:42 -07:00
pingqiu 69b41a7f16 refactor: rebind bounded replica-ready heartbeat truth
Make the heartbeat/master boundary carry explicit replica readiness truth so the registry no longer depends only on replica transport-address presence as a readiness proxy. Keep backward compatibility by falling back to the old address heuristic when older heartbeats do not send the field.

Made-with: Cursor
2026-04-04 12:06:53 -07:00
pingqiu 43dbebfa04 refactor: close bounded recovery drain and invalidation seams
Move removed-replica drain and replica-scoped invalidation onto explicit core-command paths so the widened multi-replica runtime no longer depends on coarse host-side recovery handling.

Made-with: Cursor
2026-04-04 11:01:12 -07:00
pingqiu 5fd9ec0edf refactor: widen bounded multi-replica catch-up startup ownership
Emit one core-owned start_recovery_task per primary catch-up replica so the bounded multi-replica startup path no longer depends on a single-replica assumption.

Made-with: Cursor
2026-04-04 10:21:28 -07:00
pingqiuandClaude Opus 4.6 0bcfc678d0 refactor: Batch 4 step 1 — typed PendingExecution, zero type assertions
Replace interface{} fields in runtime.PendingExecution with typed handles:
- Driver: *engine.RecoveryDriver (was interface{})
- Plan: *engine.RecoveryPlan (was interface{})
- CatchUpIO: engine.CatchUpIO (was interface{})
- RebuildIO: engine.RebuildIO (was interface{})

block_recovery.go:
- ExecutePendingCatchUp/Rebuild: direct field access (pe.Driver, pe.Plan)
  instead of type assertions (pe.Driver.(*engine.RecoveryDriver))
- CancelFunc: pe.Driver.CancelPlan(pe.Plan, reason) — no casts
- 6 type assertions removed from production path

Test files: remove Plan type assertions — fields are typed end-to-end.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 01:27:29 -07:00
pingqiuandClaude Opus 4.6 3a5fbbfded fix: Batch 3 wiring — production path uses runtime helpers, legacy isolated
H wiring: block_recovery.go now uses runtime.PendingCoordinator
- Removed local pendingRecoveryExecution type + store/take/peek/has/cancel
- ExecutePendingCatchUp/Rebuild delegate to coord.TakeCatchUp/TakeRebuild
- Shutdown uses coord.CancelAll
- Added CancelAll to PendingCoordinator

I wiring: executeCatchUpPlan/executeRebuildPlan replaced
- ExecutePendingCatchUp now calls rt.ExecuteCatchUpPlan with RecoveryManager
  as RecoveryCallbacks (OnCatchUpCompleted/OnRebuildCompleted)
- ExecutePendingRebuild follows same pattern
- Local executeCatchUpPlan/executeRebuildPlan methods removed

J structural: legacy no-core branches extracted
- executeLegacyCatchUp: wraps rt.ExecuteCatchUpPlan for v2Core==nil path
- executeLegacyRebuild: wraps rt.ExecuteRebuildPlan for v2Core==nil path
- Clear "LEGACY NO-CORE COMPATIBILITY" section with structural separation
- runCatchUp/runRebuild now branch cleanly: legacy helper vs core coordinator

Test updates: pendingRecoveryExecution → rt.PendingExecution, field casing,
Plan type assertions.

Validation: all P4, P16B, and ApplyAssignments tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 01:20:41 -07:00
pingqiuandClaude Opus 4.6 8c2485e0e9 feat: Phase 15 + Phase 16A/B — V2 core integration + checkpoint review
Phase 15: V2 core wired into BlockService
- volume_server_block.go: v2Core field, applyCoreAssignmentEvent,
  core command executors (ApplyRole, StartReceiver, ConfigureShipper,
  InvalidateSession, StartCatchUp, StartRebuild, PublishProjection)
- Assignment processing now goes through core engine → command emission
  → bounded execution, replacing direct V1 replication setup
- master_block_registry.go: ClusterHealthSummary, VolumeMode in entries
- master_server_handlers_block.go: blockStatusHandler, entryToVolumeInfo
  refactored with entryReplicaSurface

Phase 16A: Core projection surfaces
Phase 16B: Bounded closure (checkpoint review ready)

Test fixes: add v2Core to manually-constructed BlockService in
idempotence, convergence, soak, and CP13-8A tests (required because
V1 replication setup paths now delegate to core engine).

All tests pass (21s regression).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 20:58:12 -07:00
Ping QiuandClaude Opus 4.6 0e234f5c80 feat: Phase 10 CP10-1 -- NVMe/TCP target MVP, 109 tests
NVMe over Fabrics (TCP) target implementation sharing the same BlockVol
engine, fencing, replication, and failover as the existing iSCSI target.

New package: weed/storage/blockvol/nvme/ (11 files, 2,242 production LOC)
- protocol.go: PDU types, opcodes, status codes, marshal/unmarshal
- wire.go: TCP reader/writer with header bounds validation
- controller.go: IC handshake, per-queue state, command dispatch, KATO
- fabric.go: Connect (admin+IO), PropertyGet/Set, Disconnect
- identify.go: Controller/Namespace/NS list/NS descriptors (Linux 5.15)
- admin.go: SetFeatures, GetFeatures, GetLogPage (SMART/ANA), KeepAlive
- io.go: Read (C2HData), Write (inline), Flush, WriteZeros/Trim
- server.go: TCP listener, admin session registry, graceful shutdown
- adapter.go: BlockVol-to-NVMe bridge, error mapping, ANA state

Integration: NVMeConfig + CLI flags (-block.nvme.*), disabled by default.

Key design: inline-data writes only (no R2T), MaxH2CDataLength=32KB,
single ANA group coherent with BlockVol role, CNTLID session registry
for cross-connection IO queues, HostNQN continuity enforcement.

Tests: 65 dev + 44 QA adversarial = 109 total, all passing.
Bugs fixed during review: IO queue cross-connection (A), header bounds
validation (B), write payload size check (C), disconnect error (D),
stream desync prevention (E), HostNQN enforcement (F), capsule-before-IC
state guard (H), flowCtlOff SQHD timing (I).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 16:52:37 -07:00
Ping QiuandClaude Opus 4.6 9acd187587 feat: Phase 8 complete -- CP8-5 stability gate, lease grant fix, Docker e2e, 13 chaos scenarios
Phase 8 closes with all 6 checkpoints done (CP8-1 through CP8-5 + CP8-3-1):
- CP8-5: 12/12 enterprise QA scenarios PASS on real hardware (m01/M02)
- Master-authoritative lease grants (BUG-CP85-11): master renews primary
  write leases on every heartbeat response, replacing retain-until-confirmed
  assignment queue semantics that caused 30s lease expiry
- Post-rebuild WAL shipping gap fix (BUG-CP85-1): syncLSNAfterRebuild
  advances replica nextLSN so WAL entries are accepted after rebuild
- Block heartbeat startup race fix (BUG-CP85-10): dynamic blockService
  check on each tick instead of one-shot at loop start
- 8 new tests: 4 engine lease grant + 4 registry lease grant
- 13 new YAML scenarios: chaos (kill-loop, partition, disk-full),
  database integrity (sqlite crash, ext4 fsck), perf baseline,
  metrics verify, snapshot stress, expand-failover, session storm,
  role flap, 24h soak
- 12 new testrunner actions (database, fsck, grep_log, write_loop_bg,
  stop_bg, assert_metric_gt/eq/lt) + phase repeat support
- Docker compose setup + getting-started guide for block storage users
- 960+ cumulative unit tests, 24 YAML scenarios

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 21:30:14 -08:00
Ping QiuandClaude Opus 4.6 8b2b5f6f66 feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests
Wire low-level fencing primitives to master/VS control plane and CSI:

- Proto: replica/rebuild address fields on assignment/info/response messages
- Assignment queue: retain-until-confirmed (Peek+Confirm), stale epoch pruning
- VS assignment receiver: processes assignments from HeartbeatResponse
- BlockService replication: ProcessAssignments, deterministic ports (FNV hash)
- Registry replica tracking: SetReplica/ClearReplica/SwapPrimaryReplica
- CreateBlockVolume: primary + replica, enqueues assignments, single-copy mode
- Failover: lease-aware promotion, deferred timers with cancellation on reconnect
- ControllerPublish: returns fresh primary iSCSI address after failover
- Recovery: recoverBlockVolumes drains pendingRebuilds, enqueues Rebuilding
- Real integration tests on M02: failover address switch, rebuild data
  consistency, full lifecycle failover+rebuild (3 tests, all PASS)

Review fixes (12 findings, 5 High, 5 Medium, 2 Low):
- R1-1: AllocateBlockVolume returns replication ports
- R1-2: setupPrimaryReplication starts rebuild server
- R1-3: VS sends periodic block heartbeat for assignment confirmation
- R2-F1: LastLeaseGrant set before Register (no stale-lease race)
- R2-F2: Deferred promotion timers cancelled on VS reconnect
- R2-F3: SwapPrimaryReplica uses RoleToWire instead of uint32(1)
- R2-F4: DeleteBlockVolume deletes replica (best-effort)
- R2-F5: SwapPrimaryReplica computes epoch atomically under lock
- QA: SetReplica removes old replica from byServer index (BUG-QA-CP63-1)

126 CP6-3 tests (67 dev + 48 QA + 8 integration + 3 real).
Cumulative Phase 6: 352 tests. All PASS.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 00:52:05 -08:00
Ping QiuandClaude Opus 4.6 80801b0fac feat: Phase 3 — performance tuning, iSCSI session refactor, store integration
Phase 3 delivers five checkpoints:

CP1 Engine Tuning: BlockVolConfig tunables, 256-shard DirtyMap, adaptive
group commit (low-watermark immediate flush), WAL pressure handling with
backpressure and ErrWALFull timeout.

CP2 iSCSI Session Refactor: RX/TX goroutine split with respCh (cap 64),
txLoop for serialized response writes, StatSN assignment modes. Login
phase stays single-goroutine; full-duplex after login.

CP3 Store Integration: BlockVolAdapter (iscsi.BlockDevice interface),
BlockVolumeStore management, BlockService in volume_server_block.go,
CLI flags (--block.listen/dir/iqn.prefix), sw-block-attach.sh helper.

CP5 Concurrency Hardening: WAL reuse guard (LSN validation in ReadLBA),
opsOutstanding counter with beginOp/endOp + Close drain, appendWithRetry
shared by WriteLBA and TrimLBA, flusher LSN guard in FlushOnce.

Bug fixes (P3-BUG-2–11): unbounded pending queue cap, Data-Out timeout,
flusher error logging, GroupCommitter panic recovery, Close vs concurrent
ops guard, target shutdown race, WAL-full retry vs Close, WRITE SAME(16)
for XFS, MODE SENSE(10) + VPD 0xB0/0xB2 for Linux kernel compatibility.

797 tests passing (517 engine + 280 iSCSI), go vet clean.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 10:43:34 -08:00