pingqiuandClaude Opus 4.6 ebe95b6e2e fix: flusher OOM on multi-block writes + testrunner enhancements ...

Bug: flusher.go:336 allocated make([]byte, entryLen) per dirty block
instead of per unique WAL entry. A 4MB WriteLBA creates 1024 dirty map
entries (one per 4KB block), all sharing the same WAL offset. The flusher
read the full 4MB WAL entry 1024 times into separate buffers:
1024 × 4MB = 4GB per 4MB write → OOM on mkfs.ext4.

Root cause: flusher assumed 1:1 dirty-block-to-WAL-entry mapping.
WriteLBA supports multi-block writes but the flusher never deduplicated
shared WAL offsets.

Fix: deduplicate WAL reads by WalOffset in flushOnceLocked(). Multiple
dirty blocks from the same WAL entry share one read buffer and one
DecodeWALEntry call. Memory: O(WAL_entries × size) not O(blocks × size).
For a 4MB write: 4GB → 4MB.

Verified on hardware (m01/M02 25Gbps RoCE):
- Before: mkfs.ext4 → VS RSS 100MB→25GB → OOM killed
- After: mkfs.ext4 → VS RSS 129MB stable, mkfs succeeds
- pgbench TPC-B c=4: 1,248 TPS (RF=1, previously blocked by OOM)

Tests added:
- flusher_test.go: flush_multiblock_shared_wal_read (16 blocks share
  one WAL offset, flush dedup verified)
- flusher_test.go: flush_multiblock_data_correct (3 mixed multi-block
  writes, all data correct after flush)
- test/component/large_write_test.go: 7 component tests (single 4MB,
  sequential mkfs sim, concurrent, mixed sizes, production volume,
  flusher throughput 30s sustained)
- iscsi/large_write_mem_test.go: 2 iSCSI session memory tests (4MB
  R2T flow, slow device)

Testrunner enhancements (same commit — all tested on hardware):
- discover_primary action: maps primary IP → topology node name,
  supports alt_ips for multi-NIC (RoCE + management)
- NodeSpec.AltIPs field for multi-NIC node identification
- 5 new YAML scenarios: ec3, ec5, degraded sync_all/best_effort, pgbench
- All 13 hardware-verified scenarios PASS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-02 14:24:10 -07:00

..

batchio

fix: require explicit build tags for io_uring backends, add implementation logging

2026-03-10 18:19:31 -07:00

blockapi

feat: CP11B-2 explainable placement / plan API

2026-03-23 02:12:25 -07:00

blockerr

feat: CP8-3-1 durability modes + testrunner platform + 21 adversarial tests

2026-03-06 01:06:51 -08:00

csi

feat: master-backed NVMe/TCP publication (nvme_addr + nqn plumbing)

2026-03-11 17:35:43 -07:00

iscsi

fix: flusher OOM on multi-block writes + testrunner enhancements

2026-04-02 14:24:10 -07:00

monitoring

feat: CP11B-4 observability pack — health state, alerts, dashboard

2026-03-23 02:12:42 -07:00

nvme

feat: CP10B-1 NVMe/TCP RX/TX split + CP10B-2 bench/profiling fixes

2026-03-10 15:09:41 -07:00

operator

feat: Phase 10 CP10-3 -- NVMe/TCP Tier 1 optimizations, WAL admission control, benchmark platform

2026-03-09 17:44:01 -07:00

scripts

feat: Phase 4A CP4a -- simulated master, assignment sequence tests, BlockVolumeStatus

2026-03-02 09:33:45 -08:00

test

fix: flusher OOM on multi-block writes + testrunner enhancements

2026-04-02 14:24:10 -07:00

testrunner

fix: flusher OOM on multi-block writes + testrunner enhancements

2026-04-02 14:24:10 -07:00

v2bridge

fix: require CatchUp outcome, true simultaneous overlap, observability assertions

2026-03-31 15:55:37 -07:00

adapter_test.go

feat: Phase 6 CP6-2 -- CSI control-plane integration + csi-sanity/k3s validation

2026-03-04 11:01:08 -08:00

adapter.go

feat: Phase 6 CP6-2 -- CSI control-plane integration + csi-sanity/k3s validation

2026-03-04 11:01:08 -08:00

block_heartbeat_proto_test.go

feat: rebuild fallback with per-replica heartbeat state (CP13-7)

2026-03-26 16:46:31 -07:00

block_heartbeat_proto.go

feat: master-backed NVMe/TCP publication (nvme_addr + nqn plumbing)

2026-03-11 17:35:43 -07:00

block_heartbeat_test.go

feat: Phase 4A CP4b-1 -- wire types, conversion helpers, heartbeat collection

2026-03-02 09:34:00 -08:00

block_heartbeat.go

fix: stable ServerID in assignments, fail-closed on missing identity, wire into ProcessAssignments

2026-03-31 10:46:17 -07:00

blockvol_qa_test.go

fix: remove nil-optional superMu pattern, require in all FlusherConfigs

2026-03-24 00:19:25 -07:00

blockvol_test.go

feat: reconnect handshake + WAL catch-up protocol (CP13-5)

2026-03-25 15:38:06 -07:00

blockvol.go

feat: separate CommittedLSN from CheckpointLSN, close catch-up ONE CHAIN (Phase 08 P2)

2026-03-31 15:22:23 -07:00

bug_gc_panic_test.go

feat: Phase 3 — performance tuning, iSCSI session refactor, store integration

2026-03-01 10:43:34 -08:00

config_test.go

feat: Phase 10 CP10-3 -- NVMe/TCP Tier 1 optimizations, WAL admission control, benchmark platform

2026-03-09 17:44:01 -07:00

config.go

feat: tri-state IOBackend config with explicit logging and CLI flag

2026-03-10 16:02:30 -07:00

dirty_map_test.go

feat: Phase 3 — performance tuning, iSCSI session refactor, store integration

2026-03-01 10:43:34 -08:00

dirty_map.go

feat: Phase 4A CP3 -- promotion, rebuild, split-brain prevention

2026-03-01 21:26:39 -08:00

dist_group_commit_test.go

feat: reconnect handshake + WAL catch-up protocol (CP13-5)

2026-03-25 15:38:06 -07:00

dist_group_commit.go

feat: replica state machine + barrier eligibility gating (CP13-4)

2026-03-25 02:39:32 -07:00

durability_mode_test.go

feat: CP8-3-1 durability modes + testrunner platform + 21 adversarial tests

2026-03-06 01:06:51 -08:00

durability_mode.go

feat: CP8-3-1 durability modes + testrunner platform + 21 adversarial tests

2026-03-06 01:06:51 -08:00

engine_metrics.go

feat: CP11A-3 WAL hardening foundations — pressure visibility, sizing guidance, preflight

2026-03-12 19:30:59 -07:00

epoch.go

feat: Phase 4A CP1 — epoch, lease, role state machine, write gate

2026-03-01 16:00:06 -08:00

expand_test.go

feat: CP11A-2 coordinated expand protocol for replicated block volumes

2026-03-12 15:06:48 -07:00

flusher_test.go

fix: flusher OOM on multi-block writes + testrunner enhancements

2026-04-02 14:24:10 -07:00

flusher.go

fix: flusher OOM on multi-block writes + testrunner enhancements

2026-04-02 14:24:10 -07:00

group_commit_test.go

feat: Phase 3 — performance tuning, iSCSI session refactor, store integration

2026-03-01 10:43:34 -08:00

group_commit.go

feat: Phase 8 CP8-1/2/3/4 -- ops control plane, multi-replica, CSI snapshots, observability

2026-03-06 00:05:17 -08:00

health_score_test.go

feat: Phase 8 CP8-1/2/3/4 -- ops control plane, multi-replica, CSI snapshots, observability

2026-03-06 00:05:17 -08:00

health_score.go

feat: Phase 8 CP8-1/2/3/4 -- ops control plane, multi-replica, CSI snapshots, observability

2026-03-06 00:05:17 -08:00

lba_test.go

feat: add BlockVol engine and iSCSI target (Phase 1 + Phase 2)

2026-02-28 09:32:01 -08:00

lba.go

feat: add BlockVol engine and iSCSI target (Phase 1 + Phase 2)

2026-02-28 09:32:01 -08:00

lease_grant_test.go

feat: Phase 8 complete -- CP8-5 stability gate, lease grant fix, Docker e2e, 13 chaos scenarios

2026-03-07 21:30:14 -08:00

lease.go

feat: Phase 4A CP1 — epoch, lease, role state machine, write gate

2026-03-01 16:00:06 -08:00

naming_test.go

feat: Phase 6 CP6-2 -- CSI control-plane integration + csi-sanity/k3s validation

2026-03-04 11:01:08 -08:00

naming.go

feat: Phase 10 CP10-2 -- CSI NVMe/TCP node plugin, 210 tests

2026-03-08 23:02:59 -07:00

net_util_test.go

feat: wire real pinner into flusher retention + real WAL scan executor (Phase 07 P1)

2026-03-30 20:01:46 -07:00

net_util.go

fix: canonicalize host in AllocateBlockVolumeResponse (CP13-2 follow-up)

2026-03-26 19:16:45 -07:00

preset_test.go

feat: CP11B-1 provisioning presets + review fixes

2026-03-22 14:44:24 -07:00

preset.go

feat: CP11B-1 provisioning presets + review fixes

2026-03-22 14:44:24 -07:00

promotion.go

feat: Phase 8 complete -- CP8-5 stability gate, lease grant fix, Docker e2e, 13 chaos scenarios

2026-03-07 21:30:14 -08:00

qa_expand_test.go

feat: CP11A-2 coordinated expand protocol for replicated block volumes

2026-03-12 15:06:48 -07:00

qa_iobackend_config_test.go

feat: CP11A-2 coordinated expand protocol for replicated block volumes

2026-03-12 15:06:48 -07:00

qa_phase3_engine_test.go

feat: Phase 4A CP1 — epoch, lease, role state machine, write gate

2026-03-01 16:00:06 -08:00

qa_phase4a_cp1_test.go

feat: Phase 4A CP4a -- simulated master, assignment sequence tests, BlockVolumeStatus

2026-03-02 09:33:45 -08:00

qa_phase4a_cp2_test.go

feat: reconnect handshake + WAL catch-up protocol (CP13-5)

2026-03-25 15:38:06 -07:00

qa_phase4a_cp3_test.go

feat: Phase 10 CP10-3 -- NVMe/TCP Tier 1 optimizations, WAL admission control, benchmark platform

2026-03-09 17:44:01 -07:00

qa_phase4a_cp4a_test.go

feat: Phase 4A CP4a -- simulated master, assignment sequence tests, BlockVolumeStatus

2026-03-02 09:33:45 -08:00

qa_phase4a_cp4b1_test.go

test: Phase 4A CP4b-1 QA adversarial tests (19 tests)

2026-03-02 09:34:35 -08:00

qa_phase4a_cp4b4_test.go

feat: reconnect handshake + WAL catch-up protocol (CP13-5)

2026-03-25 15:38:06 -07:00

qa_resize_test.go

test: Phase 5 QA adversarial tests -- 49 tests for CHAP, resize, snapshots

2026-03-03 17:05:43 -08:00

qa_snapshot_export_adversarial_test.go

fix: CP11A-4 snapshot export/import safety — 3 bugs from review

2026-03-13 10:56:18 -07:00

qa_snapshot_export_test.go

feat: CP11A-4 snapshot export/import to S3 — artifact format, engine, and transport

2026-03-13 00:15:27 -07:00

qa_snapshot_test.go

test: Phase 5 QA adversarial tests -- 49 tests for CHAP, resize, snapshots

2026-03-03 17:05:43 -08:00

qa_storage_profile_test.go

feat: CP11A-2 coordinated expand protocol for replicated block volumes

2026-03-12 15:06:48 -07:00

qa_wal_admission_test.go

chore: Phase 10 remaining — QA WAL admission metrics tests

2026-03-11 17:35:44 -07:00

qa_wal_cp11a3_adversarial_test.go

feat: CP11B-3 safe ops — promotion hardening, preflight, manual promote

2026-03-13 17:21:17 -07:00

qa_wal_hardening_test.go

feat: CP11A-3 WAL hardening foundations — pressure visibility, sizing guidance, preflight

2026-03-12 19:30:59 -07:00

rebuild_v1_test.go

fix: sync flusher checkpointLSN after rebuild (CP13-7)

2026-03-26 17:22:55 -07:00

rebuild.go

fix: sync flusher checkpointLSN after rebuild (CP13-7)

2026-03-26 17:22:55 -07:00

recovery_test.go

feat: wire real pinner into flusher retention + real WAL scan executor (Phase 07 P1)

2026-03-30 20:01:46 -07:00

recovery.go

feat: wire real pinner into flusher retention + real WAL scan executor (Phase 07 P1)

2026-03-30 20:01:46 -07:00

repl_proto.go

feat: reconnect handshake + WAL catch-up protocol (CP13-5)

2026-03-25 15:38:06 -07:00

replica_apply.go

feat: wire real pinner into flusher retention + real WAL scan executor (Phase 07 P1)

2026-03-30 20:01:46 -07:00

replica_barrier.go

feat: durable progress truth — replicaFlushedLSN in barrier (CP13-3)

2026-03-25 01:52:35 -07:00

resize_test.go

feat: Phase 5 CP5-3 -- CHAP auth, online resize, Prometheus metrics, 12 tests

2026-03-03 01:43:42 -08:00

role.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

scrub_test.go

feat: Phase 8 CP8-1/2/3/4 -- ops control plane, multi-replica, CSI snapshots, observability

2026-03-06 00:05:17 -08:00

scrub.go

feat: Phase 8 CP8-1/2/3/4 -- ops control plane, multi-replica, CSI snapshots, observability

2026-03-06 00:05:17 -08:00

shipper_group_test.go

feat: replica state machine + barrier eligibility gating (CP13-4)

2026-03-25 02:39:32 -07:00

shipper_group.go

fix: stable ServerID in assignments, fail-closed on missing identity, wire into ProcessAssignments

2026-03-31 10:46:17 -07:00

snapshot_export_test.go

fix: CP11A-4 snapshot export/import safety — 3 bugs from review

2026-03-13 10:56:18 -07:00

snapshot_export.go

fix: ioMu data-plane exclusion for restore/import/expand

2026-03-23 20:40:41 -07:00

snapshot_test.go

feat: Phase 5 CP5-2 -- CoW snapshots, 10 tests

2026-03-03 00:53:33 -08:00

snapshot.go

feat: Phase 5 CP5-2 -- CoW snapshots, 10 tests

2026-03-03 00:53:33 -08:00

storage_profile_test.go

feat: CP11A-1 storage profile type, superblock persistence, and validation

2026-03-11 21:52:00 -07:00

storage_profile.go

feat: CP11A-1 storage profile type, superblock persistence, and validation

2026-03-11 21:52:00 -07:00

superblock_test.go

feat: CP8-3-1 durability modes + testrunner platform + 21 adversarial tests

2026-03-06 01:06:51 -08:00

superblock.go

feat: CP11A-2 coordinated expand protocol for replicated block volumes

2026-03-12 15:06:48 -07:00

sync_all_adversarial_test.go

feat: wire real pinner into flusher retention + real WAL scan executor (Phase 07 P1)

2026-03-30 20:01:46 -07:00

sync_all_bug_test.go

feat: reconnect handshake + WAL catch-up protocol (CP13-5)

2026-03-25 15:38:06 -07:00

sync_all_protocol_test.go

feat: wire real pinner into flusher retention + real WAL scan executor (Phase 07 P1)

2026-03-30 20:01:46 -07:00

wal_admission_test.go

feat: CP11A-3 WAL hardening foundations — pressure visibility, sizing guidance, preflight

2026-03-12 19:30:59 -07:00

wal_admission.go

feat: CP11A-3 WAL hardening foundations — pressure visibility, sizing guidance, preflight

2026-03-12 19:30:59 -07:00

wal_entry_test.go

feat: add BlockVol engine and iSCSI target (Phase 1 + Phase 2)

2026-02-28 09:32:01 -08:00

wal_entry.go

feat: add BlockVol engine and iSCSI target (Phase 1 + Phase 2)

2026-02-28 09:32:01 -08:00

wal_guidance_test.go

feat: CP11A-3 WAL hardening foundations — pressure visibility, sizing guidance, preflight

2026-03-12 19:30:59 -07:00

wal_guidance.go

feat: CP11A-3 WAL hardening foundations — pressure visibility, sizing guidance, preflight

2026-03-12 19:30:59 -07:00

wal_shipper.go

fix: stable ServerID in assignments, fail-closed on missing identity, wire into ProcessAssignments

2026-03-31 10:46:17 -07:00

wal_writer_test.go

feat: Phase 3 — performance tuning, iSCSI session refactor, store integration

2026-03-01 10:43:34 -08:00

wal_writer.go

feat: Phase 5 CP5-2 -- CoW snapshots, 10 tests

2026-03-03 00:53:33 -08:00

write_gate.go

feat: Phase 4A CP1 — epoch, lease, role state machine, write gate

2026-03-01 16:00:06 -08:00