pingqiu and Claude Opus 4.6
ebe95b6e2e
fix: flusher OOM on multi-block writes + testrunner enhancements
...
Bug: flusher.go:336 allocated make([]byte, entryLen) per dirty block
instead of per unique WAL entry. A 4MB WriteLBA creates 1024 dirty map
entries (one per 4KB block), all sharing the same WAL offset. The flusher
read the full 4MB WAL entry 1024 times into separate buffers:
1024 × 4MB = 4GB per 4MB write → OOM on mkfs.ext4.
Root cause: flusher assumed 1:1 dirty-block-to-WAL-entry mapping.
WriteLBA supports multi-block writes but the flusher never deduplicated
shared WAL offsets.
Fix: deduplicate WAL reads by WalOffset in flushOnceLocked(). Multiple
dirty blocks from the same WAL entry share one read buffer and one
DecodeWALEntry call. Memory: O(WAL_entries × size) not O(blocks × size).
For a 4MB write: 4GB → 4MB.
Verified on hardware (m01/M02 25Gbps RoCE):
- Before: mkfs.ext4 → VS RSS 100MB→25GB → OOM killed
- After: mkfs.ext4 → VS RSS 129MB stable, mkfs succeeds
- pgbench TPC-B c=4: 1,248 TPS (RF=1, previously blocked by OOM)
Tests added:
- flusher_test.go: flush_multiblock_shared_wal_read (16 blocks share
one WAL offset, flush dedup verified)
- flusher_test.go: flush_multiblock_data_correct (3 mixed multi-block
writes, all data correct after flush)
- test/component/large_write_test.go: 7 component tests (single 4MB,
sequential mkfs sim, concurrent, mixed sizes, production volume,
flusher throughput 30s sustained)
- iscsi/large_write_mem_test.go: 2 iSCSI session memory tests (4MB
R2T flow, slow device)
Testrunner enhancements (same commit — all tested on hardware):
- discover_primary action: maps primary IP → topology node name,
supports alt_ips for multi-NIC (RoCE + management)
- NodeSpec.AltIPs field for multi-NIC node identification
- 5 new YAML scenarios: ec3, ec5, degraded sync_all/best_effort, pgbench
- All 13 hardware-verified scenarios PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-04-02 14:24:10 -07:00
Ping Qiu and Claude Opus 4.6
e8c921d9e8
fix: remove nil-optional superMu pattern, require in all FlusherConfigs
...
superMu is mandatory for correctness — all superblock mutation+persist
must be serialized. Remove the nil guard in updateSuperblockCheckpoint
and add SuperMu to all 7 test FlusherConfig sites.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-03-24 00:19:25 -07:00
Ping Qiu and Claude Opus 4.6
09c7e40d29
feat: Phase 4A CP4a -- simulated master, assignment sequence tests, BlockVolumeStatus
...
Add SimulatedMaster test helper + 20 assignment sequence tests (8 sequence,
5 failover, 5 adversarial, 2 status). Add BlockVolumeStatus struct and
Status() method. Includes QA test files for CP1-CP4a. 940 total unit tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-03-02 09:33:45 -08:00
Ping Qiu and Claude Opus 4.6
80801b0fac
feat: Phase 3 — performance tuning, iSCSI session refactor, store integration
...
Phase 3 delivers five checkpoints:
CP1 Engine Tuning: BlockVolConfig tunables, 256-shard DirtyMap, adaptive
group commit (low-watermark immediate flush), WAL pressure handling with
backpressure and ErrWALFull timeout.
CP2 iSCSI Session Refactor: RX/TX goroutine split with respCh (cap 64),
txLoop for serialized response writes, StatSN assignment modes. Login
phase stays single-goroutine; full-duplex after login.
CP3 Store Integration: BlockVolAdapter (iscsi.BlockDevice interface),
BlockVolumeStore management, BlockService in volume_server_block.go,
CLI flags (--block.listen/dir/iqn.prefix), sw-block-attach.sh helper.
CP5 Concurrency Hardening: WAL reuse guard (LSN validation in ReadLBA),
opsOutstanding counter with beginOp/endOp + Close drain, appendWithRetry
shared by WriteLBA and TrimLBA, flusher LSN guard in FlushOnce.
Bug fixes (P3-BUG-2–11): unbounded pending queue cap, Data-Out timeout,
flusher error logging, GroupCommitter panic recovery, Close vs concurrent
ops guard, target shutdown race, WAL-full retry vs Close, WRITE SAME(16)
for XFS, MODE SENSE(10) + VPD 0xB0/0xB2 for Linux kernel compatibility.
797 tests passing (517 engine + 280 iSCSI), go vet clean.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-03-01 10:43:34 -08:00
Ping Qiu and Claude Opus 4.6
6a400f6760
feat: add BlockVol engine and iSCSI target (Phase 1 + Phase 2)
...
Phase 1: Extent-mapped block storage engine with WAL, crash recovery,
dirty map, flusher, and group commit. 174 tests, zero SeaweedFS imports.
Phase 2: Pure Go iSCSI target (RFC 7143) with PDU codec, login
negotiation, SendTargets discovery, 12 SCSI opcodes, Data-In/Out/R2T
sequencing, session management, and standalone iscsi-target binary.
164 tests. IQN->BlockDevice binding via DeviceLookup interface.
Total: 338 tests, 14.6K lines.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-02-28 09:32:01 -08:00
