Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-10-10 16:27:47 +02:00
Code Issues Packages Projects Releases Wiki Activity
13,089 Commits 202 Branches 361 Tags
512bb5bcf6178e1663dcafb9c75637cdb1c25205
Commit Graph
18 Commits
Author SHA1 Message Date
Ping QiuandClaude Opus 4.6 ed11a09a61 fix: CP11A-4 snapshot export/import safety — 3 bugs from review 
BUG-CP11A4-1 (HIGH): ImportSnapshot now rejects when active snapshots
exist. Import overwrites the extent region that non-CoW'd snapshot blocks
read from, which would silently return import data instead of snapshot-time
data. New ErrImportActiveSnapshots error and snapMu-guarded check.

BUG-CP11A4-2 (HIGH): Double import without AllowOverwrite now correctly
rejected. Import bypasses WAL so nextLSN stays at 1; added FlagImported
(Superblock.Flags bit 0) set after successful import and checked alongside
nextLSN in the non-empty gate.

BUG-CP11A4-3 (MED): Replaced fixed exportTempSnapID (0xFFFFFFFE) with
atomic sequence counter (exportTempSnapBase + exportTempSnapSeq). Each
auto-export gets a unique temp snapshot ID, preventing concurrent export
races and user snapshot ID collisions.

Also added beginOp()/endOp() lifecycle guards to both ExportSnapshot and
ImportSnapshot, and documented the non-atomic import failure semantics.

5 new regression tests + QA-EX-3 rewritten for rejection behavior.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-13 10:56:18 -07:00
Ping QiuandClaude Opus 4.6 7cc6467d09 feat: CP11A-4 snapshot export/import to S3 — artifact format, engine, and transport 
Add crash-consistent snapshot export/import for single-profile block volumes.
Export creates a temp snapshot, streams the full volume image with inline
SHA-256, and uploads to S3. Import validates manifest + checksum and writes
directly to extent region. Admin HTTP endpoints /export and /import added
to the standalone iscsi-target binary.

Engine: snapshot_export.go (manifest types, ExportSnapshot, ImportSnapshot)
S3: snapshot_s3.go (AWS SDK v1 transport, pipe-based streaming upload)
Tests: 14 engine + 9 QA adversarial = 23 new tests, all passing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-13 00:15:27 -07:00
Ping QiuandClaude Opus 4.6 1c5b658170 feat: CP11A-3 WAL hardening foundations — pressure visibility, sizing guidance, preflight 
Add PressureState() and writer wait tracking to WALAdmission, WALStatus
snapshot API on BlockVol, WAL sizing guidance pure functions, Prometheus
histogram/gauge/counter exports, and admin /status WAL fields. 23 new
tests (7 admission, 10 guidance, 6 QA adversarial).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-12 19:30:59 -07:00
Ping QiuandClaude Opus 4.6 f698b1f154 fix: reject IOBackend=io_uring in Validate(), fix wal_admit_wait metric type 
Finding 1: IOBackend=io_uring was accepted and logged as resolved but
had no runtime effect. Now rejected by Validate() until actually wired,
preventing user confusion.

Finding 2: wal_admit_wait_seconds_total was exported as GaugeFunc but
is monotonically increasing. Changed to CounterFunc to match _total
naming convention.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-11 17:35:43 -07:00
Ping QiuandClaude Opus 4.6 e22e57a3f7 feat: WAL admission metrics for visibility into write pressure behavior 
Add counters (total, soft, hard, timeout) and wait-time histogram to
WALAdmission, wired through EngineMetrics and exported as Prometheus
metrics. Six new tests verify all code paths. Nil-safe for backwards
compatibility.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-11 17:34:58 -07:00
Ping QiuandClaude Opus 4.6 9d0ec8efa3 feat: tri-state IOBackend config with explicit logging and CLI flag 
Replace UseIOUring bool with IOBackend IOBackendMode (tri-state):
- "standard" (default): sequential pread/pwrite/fdatasync
- "auto": try io_uring, fall back to standard with warning log
- "io_uring": require io_uring, fail startup if unavailable

NewIOUring now returns ErrIOUringUnavailable instead of silently
falling back — callers decide whether to fail or fall back based
on the requested mode. All mode transitions are logged:
  io backend: requested=auto selected=standard reason=...
  io backend: requested=io_uring selected=io_uring

CLI: --io-backend=standard|auto|io_uring added to iscsi-target.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-10 16:02:30 -07:00
Ping QiuandClaude Opus 4.6 3557ae283f feat: Phase 10 CP10-3 -- NVMe/TCP Tier 1 optimizations, WAL admission control, benchmark platform 
CP10-3 Tier 1 optimizations (T1-T4):
- TCP_NODELAY + 256KB socket buffers on NVMe/TCP connections
- Response batching: all C2H data chunks + CapsuleResp in single flush
- Tiered buffer pool (4KB/64KB/256KB sync.Pool) for write payloads
- Configurable MaxH2CDataLength wiring through controller/IC/chunking

BUG-CP103-1: NVMe write retry with jittered backoff for transient WAL pressure
- writeWithRetry() with bounded backoff [50/200/800ms]
- throttleOnWALPressure() pre-write delay above 90% WAL usage
- WALPressureProvider interface + NVMeAdapter.WALPressure()

BUG-CP103-2: Volume-level WAL admission control
- WALAdmission with counting semaphore (max concurrent writers)
- Soft watermark (0.7): small delay to desynchronize herd
- Hard watermark (0.9): block until flusher drains
- Single-deadline budget shared across watermark wait + semaphore
- Close-aware during both watermark and semaphore waits
- Wired into BlockVol.WriteLBA() and Trim()

Benchmark platform enhancements:
- NVMe benchmark actions and scenarios (A/B, CW sweep, IOQ sweep)
- Database benchmark actions (SQLite, pgbench)
- K8s operator QA reconciler tests
- New testrunner scenarios for HA, fault injection, CSI lifecycle

Test counts: 213 NVMe + 625 engine + operator + testrunner tests, all passing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-09 17:44:01 -07:00
Ping QiuandClaude Opus 4.6 9acd187587 feat: Phase 8 complete -- CP8-5 stability gate, lease grant fix, Docker e2e, 13 chaos scenarios 
Phase 8 closes with all 6 checkpoints done (CP8-1 through CP8-5 + CP8-3-1):
- CP8-5: 12/12 enterprise QA scenarios PASS on real hardware (m01/M02)
- Master-authoritative lease grants (BUG-CP85-11): master renews primary
  write leases on every heartbeat response, replacing retain-until-confirmed
  assignment queue semantics that caused 30s lease expiry
- Post-rebuild WAL shipping gap fix (BUG-CP85-1): syncLSNAfterRebuild
  advances replica nextLSN so WAL entries are accepted after rebuild
- Block heartbeat startup race fix (BUG-CP85-10): dynamic blockService
  check on each tick instead of one-shot at loop start
- 8 new tests: 4 engine lease grant + 4 registry lease grant
- 13 new YAML scenarios: chaos (kill-loop, partition, disk-full),
  database integrity (sqlite crash, ext4 fsck), perf baseline,
  metrics verify, snapshot stress, expand-failover, session storm,
  role flap, 24h soak
- 12 new testrunner actions (database, fsck, grep_log, write_loop_bg,
  stop_bg, assert_metric_gt/eq/lt) + phase repeat support
- Docker compose setup + getting-started guide for block storage users
- 960+ cumulative unit tests, 24 YAML scenarios

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-07 21:30:14 -08:00
Ping QiuandClaude Opus 4.6 979a9b496c feat: Phase 8 CP8-1/2/3/4 -- ops control plane, multi-replica, CSI snapshots, observability 
CP8-1: HTTP REST API (create/delete/lookup/list/assign/servers), blockapi Go
client with multi-master failover, 5 shell commands, HTML dashboard at /block/.

CP8-2: RF=2/RF=3 multi-replica support -- ShipperGroup fan-out, distributed
sync, health scoring, segment-based scrub, gated promotion (heartbeat
freshness + WAL LSN + role checks), failover/rebuild for N>2 replicas.

CP8-3: CSI snapshot + expansion -- CreateSnapshot/DeleteSnapshot/ListSnapshots
RPCs, NodeExpandVolume with iSCSI rescan, snapshot ID helpers, 20 adversarial
tests covering concurrent ops, edge cases, and error injection.

CP8-4: Observability -- EngineMetrics atomic counters for flusher/group-commit/
WAL-shipper/scrub, 10 new Prometheus metrics, barrier_lag_lsn SLO gauge,
failover/promotion/rebuild counters, request ID correlation in master gRPC
logs, baseline regression framework with 7 hard-fail conditions.

Total: 63 files, ~11.2K LOC, 160+ new tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-06 00:05:17 -08:00
Ping QiuandClaude Opus 4.6 8b2b5f6f66 feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests 
Wire low-level fencing primitives to master/VS control plane and CSI:

- Proto: replica/rebuild address fields on assignment/info/response messages
- Assignment queue: retain-until-confirmed (Peek+Confirm), stale epoch pruning
- VS assignment receiver: processes assignments from HeartbeatResponse
- BlockService replication: ProcessAssignments, deterministic ports (FNV hash)
- Registry replica tracking: SetReplica/ClearReplica/SwapPrimaryReplica
- CreateBlockVolume: primary + replica, enqueues assignments, single-copy mode
- Failover: lease-aware promotion, deferred timers with cancellation on reconnect
- ControllerPublish: returns fresh primary iSCSI address after failover
- Recovery: recoverBlockVolumes drains pendingRebuilds, enqueues Rebuilding
- Real integration tests on M02: failover address switch, rebuild data
  consistency, full lifecycle failover+rebuild (3 tests, all PASS)

Review fixes (12 findings, 5 High, 5 Medium, 2 Low):
- R1-1: AllocateBlockVolume returns replication ports
- R1-2: setupPrimaryReplication starts rebuild server
- R1-3: VS sends periodic block heartbeat for assignment confirmation
- R2-F1: LastLeaseGrant set before Register (no stale-lease race)
- R2-F2: Deferred promotion timers cancelled on VS reconnect
- R2-F3: SwapPrimaryReplica uses RoleToWire instead of uint32(1)
- R2-F4: DeleteBlockVolume deletes replica (best-effort)
- R2-F5: SwapPrimaryReplica computes epoch atomically under lock
- QA: SetReplica removes old replica from byServer index (BUG-QA-CP63-1)

126 CP6-3 tests (67 dev + 48 QA + 8 integration + 3 real).
Cumulative Phase 6: 352 tests. All PASS.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 00:52:05 -08:00
Ping QiuandClaude Opus 4.6 5a9a52f2d0 feat: Phase 6 CP6-2 -- CSI control-plane integration + csi-sanity/k3s validation 
CP6-2 wires the CSI driver to SeaweedFS master/volume-server control plane:
- Proto: block volume messages in master.proto/volume_server.proto, codegen
- Master registry: in-memory BlockVolumeRegistry with Pending->Active status,
  full/delta heartbeat, inflight lock, placement (fewest volumes)
- VS gRPC: AllocateBlockVolume/DeleteBlockVolume handlers, shared naming
- Master RPCs: CreateBlockVolume (retry up to 3 servers), Delete, Lookup
- Heartbeat: block volume fields wired into bidirectional stream
- CSI Controller: VolumeBackend interface (Local + Master), returns volume_context
- CSI Node: reads volume_context for remote targets, staged map + IQN derivation
- Mode flag: --mode=controller/node/all, --master for control-plane
- K8s manifests: csi-driver.yaml, csi-controller.yaml, csi-node.yaml

csi-sanity conformance (33 pass, 58 skip) found 6 bugs:
- BUG-SANITY-1/2/3: missing VolumeCapabilities/VolumeCapability validation
- BUG-SANITY-4: NodePublish used mount instead of bind mount
- BUG-SANITY-5: NodeUnpublish didn't remove target path
- BUG-SANITY-6: NodeUnpublish failed on unmounted path

k3s Level 4 (PVC->Pod data persistence) found 1 bug:
- BUG-K3S-1: IsLoggedIn didn't handle iscsiadm exit code 21

226 CSI tests + 54 server tests = 280 new tests, all passing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-04 11:01:08 -08:00
Ping QiuandClaude Opus 4.6 531ee764ee feat: Phase 5 CP5-3 -- CHAP auth, online resize, Prometheus metrics, 12 tests 
CHAP authentication (RFC 7143 S12.1):
- auth.go: CHAPAuthenticator with MD5 challenge-response, ValidateCHAPConfig
- login.go: multi-PDU SecurityNeg flow (challenge → verify → transit)
- main.go: -chap-user/-chap-secret CLI flags with validation

Online volume expand:
- blockvol.go: Expand() with flusher pause, snapMu TOCTOU guard, alignment check
- Rejects shrink (ErrShrinkNotSupported) and resize with active snapshots

Prometheus metrics:
- metrics.go: metricsAdapter wrapping BlockDevice, 15 metrics (counters,
  histograms, gauge funcs for WAL/dirty-map/epoch/role/snapshots)
- Dedicated prometheus.NewRegistry() per server instance

Admin HTTP endpoints:
- POST /snapshot (create/delete/restore/list)
- POST /resize (online expand)
- GET /metrics (Prometheus text format)
- VolumeSize added to /status response

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-03 01:43:42 -08:00
Ping QiuandClaude Opus 4.6 98d0e9e631 feat: Phase 5 CP5-1 -- ALUA + multipath failover, 28 tests 
Add ALUA (Asymmetric Logical Unit Access) support to the iSCSI target,
enabling dm-multipath on Linux to automatically detect path state changes
and reroute I/O during HA failover without initiator-side intervention.

- ALUAProvider interface with implicit ALUA (TPGS=0x01)
- INQUIRY byte 5 TPGS bits, VPD 0x83 with NAA+TPG+RTP descriptors
- REPORT TARGET PORT GROUPS handler (MAINTENANCE IN SA=0x0A)
- MAINTENANCE OUT rejection (implicit-only, no SET TPG)
- Standby write rejection (NOT_READY ASC=04h ASCQ=0Bh)
- RoleNone maps to Active/Optimized (standalone single-node compatibility)
- NAA-6 device identifier derived from volume UUID
- -tpg-id flag with [1,65535] validation
- dm-multipath config + setup script (group_by_tpg, ALUA prio)
- 12 unit tests + 16 QA adversarial tests + 4 integration tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 23:29:58 -08:00
Ping QiuandClaude Opus 4.6 7940e6b7c9 feat: Phase 4A CP4b-4 Windows iSCSI + instrumentation + QA tests 
Windows iSCSI Initiator compatibility:
- Add TargetPortalGroupTag to login response (RFC 7143 S13.9)
- Add REQUEST_SENSE, START_STOP_UNIT, MODE_SELECT(6/10) handlers
- Add PERSISTENT_RESERVE_IN/OUT, MAINTENANCE_IN (REPORT SUPPORTED OPCODES)
- Implement MODE SENSE caching (page 0x08) and control (page 0x0A) pages
- Fix Data-In residual underflow/overflow flags (U/O bits on final PDU)
- Rename ScsiReadCapacity16 -> ScsiServiceActionIn16 for correctness

Instrumentation and tooling:
- Add instrumentedAdapter with periodic PERF stats logging
- Add pprof endpoints on admin HTTP server (/debug/pprof/*)
- Add blockbench CLI tool for standalone block device benchmarking
- Add SCSI CDB debug logging in session dispatch

HA integration fixes:
- Move HA test replica ports to 9011-9014 to avoid conflicts
- Add QA adversarial tests for Phase 4A CP4b-4 (755 lines)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 21:17:57 -08:00
Ping QiuandClaude Opus 4.6 c39080ceaa feat: Phase 4A CP4b-4 -- HA integration tests, admin HTTP, 5 bug fixes 
Add HTTP admin server to iscsi-target binary (POST /assign, GET /status,
POST /replica, POST /rebuild) and 7 HA integration tests validating
failover, split-brain prevention, epoch fencing, and demote-under-IO.

New files:
- admin.go: HTTP admin endpoint with input validation
- ha_target.go: HATarget helper wrapping Target + admin HTTP calls
- ha_test.go: 7 HA tests (all PASS on WSL2, 67.7s total)

Bug fixes:
- BUG-CP4B4-1: CmdSN init (expCmdSN=0 not 1, first SCSI cmd was dropped)
- BUG-CP4B4-2: RoleNone->RoleReplica missing SetEpoch (WAL rejected)
- BUG-CP4B4-3: replica applyEntry didn't update vol.nextLSN (status=0)
- BUG-CP4B4-4: PID discovery killed primary instead of replica (shared
  binPath; fixed by grepping volFile)
- BUG-CP4B4-5: artifact collector overwrote primary log with replica log
  (added CollectLabeled method)

Also: 3s write deadline on WAL shipper data connection to avoid 120s TCP
retransmission timeout when replica is dead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 15:04:59 -08:00
Ping QiuandClaude Opus 4.6 09c7e40d29 feat: Phase 4A CP4a -- simulated master, assignment sequence tests, BlockVolumeStatus 
Add SimulatedMaster test helper + 20 assignment sequence tests (8 sequence,
5 failover, 5 adversarial, 2 status). Add BlockVolumeStatus struct and
Status() method. Includes QA test files for CP1-CP4a. 940 total unit tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 09:33:45 -08:00
Ping QiuandClaude Opus 4.6 6546c549eb fix: iSCSI login and discovery bugs found in WSL2 smoke test 
- Skip InitiatorAlias in negotiation (was returning NotUnderstood)
- Capture TargetName in StageLoginOp direct-jump path (iscsiadm skips
  security stage, sends CSG=LoginOp directly -- nil SCSIHandler crash)
- Add portalAddr to TargetServer for discovery responses (listener on
  [::] is not routable from WSL2 clients)
- Add -portal flag to iscsi-target binary

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 10:04:41 -08:00
Ping QiuandClaude Opus 4.6 6a400f6760 feat: add BlockVol engine and iSCSI target (Phase 1 + Phase 2) 
Phase 1: Extent-mapped block storage engine with WAL, crash recovery,
dirty map, flusher, and group commit. 174 tests, zero SeaweedFS imports.

Phase 2: Pure Go iSCSI target (RFC 7143) with PDU codec, login
negotiation, SendTargets discovery, 12 SCSI opcodes, Data-In/Out/R2T
sequencing, session management, and standalone iscsi-target binary.
164 tests. IQN->BlockDevice binding via DeviceLookup interface.

Total: 338 tests, 14.6K lines.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 09:32:01 -08:00