Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-10-18 12:08:32 +02:00
Code Issues Packages Projects Releases Wiki Activity
13,201 Commits 195 Branches 361 Tags
2b97cd04b8c8d39b36ca0c0dd50df19d2ed6086f
Commit Graph
18 Commits
Author SHA1 Message Date
pingqiuandClaude Opus 4.6 044a6d770b feat: Phase 19 — bounded working RF2 block path 
Live HTTP evidence transport, continuous Loop2 service, bounded auto
failover trigger, runtime-managed frontend export, bounded replica
repair, end-to-end RF2 handoff with continued I/O on new primary,
bounded operator HTTP surface, and CSI V2 runtime backend adapter.

11 new proof tests covering the full M6-M10 chain plus CSI create/
lookup/publish through the V2 runtime path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-05 15:12:00 -07:00
pingqiuandClaude Opus 4.6 b82df09856 feat: Phase 18 M1 — transport-backed RF2 failover runtime 
M1 milestone: failover evidence crosses transport/session seam.

Adapter seam (failover_adapter.go):
- FailoverEvidenceAdapter: query-side (promotion evidence + replica summary)
- FailoverTakeoverAdapter: execution-side (prepare + gate)
- FailoverTarget: binds NodeID + both adapters
- NewInProcessFailoverTarget: factory for in-process case

Transport seam (failover_evidence_transport.go):
- FailoverEvidenceTransport: request/response interface with nodeID routing
- FailoverEvidenceHandler: server-side registration
- InMemoryFailoverEvidenceTransport: first transport impl (in-memory)
- NewHybridInProcessFailoverTarget: transport-backed evidence + local takeover

Runtime manager (runtime_manager.go):
- InProcessRuntimeManager: participant registry + ExecuteFailover entry point
- Persisted failover snapshots/results per-volume and global-last

All failover paths (session/driver/manager) now go through adapter seam.
Old FailoverParticipant preserved as compatibility wrapper only.

Phase 18 docs: phase-18.md (M1-M5 structure), log, decisions.
Design docs updated: kernel-closure-review, claim-and-evidence ledger.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-05 13:57:08 -07:00
pingqiuandClaude Opus 4.6 b8c6944e3f feat: V2 MVP milestone — masterv2 + volumev2 + in-process failover 
V2 runtime packages:
- sw-block/runtime/masterv2: identity authority (desired state,
  heartbeat handling, promotion arbitration via SelectPromotionCandidate)
- sw-block/runtime/volumev2: per-volume micro-cluster shell (node,
  orchestrator, control session, iSCSI frontend, takeover gate,
  failover session + driver, replica summary reconstruction)
- sw-block/runtime/purev2: RF1 execution shell (engine + store +
  dispatcher + local boundary observations)
- sw-block/runtime/protocolv2: three-channel separation
  (heartbeat/assignment/query + replica summary)

V2 binaries:
- sw-block/cmd/v2singleblock: single-node RF1 block server
- sw-block/cmd/purev2rf1: minimal RF1 runtime binary

Milestone capabilities:
- RF1 write/read/sync with engine-driven mode projection
- masterv2 ↔ volumev2 heartbeat convergence + assignment reissue
- Promotion query with fresh CommittedLSN/WALHeadLSN evidence
- Replica summary for bounded takeover reconstruction
- Primary-loss reconstruction from peer summaries (fail-closed gate)
- In-process failover driver with session observability
- Local boundary observations feed engine (Committed/Durable/Checkpoint)

Design docs:
- v2-two-loop-protocol.md: identity vs data-control separation
- v2-automata-ownership-map.md: event/command ownership split
- v2-loop1-surface-draft.md: heartbeat/query/assignment field spec
- v2-volumev2-single-node-mvp.md: target layering
- v2-kernel-closure-review.md: per-volume micro-cluster principle
- v2-pure-runtime-rf1-bootstrap.md, v2-capability-map.md,
  v2-proof-and-retest-pyramid.md

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-05 13:08:02 -07:00
pingqiuandClaude Opus 4.6 cf16e53b04 feat: Phase 16M/17 + promote fixes + testrunner updates 
Phase 16M: explicit replica readiness on heartbeat seam
- master.proto: optional bool replica_ready = 19 (proto regenerated on M01)
- block_heartbeat_proto.go: write/read ReplicaReady with presence semantics
- master_block_registry.go: replicaReadyObservedFromHeartbeat prefers
  explicit proto field, falls back to address heuristic when absent
- volume_server_block.go: heartbeat emits ReplicaReady from core projection

Phase 17: host effects extraction + stop line
- phase-17-log.md: Batch 10/11 delivery notes

Promote fixes:
- master_block_failover.go: deterministic replica addrs from path hash
- qa_promote_replication_test.go: address-upgrade trigger test
- qa_promote_rejoin_live_test.go: new live rejoin test

Testrunner:
- devops.go: action improvements
- recovery-baseline-failover.yaml, suite-ha-failover.yaml: scenario updates
- cp11b3-manual-promote.yaml: promote scenario alignment
- fresh_volume_write_test.go: new component test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-05 11:38:05 -07:00
pingqiu 7855d5240c docs: add bounded productionization pilot artifacts 
Freeze the first bounded pilot/preflight/stop/rollout-review artifact set and sync the global product ledgers so productionization can start from an explicit chosen-envelope discipline instead of ad hoc rollout judgment.

Made-with: Cursor
 2026-04-04 19:01:56 -07:00
pingqiu 4f95a1e868 docs: package phase 17 product claim checkpoint 
Freeze the first Phase 17 branch/contract/policy/envelope package, add review and supported-matrix artifacts, and sync the product-completion and claim-evidence ledgers to the new bounded post-Phase-16 checkpoint.

Made-with: Cursor
 2026-04-04 18:21:16 -07:00
pingqiu 0f72c8d062 refactor: close bounded phase 16 restart truth seams 
Bind non-authoritative inventory, restart primary-truth rebasing, and sparse replica readiness retention into the heartbeat/master seam, and package the bounded finish-line checkpoint with explicit claims, non-claims, and proof commands.

Made-with: Cursor
 2026-04-04 16:13:06 -07:00
pingqiu 43dbebfa04 refactor: close bounded recovery drain and invalidation seams 
Move removed-replica drain and replica-scoped invalidation onto explicit core-command paths so the widened multi-replica runtime no longer depends on coarse host-side recovery handling.

Made-with: Cursor
 2026-04-04 11:01:12 -07:00
pingqiuandClaude Opus 4.6 11c6aaf316 feat: Batch 7 + Phase 16C-E — command dispatch extraction + engine refinements 
Batch 7: Command dispatch binding extraction
- New weed/server/blockcmd package: CommandHandler interface + DispatchCommands
- volume_server_block.go applyCoreCommandsWithAssignment delegates to dispatcher
- weed/server still owns RecordCommand, EmitCoreEvent, PublishProjection
- v2bridge NOT given command-switch or event-emission semantics

Phase 16C: Rebuilding assignment enters core command path
Phase 16D: Rebuild recovery-task startup is command-driven
Phase 16E: Catch-up recovery-task startup is command-driven

Engine refinements:
- RecoveryTarget on AssignmentDelivered event
- shouldStartRecoveryTask / shouldStartReceiver guards
- bootstrapReason: awaiting_rebuild_start

Bridge/contract updates:
- control_adapter.go: refined translation helpers
- contract.go: executor port alignment

Migration design docs (Batch 1-3 delivered, design artifacts):
- v2-first/second/third-migration-batch.md + task-pack.md
- v2-assignment-translation-unification.md
- v2-execution-muscles-inventory.md
- v2-separation-port-layer-audit.md
- v2-legacy-runtime-exit-criteria.md

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-04 02:13:08 -07:00
pingqiuandClaude Opus 4.6 8c2485e0e9 feat: Phase 15 + Phase 16A/B — V2 core integration + checkpoint review 
Phase 15: V2 core wired into BlockService
- volume_server_block.go: v2Core field, applyCoreAssignmentEvent,
  core command executors (ApplyRole, StartReceiver, ConfigureShipper,
  InvalidateSession, StartCatchUp, StartRebuild, PublishProjection)
- Assignment processing now goes through core engine → command emission
  → bounded execution, replacing direct V1 replication setup
- master_block_registry.go: ClusterHealthSummary, VolumeMode in entries
- master_server_handlers_block.go: blockStatusHandler, entryToVolumeInfo
  refactored with entryReplicaSurface

Phase 16A: Core projection surfaces
Phase 16B: Bounded closure (checkpoint review ready)

Test fixes: add v2Core to manually-constructed BlockService in
idempotence, convergence, soak, and CP13-8A tests (required because
V1 replication setup paths now delegate to core engine).

All tests pass (21s regression).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-03 20:58:12 -07:00
pingqiuandClaude Opus 4.6 34f42078fb docs: Phase 13 CP13-9 accepted + Phase 14 preparation docs 
- phase-13.md: CP13-8/8A/9 accepted with carry-forward
- phase-13-log.md: CP13-9 technical/delivery packs
- phase-13-cp9-mode-normalization.md: minor updates
- v2-protocol-claim-and-evidence.md: CP13-8/8A claims updated,
  constrained-V1-runtime interpretation rule added

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-03 16:13:03 -07:00
pingqiu fb0da91196 feat: start Phase 14 V2 core shell 
Make the first V2 core owner explicit in sw-block by freezing Phase 14 docs, mode/readiness/publication semantics, and bounded command emission rules. This turns accepted Phase 13 constraints into executable core behavior without overclaiming live runtime cutover.

Made-with: Cursor
 2026-04-03 16:11:38 -07:00
pingqiuandClaude Opus 4.6 4c7fbefe25 feat: CP13-8 PASSES — real-workload validation on RF=2 sync_all 
CP13-8 scenario results on m01/M02 (25Gbps RoCE):
  fsck_ext4:       CLEAN
  file count:      200 (assert_equal PASS)
  checksum match:  MATCH (assert_contains PASS)
  pgbench TPS:     565.69 (assert_greater PASS)
  auto-failover:   10.0.0.1:18480 → 10.0.0.3:18480

Code changes (tester + scenario):
- volume_server_block.go: readiness state, assignment lifecycle cleanup
- block_heartbeat_loop.go: readiness-aware heartbeat reporting
- store_blockvol.go: readiness tracking
- master_server_handlers_block.go: block API handler updates
- cp13-8-real-workload-validation.yaml: redesigned scenario
  (removed block_promote, use natural auto-failover flow,
  bootstrap write before wait_volume_healthy)
- testrunner/actions/devops.go: scenario action improvements
- replica_read_test.go: component-level replica read test

Phase docs: CP13-7 accepted, CP13-8/8A technical packs updated,
design docs updated for protocol closure evidence.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-03 14:24:13 -07:00
pingqiuandClaude Opus 4.6 21ccf06ef3 docs: Phase 13 CP13-1..CP13-7 technical packs, acceptance status, design updates 
- phase-13.md: CP13-1 through CP13-6 accepted, CP13-7 active
- phase-13-log.md: full technical + delivery packs for CP13-2..CP13-7
- phase-13-cp4-state-eligibility.md: refined barrier behavior table
  (Disconnected/Degraded as recovery entry points, not eligibility)
- phase-12.md: minor cross-reference updates
- Older phase docs: minor wording alignment
- Design docs: V2 development plan and completion overview updated

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-03 08:48:05 -07:00
pingqiuandClaude Opus 4.6 c7eb87c587 feat: Phase 09 — V2 execution primitives and production closure 
Engine execution layer for V2 replication protocol:
- RebuildInstaller: full state handoff (dirty map, WAL, superblock, flusher)
- TruncateToLSN: exact safety predicate (checkpointLSN == truncateLSN),
  ErrTruncationUnsafe escalation to NeedsRebuild
- SyncReceiverProgress: unconditional Store for post-rebuild alignment
- V2StatusSnapshot: CommittedLSN = nextLSN-1 for sync_all

V2 bridge real I/O executors:
- TransferFullBase: TCP streaming + RebuildInstaller + second catch-up
- TransferSnapshot: SHA-256 verified streaming to disk
- TruncateWAL: ErrTruncationUnsafe detection + escalation
- StreamWALEntries: rebuild-mode TCP apply

Engine executor interfaces:
- CatchUpIO.TruncateWAL, RebuildIO.TransferFullBase returns achievedLSN
- CatchUpExecutor truncation-only skip, NeedsRebuild escalation
- RebuildExecutor uses achievedLSN for progress tracking

Design docs reorganized: superseded planning docs removed, protocol
truths and closure map added.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 16:25:23 -07:00
pingqiuandClaude Opus 4.6 46faf0f7e3 feat: Phase 09 P0 — production execution closure plan 
Execution-closure targets:
- P1: TransferFullBase — reuse rebuild.go TCP protocol
- P2: TransferSnapshot — checkpoint image + WAL tail
- P3: TruncateWAL — AdvanceTail + superblock update
- P4: Runtime ownership — V2 orchestrator drives execution

Key reuse sources identified:
- rebuild.go: rebuildFullExtent (client), RebuildServer (server)
- wal_writer.go: AdvanceTail
- flusher.go: updateSuperblockCheckpoint
- blockvol.go: ScanWALEntries (already wired)

Slice order: full-base first (highest value), then snapshot,
then truncation, then runtime ownership.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-31 17:25:09 -07:00
pingqiuandClaude Opus 4.6 46ef79ce35 fix: stable ServerID in assignments, fail-closed on missing identity, wire into ProcessAssignments 
Finding 1: Identity no longer address-derived
- ReplicaAddr.ServerID field added (stable server identity from registry)
- BlockVolumeAssignment.ReplicaServerID field added (scalar RF=2 path)
- ControlBridge uses ServerID, NOT address, for ReplicaID
- Missing ServerID → replica skipped (fail closed), logged

Finding 2: Wired into real ProcessAssignments
- BlockService.v2Bridge field initialized in StartBlockService
- ProcessAssignments converts each assignment via v2Bridge.ConvertAssignment
  BEFORE existing V1 processing (parallel, not replacing yet)
- Logged at glog V(1)

Finding 3: Fail-closed on missing identity
- Empty ServerID in ReplicaAddrs → replica skipped with log
- Empty ReplicaServerID in scalar path → no replica created
- Test: MissingServerID_FailsClosed verifies both paths

7 tests: StableServerID, AddressChange_IdentityPreserved,
MultiReplica_StableServerIDs, MissingServerID_FailsClosed,
EpochFencing_IntegratedPath, RebuildAssignment, ReplicaAssignment

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-31 10:46:17 -07:00
pingqiuandClaude Opus 4.6 edec7098e8 feat: add V2 protocol simulator and enginev2 sender/session prototype 
Adds sw-block/ directory with:

- distsim: protocol correctness simulator (96 tests)
  - cluster model with epoch fencing, barrier semantics, commit modes
  - endpoint identity, control-plane flow, candidate eligibility
  - timeout events, timer races, same-tick ordering
  - session ownership tracking with ID-based stale fencing

- enginev2: standalone V2 sender/session implementation (63 tests)
  - per-replica Sender with identity-preserving reconciliation
  - RecoverySession with FSM phase transitions and session ID
  - execution APIs: BeginConnect, RecordHandshake, BeginCatchUp,
    RecordCatchUpProgress, CompleteSessionByID — all sender-authority-gated
  - recovery outcome branching: zero-gap, catch-up, needs-rebuild
  - assignment-intent orchestration with epoch fencing

- design docs: acceptance criteria, open questions, first-slice spec,
  protocol development process

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-28 10:38:27 -07:00