seaweedfs/sw-block/design/v2-phase-development-plan.md

# V2 Phase Development Plan

Date: 2026-04-02
Status: active
Purpose: define the execution-oriented phase plan after the current candidate-path work, with explicit module status and target phase ownership

## Why This Document Exists

The project now needs a development plan that is:

1. phase-oriented
2. execution-oriented
3. large enough to avoid overhead-heavy micro-slices
4. explicit about which module belongs to which future phase

This document is the planning bridge between:

1. `v2-product-completion-overview.md`
2. `../.private/phase/phase-08.md`
3. future implementation phases

## Planning Rules

Use these rules for all later phases:

1. one phase should close one meaningful product/engineering outcome
2. every phase must have a clear delivery object and a clear closed-loop validation mechanism
3. every slice inside a phase should also name:
   - what is delivered
   - what loop is proven closed
   - what reject shapes remain insufficient
4. phases should prefer real code/test/evidence over wording-only progress
5. later phases may reuse V1 engineering reality, but must not inherit V1 recovery semantics as truth
6. a phase is too small if it does not move the overall product-completion state clearly

## Current Baseline

Current accepted path through `Phase 13`, with `Phase 14` now the immediate next engineering focus:

1. protocol/algo truth set is strong
2. engine recovery core is strong on the chosen path
3. control-plane closure is accepted on the chosen path
4. selected product-surface rebinding is accepted on the chosen path
5. bounded production hardening is accepted on the chosen path
6. one bounded accepted path exists for:
  - `RF=2`
  - `sync_all`
  - existing master / volume-server heartbeat path
  - `blockvol` as execution backend
7. `Phase 13` has now closed the bounded `WAL V1.5` contract package for the current constrained chosen path:
  - real-workload package accepted
  - assignment/publication closure accepted
  - bounded mode normalization accepted

Phase-accounting note:

1. `Phase 08` is closed
2. `Phase 09` is closed
3. `Phase 10` is closed
4. `Phase 11` is closed
5. `Phase 12` is the accepted hardening baseline for the chosen path
6. `Phase 13` is closed and should be read as one bounded constrained-runtime contract package, not as launch approval
7. `Phase 14` is now the immediate next engineering focus:
   - explicit `V2 core` extraction
   - not more deepening of constrained-`V1` validation by default

Important interpretation rule:

1. the accepted chosen path and claim/evidence set are real
2. the current `weed/` runtime structure is not automatically the final `V2` structure
3. until `Phase 14` establishes an explicit `V2 core`, current integrated tests should be read primarily as:
   - `V1` runtime validation under `V2` constraints
   - not proof that a completed `V2 runtime` already exists
4. future phases must treat:
   - `v2-protocol-claim-and-evidence.md` as current claim authority
   - `v2_mini_core_design.md` as engineering-structure authority
   - `v2-reuse-replacement-boundary.md` as reuse vs replacement authority

This means the next phases should focus mainly on:

1. extracting the long-term `V2 core` structure explicitly
2. rebinding `weed/` into adapter / projection / backend roles
3. closing one bounded `V2`-native runtime path before productionization
4. using `Phase 13` evidence as acceptance input rather than continuing to treat constrained-`V1` validation as the main workstream

## Phase Roadmap

### Phase 09: Production Execution Closure

Goal:

1. turn validation-grade backend execution into production-grade backend execution

Must prove:

1. full-base rebuild performs real data transfer
2. snapshot rebuild performs real image transfer
3. replica-ahead path is physically executable, not only detected
4. runtime execution ownership is stronger than the current bounded candidate path

Typical outputs:

1. real `TransferFullBase`
2. real `TransferSnapshot`
3. real `TruncateWAL`
4. stronger executor/runtime integration in the live volume-server path

Verification mechanism:

1. one-chain execution tests on real backend paths
2. cleanup assertions after success/failure/cancel
3. focused adversarial tests for truncation and rebuild execution

Workload:

1. large
2. this is likely the single biggest remaining engineering phase

Status:

1. complete
2. accepted closeout exists in `../.private/phase/phase-09.md`

### Phase 10: Real Control-Plane Closure

Goal:

1. strengthen from accepted assignment-entry closure to fuller end-to-end control-plane closure

Why this is next:

1. `Phase 09` already closed the main backend execution gaps
2. the most important remaining product risk is no longer storage execution itself
3. it is now control-path completeness:
   - heartbeat / gRPC delivery
   - reassignment / result convergence
   - cleaner local identity than transport-shaped `listenAddr`
4. `Phase 10` can also absorb bounded low-severity cleanup discovered during `Phase 09` if it is directly relevant to live control/runtime ownership

Current accepted progress inside `Phase 10`:

1. `P1` accepted:
   - stable identity and control-truth closure on the chosen block assignment wire
2. `P2` accepted:
   - reassignment/result convergence through the accepted volume-server-side chosen-path ingress
3. `P3` accepted:
   - bounded repeated-assignment / idempotence cleanup on the chosen live path
4. `P4` accepted:
   - master-driven heartbeat / gRPC control-loop closure on the chosen path
5. `Phase 10` is now closed:
   - bounded end-to-end control-plane closure for the chosen path is accepted

Must prove:

1. heartbeat/gRPC-level delivery is real for the chosen path
2. failover / reassignment state converges through the real control path
3. local and remote identity are consistent enough for product use

Typical outputs:

1. stronger heartbeat/gRPC delivery proof
2. stronger result/reporting convergence
3. cleaner local identity than transport-shaped `listenAddr`

Verification mechanism:

1. real failover/reassignment tests at the fuller control-plane level
2. identity/fencing assertions through the end-to-end path

Suggested first targets:

1. keep accepted `Phase 10` control-plane closure closed
2. start `Phase 11` with one bounded product-surface rebinding slice
3. prefer selected surface proofs over broad surface explosion
4. keep any residual control-path cleanup narrow; do not reopen accepted `Phase 10` closure casually

Workload:

1. medium-large

### Phase 11: Product Surface Rebinding

Goal:

1. bind product-facing surfaces onto the V2-backed block path after backend closure is strong enough

Must prove:

1. the V2-backed backend can support selected product surfaces without semantic drift
2. reuse of V1 surfaces does not reintroduce V1 recovery truth

Candidate areas:

1. snapshot product path
2. `CSI`
3. `NVMe`
4. `iSCSI`

Recommended first slice:

1. start with bounded `snapshot product path` rebinding
2. defer `CSI` and `NVMe` / `iSCSI` until one simpler product-visible surface is already accepted

Suggested slice order:

1. `P1` snapshot product-path rebinding
2. `P2` `CSI` rebinding
3. `P3` `NVMe` / `iSCSI` front-end rebinding
4. `P4` broader workflow closure such as snapshot restore/clone if still needed

Verification mechanism:

1. selected surface integration tests
2. product-surface contract checks
3. no-overclaim review that the surface does not imply unsupported backend capability

Workload:

1. medium-large
2. can be split by product surface if needed, but only after backend closure is strong

### Phase 12: Production Hardening

Goal:

1. move from candidate-safe to production-safe

Must prove:

1. restart/recovery stability under repeated disturbance
2. long-run/soak viability
3. operational diagnosability
4. acceptable production blockers list or production-ready gate

Verification mechanism:

1. soak/adversarial runs
2. failover/restart under disturbance
3. runbook/debug validation

Workload:

1. large

Recommended initial planning cut:

1. treat `P0` as hardening-plan freeze
2. first hardening slice should likely target restart / recovery disturbance before soak or perf

Current slice order:

1. `P0` hardening-plan freeze
2. `P1` restart / recovery disturbance hardening
3. `P2` soak / long-run stability hardening
4. `P3` diagnosability / blocker accounting / runbook hardening
5. `P4` performance floor / rollout-gate hardening

Slice delivery / closed-loop bar:

1. every `Phase 12` slice must end with:
   - one bounded delivery object
   - one bounded closed-loop validation object
   - one explicit no-overclaim boundary
2. “tests exist” is not enough:
   - the tests must close the loop from disturbance/input to visible accepted truth
3. “code changed” is also not required:
   - a hardening slice may legitimately close by proving existing production code is already correct under the targeted disturbance class

Current status:

1. `P0` accepted:
   - hardening object frozen as the accepted chosen path from `Phase 09` + `Phase 10` + `Phase 11`
   - slice order frozen as `P1` / `P2` / `P3` / `P4`
   - evidence ladder frozen as disturbance correctness, soak, diagnosability, then perf/rollout gates
2. `P1` accepted:
   - acceptance object = correctness under restart/disturbance on the chosen path
   - not soak, not diagnosability, not performance, not rollout readiness
3. `P2` accepted:
   - acceptance object = bounded soak / long-run stability on the chosen path
   - repeated-cycle coherence and bounded runtime-state hygiene are accepted inside a bounded test envelope
4. `P3` accepted:
   - acceptance object = bounded diagnosability / blocker accounting / runbook hardening on the chosen path
   - bounded operator-visible diagnosis surfaces and finite blocker accounting are accepted
5. `P4` accepted:
   - acceptance object = bounded performance floor / rollout-gate hardening on the chosen path
   - not broad rollout readiness beyond the named launch envelope
6. `Phase 12` is now closed:
   - one bounded hardening package is accepted for the chosen path

Current `P4` first delivery shape:

1. proof-first hardening slice with explicit measured floor and launch-gate artifacts
2. one bounded performance package:
   - named workload envelope
   - repeatable measurement harness
   - explicit floor values
3. one explicit rollout-gate artifact:
   - finite supported launch envelope
   - cleared blockers/gates
   - remaining blockers/gates
4. current evidence shape:
   - measured floor values are tied to one named accepted workload envelope
   - cost/resource trade-offs are explicit
   - rollout discussion is bounded by an explicit finite gate package
5. current reuse boundary:
   - accepted chosen-path runtime/control/product surfaces remain stable unless perf-floor work exposes a real bug or measurement gap
   - focused benchmarks/tests and bounded launch-gate artifacts carry the main delivery burden

Closed-loop expectation for `P4` review:

1. one bounded workload envelope runs on the accepted chosen path
2. measured floor values and cost characteristics are explicit
3. launch claims map back to accepted prior slices plus the measured envelope
4. remaining rollout blockers are explicit and finite
5. claims remain bounded to measured floor / named launch envelope only

### Phase 13: V1.5 Contract Closure And Contradiction Ledger

Goal:

1. close the bounded `RF=2 sync_all` contract on the current chosen path
2. freeze what `WAL V1.5` is allowed to claim
3. classify remaining live contradictions as:
   - reusable-core bug
   - adapter-boundary bug
   - `V2`-authority bug

Delivery object:

1. one frozen `CP13-*` contract package
2. one centralized claim/evidence ledger for the active chosen path
3. one explicit contradiction/rerun queue for invalidated or narrowed evidence

Closed-loop validation:

1. protocol/unit/adversarial proofs for `CP13-1..7`
2. bounded real-workload validation, assignment/publication closure, and mode normalization are all accepted
3. explicit narrowing or restoration of claims in the centralized ledger

Non-claims:

1. not full `V2 core` extraction
2. not launch approval
3. not proof that current `weed/` structure is the final `V2` structure

Key files / ownership:

1. `sw-block/.private/phase/phase-13-*.md`
2. `sw-block/design/v2-protocol-claim-and-evidence.md`
3. `sw-block/design/v2-protocol-truths.md`

### Phase 14: V2 Core Extraction

Goal:

1. make the `V2 core` explicit as a long-term code structure
2. stop relying on implicit semantic ownership spread across runtime files

Execution rule:

1. define core-owned state and transitions first
2. freeze command-emission rules second
3. freeze projection contracts third
4. only then connect adapters

Delivery object:

1. one explicit `V2 core` package/file layout with named:
   - `state`
   - `event`
   - `command`
   - `projection`
2. one minimal real code path for:
   - `ApplyEvent()`
   - `Decide()`
   - `EmitCommands()`
3. one bounded parity package showing accepted prototype/FSM semantics are preserved

Closed-loop validation:

1. focused engine tests proving accepted claims can be represented through the new event/command core
2. parity checks against accepted prototype/FSM semantics
3. no-overclaim review that this is structural extraction, not live-path cutover

Recommended slice order:

1. `Phase 14A`: core-owned automata
   - explicit assignment / recovery / boundary / mode / publication automata
   - structural tests only
2. `Phase 14B`: command semantics
   - bounded command sequences derived from semantic state
   - still no live `weed/` execution
3. `Phase 14C`: projection contracts
   - lookup / heartbeat / debug / tester normalization from core-owned state
   - surface-consistency tests

Immediate focus inside `Phase 14`:

1. start from one complete semantic chain:
   - `mode`
   - `readiness`
   - `publication`
2. use accepted `CP13-8A` and `CP13-9` as the first hard input package

Non-claims:

1. not full live-path cutover
2. not replacement of all `weed/` logic
3. not a separate process yet

Key files / ownership:

1. `sw-block/design/v2_mini_core_design.md`
2. `sw-block/design/v2-phase14plus-semantic-framework.md`
3. `sw-block/engine/replication/`

### Phase 15: Adapter And Projection Rebinding

Goal:

1. make `weed/` a bounded adapter/projection layer instead of mixed semantic authority
2. close assignment -> readiness -> publication through named `V2` state

Delivery object:

1. one explicit adapter-boundary package on the live path
2. one explicit projection store / projection surface package for:
   - readiness
   - publication
   - diagnostics
3. one narrowed role definition where:
   - `BlockService` is closer to command executor
   - registry is closer to projection store

Closed-loop validation:

1. live-path tests proving assignment delivered != receiver ready != publish healthy unless the named readiness/projection loop is closed
2. focused regression package for the `CP13-8A` bug class
3. operator-visible projection checks rather than internal-state-only proof

Recommended slice order:

1. `Phase 15A`: minimal adapter hook
   - one narrow event path into the core
   - one bounded command path back out
   - prove no semantic split on that narrow path
2. `Phase 15B`: projection-store rebinding
   - registry / lookup / tester-facing surfaces consume core-owned projection truth
   - prove assignment delivered != ready != publish healthy on the real path

Non-claims:

1. not backend rewrite
2. not broader productization
3. not new transport matrix claims

Key files / ownership:

1. `sw-block/design/v2-reuse-replacement-boundary.md`
2. `sw-block/design/v2-phase14plus-semantic-framework.md`
3. `weed/server/volume_server_block.go`
4. `weed/server/block_heartbeat_loop.go`
5. `weed/server/master_block_registry.go`
6. `weed/server/master_block_failover.go`

### Phase 16: V2-Native Runtime Closure

Goal:

1. make the integrated runtime behave as a `V2`-owned recovery/control system
2. stop depending on a merely improved `WAL V1.5` path for correctness interpretation

Execution precondition:

1. do not enter `Phase 16` until `Phase 14` has frozen state / command / projection semantics
2. do not treat adapter rebinding alone as runtime closure

Delivery object:

1. one bounded product/runtime path where failover, recovery, publication, and selected surfaces are driven by the `V2 core` + adapter contract
2. one explicit runtime integration package that maps simulator/prototype failure classes to live-path behavior

Closed-loop validation:

1. end-to-end failover/recovery scenarios on the core-driven path
2. simulator-to-runtime consistency checks for the named failure classes that `V2` is supposed to survive
3. bounded real-workload checks on the core-driven path, not just on the legacy-integrated path

Non-claims:

1. not broad rollout approval
2. not physical split into an independent `V2 core process` unless the logic is already structurally independent

Key files / ownership:

1. `sw-block/engine/replication/`
2. `weed/server/`
3. selected `weed/storage/blockvol/v2bridge/*` files

### Cross-Phase Review Rule For `Phase 14+`

For any new transition, command, or projection rule in `Phase 14+`, require a
short justification in the delivery note or code review:

1. semantic constraint satisfied
   - which `claim / truth / CP13-*` item it is implementing
2. overclaim avoided
   - which false healthy / ready / durable / recoverable interpretation is being prevented
3. proof preserved
   - which accepted test or checkpoint remains valid because of the rule

### Productionization Program After `Phase 16`

Goal:

1. turn the accepted `Phase 16` bounded path into a bounded first-launch product envelope without reopening protocol discovery or core-ownership questions

Program slices:

1. Program `P0`: launch-envelope freeze
   - freeze the first supported launch envelope from accepted hardening + runtime-closure evidence
   - lock:
     - supported topology / transport matrix
     - explicit exclusions
     - launch-blocking vs post-launch blockers
   - reject if any launch claim outruns the measured matrix or accepted blockers/gates
2. Program `P1`: internal pilot pack
   - convert the frozen launch envelope into a limited internal pilot package
   - define:
     - pilot environment and topology
     - preflight checklist
     - success criteria
     - stop / rollback conditions
     - incident intake template tied to accepted diagnosability surfaces
   - reject if pilot success depends on tribal knowledge or undefined operator judgment
3. Program `P2`: incident-driven hardening loop
   - route pilot findings into explicit buckets:
     - config / environment issue
     - known exclusion
     - true product bug
   - keep one bounded incident ledger and one bounded fix queue
   - reject if incidents accumulate as vague notes or exclusions are silently redefined
4. Program `P3`: controlled rollout review
   - decide whether to:
     - stay in pilot
     - widen within the same launch envelope
     - block expansion
   - require explicit mapping from any expansion decision back to:
     - accepted `Phase 16` evidence
     - pilot outcomes
     - incident dispositions
   - reject if rollout broadens beyond the named envelope or reuses pilot success as generic production proof

Cross-cutting rules:

1. do not invent a `Phase 12 P5`; productionization remains separate from hardening
2. do not collapse `Phase 13-16` into generic productionization; they are engineering-structure phases
3. keep the accepted chosen path fixed unless contradiction or incident evidence exposes a real bug
4. treat known missing evidence as explicit constraints until cleared, especially:
   - failover-under-load performance
   - hours/days soak under load
   - `RF>2`
   - broad transport matrix
   - full gRPC-stream integration evidence
5. keep the roadmap aligned with:
   - `v2-protocol-claim-and-evidence.md`
   - `v2_mini_core_design.md`
   - `v2-reuse-replacement-boundary.md`

## Module Status Map


| Module area | Current status | Current owner phase | Next target phase | Notes |
| ----------- | -------------- | ------------------- | ----------------- | ----- |
| `sw-block/engine/replication` core FSM/orchestrator/driver | Strong long-term `V2 core` asset | `Phase 09` accepted, `Phase 13` active constraints | `Phase 14` | Main next work is explicit `state / event / command / projection` extraction, not reopening accepted semantics. |
| Engine executor real I/O boundary (`CatchUpIO` / `RebuildIO`) | Strong on chosen path | `Phase 09` accepted | `Phase 14/16` | Keep the boundary stable; later work is to connect it to explicit `V2 core` ownership and runtime closure. |
| `weed/storage/blockvol/v2bridge/control.go` | Strong boundary adapter on chosen path | `Phase 08/09/10` accepted | `Phase 15` | Remains a bridge between `V2` truth and runtime execution; should not accumulate new semantic authority casually. |
| `weed/storage/blockvol/v2bridge/reader.go` | Strong backend-facing adapter | `Phase 09` accepted | `Phase 15/16` | Mostly stable; later work is explicit boundary ownership and runtime proof, not new protocol semantics. |
| `weed/storage/blockvol/v2bridge/pinner.go` | Strong backend-facing adapter | `Phase 09` accepted | `Phase 15/16` | Retention safety is proven on the chosen path; later work is keeping it under explicit `V2` control boundaries. |
| `weed/storage/blockvol/v2bridge/executor.go` WAL scan | Strong backend-facing adapter | `Phase 09` accepted | `Phase 15/16` | Real execution path is closed on the chosen path; later work is core-driven runtime integration. |
| `v2bridge` `TransferFullBase` | Strong on chosen path | `Phase 09 P1` accepted | `Phase 16` | Execution closure is accepted; do not reopen casually unless core-driven runtime closure exposes a real contradiction. |
| `v2bridge` `TransferSnapshot` | Strong on chosen path | `Phase 09 P2` accepted | `Phase 16` | Execution closure is accepted; later work is bounded runtime closure rather than first-implementation discovery. |
| `v2bridge` `TruncateWAL` | Strong on chosen path | `Phase 09 P3` accepted | `Phase 16` | Narrow contract is accepted; later work is preserving that contract under explicit runtime ownership. |
| `weed/server/volume_server_block.go` V2 assignment intake | Adapter-boundary reality with accepted chosen-path closure | `Phase 10 P4` accepted, `Phase 13` contradiction pressure | `Phase 15` | This is where assignment/readiness/publication closure must become explicit adapter behavior rather than mixed service semantics. |
| `weed/server/block_recovery.go` live runtime ownership | V2-owned runtime truth on chosen path | `Phase 09/10` accepted | `Phase 15/16` | Serialized ownership is accepted; later work is making it cooperate with explicit `V2 core` and projection boundaries. |
| `weed/server/master_block_registry.go` / failover / handlers | Mixed projection/truth reality | `Phase 10-12` accepted surfaces | `Phase 15` | Should converge toward projection store + operator-visible truth surfaces rather than mixed business-logic/state storage. |
| `blockvol` WAL/flusher/checkpoint runtime | Reuse reality | Existing production code | `Phase 16` | Reuse implementation; do not let `V1` replication semantics redefine `V2` truth. |
| `blockvol` rebuild transport/server reality | Reuse with redesign boundary | Existing production code | `Phase 16` | Bounded chosen-path integration is accepted; later work is runtime closure under `V2` authority. |
| local server identity (`localServerID`) | Strong chosen-path rule with narrowed semantics | `Phase 10 P1` accepted, `Phase 13` constraints | `Phase 15` | Stable identity must remain distinct from transport address shape and explicit in later adapter/projection work. |
| Snapshot product path | Strong on chosen path | `Phase 11` accepted | `Phase 16` | Product-visible snapshot workflow is accepted on the chosen path; later work is preserving it under `V2`-native runtime closure. |
| `CSI` integration | Strong on chosen path | `Phase 11` accepted | `Phase 16` | Bounded controller/node lifecycle rebinding is accepted; later work is runtime preservation, not first rebinding. |
| `NVMe` / `iSCSI` front-ends | Strong on chosen path | `Phase 11` accepted | `Phase 16` | Publication/address truth rebinding is accepted; later work is proving the `V2`-driven runtime path beneath them. |
| Testrunner / infra / metrics | Strong support layer | existing | `Phase 13-16` | Reuse to validate contract closure, adapter contradictions, runtime closure, and later productization gates. |


## Completion-State Targets

Use these rough targets to judge whether a phase is moving the product meaningfully.


| Phase | Expected completion move |
| ----- | ------------------------ |
| `Phase 09` | from validation-grade backend execution to accepted execution closure on the chosen path |
| `Phase 10` | from bounded control-entry proof to stronger end-to-end control-plane closure |
| `Phase 11` | from backend-ready path to selected product-surface readiness |
| `Phase 12` | from candidate-safe to production-safe on one bounded chosen path |
| `Phase 13` | from scattered `WAL V1.5` evidence to one frozen contract/contradiction ledger |
| `Phase 14` | from implicit core semantics to explicit `V2 core` package/engine structure |
| `Phase 15` | from mixed runtime semantics to explicit adapter/projection closure |
| `Phase 16` | from improved chosen-path runtime to one bounded `V2`-native runtime path |
| Productionization | from bounded runtime closure to bounded launch-envelope / pilot / rollout review |


## Near-Term Execution Direction

If the goal is to maximize product completion efficiently, the recommended order is now:

1. keep `Phase 09-12` accepted closures closed and do not reopen them casually
2. keep `Phase 13` closed as the bounded `WAL V1.5` contract package
3. move next to `Phase 14` core extraction:
   - explicit `state / event / command / projection`
   - minimal `ApplyEvent() -> Decide() -> EmitCommands()` path
4. then `Phase 15` adapter/projection rebinding:
   - assignment
   - readiness
   - publication
   - diagnostics
5. then `Phase 16` bounded `V2`-native runtime closure
6. only then move to the productionization program:
   - freeze launch envelope
   - run limited internal pilot
   - harden from incidents
   - review controlled rollout

The most important near-term engineering weight should now go to:

1. making `sw-block/engine/replication` the explicit long-term `V2 core`
2. keeping `weed/` changes bounded to adapter / projection / backend roles unless they are explicitly promoted into `V2`-owned authority
3. using `CP13-1..9` as acceptance input for new `V2 core` work rather than as a reason to keep extending constrained-`V1` validation

## Short Summary

The V2 line now has accepted execution, control, product-surface, and hardening closure on one bounded chosen path.

The next development plan should not jump directly from `Phase 12` to productionization.
It should move through four bounded engineering phases first:

1. `Phase 13`: freeze the current `WAL V1.5` constrained-runtime contract package
2. `Phase 14`: extract the explicit `V2 core`
3. `Phase 15`: rebind `weed/` as adapter/projection reality
4. `Phase 16`: close one bounded `V2`-native runtime path

Only after that should the roadmap move into bounded launch-envelope freeze, internal pilot, incident-driven hardening, and controlled rollout review.