Start with one decision
Choose a workflow with a named owner and a concrete decision: a migration, a recovery policy or a capacity change. Bound the interfaces and operating conditions before expanding the scope.
A useful scope can be expressed as a comparison: under the same declared demand, does this candidate preserve the required business result while changing an operational outcome? Choose a boundary the team can inspect. A broad ambition such as “improve resilience” needs a specific workflow, interruption and acceptance condition before it becomes an experiment.
- Record the baseline and the proposed change.
- Define business correctness separately from performance.
- Name the people responsible for acceptance.
Agree the evidence contract
Identify what is observed, what is authored and what remains unknown. Use independent assertions that can fail even when the simulation is repeatable.
Write the evidence contract before observing results. Include the source population, workload, expected outcomes, execution mode and resource limits. Label synthetic cases and authored rules. Agree which evidence can support the immediate decision and which questions still require measurements from the intended deployment.
From business question to acceptance evidence
Name the decision and the person who owns its consequences.
- Pin source snapshots and model revisions.
- Keep training and verification evidence separate.
- Record excluded interfaces and untested conditions.
Make the review actionable
Compare the candidate against the baseline under the same declared workload. A useful outcome can be a rejected proposal or a better-defined qualification gap.
Review the complete outcome set, including failed, unsupported and withheld cases. A candidate that improves a headline metric but violates a business invariant has not passed. Make the review explain what changed, why the result matters and the conditions under which the conclusion could fail.
Would a revised processing path improve a synthetic payment workflow?
Hold the declared payment population and correctness assertions fixed. Compare the current path with one supported candidate change, retaining completion, exceptions and timing context.
A useful result explains both any improvement and the cases that regressed. Native payment-system compatibility and production performance need separate evidence.
- Retain scenario inputs and result references.
- Review exceptions with domain and platform owners.
- Set the next acceptance gate before broader adoption.
Choose a comparison that can fail
Good acceptance criteria can reject an attractive proposal. Separate necessary business conditions from desired operational improvements, and agree how exceptions are handled. The candidate should face the same declared workload as the baseline.
A successful rehearsal may demonstrate a narrow improvement, expose a missing model dependency or show that the proposed change is not useful. Keep all three outcomes available; otherwise the evaluation becomes a search for a preferred answer.
Read the result as a decision
| Observation | Interpretation | Next step |
|---|---|---|
| Correctness and target pass | Evidence supports the declared scenario. | Review usefulness and scope before expanding. |
| Faster, but an invariant fails | The candidate breaks a required business condition. | Reject or revise it with a new recorded comparison. |
| Inputs or coverage are incomplete | The conclusion is limited or inconclusive. | Close the evidence gap before making a broader claim. |
- Freeze correctness assertions before the run.
- Retain the declared population and exclusions.
- Review tradeoffs rather than collapsing them into one score.
Build a reviewable evidence package
A reviewer should be able to trace the decision from inputs to results without reconstructing the experiment from memory. Keep the source snapshot, baseline and candidate model identities, scenario settings, execution records and verification outcomes together.
Attach the assumptions that materially affect the conclusion. If a result depends on synthetic demand, a declared repair time or an unsupported interface, carry that limitation into the review. A local experiment does not establish customer production acceptance.
Keep the evidence and its conditions together. A later success does not erase an earlier failure or broaden the original scope.
- Preserve failed runs and unresolved differences.
- Name the domain reviewer and platform owner.
- Record the next evidence needed to broaden the conclusion.
Prepare the evaluation brief
Turn the review into a bounded next step. The brief should identify one accountable owner, one decision, a declared environment and the evidence required to judge the result.
Use the checklist to prepare the conversation. Its selections describe your planning progress only. They do not change platform controls, approve an operation or establish that an evaluation has passed.
Review preparation
0 / 6 prepared- Agree what the evaluation can resolve.
- Document what remains outside its scope.
- Choose the next acceptance gate explicitly.