Give the model a chance to fail
A simulation becomes useful when it can be contradicted. Begin with a question, an explicit model and an independent assertion about the outcome. A polished replay is not enough if the test cannot detect a mistaken assumption.
An experiment needs a possible result that would change your mind. Write the expected business outcome and the assumptions supporting it before the run. Then include the interruptions, exceptions or unfamiliar conditions most likely to expose a weakness in the model. Repeatability is valuable, but it should make a mistake easier to investigate rather than easier to overlook.
- Make the assumption visible before the run.
- Test exceptions, interruptions and unfamiliar conditions.
- Preserve failed results rather than hiding them behind a score.
Turn the mismatch into a question
A failed assertion points to something specific: missing source evidence, an incorrect mapping, an unsupported behavior or a flawed hypothesis. Inspect the cause before proposing a refinement.
Classify the mismatch before reaching for a new parameter. A missing observation, an incorrect source mapping, an authored rule and a flawed expected result require different responses. The next useful question is specific: which part of the evidence or model explains this difference, and what observation could distinguish the competing explanations?
An inspectable failure becomes a better question
State the assumption and what would contradict it.
- Trace the failure to its source and model version.
- Distinguish a bad model from a bad scenario assumption.
- Use AI assistance to propose changes within protected boundaries.
Verify the refinement independently
A candidate that fits yesterday’s failure still needs independent testing. Keep the original baseline, preserve held-out evidence and require review before publication or any separately authorized action.
A refinement can overfit the very failure that motivated it. Preserve the original outcome and declare the revised candidate, then evaluate it against appropriately protected evidence. Once verification cases have been used to tune a candidate, their role has changed; retain that history instead of continuing to describe them as untouched independent evidence.
What does a failed synthetic claim-transition assertion teach us?
Retain the case identity, source mapping, model version and expected transition. Investigate whether the mismatch comes from an incorrect mapping, an authored rule or an unsupported exception.
Revise the specific cause and test the candidate independently. Keep the original failure and explain whether the revision improved the model, narrowed its claim or exposed a need for more evidence.
- Compare original and revised behavior.
- Retain failed, withheld and inconclusive outcomes.
- Learning is a reviewed evidence loop—not permission for autonomous production change.
Keep the failure useful
A failure becomes useful when someone can inspect its context. Retain the source and model references, scenario settings, assertion, observed result and any relevant exception information. A label such as “low confidence” cannot replace that explanation.
Avoid treating all mismatches as model defects. An unsupported scenario, stale source snapshot or incorrect expectation can also produce a failed assertion. Preserve uncertainty until the cause is understood, and record the evidence that supports the eventual diagnosis.
A mismatch can point in different directions
| Possible cause | Question to ask | Useful next evidence |
|---|---|---|
| Missing source context | Was the required business event actually observed? | Reviewed source coverage and missing-event analysis. |
| Incorrect model or mapping | Did the model interpret the evidence as intended? | Pinned mapping, rule and case-level comparison. |
| Unsupported expectation | Does the assertion fit the declared domain and scope? | Independent domain review of the expected result. |
- Identify the assertion that failed.
- Trace the result to the input and model version.
- Separate observed causes from proposed explanations.
Improve the question as well as the model
Sometimes the right response is to narrow the claim or collect better evidence. If the source cannot support a required distinction, changing the model alone will not solve the problem. A useful evaluation can expose that gap before a consequential decision is made.
AI assistance may help propose a bounded refinement, but it cannot grant itself authority over protected evidence or acceptance rules. Review the actual proposed change, retain failed and withheld outcomes and judge usefulness independently of numerical success.
Keep the evidence and its conditions together. A later success does not erase an earlier failure or broaden the original scope.
- Keep the permitted change and trial budget explicit.
- Retain the old and revised hypotheses.
- Withhold a conclusion that the evidence cannot support.
Prepare a learning review
Bring one inspectable mismatch, its likely explanations and a bounded next experiment to the review. Name the person responsible for the business interpretation of the expected outcome.
The aim is a better-supported decision, not a story in which every experiment eventually passes. The checklist records preparation only. A reviewed learning loop does not authorize production changes or establish customer qualification.
Review preparation
0 / 6 prepared- State what the failure taught the team.
- Define the next independent check.
- Preserve the limits of the revised conclusion.