Skip to content
WorkspaceLet's simulate→
← Resource library
THE SCENARIO COMPARISON GUIDE / 6 min

Beyond the average.
Into the tradeoff.

Make workload, latency, retry cost and uncertainty visible together.

6 practical chaptersWorked exampleReview checklist
CORTEXSIM / VISUAL FIELD NOTESSELECT A COMPONENT
RESPONSE TIME / CONCEPTUALINCREASING RESPONSE PERCENTILE →— Baseline shape— Candidate shapeILLUSTRATION ONLY · NOT A MEASURED BENCHMARK01Workload02Baseline03Candidate04Tail + completion05Conclusion
Workload

Pin demand, input population, execution mode and observation window.

GUIDE / CORTEXSIM

Compare the shape of outcomes, not one headline number. Select a component to explore its role in the evaluation.

CHAPTER 01

Use a fair baseline

Compare alternatives under the same workload, observation window and execution mode. Measured client tests and virtual-time scenarios answer different questions.

State whether timing comes from a modeled virtual-time scenario or a measured client test. These are different observations and should not share an unlabeled chart. Keep warm-up, arrival pattern, concurrency, data distribution and observation window visible so the comparison can be interpreted.

DESIGN TAKEAWAYS
  • Pin demand and data assumptions.
  • Separate average and tail latency.
  • Record the experiment seed where applicable.
CHAPTER 02

Look beyond one metric

A faster response can create more retries, queue pressure or failed operations. Compare acceptance targets across several outcomes.

A latency improvement among successful responses may hide a higher timeout rate. Retry traffic can increase work even when useful completions do not increase. Compare denominators as well as values: how many requests were scheduled, admitted, completed, failed or left unresolved?

EXPLORE THE METHOD

Compare the shape of outcomes, not one headline number

Workload

Pin demand, input population, execution mode and observation window.

DESIGN TAKEAWAYS
  • Include timeout and retry costs.
  • Inspect completion and correctness together.
  • Show threshold breaches rather than only averages.
CHAPTER 03

Calibrate the conclusion

Use independent measurements to check the model where possible. Retain the workload match and uncertainty rather than implying universal performance.

Explain the evidence that supports the modeled service behavior and the conditions where that support is weak. A synthetic workload is useful for exploring a hypothesis, but it does not establish universal throughput or latency for a customer deployment.

ILLUSTRATIVE WORKED EXAMPLE

Is a synthetic faster response path actually completing more useful work?

REHEARSE

Use the same arrival pattern for both candidates. Inspect completion and correctness alongside response-time distribution, timeout count and retry work.

REVIEW

Reject a simple “faster” conclusion if the candidate excludes more requests from its successful-response latency calculation. Review the full denominator first.

Synthetic evaluation illustration. No customer result or benchmark is claimed.
DESIGN TAKEAWAYS
  • State which measurements support the model.
  • Keep illustrative data labeled.
  • Repeat only when inputs or uncertainty warrant it.
CHAPTER 04

Separate response time from useful work

Choose metrics that describe the business question. Response latency, queue wait, completion count, correctness and resource consumption reveal different failure modes. A single aggregate score can hide a regression that matters to the workflow.

Look at tails and exceptions alongside averages. Record timeouts rather than quietly dropping them from the comparison. If an observation window ends with work pending, carry that pending work into the interpretation.

Read metrics in context

MetricPair it withReason
Average latencyTail latency and timeout countsAn average can hide a slow or failed minority.
ThroughputCorrect completions and scheduled demandMore attempts do not necessarily mean more useful work.
Resource costQueue pressure and retry trafficA local optimization can shift cost elsewhere.
DESIGN TAKEAWAYS
  • Keep scheduled and completed counts distinct.
  • Show retries and failed operations.
  • Review correctness with the performance result.
CHAPTER 05

Read charts with their assumptions attached

The graphic in this guide is a conceptual comparison, not a benchmark. A useful measured chart should state its units, population, timing method, observation window and whether it represents modeled or observed behavior.

When repeating an experiment, preserve the earlier result and explain what changed. Additional runs can explore variability or changed inputs; selecting only the best outcome changes the evidence. Review uncertainty and workload fit before making a wider performance claim.

WHAT TRAVELS WITH THE RESULT
01Input references
02Model + scenario
03Complete outcomes
04Review + limits

Keep the evidence and its conditions together. A later success does not erase an earlier failure or broaden the original scope.

DESIGN TAKEAWAYS
  • Label illustrative and measured data separately.
  • Retain scenario settings and versions.
  • Avoid translating local measurements into production capacity promises.
CHAPTER 06

Prepare a comparison review

Bring a paired baseline and candidate with enough context for another person to understand the tradeoff. Agree the thresholds and unacceptable outcomes before inspecting the candidate result.

The review may accept a bounded improvement, reject a regression or request a more representative workload. Customer sustained-load and burst qualification remain separate from internal synthetic scenarios and bounded local checks.

Review preparation

0 / 6 prepared
Selections stay in this page session. They do not approve an evaluation, grant permissions or establish production readiness.
DESIGN TAKEAWAYS
  • Freeze workload and targets.
  • Record all outcome categories.
  • State which deployment tests still remain.
TURN THE GUIDE INTO A PLAN

Bring the question.
Build the evidence.

Define one meaningful decision, the conditions it depends on and the evidence your reviewers need.

Create your evaluation brief ↗