Use a fair baseline
Compare alternatives under the same workload, observation window and execution mode. Measured client tests and virtual-time scenarios answer different questions.
State whether timing comes from a modeled virtual-time scenario or a measured client test. These are different observations and should not share an unlabeled chart. Keep warm-up, arrival pattern, concurrency, data distribution and observation window visible so the comparison can be interpreted.
- Pin demand and data assumptions.
- Separate average and tail latency.
- Record the experiment seed where applicable.
Look beyond one metric
A faster response can create more retries, queue pressure or failed operations. Compare acceptance targets across several outcomes.
A latency improvement among successful responses may hide a higher timeout rate. Retry traffic can increase work even when useful completions do not increase. Compare denominators as well as values: how many requests were scheduled, admitted, completed, failed or left unresolved?
Compare the shape of outcomes, not one headline number
Pin demand, input population, execution mode and observation window.
- Include timeout and retry costs.
- Inspect completion and correctness together.
- Show threshold breaches rather than only averages.
Calibrate the conclusion
Use independent measurements to check the model where possible. Retain the workload match and uncertainty rather than implying universal performance.
Explain the evidence that supports the modeled service behavior and the conditions where that support is weak. A synthetic workload is useful for exploring a hypothesis, but it does not establish universal throughput or latency for a customer deployment.
Is a synthetic faster response path actually completing more useful work?
Use the same arrival pattern for both candidates. Inspect completion and correctness alongside response-time distribution, timeout count and retry work.
Reject a simple “faster” conclusion if the candidate excludes more requests from its successful-response latency calculation. Review the full denominator first.
- State which measurements support the model.
- Keep illustrative data labeled.
- Repeat only when inputs or uncertainty warrant it.
Separate response time from useful work
Choose metrics that describe the business question. Response latency, queue wait, completion count, correctness and resource consumption reveal different failure modes. A single aggregate score can hide a regression that matters to the workflow.
Look at tails and exceptions alongside averages. Record timeouts rather than quietly dropping them from the comparison. If an observation window ends with work pending, carry that pending work into the interpretation.
Read metrics in context
| Metric | Pair it with | Reason |
|---|---|---|
| Average latency | Tail latency and timeout counts | An average can hide a slow or failed minority. |
| Throughput | Correct completions and scheduled demand | More attempts do not necessarily mean more useful work. |
| Resource cost | Queue pressure and retry traffic | A local optimization can shift cost elsewhere. |
- Keep scheduled and completed counts distinct.
- Show retries and failed operations.
- Review correctness with the performance result.
Read charts with their assumptions attached
The graphic in this guide is a conceptual comparison, not a benchmark. A useful measured chart should state its units, population, timing method, observation window and whether it represents modeled or observed behavior.
When repeating an experiment, preserve the earlier result and explain what changed. Additional runs can explore variability or changed inputs; selecting only the best outcome changes the evidence. Review uncertainty and workload fit before making a wider performance claim.
Keep the evidence and its conditions together. A later success does not erase an earlier failure or broaden the original scope.
- Label illustrative and measured data separately.
- Retain scenario settings and versions.
- Avoid translating local measurements into production capacity promises.
Prepare a comparison review
Bring a paired baseline and candidate with enough context for another person to understand the tradeoff. Agree the thresholds and unacceptable outcomes before inspecting the candidate result.
The review may accept a bounded improvement, reject a regression or request a more representative workload. Customer sustained-load and burst qualification remain separate from internal synthetic scenarios and bounded local checks.
Review preparation
0 / 6 prepared- Freeze workload and targets.
- Record all outcome categories.
- State which deployment tests still remain.