Plan the decision before launching
Before launch, define the outcome, variants, baseline, smallest worthwhile effect, eligible traffic, allocation and guardrails. Those details determine whether the test can support a useful decision; total site sessions alone do not.
Protect the integrity of the result
Check event tracking, consistent assignment, variant quality and stopping rules. Looking repeatedly for a favourable result or stopping early can mislead. Don't trade qualified leads, revenue or checkout completion for a better proxy metric.
Use an alternative when a conventional test is weak
If traffic is low or tracking is unreliable, start with measurement repair, usability research or a direct fix rather than an A/B test. A carefully qualified before-and-after review or testing for a larger meaningful effect may help. A non-significant result is inconclusive, not proof of no effect.
Read an experiment honestly
Report the observed difference, uncertainty, primary and guardrail metrics, implementation quality, dates and caveats. Inconclusive results, implementation issues and trade-offs still guide decisions. Don't generalize a short-term lift or treat unstable comparisons as causal proof.
Make readiness a scoping conversation
Use this readiness conversation for a page or journey with an eligible audience, event definition, baseline period, likely implementation owner and a decision at stake. It helps identify the next step: tracking QA, research, a fix, test design or broader conversion diagnosis—not self-certify readiness.
Common questions
How much traffic is enough for an A/B test?
There is no universal threshold. Check baseline conversion, worthwhile effect, eligible traffic, allocation, variants, statistical method and metric quality to judge feasibility.
Does a statistically significant result guarantee business value?
No. Check effect size, uncertainty, guardrails, data quality and business context before you act.
How do we estimate the sample size an A/B test needs?
Estimate sample needs before launch using the baseline rate or metric variability, the smallest effect worth detecting, and the chosen power and error threshold. The smaller the effect you want to distinguish from noise, the more data is generally required. Use inputs that match the actual eligible traffic and metric, not total site visits by default. Sources and context
