How to read your results
The verdict at the top tells you, in plain English, whether your variant won, lost, or is still undecided. Behind that verdict are three numbers worth understanding: the relative lift, the p-value, and the confidence interval.
Relative lift is how much better (or worse) the variant converts compared to the control. The p-value is the probability of seeing a difference at least this large if there were truly no difference — lower is stronger evidence. The confidence interval shows the plausible range for the true lift; when that range excludes zero, your result is significant.
What statistical significance means
A statistically significant result is one that is unlikely to be explained by random chance alone. We use a two-proportion z-test to compare the conversion rate of each variant against the control. If the resulting p-value is smaller than your alpha (1 − confidence level), the difference is significant.
Significance is not the same as importance. A tiny lift can be statistically significant with enough traffic, while a large lift may not be significant if your sample is small. Always read significance alongside the size of the lift and its confidence interval.
How sample size works
Before launching a test, decide how small an effect is worth detecting — the minimum detectable effect (MDE). The smaller the effect you want to catch, the more visitors you need. Sample size also grows as your baseline conversion rate gets lower and as you demand higher confidence or power.
Power is the probability of detecting a real effect when one exists; 80% is the common default. Use the Plan a test tab to turn these inputs into a concrete number of visitors per variant and an estimated runtime based on your traffic.
Why you shouldn't stop a test early
It is tempting to end a test the moment it crosses the significance line. Don't. Repeatedly checking and stopping at the first significant moment — known as peeking — can push your real false-positive rate well above the 5% you think you're running at.
Commit to a sample size in advance and let the test run to completion. Cover full business cycles (typically whole weeks) so weekday/weekend behavior and other patterns don't skew the outcome.
Frequently asked questions
- What is statistical significance in an A/B test?
- Statistical significance means the difference between your variant and control is unlikely to be caused by random chance. When the p-value falls below your chosen alpha (for example 0.05 at 95% confidence), the result is considered statistically significant.
- What confidence level should I use?
- 95% is the standard for most product and marketing experiments — it balances the risk of false positives against how long the test must run. Use 99% when the decision is high-stakes and hard to reverse, and 90% only for low-risk, exploratory tests.
- Should I use a one-sided or two-sided test?
- Use a two-sided test when you care about detecting any change, better or worse. Use a one-sided test only when you will never ship a losing variant and only want to know if the variant beats the control. Two-sided is the safer default.
- Why can't I stop my test as soon as it hits significance?
- Checking results repeatedly and stopping the moment you see significance (called peeking) dramatically inflates your false-positive rate. Decide your sample size in advance and run the test to completion, ideally covering full weekly business cycles.
- How many visitors do I need for an A/B test?
- It depends on your baseline conversion rate and the smallest lift you want to detect. Smaller effects and lower baseline rates require far more traffic. Use the Plan a test tab to calculate the exact sample size for your situation.
- Is this A/B test calculator really free?
- Yes. It is free forever with no signup and no email gate. All calculations run entirely in your browser, so your numbers never leave your device.