Mean Comparison · Proportion Comparison · Sample Size Calculator · Statistical Power
Mean Comparison (T-Test)
Use this to compare average values between two groups of continuous numeric data — e.g. time on page, order value, average spend. Enter numbers separated by commas or spaces.
How to Read This & Common Mistakes
How to Read It
p < 0.05 means this difference is unlikely to be pure random noise — but it does not tell you how large or how important the difference is. Always look at the raw difference and relative difference together to judge whether it's worth acting on in business terms.
Degrees of freedom (df) closer to n_A + n_B means the two groups have similar variance; a noticeably lower value means the groups' spread differs more — this is normal, and the tool automatically corrects for it using Welch's t-test, so no manual adjustment is needed.
Common Mistakes
Peeking mid-test and adding more samples when the result isn't significant yet — this is the most common source of false positives. Decide on a sample size or test duration before you start, run it to completion, and don't make calls based on interim results.
Small samples (under 30 per group) with clearly non-normal data make the t-test less reliable — collect more data, or use a non-parametric method instead.
Treating "statistically significant" as the same thing as "practically important" — with a large enough sample, even a tiny, meaningless difference can produce p < 0.05. Always check the relative difference too.
Proportion Comparison (Z-Test)
Use this to compare conversion rates, click-through rates, open rates, or other proportion-based metrics between two versions.
Control Group (A)
Test Group (B)
How to Read This & Common Mistakes
How to Read It
The 95% confidence interval is more informative than the p-value alone — it tells you the plausible range for the true difference. If the interval crosses zero (one end positive, one negative), the evidence isn't strong enough to establish direction. This tool computes the interval using the Newcombe (Wilson score) method, which is more robust than the simple normal approximation (Wald interval) taught in most textbooks, especially with small samples or rates near 0%/100% — it won't produce an impossible negative lower bound.
Look at both the absolute difference (percentage points) and the relative lift (%): going from a 1% to a 2% conversion rate is a 100% relative lift, but only 1 percentage point in absolute terms — these tell different stories, so report both.
In extreme edge cases with very small samples, the p-value and confidence interval can appear to disagree — the p-value relies on a normal approximation, while the interval uses Wilson score, and they aren't built on quite the same mathematical framework. Occasionally you may see p < 0.05 while the interval still sits right around zero. This is a known property of the two methods, not a calculation error — with very small samples, treat both numbers cautiously and prefer collecting more data.
Common Mistakes
Very small conversion counts (single digits) make the normal approximation behind the p-value unreliable — treat results as indicative only and gather more data before concluding anything.
Ignoring test duration — even if you've hit your target sample size, stopping after just a day or two can leave results contaminated by time-of-week effects. Run at least one full business cycle (e.g. a full week covering both weekdays and weekends).
Looking only at relative lift, not the absolute base — a "doubling" on a tiny base often isn't as dramatic in business terms as it sounds. Convert to absolute numbers before making a decision.
Watching too many metrics at once and reporting whichever one comes up significant — if you're monitoring conversion rate, average order value, time on page, and more all at once, test enough metrics and something will look "significant" by chance alone. This is the multiple comparisons problem. Decide on one primary metric before the test starts, and treat everything else as supporting context — not as the basis for your verdict.
Sample Size Calculator
Estimate the sample size needed per group based on your baseline conversion rate and the minimum effect you want to be able to detect (MDE). This is the formula to use for actual A/B testing.
When your total population is limited (e.g. an internal staff survey or an existing customer list), calculate the required sample size using the finite population correction.
How to Read This & Common Mistakes
The Difference Between the Two Modes (easiest thing to mix up)
A/B Test Mode answers: "How many people do I need per group to reliably detect a difference of a given size between two versions?" This involves the concept of "detecting an effect," so you first assume a baseline conversion rate and the minimum lift you want to catch (MDE).
Population Survey Mode answers a completely different question: "I have a population of known size — how many do I need to sample to keep my survey's margin of error within a given range?" There's no "comparing two groups" concept here — it's purely descriptive sampling.
The two formulas are not interchangeable — using the population-survey sample size for an A/B test will almost always underestimate what you actually need, because it doesn't account for the statistical power required to detect a difference between two groups.
The A/B Test Mode formula assumes an even 50/50 traffic split between the two groups. If your test design uses an uneven split (e.g. 90/10), the total sample size you'll actually need is higher than what's shown here — that requires a different, unequal-allocation version of the formula.
Common Mistakes
The smaller the MDE, the larger the required sample size — and not linearly. If you're trying to detect a very small difference, the sample size can become unrealistic; reconsider whether the effect size you're targeting is reasonable.
Even after reaching the calculated sample size, run at least one full business cycle — stopping after just 2 days can still leave results contaminated by time effects (e.g. a different weekend traffic mix).
Statistical Power Calculator
Given a per-group sample size, work out the statistical power your test can achieve — the probability you'll detect a real difference if one exists.
How to Read This & Common Mistakes
How to Read It
Power below 80% means: even if the difference you entered genuinely exists between the two versions, your test design has a substantial chance of failing to detect it (a false negative). This is not evidence of "no difference" — it just means this particular test design lacks the ability to find it.
Common Mistakes
Post-hoc power calculations — many people, after a test comes back non-significant, calculate power afterward to explain away the result ("power was too low, so it doesn't mean there's no difference"). This use is contested in the statistics field. The correct use is to calculate power before the experiment starts, to plan sample size — not to defend a result after the fact.
Treating "power is insufficient" as evidence that "the two groups have no difference" — these are two different things: insufficient power only means this particular experiment lacked the ability to detect a difference; it says nothing about whether a difference actually exists.