A/B Test Sample Size and Significance: A Startup Guide

To find the sample size for an A/B test, you need three inputs: your baseline conversion rate, the minimum lift you care about detecting (the minimum detectable effect), and your confidence and power targets (usually 95 percent and 80 percent). Plug them into any sample size calculator; it returns the visitors per variant you must collect before reading the result. Decide that number before you launch and do not peek early.

This is the statistics layer under A/B testing - the part that decides whether a result is real or noise. If you want the practical "what to test" playbook, the A/B testing landing pages guide covers that; this piece answers the question that sinks most startup tests: do I even have enough traffic to trust the number?

How Do You Calculate a/B Test Sample Size?

Sample size is a function of three things you set before the test, not something you discover after. Give a calculator these and it returns the visitors needed per variant:

• Baseline conversion rate - the current rate of the metric you are testing (say 4 percent).
• Minimum detectable effect (MDE) - the smallest lift worth detecting, expressed as relative or absolute change. Detecting a 20 percent relative lift needs far fewer users than detecting a 2 percent one.
• Significance and power - typically 95 percent confidence (5 percent false-positive rate) and 80 percent power (20 percent chance of missing a real effect).

The relationship that matters intuitively: smaller effects and lower baseline rates both demand dramatically more traffic. A test to detect a small lift on a rare conversion can need hundreds of thousands of visitors - which is exactly why most startup tests never reach significance.

Those figures are approximate and meant to show the shape: halving the effect you want to detect roughly quadruples the traffic you need. Always run the exact numbers in a calculator for your baseline.

What Is Statistical Significance in an a/B Test?

Statistical significance means the difference you observed is unlikely to be due to random chance. The common threshold is a p-value below 0.05, which corresponds to 95 percent confidence: if there were truly no difference between variants, you would see a result this extreme less than 5 percent of the time.

Significance is necessary but not sufficient. Two traps catch startups:

• Significant but tiny. With enough traffic even a meaningless 0.3 percent lift becomes "significant." Always ask whether the effect size is worth shipping, not just whether p is under 0.05.
• Not significant is not the same as no difference. An underpowered test that fails to reach significance has told you nothing, not that the variants are equal.

Why Do Most Startup a/B Tests Never Reach Significance?

Because startups have thin traffic and want to detect small effects, which is the exact combination that demands enormous sample sizes. A startup with a few thousand visitors a month trying to detect a 5 percent lift would need to run one test for a year. The math simply does not work at low volume, and pretending it does produces confident conclusions from noise.

What to do instead when traffic is the constraint:

1. Test bigger swings. Test radically different variants, not button colors. A large MDE needs far less traffic, and early-stage products have big things worth changing.
2. Move the test up the funnel. Test on a higher-traffic step (the landing page, not the checkout) where you have the volume to reach power.
3. Use a proxy metric with more events. If purchases are rare, test against add-to-cart or signup, which happen far more often, then validate downstream.
4. Accept qualitative methods. Below a real traffic threshold, user interviews and session replay beat a chronically underpowered test. The CRO testing framework covers how to decide when quant testing is even viable.

Why Should You Not Peek at a/B Test Results Early?

Peeking - checking the result repeatedly and stopping the moment it looks significant - inflates your false-positive rate far above the 5 percent you think you set. Every look is another chance for random noise to cross the threshold, so a test you peek at daily can hit "significance" by chance even when there is no real effect.

The fixes:

• Fix the sample size and duration up front and only read the result once you hit it.
• Run at least one to two full business cycles (usually one to two weeks) so day-of-week effects average out, even if you hit sample size sooner.
• If you need to monitor continuously, use a tool built for sequential testing or Bayesian methods, which are designed to be checked repeatedly - do not apply peeking to a fixed-horizon frequentist test.

Getting the statistics right is what makes a testing program compound instead of mislead. It pairs directly with running more tests safely - see experiment velocity for startups for how to increase throughput without lowering this bar, and the growth experimentation framework for the surrounding process.

TL;DR

• Sample size needs three inputs: baseline conversion rate, minimum detectable effect (MDE), and significance/power (usually 95% and 80%). Set them, then calculate before launch.
• Smaller effects and lower baselines demand far more traffic - halving the MDE roughly quadruples the visitors needed.
• Significance (p < 0.05) means unlikely-by-chance, not important; check effect size too, and a non-significant underpowered test tells you nothing.
• Most startup tests fail because thin traffic plus small target effects need huge samples - test bigger swings, higher-traffic steps, or proxy metrics.
• Do not peek: fix sample size and duration up front, run full business cycles, and use sequential or Bayesian tools if you must monitor continuously.

FAQ

How Do You Calculate Sample Size for an a/B Test?

You need three inputs: your baseline conversion rate, the minimum detectable effect (the smallest lift worth detecting), and your significance and power targets, typically 95 percent confidence and 80 percent power. Enter them into any sample size calculator and it returns the number of visitors required per variant. Decide that number before launching, because smaller target effects and lower baseline rates both sharply increase the traffic you need.

What Is a Good Sample Size for an a/B Test?

There is no universal number - it depends entirely on your baseline rate and the lift you want to detect. As a shape, a 5 percent baseline needs roughly 1,700 visitors per variant to detect a 50 percent relative lift, about 9,000 for a 20 percent lift, and around 36,000 for a 10 percent lift. Run your exact baseline through a calculator rather than copying a figure.

What Does Statistical Significance Mean in a/B Testing?

Statistical significance means the observed difference between variants is unlikely to be due to random chance, commonly defined as a p-value below 0.05, or 95 percent confidence. It is necessary but not sufficient: a significant result can still be too small to be worth shipping, and a non-significant result from an underpowered test tells you nothing rather than proving the variants are equal. For a full treatment of p-values, confidence, peeking, and significance versus practical impact, see our statistical significance in A/B testing guide.

Why Do Startup a/B Tests Fail to Reach Significance?

Because startups usually have thin traffic and want to detect small effects, and that combination requires very large sample sizes. A few thousand monthly visitors cannot detect a 5 percent lift in a reasonable window. The fix is to test bigger swings, move tests to higher-traffic funnel steps, use higher-frequency proxy metrics, or rely on qualitative methods like interviews and session replay when volume is too low for valid testing.

Why Is Peeking at a/B Test Results a Problem?

Peeking - repeatedly checking and stopping as soon as a result looks significant - inflates your false-positive rate well above the 5 percent you intended, because each look is another chance for noise to cross the threshold. Fix the sample size and duration up front and read the result once, run at least one to two full business cycles, and use sequential or Bayesian testing tools if you genuinely need to monitor continuously.