Statistical significance is not a stopping rule. That's the most important thing most articles about A/B testing get wrong. Crossing the 95% confidence threshold tells you the result is unlikely to be noise — it doesn't tell you whether you've run the test long enough, collected enough data, or accounted for the decisions that come after the data is collected.
Knowing when to stop is as important as knowing what to test. Get the timing wrong and the data will mislead you.
Marketers investing in conversion optimisation platforms see an average ROI of 228.60% — essentially unchanged from 231.08% in 2025, but consistently well over 2×. That return compounds only when winners are shipped promptly and new tests are launched. Stopping tests on time is what keeps the engine running. — 2026 Growth Marketer Survey
Pre-Define Your Stopping Criteria
The only reliable way to stop a test correctly is to define the stopping criteria before the test starts. Write down three numbers before you press launch:
- Minimum runtime: typically two full business cycles (usually two calendar weeks) to smooth out weekday/weekend and week-over-week variation
- Minimum sample size per variant: calculated from your baseline conversion rate, target minimum detectable effect, and required statistical power (usually 80–90%)
- Maximum runtime: a hard ceiling — usually 4–6 weeks — after which the test is called regardless of significance, to prevent seasonal contamination
Once you've written these numbers down, commit to them. Don't revisit the criteria mid-test because the early data looks interesting.
Why Peeking Causes False Positives
Every time you look at a test before it reaches its minimum runtime, you're making an implicit decision: should I stop now? The more often you check, the more likely you are to catch a false positive — a moment where the results look significant purely by chance, even when the true effect is zero.
Research on this "peeking problem" shows that teams who check results daily can experience effective false positive rates exceeding 25%, even when their nominal significance threshold is 95%. That means roughly one in four apparent "winners" they ship is noise.
The Business Decision Layer
Statistical significance tells you the result is real. It doesn't tell you whether to act on it. Before declaring a winner, ask three additional questions:
- Is the absolute conversion rate change large enough to be commercially meaningful? A 0.1% lift on a low-traffic page may not justify the engineering cost of shipping the variant.
- Does the winner hold across your most important segments — device type, traffic source, new vs. returning visitor? A variant that wins on desktop but loses on mobile isn't a clear winner.
- Does the winning variant have any negative effects on secondary metrics — time on site, downstream funnel conversion, support volume? A short-form that converts more sign-ups but produces lower-quality leads is a trade-off, not a win.
When to Call a Flat Result
Not all tests produce clear winners. If a test runs to its maximum runtime and neither variant significantly outperforms the other, that result is still information: the change you tested doesn't matter enough to move the needle for your audience. File it, update your hypotheses about what this audience responds to, and move to the next test.
The goal is not to win every test. The goal is to run enough correctly designed tests that the aggregate impact compounds. The 228% platform ROI in the survey data comes from programmes that ship winners quickly, call flat results honestly, and keep the test queue moving.