Traffic Splitting & Statistics

When a new visitor arrives, Iteratist assigns them to a variant randomly, weighted by the traffic split percentages you've configured. The assignment is then tied to the visitor's unique ID — so the same visitor always sees the same variant on every subsequent visit. This prevents a single person from being counted as both a control and variant visitor, which would contaminate your results.

The 50/50 setting is a probability for each visitor — not a guarantee of the final ratio. Each visitor independently has a 50% chance of being assigned to either variant, similar to flipping a coin. With any finite number of visitors, the actual split will fluctuate around the target — just as flipping a coin 100 times rarely gives exactly 50 heads.

With 200 visitors the split might naturally land at 47/53. With 2,000 visitors it will be much closer to 50/50. This is normal statistical variance and does not affect the validity of your results — the significance calculation accounts for unequal sample sizes automatically.

Iteratist uses a two-proportion z-test to compare conversion rates between the control and each variant. The z-score measures how many standard deviations apart the two conversion rates are, taking into account the sample sizes of both groups. This gives a confidence percentage — the probability that the observed difference is real and not due to random chance.

A minimum of 30 visitors per variant and at least 3 total conversions are required before any significance calculation is shown, to prevent misleading early results.

Iteratist requires 95% statistical confidence before declaring a winner. This means there is a 95% probability that the observed conversion rate difference is genuine and not a result of random variation — and a 5% chance it could be a false positive. This is the standard threshold used across the testing industry and balances the need for certainty against the time cost of running tests longer.

A z-score measures how far apart two conversion rates are relative to the natural variation you'd expect from random chance. A z-score of 1.96 or above corresponds to 95% confidence. A z-score of 2.576 or above corresponds to 99% confidence. The higher the z-score, the more certain you can be that the difference between variants is real. Iteratist shows you the current z-score alongside the confidence percentage in the results report.

Once a winner is declared for a given traffic source, Iteratist automatically routes 100% of future traffic from that source to the winning variant. This happens in real time with no manual action required. The traffic source is marked as "Optimised" in the report, and a post-optimisation conversion rate is tracked separately so you can measure the uplift.

Importantly, winners are locked permanently. Adding more data after declaration does not and cannot revert a declared winner — the decision is irreversible once the 95% threshold is crossed.

Learning means the experiment is still running for this traffic source and hasn't yet gathered enough data to declare a winner with statistical confidence. Traffic is being split across variants according to your traffic split settings.

Optimised means a winner has been declared for this source with at least 95% confidence. All traffic from this source is now being served the winning variant automatically. The experiment has effectively "graduated" for this source.

As a rule of thumb, aim for at least 100 conversions per variant — ideally 200 or more. The number of visitors you need depends entirely on your baseline conversion rate and the size of the effect you're testing. A page converting at 1% needs far more visitors to detect a 0.2% improvement than a page converting at 10%.

Iteratist's significance calculator handles this automatically. Watch the confidence percentage rise as data accumulates — when it crosses 95%, you have your answer.