Put this into practice
Run A/B tests and personalize your store without code. 14-day free trial, no credit card.
Start free trial →Run A/B tests and personalize your store without code. 14-day free trial, no credit card.
Start free trial →Regression to the mean is a statistical phenomenon where extreme measurements — unusually high or unusually low values — tend to be followed by values closer to the long-run average on subsequent measurements. First described by Francis Galton in the 19th century, it occurs because extreme values often reflect a combination of a true underlying signal plus random chance. When you re-measure the same thing, the random component varies, pulling the result back toward the true average. In A/B testing and conversion optimisation, regression to the mean causes teams to over-react to short-term fluctuations in conversion rate and reach premature conclusions.
Ecommerce conversion rates fluctuate daily due to traffic composition, device mix, weather, promotional activity, and dozens of other factors. A product page that converts at 8% on Tuesday (an unusually good day) will likely convert at 5–6% on Thursday — not because anything changed, but because Tuesday was an outlier. If you start an A/B test on an unusually good day and compare Thursday's results to Tuesday's baseline, regression to the mean can make your variant look like a loser even if it is neutral or positive.
More dangerously, regression to the mean causes "false wins" when teams cherry-pick good performance windows to start tests. If you begin a test during a week when traffic quality is high (post-promotional surge of engaged buyers), the control group establishes an artificially high baseline. As conditions normalise, the baseline regresses — your variant's stable performance looks like a dramatic lift, when in reality nothing changed.
For Indian D2C brands, the conversion rate volatility around festive seasons (Diwali, Eid, Republic Day) is extreme. Any test running across a major sale event must account for the fact that conversion rates during the sale will not represent normal traffic behaviour.
A beauty brand notices their conversion rate spiked from 3.2% to 6.8% during a influencer campaign week. A junior analyst argues they should start an A/B test immediately to "ride the momentum." A more experienced team member pauses the test launch, noting that the spike is likely to regress to the 3.2–3.8% typical range once influencer traffic normalises. They start the test the following week and set the baseline from the first 3 days of the experiment itself. When the test runs, control settles at 3.5% and the variant at 3.9% — a real 11% lift that holds post-launch. Had they started mid-influencer-spike, the "baseline" of 6.8% would have made 3.9% look like a catastrophic 43% drop.
Regression to the mean is one of the primary reasons A/B tests require adequate sample sizes and run durations — with small samples, extreme results that haven't had time to regress can look statistically significant. Proper randomisation, simultaneous control measurement, and adequate test duration all mitigate regression to the mean effects in practice.
Run smarter A/B tests with CustomFit.ai — 14-day free trial, no credit card required.