Run A/B tests and personalize your store without code. 14-day free trial, no credit card.
Start free trial →In-depth guides and case studies where this concept is put to work.
Hypothesis testing is the formal statistical process used to determine whether the difference in performance between a control and a variant in an A/B test is real (attributable to the change) or merely a product of random variation. It involves formulating a null hypothesis (no effect) and an alternative hypothesis (a real effect exists), collecting data, calculating a test statistic and p-value, and deciding whether to reject the null hypothesis based on a pre-set significance threshold.
The process follows these steps:
Every A/B test is a hypothesis test, whether you think of it that way or not. The difference between a rigorous testing program and a p-hacking exercise is whether teams follow the formal structure: setting α before the test, committing to a sample size, and not adjusting the hypothesis based on early results. D2C brands that skip this rigor end up with a portfolio of "winning" tests that don't hold up when shipped — because each result was actually noise that happened to cross an arbitrary threshold.
Nykaa's analytics team ran an experiment on their beauty editorial pages, testing whether adding a "Shop This Look" CTA below editorial content would increase product page visits. Their null hypothesis: adding the CTA does not change the CTR to product pages. Their alternative hypothesis: the CTA increases product page CTR. After setting α = 0.05 and collecting 28 days of data, the p-value came in at 0.019 — below the 0.05 threshold. They rejected the null hypothesis and shipped the CTA, which went on to drive a measurable increase in product discovery sessions.
A/B testing is applied hypothesis testing. Every time you set a significance threshold, run a test to sample size, and check a p-value, you are executing a formal hypothesis test. Most A/B testing platforms automate the calculation but not the discipline — the rigor of defining hypotheses upfront, respecting run time, and interpreting results honestly is the responsibility of the practitioner.
Run smarter A/B tests with CustomFit.ai — 14-day free trial, no credit card required.