The null hypothesis (H₀) in A/B testing is the default assumption that there is no real difference in the target metric between the control and the variant. It represents the skeptical position: any observed difference in conversion rates is simply due to random sampling variation, not a genuine effect of the change being tested. Statistical hypothesis testing is designed to either reject the null hypothesis (when evidence is strong enough) or fail to reject it (when evidence is insufficient).
How the Null Hypothesis Works in Practice
In a conversion rate experiment:
- H₀: Variant conversion rate = Control conversion rate (no real difference)
- H₁ (alternative): Variant conversion rate ≠ Control conversion rate (a real difference exists)
The test collects data and computes a p-value — the probability of observing a difference at least as large as the one measured, assuming H₀ is true. If p < 0.05 (the typical threshold), you reject H₀ and conclude the difference is statistically significant.
Failing to reject H₀ is not the same as proving H₀ is true. It simply means you did not collect enough evidence to rule out chance.
Why the Null Hypothesis Matters for Ecommerce
The null hypothesis is the conceptual safeguard against confirmation bias. Without it, teams would interpret any positive-looking data as proof that their variant works — ignoring the ever-present possibility of random noise. For D2C brands, the cost of a false rejection (shipping a change that doesn't actually help) is measured in lost revenue and wasted engineering cycles. Respecting the null hypothesis — and only rejecting it when evidence is genuinely strong — is what makes an optimization program trustworthy rather than illusory.
Real-World Example
A Shopify apparel brand added a "As Seen In" media logo strip to their homepage, expecting it to build trust and improve conversion. The null hypothesis: adding the logo strip does not change homepage conversion rate. After 18 days and 24,000 visitors per variant, the p-value was 0.21 — far above the 0.05 threshold. The team failed to reject the null hypothesis. Rather than shipping the change "because it probably helps a little," they moved on to a higher-impact hypothesis. This discipline kept their test backlog focused and their shipped changes credible.
Common Mistakes with the Null Hypothesis
- Treating "fail to reject" as "proven false": an inconclusive test doesn't mean your hypothesis was wrong; it may just mean the test was underpowered.
- Adjusting the threshold after seeing results: setting α = 0.1 because your p-value came in at 0.08 is p-hacking.
- Testing the wrong null: if your primary metric is conversion rate but you reject the null on a secondary metric (e.g., time on page), that result doesn't justify shipping the variant.
- Running multiple tests without correction: testing 20 variants simultaneously means you'd expect one false rejection at α = 0.05 even if all variants had no real effect.
Null Hypothesis in A/B Testing
Every A/B test implicitly operates under a null hypothesis. Well-run experimentation programs make this explicit: teams write out H₀ and H₁ as part of the test brief, set α in advance, and document both the hypothesis and the statistical outcome so organizational learning accumulates correctly over time.
Run smarter A/B tests with CustomFit.ai — 14-day free trial, no credit card required.