Put this into practice
Run A/B tests and personalize your store without code. 14-day free trial, no credit card.
Start free trial →Run A/B tests and personalize your store without code. 14-day free trial, no credit card.
Start free trial →The chi-square test (χ² test) is a statistical test used to determine whether observed differences in categorical outcomes — such as conversion rates — between two or more groups are statistically significant or likely due to random chance. In A/B testing, it is commonly applied to binary conversion data: the number of visitors who converted versus those who did not, in the control versus the variant.
For a 2×2 contingency table (control vs. variant, converted vs. not converted):
χ² = Σ [(Observed − Expected)² / Expected]
| Group | Converted | Not Converted | Total |
|---|---|---|---|
| Control | O₁ | O₂ | N₁ |
| Variant | O₃ | O₄ | N₂ |
Where Expected = (Row Total × Column Total) / Grand Total
The resulting χ² value is compared against a critical value table (or converted to a p-value) for the given degrees of freedom (df = 1 for a 2×2 table) to determine significance.
Most A/B testing tools perform this calculation automatically — understanding the mechanics helps you interpret the output intelligently.
The chi-square test is the workhorse of conversion rate testing because conversion is a binary event: a visitor either buys or doesn't. It's well-suited to large samples, which is the typical condition in ecommerce A/B tests. For Shopify and D2C brands, understanding that your testing platform is running a chi-square (or equivalent z-test for proportions) under the hood helps you trust the results — and spot when the test was underpowered or terminated too early.
Pilgrim ran a checkout test comparing a single-page checkout (variant) against their existing multi-step checkout (control). Over 21 days, 12,400 visitors reached checkout in the control group, of which 6,820 completed the purchase (55% completion rate). The variant saw 12,600 visitors with 7,434 completions (59% completion rate). Running a chi-square test on these four numbers produced χ² = 48.3 with p < 0.001 — extremely strong evidence that the single-page checkout genuinely outperformed the multi-step version, not a chance result.
Most A/B testing platforms use either a chi-square test or an equivalent z-test for proportions (which gives nearly identical results at large sample sizes) to assess conversion rate significance. If you're building your own analysis in Python or R, scipy.stats.chi2_contingency and statsmodels.stats.proportion.proportions_ztest are the standard implementations.
Run smarter A/B tests with CustomFit.ai — 14-day free trial, no credit card required.