The Minimum Detectable Effect (MDE) is the smallest relative improvement in a target metric that an A/B test is designed and powered to detect reliably. Before launching an experiment, you specify the MDE to answer: "What is the smallest lift I care about detecting?" This determines how many visitors you need per variant (sample size) and how long the test must run. A smaller MDE requires a larger sample size; a larger MDE requires less traffic but only catches bigger improvements.
The MDE is an input to the sample size formula. Most power calculators ask for:
- Baseline conversion rate
- MDE (as a relative percentage, e.g., 10% means detecting a lift from 3.0% to 3.3%)
- Desired statistical power (typically 80%)
- Significance level (typically 95%)
As a rough guideline, smaller MDE → larger sample size → longer test:
- MDE = 5% relative lift at 3% baseline → ~50,000 visitors per variant
- MDE = 10% relative lift at 3% baseline → ~13,000 visitors per variant
- MDE = 20% relative lift at 3% baseline → ~3,400 visitors per variant
Why MDE Matters for Ecommerce
MDE is the honest answer to "what can we actually test?" For a Shopify store with 2,000 monthly visitors to a product page, the practical MDE is very large — say, 20–30% relative lift — because detecting smaller effects requires more traffic than the page sees in a reasonable timeframe. Setting an unrealistically small MDE (e.g., 2% lift) when you have limited traffic means your test will either run for 18 months or produce an underpowered, unreliable result. Being honest about MDE forces teams to either test only high-impact hypotheses on low-traffic pages, or reserve small-effect testing for high-traffic pages where the sample size is achievable.
Real-World Example
Nykaa's CRO team wanted to test a small copy change on their product page — adjusting the product subtitle to include a key benefit. Their baseline add-to-cart rate was 5.2%. Their product page received 15,000 visitors per week. Running their power calculator with:
- Baseline: 5.2%
- MDE: 5% relative lift (detecting a change from 5.2% to 5.46%)
- Power: 80%
- Significance: 95%
Output: 24,000 visitors per variant = ~3.2 weeks. A 5% MDE was achievable. Had they set MDE at 2%, they would have needed 150,000+ visitors per variant — nearly 20 weeks. They confirmed 5% was a meaningful commercial threshold and ran the test.
How to Set MDE Correctly
- Define the minimum commercially meaningful lift for the test — what CR improvement would justify shipping the change and any ongoing maintenance?
- Use your store's actual traffic data to sanity-check whether a given MDE is achievable in 4–6 weeks.
- Set MDE based on what's detectable, not what's desirable — wishing for a 2% MDE on low-traffic pages doesn't make it statistically feasible.
- Document the MDE in the test brief before launch, alongside the hypothesis and primary metric.
- Adjust the test hierarchy by MDE and traffic: run low-MDE tests (requiring large samples) on high-traffic pages; run high-MDE tests on low-traffic pages where only major changes are worth testing.
MDE in A/B Testing
MDE is the threshold that connects the statistical world (sample size, power, significance) to the commercial world (is this lift worth detecting and acting on?). It is the most under-used concept in ecommerce experimentation and the most important one for avoiding underpowered tests that can't answer the questions they were designed to answer.
Run smarter A/B tests with CustomFit.ai — 14-day free trial, no credit card required.