Learn exactly how to run A/B tests on your website — from forming a hypothesis to reading results. A practical, jargon-free guide with real examples for D2C and ecommerce marketers.
Concepts referenced in this article, defined.
Run rigorous A/B tests and personalize every visit on Shopify or any storefront — no engineers required.
Running an A/B test means creating two or more versions of a web page, splitting your traffic between them, and measuring which version drives more of your target outcome. It's the most reliable way to make decisions about your website based on data rather than opinions.
This guide walks through exactly how to run an A/B test — from choosing what to test to reading your results.
Not all A/B tests are equal. The highest-value tests change something that affects whether a visitor converts — not just whether they click.
High-impact elements to test:
How to pick your first test: Look at your analytics funnel. Where do you lose the most visitors? If 60% of visitors who reach your product page don't add to cart, start there — even a 1% improvement in add-to-cart rate has compounding revenue impact.
Every A/B test needs a hypothesis in this form:
"If we [change X], we expect [outcome Y] because [reason Z]."
Example: "If we change the add-to-cart button copy from 'Add to Cart' to 'Get Yours Now', we expect add-to-cart rate to increase because the more action-oriented copy creates urgency."
The hypothesis keeps your test focused and ensures you'll learn something useful whether the test wins or loses.
Before you launch, decide what "winning" means. Your primary metric should be a business outcome, not a proxy:
| Proxy metric (weak) | Business metric (strong) |
|---|---|
| Click-through rate | Add-to-cart rate |
| Time on page | Checkout completion rate |
| Bounce rate | Revenue per visitor |
| Scroll depth | Conversion rate |
Pick one primary metric and stick to it. Secondary metrics (AOV, pages per session) can provide context but shouldn't determine the winner.
Use a sample size calculator before launching. You need to know:
Rough guide: If your current conversion rate is 2%, you need approximately 5,000 visitors per variant to detect a 5% relative improvement (i.e., from 2.0% to 2.1%) with 95% confidence.
CustomFit.ai shows your test's projected time-to-significance based on your current traffic volume.
Using CustomFit.ai's visual editor:
Rule: Change one meaningful thing per variant. If you change the headline, button copy, and hero image all at once, you won't know which change drove the result.
Traffic split: Start with 50/50 (control vs. variant). If you're nervous about showing an untested experience to half your visitors, start with 80/20 — but know this will take longer to reach significance.
Audience: By default, run on all visitors. Once you're comfortable with A/B testing, segment by traffic source, device type, or behavioral signals for more targeted tests.
Exclusions: Exclude internal team IP addresses, logged-in staff, and bot traffic from your test data.
Hit publish. Do not check results for at least 7 days.
Peeking at results early and stopping a test when you see a winning trend (called "peeking") is one of the most common A/B testing mistakes. Statistical significance fluctuates early in a test — a variant that looks like it's winning on day 3 may reverse by day 10.
Minimum test duration: 14 days, always. This captures two full weeks of visitor behavior, including the weekend/weekday variation that affects most websites.
After your test runs for at least 14 days and reaches the required sample size, read your results:
If you have a clear winner (95%+ confidence): Ship the winning variant to 100% of visitors with one click. Document what you tested, what won, and why you think it won. Use the insight to generate your next hypothesis.
If the test is inconclusive: Neither version is statistically better. This is still valuable — it tells you that the element you tested doesn't significantly impact conversions. Move on to testing something with higher potential impact.
If the variant loses: The losing insight is as valuable as the winning one. Document why the change might have hurt performance and use that to refine your mental model of your audience.
Stopping too early: Significance fluctuates. Always run for the full minimum duration.
Testing too many things: Multivariate tests require exponentially more traffic. Start with clean A/B tests.
Ignoring seasonality: Don't run a test during a major sale, holiday, or traffic spike — the aberrant conditions will skew your results.
Testing low-traffic pages: A/B testing requires traffic. If the page you want to test gets 500 visitors per month, you'll need 4+ months to reach significance. Focus tests on your highest-traffic pages first.
Continue reading:
Run your first A/B test today — start your free CustomFit.ai trial. First test live in under 30 minutes.