Concepts referenced in this article, defined.
Run rigorous A/B tests and personalize every visit on Shopify or any storefront โ no engineers required.
A CRO case study documents a completed A/B test or optimization experiment in enough detail that any future team member can understand what was tested, why, what happened, and what to do next. Most ecommerce teams run tests but fail to document them systematically โ which means learnings evaporate when team members leave, and the same failed experiments get repeated. A well-maintained case study library is a compounding asset: after 50 documented experiments, your team has a proprietary playbook that external agencies and competitors cannot replicate.
Consider two scenarios:
Scenario A (No documentation): Your growth team tests a new product page layout in Q1. It loses. In Q3, a new designer proposes the same layout change. Nobody remembers the Q1 test. The team re-runs the experiment and re-discovers the same result โ wasting 4 weeks of test capacity.
Scenario B (With documentation): The Q1 test is in the case study library. When the designer proposes the change in Q3, the growth lead can pull up the documented test in 5 minutes and share: "We tried this in March โ here's why it didn't work." The team moves on to a new hypothesis in minutes.
Scenario B teams run 20โ30% more net-new experiments per year because they are not re-running known failures.
Use this template for every completed experiment. Copy it into Notion, Confluence, Google Docs, or Airtable.
Date completed: [Month Year] Test owner: [Name] Page(s) tested: [Product page / Checkout / Email / Homepage / etc.] Test platform: [CustomFit.ai / Optimizely / VWO / Google Optimize / etc.]
What data identified this as a problem worth testing?
Example: "Product page funnel data shows 92% of visitors who reach the product page leave without adding to cart. Heatmap data shows 60% of mobile visitors do not scroll past the first screen. The add-to-cart button is below the scroll fold on mobile โ a likely friction point."
Complete the statement: "I believe [specific change] will [improve metric] because [data-backed reason]."
Example: "I believe adding a sticky 'Add to Cart' bar at the bottom of the mobile product page will increase mobile add-to-cart rate because the button will always be visible regardless of scroll position, reducing the friction of finding the CTA."
Control (Variant A): [Describe the current state] Example: "Standard product page โ add-to-cart button visible only in the static product block, approximately 600px from the top of the page on mobile."
Variant (Variant B): [Describe the change] Example: "Product page with sticky bottom bar on mobile containing product name, price, and 'Add to Cart' button. Bar appears on scroll and disappears on scroll back to top."
Traffic split: [50/50 / 60/40 / etc.] Audience: [All visitors / Mobile only / New visitors / etc.] Primary metric: [Add-to-cart rate / CVR / Checkout completion / etc.] Guardrail metric: [Bounce rate / Revenue per visitor / etc.] Start date: [Date] End date: [Date] Total visitors per variant: [Number]
Baseline metric: [e.g., "Mobile add-to-cart rate: 6.2%"] Minimum detectable effect (MDE): [e.g., "1.5 percentage points (from 6.2% to 7.7% or better)"] Required sample size: [e.g., "1,800 visitors per variant"] Target confidence level: [95% / 90%]
| Metric | Control (A) | Variant (B) | Lift | Significance |
|---|---|---|---|---|
| Add-to-cart rate | 6.2% | 8.1% | +30.6% | 97% (significant) |
| Mobile CVR | 1.8% | 2.3% | +27.8% | 95% (significant) |
| Bounce rate | 54% | 55% | +1.8% | Not significant |
| Revenue per visitor | โน43.20 | โน56.70 | +31.3% | 96% (significant) |
Winner: Variant B (sticky mobile CTA)
Translate the result into revenue impact at current traffic levels.
Example: "At 8,000 mobile product page visitors/month, moving from 6.2% to 8.1% add-to-cart rate generates approximately 152 additional add-to-carts/month. At 45% checkout completion and โน1,200 AOV, this represents approximately โน82,000 additional monthly revenue."
Go beyond the number โ explain the mechanism.
Example: "The sticky bar reduced the cognitive effort required to add to cart at any point during product page exploration. Mobile visitors who read reviews (which appear below the fold) previously had to scroll back up to find the CTA โ eliminating this step reduced drop-off during the consideration phase."
The most important insight from this test, applicable beyond this specific experiment.
Example: "Reducing the physical effort required to take action (not just the psychological hesitation) is a meaningful CRO lever. Mobile users experience more physical friction from scrolling than desktop users โ sticky CTAs should be standard on all mobile product pages, not tested on a case-by-case basis."
What should future readers know about the constraints on these results?
Example: "Test ran during the Navratri sale period when traffic composition (higher paid traffic, higher purchase intent) may have inflated the absolute CVR numbers. The relative lift (+30.6% add-to-cart rate) is likely generalizable, but the absolute metric values should be recalibrated against normal traffic baselines."
What test does this result suggest?
Example:
When a test returns no statistically significant result, use this condensed version:
Why this still deserves documentation:
Additional field for inconclusive tests:
Why we think this didn't work: Example: "The delivery countdown timer was added to the product page, but session recordings show users scrolling past it without engaging. The timer may not be visible enough (small font) or the specific delivery dates may not be relevant to browsers who have not yet decided to buy. Alternative hypothesis: delivery urgency matters most in the cart, not on the product page."
Tagging system for easy retrieval:
Quarterly digest: Every quarter, compile a "What We Learned" summary with:
Share this digest with the founding team, investors (if relevant), and all stakeholders who fund CRO resources.
Write the "Why" section immediately after the test. The mechanism explanation is hardest to reconstruct weeks later. Write it while the data and session recordings are fresh.
Include screenshots. Screenshots of control vs. variant, plus the results dashboard, make case studies far more useful for future reference.
Keep it factual. Avoid editorializing. "Variant B significantly outperformed the control" is factual. "Variant B proved that sticky CTAs are always better" is an overclaim. Let the data speak.
Link to the test in your platform. If your A/B testing platform (CustomFit.ai, Optimizely, VWO) stores historical test data, link the case study to the live test dashboard so readers can access the raw data.
Related reading: