Novelty effect is a cognitive bias in A/B testing where a new variant performs artificially better (or occasionally worse) not because of its inherent quality, but because it is unfamiliar — users interact with it differently out of curiosity or novelty. Returning visitors who are accustomed to the original design notice the change and engage with it more than they normally would; new visitors are unaffected. As time passes and returning visitors habituate to the variant, the elevated engagement fades, and conversion rates regress toward the variant's true long-run performance.
Why Novelty Effect Matters for Ecommerce
For D2C brands with a loyal returning customer base, novelty effect can produce misleading A/B test wins that fail to hold post-launch. You ship a "winner" with a 12% conversion lift, but three weeks after full launch, the lift has evaporated — or worse, reversed. The team wasted engineering resources shipping a change that doesn't help.
Novelty effect is most pronounced for tests involving changes that returning users will notice: homepage redesigns, navigation restructures, new interactive elements (video backgrounds, sticky headers), or chat widget additions. Functional tests on typically unnoticed elements (backend algorithm changes, pricing display formatting, meta-description updates) are less susceptible.
For Indian ecommerce brands with high repeat purchase rates — subscription boxes, supplements, baby care products (Mamaearth, The Good Bug) — novelty effect is a meaningful concern. A loyal customer who buys monthly represents a returning user; if 40% of your test traffic is repeat buyers, novelty effect can significantly distort results.
Real-World Example
A direct-to-consumer tea brand adds a persistent chat bubble with a "Find Your Perfect Blend" quiz to their homepage. In the A/B test (4 days, 6,000 visitors), the chat variant shows a 23% lift in quiz engagement and a 9% lift in first-purchase conversion. Excited, they ship it. Three weeks later, engagement with the chat bubble has dropped 70% — returning customers who tried the quiz once no longer interact with it. First-purchase conversion (new visitors only) holds at 6%, a more modest 4% lift. The initial 9% was partly novelty. Had the team segmented results by new vs. returning visitors from day one, they would have seen the 4% new-visitor lift clearly and not over-estimated the impact.
How to Improve / Optimize Novelty Effect
- Segment results by new vs. returning visitors. Novelty effect only affects returning users; new visitors have no prior experience with the control. If your A/B test shows a large lift for returning visitors but not new visitors, novelty effect is likely at work.
- Run tests long enough to outlast novelty. For sites with a 2–3 week average return visit cycle, tests should run at least 3–4 weeks to capture both novelty and post-novelty behaviour. Shorter tests on high-returning-user sites over-index on novelty.
- Look at performance over time within the test. Plot daily conversion rates for the variant across the test period. If the lift is highest in week 1 and declining in weeks 2–3, novelty effect is visible in the trend line.
- Prioritise new visitor results as the primary metric. Since new visitors are unaffected by novelty, their conversion rate in the test is a cleaner signal of the variant's true long-run impact. Use returning visitor data as a secondary sanity check.
- Use a holdback test post-launch. After shipping a winner, keep 5–10% of traffic on the original for 30 days. If the lift holds in the holdback comparison, novelty effect is not a concern. If it fades, you have evidence of a novelty-driven result.
Novelty Effect in A/B Testing
Novelty effect is the temporal equivalent of selection bias — it corrupts results not through wrong assignment but through wrong timing. It is particularly relevant for high-frequency ecommerce platforms where returning users make up a significant share of traffic. Well-designed tests account for it through longer runtime, new-visitor segmentation, and post-launch holdbacks.
Run smarter A/B tests with CustomFit.ai — 14-day free trial, no credit card required.