Sequential testing is a statistical approach to A/B testing that allows the experimenter to continuously monitor results and make a decision to stop the test — declaring a winner or a null result — at any point during data collection, without inflating the false positive rate. Traditional frequentist tests require a fixed sample size determined before the experiment; peeking at results early and stopping when significance is reached violates those assumptions and produces unreliable conclusions. Sequential testing solves this by using adjusted significance boundaries (called spending functions) that account for multiple looks at the data.
The most common implementation is the Sequential Probability Ratio Test (SPRT) and always-valid inference methods used by tools like Optimizely's Stats Engine.
Why Sequential Testing Matters for Ecommerce
The ability to stop a test early — or continue it confidently — is commercially valuable for D2C brands operating on tight calendars. If a variant is clearly winning after 5 days, waiting 15 more days for a traditional test to conclude means leaving revenue on the table. Conversely, if a test is clearly going nowhere, stopping early frees traffic for more promising experiments.
Indian ecommerce brands face particularly acute version of this problem during festive seasons. A test running through Navratri, Dussehra, and Diwali experiences dramatically different traffic patterns across its duration. Sequential testing lets teams monitor actively and pull the plug — or ship winners — when the data warrants it, rather than committing to a pre-set duration that may span multiple distinct consumer behaviour windows.
Sequential testing also reduces the risk of "peeking" mistakes that lead teams to ship losers. Many ecommerce teams check their dashboards daily and stop tests when they see "significance," whether or not they planned to. Sequential methods formalise this behaviour with statistical rigour.
Real-World Example
Mamaearth runs a sequential A/B test on their homepage hero banner for their upcoming Holi sale. Control shows a general "natural ingredients" message; Variant B shows a Holi-specific promotion. Using sequential testing, they monitor daily. After 3 days, the Holi variant has a clear, statistically valid lift of 11% in click-through rate. Under a traditional fixed-horizon test, they would have needed 8 more days to reach the pre-determined sample size. Instead, they ship the winner on Day 4 — giving the Holi banner 9 days of live time before the festival instead of 1 day. The estimated revenue impact of shipping early: ₹8 lakh in additional gross merchandise value.
How to Improve / Optimize Sequential Testing
- Use a tool that natively supports sequential methods. Don't manually implement early stopping on a tool designed for fixed-horizon testing — the maths are non-trivial. Use platforms with built-in always-valid inference or Bayesian continuous monitoring.
- Understand what "early stop" means statistically. An early stop on sequential tests is valid — but it means you observed an extreme result quickly. The effect size estimate from a very early stop can be inflated (winner's curse). Plan to validate the lift post-ship with holdout analysis.
- Set minimum runtime floors. Even with sequential testing, stopping on Day 1 is almost always premature — sample sizes are too small and day-of-week effects dominate. Set a minimum of 3–5 days before the test can stop, regardless of significance.
- Apply sequential methods consistently. If your team only stops tests early when they win (not when they lose), you are introducing survivorship bias. Sequential stopping rules should apply symmetrically — stop for futility as well as significance.
- Document your sequential testing protocol. Record the spending function used, the alpha and power levels, and the minimum runtime before the test starts. This prevents post-hoc rationalisation of stopping decisions.
Sequential Testing in A/B Testing
Sequential testing is a direct response to the most common A/B testing mistake: peeking. It formalises continuous monitoring within a statistically valid framework. Modern experimentation platforms including CustomFit.ai support sequential and always-valid inference approaches, making continuous monitoring safe without the penalty of inflated Type I error rates.
Run smarter A/B tests with CustomFit.ai — 14-day free trial, no credit card required.