Ready to Test? How Long Should It Run?
A/B tests (as well as other forms of testing including those of the multivariate variety) can run for any length of time but ideally will do so for at least a full seven-day week (and typically no longer than a few months).
When tests are run for too long, additional variables can influence the results (like seasonal changes or promotions). Since accurate test results are the goal, it is important that businesses set up a timeframe prior to execution. Although this timeframe will vary depending on each business and its specific goals, here are some tips for getting started:
Test by Sample Size
One way to decide the length of tests is by sample size, as test results won’t be accurate if the testing audience is too small. Fortunately, there are many tools available to help marketers figure out the right sample size. Optimizely’s Sample Size Calculator, for instance helps determine the traffic needed for a marketer’s conversion rate tests. Another option comes from Visual Website Optimizer, which offers an A/B Split and Multivariate Test Duration Calculator.
Test by Purchase Cycle
In addition to sample size-based tests, marketers can also create tests based on purchase cycles. For example, if the typical purchase cycle of a business is three weeks, the test should be set to run for that time frame, otherwise the test results will be polluted with data from before and/or after the typical purchase cycle, making the data unreliable.
Test with a Set Time
Lastly, marketers can set up tests to run for a specific amount of time. For example, if a retailer is testing what type of promotions will boost sales between Black Friday and the Christmas holiday, the retailer will only want to run tests during that time frame in order to get accurate results that can help him or her optimize their strategies for the next year.
Good Testing Software Will Help
Although testing time frames will vary depending on each company's unique goals, the right testing software can help marketers identify how their tests should run. Many solutions offer test duration calculators like the two mentioned above, while other solutions may provide expertise from optimization professionals.