Skip to Main Content

Ready to Test? How Long Should It Run?

Posted on 3.27.2016

A/B tests (as well as other forms of testing including those of the multivariate variety) can run for any length of time but ideally will do so for at least a full seven-day week (and typically no longer than a few months).

When tests are run for too long, additional variables can influence the results (like seasonal changes or promotions). Since accurate test results are the goal, it is important that businesses set up a timeframe prior to execution. Although this timeframe will vary depending on each business and its specific goals, here are some tips for getting started:

Test by Sample Size

One way to decide the length of tests is by sample size, as test results won’t be accurate if the testing audience is too small. Fortunately, there are many tools available to help marketers figure out the right sample size. Optimizely’s Sample Size Calculator, for instance helps determine the traffic needed for a marketer’s conversion rate tests. Another option comes from Visual Website Optimizer, which offers an A/B Split and Multivariate Test Duration Calculator.

Test by Purchase Cycle

In addition to sample size-based tests, marketers can also create tests based on purchase cycles. For example, if the typical purchase cycle of a business is three weeks, the test should be set to run for that time frame, otherwise the test results will be polluted with data from before and/or after the typical purchase cycle, making the data unreliable.

Test with a Set Time

Lastly, marketers can set up tests to run for a specific amount of time. For example, if a retailer is testing what type of promotions will boost sales between Black Friday and the Christmas holiday, the retailer will only want to run tests during that time frame in order to get accurate results that can help him or her optimize their strategies for the next year.

Good Testing Software Will Help

Although testing time frames will vary depending on each company's unique goals, the right testing software can help marketers identify how their tests should run. Many solutions offer test duration calculators like the two mentioned above, while other solutions may provide expertise from optimization professionals.

Today's Top Picks for Our Readers:
Recommended by Recommended by NetLine

Leave Your Comment

Login to Comment

Become a Member

Not already a part of our community?
Sign up to participate in the discussion. It's free and quick.

Sign Up


Leave a comment
    Load more comments
    New code