A Crash Course on A-B Split Testing

The most basic technique available for testing and validating landing pages is called A-B split testing. The name comes from the fact that two versions of your landing page ("A" and "B") are being tested. Split testing refers to the random assignment of new visitors to either version A or version B.

In other words, the traffic is split (unbeknownst to each individual user) and the different versions are shown in parallel throughout the data collection period (usually in equal proportions).

Parallel timing and random visitor assignment are important requirements in A-B split testing. In contrast to sequential tests, in which one variation is run for a period of time and then replaced with the second variation, testing your landing page variations in parallel allows you to control for as many outside factors as possible.

This includes seasonality, competitive activity, fluctuations in traffic volume and countless other circumstances that could affect your conversion rate and pollute your data. Likewise, the random assignment of new visitors to a particular landing page design is also critical, since randomness is the basis for the probability theory that underlies the statistical analysis of the results.

Split Testing Fundamentals
In split testing terminology, version A is usually defined as your original control page, or baseline (commonly called the champion version). The other version is the alternative (commonly called the challenger). If the challenger proves to be better than the champion, the challenger replaces the champion after the test and becomes the new champion to beat in any subsequent tests.

It is also possible to have more than two versions in a split test. For example, if you had one original and two alternative versions, you would have an A-B-C split test, and so on. In practice, split tests rarely have more than 10 versions of the page. The variable in your split test can be very granular (e.g. a single change such as headline text), or it can be a wholepage redesign of your landing page that is radically different from the current page.

A-B Split Testing Advantages

• Ease of test design

Unlike more complicated multivariate tests, split tests do not have to be carefully designed or balanced. You simply decide how many versions you want to test, and then split the available traffic evenly among them. No follow-up tests are required to verify the results - the best performer in the test is declared the winner once enough data is collected.

• Ease of implementation

Many software packages are available to support simple split tests. If you are testing granular test elements, you can design, set up your test and be collecting data literally within a matter of minutes. This can be done in most cases without support from your IT department or others within the company. You may even be able to collect the data you need with your existing Web analytics tools, without the use of additional landing page testing tools.

• Ease of analysis

Only very simple statistical tests are needed to determine the winner. Basically, all you have to do is compare the baseline version to each challenger to see if you have reached your desired statistical confidence level.

• Ease of explanation

No complicated analyses or charts are needed to present your results to others. You can simply declare that you are very confident that a particular version is better than another. You can also give a likely range of percentage improvement (based on the amount of data you have collected and the width of the error bars).

• Flexibility in defining the variable values

In whole-page split tests, you have complete flexibility in how different the proposed alternatives are. For example, in one alternative, you may simply choose to test a different headline. In another you may completely restructure everything about the page (layout, color scheme, sales copy, offer and call-to-action). This ability to mix and match allows you to test a range of evolutionary and revolutionary alternatives in one test, without being constrained by the more granular definition of variables in a multivariate test.

• Useful in low data rate tests

If your landing page only has a few conversions per day, you simply cannot use more advanced tuning methods. But with the proper selection of the test variable and alternative values, you can still achieve significant results in a split test. Improvements in the double or even triple digits are not uncommon.

A-B Split Testing Disadvantages

• Limited number of versions

By necessity, the typical split test includes only a small number of variations. But if you did your homework properly, you probably came up with dozens of potential issues with your landing page, and likewise constructed many alternative variations to test.

However, with the limited scope of split testing, you will be reduced to testing your ideas one at a time. You will also be forced to guess which ideas to test first (based on your intuition about which ones might make the most difference). In other tuning methods, you may be able to test many of your key ideas at once and find all of the changes that improve your conversion rate in one test.

• Does not consider context or variable interactions

Split tests consider only one variable at a time, making it impossible to detect variable interactions (how combinations of variables influence each other). A series of split tests covering several variables is not the same as a multivariate test with the same variables.

Depending on the variable interactions, you may not be able to find the best-performing combination of variables on the page at all. Whether you do depends on the order in which you conduct your split tests, and the exact nature of the interactions.

• No way to discover the importance of page elements

Often, you may choose very coarse variables for your split test. Because of the limited data rate, you are forced to make your best guess at page elements that might improve performance. These elements may actually involve many simultaneous changes to your landing page. In the extreme case of a whole-page redesign, you may have changed dozens of details on the page in question and defined them as a single alternative version.

However, the same flexibility that allows you to do this also limits your ability to interpret the results and attribute credit for the conversion improvement to any particular change that you made. Was it the button color? Or was it the headline change? Or was it the different offer? You will never know. By squashing multiple changes into one page, you have confounded their effects and lost the ability to look at them separately.

• Inefficient data collection

Multivariate tests are often carefully constructed in order to get the most information from a smaller data sample. In effect, they allow you to more efficiently conduct multiple split tests simultaneously, and even to detect certain kinds of variable interactions. Conducting multiple split tests back-toback is the least efficient kind of data collection - none of the information from a previous test can be reused to draw conclusions about the other variables that you may want to
test in the future.

About the Author: Tim Ash is CEO of SiteTuners.com, a landing page optimization firm that offers conversion consulting, full-service guaranteed-improvement tests, and tools to improve conversion. SiteTuners' interactive Express Reviews of a landing page can quickly identify major conversion issues. Ash is a frequent speaker at Internet marketing conferences. He is a contributing columnist to several industry publications and websites, and is also the author of the bestselling book "Landing Page Optimization."