The concept of testing has burst onto the Internet scene in a
big way. The push for improved ROI and the greater availability
of tools and technology to aid the testing process has fueled
the excitement. But can this “next great thing” be the solution to all of
our problems online?
Testing has actually been around for a long time. Nearly a century
ago, direct-marketers were conducting fairly sophisticated in-market
tests using coded coupons in newspaper advertisements and advertisers
have been leveraging testing methods such as “split-runs” in print
and broadcast advertising for decades.
On the Internet, however, testing can be taken to a whole new level
of depth, sophistication, and speed that just isn't practical or cost-effective
in the offline world. In an online environment, nearly anything can
be tested with enough thoughtful planning, the right methodology and
the right tools.
Sequential Testing
Sequential testing is the most common form utilized on the Web today
— maybe because it matches what we humans tend to do naturally or
as a matter of course. In sequential testing, you simply measure ratios,
trends or performance-bands over time, make changes to your
approach then see whether those measurements go up or down.
For example, you might track your checkout abandonment ratios
on a weekly basis and discover that the performance-band consistently
hovers around 30-40 percent. You might then decide to make some
improvements to your checkout process to see if you can positively
impact performance. After enhancing the usability of your checkout
forms, adding another payment method and including some reassuring
messaging throughout the process, you monitor the ratios and see
that you’ve reduced checkout abandonment to 20-25 percent.
This type of testing is fairly easy and intuitive for most people. You
generally don’t need any additional tools or vendors beyond your current
analytics solutions and maybe a spreadsheet or two. On the
downside, however, sequential testing takes time. It can take weeks to
establish baseline trends and even more time to see conclusive
changes to those trends. And that linear expanse of time introduces
another variable that can call into question the validity of your results.
A lot can change in the marketplace over a short period of time.
With sequential testing, this makes it very difficult to know whether
changes in trend-performance are due to your specific actions or
external marketplace dynamics. In certain fast-moving markets, the
results of a sequential test could be completely unreliable if market
conditions are changing dramatically throughout the testing period.
Split-Testing or A/B Testing
Split-testing — or A/B testing, as it’s often referred to — addresses the
deficiencies in sequential testing by eliminating the time variable.
With split-testing, two approaches are tested at the same time, in
parallel. In theory, both approaches will then be operating under the
exact same market conditions, producing a more-reliable result.
For example, you may want to test whether a new call-to-action
on your shopping-cart page will have a positive influence on the ratio
of visitors proceeding to checkout. With split-testing, you can rotate
your current call-to-action and the new call-to-action on a visitor-byvisitor
basis. In effect, half of your visitors get the current messaging
and the other half get the new messaging. By tracking the behavior of
the visitors to each treatment, you can see which call-to-action produces
the better result. Because the treatments are being rotated
between every other visitor in the same time period, the influence of
external factors is less of a concern.
In addition to reliable results, split-testing also tends to be highly
time effective. Whereas a sequential test might require a month to
complete, a split-test might only require a week.
However, there’s generally a little more back-end work involved
with split-testing. While the concept of testing “A” versus “B”, or even
“A” versus “B” versus “C” on a rotational basis is pretty straightforward
and easy to grasp, actually executing the tests usually requires
some additional tools. These specialized tools handle the rotational
serving of whatever you’re trying to test while also facilitating the
proper tracking of the end-results. Once you’ve engaged a vendor or
implemented a split-testing tool, conducting split-tests is a relatively
simple matter.
Of course, the downside to split-testing — if there really is a downside
— is that only limited variations can be tested with accuracy and
in a timely manner. In most cases, you’re only able to split-test two
or three variations of a single element at one time and produce a reliable
result.
You can, of course, split-test wholly different pages with different
messaging, layouts and advertisements and ultimately understand
which page performs best. But you won’t know which specific elements
within those treatments are actually influencing the result. Was
it the different headline? Or maybe it was the different layout? Or
maybe it was the offer?
Multivariate Testing
Multivariate testing is a relative newcomer to the Internet. Like
split-testing, multivariate testing allows for the testing of different
variations in the same timeframe. But multivariate testing allows you
to test dozens, even hundreds or thousands of different variables all at
the same time. Using sophisticated mathematical algorithms, multivariate
testing can isolate the influence of individual variables, as well
as measure the variables’ influences on each other.
Through multivariate testing, it's possible to determine the bestperforming
combination of multiple variables.
For example, you might want to develop an improved landing
page for a major online advertising initiative. You know that a number
of different page elements could impact performance — the headline,
the imagery, the layout, the call-to-action, the components of the
offer, etc. In fact, you might have two or three variations of each of
these elements to test. Of course, split-testing each variation of each
individual element would take forever. Through multivariate testing,
you could test them all at the same time, and ultimately understand
not only the influence of the individual variations, but also which
combination of all the different variables produces the best results.
As you might have guessed, there's a lot going on behind-thescenes
with multivariate testing. The variations you want to test must
be defined and combined into a smaller number of “test recipes” to
run. After performance data is gathered on the test recipes, the real fun
can begin — applying the statistical algorithms to determine the isolated
and combined influences of the elements and variables.
While multivariate testing can be extremely powerful and efficient,
it can also be a fairly complex practice. Proper experimental design
and execution on a large scale usually requires specialized expertise
and tools. Rest assured, however, that there are a number of vendors
and tools available to help. In fact, the tools have advanced at such a
rapid pace that multivariate testing is now within reach for smaller
sites and adventurous do-it-yourselfers.
Testing is not a Panacea
With all of the hype surrounding testing, it’s easy to see it as the solution
to everything. But, just like anything else in online business, it’s
just another piece of the puzzle. And doing your homework makes
testing easier and far more productive.
You only have a limited amount of time and resources, so you have
to prioritize. You shouldn't just test for the sake of testing. Identify
where testing will produce the greatest impact. Will it be more profitable
to spend time and money testing your landing pages or aspects
of your checkout process? Should you test changes to your homepage
or product pages? Where is the greatest opportunity for profitable
improvement? Even the most sophisticated testing methods will yield
only marginal results when applied to the wrong areas.
Understand your target prospects and competitors. Without a
solid foundation of marketing research and competitive analysis, you
might test 100 different things that your prospects fail to find particularly
important or differentiating. Of course, you’ll be able to determine
the best of the lot — but the whole lot might be lacking.
Often, you will need to look well beyond the test results to understand
the true impact to your business. For example, while a certain
offer might produce a higher conversion rate and more orders, it
might produce less profit once all variable costs are considered.
Another combination of elements might produce more short-term
sales, but customers with lower lifetime values, higher support costs
or higher return rates. You should always try to expand on your test
findings to assess the full impact to your firm’s profitability.
The bottom line is that testing—in all of its various forms—can be
very powerful when applied properly. But testing can’t solve all of our
problems and we can’t abdicate our responsibilities to the testing
process. We still have to do our homework, prioritize, and ensure that
our long-term business objectives are being met.
About the Author:
Rafe VanDenBerg is the founder and president of Business Development
Xcellerator, Inc. The creator and architect of the Xcelleration Process, he
helps companies large and small to optimize and improve their online
sales and marketing-ultimately producing more growth and profits, with
greater leverage and scalability.
