I often hear heated
arguments over the
merits of various testing
Many conversion consultants
brush aside multivariate testing
as akin to simply mixing
up the features of a landing
page in a blender and seeing
what combination works best.
They insist that multivariate
testing requires little thought
and is unlikely to produce good results. They stress their
own brilliant breakthrough ideas as being at the heart of any
significant increase in conversion.
In reality the matter is much more nuanced.
Thinking vs. Testing
Not everything should be tested. There are many obvious
elements (or at least commonly understood things in the
conversion improvement community) that should immediately
be changed about a landing page. Most know it is common
sense, but have not bothered to do anything about it.
The most common reasons for ignoring known landing
page problems are that you are too busy, embarrassed about
creating the problem in the first place, or too familiar with
the page to see it clearly. Once you can get past those issues
and are ready to roll up your sleeves, there are several
avenues you can pursue to dramatically improve critical
• Best practices reviews — Often, using well-known best
practices can quickly eliminate the most glaring problems.
SiteTuners offers an inexpensive live interactive Express
Review of your site via an online meeting. The
recording can be shared and circulated to others on your
team. The recommendations can be put into practice
immediately and will often result in a double-digit percentage
improvement in conversion rates.
• Using tools to find problems — Sometimes you might
not even be aware of the problems with your landing page.
They come in many variations including usability, copywriting
and messaging, graphic design, value proposition,
trust and credibility. There is an explosion of new tools
available to help diagnose problems. They include Click-
Tale.com, CrazyEgg.com, CrossBrowserTesting.com, and
our own AttentionWizard.com visual attention prediction
tool. Most of these tools are free or very inexpensive.
• Using people to find problems — Many times, the only
reason you can’t find problems is that you are too close to
them. You may have even had a hand in their creation.
Since you are very familiar with the page, you cannot see
it from the perspective of a new or casual visitor. Conducting
some informal user testing will provide a wealth
of knowledge about what is wrong. Simply sit three to
five people in front of your page and ask them to speak
out loud while trying to accomplish a pre-defined task.
You can even work with a hand-drawn wireframe or
post-it note prototype. Tools like Loop11.com can be
used to recruit online participants and design tests.
UserTesting.com will quickly and inexpensively recruit
people and record their sessions for your review.
Multivariate Testing vs. A-B Split Testing
Battles fought over which is the best landing page testing
approach can be heated and passionate. Some people claim
that the ease of split testing makes it much more likely to be
frequently used and, therefore, more valuable. Others swear
by the advantages of large-scale multivariate testing.
Split tests have several advantages:
• Ease of design — Simply decide how many versions you
want to test, and then split the available traffic evenly
• Ease of implementation — Many content management
and software packages support simple split tests.
• Ease of explanation — No complicated analyses or
charts are needed to present results to others.
• Useful in low-traffic situations — If your landing page
only has a few conversions per day, you simply cannot
use more advanced tuning methods.
Multivariate testing has its own advantages:
• Ability to try large number of unique pages — Some
systems allow you to test millions of page versions if you
have enough traffic.
• Speed of data collection — Running a multivariate test
is much faster than running a comparable number of
split tests back-to-back.
• Taking context into account — Some multivariate tools
take context (variable interactions) into account and
consider how well combinations of your page elements
Take a holistic approach and use a bit of brilliance combined
with occasional blending to get the best results out of
your landing page optimization program. To ignore any of
the available tools or approaches would be like playing golf
with only a single club in your bag — and just as unlikely
to produce a good outcome. So get over the arguments and
the posturing and get to work!
About the Author: Tim Ash is CEO of
optimization firm that
tools to improve
Reviews of a landing
page can quickly
conversion issues. Ash
is a frequent speaker
at Internet marketing
conferences. He is a
to several industry
websites, and is also
the author of the bestselling