Gated Split Testing: What Pages Need Before Testing

Marty Greif
by Marty Greif 02 Jun, 2016

There's a fraction of the conversion community that actively asks marketers to test everything.



You should adhere to that advice if you meet the following criteria:


+ You don't have the technical savvy to find a core set of pages to test.


+ You have a shortage of projects on the backburner, and too much time on your hands.


+ You don't care whether the projects you undertake actually have an impact on the bottom line.


In reality, the belief that marketers should test everything is probably a deterrent to testing in general. People who might have started get intimidated by the daunting thought of testing all changes. What is really needed is a rational entry point for split testing. If marketers want split testing to cover areas that matter, they can start with this process:




It's time to get familiar with the elements that should be checked before running a test.



Criticality: Improving this page will move the needle


Each split test usually takes weeks to run. Optimizers don't want to skimp on statistical significance; 90-95 percent is the threshold conversion professionals use, and that means weeks running a test on a page with moderate traffic. That said, the first thing a company will want to know is how critical the page is for the website. If improving a page will not result in significant gains, brands are better off testing other pages. Criticality can be thought of as a function of two things: the amount of traffic a page receives, and how high the page's bounce rate (especially if it is a page designed for navigation). If optimizers find just the high-traffic pages, those pages may be perfectly fine from an engagement standpoint.


If optimizers find just the pages with high bounce rate, the page may not be a common pathway for visitors. If they find the pages that have both a significant amount of traffic, and a high bounce rate, (visitors don't stick around), however, then that means marketers have found their trouble areas. Those are the candidates for pages that need to be fixed.


If you use Google Analytics, there's a hidden gem called "Weighted Sort," which makes finding those pages 10 times easier.


Baseline Functionality: All the functionality on the page works


It's not time to fire up Google Content Experiments, VWO or Optimizely and run a test just yet. Before then, optimizers need to learn what devices and browsers get used to access candidate pages to test. Once that data is in hand, optimizers can check if those pages function properly. Remember, this isn't how convincing a page is yet, or whether the call-to-action (CTA) is the right size or color. This is "can the users technically get to the next step?" A significant amount of the time, what brands will find is that they need a fix at this stage. For instance, maybe the form's submit field doesn't fire on mobile, maybe the JavaScript being used doesn't play well with IE8 and a lot of the site's visitors still use the browser, or maybe they'll find any of a dozen things that are actually broken.


Enterprises need to fix those BEFORE they think about running split tests.


Visual Load: Everything a user needs on the page can be seen


At this stage, optimizers will want to check the visual load. That is, can visitors actually read the page, and can they "see"¬ù what items they can interact with? Any marketer worth his or her salt will ensure that, at minimum, the CTA is present and looks clickable, and that most of the audience can read the content without issue.


Cognitive Load: The page doesn't overwhelm visitors


Next, optimizers will want to make sure they don't set off what Daniel Kahneman calls "system 2." Let's back up a bit.


Let's assume that most of a day, a marketer runs through a set of automatic actions. He eats breakfast. He takes a bath. He brushes his teeth. He gets to work on time. All of those actions he can pretty much do without any mental effort are "system 1." Then, at work, let's assume somebody asks him details about a report he made four months ago. He's likely to remember the details, but only after significant thought. That's "system 2" and it means conversion death if triggered.


Optimizers want to make sure their pages avoid these before testing begins:


+ Significantly deviating from the convention, as established by other sites


+ Presenting too many options without clear differentiation


+ Failing to reduce the number of elements on a page




If the page fails any of those four items, it's time to hit the reset button. However, if the pages passed all four, it's time to figure out exactly what should be tested.



Motor Load: It's easy for visitors to take the next step


B.J. Fogg, of Persuasive Technology fame, likes to say that you should put "hot triggers in front of motivated people."


Making a trigger "hot" is usually a function of how easy something is to interact with. What marketers can test, then, comes down to a few items for ease of use:


+ Different sizes at which buttons/mechanisms can be manipulated properly by a site's visitors


+ Grouping of tasks to minimize switching costs between actions


+ Types of interaction people can accomplish on the site Broadly, these affect motor load; how much of an "interaction tax" visitors need to take to do what they need to do. Marketers should be testing pages to minimize that "tax."


Motivation: Visitors are persuaded to convert


Robert Cialdini, author of "Influence: The Psychology of Persuasion," discusses six things that affect how persuasive you are. The CliffsNotes version, modified for Web marketers, goes like this:


Liking - Is the content written in the user's language?


Scarcity - Are you using persuasion elements like timed offers to push visitors to act?


Commitment - Are you getting visitors to commit smaller actions before asking for bigger ones?


Authority - Are you borrowing "trust" from logos of known clients and security symbols, and are they above the fold?


Reciprocity - Are you offering PDF guides for top-of-the funnel visitors, so they'll "return the favor"?


Social Proof - Are you displaying reviews, testimonials, or number of clients served?


A significant amount of a marketer's test plans "live" here - persuading people to act. Motivation and interaction mechanisms are perfect items to test against, you just need to make sure your pages are worth testing first.