Should You A/B Test That? A Mental Checklist...
Testing a website "experience" (or any of the elements on any of its landing pages, product pages, or information pages) in order to increase conversion and engagement is a smart decision, but it may not be the "right" decision in every instance.
While the benefits are many, and the potential insights that can be gained from any test are incredibly valuable (whether they are failures or successes), just because you can test does not always mean that you should. Often in the heat of the moment we tend to say "yes" to tests because, of course, we are and want to be a responsible steward of our enterprise and the user experience. But imagine the savings in resources (time, money, effort) if the answer to the "Should I A/B test that?" question were "no" more often. It might enable you to concentrate on only those test that are significant enough to really mean something on a fundamental level to the business or enterprise itself.
Here's a quick 3-point mental checklist to determine the worthiness of a test of any digital element or experience when you need to make a decision about the worthiness of a test:
1) Plenty of Traffic (and Data) Exists For This Test
It is often necessary to have thousands (if not tens of thousands) of visits to be "confident" that one version of a webpage experience or specific digital element is better than another. Using a sample size calculator to determine what that number is for the specific test will show the audience size required based on the current conversion rate, the expected lift, and the degree of confidence required.
2) Many Similar Tests Can Also be Conducted
The current minimalist design trend sometimes limits many enterprises options for testing the presence, removal or modification of elements on their sites or with the digital experience they produce and provide their audience. If you don't have many things to test on a site, however, it may not be worth testing at all. Unless you can come up with a long and detailed list of ideas and be able to group them into something more meaningful it is difficult to know with certainty what was responsible for improvement.
3) Clear Path to Improvements in Key Metrics
Before you can run any kind of test, you have to understand what matters most to the enterprise. Depending on the specific objective (e.g. increasing average order value, or acquiring email newsletter subscribers) different elements will be more critical to the digital experience for each. Check out these S.M.A.R.T. metrics if you're not sure what to measure.
While there are a wealth of articles and advice online telling you to test every page and experience developed and to A/B test every variation, you don’t have to. For many startups and growing online businesses there just isn’t enough traffic early on to create an accurate sampling with measurable results, and initially tend to test things that in the greater scheme of things don't really matter much. Instead, focus on growing your business. As traffic levels increase, commit to learning more about your customers and you'll start discovering testing variations to achieve significant improvement.
Do you use A/B testing on your site or landing pages right now? Have you found issues with the quality of your results?
Share your thoughts with me in the comments below!