Skip to Main Content

Ratings, Bias & the Performance Factor

Posted on 11.19.2014

:: By Isaac Rothstein, Infinite Conversions ::


Thanks to the Internet, consumers' voices have never been so easy to hear. Go back 20 years and it was almost impossible for consumers to research a company they were thinking of buying from unless they just happened to know someone who had experiences with that business. Today, the Internet is bursting with independent customer reviews for even the most minor of commercial businesses.

What's more, major companies use software tools to help them keep track of the loyalty of their customer relationships. This kind of closed-loop process for feedback allows companies ascertain whom their key clients might be, which particular services or products cause the most problems or are thought to be the best, any issues that need dealing with swiftly and any opportunities for service improvements.

The one problem with such closed-loop practices is that bias becomes an inherent problem. If a company allows bias to creep into the company’s results, those results will be flawed.

Here are some issues that all business owners need to be aware of to prevent bias from skewing their results:

1. Bias from the responder

Do the people who respond to the company’s survey represent a true picture of its clientèle? If people are much more likely to answer the company’s survey when they are pleased with the company’s services than if they are displeased, the company’s results will not reflect how people truly feel about the business. Say that 60 percent of people who responded to the company survey expressed their satisfaction, but only 20 percent said they weren't too thrilled. That's a 40 percent success rate … isn't it? But what if 80 percent of people who were unhappy ignored the survey, and 60 percent who were happy responded. That means that the 20 percent who complained only represented 20 percent of its unhappy customers, and the 60 percent who were happy represented 60 percent. The company is not as appreciated as once thought.

Businesses cannot get everyone to respond to their surveys, but try and make sure that people are just as likely to respond to its surveys following a bad experience as they are following a good experience, and vice versa.

2. Workforce interference

Employees can be made to feel pressured into obtaining good feedback, as often surveys are as much about them as they are about the company. Employees may therefore try and influence survey takers to only write positive things, or will load the system in their favor by failing to submit bad reviews, or even leaving fake surveys that outline positive experiences.

Prevent this by using automated tools as much as they can, and making the manipulation of figures by employees a disciplinary matter.

3. The kindness of strangers

Often people do not want to cause offense, or to create ructions. This leads them to keep quiet about bad experiences, or to heap praise on services when none was deserved. Additionally, small companies may be fearful of the consequences of leaving bad feedback, and often people don't respond when bad experiences are encountered as they feel it will make no difference.

The company can take steps to combat this by making assurances regarding confidentiality. Don't include questions about the nature of the responder that would make identification possible or would lead the responder to think that they could be identified.

Finding the Bias Factor

Bias can be difficult to pin down and eradicate. Usually, the only way of noticing bias is when customer feedback ratings do not match customer behavior. For example, if the company owned a hotel, and the hotel’s TripAdvisor rating was 4.9. yet the rooms were seldom occupied. When behavior doesn't match what the hotel’s scores suggest one should be observing, bias is usually a factor.

Market research surveys have a bad rap (because they’re annoying). It’s common for companies to spam their customer list with phone calls and really long email surveys.

Stop badgering people. They’ll be scared away.

But don’t be overly cautious and quiet, either. People want to share feedback about the company. They’re happy to help, and they feel good knowing that the company values their opinion.

Just be sure to follow the golden rule of Web surveys:

Keep it short.

Don’t worry about being scientifically accurate. Even the most sophisticated survey tools have natural biases in the data. Just make sure the company is focused on the information that matters most for creating the best user experience possible.

Survey Tips

Now, since it’s clear why some surveys aren’t accurate at all. Here are some tips to put into practice. 

1. All Surveys Must Have a Purpose

Don’t just ask questions because the company feels curious. Whenever the company is investing its time, money or other resources, all questions should have a direct influence on the financial metrics of the business. The questions that should be asked include the topics: user experience, usability and buying decision journeys. After this is done, the company should apply this feedback into actionable revenue-generating changes for the company’s site.

2. The questions should be simple

Don’t masquerade the questions in verbose language. Don’t use overly technical language. Be upfront. Never assume that the company’s website visitors are aware of what they  are talking about.

3. Consistency is key

Utilizing a rating scale is a strong way to grasp and gain the attention of the site’s users. Should the company decide to take advantage of a rating scale, the company needs to ensure that they remain consistent. Should the company modify its rating scale, the company’s respondents will end up confused.

4. Logic is King

Commence with an introduction that motivates the company’s website visitors to being the survey. Start with broad questions and then narrow it down. Remember to ask for contact information as well as demographic information at the end of the company’s survey.

5. Offer a ‘thank you’

The company’s users should be rewarded for spending their own time completing the company’s survey and thus completing its survey. Offer them a discount or promotional code. A great way to increase response rates is by way of using incentives. However, bear in mind — incentives also do influence biases in the data. People are motivated by different things. 


This is a guest post by Isaac Rothstein. Isaac is an analyst at Infinite Conversions, a digital focusing on improving website’s real-world financial metrics through conversion rate optimization.

 Request Website Magazine's Free Weekly Newsletters 

Leave Your Comment

Login to Comment

Become a Member

Not already a part of our community?
Sign up to participate in the discussion. It's free and quick.

Sign Up

 

Leave a comment
    Load more comments
    New code

  • 5 Next-Generation Supplier Strategies

    Tipalti