Thursday, February 26, 2009

Survey SAYS...!

You supply the buzzer sound. I'm out of onomatapoeic juice.

I get surveys sometimes. I'm sure you do too. I hate them. I don't mind answering questions. It can be kinda fun. I do mind answering leading questions or being overly constricted in how I respond.

Now, I construct surveys as part of my job, so I know from whence I speak. But my goal in constructing a question and a list or range of possible responses is to gather accurate information. Some surveys, on the other hand, seem designed to get you say you liked something.

Let's play.

Here's a question in an "honest" survey, using a standard Likert scale.

Please rate your experience with our Customer Service group, with 1 being least satisfied and 5 being most satisfied.

1 2 3 4 5

This is straightforward and allows the respondent to be as objective as possible. Most respondents will select 3 or 4, while some malcontents (or those who are actually unhappy with the service or, just as likely, the product) will select something lower. The rare Sally Hawkins will choose 5. All-in-all, and assuming a meaningful cross-section of customers, you should get a reasonably accurate assessment of your performance as perceived by your customers.

This brings me to a survey that I receive regularly from a company that services me. (Service me, baby.) I never complete it. I'll tell you why. Here's their scale.

Check out the difference in this scale. This is designed to get you to say you were satisifed, and very satisified, at that. First off, there is no middle answer. In a normal Likert scale, there is an odd number of choices, so a respondent can remain entirely neutral by selecting the middle button. Eh.

Now, the question leading into this scale has the following "explanatory" text. "In the scale below, 1 is Very Dissatisfied, 5 is Neutral, 10 is Very Satisfied and N/A is Not Applicable." But graphically, there is no neutral. Note the line creating a strong delineation between satisfied and dissatisfied. There is no true neutral here. And if 5 were truly neutral, then the scale should be 1-9 or 0-10. This is constructed so that you have to fall on one side or the either.

Wait, there's more. In addition to the line separating satisfied from dissatisfied, they've added an N/A button. So this actually does create an odd number of choices. But note the placement. It's on the far right, making 6 the choice in the exact center of the scale. Respondents with a neutral experience are even more likely now to select 6. The instruction is telling you that 5 is neutral but your eyes are telling you that 6 is. You also don't want to select 5 because it's clearly not neutral, what with that vertical line telling you that it really means you were dissatisfied and the fact that it's actually physically left of center.

One more thing: In a normal Likert scale, the numbers' meaning may be spelled out even more clearly for you. 1=Very Dissatisfied, 2=Somewhat Dissatisfied, 3=Neutral, 4=Somewhat Satisfied, 5=Very Satisfied. The only text in this scale is "Very Dissatisfied" and "Very Satisfied". You're not going to say you were very dissatisfied, are you? People will tend to err on the side of okayness rather than crumminess, unless the rep was a real prick.

The end result is way more positive response than if the survey were constructed in a purely objective fashion, which was easily achievable, as demonstrated earlier. And the company can now claim that, say, 95% of respondents were satisfied with their performance. It looks good, but it probably isn't true.

It's possible that all this is done to weight the survey against cranks. I, for one, don't usually answer these unless I have a complaint. This may be a way to offset that. But if you have to do that, then you have worthless data from the get-go.

As if this weren't bad enough, here's another question in the survey.

Did the Customer Support representative provide exceptional service to you in resolving your service request?

Yes
No

So if I answer "Yes", I'm saying they were "exceptional". If I answer "No", I'm saying what? They were crummy? This reminds me of the bit that Stephen Colbert does where he has a U.S. congressperson in for an interview and asks them "George W. Bush: Great President or The Greatest President?"

The congressperson sputters. The audience snickers. Colbert says "I'll just put you down for 'greatest'."

1 comment:

Mrs. Chili said...

What if my problem was solved, but I was dissatisfied with the service overall (hello, Comcast?). I generally dislike surveys, and for many of the reasons you list here.