Survey Techniques for Authentic Quantitative Customer Research

During the last week of every month, I log into my Blue Shield of California member portal to pay my health insurance bill.

Every time I log out of the portal, without fail, a small separate window with a survey pops-up on my screen. 

I have to confess that I often ignore the survey, even though it only has six questions. When I read the survey the first time I wasn’t sure if it was asking me about my experience as a customer or if I needed assistance finding information—it’s vague. Not to mention the 0 to 10 scale of how satisfied I was with my experience, I’d rather just answer a definitive ‘yes’ or ‘no’. 

Michael Murakami, research manager for Airbnb, puts that kind of vagueness on his list of nine survey mistakes to avoid when you’re trying to gather quantitative customer research. A little background on Murakami: he leads a survey science team that identifies and spreads best practices across the multinational corporation.

Michael teamed up with Qualtrics to share his insider knowledge on how to create effective surveys by listing nine errors companies can steer clear of. 

1. A respondent should have to spend no more than three to five minutes on a survey. Michael says,  “any longer and the survey feels like a chore.”

Oscar Health, a New York-based health insurance company, uses a survey to onboard its customers. The survey is very specific and direct to help their customers choose the best health plan for them at a whopping six questions. 

2. Avoid agree-to-disagree questions. Michael says these questions are inherently biased because they don’t validate the statement itself, but how much the customer agrees with it. 

In this question, Teladoc already assumes their consultation experience is less time consuming for the customer rather than asking if it really does take less time:

3. Don’t use double-barreled questions. The biggest issue with these is that you can’t know which question the respondent is answering. Michael suggests never beginning a question with the phrase, “Based on X, what do you think about Y?”.

In a Fitbit user questionnaire, the company wanted to understand their customer’s product experience. But with the question below, the company won’t know if the customer uses the product to connect with friends, participate in group challenges, or meet personal goals. 

4 & 5. Don’t use 11 point scales, and label each mark (1,2,3…). Five to seven-point scales are typically more evenly spaced, easier to use, and better for analytical research. 

Referring back to my Blue Shield example, the 10 point scale doesn’t let me accurately show how satisfied I am with my experience. Plus, it’s not pleasing to the eye. 

6. Using brand metrics to evaluate product changes or experiments isn’t effective, Michael says, because brand building takes place over years, not weeks or months. 

I’m revisiting the Fitbit survey here because this question is too open-ended and doesn’t give enough information for the customer to refer to when responding. It just asks us to rate the brand as a whole. Instead of asking whether the app, in general, is intuitive ask about specific features of the product. 

7. Michael advises against elevating net promoter scores above other survey metrics because the question is often poorly designed, double-barreled, or unclear. 

Wellable, an employee wellness program, hosted a challenge for biotechnology and life science companies and conducted a survey on participants. I would reword this question to get a better understanding of what specifically worked for the customer rather than ask how “likely” they would use it. Specific information is more helpful for your team to analyze. 

8. When crafting the questions within a survey don’t generalize, be specific, be concrete, and only ask questions about the present or recent past. Avoid questions that put business concepts or ideas in front of customers who don’t have experience in the subject. 

Meru Health uses a confidential test for potential clients to determine whether or not they’re suffering from depression or anxiety. The questions and answers are too general. I’m left wondering how they can accurately understand their patient’s needs.

9. Unless you’re going to create a specific survey for multiple locations across the country, taking into consideration each unique marketplace, local economy, cultural, native language, etc. Michael strongly suggests not making surveys global. 

TotalWellness, a content creator, has a survey on its site that opens up to anyone, with an incentive, that doesn’t help the company understand its target audience. 

Michael’s list of nine don’ts just scratches the surface of developing engaging surveys that garner the most accurate and useful information for your team. In order to help our clients, EMPATH often conducts customer research through short, concise surveys that get solid response rates. If you’re looking for the best way to create approachable surveys and tips on getting them in front of your customers, let’s talk.