Response bias occurs when a respondent answers a question in a way they think you want them to. It can also be caused by question or response wording, layout or order. While there is no way to completely eliminate response bias, the survey creation tips below will at least help you minimize bias and gather more accurate and complete data.
Tip #1: Clear Language
When designing a survey, make sure you understand your audience and the language that they use and comprehend. If you are a tech company and you want to know how useful your services are, be aware of the terms and language you are using and avoid overly-technical jargon that may be above your audience’s understanding. Pre-testing your surveys on a sample audience is also important to double check that your respondents understand what you’re trying to convey/ask.
Tip #2: Check for Inherent Bias
Framing a question properly is key to collecting accurate feedback. Behavioral researcher Dan Ariely gave a presentation on the effect of question design on questionnaire results. In his example, researchers wanted to know why organ donation rates differed greatly between countries. They found the primary difference lay in the format of the organ donation question. The countries with the highest organ donation rates asked people to ‘opt-out’ of organ donation, while asking people to ‘opt-in’ to donating their organs resulted in lower volunteer rates. This simple difference resulted in donation rates of 4-20% for the ‘opt-ins’ and close to 100% for the ‘opt-outs’. Be aware of this type of bias when you are wording your questions.
Tip #3: Offer Enough Options
Another key point is to include an option such as ‘other’, ‘I don’t know’, or ‘does not apply’ to make sure you aren’t getting a ‘false positive’ because respondents did not have the option to choose ‘not applicable’. Along these same lines, you also want to keep open-ended questions to a minimum in order to prevent survey fatigue and optimize the quality of the responses you receive.
Tip #4: Avoid Question Order Bias
It’s a fact that the order of your questions will affect your results. This is especially true for longer surveys where respondent fatigue comes into play. We’ve all been there – at the end of a long questionnaire you just start picking any old answer to get to the end. One of the easiest ways to avoid response bias due to question order is to randomize your questions as well as your question options. This will help minimize the “A-B-A-B” effect of respondents randomly clicking options.
Another rule of thumb related to bias based on question order is to offer up open-ended questions before closed questions on the same subject. Researcher Sam Mcfarland found that when you start with a closed question, you may affect how the respondent will answer a subsequent open-ended question on the same topic because the earlier question has primed them to focus on that issue.
We hope these tips have helped you understand how to design better surveys. Now go build your own with a Checkbox free trial or contact us for more information!