Introduction
Survey design is filled with many pitfalls that negatively impact the quality of your research. Response bias is one of these pitfalls. Response bias is a general term for when respondents answer inaccurately or falsely to questions and it covers a wide range of effects and influences. This bias can be intentional or accidental but with the biased responses the survey becomes less useful.It should be noted that response bias is different to selection bias. Selection bias or sampling bias occurs when you only capture responses from a certain segment of your audience which skewed your results. You can avoid selection bias through strategic use of targeting. On the other hand, response bias is caused predominantly by survey design when participants are self-reporting on surveys.
In this article, we will examine just some of the types of response biases that exist. By understanding these biases and utilising better survey design, you can minimise the effects and produce higher quality research for your business.
Acquiescence bias
You can think of the acquiescence bias as the “yes-man” bias. Essentially, the respondent will answer yes and agree with all questions within the survey even if they are contradictory. An example of phenomenon is a respondent answering yes to the question “Are you outgoing and social?” and then later in the same survey, answering yes to the question “Which do you prefer a quiet night in rather than a raging party?”.The interesting aspect of acquiescence bias is that both answers can be truthful and genuine answers in the perspective of the respondent. Many people like to think of themselves as likeable social individuals. However, it is also very likely that the respondent had enjoyed a nice night-in recently, hence why they answered the second question yes too. There are a number of ways to avoid acquiescence bias. You should firstly avoid leading questions, as well as varying the question types and using Likert scale questions.
Courtesy Bias
Courtesy bias is a type of response bias that occurs when respondents tend to not fully state their unhappiness with a service or product as an attempt to be polite or courteous toward the researcher. It is a common bias in qualitative research methodology as well as in face-to-face interviews. The best ways to avoid it are by actively asking for negative experiences or issues in the survey.
Demand Bias
Demand bias comes from the respondents being influenced simply by being part of the research, particularly when they are involved in longitudinal surveys or research communities. This most often occurs when the respondents try to understand the purpose of the surveys and attempt to give the right answer. One example is if a respondent is involved in a longitudinal study regarding public transport, they may between the intervening period between surveys change their habits to better support the purpose of the survey. Respondents may do this consciously or subconsciously but regardless it weakens the quality of the research. To prevent this, you should as a researcher keep respondents in the dark about the specific purpose behind your research in order to get the most accurate results from your respondents.
Extreme responding
Unsurprisingly, this bias occurs when respondents provide extreme answers, whether it be positive or negative. This happens predominantly on likert scale questions, in which people will go for either the highest or lowest options. Cultural behaviour and norms is one cause of this bias, that is some cultures avoid middle-ground answers, while others favour them. Another cause is indifference on the part of the respondent. However, questions that spark a strong emotional response can lead to respondents going for extreme answers. Thus, a good tactic to follow is to be careful when wording sensitive questions.
Question order bias
This is a bias that occurs when the initial questions of your survey influences the answers your respondents give to the subsequent questions later on in the survey. This can also be known as “priming”. That happens due to a desire from respondents to give internally consistent answers. Take these two questions:
- How happy are you in your life overall?
- How happy are you with your marriage?
The ordering of these questions are very important. Studies have shown that by asking specific questions first, respondents are more likely to answer the same way for a more general question afterwards. E.g. it is very possible that a respondent could be happy in their marriage but not be happy with their life overall. However, if the marriage question is asked first, they are more likely to answer the second question inaccurately, for the sake of appearing consistent.Notably, this bias can be caused not only by the order of questions in a survey but by the order of answer options too. To minimise this bais, a researcher should logically group your questions around a common topic, randomise the order of their answer options and attempt to make your questions as engaging as possible, while still being clear.
Social desirability bias
This bias is linked with acquiescence bias and occurs because respondents want to be perceived in their best light. Therefore, respondents may exaggerate their habits, beliefs, and personal preferences so they are more socially attractive, even when surveys are anonymous. The downside of this is that habits, lifestyle choices and opinions that are viewed negatively by society may be misrepresented. The best way to avoid respondents doing this is to be as neutral when wording questions. If you are running an anonymous survey it is also important to highlight
Non-response bias
Non-response bias occurs when people are unable and unwilling to respond to a survey. This can be due to a factor making them differ greatly from people who respond. This can also be called participation bias. A reason for this could be due to a poorly constructed survey or the targeting carried out by the questionnaire. An example of this would be to ask non-alcohol drinkers what their favourite brand of alcohol is. The opinions of the participants will be disproportionate to the population and will result in a biased data set.
Checklist
- Short, clear and neutral questions
- Vary question types
- Avoid leading answer options
- Remove brand identifiers
- Short and relevant time period
Conclusion
There are many forms of bias that occur in surveys and market research that can skew the quality and accuracy of your research. Your initial survey design may fall into the pitfalls of a number of the biases we have listed above. In fact, these biases often intersect with each other and it may not always be possible to rid all of your research of these pitfalls. Regardless, understanding these biases and reviewing your survey design is just the first step in limiting the effect of response bias on your research. Following the steps outlined and the survey design recommendations will aid in eliminating biases which can occur.