In a recent article, we cover the common question types in market research surveys. One of those question types were preference survey questions, also known as rank order or ranking questions. A ranking question asks respondents to order answer choices by way of preference. This allows you to not only understand how respondents feel about each answer option, but it also helps you understand each one’s relative popularity. If utilized correctly, preference questions can provide a niche understanding for researchers.
As the name would suggest, preference questions provide information on respondent’s preferences, pitting one option against the other. Preference questions are very useful for understanding the most important features/issues to a respondent and how they relate to each other. Since preference questions are often referred to as ranking questions, a common mistake is thinking they are the same as rating questions. Preference survey questions differ from rating questions in that participants are forced into providing a preference for one answer option over another. In a rating question, two answer options may be given the same value.
A preference question would look like: “When shopping online for any product, please rank the following based on their level of importance”
A (matrix) rating question would look like: “On a scale of 1-5, please rate the following online shopping features”
A respondent may answer “5” for all the features in the rating question – which will tell you how satisfied a shopper is, whereas survey ranking questions may provide insight on what the most appealing feature is. A preference question is necessary if one of your research objectives is knowing what issues to prioritize or what experience is most favourable to the respondent.
Bare in mind, that they are relative to the options within the question. If you are asking what experience is the most important and none of the options are particularly important to the respondent, then assuming the top preference option is very important is incorrect. Hence you should use preference questions in conjunction with rating questions.
Here are some useful tips for writing preference questions.
The best practice is to aim for 6 answer options. This is because most people are good at identifying their top 3 preferences and their bottom 3 preferences. The middle preferences tend to be more difficult to identify as a respondent and therefore aren’t as reliable, can you at this moment in time identify your 5th favourite tea brand? No, it takes time and thought. Thus 6 answer options makes it easier and quicker for the respondent to answer, which will increase completion rates. It also makes the data more reliable, since it is less likely than someone just selecting an answer option to get the question over with quickly.
The maximum amount of answer options you should consider is 10. If you need to use more, you should then group the answer options. For example, if you have a list of 35 product attributes, try to think of 4 or 5 categories that contain all of those attributes and ask your respondents to rank the answer options within each category and then rank the categories overall.
As outlined above, respondents will usually have the strongest opinion on their top preferences, after that, their preferences become more flexible. Therefore, in your analysis of preference questions, weigh the top preference breakdown higher than the middle or lowest.
Sometimes, an answer option that you put in won’t be of relevance or known to your respondent. It’s unfair to commit them to rank this, as it will likely end up at the end. That doesn’t mean it is least important or favourable, just that the user wasn’t knowledgeable enough to rank it correctly. A suitable solution is providing a “not applicable” option to cancel out any answer option that the user doesn’t want to rank. This will help make your results more accurate and put less pressure on your respondents for complicated preference questions.
Preference questions are just one tool that any researcher should utilise to get the most out of their survey. Having a variety of question types helps keep respondents engaged and also allows for a more diverse range of data collected from a survey. Preference questions are excellent when looking to get comparative data on options, particularly if you are looking to find a prioritization or preference on features. However, don’t forget to use preference questions in conjunction with other question types to ensure you get the most accurate research.