Introduction
A ranking styled question asks respondents to order answer choices by way of preference. This allows you to not only understand how respondents feel about each answer option but also helps you understand each one’s relative popularity. If utilized correctly, preference questions can provide a niche understanding for researchers.
Benefits
As the name would suggest, preference questions provide information on respondents’ preferences, pitting one option against the other. Preference questions are very useful for understanding the most important features/issues to a respondent and how they relate to each other. Since preference questions are often referred to as ranking questions, a common mistake is thinking they are the same as rating questions. Preference survey questions differ from rating questions in that participants are forced into providing a preference for one answer option over another. In a rating question, two answer options may be given the same value.
For example:
A preference question would look like: “When shopping online for any product, please rank the following based on their level of importance”
- Free Shipping
- Tailored Offers And Deals
- Speed of Checkout
- Ease of Navigation
A (matrix) rating question would look like: “On a scale of 1-5, please rate the following online shopping features”
- Free Shipping
- Tailored Offers And Deals
- Speed of Checkout
- Ease of Navigation
A respondent may answer “5” for all the features in the rating question – which will tell you how satisfied a shopper is, whereas survey ranking questions may provide insight on what the most appealing feature is. A preference question is necessary if one of your research objectives is knowing what issues to prioritize or what experience is most favourable to the respondent.
It is important to bear in mind, that they are relative to the options within the question. If you are asking what experience is the most important and none of the options is particularly important to the respondent, then assuming the top preference option is very important is incorrect. Hence you should use preference questions in conjunction with rating questions. This way it will allow you to compare what the respondent deems to be important with a rating question and then how important it is relative to all other options with a preference question.
Useful Tips
Here are some useful tips for writing preference questions.
- Limit the answer options in your question
The best practice is to aim for 6 answer options. This is because most people are good at identifying their top 3 preferences and their bottom 3 preferences. The middle preferences tend to be more difficult to identify as a respondent and therefore aren’t as reliable, can you at this moment in time identify your 5th favourite tea brand? No, it takes time and thought.
Designing your survey questions in this manner will make it easier and quicker for the respondent to answer, which will increase completion rates. It also makes the data more reliable, since it is less likely than someone just selecting an answer option to get the question over with quickly.
The maximum amount of answer options you should consider is 10. If you need to use more, you should then group the answer options.
For example, if you have a list of 35 product attributes, try to think of 4 or 5 categories that contain all of those attributes and ask your respondents to rank the answer options within each category and then rank the categories overall.
- Don’t read too much into the lower rankings
As outlined above, respondents will usually have the strongest opinion on their top preferences, after that, their preferences become more flexible. Therefore, in your analysis of preference questions, weigh the top preference breakdown higher than the middle or lowest.
- Don’t force an answer
Sometimes, an answer option that you put in won’t be of relevance or known to your respondent. It’s unfair to commit them to rank this, as it will likely end up at the end. That doesn’t mean it is least important or favourable, just that the user wasn’t knowledgeable enough to rank it correctly. A suitable solution is providing a “not applicable” option to cancel out any answer option that the user doesn’t want to rank. This will help make your results more accurate and put less pressure on your respondents for complicated preference questions.
Conclusion
Preference questions are just one tool that any researcher should utilise to get the most out of their survey. Having a variety of question types helps keep respondents engaged and also allows for a more diverse range of data collected from a survey. Preference questions are excellent when looking to get comparative data on options, particularly if you are looking to find a prioritization or preference on features. However, don’t forget to use preference questions in conjunction with other question types to ensure you get the most accurate research.
If you are unsure or need help scripting your questions, our Account Managers are on hand to help you out and give you a helping hand to ensure that you are asking the right questions to get the best insights - you can contact us at accounts@bounceinsights.com for any queries you may have!