Introduction
Whether it’s just an “additional comments” question or a question delving into a complex issue, open-ended questions are arguably the one of the first tools researchers reach for when building a survey. Open ended questions can answer the why that can multiply the value of your research and insights tenfold. In this article, we cover the basics of this key tool to ensure you are using it correctly and strategically to get the most out of your research.
What is an open ended question?
One of the first lessons for new survey designers is learning the difference between open ended questions and close ended questions. The difference is straightforward; a closed-ended question includes a predefined list of answer options, while an open-ended question asks the respondent to provide an answer in their own words. Another way to look at this is that close ended questions provide quantitative research, while open-ended questions provide quantitative research. There may be some odd cases where a close ended question has an open ended component e.g. the other option in a list of predefined responses, allowing users to write their own response. However overall, you should be able to easily categorize your survey questions as open or close ended questions.
The Good and the Bad of Open-Ended Questions
The Good
- Open-ended questions can provide more holistic and comprehensive data than close-ended questions. The data is more diverse and unexpected as users can put their own mark on their responses and therefore your research can identify more nuanced takes.
- They empower the respondent. Often respondents can feel frustrated and confused by surveys if they have to select responses for close-ended questions that don’t reflect their actual opinion or experience. Open-ended questions avoid this, allowing respondents to properly articulate their thoughts. Many respondents find this freeing and enjoy using their voice to give feedback and air grievances.
- Respondents can provide a lot more data in open-ended questions. This is especially true if there is no limit in the text boxes used in the survey. Some respondents can go on to write long paragraphs and more, providing a huge amount of understanding. However this data is qualitative data, not quantitative.
The Bad
- Open ended questions often take more time and effort for the respondent to complete than a close ended question. Firstly the response is entirely respondent-generated, as opposed to close-ended questions where the responses are generated by the survey designer. Thus the respondents have to put more thought into the answer than normal. Additionally, responses take more time to be inputted since the respondents are most likely typing the response out, compared to pressing a button for a close ended question. More time spent on questions means longer surveys and this will increase drop-off and decrease your response rate.
- Analysing responses to open ended questions can take more time and effort than regular close ended responses. While close-ended responses can easily and quickly be combined and presented in charts and metrics, open-ended questions either need to be viewed individually or put through a text analysis tool, such as a higher chance of misinterpretation.
- Open ended questions can be more subjective than close ended questions. This has two impacts. Firstly, respondents may give a response that doesn’t actually answer the question. This can be avoided by improving the phrasing of your question and through testing. Secondly, the researcher analysing the data could misinterpret the sentiment the respondent was giving. This is because open-ended responses are difficult to compare to one another and it is easy to misidentify the tone and context the respondent was providing.
When should you use them?
A general rule of thumb is to use open ended questions sparingly. As mentioned above, open-ended questions take longer to answer for respondents and therefore if you are trying to keep your survey shorter, which all researchers should, you should aim to have as few open-ended questions as possible. Additionally, the fewer open ended questions you have to analyse, the easier and quicker it will be to analyse the data collected.
There are also two types of open-questions; supplementary and freestanding. Supplementary open-questions are questions that relate to another question, usually a close-ended question. For example, after the close-ended question “please rate our product”, you may follow up with the open-ended question “What was the reasoning behind your rating?”. In deciding whether you want to have a supplementary open-ended question, ask yourself how much nuance you are receiving from your initial question and if you are not satisfied, then go with a follow up open ended question.
Freestanding questions are open-ended questions that don’t directly relate to another question within the survey. A basic guideline is one freestanding open ended question per survey, unless you have a strong justification to add more. Strong justification means that you would not be able to answer the question you want to ask without an open-ended question. Sometimes, you may have very good reasons for open-ended questions; such as wanting respondents to highlight opportunities that you may have otherwise overlooked. Another reason to use open-ended questions is when you are dealing with complex or in depth topics and responses.
Tips
- Analysing responses to open ended questions can be time-consuming if you don’t know what you are looking for. Viewing 1000 responses individually is the best use of your time so consider using a text analysis tool, such as the word cloud feature used on the Bounce Dashboard. Word clouds allow you to get a sense of what the most common sentiments were and then you can alter your approach when viewing the actual response to focus on what are the most common sentiments.
- Before setting any open-ended question, you should brainstorm possible close-ended questions that could replace that question. Even if you don’t replace the open-ended question, you should use the close-ended question to complement your open-ended question.
- Ensure your open ended questions are not double barreled questions. A double barreled question, or questions that are asking more than one question, is a common pitfall when you are trying to reduce the number of open-ended questions, thus bundling them together. However this just causes confusion for respondents and reduces the quality of the answers. You can learn more about these types of questions here.
Conclusions
Open-ended questions are the cornerstone of qualitative research and can be very valuable to brands and businesses. Unfortunately, if they are misused or overused, they can have a detrimental effect on your research. Open-ended questions in small amounts can help you stay connected to your customer, while still keeping the workload on you and your respondent to a minimum.
