Measuring the Response Quality of Online Open-Ended Questions in Linguistics Complexity

Open-ended questions (OEQs) are important survey tools for social scientists, but their response quality is often disputed due to the additional load they impose on the respondent. To find ideally worded questions and survey strategies that encourage high-quality responses to OEQs, the quality of textual responses is often assessed.

Response length and response latency are often used as measures of response quality, but they do not provide enough information on interpretability and richness of the responses. In this study, we propose a novel way of evaluating the data quality of open-ended responses by leveraging approaches from Natural Language Processing (NLP), measuring different linguistic complexity features of responses. Using various automatically generated linguistic features, we compared the quality of responses to distinctly worded sets of questions related to people’s uncertainty about their intention of having children.