Skip to main content
Article | Sustainably Engaged

Survey best practice: The allure of the comment

Talent
N/A

By Adam Zuckerman | July 17, 2019

Surveys produce numbers: quantitative data. This kind of information enables ranking, sorting, benchmarking, tracking and the kind of predictive analytics that have exploded in popularity over the last several years. But surveys can also produce qualitative data: free-form comments from individual employees that narratively describe a challenge or proposed solution.

The qualitative versus quantitative quandary

While quantitative data is generally thought of as the main driver of business decisions, a single free-form comment – especially if well-crafted and provocative – can capture the attention and motivate change in a way that numbers alone often fail to do. For this reason, many leaders rely heavily on the open-ended comments generated from an employee survey. Some, in fact, read the comments before looking at the quantitative data, and base their decisions and actions primarily on this unstructured information.

At Willis Towers Watson, we believe this qualitative information is extremely valuable, but must be viewed in the right way and within the proper context. We recommend using open-ended questions to supplement, rather than replace (or even rival) the quantitative data in forming overall conclusions about the employee experience. This is for several reasons:

  1. In general, a smaller percentage of employees will respond to an open-ended text question versus a multiple-choice question. In fact, our data shows that only about half of employees who return a survey make the additional effort to provide a comment. The other half of employees simply leave that section blank. In practice, this means you have a far less representative population contributing to the qualitative data.
  2. When employees do make the effort to respond to an open-ended question, they rarely do so in order to extol the company’s virtues. Instead, the overwhelming majority of open-ended comments produced from any employee survey, in any company no matter how great, tend to be negative - essentially complaints or “areas of opportunity.” This bias does not exist with multiple-choice questions.
  3. Although not impossible (see below), it is difficult to track and benchmark qualitative data in a way that is valid and reliable over time, especially across multiple languages. This means that vital context for interpreting comments and deciding on their implications is much rarer.

For these reasons, we recommend using open-ended comments as “seasoning” when interpreting employee survey results. In other words, the quantitative data, which is more representative, less biased, and benchmark-able, is used to first identify priority areas, and only then is the qualitative data used to help flesh out and provide examples of (and potential ways to address) identified priorities.

Reviewing open-ended comments need not be an entirely manual process any more. Advances in machine learning and natural language processing have enabled the creation of terrific new tools that automatically identify the underlying themes and sentiment expressed through employees comments, and allow for advanced analytics that link both qualitative and quantitative data and even benchmarking.

Sign in to your Willis Towers Watson Employee Engagement Software account today or contact us for a demo of our natural language processing capabilities.

Author

Product Leader
Willis Towers Watson Employee Engagement Software

Adam is responsible for the overall development and direction of Willis Towers Watson Employee Engagement Software. His goal is to create the world’s greatest software for delivering insight and enabling actions that enhance employee experience, company culture, and business performance. Outside of work, Adam enjoys off-roading in his Jeep and spending time with his family. Follow Adam on Twitter.


Contact Us
Related content tags, list of links Article Sustainably Engaged Talent