Guidance and tools for analyzing freeform survey responses #10

In some surveys we ask developers open ended questions and allow freeform text responses. This can be very useful when we're looking for what's top of mind on a particular topic, and can give us output like this, from the MDN Browser Compatibility Report 2020:

The has a section on Categorizing Responses which explains how it was done.

State of CSS also has freeform questions, and we might want to ask freeform questions in MDN short surveys.

A basic method for analyzing results like these is:

• Put all raw responses into a spreadsheet, with columns for categories, and a "done" checkbox
• Sort the responses alphabetically and look for groups of similar responses
• Add categories for the large groups, checking "done" for responses that have been fully categorized
• Continue to add smaller and smaller categories until they would be too small (<1% of responses, for example)
• Keep going until all "done" are checked, using "misc", "spam" and similar categories to handle the long tail

Based on that, it's then possible to treat the responses similar to the results of a multiple choice question.

This method could be improved by tooling to help get the broad strokes right in at least two ways:

• Generating a word cloud to quickly identify the most common words
• Quickly assigning categories based on string matching

Both the guidance and the tools is something this group could provide to make this more effective.