Written by: Sheila B. Robinson
Surveys are easy to design, right? Not so. Effective survey design is a lot more than just brainstorming a list of questions.
A well-designed employee survey will yield data-informed insights that are both meaningful and actionable.
Whether it’s employee morale, engagement, or reactions to new policies you need to understand, the way to unlock high quality insights is to craft an effective employee survey.
Why attention to survey design is so important
In 1941, Donald Rugg conducted an experiment on whether the wording of survey questions makes a difference in how people answer. He asked two versions of a question:
A. Do you think the United States should allow speech against democracy?
B. Do you think the United States should forbid speeches against democracy?
If someone answered “yes” to question A, they would answer “no” to B, believing that the US should “not allow” those speeches. But that’s not what Rugg found. There was a substantial difference in responses, even though the groups surveyed were randomly selected and representative of the American public. More people said “no” to version B than said “yes” to version A. Why? Simply put, words matter, and people have feelings about them. “Forbid” can stir up negative emotions, while “allow” feels more positive. The point? How we ask questions—the words and phrases we choose—influences how people respond.
How to craft the right questions for your employee survey
#1 – Start with a broad question and a purpose
Broad questions might be:
- “How do employees feel about the new policy changes?”
- “What are employees’ perceptions of our company’s strengths?”
- “How is the new training program working?”
The purpose of your employee survey might be a combination of answering these broad questions and informing decisions that rest on that data. A survey purpose statement might be: “The purpose of this survey is to understand how employees feel about the new policy changes and inform decisions about how we might refine the policies.”
#2 – Know your respondents
Consider who will be invited to complete your survey:
- What do you know about them?
- What are their working conditions?
- What is the organizational culture like?
- Are people happy to be working here?
- What is going on that may influence how people respond to the survey (e.g., contract negotiations, union activity, sales challenges, market fluctuations)?
- Will they be able to access the survey and have time to complete it?
- Do they want to share their perceptions?
There are dozens of reflective questions to consider before designing your survey. The more you know about respondents and how they might react, the better your response rate may be and the less prone respondents will be to survey fatigue.
#3 – Consider how long to make your survey.
This is not about how many questions to include, but rather how much time it will take respondents to complete the survey. Five minutes? Ten? Twenty? If information needs are substantial requiring a longer survey, consider administering it during part of the workday, such as a staff meeting, where employees could be asked to spend time completing it.
#4 – Craft survey items aligned with broad question(s) and purpose
Here’s where a question brainstorming session is appropriate. Don’t worry about specific question types—open or closed ended, rating scale, check-all-that-apply, etc.—or what the response options or scale will be. Just brainstorm. Then, pare down the list by checking each question against the survey purpose. Eliminate any item that doesn’t align with the survey purpose. Next, prioritize. Given the amount of time you want respondents to spend completing the survey, if there isn’t space for all questions, which are most important?
#5 – Refine the questions
For each item, consider what you are actually trying to measure, and then determine the question type. Will it be open-ended? Will it feature a rating scale? Multiple choice options? There are implications for each. Open-ended questions are more time-consuming to analyze and require skill in qualitative data analysis. Closed-ended questions are easy to analyze with basic statistics. As you figure out how to ask the question and what question type to use, be aware that this is where most mistakes are made and where questions become problematic for respondents. Some points to keep in mind:
Respondents must understand each question.
Respondents must know what you mean by each word and phrase. Consider this example: “How many times in the last month did you read a magazine?” This can be tricky for respondents. What counts as “read a magazine?” One article? Two? Cover to cover? If respondents have difficulty interpreting a question, they may not put forth the effort to reread it or think more deeply to come up with a response. They may get frustrated and quit the survey before completing it. Survey questions should be clear, concise, and easy to answer.
Ask one thing at a time.
Ensure that you’re asking one question at a time. Here’s an item from a survey from an HR department to the company’s customers:
Was the information you received, timely, relevant, and appropriate? YES / NO
This item is asking three different questions! It’s possible that the information received was timely but not relevant, or relevant but not timely.
Another example from a hotel asks customers to rate whether staff members – front desk, bell service, housekeeping – were friendly and efficient. Again, it’s possible for staff to be friendly but not efficient, or efficient but not friendly. These response options are not mutually exclusive.
Why does this happen? Of course, in order to get a good response rate, surveys must be relatively short. Survey writers take shortcuts trying to squeeze too much into one item in order to create a survey with fewer items. But when questions are confusing and survey respondents must read them several times just to understand, it actually takes longer to respond to one item than it would to respond to three simple items. (This is why you consider how much time you want respondents to spend vs how many questions the survey should contain.)
There must be a response option for everyone.
If you ask about their favorite flavor of ice cream, then only offer chocolate, vanilla, or strawberry, respondents will be frustrated if their favorite is butterscotch and they don’t find that among the choices. If you’re asking more sensitive questions, such as demographic questions, consider very carefully which options to provide. A question about relationship status needs to have options other than “married” or “single,” since there are many other ways to be in a relationship. Keep language respectful too. Adding “other” as an option when you’re asking about personal traits or attributes (e.g., race/ethnicity, gender) is not respectful. No one is an “other.” No matter what type of question you ask, respondents must see themselves in the survey (read: find a response option that matches their thoughts or experiences) to be engaged and to want to offer their data.
#6 – Test your survey and plan for administration
Employee surveys must be pretested before being administered. Prestesting can be as simple as showing the survey to a few colleagues and asking whether the questions make sense, or it can involve actual pilot testing of the survey with a small, select group of respondents. One way to test a survey with respondents is to do a “think aloud” where a respondent takes the survey and shares with you what they’re thinking as they read and consider how to respond to each question. In this way, you can shine a light on any questions that cause confusion or misinterpretation. Also, consider how respondents will access your survey. Online? On paper by mail? Over the phone? Ensure that the way you want to administer the survey matches what respondents have both the time for and access to.
You’ll know you have a well-crafted employee survey when you achieve a good response rate and collect data that yields important insights that inform key decisions in the workplace.
_____
About the author:
Sheila B. Robinson, Ed.D., of Custom Professional Learning, LLC, is a speaker, educator, program evaluator, and consultant with a passion for the science of teaching and learning, presentations, and asking good questions. Through her talks, professional development workshops and university courses, Sheila teaches people how to make learning stick and how to ask good questions, along with program evaluation, survey design, data visualization, audience engagement, and presentation design.
Sheila is author of Designing Quality Survey Questions (SAGE Publications, 2019), and writes a popular blog with numerous articles on survey design, learning, presentations and other topics. Sheila is also a Certified Presentation Specialist (CPS)™, Vice President of the Presentation Guild, and Senior Design and Facilitation Consultant with Evergreen Data.
