I spend a lot of time developing data collection instruments for my clients, particularly surveys. Most surveys I create these days are online, but some are still paper-based depending on who the audience is for them. I enjoy crafting surveys and really am happy to do it for my clients. I especially like it when they successfully render a strong response rate!
But…….sometimes I have clients who have small budgets (and little time) and want to create their own surveys with some pointers from me. Recently this happened and I decided it’s time to put something down on paper. This is what I came up with.
How to Create a Survey
Entire books are dedicated to developing surveys. The following are just some of the steps to take to create a reliable instrument that will produce valid information.
Step 1. How will you use the information?
Will information be shared with a large audience or be for internal use only? Be sure to identify the survey’s purpose as early as possible, not only to set the tone for the survey but also to keep the questions focused.
Step 2. Who do you need to know it from?
This is your unit of analysis. Are you interested in directors, board members, volunteers, clients, training attendees, trainers (the list goes on…)? Your answers will frame the types of questions you ask and the type of survey you will give. For example, Who can give you the answers you need? Will your audience be more likely to respond to a phone, e-mail, or paper/pencil survey?
Step 3. What do you need to know?
Answering this will help determine the types of questions you need to include in your survey. Are you going after activities and outputs, or outcomes that indicate a change has occurred? Are you interested in attitudes and opinions (e.g., conference survey) or performance and characteristics (e.g., behavioral survey )?
Step 4. What is the best way to ask your questions?
Survey questions, which fall primarily into the four categories below, can be closed-ended or open-ended. Closed-ended questions are easier to analyze but often are harder to create because researchers must have a keen understanding of the issue at hand to provide the most appropriate list of options. Here are the most popular question types with what they measure and both open- and close-ended examples:
||Please describe how satisfied you were with the training, and why…
||On a scale from 1 to 5, please rate how effective…
||Actual behavior or anticipated behavior
||Estimate the number of volunteers you connected…
||Of the following choices, how many volunteers did you connect…
||Possession of facts or understanding
||Please describe how to make your family volunteer program envelop the ideas of Neighboring…
||Is creating family-friendly opportunities for low-income residents a key concept of Neighboring…
||Fill in the blank for the proportion of FTEs that focus on episodic volunteer programs…
||Of the following choices, how long have you been employed at the VC…
When using close-ended questions, it is essential that they are:
- Mutually exclusive (e.g., under 1 year, 1-3 years, 4-6 years, over 6 years)
- Distinctly different choices (e.g., hits in newspaper and hits in media are nearly the same)
- Exhaustive, making sure to include all the relevant alternatives, because leaving out a choice can give misleading results; it is always possible to add ”other” and “don’t know” responses to encompass any answers you might have inadvertently left out
It is best to keep surveys short to not overburden respondents. To select your questions, divide them into three groups: Must have, useful to have, nice to have. Then, discard the last group, unless the other two groups are very short.
- Do not put two questions into one (“Do you wear pants or shorts?” An answer of yes means the respondent wears shorts, pants, or both even though they might only wear one or the other and never both at the same time)
- Avoid emotionally charged words
- Avoid leading questions that point toward a certain answer; the way you write a question may change the responses given, so make sure wording does not favor one answer choice over another
- Be aware of cultural factors
- Avoid technical terms and acronyms, unless you are absolutely sure that respondents know what they mean
Step 1. Survey introduction
To help respondents feel more comfortable, write a cover letter, e-mail introduction, or speaking prompt to introduce the survey, and include:
- Purpose and how data will be used to show participation is worthwhile
- Incentive, if any (to boost response rate)
- Confidentiality and anonymity levels and statement that responses are combined with many others to help inform about the overall group of respondents, not the individuals
- Completion instructions
- Time commitment
- Return instructions
- Note of gratitude for their time
- Conditions of participating (Is it required for membership?)
Step 2. Format
Carefully consider the layout of your questionnaire. This is especially important for paper/pencil and e-mail surveys. Use plenty of white space, an easy-to-read font, and sufficient instructions to make the survey easy to complete.
Step 3. Pilot
A trial can uncover unanticipated problems with question wording, instructions, or definitions. A pilot also allows you to determine if the interviewees understand your questions and are able to give useful answers.
Step 4. Follow-up
To boost the response rate, along with the cover letter highlighting the importance of the survey:
- Send a reminder to non-respondents one week after your close date
- Send a second copy of the survey one week after the initial reminder
- Make follow-up phone calls to a sample of non-respondents to see why they didn’t respond, and attempt to secure their participation by sending them another copy of the survey
RESOURCES & REFERENCES
 Confidentiality: the data won’t be shared with anyone, but the researcher knows who said what statements. Anonymity: nobody, including you, knows who gave what statements.