How to Write Effective Questions for Decision-Making, Evaluation, and Research at SJCG
hostedbytheCentreforAppliedHealthResearch
Do you create surveys or write questions to gather feedback or information about SJCG clients, staff, students, or volunteers?
hostedbytheCentreforAppliedHealthResearch
Do you create surveys or write questions to gather feedback or information about SJCG clients, staff, students, or volunteers?
Interested or curious to learn how to create a survey?
hostedbytheCentreforAppliedHealthResearch Four Stages for Creating Your Next Effective Survey at SJCG
Understand Survey Basics
hostedbytheCentreforAppliedHealthResearch Four Stages for Creating Your Next Effective Survey at SJCG
Determine
Purpose of Survey Understand Survey Basics
hostedbytheCentreforAppliedHealthResearch Four Stages for Creating Your Next Effective Survey at SJCG
Determine Purpose of Survey Avoid 3 Common Survey Mistakes Understand Survey Basics
hostedbytheCentreforAppliedHealthResearch Four Stages for Creating Your Next Effective Survey at SJCG
Determine Purpose of Survey Avoid 3 Common Survey Mistakes Pre-test Survey Understand Survey Basics
hostedbytheCentreforAppliedHealthResearch Four Stages for Creating Your Next Effective Survey at SJCG
Determine
Purpose of Survey Avoid 3 Common Survey Mistakes Pre-test Survey Understand Survey Basics
Let’s go through each of the four stages
What is a survey?
Understand SurveyBasics
hostedbytheCentreforAppliedHealthResearch
A survey (also called a questionnaire) is a set of questions used to collect data from an entire population or from a sample of that population.
For more information on populations versus samples, see the previous SJCG Research & Evaluation Month topic, Week 2: “Reaching More Participants.”
Two Types of Questions in Surveys
Closed-ended Provide fixed response options (e.g., multiple choice, rating scales)
Understand SurveyBasics
hostedbytheCentreforAppliedHealthResearch
Two Types of Questions in Surveys
Closed-ended
Understand SurveyBasics
Open-ended
Provide fixed response options (e.g., multiple choice, rating scales)
hostedbytheCentreforAppliedHealthResearch
Allow respondents to answer in their own words
Two Types of Questions in Surveys
Closed-ended
Provide fixed response options (e.g., multiple choice, rating scales)
Open-ended
Allow respondents to answer in their own words
Understand SurveyBasics
hostedbytheCentreforAppliedHealthResearch
Researchers widely recommended a balance of ~80% closed-ended and ~20% open-ended for surveys
Understand SurveyBasics
hostedbytheCentreforAppliedHealthResearch
How creating better surveys impacts you at SJCG
Better surveys can lead to better informed decisions at SJCG.
Whether you are using the data to make decisions or are impacted by those decisions, improving survey quality at SJCG ultimately benefits everyone.
Understand SurveyBasics
hostedbytheCentreforAppliedHealthResearch
Interviews & Focus Groups versus Surveys
Interviews & Focus Groups Surveys
Mostly open-ended questions
Mostly closed-ended questions
Understand SurveyBasics
hostedbytheCentreforAppliedHealthResearch
Interviews & Focus Groups versus Surveys
Interviews & Focus Groups
Mostly open-ended questions
Surveys
Mostly closed-ended questions
Shared Characteristics
Many survey design principles can also apply to interview/focus group questions
Understand SurveyBasics
hostedbytheCentreforAppliedHealthResearch
...do you even need a survey?
Surveys are not always the best choice for every research or evaluation goal.
Consider alternatives
Interviews or focus groups
Existing data sources that may already answer your question
Determine PurposeofSurvey
hostedbytheCentreforAppliedHealthResearch
Determine the purpose of your survey by asking yourself:
1.What decision will this information support?
2.What do we actually need to know (not just what we ’ re curious about)?
Avoid “scope creep”
Determine PurposeofSurvey
hostedbytheCentreforAppliedHealthResearch
Scope creep occurs when a project, in this case a survey, expands beyond its original goals by adding questions that are “nice to know”, but not essential to the purpose of the survey.
Determine PurposeofSurvey
hostedbytheCentreforAppliedHealthResearch
How you can tell if your survey has a clear purpose?
A good tell that your survey has a clear purpose is that it’s easy to determine whether each question is necessary. If you frequently find yourself asking, “Do we need this question?”, it often means the purpose of the survey needs to be refined.
Avoid3Common SurveyMistakes
hostedbytheCentreforAppliedHealthResearch
The 3 most common survey design issues seen most often at
SJCG
If you recognize any of these upcoming 3 in your own work, don’t worry, these are common and fixable.
Avoid3Common SurveyMistakes
hostedbytheCentreforAppliedHealthResearch
The 3 most common survey design issues seen most often at SJCG
Double-barrelled questions 1
These occur when a single question asks about two different things.
Avoid3Common SurveyMistakes
hostedbytheCentreforAppliedHealthResearch
The 3 most common survey design issues seen most often at SJCG
Double-barrelled questions 1
These occur when a single question asks about two different things.
Split into two questions:
“Was the training clear and helpful?”
“Was the training clear?”
“Was the training helpful?”
Avoid3Common SurveyMistakes
hostedbytheCentreforAppliedHealthResearch
The 3 most common survey design issues seen most often at SJCG
2
Overlapping response options
Most commonly seen among age ranges or time frames.
Avoid3Common SurveyMistakes
hostedbytheCentreforAppliedHealthResearch
The 3 most common survey design issues seen most often at
SJCG
Overlapping response options 2
Most commonly seen among age ranges or time frames.
“How many hours do you work per week?”
0 – 20 hours / 20 – 39 hours /
40 – 59 hours / 60 hours or more
“How many hours do you work per week?”
0 – 19 hours / 20 – 39 hours / 40 – 59 hours / 60 hours or more
Avoid3Common SurveyMistakes
hostedbytheCentreforAppliedHealthResearch
The 3 most common survey design issues seen most often at SJCG
Undefined constructs
A construct is an abstract concept you want to measure, like satisfaction. Problems in survey design typically arise when the construct underlying the chosen survey purpose is left vague or undefined in the survey design process.
Avoid3Common SurveyMistakes
hostedbytheCentreforAppliedHealthResearch
The 3 most common survey design issues seen most often at SJCG
Undefined constructs 3
Take “staff satisfaction” with a new SJCG workshop. What does that actually mean to you and your survey design team?
Avoid3Common SurveyMistakes
hostedbytheCentreforAppliedHealthResearch
The 3 most common survey design issues seen most often at
SJCG
Undefined constructs 3
Take “staff satisfaction” with a new SJCG workshop. What does that actually mean to you and your survey design team?
How engaging the workshop felt to SJCG staff?
Avoid3Common SurveyMistakes
hostedbytheCentreforAppliedHealthResearch
The 3 most common survey design issues seen most often at SJCG
Undefined constructs 3
Take “staff satisfaction” with a new SJCG workshop. What does that actually mean to you and your survey design team?
How engaging the workshop felt to SJCG staff?
How useful it was for their job?
Avoid3Common SurveyMistakes
hostedbytheCentreforAppliedHealthResearch
The 3 most common survey design issues seen most often at SJCG
Undefined constructs 3
Take “staff satisfaction” with a new SJCG workshop. What does that actually mean to you and your survey design team?
How engaging the workshop felt to SJCG staff?
How useful it was for their job? A mixture of these two?
Avoid3Common SurveyMistakes
hostedbytheCentreforAppliedHealthResearch
The 3 most common survey design issues seen most often at SJCG
Undefined constructs 3
Take “staff satisfaction” with a new SJCG workshop. What does that actually mean to you and your survey design team?
How engaging the workshop felt to SJCG staff?
How useful it was for their job? A mixture of these two? Or something else entirely?
Avoid3Common SurveyMistakes
hostedbytheCentreforAppliedHealthResearch
The 3 most common survey design issues seen most often at SJCG
Undefined constructs 3
At the beginning of your survey design process, define your constructs, then determine each question to measure specific parts of that definition. That way, your results reflect what you actually set out to understand in your survey - the purpose of your survey.
hostedbytheCentreforAppliedHealthResearch Pre-testSurvey
Before launching your survey, here are 3 different ways you can pre-test your survey:
1. At a minimum, ask teammates or peers to complete the survey to catch awkward wording, unclear instructions, and technical issues.
Pre-testSurvey
Before launching your survey, here are 3 different ways you can pre-test your survey:
hostedbytheCentreforAppliedHealthResearch
1. At a minimum, ask teammates or peers to complete the survey to catch awkward wording, unclear instructions, and technical issues.
2. Conduct a pilot test with individuals who closely resemble your target respondents to ensure the questions are understandable and interpreted as intended.
Pre-testSurvey
Before launching your survey, here are 3 different ways you can pre-test your survey:
hostedbytheCentreforAppliedHealthResearch
1. At a minimum, ask teammates or peers to complete the survey to catch awkward wording, unclear instructions, and technical issues.
2. Conduct a pilot test with individuals who closely resemble your target respondents to ensure the questions are understandable and interpreted as intended.
3. Seek input from researchers or subject-matter experts (SMEs) to strengthen survey design and content. For example, the Centre for Applied Health Research (CAHR) is available to review surveys and provide feedback for anyone at SJCG.
Bonus Tip for Pre-testing Your Survey
Use multiple forms of pretesting rather than relying on a single source of feedback.
If you can, combining perspectives from peers, target respondents, and experts provides better feedback for survey revisions.
