Step 1: Decide on a sampling strategy
The people whose views or experiences you want to understand are called the ‘target population’ for a survey. The people who you are able to contact are the ‘sample frame’.
Sampling for surveys
If the target population is small and easy to reach (for example, Year 9-10 students at our school), they can all be invited to take part in the survey. If the target population is large (for example, all Year 9-10 students in NSW), a carefully chosen sample can be used to represent the population or particular contexts within it.
If you only hear from a sample of the population, you want this sample to allow you to draw inferences about the broader group. Regardless of how you build your sample (see below), the survey should:
- be easy for people to respond to if they choose to participate (follow the link below to Step 2: Choose a survey format).
- only ask questions that are clear and relevant to the respondent (follow the link below to Step 3: Design the questionnaire).
- be compelling and target the right people (Step 4: Invite people to participate).
Strategies for drawing a sample
A census is a 'full coverage sample frame', where we seek a response from everyone in our target population.
A random sample is where everyone in the target population has an equal and random chance of being invited to participate in the survey. Microsoft Excel has a few random number generator functions that can help with this. There is also open source randomising software, such as the Research Randomizer.
A systematic sample is similar to a random sample, but where participants are selected according to a rule or pattern. For example, a systematic sample may be drawn by selecting every 10th participant from a list. The 10th, 20th, 30th and so on will be selected to become the members of the sample group.
A stratified sample refers to dividing the target population into sub-samples (‘quota’ groups), then applying an appropriate sampling strategy within each group. This helps ensure that the diversity in the target population is reflected in the sample (for small, medium and large schools; secondary, primary and central schools; urban, regional and remote locations and so on).
A purposive sample refers to narrowing the sample frame just to a particular sub-group, based on the purpose of the study. For example, instead of surveying all our Year 9-10 students, you might just survey those who have withdrawn from a subject or changed their enrolment part way through a year.
A convenience sample is where the where the sample frame is based on people we can contact easily and conveniently, based on their availability or our resources. In convenience sampling, you cannot assume that the people you hear from (for example, parents and carers who pick their children up from the school gate) are representative of the broader population (all parents and carers). Convenience sampling tends to be used when you need a general sense of views in the population, rather than drawing inferences about the population as a whole.
A snowball sample is an iterative sampling strategy where you start with a small number of people who meet your criteria and then build from there, based on advice from those early respondents. If you want to hear from students who cycle to school, for example, you might start by greeting those who arrive at the bike shed before school on a few mornings, or pegging a note to their bicycles if you miss them. Those students are asked to promote the survey among others they have seen riding to school, or to suggest where other students park their bikes so they can be recruited as well. This process continues until you have obtained sufficient interviews or exhausted all possible leads. Snowball sampling is suitable when the target population is hard to find or there are few other ways of building a sample frame.
To achieve an adequate sample size and coverage, it may be necessary to use different approaches with different groups in a population of interest. For example, you might do a random sample of all students, complemented by a purposive or snowball sample of particular students whose experiences are uncommon but important to understand for the purpose of the evaluation.
Response rates
The response rate for a survey is expressed as a percentage: the number of people who responded to the survey, as a proportion of the total number of people invited to participate.
Response rates can be hard to predict, but 30% is a good target for surveys that are voluntary and don’t have any material incentive attached (such as a prize draw). Response rates can exceed 85% when the target population is motivated or feels obliged to respond, and the survey is well-executed. However, they can fall below 2% (1 response for every 50 invitations sent) when motivation is low, the questionnaire is poorly designed or contact details are unreliable. For tips on maximising response rates, follow the link below to Step 4: Invite people to participate.
Sample sizes
One of the most common questions when planning a survey is: “How big a sample do I need?” Like many things in research and evaluation, the answer is: “It depends.”
The two main driving factors behind sample size calculation are the population size and the margin of error you are prepared to accept in our analysis.
If the population of interest is large (say, 5,000 Year 9-10 students across a network of schools) and a random sample can be drawn, you may only need to hear from 10%-20% of people. The smaller our population of interest, the closer you need to get to a census in order to have the same level of confidence.
For more on this topic, refer to the notes in the School Excellence Framework evidence guide (linked below) on confidence intervals and statistical significance. A sample size calculator tool and accompanying notes can also be found at Creative Research Systems, linked below.
With larger target populations, where you are making choices about your required sample size, a third consideration that might drive up our sample size is diversity within the population.
- The more you want to ‘slice and dice’ survey results, comparing one sub-group with another, the larger your sample needs to be. In the example of 5,000 Year 9-10 students across a network of schools, if you know you will want to analyse responses for each school individually (rather than just the whole network), you need to factor this into your sampling to ensure that you have sufficient responses from each school.
- If you know you want to isolate the views of a sub-group that is not highly represented in the target population (for example, students who have moved school since the start of Year 7), you might need to ‘oversample’ them.
For advice on these and other technical sampling matters, contact evaluationsupport@cese.nsw.edu.au