Evaluating curriculum implementation
Guidance for schools to use evaluative thinking practices to plan, monitor and evaluate activities aligned to the phases of curriculum implementation.
Evaluative thinking is a disciplined approach to inquiry and reflective practice that helps us make sound judgements using good evidence, as a matter of habit (CESE, 2021).
Key principles of evaluative thinking include:
- suspending judgement and being aware of potential bias
- asking important questions
- using existing evidence well
- strengthening the evidence base.
Target audience
This resource is designed for school principals, school leadership teams and school staff. Directors, Educational Leadership (DELs), Principals, School Leadership (PSLs), Principals in Residence (PIRs) and Principal Coach Mentors (PCMs) can also use this resource to guide schools with planning, monitoring and evaluating their School Excellence Plan (SEP) and the processes of the QDAI framework.
What and why
Curriculum implementation provides an opportunity for schools to place curriculum at the centre of school planning. Effective curriculum implementation drives student growth and attainment, and school improvement. This requires:
- evaluating curriculum implementation activities that focus on the Leading, Teaching and Learning domains within the School Excellence Framework.
- a focus on curriculum implementation in areas such as educational leadership, staff capabilities, learning and development, improvement goals, policy, data skills and use, classroom practice, and assessment.
When and how to use
Understanding the principles of evaluative thinking and applying effective evaluation processes enables schools to investigate their curriculum implementation initiatives in a meaningful way. Effective curriculum implementation is an iterative and continuous improvement process, occurring simultaneously for different syllabuses as they are made available.
This resource can be used to:
- build staff understanding of what constitutes effective evaluation practices
- strengthen the reliability and validity of evaluation of curriculum implementation practices
- deepen awareness of the principles of evaluative thinking to analyse data effectively
- apply an evaluative mindset to the core processes of aligning curriculum implementation to the School Excellence Plan (SEP) and in developing implementation and progress monitoring (IPM)
- access the Evaluation resource hub to learn more about Evaluative thinking.
Contact
Email questions, comments and feedback about this resource to contactcurriculumimplementation@det.nsw.edu.au by using the subject line ‘School planning for curriculum implementation’.
Questions
To effectively evaluate curriculum implementation, school leaders and teachers need to ask purposeful, well-crafted questions. The quality of the data collected can be strengthened by the quality of the questions asked.
Important considerations include:
- clarifying questions upfront to target the analysis process
- ensuring questions are focused and succinct
- tailoring questions to a phase of the curriculum implementation journey and the type of evaluation being conducted (process or outcome evaluation)
- asking questions that lead to clear, informative data.
- To what extent has our school delivered our intended curriculum implementation plan? How do we know?
- How is our curriculum implementation plan aligned to the priorities of Our Plan for NSW Public Education?
- What progress has been made towards our identified milestones and goals?
- What practices have proven particularly effective during implementation, and should these be scaled or adapted more broadly?
- What factors in the working environment – such as leadership, collaboration, or resources – have enabled or presented a barrier to curriculum implementation? What lessons have we learned?
- What new insights have emerged through implementation that may prompt a review of current priorities, approaches, or focus areas?
Data sources
Once questions have been established, schools need to ensure that the data they are collecting can answer the question being asked.
Considerations include:
- the type of data (qualitative, quantitative)
- the collection method (self-report, observation, assessment)
- the scale (granular – student, class, teacher or aggregate – whole-school)
- collecting baseline data to determine the progress made.
Where possible, schools should triangulate multiple sources of data to ensure perspectives are balanced.
Defining evidence types
Evidence of activity:
- What has taken place for curriculum implementation?
- What did we do?
Evidence of process quality:
- the quality of the curriculum implementation activities
- How well did we do it?
Evidence of impact:
- progress in achievement of curriculum implementation outcomes when comparing our baseline data
- What difference did it make?
Examples of data that schools could consider using to support the evaluation of curriculum implementation activities.
| Evidence of activity | Evidence of process quality | Evidence of impact |
|---|---|---|
| Professional learning records | Document analysis (teaching and learning programs) |
Document analysis of teaching and learning programs |
| Classroom observations | Classroom observations | Classroom observations |
| Staff or faculty meeting minutes | Professional learning exit slips | Professional learning exit slips |
| Assessment schedules | Perceptual data: student and parent surveys | Perceptual data: student and parent surveys |
| Scope and sequence documents | Focus group responses | Internal and external student assessment data |
| Student work samples | Moderation feedback sessions | Focus group responses |
| Financial records | Whole-school processes and procedures | NAPLAN (value-add, expected growth) |
QDAI framework
The QDAI is an evaluative thinking framework to guide evaluation and support the systematic collection and interpretation of evidence for curriculum implementation.
The following tools can be used to guide schools with using and reviewing the framework.
The following questions can be used to guide your curriculum implementation evaluation practices.
When planning to evaluate curriculum implementation, consider:
- In what ways has your school leadership team provided opportunities to enhance evaluative thinking practices?
- How is evaluative thinking used to monitor curriculum implementation across your school?
- What opportunities are created to discuss potential areas of bias when planning evaluation?
- Is the focus of your evaluation clear (aligned to SEP initiative; an existing problem to be investigated further; an important question to answer)?
- What evidence needs to be obtained? For example evidence of activity, process quality, or outcomes?
- How can the QDAI framework be applied to support your evaluation process?
- Has your team considered if existing evidence sources and evaluation tools could be utilised?
- Who will be involved in the evaluation process? Who will lead the evaluation? Who is on the evaluation team? Audience?
- Are the interests of all your stakeholders represented (parents and carers, diversity of students, and so on)?
When using evaluation tools to gather data and evidence of the efficacy and impact of curriculum implementation, consider:
- Is the method or evaluation tool suitable, reliable and valid?
- Does the chosen tool consistently measure what it is supposed to measure?
- Does the tool allow reliability across time (test/retest reliability), across items (internal reliability) or across evaluators (inter/rater reliability)?
- Does the tool accurately measure what it is supposed to measure?
- What critical factors may need to be controlled so that results are valid and reliable?
- Is your ‘inquiry’ or ‘key evaluation question’ focus narrow?
- Is the sample size appropriate? Does it broadly represent your target audience?
- Has more than one evaluation point or method to collect evidence been obtained?
- Have your evaluation tools been designed in a collaborative way?
When analysing results to determine meaningful evidence of curriculum implementation, consider:
- Are findings generalisable across your context (school, faculties or stages, teams, classes)?
- Has collaborative analysis of the data been undertaken?
- Have results been triangulated with other sets of conceptually similar data?
- Can you determine the ‘effect’ size?
- Do the findings prove any statistical correlation or causation? How do you know?
When determining the next steps:
- Do the evaluation findings have ‘real world’ significance to your context?
- How do the findings inform your next steps?
- Where are the opportunities to strengthen and scale your success? In what ways will your solutions for ongoing areas of concern be addressed?
- What systems and processes will continue to be used as a part of your next iteration of the curriculum implementation improvement cycle?
Evaluation tools
The following evaluation tools could be used by schools to guide the collection of data to evaluate curriculum implementation activities.
Surveys
Surveys are a consistent set of questions used to gather data about experiences, opinions beliefs and motivations. Evaluation resources – surveys explains the purpose and effective use of surveys and provides a range of sample surveys which schools may use and/or adapt for their own contexts and needs.
Observation
A structured observation is an organised process for watching and recording events or behaviours that occur in a particular setting (for example: learning space, playground, meetings). Evaluation resources – observation includes an observation template that schools can contextualise, adapt and draw from to meet the needs of each school.
Focus groups
A focus group is a guided conversation that helps gather people’s thoughts, feelings, opinions, and experiences. Evaluation resources – focus groups provides guided advice to schools on how to conduct a focus group, including a suggested planning and collation template that can be contextualised to meet individual school needs.
Document analysis
A document analysis is a structured way of reviewing existing documents to help answer specific evaluation questions. Evaluation resources – document analysis includes a template schools can use to complete document analysis.
Resources
The following tools could be used by schools to evaluate curriculum implementation resources and consider evaluative practices.