NSW Literacy and Numeracy Action Plan 2012-2016 evaluation
This report was originally published 02 June 2017.
This report presents the findings of the evaluation of the NSW Literacy and Numeracy Action Plan (Action Plan). The Action Plan was implemented in targeted schools in NSW over the five-year period, 2012-2016, with an independent evaluation being conducted over this period. The evaluation reports on the extent to which student literacy and numeracy performance improved, factors that may have led to any improvement and the extent to which any improvement achieved was cost-effective.
The NSW Literacy and Numeracy Action Plan, 2012-2016
Through the Action Plan, the NSW Government progressively allocated $261 million to meet the needs of some 41,392 Kindergarten to Year 2 students in 448 targeted schools over the period 2012 to 2016.
The schools targeted were serving communities characterised by aggregated social disadvantage, and were among the lowest-achieving in NSW.
Targeted schools were provided resourcing to:
- support the explicit assessment of the learning needs of students especially on entry into Kindergarten
- provide classroom-based professional development for teachers in personalised learning and diagnostic assessment
- adopt the use of a three-tiered response to interventions for those children who need special attention
- focus on whole school instructional leadership, including the appointment of Instructional Leaders, Literacy and Numeracy, within the government school system, and equivalent positions in the Catholic school sector.
The Action Plan, as represented in Figure ES1 aimed to increase the literacy and numeracy outcomes for students in the targeted schools, and to reduce the influence of socio-economic status as a key determinant of students’ academic performance.
2. Evaluation Design
Objective of the evaluation
The objective of the evaluation was to provide comprehensive responses to three key questions:
- To what extent has student literacy and numeracy performance improved?
- What specific factors led to the outcomes achieved?
- To what extent were the outcomes achieved cost-effective?
Data gathering methods
Six longitudinal case studies were visited annually from 2012 to 2016, with in-school interviews and focus groups being conducted in 66 different schools across the three sectors from 2013 to 2016. Online surveys with 1,005 responses from instructional leaders and 1,159 responses from principals were conducted. Analyses of Kindergarten to Year 2 performance data for Reading (Aspects of Text), Comprehension, Writing and Numeracy, as measured by teacher benchmarking against the NSW Literacy and Numeracy Continua were undertaken. Year 3 NAPLAN Reading, Writing and Numeracy data are also reported as a lagging indicator of student learning outcomes. In addition, 10,791 responses were received from online student attitude surveys, conducted from 2013-2016.
3. To what extent has there been an improvement in student learning outcomes 2012-2016?
In the government sector, where reliable data are available since 2013, the percentage of students at or above the expected end of year standard has increased in Reading by 24 percentage points in Kindergarten, 27 percentage points in Year 1 and 20 percentage points at Year 2 level between 2013 and 2016. A similar rate of improvement was observed in the non-government sector. While these results are encouraging, the data was not able to be compared against data from non-Action Plan schools.
The vast majority of principals also reported that they have observed measured growth in students’ literacy (94% great or moderate improvement) and numeracy (87% great or moderate improvement), basing these observations not only on the K-2 Continua data but also against a range of standardised tests and school based assessments. The great majority (94%) of instructional leaders also believed the Action Plan had brought about positive changes in K-2 literacy and numeracy teaching and learning in their school in 2016.
In contrast to the results from the Literacy and Numeracy Continua, the Year 3 NAPLAN results (see Table ES3), as a lagging indicator of student outcomes, showed no significant change over time for the full cohort of participating schools, and no significant closing of the gap between targeted and non- targeted schools.
To investigate further whether the Action Plan had a measurable impact on student NAPLAN outcomes, the Centre for Education Statistics and Evaluation (CESE) conducted two analyses using multilevel mixed-effects regression models.
The apparent disparity between the improving K-2 results and the static NAPLAN results requires further consideration in the implementation of the Literacy and Numeracy Strategy, 2017 – 2020.
4. What specific factors led to the outcomes achieved?
- Changes in teaching and learning of literacy and numeracy resulting from the Action Plan.
- Changes in principal practices in implementing the Action Plan.
- Changing use of paraprofessionals.
- The contribution of target setting for literacy and numeracy.
- Increased teacher collaborative planning.
Key Elements of the NSW Literacy and Numeracy Action Plan
- Use of diagnostic assessment.
- Implementation of personalised learning (and differentiated teaching).
- Effectiveness of implementation of tiered interventions.
- The role of the Instructional Leader in targeted schools.
- Building the capacity of teachers and school executives to meet the needs of low performing students.
Other factors impacting on the implementation of the Literacy and Numeracy Action Plan
- Impact of the Action Plan on parent and community engagement in participating schools.
- Sustainability of the impact of the Action Plan post-2016.
6. Discussion and Conclusions
How successful has the Action Plan been in achieving its objectives?
While the improvement in student learning outcomes from the Action Plan may not have been as great as desired, this does not mean that the Action Plan did not provide a range of benefits for students, teachers and schools. Rather, when its impact on the quality of teaching and learning and school culture is considered, the evidence demonstrates widespread impact on a range of school practices, including:
- Evidence-informed practice, personalised and student-centred learning.
- Enhanced teacher capacity to tailor learning experiences according to identified student need.
- Greater sense of collective responsibility for student outcomes.
- More tailored use of interventions for students at risk.
- More appropriate use of specialist and paraprofessional staff.
- Stronger accountability for outcomes and understanding and acceptance of critical reflection on the effectiveness of practice.
The importance of these changes should not be under-estimated, nor should the difficulty in shifting entrenched school and community attitudes and expectations about what “normal” practice should be. The Action Plan has been an effective catalyst for schools to examine their practices and adopt more appropriate ways of working.
What can be learned from the Action Plan 2012-2016?
The importance of targeted sectoral support
The chances of success are greatly enhanced when the sectoral authorities prepare the principal and staff for participation in the Action Plan by making clear what is expected of them, not only in terms of compliance with accountability requirements but also the outcomes that are intended.
The sectoral authority also has a critical role in ensuring that both the principal and key staff have a sound understanding of the research base that underpins the Action Plan, including contemporary views on what constitutes best practice pedagogy in the early years of schooling.
Empowering school leadership to drive cultural change
Successful schools understood why they had been targeted for assistance, and why they needed to fundamentally change what they were doing to break their long-standing patterns of under-achievement.
Successful Action Plan schools were characterised by principal leadership that was inclusive and supportive. Successful principals also understood the need for transparency of decision making, underpinned by evidence. They developed the notion of the school staff as a team with collective responsibility for whole school direction and achievement.
The pivotal role of instructional leaders
Instructional leaders have been successful because they have had the time and expertise to focus solely on building capacity of both school executive and classroom teachers in early learning in literacy and numeracy, while ensuring that student data are the enduring source of evidence upon which informed decisions are made about teaching and learning.
Ensuring a focus on quality teaching and learning in classrooms
Successful Action Plan schools implicitly accepted that it was the quality of teaching and learning that “makes the difference” in students’ outcomes, rather than blaming poor results on the students’ backgrounds.
The pervasive impact of classroom teachers on student learning achievement
Successful Action Plan schools understood that improving student learning often required a completely different way of doing business. Building teacher confidence and competence needs to remain a high priority. Teachers in successful Action Plan schools understood how to effectively differentiate lessons to address student needs.
Increased focus on data as the basis of planning and pedagogy
One of the strongest outcomes for teachers and school executives across all sectors has been a substantial increase of focus on data analysis as the basis for planning at whole school, grade/stage and classroom levels. Importantly, formative assessment and analysis are integrated into teachers’ normal practice, rather than an add-on to it.
The emphasis on student learning data has undoubtedly helped many schools to move closer to a stage where evidence-based practice is the norm in many targeted schools. However, there is scope for further development of teachers’ skills in deep analysis of student learning data.
The Action Plan demonstrated that improving the quality of teaching and learning in the classroom can be achieved through enhancing teacher and school leader’s capacity to implement the key priorities areas emphasised during the Action Plan, including enhanced diagnostic assessment, greater application of differentiated teaching and personalised learning, and tiered interventions when required. The focus on building teacher and executive capacity has been fundamental to the successful implementation of the Action Plan.
In this regard, the following seven points summarise the key lessons learned from the Action Plan that have relevance to future initiatives:
- School improvement requires changing school culture, not simply adding additional programs.
- Achieving enhanced teacher quality requires a different approach to teacher professional learning: it must be focussed on developing the teachers’ confidence and competence to address identified student needs.
- Structured reflection on instructional practice must become routine, and not an add-on to teacher’s work but an integral part of it.
- The strong focus on enhanced instructional leadership K-2 has been essential for enhancing the quality of teaching and learning in classrooms.
- The additional time given for collaborative teacher planning and the additional expertise for intervention in classrooms are necessary for successful implementation.
- The strong support and accountability provided as a result of the Action Plan have been essential for providing consistency and coherence to schools’ efforts.
- The provision by sectors of tailored pressure through accountability for student results and simultaneous professional support, in accordance with school needs, continues to be a key ingredient in each school’s ongoing success both in the short and mid-term.