This literature review was originally published 11 December 2014.

Summary

Background

Continuous, systemic school improvement is increasingly seen as essential by education systems. The focus on school improvement is driven in part by a growing awareness of international educational performance (through, for instance, the Programme for International Student Assessment, or PISA), and major comparative works published by organisations such as the OECD and McKinsey & Company. One recent OECD report argues that assessment and evaluation of students, teachers and schools are ‘becoming critical’ to establishing high performance and providing feedback, with the ultimate goal of improving student outcomes¹.

In his foreword to the 2007 McKinsey & Company paper, How the world’s best performing school systems come out on top, Andreas Schleicher explained the economic imperative sitting behind the drive for improvement:

The capacity of countries … to compete in the global knowledge economy increasingly depends on whether they can meet a fast-growing demand for high-level skills. This, in turns, hinges on significant improvements in the quality of schooling outcomes and a more equitable distribution in learning opportunities².

Although it is widely accepted as necessary, systemic and continuous improvement is also acknowledged to be a complex process, requiring action over many domains. Many large initiatives for school improvement fail because they do not change day-to-day school practices, which are ‘recognised as remarkably impervious to, and self-protective against, fluctuating external policies and agendas’³.

A related challenge is sustaining change once it has been enacted: sustainability is essential if Australia wants to become and remain a top educational performer. Sustaining change means building capacity within schools, to ensure that teachers and schools are adaptive, capable of continuous learning, and can take charge of change⁴. As part of this process, schools and districts must work together to share best practice⁵ . This ‘systemic’ approach is not new, however Barber and Fullan have argued that what is needed is a shift from systems thinking to systems action — for the strategic, powerful pursuit of improvement in practice⁶.

There is evidence that substantial, long-lasting change is possible. McKinsey & Company’s 2010 report, How the world’s most improved school systems keep getting better, identifies a range of improved education systems from across the world that have made sustained improvements from a wide range of starting points⁷. For instance, Singapore has transformed its education system from ‘fair’ to ‘great’ in a twelve-year period; and Ontario, Canada moved from ‘good’ to ‘great’ within the space of ten years⁸.

The McKinsey report also provides guidance as to the kinds of practices improving educational systems have undertaken, to foster that improvement. These practices include building teacher and principal technical skills; student assessment; use of data systems; facilitation of improvement through policy and law as well as revision of curriculum and standards; and ensuring appropriate reward and remunerations structures for teachers and principals⁹.

In recent years, many school systems have developed school improvement or performance frameworks as a means of:

• identifying the core components of the school improvement process
• supporting schools through this process, and
• assessing schools’ performance against core components.

All states and territories in Australia have developed such frameworks. Australian interest has sharpened recently, with the development of a National School Improvement Tool (NSIT), intended for implementation as part of the National Plan for School Improvement.

Many of these frameworks use standards as policy levers. These are useful as they provide a clear articulation of good practice; they support self-reflection and assessment; and, as they are not relative, they support system-wide improvement. For instance, the NSIT describes school performance at ‘Low’, ‘Medium’, ‘High’ or ‘Outstanding’ levels. This allows schools to ‘make judgements about where they are on their improvement journeys, to set goals and design strategies for improvement, and to monitor and demonstrate school improvement over time’¹⁰.

This paper considers the research literature related to school improvement frameworks, to identify the core components and processes of such frameworks, and to assess evidence of their efficacy. While there is a substantial body of research bearing on the development of improvement frameworks, the evidence regarding the effect of frameworks on student outcomes is comparatively slight and inconclusive. This is because it is much easier to describe a framework than it is to test whether its implementation improves student outcomes. However, as outlined below, many of the individual components of school improvement frameworks are underpinned by robust evidence.

How is improvement measured?

The international evidence indicates that school improvement is best measured with reference to both student outcomes and school practices or processes, rather than by focusing exclusively on one or the other. Raudenbush argues that no matter how sophisticated analysis of student outcome data is, there will always be limitations as to how much the data can tell us, claiming that ‘to be successful, accountability must be informed by other sources of information … in particular, information on organizational and instructional practice’¹¹. Similarly, Elmore argues that accountability systems must go beyond testing and regulation, to actively engage those who work in schools through ‘explicit strategies for developing and deploying knowledge and skill in classrooms and schools’¹².

A dual approach (encompassing both outcomes and practices) is already evident in some school systems. For instance, the Nova Scotia School Accreditation Program (NSSAP) requires schools to select one area of practice alongside one area of student achievement for improvement. Working in ‘professional learning communities’, staff members collectively establish goals for these focal areas, while the school more broadly establishes strategies to meet them¹³.

The evidence so far: The impact of improvement frameworks on learning

It is difficult to find evidence about the operation or the efficacy of school improvement and accreditation frameworks, for a number of reasons.

First, it is difficult to isolate the impact of a school improvement system when it has been implemented nation-wide, with no control groups, at the same time as other major policy changes¹²⁷. In the United States, school accreditation takes place against a backdrop of Federal standard-setting (particularly, No Child Left Behind) and it can be difficult to disentangle the impact of state-based accreditation from that of Federal measures.

Further, the range of models, and the range of contexts in which they have been implemented, make it difficult to isolate and identify ‘what works’¹²⁸. For instance, there is some evidence available in relation to the impact of school inspections, but these inspections may occur quite separately from any explicit school improvement or accreditation agenda. Some of these studies cite ‘plausible’ evidence that school inspections lead to school improvement and teachers’ behavioural change but findings are far from conclusive: one literature review found that school inspections had both small positive and negative effects on student outcomes¹²⁹. The authors of that review concluded that researchers still ‘do not know how school inspections drive improvement of schools and which types of approaches are most effective and cause the least unintended consequences’¹³⁰. Findings in relation to accreditation programs are also mixed: while many teachers and leaders find accreditation to be a useful process that has enhanced the overall quality of their schools¹³¹, others point to the stress and anxiety that can result from inspection and evaluation processes¹³².

Few studies have empirically assessed the impact of school improvement frameworks on student outcomes, and those that have been undertaken present inconclusive results¹³³. Studies tend to use surveys or interviews of staff rather than performing any analysis of student outcomes¹³⁴. Repeated evaluations of the Southern Association of Colleges and Schools, which operated within a school-improvement framework, did assess student outcomes, but found no difference in school performance data (as measured by standardised achievement tests in reading and mathematics), between accredited and nonaccredited schools¹³⁵.

A number of studies have found that teaching and learning are the school elements that benefit least from accreditation or improvement frameworks. Such findings have been reported in:

• Nova Scotia, where some of the lowest-scoring survey items were those relating to the impact of the School Accreditation Program on teacher practice and student achievement¹³⁶;
• New England, where interviewees ‘strongly asserted’ the benefits of the program for teachers and the school, but held conflicting views as to the impact of accreditation on students¹³⁷.

A similar finding was made in Queensland, where a Masters developed tool (similar to the NSIT) was used to evaluate school performance across eight domains, over time. All Queensland schools were audited with the tool in 2010, and 25 per cent of schools were re-audited in 2011. While there were improvements across some areas after the 12-month period, the teacher practice domain showed the least improvement¹³⁸.

These findings seem at odds with the evidence about the impact best practice (such as high expectations, professional development in data skills and instructional leadership) can have on teacher practices and student outcomes. This may indicate that problems arise not in the content of these frameworks, but in their implementation. Hattie’s synthesis of 800 metaanalyses identifies the challenge of realising the results of any educational initiative where it matters most, finding that while professional development was likely to change teacher learning (with an effect size of 0.90), it was less likely to change teacher behaviour (0.60) and even less likely to have an impact on student learning (0.37)¹³⁹.

Enabling and sustaining school improvement through cultural change

In the view of some educational experts, accountability measures are best used as means to an end, rather than the end itself. Pressure and accountability, when divorced from support and other goals (such as development of capacity) can have a negative effect¹⁴⁰. Elmore suggests that this may be because teachers are already operating ‘more or less at the limit of their knowledge and pedagogical skill’, and adding pressure, without also providing support or guidance as to how to reach goals, may have little impact¹⁴¹. Fullan similarly argues that using test results alone to punish or reward schools ‘assumes that educators will respond to these prods by putting in the effort to make the necessary changes … it assumes that educators have the capacity or will be motivated to develop the skills and competencies to get better results’¹⁴². Drivers of school improvement are far more likely to be successful if they foster intrinsic motivation; engage educators and students; inspire team work; and affect all teachers and students¹⁴³.

There is some support for this in the McKinsey study, which found that teachers in successful systems received 56 per cent of all support initiatives, but only 3 per cent of accountability measures, such as teacher appraisals. Teachers in these systems were held accountable through their students’ learning and collaborative practice with their peers:

By developing a shared concept of what good practice looks like, and basing it on a fact-based inquiry into what works best to help students learn, teachers hold each other accountable to adhering to those accepted practices¹⁴⁴.

A number of education systems provide support for their schools as part of their improvement or accreditation frameworks. For instance, in the Northern Territory, a coaching model is used to develop principals’ skills¹⁴⁵; in South Australia, the Department of Education and Children’s Services is responsible for developing workforce capabilities and system capacity as part of its Improvement and Accountability Framework¹⁴⁶; and in Victoria, the Performance and Development Culture framework encourages effective induction and mentoring support for teachers¹⁴⁷.

Cultural change of the type described by Fullan may be difficult to achieve, but it is possible. Moreover, it is essential to sustaining educational improvements. Tucker observes that a ‘sustained emphasis on education quality … carries enormous implications’ in terms of garnering support at all levels, from government to educators and the broader community¹⁴⁸.

The McKinsey study identified 13 ‘sustained improvers’ – systems with at least five years of consistent rises in student performance across multiple data points and subjects. As the study reports:

For a system’s improvement journey to be sustained over the long term, the improvements have to be integrated into the very fabric of the system pedagogy¹⁴⁹.