Most significant change

Most significant change (MSC) is a participatory approach to monitoring and evaluating change – participatory in that those who have been involved in an innovation process can tell their story in their own words, and also because they are involved in the analysis and sense-making process.

MSC was first developed in international community development contexts. It involves the gathering of rich stories from the field through a systematic process of data collection and story building. The final MSC narratives can be used for evaluation and to inform future planning. It is easily adaptable to school contexts for evaluating cycles of innovations as they are in progress.

During or at the end of a process of innovation, MSC gathers data through interviews with participants and a range of stakeholders about changes that have taken place so far. Then this data is shaped through consultation into a narrative of change that can be shared with the wider community.

The final set of stories represents the collective view about the most important changes that have taken place, through the eyes of key stakeholders and participants. The process of story-telling and selection captures what people most value and how they make sense of what has happened.

Why use most significant change?

MSC is a collaborative and participatory approach to evaluation so that all the people who are involved in a project can analyse the impact of the work together. Students, teachers, non-teaching staff, parents, community members, mentors, school executive can all have their views incorporated into the process. It is open ended, allowing participants to voice what is important to them in their own words. This is in contrast to other approaches, where the key criteria for ‘success’, ‘effectiveness’ or ‘impact’ are pre-determined by the project or the evaluators.

MSC is a flexible approach that can be used in a single site or across multiple sites (for example, classes, schools) as long as they are involved in a similar program or innovation. Each site will have their own MSC story (or set of stories), and stories can be shared and circulated across sites. The final MSC stories can be shared with the school and wider community.

If it is used regularly (for example, annually) in a single site, through multiple iterations of an innovation program, it can map changes in priorities over time and develop a cumulative set of stories about the identified domain of change. This allows an organisation to adjust the focus of attention and the success criteria that are used to determine its impact.

Strengths and limitations of most significant change

In evaluation MSC helps decision makers focus on the most important findings for participants in the process, rather than wading through large amounts of complex information and trying to determine what is important themselves. MSC emphasises people’s experiences of innovation.

It can identify any sorts of changes, positive and negative, as long as these are identified by participants as significant. It is open as to how change is experienced, understood and articulated by participants. That said, the MSC guide (see link below) estimates that 90-95% of stories will be positive. Other lines of questioning, or other methods, will need to be pursued to understand project limitations and areas for improvement.

MSC is not intended to be a stand-alone evaluation technique – it is supposed to be complementary to other approaches, and sits particularly well alongside quantitative analysis. MSC does not aim to generalise across multiple sites or contexts, or present an ‘average’. It produces specific stories about a specific time and place, and cannot reliably predict what will happen in a different time and place even when the same innovation is attempted. However it can generate broad insights across sites about a particular innovation.

MSC does not rely on ‘hard’ evidence of test results and other objective measures but on the subjective lived experiences of people. Rather, MSC allows discovery of important aspects of innovation processes that may not be picked up in other data.

MSC relies on people’s willingness to be open and honest about their experiences in an interview or focus group. This has implications for the selection of participants and facilitator. It is important to select a facilitator who is seen as objective and is not intimidating or in an unequal relation of power to the participants.

How do I use most significant change?

Use MSC to identify and evaluate people’s experiences during or after an innovation cycle. Possible stages include:

  • Select facilitator – someone who is:
    • adept at interviewing and leading open discussions;
    • able to draft a story of change from the data;
    • willing to consult with stakeholder participants on drafts of the MSC story.
  • Identify the points in time during the innovation cycle when it will be appropriate to conduct interviews for example, usually at least half way through and/or at the end.
  • Prepare your key question for the story collection, via semi-structured interviews with students and teachers (and potentially other stakeholders such as parents). Aim for sufficient time for wide-ranging discussion in the interview (for example, at least 30 minutes per person in the interview), remembering that one story often leads to another, and then to another.

The wording of the question is important. Lead with an open question that focuses on a time span (for example, ‘over the last month’, ‘while we have been studying the wetland’), invites people’s judgments (for example, ‘what do you think’), asks people to be selective (for example, ‘the most important’), ask people to identify aspects that are different from before (for example, ‘change’), identify a specific focus of change (for example, ‘in how you are learning’), and sets some boundaries (for example, ‘about the environment’, ‘about our world’, ‘about Science’).

For example, ‘Looking back over the last month, what do you think was the most significant change in how students are learning about the environment?’

Other questions might follow from the answers – ‘Tell me more about…’ or ‘Was that more important than…’ or ‘How is that different from…’ or ‘Why is that significant for you?’.

Select participants to tell their stories – Selecting a diversity of participants will provide more diverse responses and make it easier to identify change across groups involved in the innovation.

  • Conduct interviews with students, teachers, parents or other stakeholders. One-to-one interviews are often best, but small group interviews (2-3 people) can also work, depending on the topic and the existing level of trust between participants. Large group discussions can become unwieldy for MSC, particularly as the process is designed to elicit individual stories from a personal perspective, rather than group consensus at the data collection stage. Audio recording and/or having a note taker in the room is a good way to ensure that the main points can be checked and identified later, and that the story is (as much as possible) told in the words of the participant.
  • Draft a descriptive story of change about the innovation. Write it using participant’s words as much as possible, from the first person. Include enough information so that people who know nothing about the site or the innovation can make sense of the story. Start with a brief description of the context, site and innovation. Then go on to describe the experiences of participants about what has changed for them, and why this is significant to them. You will need to write several stories. Most MSC stories are less than a page – two pages at the most.

Note: It is also possible to ask people to write their own story and bring it with them, however the process of telling the story to someone else often brings out richer a and details.

  • Verify the stories by revisiting the site and taking the draft stories back to the original participants. Ask them to check whether the story accurately represents the most significant experiences that were shared during the interview. This could be done in a story workshop.
  • ‘Summary by selection’ – Using a stakeholder panel meeting or a story workshop, the aim here is to review the draft stories of change that are significant at the more granular level, and select a subset that are widely valued the ‘most important’ for the innovation. It is important not only to document which stories are selected, but 'why' they are selected, as this reveals a great deal about 'values' (for example, what the selection committee considers to be valuable, and on what basis).
  • Sharing the stories. This can be done via a website or a publication.

When do I use most significant change?

Use most significant change to identify and evaluate people’s experiences during or after an innovation cycle. Below is an example of use at different points in time:

Throughout the project/studyAt the end of the project/study
Purposes

To see how the innovation is travelling at different points in time, perhaps with different stakeholders To identify emerging significant impacts and fine-tune the innovation while it is in progress

To identify what the most important impacts of an innovation have been across participant groups

To build a shared story of change about a particular innovation to share with the community To decide on next strategic directions for innovation

Examples

Asking students across several classes how they are experiencing cross-curriculum inquiry learning so far, building and testing a provisional story of change (for example, perhaps students report main impact as difficulties in setting goals and organising time) Asking teachers across several year levels or subject areas how they are experiencing cross-curriculum inquiry learning so far, building and testing a provisional story of change (for example, perhaps teachers identify different sorts of scaffolding that students are requiring)

Asking students, teachers and other stakeholders how they experienced the cross curriculum inquiry learning unit this semester; building/ revising and testing a collective story of change (for example, perhaps increased autonomy and responsibility are identified by both groups, with the caveat that students and teachers both needed time and support to develop these; next focus to be …..)
How long does it take and with whom?

About 2 weeks, with students and teachers.

About 1 month or more, with students, teachers and the community.

What the research says

MSC was the key qualitative method used in the evaluation of 'Teaching Teachers for the Future', a national ICT innovation project in thirty nine university pre-service teacher education courses (2012). Research found that 'Teaching Teachers for the Future' project embedding of ICTE within curriculum method units and assessment increased the capacities of participating teacher educators and that pre-service teachers increased their confidence in using ICT in the classroom, and facilitating student use of ICT. Using MSC at each site as part of the evaluation strategy created detailed accounts of the different ways that the universities had achieved this result.

Sustainability Victoria used MSC with schools to evaluate the impacts in each site of participating in the 'Resource Smart initiative' (2009). The overwhelming finding was that schools wanted more time to learn, network and share their experiences, challenges and solutions – with other schools and with service partners. The MSC stories also identified the value of having an overarching framework for their sustainability activities, and how engagement and motivation was enhanced through student initiated projects, teachers as ‘motivators,’ celebrations and local networking.

Resources

Examples of MSC stories can be found in the resources listed below:

Return to top of page