Disciplines of evaluative thinking
Evaluative thinking is part mindset, part skill set. The values underpinning evaluative thinking are:
- curiosity - where we are willing to ask questions that might not have easy answers
- ambition - as we continually work to improve our processes and our impact
- courage - as we question and challenge existing practice in ourselves and others
- humility - recognising that insight can come from a wide range of sources and there is always more to learn
- honesty - where we do not seek to bend the facts to suit ourselves or cover up 'inconvenient truths'.
Discipline 1 - Suspending judgement
Most people are susceptible to optimism bias when they introduce a change and have high hopes for the impact it will have. Sometimes this can start to run at odds to evaluative thinking.
In the words of Earl and Timperley:
'Innovations may … be the ‘brainchild’ of individuals or groups who are strong advocates for particular directions, follow their instincts and … have little interest in alternative perspectives. Evaluative thinking is not consistent with only considering positive evidence or with approaches and directions that are already decided or entrenched to produce ‘good news’.'
Evaluative thinking for successful education innovation, 2015, page 34. For more, read the full text.
Suspending judgement is not about being a cynic, or a fence-sitter who is unable to make a decision. It is about:
- being open to the possibility that things are not as we would like them to be
- being open to alternative explanations about why something might be the case
- knowing when the evidence is not adequate to make a conclusion
- appreciating the value of a well-considered 'I don't know', rather than feeling compelled to jump to a false positive or false negative
- being willing to challenge taken-for-granted ideas, using evidence.
Discipline 2 - Asking important questions
Effective evaluation starts with good questions. We need to focus on questions that will shine a light on important matters and help to strengthen our impact.
Three helpful lines of inquiry that engage our evaluative thinking include:
- why are we doing this? Questions about our rationale, goal alignment, problem definition and our assumptions about cause and effect. For more information, go to logic modelling.
- is it working? Questions about who is being affected by the actions, in what way, under what circumstances. For more information, go to outcome evaluation.
- what are we learning? Recognising that we can learn just as much from a disappointment or a failure as we can from success.
Other questions may well follow, including questions about equity, value for money and sustainability.
Discipline 3 - Using existing evidence well
There is a great deal of data available to schools and it is important to get the best value possible from existing data before collecting more. The skills of working with data include:
- awareness - knowing what useful data exists where, and how to access it
- analysis - knowing how to handle and interpret different types of data (for example, qualitative and quantitative)
- synthesis - being able to critically analyse data from a variety of sources to distil the salient points
- integration - using evidence collected throughout the year to inform decision making, actions and reflection.
This resource hub contains general guidance on turning data into evidence, as well as detailed advice on getting the most out of specific data sources available both at the school level and system-wide.
Discipline 4 - Strengthening our evidence base
Every dataset and data type has its own advantages and limitations. There is no single source of truth that tells us everything about everything. Being aware of the specific strengths and weaknesses of our data helps us to avoid underestimating or overstating the strength of the evidence base we are working with.
Any additional data collection needs to be undertaken carefully, and with a clear purpose. When collecting additional data, it is best to focus on addressing our 'blind spots'.
This resource hub provides suggestions for collecting additional data, including surveys, focus groups and interviews and observation. The best approach is to incorporate this kind of data collection into the implementation of a new initiative, as part of a broader commitment to continuous improvement. Planning the nature and timing of information gathering from the outset will help the evaluation produce useful data to inform later decision making. For more detail, read turning data into evidence and setting the scope of an evaluation.