Causality, contribution and effect
It’s important to use evidence to explore why changes do or don’t take place. There will always be a number of possible explanations or theories for what has happened. Sometimes it might look like a new approach has had an effect, when in fact something else is causing the observed changes.
"Whenever a theory appears to you as the only possible one, take this as a sign that you have neither understood the theory nor the problem which it was intended to solve."
Karl Popper, philosopher of science
As human beings, we sometimes have a tendency to take credit when things go well but point the finger at factors outside of our control when things don’t go as we had hoped. This is what psychologists refer to as the self-serving bias, and it’s easy to fall into it without noticing. Follow the link below to cognitive bias for more information.
Example: The school has introduced some new literacy teaching strategies as a whole school initiative. NAPLAN reading scores improved in Year 3, but not in Year 5.
- Does this mean the strategies are not effective for older students?
- Or that the strategies are effective for all students but just weren’t implemented properly in Stage 3?
- Or is there a particular cohort of students in Year 5 this year that have always had difficulty with reading?
Some of these explanations may be able to co-exist, while others may contradict each other. Being able to figure out which explanations are valid and plausible helps to increase evaluation reliability.
Evidence that supports our theories is often easier to find. Our brains are naturally geared to find connections that reinforce the way we already see things. This is what psychologists call confirmation bias, and it’s an important one to be aware of.
Example: My colleagues and I are teaching a unit of work differently this year. Two weeks into the unit I see a student’s mastery of one of the key concepts really take off. I conclude ‘This strategy really works!’.
In this scenario, I might be right – it’s possible that the strategy is a good one for this student and is making a real difference. However, it’s also possible that something entirely different has affected the student’s learning, or that he or she would have responded equally well to the way we taught it last year.
When we have a situation like this, it helps to be aware of our mental model and take it to its logical conclusion. What it would look like if this strategy was really working? What would be the other impacts and flow-on effects – not just for this student but for all my students? Then we can see if other data matches these flow-on effects.
Building a logic model is one technique that can help us map our intended flow of cause and effect and articulate the underlying assumptions. This allows us to better target which data to pay attention to.
Evidence that contradicts a theory is hugely useful. Finding one example that challenges our assumptions can sometimes provide just as much insight, if not more, than 100 examples that confirm what we are already thinking.
Contradicting or falsifying a theory allows us to modify it or remove it from our thinking, helping us focus on explanations that can better account for what has happened.
The video below provides a good illustration of this. The closing line is particularly relevant:
'If we think that something is true, we should try as hard as we can to disprove it. Only then can we really get at the truth, and not fool ourselves.'
Can you solve this? 2-4-8 video
The following video shows people trying to guess a rule for numbers 2,4,8. It runs for 4:33 minutes.
To properly test a theory, it?s best to have experimental conditions. The purest form of evidence, from a scientific perspective, comes from isolating and testing just one theory or proposition, controlling for all other factors.
In educational settings, designs like this require careful planning and consideration of numerous methodological and ethical constraints.
Experimental or quasi-experimental designs in education often involve focusing on two groups of similar students who are all working towards the same learning goals.
One group of students is taught using different teaching strategies or in different learning environments. Later differences in their learning outcomes are then taken to be the effect of the ?intervention?, as there are no other major factors to account for.
- In a Randomised Controlled Trial (RCT), students have equal chance of being allocated to the different teaching strategies or learning environments, where one of them is the ?control?, which is the base state or point of comparison for the others.
- In a quasi-experimental design, the evaluators are trying to find a way to simulate experimental conditions. For example, there might be a group of schools who are all using ?teaching strategy X? that they believe is the best way to go. The evaluators might try to simulate a control group by looking at comparable data for students at similar schools where ?strategy X? is not being used.
Experimental conditions like this might not often occur within one school, particularly where whole-school strategies affect all students at the same time. Trials like this also tend to be quite complicated to organise, requiring large sample sizes and complex statistical analyses.
In practice, it can also be quite difficult to find a suitable comparison group, or to keep the teaching practices consistent within each group in a trial. Ethical considerations also come into play if we know that we are withholding access to something of potential benefit. These are the kind of issues that professional evaluators wrestle with when trying to optimise their evaluation design.
However, within a school, there are occasions where we might want to draw on the research evidence from experimental designs. There are also times where it might be possible to use some of the principles from experimental design into our innovation processes.
Example: My school has funds available for new furniture and a refit of two new classrooms. I have four stage 3 classes, and am keen to understand the impact of the new learning environments on student engagement and outcomes. Two of the stage 3 teachers have been immersing themselves in the research on the link between pedagogy, learning spaces and future-focused learning. Rather than giving these two teachers the new classrooms, I structure my classes in stage three using a 2x2 design:
|Teacher is traditional and doesn't know this research well||Teacher loves to innovate and is immersed in this research|
|New classroom environment||Mr Davis||Mrs Tang|
|Traditional classroom environment||Mr Gianopolis||Mrs Allen|
Read a description of the table above.