Causality, contribution and effect
It’s important to use evidence to explore why changes do or don’t take place. There will always be a number of possible explanations or theories for what has happened. Sometimes it might look like a new approach has had an effect, when in fact something else is causing the observed changes.
"Whenever a theory appears to you as the only possible one, take this as a sign that you have neither understood the theory nor the problem which it was intended to solve."
Karl Popper, philosopher of science
As human beings, we sometimes have a tendency to take credit when things go well but point the finger at factors outside of our control when things don’t go as we had hoped. This is what psychologists refer to as the self-serving bias, and it’s easy to fall into it without noticing. Follow the link below to cognitive bias for more information.
Example: The school has introduced some new literacy teaching strategies as a whole school initiative. NAPLAN reading scores improved in Year 3, but not in Year 5.
- Does this mean the strategies are not effective for older students?
- Or that the strategies are effective for all students but just weren’t implemented properly in Stage 3?
- Or is there a particular cohort of students in Year 5 this year that have always had difficulty with reading?
Some of these explanations may be able to co-exist, while others may contradict each other. Being able to figure out which explanations are valid and plausible helps to increase evaluation reliability.
Supporting evidence
Evidence that supports our theories is often easier to find. Our brains are naturally geared to find connections that reinforce the way we already see things. This is what psychologists call confirmation bias, and it’s an important one to be aware of.
Example: My colleagues and I are teaching a unit of work differently this year. Two weeks into the unit I see a student’s mastery of one of the key concepts really take off. I conclude ‘This strategy really works!’.
In this scenario, I might be right – it’s possible that the strategy is a good one for this student and is making a real difference. However, it’s also possible that something entirely different has affected the student’s learning, or that he or she would have responded equally well to the way we taught it last year.
When we have a situation like this, it helps to be aware of our mental model and take it to its logical conclusion. What it would look like if this strategy was really working? What would be the other impacts and flow-on effects – not just for this student but for all my students? Then we can see if other data matches these flow-on effects.
Building a logic model is one technique that can help us map our intended flow of cause and effect and articulate the underlying assumptions. This allows us to better target which data to pay attention to.
Contradictory evidence
Evidence that contradicts a theory is hugely useful. Finding one example that challenges our assumptions can sometimes provide just as much insight, if not more, than 100 examples that confirm what we are already thinking.
Contradicting or falsifying a theory allows us to modify it or remove it from our thinking, helping us focus on explanations that can better account for what has happened.
The video below provides a good illustration of this. The closing line is particularly relevant:
'If we think that something is true, we should try as hard as we can to disprove it. Only then can we really get at the truth, and not fool ourselves.'
Can you solve this? 2-4-8 video
The following video shows people trying to guess a rule for numbers 2,4,8. It runs for 4:33 minutes.
[In this video the interviewer is outdoors in a park. At different times, he stops four different couples and one man on his own. He gives them the numbers 2-4-8 and asks them to work out his number rule. He asks them to propose three numbers of their own in a sequence that follows the rule and to tell him what his rule is. The video jumps around to show him asking all of the people the question before showing any of their responses.]
Interviewer: Name unknown
[A young man with a beard, blue shirt and jumper with sunglasses tucked into his shirt neck]
I’m gonna give you guys three numbers. A three number sequence and I have a rule in mind that these three numbers obey and I want you to try to figure out what that rule is.
But the way that you can get information is by proposing your own set of three numbers, to which will say yes that follows my rule or no it doesn’t follow my rule. And then you can propose what you think the rule is.
Is that fair?
Couple 1:
[A young man in a dark corduroy coat and a young woman with a white beanie and leopard print scarf.]
Yep
Interviewer:
Okay, so here are the three numbers: 2, 4, 8.
Couple 1:
Woman: [repeats] 2, 4, 8
Interviewer:
You don’t need to continue the sequence, you can propose a totally different sequence, whatever you want to propose and I will simply say yes or no that follows my rule.
Man:
[A middle-aged man in a baseball cap and denim jacket.]
2, 4, 8 … 16, 32
Couple 2:
[A young man in a grey sweatshirt and padded vest with a young woman with long brown hair, wearing a parka and a baseball cap with NY on it.]
16, 32 and 64
Interviewer:
Those also follow my rule.
Couple 2:
Okay
Interviewer:
What’s the rule?
Couple 1:
Ah, multiply by 2?
Interviewer:
That is not my rule.
Couple 2:
What?
Interviewer:
That’s not my rule. But you’re allowed, if you want, propose three other numbers.
Couple 1:
3, 6, 12
Couple 3:
[An older man wearing a baseball cap, glasses and a leather jacket with an older woman in a pink parka and white framed sunglasses.]
3, 6, 12 .. ah ha
Interviewer:
3, 6, 12 follows my rule
Couple 3:
10, 20, 40
Couple 1:
10, 20, 40
Interviewer:
That follows the rule
Couple 1:
I'm also multiplying by 2
Interviewer:
I know, I know what you are doing and yes it follows my rule, but no it’s not my rule
Man:
5, 10
Couple 1:
10, and 20
Interviewer:
Follows my rule
Man:
100
Couple 3:
200, 400
Interviewer:
Follows my rule
Man:
500, 1000, 2000
Interviewer:
Follows my rule
Man:
Want me to keep going?
Couple 1:
Do I just keep going … you going to tell me your rule …? [laughter]
Man:
Am I doing it the wrong way, am I approaching this the wrong way?
Interviewer:
You’re totally fine but you are approaching it the way most people approach it.
Think strategically about this, you want information, I have information, the point of the three numbers, right is to allow you to figure out what the rule is.
Couple 2:
Okay
Man:
Okay, I am going to give you the numbers that I don’t think fits the sequence and then see what you say. So I’ll say 2, 4, 7
Interviewer:
Fits my rule
Man:
So whatever I propose is right?
Couple 2:
So is your rule like, you can propose any number?
Couple 1:
So the rule’s anything we say is yes?
Interviewer:
No
Man:
Dammit
Interviewer:
But you are on the right track now. Hit me with three numbers.
Couple 1 - woman:
3, 6, 9
Interviewer:
Follows my rule
Couple 1 – man:
Oh, that didn’t follow my rule
Interviewer:
This is good right?
Couple 2:
5, 10, 15
Interviewer:
That follows my rule
Couple 2:
What? Oh. Really?
Interviewer:
Yeah
Couple 3:
I don’t believe this. [laughter]
Couple 2:
1, 2, 3 ?
Couple 1:
1, 2, 3?
Interviewer:
Follows the rule
Couple 2:
What about 7, 8, 9?
Interviewer:
Follows the [laughter], yes that follows the rule
Man:
8, 16, 39
Interviewer:
Fits the rule
Man:
Excellent.
Interviewer:
But we are no closer to the rule!! [laughter] I want you to get to the rule.
Couple 2:
How about 1, 7, 13
Interviewer:
Follows the rule
Couple 2:
What? 11, 12, 13
Interviewer:
Follows the rule
Couple 2 - woman:
How does this make sense? I don’t know how to do this.
Couple 2 – man:
10, 9, 8
Interviewer:
Does not follow the rule>
Couple 2:
10, 9, 8 does not.
Couple 2:
Oh, so is it all in ascending order?
Interviewer:
Booyah! Up top [interviewer and couple high-five]
Couple 2:
Okay
Interviewer:
Booyah! Up top [interviewer and couple high-five]
Interviewer:
Yes! First ones to get it. You guys nailed it, that’s the rule. That’s the rule! Numbers in increasing order.
Couple 1:
Ohhhhh
Interviewer:
Numbers in ascending order
Couple 4:
[An older man in bright blue wind jacket, grey hair and sunglasses with an older woman with auburn hair, grey wind jacket and a green scarf.]
Oh, so much easier than we thought.
Interviewer:
1, 2, 3, 4, 5, 6, 10, 15, 25. Doesn’t matter. Any numbers in ascending order.
I was inspired to make this video by the book ‘The black swan’, by Nassim Taleb. Now the black swan is a metaphor for the unknown and the unexpected. I mean, In the old world the theory was that all swans were white. So each instance of a white swan would make you think, yeah, that theory is pretty good. But the point is you can never prove a theory true and in fact when people found Australia they realised that there were black swans. What was interesting for me was that everyone I spoke to came up with a rule very early on and then only proposed numbers that fit with that rule they were thinking.
Interviewer:
I was looking for you guys to propose a set of numbers that didn’t follow your rule and didn’t follow your rule.
Couple 4:
Oh right
Interviewer:
I was looking for you guys not to try to confirm what you believe.
You are always asking something where you expect the answer to be yes, right?
Couple 2:
Yeah
Interviewer:
Like you’re trying to get at it
Couple 2:
But instead you want to get the ‘no’.
Interviewer:
You want to get the ‘no’ because that’s much more informational for you than the ‘yes’. Like if everything is given a yes …
Couple 2:
No, that is true, that is really true once you say that.
Interviewer:
That is so important about the scientific method. We set out to disprove our theories. And it’s when we can’t disprove them that we say, ‘this must be getting at something really true about our reality’. So I think we should do that in all aspects of our lives. If you think that something is true you should try as hard as you can to disprove it. Only then can you really get at the truth and not fool yourself.
[End of transcript]
Experimental designs
To properly test a theory, it?s best to have experimental conditions. The purest form of evidence, from a scientific perspective, comes from isolating and testing just one theory or proposition, controlling for all other factors.
In educational settings, designs like this require careful planning and consideration of numerous methodological and ethical constraints.
Experimental or quasi-experimental designs in education often involve focusing on two groups of similar students who are all working towards the same learning goals.
One group of students is taught using different teaching strategies or in different learning environments. Later differences in their learning outcomes are then taken to be the effect of the ?intervention?, as there are no other major factors to account for.
- In a Randomised Controlled Trial (RCT), students have equal chance of being allocated to the different teaching strategies or learning environments, where one of them is the ?control?, which is the base state or point of comparison for the others.
- In a quasi-experimental design, the evaluators are trying to find a way to simulate experimental conditions. For example, there might be a group of schools who are all using ?teaching strategy X? that they believe is the best way to go. The evaluators might try to simulate a control group by looking at comparable data for students at similar schools where ?strategy X? is not being used.
Experimental conditions like this might not often occur within one school, particularly where whole-school strategies affect all students at the same time. Trials like this also tend to be quite complicated to organise, requiring large sample sizes and complex statistical analyses.
In practice, it can also be quite difficult to find a suitable comparison group, or to keep the teaching practices consistent within each group in a trial. Ethical considerations also come into play if we know that we are withholding access to something of potential benefit. These are the kind of issues that professional evaluators wrestle with when trying to optimise their evaluation design.
However, within a school, there are occasions where we might want to draw on the research evidence from experimental designs. There are also times where it might be possible to use some of the principles from experimental design into our innovation processes.
Example: My school has funds available for new furniture and a refit of two new classrooms. I have four stage 3 classes, and am keen to understand the impact of the new learning environments on student engagement and outcomes. Two of the stage 3 teachers have been immersing themselves in the research on the link between pedagogy, learning spaces and future-focused learning. Rather than giving these two teachers the new classrooms, I structure my classes in stage three using a 2x2 design:
| Teacher is traditional and doesn't know this research well | Teacher loves to innovate and is immersed in this research | |
|---|---|---|
| New classroom environment | Mr Davis | Mrs Tang |
| Traditional classroom environment | Mr Gianopolis | Mrs Allen |
Read a description of the table above.