Using data in teaching practice CESE: You're listening to CESE's What Works Best podcast. In today's episode, Ben North, a public high school head teacher, talks to high school deputy principals Karyn O'Brien and Daniel French. Ben, Karyn and Daniel are currently on secondment at CESE, and they sat down to talk about the use of data in professional practice. Ben North: Okay, so we're here to talk about what works best, and one of the seven key themes from CESE's What Works Best publication is data and the use of data by teachers and school leaders. So, what I really want to know is, what can teachers and school leaders gain from using data in their professional practice? Dan French: Yeah Ben, firstly I think it's important for us to acknowledge that data is more than just NAPLAN scores. This is just one of the many sources of data that teachers and school leaders can use. So, you look at classrooms around the state, and we've got thousands of teachers that collect and act upon formative data with their students every single day. This too is a really valuable source of data. So, ultimately data includes all those forms which allow teachers to make informed judgements on where they might go next with their lessons, or with their students, or the other question of, how well did we do that? So, teachers can gain really significant information about their students with their classes, in order to provide tailored support for every student. The flipside of that with school leaders, is that school leaders can use that data that they've collected or that they've analysed in their practice for their strategic leadership across the school to identify areas of need. And, to implement evidence informed practices and effective evaluation to monitor the collective effort of everyone in the school. Ben North: Okay, all right, so you said evidence based practice? What does that mean? Dan French: Yeah, I think that's a really good question Ben. So, evidence informed practices or evidence based practices are those that have been tried and tested to show improvement in student learning and school improvement. So, when we identify a need, and whether that's at an individual student level, or at a whole school level, we have a suite of available strategies at our disposal which we can implement. So, the challenge is choosing the strategy or the practice, which is best going to address these areas of need. So, because we know that not all practices and strategies are created equal, again the challenge is to choose the right one. And, this really does highlight the importance of teachers as researchers. So, finding the most effective practice for that particular need is the challenge for teachers, and for school leaders, but the great news, and the good news is that we have an evergrowing and quite a large repository of an evidence base available to us that we can draw upon. So, for instance, we know what works best for literacy and numeracy, and it's a matter of going in and delving into that detail, to work out how it applies to our particular identified need. Ben North: So, I guess I'm going to ask a hard question here. If data is all about evidence, what is the evidence that using data makes a difference for students? Dan French: Well, if you look at a lot of the research that's been done, particularly over the last 20 years, you look at Professor Hopkins out of the UK, he talks about the fact that the effect size of teachers using data is zero-point-nine. Now, considering the average growth of students is zero-point-two to zero-point-four, that's significant. If you look at the work of Wiliam, he talks about the fact that by teachers using the data of their students, you're looking at eight additional months of growth over 12 months. And, then looking at Helen Timperley, for example, she talks about when you're using writing growth, when you're looking at writing and using writing data that you can have six times the average growth by looking at that, and about three-and-a-half times the average growth, when you're looking at data around reading. So, that kind of research is pretty significant to show the impact that teachers and leadership groups using data can have on student growth and student learning. Ben North: So, I mean, that's more than just having a few graphs and numbers, that's then using the data to turn it into knowledge I guess. Dan French: Yeah, that's right, and I think what schools often struggle with is it's one thing to be able to have the data there in front of them, and it's one thing to analyse it, but what do we do with it next? And, that's where you've got CESE with publications, What Works Best, these six effective practices in high growth schools, you've got the effective practices toolkit and that website. That’s providing the next step for schools, so that they know, we've got this set of data, both internal and external data, what do we do next? What can we do to improve the learning of our students? And, those resources help to provide that. Karyn O’Brien: And, Daniel just mentioned using internal and external data as being important in making decisions, so in my work with schools this year I'm making sure that they are looking at the data that they have in Scout reports that reflect performance in NAPLAN and HSC. But, also for those schools that can participate in the Tell Them From Me program, also looking at that data in correlation with the Scout data, because I think it's important to look at the whole child. And, we all have opened up reports in SMART or Scout and had a look at an individual student and go, "Well, I know why that student went backwards in year nine writing and numeracy, because this has happened in their home life." And, I think that we can do that with all the students in our schools, and that that teacher judgement or analysis of the data is really important, not just looking at the raw figures. Ben North: So essentially, it's helping us know students better. Karyn O’Brien: That's right. Ben North: All right, something I see a lot of on places like Twitter is often statement along the lines that, "Data and evidence, it undermines a teacher's professional judgement. It's done by academics and researchers who don't really know anything about teaching apparently, and that it's just it's not the same as being there in the moment as a teacher." What do you say to a statement like that? Dan French: Well, I think that ultimately data and evidence is what should inform our teacher judgements. If you're not using data or evidence to inform teacher judgements, then what are you using? If you're not using those methods, then you're really going off a gut feel and we know that gut feels, while they might be good in the moment, don't necessarily lead to practices which can enhance student outcomes. So, I would say that sure, you're always - and I think this is part of the issue is that people can often confuse data as being externalised standardised tests, and that that's purely what they think data is. Whereas, data's more than that, and we talk about Pasi Sahlberg with small data. If it is that you're forming a judgment on a student just through oral questioning, you're building up a bank of a data as a result of that. And, by bringing that sort of data, the small data and the big data together, that's when you can start informing what judgements you're making about where you're going to next, or making judgements around, well how effective was that lesson today? So yeah, I think that's the importance of using data. Karyn O’Brien: Yeah, so I agree with Daniel, there's this cynicism around the use of data, whether it's used for accountability, and whether there's going to be league tables, and schools being accountable for that. But, I think once you have that conversation with teachers, and you look at an individual student data, and in Scout one of the favourite reports we have is a report called Individual Student Growth, for each of the domains of NAPLAN. And, you can see a student's growth from year three to five, from five to seven, seven to nine. And, it's really powerful to show that individual student's report, and teachers really like that, because you're focusing on that child. And, you can look at that student and think, well they had a difficult year, their parents went through a marriage breakup, that might have impacted on them. Or, they got some additional support, they were engaged in a reading program at school, and look at the impact that's made to that student's performance. I think if you always bring it down to the individual student and what it can do for them, the importance of looking at the data, and using it to inform where you go to next, you will always be able to win teachers over. Ben North: And, I guess that's what we're here for. Karyn O’Brien: That's right. Ben North: Is to make a difference for students, and to help them as much as we can. So, I mean for a lot of teachers I guess, words like mean and median and mode, they might not have really heard it since year eight maths. But, there's more to it than that I guess in understanding and explaining, it's something that we can kind of learn about. How do you go about explaining those concepts to teachers in a relevant, meaningful way? Dan French: Yeah, well I mean one of the things that we talk about with Scout is confidence intervals with the value-added model. And, with the confidence interval, you essentially have a margin of error, so we think that this is our best estimate of where students are; however, it could fall within this range. So, the way that I generally explain that is I say, "Well, it's a Friday afternoon, you've just had - it's been a windy day outside, you've got kids that are a little bit crazy perhaps. You might have had a parent complain about who knows what, and all you're worried about is getting to the coffee shop in the afternoon, just to try and -" Karyn O’Brien: Coffee shop? Dan French: Coffee shop, generally coffee shop. Karyn O’Brien: Okay. Ben North: Yeah, coffee shop, sure. Dan French: In order to maybe have a break and just wind down. And I say, "Look, I reckon all of you here could probably guess within a minute or two how long it would take to get to your local coffee shop. And, it might be that along the way, you're so tired that you trip over and that takes you an extra 30 seconds, and that's where your margin of error sort of comes into play." Ben North: Before you come to the coffee shop, this is? Dan French: Yeah, you're left on the way to the coffee shop. Ben North: Yeah, on my way, all right, got it. Dan French: And, that's a nice little way of sort of helping people understand the fact that with a value-added model for instance, yes this is our best estimate of where students are, but over a period of two years, there's a lot of variables that can happen. And, that's why we say there's a margin of error with that. Karyn O’Brien: Yeah, and in terms of a mean or an average, we use the example of the results from a running race, and majority of students get 27, 28, 30 seconds, but there's an 82 that Trevor got. And, so you look at that data set and you go, now was that 82 an error - a data entry error, or did Trevor really run the race at that slow a pace? And, maybe it's that Trevor has some special needs and he has a physical disability, he wanted to be involved in the race, and he ran it, but he ran it - he was an outlier in terms of the results. So, doing an average of those results isn't going to give you a true indication of the measure of central tendency, so it would be better then, to have a look at the median, where 50% are below and 50% are above, and maybe that comes out at 28, which is a much better indicator of the measure of central tendency for that running race. Ben North: And, that works? Karyn O’Brien: Yeah. Ben North: While it's often explaining to teachers about a trend, which is essentially the interpret - Karyn O’Brien: Interpreting. Ben North: Interpreting the data, and essentially, what direction is it heading? Is it heading upwards? Is it down? Has it moved? Often when we talk about trends in fashion, in that wearing certain items of clothing might be on the way in, and that certain people are wearing cargo pants or high wasted jeans or short shorts. Dan French: Or Van shoes that are back in at the moment. Ben North: Yeah. Karyn O’Brien: Or blue suits. Ben North: And, blue suits, and whether or not they are on the way out, and the couple of teachers - Karyn O’Brien: No, I think they're very on trend. Ben North: Well, I don't know, the couple of teachers wearing blue suits, maybe they're on the way out. But yeah, in the sort of are they on the way in, or are they on the way out, or are you Warrick Capper and you're still wearing short shorts, regardless of what decade it is. I mean, things like that, they do make these terms a lot more approachable. Dan French: And, it breaks down that fear factor associated with the term, right. Ben North: Yeah, it's being able to get onto that ladder, and feel confident in what you're doing. Karyn O’Brien: That's right. Ben North: All right, well to help inform teachers, what are some of the data and evidence sources that we have available at our fingertips for people working in the Department of Education in New South Wales? Karyn O’Brien: Well, Scout is the new business intelligence for education, and it's a cloud based product that draws information from a range of sources. So, we have our ERN data from schools, information about students and their parents and carers. It also draws data from NESA and the MyPL for teachers, and also has information on local government areas, on health and communities et cetera. So, it has a wide range of sources of information that can create reports that are relevant to schools. The other one is Tell Them From Me, which is a survey that's conducted once or twice a year, with student groups as well as parents and teachers, so that you can feedback about how those groups are feeling about their school community. Ben North: So, it's like wellbeing and …? Karyn O’Brien: Wellbeing, student engagement, a sense of belonging, how they feel about academic rigour, intellectual challenge, all those questions are in Tell Them From Me. And, from a teaching point of view, it's about how well they think that they're going as a teacher, and what impact they believe they're having, and how supported they feel at a school level. And, the parents survey is asking them about how well they think their school or their schools catering for their child's needs and if they're happy with that. Ben North: Sure, okay, Daniel? Dan French: Yeah, I mean you've got programs such as SMART, which is obviously fairly similar to Scout in a lot of ways, providing NAPLAN and HSC data that schools can use to even get down to an item analysis level. It's like an individual questioning. Ben North: Like a single question in an exam? Dan French: That's right, so that will sort of enable teachers and leadership groups in schools to work out exactly where their students need to go next. You've also got programs such as the results analysis package through NESA, which provide a finer breakdown on HSC performance of students. So, that's obviously really applicable to secondary schools. And, then we've also got a wealth of data and information through our CESE website, and as I said before, not only does it give you a lot of statistics around the fact, for example, in 2016 we had 31% of teachers in the state with 21 years of experience or more in teaching. There's all that sort of information that can sort of support where schools are going. Whether it's their staff profile, or whether it's in relation to trends with student attendance. So, really setting up that overall context about what's going on within our system, but then looking at how that might reflect in our school. Karyn O’Brien: And, I guess there's very rich internal sources of data as well, so. Ben North: Yeah, I was going to ask, what's within a school that a principal or a teacher might already have there and might even not just be within a school, but within someone's classroom? Karyn O’Brien: Well, at a school level, we would conduct parent surveys twice a year, we would rely on teachers to provide feedback on classroom practice and formative assessment et cetera. So, that's very valuable data. Dan French: Yeah, I mean Pasi Sahlberg talks about it as small data, doesn't he, to the point where even if you're just building up a narrative around a particular student, so understanding a student. Not necessarily doing a survey with them, or even doing a formal assessment, just by understanding stories and narratives around individual students, you're essentially collecting data. Sure, it's more in an informal sense, but it's still helping you to form a picture around that particular student or group of students. Karyn O’Brien: Yes, and schools have systems in place like Central or what's another one? Academy, or - Ben North: Canvas. Karyn O’Brien: Canvas, yeah, where they've got information, often wellbeing information on students that they can use as evidence for wellbeing practices within schools. Dan French: And, it's really through collaborative practice within schools. Again, part of the school excellence framework, that when that data is shared between teachers, between departments, between stages, that's when we see impact on student outcomes. Karyn O’Brien: That's right, yeah. Ben North: All right, probably something that I really love about teaching is that ability to know that you've made a difference, but we don't always know that we have made a difference. How can we use those sources of information to tell what kind of impact that we've made? Dan French: Yeah, I think one really good example is pre, and post-testing, so often you'll find that a lot of teachers and a lot of schools really focus on the post-tests, so it might be summative assessment at the end of a piece of work for instance. But, I think that the pre-test is one that's often overlooked, but that is a really valuable source of data and information. Because, not only is it going to provide you with an avenue of knowing where to go next with your students, based on where they currently are for a new unit of work as an example, but what it also does is gives you some baseline data. So, if you provide a pre-test to students at the beginning of a unit, they go ahead and they complete that, you've got the results, set them to one side after you've decided where you might go to next with the unit. And, then given them almost the same test at the end of the unit, or assessment at the end of the unit, and then comparing the two. Because, what you'll find is you'll be able to actually look at the growth over that time. So, if you want to measure your impact, rather than thinking purely about performance through summative assessment at the end, what you're actually doing is being able to measure okay, in week one they were at this point, when they did their pre-test. Now, it's week six, they're at this point, they've actually grown all this, and been able to show and demonstrate all these new skills and mastery of content. So, that's a really effective, very simple, very simple, and the pre-test and the post-test don't have to be extensive, it might be it could be as simple as a multiple choice, or it could be as simple as asking them to do a short writing piece around what they may or may not know. So, I think that that's a good way of being able to measure that impact that you talk about. Ben North: Yeah, I mean what a powerful thing, not just for teachers to be able to see that impact, but I imagine for students to see, and for parents and family to see how they've grown. Dan French: Yeah, and again it breaks down the silo or looking at students' education purely from the perspective of performance, and actually looks at value-added and growth as an approach. Ben North: Sure, as student growth and progression. Dan French: Yeah. Ben North: Okay, Karyn I was wondering if you could tell me a bit of a story I guess, or an example of something that you've done within your school context, something you may have either participated in or led, that might be a good example of what we're talking about today? Karyn O’Brien: I can give you a couple of examples of how we've used Tell Them From Me data from students. So, we jumped on board Tell Them From Me fairly early, we participated in the pilot in 2013. And, I think data in 2014 showed us that - there's one question about student engagement and participation in extracurricular activities. We thought everyone participated, we've got such a huge band program, but in fact the figures were quite surprisingly low for us. So, we decided to then look at what we provided and to ask staff, is there anything that you'd like to offer to engage students in co-curricular activities? Because, there's a strong correlation between participating in co-curricular activities and academic achievement I think. And, so teachers volunteered. We started up a creative club, which focused on creative writing and graphic design, and they met in the library in the afternoon. We started up a run club. I didn't participate in that, but it had teachers and students who'd meet at 6:30 on a Tuesday morning and they'd go for a run, and they created a great sense of community there, and that they'd started to participate in colour runs on the weekends et cetera. So, we went beyond the traditional debating and public speaking and band, and we expanded our dance program, and we also had a greater range of things for students to participate in. And, that was a direct response to looking at the Tell Them From Me data. Another example is, we looked at student engagement in their learning on Tell Them From Me, and we noticed a dip in year nine, and it's a fairly well recognised dip that year nine students go through. But, we also found that there was a dip in year ten as well, which worried us. And, then we started to think about with the demise of the school certificate and the RoSA being less important as a credential for students and families; we then thought about having a middle and a senior school. So, we had a very defined middle school of seven to nine, and we included year ten as part of the senior school, including changing the senior uniform and the expectations of them in terms of preparing for the HSC years in year 11 and 12. And, it made a big difference to their engagement, and we could see evidence in the Tell Them From Me data in subsequent years, that that had made a difference to their engagement. Ben North: And, I'm sure that would have made a difference that you could have seen as well. Karyn O’Brien: Yes. Ben North: All right, look this has been really interesting, and I think this is something that even though I guess I've been involved with it within my own teaching practice, and also within school leadership, it's something that I would always want to find out more and get better at. And, that's what we want to do as teachers, like students, we always want to get better at things. So, how can I learn more? If I'm a teacher in a school, how can I learn more about making using of data and evidence in my teaching practice? Karyn O’Brien: Well, when I went for this job at CESE, people said to me, "Oh, you must be a data expert to be able to work at CESE and with the Scout team." I said, "No, in fact I'm a data dummy, which makes me a very good candidate for this job, because I'm going out to school teachers and school leaders who are possibly the same level of data understanding as I do." And, for you both, it rolls off your tongue, but for me, I've needed a bit of work. So, in the last six months, we've been working with a colleague to develop a resource called Using Data with Confidence. And, by helping to assist the development of that product with a school focus and a data dummy focus, it's been really successful. Now, I can explain the charts in Scout very confidently, I can explain the confidence interval. I now understand what a central measure of tendency is, and what the difference between a mode, median and the other one is. Oh, what is it? Mode, median - Dan French: Mode, median and mean. Karyn O’Brien: Mean. Ben North: Mean - average. Karyn O’Brien: Thank you, mean average. I do know it, I'm just under pressure here. Ben North: I'm sure. Karyn O’Brien: But, I look at data differently now and so I can read a chart, I can understand it, there's also a tool as part of that resource Using Data with Confidence, which allows you to calculate a class set of assessments, and it will spit out a standard deviation and a box plot; I can explain what one of those is. And, so I understand the importance of using it, and I have the language to do that, which I didn't have at the start of the year. That resource is going to be invaluable to teachers as it gets rolled out from CESE very soon. Dan French: And, particularly because the teaching standard, so teaching standard three and also five, expect teachers to be able to use student assessment data. And, when you look at the school excellence framework with data skills and use, one of the excelling statements talks about teachers having a sophisticated understanding of data literacy concepts. And, yet part of the issue, and a lot of teachers say this to us, is that it's not something that they ever did at university, there's no training for data skills and data analysis. So, Using Data with Confidence is going to really help fill that void, and help pick up the slack that's there, so that teachers feel much more confident, because people are scared of data sometimes. Ben North: Yeah, I was going to ask that, like I mean I certainly know amongst - I'm sure there's plenty of English teachers like me, and there'd be some primary teachers or PE or creative arts that might feel that that's why they went into their subject areas, was to not to do mathematics, or not do numbers. Karyn O’Brien: But, I find even with maths teachers, they go, "I'm a mathematician, not a statistician." So, sometimes they can struggle with the concept. Dan French: Yeah, and it really is breaking down that perception that data is maths, and it actually isn't, it's evidence based practices, which every teacher needs to be involved in. Karyn O’Brien: Well Daniel, you explain it as the evidence life cycle, isn't it, the data life cycle? Dan French: Information life cycle, yeah. Karyn O’Brien: Information life cycle, do you want to explain that? Dan French: Yeah, which is basically taking data and turning that into information, which you can actually understand, and then analysing that information to then building up knowledge and insight. Once you've then built up that knowledge and insight, actually disseminating that to others, so building the capacity of others so that we've got collective wisdom. And, we've got a shared value consensus around yep, we think that year five reading is our greatest area of need, let's go towards improving that area, so. Ben North: So, making good education decisions and teaching decisions. Dan French: Yeah, yeah. Karyn O’Brien: I think Daniel mentioned the teaching standards as well, and remember there's four levels of those, and it starts with the graduate teacher. So, I think that more needs to be done with graduate teachers in understanding data and how to use it in a classroom setting, and a school setting, prior to them becoming fully fledged teachers. Ben North: Absolutely, and I guess that's then a journey that teachers can go on, and there's obviously we start with being graduates, we work towards proficiency, but I imagine for a lot of school leavers and those that may be in that sort of highly accomplished or lead zone, then those higher levels of accreditation would be important as well. Dan French: Yeah, because you look at the lead standards or highly accomplished. What's expected of you at those standards is that you are building the capacity of everyone in your school. And, not only that, it's if you're looking at undertaking higher accreditation, you are going to need to use data in order to demonstrate with evidence that you've actually had an impact on student learning. Karyn O’Brien: And, teacher learning. Dan French: And, teacher learning, that's right. So, it really is it really does sort of apply to all levels of schooling, all levels of teaching, both at a classroom level, and at a strategic level. So, the importance can't be sort of understated I suppose. Ben North: And, I think too, from teachers I've worked with in all stages of their career too, that confidence and self-efficacy is so important, because in doing that teachers I think, and certainly in my own journey and understanding some of these aspects, is you get a lot more confident. You also get a greater sense of agency in understanding it. So, rather than feeling like data is being done to you by somebody else, you're the one working through it, analysing it, and understanding it to draw conclusions for yourself. Dan French: Yeah, and it also breaks through, that it's not just that thing that you do one time in the year. Karyn O’Brien: Yeah, summative thing. Dan French: You don't just walk in and analyse NAPLAN results, oh that's data done for the year. It's something that's ongoing, and as you've said, because of that confidence, being able to do it once, you're able to do it multiple times throughout the point of the learning journey. Ben North: Oh, and for me too, I mean, using those practices are things that I've gotten better at over time, but I still distinctly remember the first time, and I think I was a secondyear teacher. I did some formative assessment, essentially a pre-test, or pre-assessment with a really capable English class in my first school in the western suburbs. And by doing that, was able to realise that most of the students had mastery over about three weeks of the course's content already out of six, so we had about half the unit done. And, that meant that rather than just simply teaching the same thing again that they had an understanding of, that we could just compress that down, focus on areas that students might be weak at, but it gave us so much space to do engaging extension type work that helped push their learning and growing. And, that stuck with me, that's become one of my teaching practices thereon in. I think too, it's hard as teachers and school leaders, we do get caught up in the very difficult challenge of essentially business as usual, that teaching and learning keeps happening every day, and that within our class, and that we might have six periods and 30 students in a class. We're just overwhelmed by everything that is happening, but I find it very powerful to just be able to take that moment to step back and reflect on the information, and realise those kinds of things. Karyn O’Brien: And, teachers are really busy people, and I like the fact that in the Using Data with Confidence resource pack, there's this educator calculator, where it's formed from an Excel spreadsheet, where they can just enter their own data. They don't have to do the calculations behind it, and I think they will really enjoy doing that, and seeing the results for a class assessment or an across the year assessment that may have been done within a faculty. And, by giving teachers those tools that are going to make their life easier, then they're going to have an understanding of data, and be able to use the information that they gained from it to help students improve their learning. Dan French: Yeah, and really and like a practical measure - Karyn O’Brien: Yeah, that's right. Dan French: - is really setting some time aside as well. You give yourself a couple of hours to work through your data. Don't sort of set yourself up so that you're going to be interrupted all the time, because you lose your train of thought. Really putting aside a couple of hours, whether it's at the end of the day or before school, before people start moving in. Because, it's at those points that you really start to pick up on the trends and what's significant, what's not. So, that's a practical thing that people can do, is really put those couple of hours aside to go through that. Ben North: Ben, Karyn, thanks for joining me today, and thanks for listening. CESE: Thanks for listening to CESE's What Works Best podcast. Don't forget you can subscribe to CESE on iTunes. For more great content, head to our website at cese-dot-nswdot-gov-dot-au. You can also follow us on Twitter and SoundCloud. Thanks for listening.