Long reads

How well are we learning from NAPLAN?

How well are we learning from NAPLAN?

[Image ©Shutterstock/ DGLimages]

Some Australian schools and school systems have seen greater improvements in NAPLAN results than others. How well do we understand where improvements are occurring and why?

There has been no higher priority in Australian school education over the past two decades than the improvement of students’ literacy and numeracy levels. This has been the goal of a range of state, territory and Australian government initiatives and programs, including national partnership agreements, the National Assessment Program-Literacy and Numeracy (NAPLAN) and the My School website. Other initiatives to close gaps for Indigenous and disadvantaged students and to improve outcomes in the early years of school have shared the objective of increasing students’ literacy and numeracy levels.

Despite the high priority given to improving literacy and numeracy skills, evidence from the OECD’s Programme for International Student Assessment (PISA) shows a steady decline in the average reading and mathematical literacy levels of Australian 15-year-olds since 2000. These declines occurred in an absolute sense, not simply in relation to performances in other countries. In mathematical literacy the decline was dramatic; while Australia was one of a handful of high performing countries in 2000, by 2015 mathematical literacy levels had declined to about the OECD average.

Interestingly, these declines are not reflected in Year 9 NAPLAN performances, which were unchanged between 2008 and 2016, raising a question about why these two assessment programs reach different conclusions.  

One difference between PISA and NAPLAN is that, while PISA assesses a national sample of 15-year-olds, NAPLAN assesses all students in Years 3, 5, 7 and 9. The most likely explanation for the different findings of these two programs, however, is that they assess different kinds of skills. The focus in PISA is not on basic reading and numeracy skills, but on students’ abilities to apply higher-level skills in reading and mathematics to the solution of real-world problems.

NAPLAN makes it possible to explore the reading and numeracy skills of Australian students in some detail. Because NAPLAN assesses all students rather than samples of students, it is possible to study changes in performance at the level of individual schools as well as school systems – including schools and systems in which results have improved since 2008. This, in turn, introduces the possibility of identifying policies and practices that may have led to these improvements.

But how well are we capitalising on this opportunity? Do we know where improvements have occurred and do we understand why?  

A starting point is to consider overall national trends in NAPLAN since 2008. These are shown for reading and numeracy in Figure 1. It can be seen from these graphs that, in Years 7 and 9, there was no significant change in mean reading or numeracy levels between 2008 and 2016. The lines for these year levels are essentially flat.

This is not the case at Years 3 and 5. Between 2008 and 2016, the national means in Year 5 reading and numeracy increased by 17 points on the NAPLAN scale. An even greater improvement (25 points) occurred in Year 3 reading, however there was no improvement in numeracy over this period.

Figure 1.  National reading and numeracy means (NAPLAN)

In the context of Australia’s PISA results for 15-year-olds, these improvements in basic literacy and numeracy skills in primary schools are encouraging. In Year 3, in particular, there have been strong improvements in average reading levels.

But how well do we understand the reasons for these improvements? Do we believe that they are sustainable into the future? Are they the result of changed educational policies since 2008? Are they the result of more effective teaching practices and, if so, do we understand the changes that were made and the factors that influenced those changes? Are improvements due to changes that occurred in schools or can they be traced to the years before school? 

Another interesting question is whether these gains were uniform across the nation. Did some states or territories experience greater improvements than others?

The answer to this question is that one state, Queensland, experienced significantly greater improvements in Year 3 and Year 5 reading and numeracy levels than other jurisdictions. Figure 2 shows the relative performances of Queensland students – that is, how far the Queensland mean was below the national mean – in each year between 2008 and 2016.    

Figure 2.  Queensland reading and numeracy means relative to national mean (NAPLAN)

It can be seen from Figure 2 that there were very significant improvements in the reading and numeracy levels of Queensland Year 3 students between 2008 and 2016. It seems likely that the introduction of a Preparatory year in Queensland from 2008 provided part of the reason for this improvement, however much of the observed improvement occurred prior to this first cohort entering Year 3 in 2011 and Year 5 in 2013. It is also clear that the strong gains in Queensland primary schools were not reflected in the performances of secondary school students.

How well do we understand why improvements in Queensland primary schools were so much greater than in the rest of the country? Were there particular policies and/or practices that produced these improvements? Are there lessons for other jurisdictions?

The other state in which there were unusually large improvements in NAPLAN results between 2008 and 2016 was Western Australia. However, the gains in that state were largely in secondary schools and, in particular, in Year 9 (see Figure 3).

Figure 3.  Western Australian reading and numeracy means relative to national mean (NAPLAN)

In 2008, the mean Year 9 performance in Western Australia was significantly below the national mean in both reading and numeracy. By 2016, performance was significantly above the national mean. (Recall from Figure 1 that, for the country as a whole, there was very little change in Year 9 performances over this period.)

Once again, there is a question about the reasons for this improvement. During this period, Western Australia introduced a requirement that students demonstrate a minimum standard of literacy and numeracy for the award of the Western Australian Certificate of Education. Students can meet this requirement through their performance on Year 9 NAPLAN. However, this requirement is unlikely to explain fully the steady improvements in Year 9 performances in Western Australia dating back as far as 2008.

Learning as a profession

Results from NAPLAN and from international studies such as PISA and the IEA’s Trends in International Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy Study (PIRLS) provide a variety of national learning opportunities.

What can we learn as a profession from national improvements in NAPLAN in Year 3 reading; exceptional improvements in Queensland primary schools since 2008; and exceptional improvements in Year 9 in Western Australia?

What can we learn from the fact that Australian secondary students, while maintaining their average performances in NAPLAN, have become steadily less able to apply higher-level reading and mathematics skills to solve real-world problems? Do we know why this is happening? Does it require a policy response?

What can we learn from the fact that, while levels of reading and mathematical literacy as measured by PISA have declined nationally, a much smaller decline has occurred in Victoria (and in the case of scientific literacy, there has been no decline at all)? Is there an explanation? Are there lessons that could be learnt from Victoria?

Are we learning from individual schools’ improvements in NAPLAN? The My School website was introduced in part as an opportunity to compare the performances of schools serving students from similar socioeconomic backgrounds. What national lessons are we learning from such comparisons? While individual schools and school systems may have views about why improvements have occurred, how reliable are these?

With evidence that reading and numeracy levels have stagnated or declined in many Australian jurisdictions, perhaps it is time for a more systematic effort to identify and understand where gains are being made in our schools.

Comments

  • So many questions and so few answers, I would have thought that with all the research going into this field we might have actually learnt something by now!

    09 May 2017
  • Hi Geoff
    I work as an educational consultant across Victoria, Queensland and Western Australia. What I have noticed in QLD and WA which is not occurring in Victoria is that schools have been instructed by the state government (whether it is explicitly or implicitly I don’t know) to teach to the test to get their school results up. Teachers constantly complain to me regardless of the school I visit about the time taken from their teaching to ensure students are ready for NAPLAN. I am not sure if the schools are embedding good literacy and numeracy practices throughout their schools or if they are masking this by ensuring their NAPLAN results look good. I think a bit more digging needs to be done on a school by school basis.

    09 May 2017
  • There are always so many more questions than answers when we test kids and NAPLAN is no exception. It is useful to examine overarching trends over a period of time, but what is happening at the school level is something else entirely. Using statistics like this is not appropriate nor does it produce good information. Teaching to the test is common across Australia and the tests are notorious for the error rate. Testing and the effects of testing do not account for individual issues on the day - lack of sleep, no breakfast, dissent and dysfunctional family life, anxiety, ... If the tests go online then other issues such as using a keyboard and mouse, hand-eye coordination and difficulties reading from the screen will have an impact on the overall results. By all means test our kids to get an overall picture of performance, but don’t link it to funding and the branding of schools in a negative way because that is detrimental to everyone.

    09 May 2017
  • Dear Geoff,

    The Year 9 NAPLAN assessment is essentially a waste of time in the minds of many students. Unless the school has an embedded examination culture as part of its assessment and reporting routine many students are unlikely to make the effort to treat the tests with the level of sincerity required. That some state education departments are attaching achievements in these assessments to future pathway access speaks volumes as to the effectiveness of assessing Year 9 students in this manner. To permit the data gained from three days of “one size fits all” testing to so strongly influence perceptions of education is to commit a fraud upon the Australian public. NAPLAN has a place but its data has to be considered in the context that the tests are not conducted under the same clinical conditions as Year 12 assessments, the various curricula are not taught at the same time across the country and some schools are sacrificing valuable learning time hammering students with test driven preparation. If our role as educators is to prepare today’s five-year-old for the world of 2030 and beyond then we need to work at developing critical and adaptive thinkers who can and will lead, create, empathise and collaborate. Good luck testing this with NAPLAN.

    10 May 2017
  • Thought provoking article. Regardless of what metrics we use to measure student growth, we need to be giving all our students the skills they need to apply higher-level skills in reading and mathematics to the solution of real-world problems in all curriculum areas.

    21 May 2017

Leave a comment





School community