Informing policy and practice

Informing policy and practice

Large-scale assessments have become increasingly used as tools for monitoring the effectiveness of educational systems. International assessments like the Programme for International Student Assessment (PISA), the Trends in International Mathematics and Science Study (TIMSS), the Progress in International Reading Literacy Study (PIRLS) and the Civic Education Study (CIVED), and national assessments like the National Assessment Program – Literacy and Numeracy (NAPLAN) and the NAP sample studies in science literacy, civics and citizenship and ICT literacy differ in purpose, approach and methods. Even so, they share the common features of utilising a common assessment tool administered to large samples or populations of students under uniform conditions. They also typically utilise similar features of test design and methods of analysis and reporting, although they differ in much of the detailed aspects of design and method.

Given ACER’s close involvement in such large-scale assessments, and drawing on the capabilities of ACER in analysis and reporting, ACER is perhaps uniquely positioned to undertake a synthesis and appraisal of data from PISA, TIMSS, PIRLS, CIVED and the various National Assessment Programs. That synthesis and appraisal, published in the Measure for Measure report released in August, addresses assessments in Australia conducted over the past 20 years.

Data from international and national assessment studies can be used in many ways. In Measure for Measure, we have focused on examining trends. Since the assessment instruments are equated over time, it is possible to interpret changes in achievement in relation to changes in policy, provision, practice and context.

Of course, it is also useful to compare statistics from international and national assessments at a point in time and we have also made use of those sorts of comparisons – including comparisons by country, jurisdiction and student groupings in terms of personal, social and geographic characteristics.

Comparing rankings of countries or jurisdictions…can be misleading. One cannot be sure whether a country missed a place on the podium by a millimetre or a kilometre.

Any data obtained from assessments has associated uncertainty arising from measurement error and sampling error. We have been careful in Measure for Measure to emphasise the uncertainty in our estimates so that we do not claim differences about which we cannot be sure, and we have made little use of rankings. Comparing rankings of countries or jurisdictions typically does not take such uncertainties into account and can be misleading. One cannot be sure whether a country missed a place on the podium by a millimetre or a kilometre. Rankings can also change as a result of new entrants to or withdrawals from the tournament. Neither can one be sure whether differences among countries are a consequence of policies in education systems or practices in classrooms or a consequence of other factors.

Achievement in reading

Our synthesis and appraisal indicates a small decline in reading achievement by Australian 15-year-olds from 2000 to 2009. Measure for Measure reveals an increase in between-school variation in student scores and the association of the between-school differences with the average socioeconomic background of the students at each school; and a decline in reading achievement that was not evident in all jurisdictions which suggests that there could be some organisational and curricular aspects of school systems associated with the decline in reading achievement. NAPLAN reading data show that there has been a steady improvement in reading achievement among Year 3 students and a smaller improvement in Year 5 reading achievement from 2008 to 2012.

Achievement in maths

Measure for Measure indicates a small decline in the mathematics achievement of 15-year-old school students, stability in the mathematics achievement of Year 8 students and a small improvement in the mathematics achievement of primary school students (but still lower than countries such as England and the United States).

Achievement in science

Our synthesis and appraisal reveals the science achievement of secondary school students in Year 8 did not change from 1994/5 to 2010/11 but also that Australia has a relatively wide dispersion of scores. The science achievement of primary school students remains relatively stable.

Measure for Measure shows that data from international and national large-scale assessments can provide broad indications of the progress and status of school systems in terms of student outcomes. We think the most fruitful way to use such data is to examine improvements and declines in achievement over time, and to relate those changes to developments in policy, practice and context that took place in the immediately preceding years. This seems most likely to generate insights that can inform policy and practice.

Read the full report:
Measure for Measure: A review of outcomes of school education in Australia, by Dr John Ainley and Eveline Gebhardt, is available at < http://works.bepress.com/john_ainley/164 >