Posted
This year's NAPLAN tests are so flawed the results of
1 million students "should be discarded", according to two
international experts who have issued scathing criticism of the national
system.
Key points:
- Two US professors say online and pen-and-paper tests cannot be compared
- They say 2018 NAPLAN results are of "very limited use" and should be discarded
- Sydney University expert says tests can be compared if the right scaling methods are used
US professors Les Perelman and Walt Haney, both renowned authorities on assessment, claim the results are of "very limited use" to parents, teachers and schools.
The professors' findings are contained in a report commissioned by the NSW Teachers Federation following the recent furore surrounding the comparability of online and pen-and-paper test results.
The Australian Curriculum and Assessment Authority (ACARA) assured schools its methods for scaling and equating test scores are statistically sound.
But Professor Haney, the former longtime head of the Centre for the Study of Testing at Boston College, said numerous studies had shown there were "enormous" differences between the two modes.
"There's no way that you can successfully equate the results for a large number of schools, much less for a large number of students, to take into account the well-documented effects of mode of administration on the results of the test," Professor Haney told the ABC.
"If the administration there is trying to claim that the results of the computer tests can be reasonably compared with the results of past tests, I would say that on the basis of research that I've done and that I know of, that is potentially very misleading.
"I would say that there may have been some naive people in the leadership of education administration in Australia who were simply not aware of what is traditionally required to reliably equate the results of different test administrations."
Do you know more about this story? Email Specialist.Team@abc.net.au
Earlier this month, some state governments issued fierce criticism of ACARA's handling of the transition to online testing.
One in five students in years 3, 5, 7 and 9 completed NAPLAN online for the first time this year.
ACARA declined to comment on the US professors' criticism but says its Measurement Advisory Group has extensively reviewed its statistical methods and has guaranteed the integrity of this year's results.
Dr Perelman was the author of the report entitled Problems in the Design and Administration of the 2018 NAPLAN, and Professor Haney was a contributor.
The report said: "Because the computer-adaptive testing and pen-and-paper tests are not comparable, posting results in the My School database would be inappropriate and misleading.
"In sum, the 2018 NAPLAN results should be discarded.
"ACARA must take responsibility for what can only be described as educational and statistical incompetence."
The report said the most concerning aspect of this year's NAPLAN was the online version of the grammar and punctuation test.
Students who achieved high scores in reading were prompted to answer more difficult grammar and punctuation questions.
Students who were not as proficient in grammar and punctuation as they were in reading were highly disadvantaged under the test method, the report said.
In addition, students who had better word processing skills were at an advantage in the online test.
The professors called for a public review of the studies used by ACARA to conclude that the two sets of test results could be equated.
But not all local experts agree with the US professors.
Jim Tognolini, the director of Sydney University's Educational Measurement and Assessment Hub, said online and pen-and-paper tests could be compared given the right scaling methods were employed.
"To say that comparisons can't be made is just not right," Professor Tognolini said.
"The whole comparison issue to me is not that big of an issue. I think this idea this year's results are unreliable is just not right."
No comments:
Post a Comment