Yahoo Finance 'Punishment by statistics': the father who foresaw A-level algorithm flaws
When the English exams regulator set out how grades would be awarded following the cancellation of GCSE and A-levels this summer, there was suspicion of its “standardisation model” – a mysterious algorithm designed to avoid grade inflation.
Ofqual probably reckoned on a bit of pushback from the Labour party and maybe the teaching unions too. But in the end it was a statistics-savvy dad worried about his own son’s chances of getting into Oxbridge who correctly predicted that standardisation would result in 39% of students being downgraded, with disadvantaged pupils worst affected.
Huy Duong came to the UK as a teenager with his parents in the 1980s, a refugee from then-communist Vietnam. Now a father of three and an IT consultant, he told the Guardian that his formative years made him inherently opposed to anything which felt opaque and undemocratic. And that included “collective punishment by statistics”.
His eldest child, Thanh, attends Matthew Arnold school, a large, non-selective comprehensive in Oxford rated “good” by Ofsted.
Smart and with a mathematical mind, this summer Thanh should have been taking A-levels in maths, further maths and chemistry. He had a conditional offer to study engineering at Cambridge, which required two A*s and an A – a tough ask, but one father and son agreed was eminently achievable.
Then came lockdown. Exams were off and Thanh’s teachers were asked to award grades on the basis of mocks and assessments, and told to rank students from best to worst in each subject. This data was then fed into Ofqual’s computer system, designed to eliminate teacher bias.
The algorithm checked the results weren’t wildly different from the past and the expected national grade distribution for the subject, downgrading outliers. Ofqual tried to reassure students by saying the national results would be “broadly in line with previous years” and might be slightly higher.
But Duong wondered what it would mean for Thanh and his classmates. Using what scant information Ofqual had made public on its methodology, and an understanding of statistics gained from his PhD in physics, he analysed the 2017-2019 A-level data for Matthew Arnold school.
With a little help from his sister, a statistician at the Medical Research Council, he came to the conclusion that that there was “virtually no chance of providing grades to the students in a way that satisfies the double criteria of being fair to the individuals and controlling grade inflation nationally”.
He warned the education select committee that 39% of grades between A* and D would be lower than the teacher assessments, and last week he shared his findings with the Guardian. He repeatedly sent emails to Ofqual only to receive a stock reply.
Publicly, Ofqual insisted that Duong was wrong. In what may now appear to be a carefully worded statement, a spokesperson said: “We expect the majority of grades students receive will be the same as their centre assessment grades.”
But on Thursday morning Ofqual admitted that nearly two in five (39.1%) pupils in the country saw their A-level grades downgraded from their teachers’ estimates. Duong had been bang on.
At home in Oxford, he was in no mood to celebrate being right, however. Shortly after 8am, Thanh received his results: an A in maths, A* in further maths and an A in chemistry, just short of his Cambridge offer.
A “ridiculous” set of grades, said his father: “[Thanh] has consistently scored A* in tests and both mocks in maths. He finds mathematics much easier than further maths. He has also done quite well in the UK Maths Challenge [a national contest] and mathematical olympiad.”
“Ofqual wants to control grade inflation, which is fair enough,” said Duong. “But how confident are they that they are not downgrading the wrong people?”
Ofqual had a “public duty” to be open about how confident it was about its data, said Duong – “otherwise it would be like the Department of Health and Social Care imposing a vaccination programme on all school children without any data on toxicity and efficacy”.
He had another analogy to explain his objections to standardisation: “Suppose you have 100 cars travelling on a motorway and 41 of them broke the speed limit. In the interest of road safety, as a matter of principle, it is correct to fine the 41 speeding drivers.
“However, suppose that in practice, for whatever reason, the speed trap catches the wrong car, say, 25% of the time. As a result, about 10 of the drivers caught by the speed trap weren’t speeding at all but have been wrongly accused. What sort of democratic society would accept that? It sounds like collective punishment by statistics.”
Ofqual continues to insist its standardisation arrangements “are the fairest possible to facilitate students progressing on to further study or employment as planned”. On Thursday it published a 319-page document explaining its methodology.
It was little solace to Thanh, even after Cambridge decided to accept him regardless. Opening the email with his results was a “stomach-dropping experience”, he said. “These results will affect us for at least the next 10 years, getting my first job, although I recognise my situation isn’t the worst … The results can’t be as reliable [as previous years] because it’s something … manufactured rather than what we have set out to do and then achieved.”
Duong’s crusade for transparency continues. “I came here as a teenager with my parents in the 1980s. In that time, Vietnam was a communist country. We came here as political refugees to a democratic country, so I have a very strong idea of how democratic societies should work and this doesn’t look like it,” he said.
