Wednesday 19 August 2020

Don't blame the algorithm, blame the people who designed it.

One idle thought I'd had was, if we are going to allow teachers to hand out GSCE and A-level grades, a good way to prevent grade inflation (or deflation) would be to bear in mind that a school's overall results don't change much from year to year and each school is allowed (or is obliged) to award the same number of each grade in each subject as pupils achieved in 2019.

So if last year, 50% of pupils achieved an A grade in GSCE maths at a particular school, this year, the teachers are allowed to give 50% of pupils who would have taken maths GSCE an A grade.

Still a bit unfair, but to be honest, exam results always are a bit unfair; exam marking is fairly subjective. Some people argue that the whole exam system is unfair (I've no complaints myself, I usually did very well in exams). And sure, pupils at private schools will get given better grades, but they would have got those anyway, so that's not really an objection to the algorithm.

It turns out that that is what was supposed to happen:

From the BBC:

Teachers were asked to supply for each pupil for every subject:
- An estimated grade
- A ranking compared with every other pupil at the school within that same estimated grade

These were put through an algorithm - or mathematical procedure. The biggest element in that was the school's performances in each subject over the previous three years. The idea was that the grades this year - even without exams - would be consistent with how schools had done in the past.


So far so good, but the idiots didn't design the algorithm properly, and pupils at independent schools were awarded 4.5% more grades at A and above than the pupils in the previous year.

That is a bit of a smoking gun, if you ask me.

13 comments:

Bayard said...

While I've never been a fan of nationalisation or centralisation, I have always thought it bonkers to have competing boards setting national exams. The temptation will always be to make their exams slightly easier and so give schools an incentive to use them father than another board.
My impression was, when I was at school, that the exam boards already did more or less what you propose, so your chances of getting an A grade were much improved if you were in a particularly dim year. That was over forty years ago, back in the days of "O" levels.

mombers said...

Terrible idea. At what point does a school get deemed to be allowed (or forced) to award a different distribution of grades?

Sobers said...

The reason the private schools got better grades was that the algorithm put more weight on the teachers predicted grade when the numbers of pupils were low, on the basis that its statistically a bad idea to make assumptions on such small numbers. One poor result last year in a group of (say) 10 pupils could mean very unfair outcomes. This algorithm setting applied to both State and private schools, but obviously private schools have far lower numbers of pupils per year and per subject so were far more likely to have their results based on the teacher assessment than State schools and colleges. Some private schools in some subjects could have as few as a handful of pupils taking a particular subject, so trying to apply some statistical framework so such small numbers would never work.

Lola said...

S. So what we have then is a total inability of the political class - the mouthpieces for the quango state - to articulate clearly how and why the algorithm + teacher assessment would be the least worst outcome?

Mark Wadsworth said...

B, but there was prestige for a passing an exam set by a "harder" board.

M, the idea is to try and give pupils the results they would have got. And it is quite simply true that the girls at my daughter's school 'would have got' 90% As and A*s.

S, "the algorithm put more weight on the teachers predicted grade when the numbers of pupils were low"

Fair point, but that's easily fixed. We separate out 'popular' subjects where there is a sufficient sample size, and all the minor subjects get lumped into one.

So if pupils at that school last year got a total of 8 As, 14 Bs etc in a rag bag of Ancient Greek, Mandarin Chinese, Astronomy, whatever, this year the teachers give out 8 As, 14 Bs etc to pupils who would have taken Advanced Maths, Domestic Science or Beekeeping.

L, it wasn't a good outcome, it gave private school pupils an ever bigger advantage than they already have.

Lola said...

MW. Maybe that would not be an issue if all schools were 'private'.? That is not nationalised?

Mark Wadsworth said...

L, the problem arose because of the lock down. The problem is 'fairness' between good schools and not-so-good schools.

Even if all schools were 'private' there would still be good ones (my girl's school) and OK ones (my boy's school).

And the schools that do a lot of weird and wonderful subjects would still have the slight advantage as explained by Sobers above.

Lola said...

MW Of course. It's be 'a market'. 'Prices' would rule. Including wages of teachers. Teachers would love that....not.

Nessimmersion said...

Its not just UK, it also affect International Baccalaureate as well.
Interesting to find out the origin of the algo as it were

Bayard said...

"but there was prestige for a passing an exam set by a "harder" board."

I remember being told that at the time, but no-one ever asked me which board had set my O and A levels.

Mark Wadsworth said...

B, your O level results are only important for getting on the next level (A level, college, job, training etc), after that nobody cares, least of all which board it was.

At the time, it seemed important. But now I wouldn't even remember which board's exams I took, I'd have to go and look at my certificate.

Bayard said...

Well, yes, but I was still at the same school where I did my O levels when I did my A levels, so they wouldn't really have cared which board it was, I suppose it's possible that the universities I applied to knew which board my school used.

Mark Wadsworth said...

B, fair point re staying at same school. Maybe universities care, or cared, a little bit which Board? It seemed important at the time, but years later, I wonder whether it did.