My Lords, I start by apologising for the absence from this debate of the noble Lord, Lord Bew, who has been delayed on his flight from Northern Ireland by weather. He was very keen to be here and will greatly regret that he has missed this debate.
I have four amendments in this group, beginning with Amendment 187. I can describe them most concisely as a range of options to de-fang the National Student Survey as an ingredient in the TEF. The options range from requiring parliamentary approval of the scheme proposed under Clause 25, to an independent inquiry into the statistical validity of NSS data and, finally, the nuclear option—that the Committee does not agree to Clause 25 standing part of the Bill.
I shall start where we left off in an excellent debate touching on these issues last Wednesday. That debate had a rather wider proposition at its heart: that the link between the TEF and the ability of universities to raise fees should not come into being straight away. They would be given time for the TEF—and the statistical ingredients and metrics within it—to be properly got right. I sympathise very much with that view, but it is not the question today.
In the debate last Wednesday, a majority were certainly critical of the metrics being used—of whether the things the National Student Survey asks students are indeed a good way of measuring the quality of teaching in an institution. Some pretty key difficulties were raised. For example, there seems to be very little correlation—or no correlation, according to a paper by the Royal Statistical Society— between the scores achieved in the NSS by an institution and the quality of its degree results. That seems a bit worrying to many people. Those who defended the NSS did not actually argue that it was perfect—the noble Lord, Lord Willetts, was very frank. It is not perfect. They made the reasonable point that if we wait for perfection on this earth we get nowhere very much, and therefore argued that we should include these metrics.
As I said, I shall not go over that argument again in detail this afternoon, though we shall probably come back to it on Report. However, I have to be absolutely clear: my worries about the NSS are not primarily related to whether the metrics are good metrics for deciding teaching quality, or whether they are the best
available, or any of those things; they are pretty well purely statistical. When the NSS survey results are compared, they do not reliably reflect the opinions of students in differing institutions as to the quality of the teaching they are getting. These are statistically flawed results, as well as, arguably, being flawed as metrics.
I am in danger of going on all night and being extremely boring. I know the Committee will have a limited appetite for a great deal of statistical discourse—although if there is anybody who shares my nerdish love of these things, they should read two documents by the Government’s own ONS on the statistical basis. They should also read the excellent document by the Royal Statistical Society, which analyses this matter in detail.
I shall just mention one or two problems that are relatively easy to comprehend. The response rates to the NSS vary greatly between different institutions. It is perfectly clear from what we know that the non-responders are not the same as the responders and, in particular, that ethnic minorities are greatly under-represented in the responses. This can have a terrific effect on the results. Let us suppose that in one year there is a 70% response rate, giving a result of 60% satisfied. If that 70% response rate had gone up to 100%, the whole of the remaining 30% might have been satisfied or all of the non-responders might have been not satisfied. So the true result could vary by 30% each way—60% in total—from the result given by the NSS. There are particular problems with sample sizes in small institutions such as my own—Trinity Laban. Music students are our biggest group of students—there are 112 of them—and the statistical margin of error for that number is very large.
4.30 pm
A rather more complex but very important point is that in the NSS the results for nearly all institutions are very clustered, so very tiny changes, which may be no more than statistical noise, can make enormous differences to where you appear in the league table. They could very easily move you down from gold to silver or from silver to bronze. These are simply not reliable statistics on which to base facts. The ONS concluded that,
“given the confidence intervals … it is likely that comparisons of raw data … at this level would not be … significant”,
yet the Government are using insignificant data to make a very significant decision about the category into which a university falls and therefore, in time, how much it will be able to charge in fees, as well as how immediately attractive it will seem to students thinking of applying to it.
I accept that the Government have slowly started to recognise the inadequacy of these numbers. In their latest instructions to assessors, they said:
“Assessors should be careful not to overweight information coming from the NSS”.
I would put it a lot stronger than that—I might even say, “Throw it in the waste paper basket”—but they did make the concession that it should not be overweighted and that other things should be relied on. One thing on which they can rely is the submissions made by
institutions, in which they lay out, according to a formula, the strengths of their teaching. To my mind, that submission procedure should be accorded much more weight than the statistics of the NSS in particular, and the metrics much less weight, if we are to get a TEF that works.
I conclude with two brief observations. Once upon a time, everybody thought that opinion polls were to be relied on, but we all know now that they were not. I am in quite a fortunate position because I said before the 2015 election that the polls were unlikely to get it right and they did not. I distrusted the polls on Brexit and they were wrong. I also distrusted the polls on the American presidential election, and those too were wrong. That is due to perfectly simple statistical things—which, again, we see in relation to the NSS—such as unrepresentative samples, poor response rates and so on. The opinion polls got it wrong and the NSS will get it wrong for the TEF. As a result, the TEF will be damaged and I shall be sad to see that.
Finally, as I said, I am a nerd, and I have peculiar Sunday reading. Last night I was reading a rather remarkable book called Weapons of Math Destruction by Cathy O’Neil, who has a PhD in maths from Harvard and is an ex-quant in the financial services industry. On page one, she describes the introduction of a scheme to improve teaching in the worst schools in the worst areas of Washington DC. That is something we would all want to do, just as we all want to see an effective TEF improve teaching in our universities. She follows through the steps by which that system, based on wrongly interpreted mathematical statistics, had led to the sacking of one of the best teachers in one of those deprived areas. It did not hurt the teacher, who got a job straightaway in one of the best schools and areas of Washington DC, but, my God, it hurt her pupils. This is the kind of road I fear we are going down. Your Lordships will find many other examples in her book.
The NSS in the TEF is using—or rather, abusing—statistics for a purpose for which the NSS was never designed. My amendments are designed to reduce that risk for good colleges with good teaching that are in danger of falling foul of a statistical lottery. I beg to move.