[2023] UKUT 157 (AAC)

Mr Hilton’s evidence was contrary to Ms Walton’s evidence that test candidates were “incapable of recalling how many questions were in a test”, and

if tutors wanted to know how many questions were in a test, they could simply ask candidates who had sat the test,

as well as Mr Coombs’ request that Mr Hilton’s evidence be “disregarded”.

Knowing the number of questions candidates need to complete is of no benefit to tutors but is directly related to reliability. In a multiple-choice questions, one answer is always correct. Therefore, a random guess has a one in five probability of being correct. Candidates are always instructed to keep careful note of the time and in the last minute complete any remaining questions by guessing.

Mr Hilton’s evidence was that some children fail to complete the test. He provided no evidence that the individuals who failed to complete the test were tutored and yet were unaware that they could gain, on average 20% marks in the sections they did not complete. He argued there exist tutors unaware of the simple and effective tactic of guessing any remaining questions during the last minute and that these tutors could use drop off rates as the basis of coaching children to start guessing after completing a given number of questions [H486 PDF502 §21]. This conflicted with Miss Walton’s evidence that these children were incapable of even recalling how many questions there were in the test.

Mr Hilton claimed that if the test reliability was disclosed that tutors could use this to reverse engineer the number of questions in the test. This is true, however an indication of the complexity of calculating this can be found in the appendix to my submission [A69 PDF78]. If knowing the number of questions was of any benefit to tutors, they would simply ask someone who took the test to provide this.

Mr Hilton also said that information about the completion rates could be used to work out which question number to start guessing from, instead of the tried and tested method (which from personal experience I can date back to the 1970s) to guess all outstanding questions during the final minute or two of the test.

In summary, Mr Hilton’s suggestions as to how tutors could benefit from disclosure of any of the information contained in the detailed statistical analysis, or the PowerPoint presentation summarising it amount to the sort of cunning plan one would expect from the character Baldrick in Black Adder and should be discounted accordingly.

would not work if (as Ms Walton’s evidence suggested, and as the FTT appeared to accept at [73]) candidates would not know what question “number” they had reached (because the questions were designed to obscure this – see paragraph 26 of Ms Walton’s 31 March 2021 witness statement, quoted at paragraph 38 above); and

boiled down to what Mr Coombs said was a well-known tactic of advising candidates to guess the remaining questions once time ran short (for which knowledge of the number of questions in the test was not required).

Mr Coombs more generally challenged the proposition that tutors knowing the number of questions in a test would give rise to any material advantage for tutored candidates; he reasoned that this was intuitively unlikely as it was something tutors could ask their pupils to tell them after they had taken the test (despite Ms Walton’s evidence about this being discouraged – again, see paragraph 26 of her 31 March 2021 witness statement).