Extraordinary Circumstances Demand Extraordinary Measures
Many northern hemisphere countries use examinations during the summer months as their main method to recognise and award student achievement. In the UK, statutory qualifications, such as GCSEs, A Levels and Scottish equivalents, have exams during the summer and the grades awarded for these are published on Results Days in July / August.
2020 has not given any occasion to ponder the merits of using exams as the customary assessment tool of schools and universities. The speed of this pandemic has required quick thinking from the educational sector; extraordinary circumstances have demanded extraordinary measures. 2020 has demanded discussion and decision-making in a different area of exam assessment; how to assess whilst imposing lockdown and social distancing.
Continue, Change, Postpone or Cancel Exams
Like the sword of Damocles hanging over their governments, countries have had to align their response to the pandemic with their customary use of exams. Four different responses were seen:
- to continue with the exams, such as Hong Kong’s Diploma of Secondary Education
- to change them, such as the Advanced Placement tests in the USA
- to postpone them, such as the Gāokăo university entrance exams in China
- to cancel them, such as in Norway and the UK.
The UK’s Choice
On 18 March, the UK government announced the decision to cancel exams as well as close all schools and colleges. On 27 March, Education secretary Gavin Williamson reassured the public that the government would do all it could to ensure students received their results in summer. Students could not attend school, nor sit any exams, but they would still be awarded with grades in the summer, their ‘ticket’ to their futures. The scheduled Results Days would still occur, such as
- 4 August for the Scottish Highers
- 13 August for A Levels
- and 20 August for GCSEs.
Don’t call them Results Days
But this year, these dates are not really the days when students’ results are published, these are the days when young people find out the grades assigned to them. The grades being receiving are not ‘results’. They are not the result of students’ hard work and application, nor the result of their teachers’ professionalism and assessment. They are mainly the product of speculation and statistical modelling; they are calculated estimates.
Awarding a Grade without an Exam
Secretary of State Gavin Williamson’s message of reassurance showed the challenge for the UK education sector; how to award a grade without an exam. Some countries had the luxury of being able to assign a grade based on other assessments which had taken place, such as in Norway where ongoing assessed classwork constitutes 80% of the final result.
A Level and GCSE qualifications reform (2010-2015) denied England and Wales this benefit. These reforms reduced or eliminated the amount of non-exam assessment (coursework) and made these courses linear, rather than modular, with terminal assessments. This has meant that assessment for these qualifications is now at the very end of the course, in the summer, by exams. The implementation of these qualifications reforms was fully completed in the summer of 2020; the least fortuitous timing in retrospect.
The method decided upon to award grades for A Levels, GCSEs and Scottish equivalents was a two-pronged approach of applying statistical modelling to the grades and ranks generated by teachers (the centre assessed grade). Teachers were asked to use professional judgement to estimate the grades students would most likely have achieved if they had sat exams in the summer. Then the various exam boards would then augment these by applying statistical models or algorithms from the governmental regulatory bodies (such as SQA in Scotland, Qualifications Wales and Ofqual in England).
Set up to Fail?
Very soon after this two-stage method was decided, the Chief Regulator of Ofqual wrote a letter to GCSE, AS and A level students. She stated that standardisation would be needed to address those schools and colleges which were more generous or harsh than others in estimating their centre assessed grades (CAGs). This letter was sent in April, two months prior to the submission of the CAGs. There was an expectation that CAGs would need to be adjusted before they were written.
There is a well reported gulf between predicted and achieved A Level grades, particularly when the predictions are used for transition to university (e.g. UCAS predictions). However, it could be argued that anticipating the over / under-estimation grades in this unique instance, with a different end-user, was an unreasonable presumption and setting teachers up to fail. To reverse the ‘set up to fail syndrome’, Manzoni and Barsoux recommend that managers and leaders challenge their own assumptions. Presuming that some would be generous or harsh before they were, should have prompted the regulatory bodies to provide better guidance, rather than stating this as a reason for requiring statistical modelling.
Standardisation, Moderation or Adjustment?
The nomenclature used to explain the use of algorithms to arrive at the calculated grades has referred to the processes of standardisation and moderation.
Normally, for high stakes assessments like Scottish Highers, A Levels and GCSEs, standardisation is the process taken to monitor the standards and decisions made by examiners and to ensure that they are consistent. It confirms that everyone who is examining the qualification sings from the same hymn sheet.
Moderation is a similar, but vertical process usually reserved for coursework. This process allows a moderator to review a sample of the centre marked work, to judge that its application of the standards is consistent with that set by the exam boards. When this consistency is not present, scrutiny and regression ensue.
This summer, neither processes are relevant as it has not been possible for the standards to have been set and applied by the exam boards. Standardisation and moderation are not what is occurring when the regulatory bodies have adjusted the CAGs. It is a process of adjustment or augmentation and should be named as such.
We need fairness
Roger Taylor, Ofqual’s Chair, justified that the process of adjusting CAGs through statistical modelling is “the fairest possible way to recognise students’ achievements this year”. This intention to be fair is commendable; we all need fairness.
Neuroscience has shown how our brains are hard-wired to respond with happiness to fair treatment and with disgust or protest to unfairness. Additionally, parents and teachers know that young people already have an enhanced need for fairness in their lives. But there is nothing fair about COVID-19.
But Fair to Who?
Adjustment to CAGs has been undertaken to promote equity. But it can be queried who the beneficiary of this fairness is. BAME students, those from low higher education participation rates, women, children with special educational needs or disabilities? None of these groups were mentioned as the intended recipient of the fairness measures.
Instead, creating fairness across place and time were the focuses; to create fairness between schools and colleges and to maintain equity with grades awarded in previous years and future recipients of these qualifications. Both laudable and worthy focuses, but given the circumstances, it can be queried whether these should have been the priorities.
But Nothing Compares to This
Whilst loathing to use one of Dictionary.com’s ‘pandemic phrases people hate’, it has to be asserted that these are unprecedented times. Students will be receiving grades without exams.
Every intention is that this summer will be the only occasion when exams are cancelled, so this situation is unique. But it is simply overstated to suggest that defending the CAGs without adjustment “would create a perpetual unfairness between this year’s grades compared to past and future generations”. Aiming for consistency over time is helpful but it can be queried whether this should have been such a priority for the qualifications regulatory bodies in this exceptional time.
Anchoring in the Past
To provide consistency over time, historical data of exam performance from the schools and colleges was used to confirm or adjust the CAGs. Both SQA and Ofqual have said that full, exact algorithmic details would be published to allow statisticians to understand how this was applied. Time is of the essence here.
The last three years of a centre’s A Level results and two years of GCSE results have been used. This historical data is applied to individual subjects, like English and Maths, rather than adjustments being made to the overall performance of the school (which often form their marketing headlines). The aim is to equate the grades to previous results in that subject at that school.
Anchoring this year’s grades to past performance is easier to achieve for more popular subjects, such as Business and Psychology at A Level, as there was sufficient previous data; less likely to be feasible for smaller subjects like Spanish and Music GCSEs. The calculated grades may show that some subjects are the victims of their own success and popularity.
Appeals to the adjustments can refer to changes in the school or college, such as being changed into an academy. It is not known yet whether the ongoing, incremental improvements in teaching and learning will be grounds for appeal, even though this is one area where nothing is static.
Dropping down the waterfall
The adjustments will move students between grades to preserve the historical proportions of them at any one grade; a reported 39% of A Level grades will be downgraded. Called the ‘waterfall effect’ by SQA, if more students receive grade A than in the past, some will be downgraded to B. This may then produce too many at Grade B, leaving those at the bottom of the rank order to be downgraded to C, and so on.
The waterfall effect seen in the Scottish qualifications grades this summer had greatest impact on those from the most deprived areas. More than 30% of them pooled at the bottom of the waterfall, with a fail grade in their Highers; they failed the qualification they sat no exam for.
It’s not personal
The algorithms chosen to secure consistency across schools and over time are impersonal; their application is not personal to the individual students. But the grades they produce are acutely personal; they form the student’s ‘ticket’ to their future. Grades permanently define their holders.
Forewarned is forearmed
Teachers will be on the frontline this summer. They will be the ones speaking with individual students about their grades, supporting their mental health, guiding them into their futures, such as university. They will have received the calculated grades mere hours before having to support the students. Little opportunity available to interrogate these grades and check the outcomes of the regulatory checkers, despite A Level and GCSE outcomes already been made available to journalists.
Teachers will be the public face of the calculated grades, the initial target for any dissatisfaction from their recipients, without full knowledge of the methodology used to generate them. It is unfair for them to be the patsy or fall guy for the outcomes of a process they have not yet been a party to.
What can be learnt from Scotland?
The Chief Executive of the SQA declared that the statistical modelling it had used for its calculated grades had granted ‘fairness to learners’. However, recipients of these grades published on 4 August largely did not agree, leading to protests and an unreserved apology from the First Minister of Scotland, Nicola Sturgeon, on 10 August.
Three key messages can be learnt from Scotland’s experience prior to the publication of A Level and GCSE grades in England and Wales:
- be correct with the names used, such as calculated grades and adjustment
- interrogate who the algorithms have been fair to
- publish the algorithms used prior to the release of the grades.
Knowing that we respond with disgust and protest to unfairness, it should be an expectation that when the A Level and GCSE grades are published, the public are guided to understand how the CAGs were adjusted to promote fairness across time and place.
Biography Blurb
By Vanessa Evagora, Ed.D student at the University of Buckingham
Vanessa is Head of Psychology, EPQ and Higher Education Applications at an independent boarding school and is a Fellow of the Chartered Institute of Educational Assessors. She is undertaking the Ed.D at the University of Buckingham, measuring the predictive and construct validity of high stakes assessment of Psychology.
References
Adams, R., 2020. Nearly 40% of A-level result predictions to be downgraded in England. The Guardian https://www.theguardian.com/education/2020/aug/07/a-level-result-predictions-to-be-downgraded-england
Collier, S., 2020. A Message For All GCSE, AS And A Level Students This Summer. [online] Ofqual.gov.uk. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/878854/Letter_to_students_-_Summer_2020_grading.pdf
Dictionary.com Contributors, 2020. 8 Pandemic Words & Phrases People Absolutely Never Want To Hear Again. [online] Dictionary.com. https://www.dictionary.com/e/pandemic-words-people-hate/
Manzoni, J. and Barsoux, J., 2002. The Set-Up-To-Fail Syndrome. Boston, Mass.: Harvard Business School Press.
Tabibnia, G., Satpute, A. and Lieberman, M., 2008. The Sunny Side of Fairness. Psychological Science, 19(4), pp.339-347.
Taylor, R., 2020. Ofqual chief: ‘This is the fairest way to award grades without exams’. Sunday Telegraph https://www.telegraph.co.uk/news/2020/08/08/ofqual-chief-fairest-way-award-grades-without-exams/
UCAS, 2016. Factors Associated With Predicted And Achieved A Level Attainment. [online] UCAS. https://www.ucas.com/file/71796/download?token=D4uuSzur
Leave a Reply