The marking process for the Leaving Cert exams is rushed, unfair and risks compromising the accuracy of students' grades, according to an internal State Examinations Commission report.
The concerns stem from the way marking schemes for individual State exams are altered to ensure up to 100,000 students achieve grades broadly consistent from year to year.
The unpublished research report produced by the commission outlines key ways this form of standard-setting makes the final marking scheme for exams “less valid” and “reduces fairness” by benefiting some students and penalising others.
Among the key concerns outlined in the paper are that the process currently used by the commission “distorts” the marking by changing the weighting attached to individual questions, rendering the marking scheme “less valid”.
The process is described as being of “questionable fairness” as it can lead to some students benefiting or losing out more than others due to changes to marking schemes.
It diverts attention from a focus on accuracy in the marking of exams to a focus on achieving a similar proportion of grades year-on-year and “compromises the accuracy of marking” because, while examiners are trained to mark accurately using the original marking scheme, they receive no training on applying changes to marking schemes.
Altering marking schemes
The process lacks “precision and efficiency” in marking exams and impacts negatively on the recruitment and retention of examiners, the report states.
Most exam authorities seek to ensure exams are of a similar level of difficulty for candidates each year and that grades are broadly consistent.
However, the commission’s method of altering marking schemes for exams is very unusual internationally.
It involves diverting marks away from questions that are “easy” if too many students are scoring high grades.
Conversely, it involves giving extra marks for more challenging questions, if too many students are scoring lower grades.
The precise details of how these marking schemes are applied and subsequently changed has been hidden from public view until now.
The commission has previously refused to release details under the Freedom of Information (FoI) Act of how marking schemes for individual exams have been changed.
However, this report, seen by The Irish Times, says the only benefits of maintaining the current process are that it has “the appearance of objectivity”, is “familiar”, “enjoys widespread public understanding” and “may contribute to keeping the standard of the paper consistent”.
‘Validity’ of results
Prof Michael O’Leary, director of the centre for assessment research policy at Dublin City University (DCU), said the internal report highlighted an urgent need for a more robust and transparent system.
“The suspicion that it may be undermining the validity of what a mark on an exam means and disadvantaging some students is worrying,” he said.
In response to queries, the State Examinations Commission said it could provide “absolute reassurance” the existing exam standard-setting process was “fit for purpose, represents a valid and reliable approach to the marking of candidates’ work and is fair to all candidates”.
It said the document seen by The Irish Times was an early draft from 2018 of an internal research paper and was intended to provoke initial discussion, consideration and debate on what is a “complex and challenging process” in any examinations system.
In the development of the paper, it said a “deliberately critical lens” was applied to the current standard-setting process in order to “stimulate debate”.
“The paper has been subject to further revision, remains a work in progress, and is part of an ongoing review in the context of senior cycle reform and development,” the commission added.
