A new study rejecting research suggestions that Wikipedia content shapes the judgments of Irish High Court judges has been welcomed by the President of the High Court.
Mr Justice David Barniville said those with practical knowledge of judicial decision-making “will know that the claim that judges rely on Wikipedia in preparing their judgments in any material way is plainly wrong”.
“I welcome this detailed analysis which confirms that such claims are wholly inaccurate”.
However, Dr Brian Flanagan, the lead author of the disputed research said it was “robust” and he rejected the claim that his methodology was flawed.
Two men arrested following cocaine haul in Limerick
Negative views of lawyers rely on ‘cliches and tropes’, says Bar council chairman
Receiver settles case against former Dublin Lord Mayor Nial Ring and his wife
Criminal lawyers divided on whether charges should have been brought against Conor McGregor
High Court judges in Ireland have repeatedly maintained that submissions from lawyers, not internet searches by judges or their assistants, primarily drive citations in judgments.
Compiled by a High Court judge, Mr Justice Richard Humphreys, and a team of serving and former judicial assistants, the new study disputes findings of research by academics at Maynooth University (MU) and two US third level institutions.
The original research paper, published last summer, arose after researchers arranged for 150 new Wikipedia legal articles on Irish Supreme Court decisions to be written by law students. Half were placed online and the rest were kept offline and treated as the control group.
The researchers looked at two measures – whether the cases on Wikipedia were more likely to be cited as precedents by subsequent judicial decisions, and whether the argumentation in court judgments echoed the linguistic content of the new Wikipedia pages.
Their key finding was that getting a Wikipedia article increased a case’s citations by more than 20 per cent.
When an amended version of the paper was published earlier this year, Dr Flanagan, associate professor of the school of law and criminology at MU and lead author of the research, rejected Mr Justice Humphreys’ assertion of a watering down of its main findings.
In a press release on Tuesday announcing the new study, Mr Justice Humphreys said there were “fundamental problems” with the original research paper which render its conclusions “unreliable”.
Those problems lay both with a “flawed experiment” and with “flawed speculation”, which seemed to derive from “a simple lack of knowledge regarding the practical process of judgment production”, as to the meaning of the results.
He did not at all believe there was bad faith by the original authors but it was important “that public trust in our legal system is not undermined by poor research”, he said.
The new study, he outlined, compared citations of cases over a two-year period before Wikipedia articles were created with a two-year period thereafter.
That showed citations in judgments changed by exactly the same amount in respect of cases where articles had been created on Wikipedia as applied where no articles had been created, he said.
“The evidence supports the conclusion that there is simply no Wikipedia effect whatsoever.”
The study tracked the origin of case citations in a representative group of judgments, and found a large majority of citations arose from written or oral submissions, or materials referred to in such submissions. “In virtually all the judgments studied, there was objective evidence for a source of the case citation which had nothing to do with Wikipedia.”
The study sets out a detailed critique of the original paper, including concerning the use of “distant historic citations” which is alleged to undermine its reliability. It describes as “problematic” the use of a commercial website to obtain citation information rather than the official courts site. The methodology implied by the published data implies it was originally intended to consult the official website but no data on this has been published, it states.
Claims that the conclusions in the original paper had not changed were described as “lacking credibility”.
The study claims the original paper’s “speculations” are exaggerated and not supported by the evidence and that its conclusions are “inappropriately moralising”.
In a statement reacting to the Humphreys study, Dr Flanagan disputed the methodology used in the research was “flawed”.
He said the study claims the research methodology was flawed in two respects in particular – that commercial data was used instead of the courts’ own records and that too long a period of citations “before” the Wikipedia articles was used and that this determined the results.
“In fact, our analysis is robust to both these critiques,” he said. The commercial data used for the analysis was cross-checked against the court’s own records and the researchers found “only small differences that didn’t change our conclusions”.
From the analysis, it was “plain to see” the treatment (Wikipedia) articles jump in citations immediately after being added to Wikipedia, and this jump is “notably different to any period in the two years beforehand”, he said.
“While we disagree with these headline points from this paper, we also understand that the authors have gathered additional data on the citation context from judgments and we welcome that discussion,” Dr Flanagan said.
A summary version of the Humphreys paper was published on Tuesday in the Irish Law Times and the full paper has been submitted, by invitation, to the Cambridge Handbook of Experimental Jurisprudence. The full paper is available online at papers.ssrn.com as is the research paper.
