The Economist summarized the crisis in academia last year when they looked at how difficult it is to replicate the results of peer reviewed studies.
“There are errors in a lot more of the scientific papers being published, written about and acted on than anyone would normally suppose, or like to think...Fraud is very likely second to incompetence in generating erroneous results, though it is hard to tell for certain. Dr Daniele Fanelli [University of Edinborough] has looked at 21 different surveys of academics (mostly in the biomedical sciences but also in civil engineering, chemistry and economics) carried out between 1987 and 2008. Only 2% of respondents admitted falsifying or fabricating data, but 28% of respondents claimed to know of colleagues who engaged in questionable research practices.”
Study design itself leads to incorrect results, as well - but this has been a problem since the beginning of academia itself. There are many, many problems with treating the work of academics as dogma.
Unfortunately, that’s what we tend to do. An appeal to authority is both comforting as an argument and a famous logical fallacy. The work of academics is important and academia itself adds massive value to society, but pundits and politicians bestow an unassailable quality to most of their conclusions.
“Economists agree” is a phrase common in the modern political lexicon, one that’s used to wave away objections to a separate group of people with more education and credentials. It’s not always the fallacy that the argument to authority would imply, but it’s used in much the same manner. Both Republicans and Democrats have employed the phrase freely, and it’s used by non-politicians as well; on issues like tax cuts, school choice, and even the question of how to measure a consensus.
Inherent in this phrase is that there is debate-ending value to what a group of academics believe on a subject, and that these people are unbiased arbiters of truth. Unsaid is what motivates the academics. Paul Krugman is fond of saying that “the facts have a liberal bias.” He might be right. Blank-slate unbiased truth-seekers might be drawn to academia and then, once they’ve really delved in-depth on the facts, declare themselves liberals due to the overwhelming nature of the evidence. Academics self-identify as liberals at a higher rate than any other profession.
There are other explanations. Moral theorist Jonathan Haidt posits that people are not machines of meta-rationality, and they bring prior intuitions into any area of study. While academia might attract personality types that are more amenable to introspection, that doesn’t meant that academics are immune to this universal human trait. Just like everyone else, they arrive into their academic career with predisposed notions of how the world works. Some of them can re-examine themselves and change with evidence. A lot of them do not.
This matters because policymakers in Washington, D.C. often outsource to academics and other experts. What’s more, there’s a strong urge to claim that there’s a widespread academic agreement on a topic that that is both hotly debated and prone to fluctuation.
Take the letter signed put together by the Economic Policy Institute and signed by 73 economists - five of them Nobel laureates - urging Congress to pass Barack Obama’s proposal to raise the minimum wage. It’s a powerful argument for a lawmaker to say that they’ve got five Nobel Prize winners among dozens of other economists who think that President Obama’s proposal is optimal policy in this political environment. But it’s likely that many of these economists have based their reasoning on flawed research, and many more are looking through a lens of ideological priors that have colored their take on the evidence.
This isn’t a partisan issue. More than 80% of economists have been found to favor free trade and raising the retirement age. Ideological priors and value judgments enter the debate before anyone puts a pen to paper to write a study, regardless of the outcome being agreeable to any partisan.
What this means is that there’s a high bar to clear. This is not to deny that a true consensus on divisive issues cannot be achieved by academics. Perhaps it’s at some kind of 80% line (and perhaps global warming - an elephant in the room when it comes to this column and much conservative distrust of academics!), or perhaps it’s lower. But the emerging field that casts doubt on the conclusions of a lot of academic research, combined with a Haidt-informed view of ideological priors and what we already know about bias and survey design, means that we must be aware of a finger on the scales. Beware politicians waving bare-majority “consensus” surveys of scientists, or letters signed by an impressive-sounding number of academics.
But all of that? It might just be my priors talking.