How Few Papers Ever Get Cited? It’s Bad, But Not THAT Bad

It’s an academic wasteland at times: Four out of five papers in the humanities never get cited, according to one study. (Photo by Jez Arnold)

This article by Dahlia Remler originally appeared on her personal blog, DahliaRemler.com, under the title “Are 90% of academic papers really never cited? Searching citations about academic citations reveals the good, the bad and the ugly.” It appears here with her permission.

That lack of support was a problem for me. I did not doubt the basic truth that many academic papers are uncited. But to be sure 90 percent was not urban legend and to learn the context and caveats, I needed to find the original research paper. I was not the only one who wanted the supporting evidence.  So, I dove into Google scholar, searching the disparaged academic literature for articles on academic citation rates.

What’s the truth?

Many academic articles are never cited, although I could not find any study with a result as high as 90 percent. Non-citation rates vary enormously by field. “Only” 12 percent of medicine articles are not cited, compared to about 82 percent (!) for the humanities. It’s 27 percent for natural sciences and 32 percent for social sciences (cite). For everything except humanities, those numbers are far from 90 percent but they are still high: One third of social science articles go uncited!

Ten points for academia’s critics.

(Before we slash humanities departments, though, remember that much of their most prestigious research is published in books. On the other hand, at least in literature, many books are rarely cited, too.)

The uncited rate is also sensitive to other factors: how long a window is used to check for citations (e.g., five years); when the article whose cites are being counted was published (2000s or 1990s); and what counts as a citation. The uncited rates I gave as “the” rates are really five-year citation rates in all Thomson’s Web of Science journals, and that is not comprehensive. The details of whether to include self-citations, non-academic articles, and so on, also matter.

Another reason for the various uncitedness rates floating around is confusion between the average citation rates of journals and citation rates of articles. Within a given journal, some articles have many citations while others have few and many have zero—citations within a given journal are skewed.  The average rate of citations for a whole journal, the impact factor, is pulled up by the few articles with many citations. Focusing on the impact factor will make it seem like more articles get cited than actually do. Ironically, a Chronicle of Higher Education article bemoaning the low rate of citations under-stated its case by assuming the average citation rate for journals applied to articles.

Clearly, academic articles have a serious problem. But my experiences highlighted another bad thing about academic articles—and a really good thing.

I had a hard time finding the rates at which articles were uncited, because the overwhelming majority of relevant articles were about other things, such as the effect of time windows, different inclusion criteria for citations, whether the Internet has changed citation practices and so on. Those are all good things to investigate, but in the grand scheme of things, they are not as important as the large share of articles going uncited altogether. Another point for academia’s critics, who contend that academics worry about small things no one else cares about and miss the big things.

But my experience also showed what’s great about academic articles. You get to learn how people reached their conclusions and judge the methods yourself. You also get the assurance that knowledgeable people paid attention to how things were done and the validity of the conclusions. Contrast the accuracy and information in the academic articles that I have linked to with the figures from non-academic outlets that darted around the Internet.

For everything except humanities, those numbers are far from 90 percent but they are still high: One third of social science articles go uncited!

And what about that 90 percent figure? It came from an article by an expert in citation analysis, Lokman Meho, then at Indiana University, in Physics Today, a (journalistic) magazine associated with the American Institute of Physics. No wonder it got (inaccurately) described in cyberspace as a “study at Indiana University.”

Meho explained the 90 percent by email, “The first paragraph of the article was written by the editor of the magazine and not me. If I recall correctly, he got the figures from/during a lecture he attended in the UK in which the presenter claimed that 90% of the published papers goes uncited and 50% remains unread other than by authors/referees/editors.” Meho noted that the 90 percent figure was about right for the humanities but not other fields.

Five points for academia’s supporters. No editor could do anything remotely like that to an academic article. (It’s bad journalism too.) In academic articles, all methods are explained and all claims are supposed to be evaluated by other experts.

Academic publication needs fixing. Even the 12 percent uncited rate for medicine seems large to me, particularly given what medical research costs. The one-third rate for social science and more than 80 percent for humanities are really troubling. But whatever we do, let’s preserve somewhere what’s good about academic articles — full descriptions of methods and expert evaluation.