Literature Reviews Are Already Broken, So Let’s Kill Them

There’s surely some good stuff in these. Shame I can’t be bothered with characterizing it accurately.

I use an application that crawls the web to notify me whenever my name is newly mentioned on an accessible web page. Like others using the app, probably, I had initially hoped it would deliver good news, such as favourable (or, at least neutral) references to my publications. More often, however, the news is bad. A substantial proportion of the notices inform me of yet another Russian website offering one of my books for easy download, either for free or for a small payment (to them).

Other bad news arrives when other researchers reference one of my publications, but misrepresent it. Recently, three authors uploaded the galley proofs of an accepted journal article. Perhaps they violated journal policy in doing so, but their advance posting allowed me the opportunity to check their representation of my work in their literature review. The authors’ lit review was of a too-common type. It alleged serious shortcomings in all the previous works they reviewed, while they bragged that their work was superior in every respect. In the history of the world, no one had done as good a job with the topic as they had just done.

This article by Richard P. Phelps originally appeared on the LSE Impact of Social Sciences blog as To save the research literature, get rid of the literature review and is reposted under the Creative Commons license (CC BY 3.0).

Problem was, their characterization of my work was completely wrong. I contacted the authors to complain, and heard back from one. He justified their comments by asserting that he and I interpreted the meaning of a key word differently. I responded that while that may have been so, I was using the standard dictionary definition of the word, and most other readers were likely to do the same.

A few months later, I saw the authors’ final version of the article in print. They had responded to my complaint by excising the part of the original misrepresentation we directly discussed in our correspondence. But, they retained the rest, which was just as false. Moreover, they added others while increasing the length of the critique eight-fold. It was an “in your face” response, as if the authors were trying to tell me that they could say whatever they pleased about my work.

And, they could. Certainly, I should have contacted the editors directly with my original complaint instead of the authors, who had already revealed a lack of goodwill in their original version of the article. But relatively few authors post galley proofs in advance of publication. In most cases, those whose work is misrepresented do not know about it until after the article is published.

At this journal, and many others, editors insist that manuscript authors demonstrate originality and provide thorough literature reviews. But, I suspect, the literature reviews themselves are not then reviewed for accuracy. Nor usually could they be. It is difficult enough for journals to find expert volunteer reviewers willing to sacrifice their time to review the study placed before them, much less familiarize themselves with an entire research literature. To really know a research literature can take years.

This state of affairs leaves literature reviewers free to write pretty much anything they please (so long as they do not happen to misrepresent the work of one of the their manuscript’s reviewers). The authors of each new journal article can trash previous work on a topic and, over time, more knowledge may be excised from the collective working memory than added. It is research cannibalism.

A quarter-century ago, I wrote a major report for a US government research bureau. In general terms, it was a benefit-cost analysis, which concluded a certain public programme would cost between US$X and US$Y dollars if implemented as proposed. My estimates did not sit well with a powerful group of researchers who opposed the programme. So, they began attacking the report and my estimates by, naturally, accusing me of having neglected to include certain cost elements, thereby underestimating.

Despite hundreds of hours of effort, numerous letters, email messages, and telephone calls on my part over the years, false claims of missing cost elements have continued unabated for a quarter century, though the identity of the alleged missing element may change. Each time I apparently manage, through considerable effort, to quash one accusation of a neglected cost element, another pops up. Naturally, each misrepresentation in the research literature feeds others each time it is cited. At this point misrepresentations far outnumber accurate representations in the research literature. At some point, however, I will die. Then, no one will be left to complain, and the more popular version of reality may assume uncontested dominance in the research literature.

Literature review falsehoods may fester and proliferate because those who can refute them are given no opportunity to do so. I, for one, have never been invited by a journal to comment on a forthcoming literature review in which my research work has been critiqued. When only one side is allowed to talk, it can say anything it pleases.

As distasteful as misrepresentative literature reviews may be, they at least bear the positive quality of effort. The laziest literature reviews are those falsely declaring previous work on a topic to be nonexistent — “dismissive reviews”. When I first began to search for the magnitude of the dismissive review problem, I expected to find hundreds; I found hundreds of thousands. I had assumed they were a problem at the margins of scholarly research, isolated among the few per cent of amoral bad pennies one finds in any profession. But, no; the problem seems to be pervasive.

You can see for yourself. Access a web database that allows searching by phrase (e.g. Google, Yahoo Search, Bing) and try some of these: “this is the first study”; “no previous studies”; “paucity of research”; “there have been no studies”; “few studies”; “little research”; or variations thereof.

I don’t believe that journals can responsibly review manuscript literature reviews for accuracy. And, because they cannot, literature reviewers are free to trash or ignore competing work. Thus, ironically, literature reviews may degrade our knowledge base more than preserve it.

I propose that journals drop the literature review requirement. Why?

• For a variety of reasons, some innocent and some not, many (perhaps most) researchers no longer conduct valid and thorough literature reviews.
• Given the huge, constantly growing size of the research literature, thorough literature reviews require considerably more time than all but a few researchers have available.
• Most journal reviewers skip past manuscripts’ literature reviews. Busy and unpaid, they cannot afford the time to review an entire research literature on their own in order to verify an author’s claims about it.
• The lack of verification of manuscripts’ literature reviews allows less scrupulous scholars to misrepresent previous research and showcase their own as better.
• It allows others to declare previous research nonexistent (or so inferior as to not be worth mentioning), thus facilitating a reduction, rather than an increase, in humanity’s understanding of the world.
• Given how easy it is to misrepresent previous research, those with the resources to publish more than others can saturate media coverage to suppress opposing ideas and rivals.

In addition, I would argue that the demand for originality in every journal article is not only unreasonable but short-sighted. Arguably, we currently have greater need of replication studies — to help weed out the large quantity of irreplicable (and probably false) published research. As at least one wise person has said:

“It isn’t what we don’t know that gives us trouble; it’s what we know that ain’t so.” (Attributed to Mark Twain)