Why Do We Still Have Journals?

The past few years have seen a massive proliferation of scholarly journals. The number of ‘‘management’’ journals indexed by the Web of Knowledge has tripled in the past dozen years, and new non-standard journals seem to emerge daily, providing readers with a continuously expanding inventory of new articles. Authors face a kaleidoscopic array of publication outlets, including traditional journals, research annuals, and online-only journals. Some formats take advantage of the Web to communicate scientific findings in new ways and with great speed, creating new possibilities for dialogue among scholars. In principle, new methods of distribution could greatly accelerate the advance of social science beyond what is possible with traditional journals published at traditional intervals.

In this essay, I argue that the core technology of journals is not their distribution but their review process. The organization of the review process reflects assumptions about what a contribution is and how it should be evaluated. Different review processes thereby create incentives for different kinds of work. Yet the shape of this incentive system and the kinds of work it promotes receive little discussion, beyond clichés about ‘‘publish or perish.’’ It’s time for a broader dialogue about how we connect the aims of the social science enterprise to the incentive systems embodied in our system of journals. This essay aims to open that dialogue.

What’s the Matter with Journals?

The traditional system of academic journals is full of pathologies. The peer review process can be slow. Even the best-managed journals are likely to take one to two months for each round of review. Editors have to assign reviewers, consider their evaluations, and render a thoughtful judgment. Authors must respond, sometimes collecting and analyzing more data. Optimistically, papers are likely to take up to a year from submission to appearing in print. In poorly run journals, each round of reviews can take months, and authors can be dragged through round after round of review, sometimes ending with a rejection. Unfortunately, this can happen in spite of the best efforts of editors and reviewers. For untenured authors, that can be hazardous to their careers.

This editorial essay by Gerald F. Davis was taken from the June 2014 edition of Administrative Science Quarterly, which he edits. The journal is owned and managed by the Samuel Curtis Johnson Graduate School of Management at Cornell University.

Access to journals can be costly. The excessive subscription prices charged by some publishers, and their excessively high margins, have led to a worldwide boycott of their journals by mathematicians. Many have questioned why authors and reviewers contribute their labor for free, while those who merely provide the infrastructure for publication reap such great financial returns.

Some journals are elitist. Randy Schekman, who won the Nobel Prize for Physiology or Medicine in 2013, used his new visibility to launch a boycott of ‘‘luxury journals,’’ particularly Nature, Cell, and Science. These journals, he argued, exercise disproportionate influence based on their elite status. Competition for their imprimatur and the resulting career rewards creates incentives for bad science. Editors of such journals can have undue leverage over the direction of the field.

The academic career system in which evaluations are based on publication tallies can encourage quantity over quality. Journals play a central role in the system of academic evaluations and career advancement. When professors are assessed for tenure on the basis of counts of articles in particular high-status journals, it creates incentives for bad scientific practice. The recent swell of retractions and scandals, well-documented elsewhere, attests to the hazards of this situation. Smart people with advanced degrees can be quite cunning in finding ways to get published, and not all of them entail expanding the stock of human knowledge.

Perhaps most importantly, the format of traditional journals seems out of sync with the rhythms of knowledge production. There is something atavistic about journals consisting of a limited number of written articles, of relatively fixed formats and lengths, published at regular intervals. To put it pointedly, why is ‘‘quarterly’’ the right interval for findings and insights about administrative science? Why not annually, or hourly, or ‘‘whenever’’? The pragmatic answer is mundane: back in the days when scholars ‘‘subscribed’’ to journals, which arrived via mail as physical artifacts, practical constraints on size mandated it. This is ‘‘an ideal system for sharing knowledge, said the quantum physicist Michael Nielsen, only ‘if you’re stuck with 17th-century technology.’’’

Indeed, the first scientific journal and the oldest newspaper in the English language were both founded in 1665, and both consisted of motley collections of, in effect, letters from correspondents. The first edition of the Philosophical Transactions of the Royal Society included articles titled ‘‘A spot in one of the belts of Jupiter,’’ ‘‘An account of a very odd monstrous calf,’’ and ‘‘Of a peculiar lead-ore of Germany, and the use thereof.’’ Newspapers at this early stage also had an epistolary flavor.

Information Technology and Scholarly Communication

New technologies of communication should enable new ways of sharing and advancing knowledge. Newspapers have been radically transformed by the Internet revolution, adapting their format to continuous updating, color, video, and opportunities for feedback and debate by readers. Yet academic journals still bear the imprints of their origins, and most look little different today than they did 50 years ago.

Perhaps the most interesting experiments have been in open access journals, such as those published by the nonprofit Public Library of Science (PLOS). Unlike traditional journals, which are typically funded by university library subscriptions that grant access to faculty and students, open access journals do not have a paywall. Instead, scholars pay to have their accepted articles posted online, making them available to the public for free in perpetuity. That is, authors rather than readers provide the revenues that fund the journal.

The review process can also diverge from tradition. At PLOS ONE, an ecumenical journal that publishes papers in all scientific fields (including psychology and sociology), articles are not evaluated on their novelty or the magnitude of their contribution. In the words of PLOS co-founder Michael Eisen, ‘‘Rejecting papers that are technically sound is a relic of the age of printed journals, whose costs scaled with the number of papers they published and whose table of contents served as the primary way people found articles of interest. But we are no longer limited by the number of articles we can publish, and people primarily find papers of interest by searching, not browsing. So PLOS ONE asks its reviewers only to assess whether the paper is a legitimate work of science. If it is, it is published.’’ Assessments of novelty and contribution happen after publication, by readers, rather than by editors and reviewers before publication. Potential readers can see how often an article has been downloaded and cited, as proxies for quality, and readers can leave comments.

Since its start in 2007, PLOS ONE has published nearly 100,000 articles on a vast array of scientific topics, including 3,000 in the domain of sociology (broadly construed). Articles can be sorted by recency or popularity. Perhaps unsurprisingly, the most popular articles are often the most sensational—for example, one asserting that, in contrast to much prior research, vegetarianism is associated with cancer, allergies, and mental illness, which was viewed an astonishing 75,000 times in its first two months.

The rationale behind PLOS ONE is persuasive, and the journal is a noble experiment. In some sense, PLOS ONE is not so much a journal as a platform for sharing articles that have surpassed a minimum threshold. Given that reviewers are assessing only the technical competence and not the level of novelty or insight of submissions, it is heavily reliant on what happens after publication to identify substantive contributions.

One could imagine a model of science in which all articles in all fields are published on PLOS ONE and then the necessary sorting for insight and contribution happens afterward. We could, in principle, dispense with traditional journals entirely. It is not obvious, however, how the process for separating out the breakthrough articles from the quotidian or the genuinely flawed might happen after the review process is over and the paper is in print (virtually), at least under the current academic incentive system. High-quality interlocutors seem unlikely to engage such papers after publication the way reviewers would engage them before publication, and authors are under no obligation to respond. But there is certainly room for many formats of journal publication today.

The Proliferation of New Journals

The low barriers to entry for online publishing have led to a vast proliferation of new journals that do not necessarily share in the idealism of the open access movement. In 2001, the Institute for Scientific Information indexed 61 ‘‘management’’ journals in its citation analysis. By 2013 this number had almost tripled, to 174 journals. More are added every year. Some journals publish a small number of articles; for instance, ASQ publishes roughly 20 papers per year. Others operate on a far larger scale, publishing 100 or more articles per year. According to the Web of Knowledge, the

Journal of Business Ethics (which is not indexed as a ‘‘management’’ journal) typically publishes over 300 articles per year, and in 2009 it published 536 papers. Together, the 174 ‘‘management’’ journals published roughly 34,000 articles between 2010 and 2013, or more than 8,000 per year. Beyond these journals, of course, there are hundreds of other journals that publish management-relevant research. These range from respected outlets in disciplines such as psychology and sociology to sketchy for-profit ‘‘vanity’’ players enabled by the near-zero barriers to entry in creating an online journal today. (Many readers are familiar with the deluge of emailed offers to submit to these journals or serve on their editorial boards.)

The proliferation of journals and other outlets has greatly exacerbated the problem of finding the right works to read. A search on the phrase ‘‘corporate governance’’ at SSRN.com yields 10,000 papers; clearly no human being could read more than a tiny fraction of this output. On the Web, articles all look more or less equivalent. There is no special font to signal quality or rigor. Those outside the field may not be aware of the distinction between peer-reviewed and non-peer-reviewed journals, and even the category of peer review includes a wide variety of practices.

There are several competing lists and rankings of journals that purport to distinguish journal quality and that might help readers narrow their search. ISI’s annual Journal Citation Report, particularly the ‘‘Impact Factor,’’ receives the most attention within the field, reporting the average number of times articles published in the previous two years were cited in a given year. In addition, lists such as the Financial Times 45 are used to distinguish ‘‘elite’’ journals for the purposes of evaluating the intellectual output of business schools (the FT 45 list includes the prolific Journal of Business Ethics). In many countries, schools and funding agencies have created criteria for evaluating faculty that rely on journal impact. Funding, careers, and status are at stake in journal evaluations. Some institutions pay a bounty for faculty to publish in particular journals; the savvier institutions might even pay a sliding scale based on the journal’s status.

It isn’t surprising that this system has led to its own pathologies. Authors have on occasion resorted to non-standard techniques to publish and inflate the impact of their work. The traditional methods of fraud are well known, such as faking data, post-hoc hypothesizing, and so on. Practitioners of this dark art occasionally end up as the subject of cover stories in the New York Times Magazine. Inventive ‘‘authors’’ have discovered that papers consisting entirely of computer-generated gibberish can end up being published. Leading social scientists have recently proposed standards for combating fraud and enhancing the transparency of social science, and such efforts are gaining traction.

Slightly less well known are techniques for increasing indicators of impact. SSRN.com ranks papers based on the number of downloads, and it guards against multiple downloads from the same IP address. Some scholars include download figures from SSRN on their CVs. But it is not difficult or costly to use Amazon MTurk to hire a few hundred people to download each of one’s papers from diverse IP addresses.

Google Scholar has become a widely used alternative to Web of Knowledge and serves as the default portal to the literature for many younger scholars. But there is now a cottage industry of scholarly papers documenting methods for gaming Google Scholar. One paper created a fake author, Ike Antkare (‘‘I can’t care’’), and turned him into one of the most cited scientists in history, according to Google Scholar. A second, titled ‘‘Manipulating Google Scholar citations and Google Scholar metrics: Simple, easy and tempting,’’ serves as a how-to guide and cautionary tale. The ‘‘Scholarly Kitchen’’ blog is an excellent source of insight into these schemes.

Journal editors have also used untoward methods to enhance their journal’s status. One method is for editors to publish ‘‘survey’’ articles in other journals indexed by ISI to cite many or most recent articles in their own journal. Another is to create citation cartels in which journals mutually (and gratuitously) cite each other’s recent articles. A third is for editors to serve as guest editors of other journals and induce authors to cite the guest editor’s home journal. All of these methods serve to spuriously inflate a journal’s Impact Factor and to render it increasingly sketchy as a valid indicator of scholarly merit.

These methods have been documented to account for the astounding rise of one mathematics journal. It may not be surprising that this happens in management journals as well. Interested readers are encouraged to consult the Web of Knowledge’s Journal Citation Report and drill down on the source of some highly ranked journals’ citations. I was quite surprised to discover unexceptional articles in unheralded journals that had been cited more often than Watson and Crick’s paper reporting the discovery of the structure of DNA. The citing articles often included text along the lines of ‘‘Management is an important topic’’ followed by parenthetical citations to a dozen recent articles in the offending journal.

>

The low barriers to entry for online publishing have led to a vast proliferation of new journals that do not necessarily share in the idealism of the open access movement

Perhaps all of this is the predictable outcome of a system that rewards scholars for publications and quantifiable impact when quality is hard to judge on its own. Authors and editors have incentives for trickery, and these have been amplified by efforts of funding agencies to quantify and reward research output. Certainly, as long as using Google Scholar and ISI is easier than actually reading papers, academic career incentives may not be well aligned with creating meticulous and high-quality work.

The Place of Journals in the Enterprise of Social Science

After this brief and selective summary of some of the problems of social science journals, it would be easy to be cynical. Largely absent from discussions about journal pathologies, however, is a strong sense of the place of journals in the enterprise of social science. Why do we have scholarly journals anyway?

Journals are vehicles for preserving, communicating, and advancing ideas and findings. Journals are a particular artifact for conveying knowledge, and they reflect their Baroque-era pedigree. In a Web-enabled world, there are other ways that scientific communication might be accomplished, and there are alternative models out there. One might post all of one’s papers on a webpage, or deposit them in a non-reviewed online archive, updated as appropriate, or publish everything that makes it through the review process on PLOS ONE, where post-publication commentary is welcome and impact is judged afterward.

More broadly, the range of alternative vehicles for conveying ideas and findings is vast, from monographs and edited collections to journal articles and conference proceedings to blog posts and tweets. Different vehicles have different affordances, and it makes sense to match the vehicle to what is being conveyed. Das Kapital took three volumes; Watson and Crick’s discovery of the structure of DNA was conveyed in 12 short paragraphs on two pages of Nature (and properly edited, it might have made a good tweet). What, then, are the distinctive affordances of traditional journals? Why do authors, reviewers, and editors contribute so much cognitive and emotional energy to keeping this thing going?

At their best, journals accomplish three things: certifying, convening, and curating. Certifying is what the review process does, validating articles as having made it through a vetting process (however organized). Convening means that specific journals are able to bring together interested and engaged scholars in a way that the abstract endeavor of organizational scholarship cannot. The membership of the editorial board reflects a journal’s ability to attract the voluntary and mostly anonymous labor of outstanding scholars. Ideally, scholars will regard a journal as a community (but not a club). Curating suggests that what is published in a particular journal is likely to be worth reading. In a field in which 8,000 or more papers are published every year, it is helpful to have the assurance that papers in a specific journal will be worth your time.

Journals can also serve a civilizing function. Through their editorial practices, journals can enhance the legibility of arguments and findings. Graduate programs rarely teach students how to write well, and good scientists are not always good writers. (Many of us believe that our Stata output ought to speak for itself and that the words surrounding the tables are mostly ornamental.) Although many journals have dispensed with the close editing of articles entirely, those that continue to do so serve a civilizing function by training new authors in how to write for an audience.

The review process is what ASQ’s founding editor James D. Thompson would call a core technology for journals. Journals organize the review process in many different ways, reflecting assumptions about what the journal is trying to accomplish and what qualifies a paper to be published. Here are a half-dozen possibilities that I have seen (in various combinations) at different journals: (1) Accuracy: papers have a true intrinsic value; the goal of the review process is to identify those whose value is above a particular threshold; (2) Impact: the value of papers is uncertain ex ante; the goal of the review process is to identify those likely to be highly cited; (3) Development: the value of papers is altered by the review process itself; the goal of the review process is to identify promising papers and make them good enough to end up in print; (4) Innovation: papers exist to advance the state of the field through new methods, new findings, new insights, new theory; the goal of the review process is to distinguish the innovative from the mundane and the merely wrong; (5) Keeping score: papers are markers of achievement in the academic careers of their authors; the goal of the review process is to provide a reasonable judgment while minimizing the trauma to the author; and (6) Community: papers are convening devices for a community of scholars; the goal of the review process is to inform and refine the taste and judgment of the participants in the scholarly enterprise. These are not mutually exclusive, and different journals emphasize different combinations of values. It is fair to say that there isn’t broad consensus in the field around which of these are the right values.

How we organize our journals shapes the incentive structures of the field. The design question is how to structure journals so as to advance the field. How do we create a journal ecosystem that encourages field-advancing research? This is a big question, worthy of collective discussion.

Implications for Journals

The organization of the review process reflects a view of what research should look like. It encourages some kinds of work and discourages others. For instance, a single-round review process, as is typical at PLOS ONE, fits well with a view of articles as having an intrinsic merit, such as straightforward empirical documentation of a regularity. The review process can be a simple hurdle: pass or fail.

ASQ’s review process, in contrast, works well for developing work that highlights theoretical contributions or puzzle solving, in which insights improve through back-and-forth exchanges with reviewers. Qualitative work, theoretical work, or studies unpacking mechanisms may be more likely to benefit from this format. Conversely, such work may not reach its potential in an up-or-out review process.

Implications for Authors

In light of the fact that there are many outlets available with different formats, aimed at different audiences, authors should calibrate their publication strategies to the types of contributions they are aiming to make. Different kinds of articles fit with different kinds of outlets. Charles Perrow (1985) described this in his article ‘‘Journaling careers,’’ which provided an astute assessment of how and when to place articles in different types of journals and books. For innovative theoretical contributions intended to stand the test of time, such as dissertation articles, ASQ may be a good choice. For primarily empirical pieces and new findings in which speed of publication is essential, PLOS ONE may be a good outlet. For more speculative or theoretically adventurous work for which solid empirical grounding is lacking, edited collections will be a better bet than journals.

Implications for the Field

What kind of social science do we want? At the level of the field, there is room for many kinds of contributions, and it is reasonable for journals and other kinds of outlets to have a division of labor. But it is worth being cognizant of the incentive structures created when different review processes articulate with different systems for evaluating academics. Our vision of progress in the social sciences needs to be reflected in how we organize our journals.

In spite of their shortcomings and the proliferation of alternative methods of scholarly communication, traditional journals still serve essential purposes in the scientific enterprise. They can certify contributions, convene scholarly communities, and curate works that are worth reading—sometimes by helping scientists become authors who know how to write for an audience. At their best, they are not just neutral vehicles for conveying findings, but active forces for advancing scholarship. For now, they are still indispensable.

***

Acknowledgments: This paper benefited enormously from conversations and data analyses with Hyunzoo Chai and Johan Chu. I am grateful for their insights. I also thank Chris Marquis for his thoughtful comments.