Is Peer Review an Achilles Heel for Interdisciplinary Work?
Is competent peer review a missing piece in the puzzle of getting interdisciplinary proposals greenlighted?
This article by Gabriele Bammer originally appeared on the LSE Impact of Social Sciences blog as “Why are interdisciplinary research proposals less likely to be funded? Lack of adequate peer review may be a factor” and is reposted under the Creative Commons license (CC BY 3.0).
They cleverly applied a technique from evolutionary biology that examines relatedness between biological lineages, using a hierarchical classification of research fields rather than an evolutionary tree. The relative representation of different field of research codes and their degree of difference were used as a proxy measure for interdisciplinarity. The results, based on five years of data from the Australian Research Council’s Discovery program, are robust and are unaffected when number of collaborators, primary research field and type of institution are taken into account.
What does it mean? Both main potential interpretations raise concerns and require action. The results may support the grumbling of interdisciplinary researchers about an unfair review processes, where reviewers with no interdisciplinary research experience hold sway over which interdisciplinary research is published. It is also possible that interdisciplinary research proposals are not as good as those which are discipline based.
What is required for adequate peer-review of interdisciplinary grant applications? I recently laid out the key issues in a paper for Palgrave Communications. This paper argued that only involving reviewers from the disciplines was likely to significantly short-change an interdisciplinary proposal.
First, the unknowns (expressed as research questions) that are addressed in interdisciplinary proposals may include issues of concern to disciplines, but are unlikely to be confined to those. Interdisciplinary research is also likely to address unknowns that worry stakeholders and unknowns that are central to the problem, both of which may well be outside the purview of disciplines. For example, in research I conducted on the feasibility of prescribing pharmaceutical heroin to dependent heroin users as a new treatment option, we addressed concerns raised by police, one of the major stakeholders, about the possibility that heroin prescription would cause a “honeypot effect”, an issue that was not on the radar of any of the disciplines involved. Similarly we addressed concerns about fostering more permissive attitudes to illicit drug use, which are integral to changing prohibitionist policies. Such issues may seem irrelevant to discipline-based reviewers and certainly cannot be adequately assessed by them.
Further, the discipline based unknowns that are relevant to the interdisciplinary research, such as estimating the number of heroin users (demography), assessing the ethics of heroin prescription (philosophy), and estimating the likely impact on the illicit drug market (economics) may seem pedestrian to discipline-based researchers, even though they are critical to assessing an interdisciplinary problem such as the feasibility of heroin prescription. Indeed in this project only the economists undertook research that was ground-breaking from a disciplinary perspective.
Discipline-based researchers are also ill-equipped to evaluate the integrative processes that an interdisciplinary proposal plans to use, such as assessing the systems view taken, the rationale by which disciplines and stakeholders were chosen for consideration in the study and how all the different insights will be combined. Along with the different unknowns addressed, integrative methods are what set interdisciplinary research apart.
These considerations are relevant to one kind of interdisciplinary research, where several diverse disciplines and stakeholders tackle a complex real-world problem. But there are also other kinds of interdisciplinary research, ranging from one person borrowing from other disciplines to tackle a problem in a new way (as Bromham and colleagues did in their paper), to teams working at the interface of two or more disciplines (which may lead to the development of a new discipline, as when quantum physics and biology combine to form quantum biology), to closely related disciplines converging to develop a new technology. Different peer review processes may well be required to do justice to these different kinds of interdisciplinarity.
At the moment these issues are in the funders’ too hard basket. And who can blame them? Apart from complaints about the inadequacy of current processes, the interdisciplinary community is too unorganized to suggest adequate ways forward. Key tasks include providing a usable definition of interdisciplinarity (that encompasses the main kinds), highlighting central issues for peer-review of each kind, and providing an adequate college of peers to undertake effective peer-review.
And what if the results by Bromham and colleagues result from the quality of the interdisciplinary research proposals submitted for funding not being of a high enough standard? That’s also the purview of the interdisciplinary community. Just as the disciplines police their own standards, interdisciplinarians need to make clear which parameters mean that interdisciplinary research is outstanding and which mean it is poor.
If we don’t organize, our research will always be judged by outsiders, who are less well equipped to do so than we ourselves are.
