Don’t Mistake Cruelty for Rigor in Peer Review
Democracy has been called the least worst system of government. Peer review is the least worst system for assessing the merit of scientific work.
Peer review is the written evaluation of a paper by other experts in the field. Though this sounds like assessment by equals, the power imbalance created by the roles of reviewer and reviewed distorts the relationship and affects the tone of the review. Reviews can be patronizing, demanding and unkind.
It is painful to read harshly worded criticism of work that has taken a team hundreds or thousands of hours and been submitted hopefully and in good faith. From our experience, we know that reviews can be accurate, robust and make every scientific point while using language and tone that is helpful and supportive.
Supportive review
We are a team of editors of an open-access Canadian kidney journal, the Canadian Journal of Kidney Health and Disease. When we founded our journal in 2014, supportive review was the first of our guiding principles. Since then, we have written supportively as editors, selected reviewers who write supportively and participated in training the next generation of Canadian kidney scientists to conduct reviews that are complete, rigorous and kind.
Supported by a larger group of like-minded people from multiple disciplines, we recently published an editorial outlining these principles. A dozen other kidney journals expressed their support for the idea, with Nature Reviews Nephrology, NDT and Pediatric Nephrology publishing co-ordinated editorials recommitting to principles of constructive criticism.
The long process of research
Scientific papers condense a large amount of work into a structured format, usually no longer than four to eight times the length of this article. The work of a paper starts with an idea that may be developed by the team for a year or more before it crystallizes into an application for funding, which may go through rounds of revisions.
Once funded, people and budgets are assigned to the project and the work proceeds. The work can involve the time of multiple team members for months and even years.
When the work is complete, they write a paper, detailing what they did, how and why, what they found and what they think it means. This paper itself is often the product of hundreds of hours of work, with multiple authors contributing their specific expertise and working on the messaging of the whole.
The journal receives the manuscript and assigns an editor, who assigns peer reviewers. Peer reviewers are other scientists working on similar topics. They must be totally unconnected with the people writing the paper. With notable exceptions, most journals employ single-masked peer review: the reviewer sees the authorship of the paper but the authors of the paper will not see who wrote the review.
Peer reviewers are not paid or rewarded for their review of the manuscript — they take it on as part of the work of academic life. Essentially, it is an unrewarded activity performed by people who are themselves authors. It varies by discipline, but in biomedicine, they may spend three to six hours on a review.
Harsh reviews
How does this altruistic activity, undertaken by a reviewer who is very familiar with the author role, lead to such pain and frustration for other authors?
We think that scientists sometimes confuse harshness with intellectual rigor and that a reviewer’s experience of harshness in reviews of their own work, amplified by the power imbalance between reviewer and reviewed, leads to perpetuation of harsh and unhelpful review. Other reviewers and editors avoid these pitfalls entirely.
“It looks to me like one of your first attempts at scientific publishing, and I can understand that you are also writing in a non-native language,” wrote one anonymous reviewer to a mid-career woman scientist with 13 first-author peer-reviewed publications. “I just want to give up today,” she wrote.
But she won’t. Scientists are prepared to receive this kind of feedback and be hurt over and over in the name of science. As editors, we believe there is a better way — that feedback should be rigorous, but will be more readily incorporated if kindly given, to the advancement of science.
These are not new ideas. In 2006, professor Mohan Dutta suggested 10 commandments for reviewers, all of which focus on the collaborative nature of relationship between reviewer and reviewed. Advice for reviewers often includes a recommendation to write constructively, though sometimes this is phrased as something like “write constructively, and then turn to criticism,” as if those are mutually exclusive.
We can take this principle further and — thanks to our community of reviewers in kidney medicine — we and other kidney journals make a commitment to kindness in review. Dutta’s 10th commandment is “do unto others as you would have them do unto you.” Every branch of science would be improved by implementing this idea.
