A Primer for the Beginning Peer Reviewer
(Photo: Niabot/CC BY-SA 3.0)
The first time I was asked to review a paper, I was extremely honored (Finally! I am a Real Scientist!) but not sure what to do at all. There were not yet a lot of resources available online, it had not been part of my schooling, and I did not have a lot of experience in how to critically read a paper. So I started out by pointing out some simple errors or typos. It took years of practice to be comfortable enough to suggest more serious edits to other people’s manuscripts, such as flaws in the design of the study, lack of controls, and over-interpretation of results. Even now, many years and about hundred peer reviews later, I am still not always sure if my reviews strike the right balance between being critical and fair. But I definitely do my best!
This post by Elisabeth Bik originally appeared at the Publons blog under the title “Peer review essentials for the beginning peer reviewer.” Publons’ mission is to speed up science by working with peer reviewers, publishers, and research institutions to make peer review a rewarding activity.
How to critically read a manuscript
So how do you write a good peer review? To help the inexperienced peer-reviewer, I’ve made a list of general questions to ask when you are reading the paper. Asking these questions should help you form an opinion about the paper, even if you have no idea where to start. It’s the list that I wished I had access to when I started my first peer review. Here we go:
- Do you have a conflict of interest when reviewing this paper? Do you collaborate with these authors, are they your personal friends, or are they direct competitors? Have you reviewed (and rejected) this paper before? If so, you need to decline this peer review and let the editors know.
- Do the title and abstract cover the main aspects of the work, would it spark interest to the right audience?
- Is the Introduction easy to follow for most readers of this particular journal? Does it cite the appropriate papers? Does it provide a hypothesis or aim of the study?
- Does the Methods section provide enough details for the general reader to repeat the experiments?
- If you skip the Methods, does the Results section give the right amount of detail to understand the basic details of the experiments?
- Do the Results refer to the figures in a logical order? Do the numbers in the tables add up correctly? Are any figures/tables mislabeled or unclear?
- Given the data that was obtained in this study, did the authors perform all the logical analyses? Did they include the proper controls?
- Does the Discussion address the main findings, and does it give proper recognition to similar work in this field?
- In general, is the paper easy to follow and does it have a logical flow? Are there any language issues?
- Did the authors make all their data (e.g. sequence reads, code, questionnaires used) available for the readers?
- Is this paper novel and an advancement of the field, or have other people done very similar work?
- Finally (and hopefully you will never have to answer yes to any of these question): Does the paper raise any ethical concerns? Any suspicion of plagiarism (text or experiments), duplicated or tampered images, lack of institutional review board approval, unethical animal experiments, or “dual use of research concern”?
Writing the peer review
As I am reading the manuscript for the first time, I will have a text editor open in which I immediately write down small comments on specific parts of the manuscript, such as a typo in line 15 or an unclear sentence in the introduction. While I go through the paper, I will start to write down more general thoughts as well, such as remarks about the length of the introduction or a misinterpretation of results. After reading the whole paper, I will then re-read the abstract to see if it correctly captured hypothesis, experiments, results and interpretation. At the end of my read-through, I try to structure my peer review into three parts.
- Summary: A couple of sentences describing what the authors did, in my own words. This is especially helpful to refresh your memory when you will be asked to do another round of review on a paper you already reviewed before. Here, I will also give a general opinion about the paper, without mentioning if I think it should be rejected or accepted with edits.
- General comments: Some broad thoughts about the paper, such as: novelty of the findings, length of discussion, order of results, any concerns about data analysis or interpretation, language issues, etc. I will usually have about three to five numbered bullet points here.
- Specific comments. A numbered list of issues that refer to a very specific portion of the text or figures. Here, I might point out typos, missing definitions or abbreviations, unclear sentences, a missing reference, or suggestions to improve a figure or table. Usually, this part of my review will have about 20 remarks (but sometimes much more!). Even if this part is a long list, most of these points should be very easy to address by the authors.
It is important to number your remarks, making it easier for the authors to respond to each one of them.
In your review, the most important thing to keep in mind is to remain friendly and reasonable. You should feel no regret publishing your review under your full name. On the other hand, you do have the right to ask the authors to make primary data publicly available, perform some small and easy additional experiments or analysis, or change the layout and order of their graphs. Depending on the scope of the journal, it is however not reasonable in most cases to ask the authors to do large amounts of additional work. If you think the science is good, it should be published. There is always a need for additional experiments, but that can be put into another paper.
Submitting the peer review
Once you have written your review, you will have to upload it into the journal’s reviewer interface. Most of them will have a box where you can assign the paper to one of 4 categories: accept without edits (only to be selected if you reviewed The Perfect Paper!), accept with minor edits (addressing typos, unclear sentences, or a small figure edit), accept with major edits (addressing bigger issues such as changes to introduction scope, interpretation of results, additional graphs or analyses) or reject (if the manuscript was not novel at all, not suitable for the scope of that journal, or contains plagiarism or other questionable practices).
Most of the papers I have reviewed were classified as “accept with major edits”; I have selected the “reject” category less often. On the journal’s website, there is usually also a box where you can give specific comments to the editor; these will not be forwarded to the authors. Here you can state your personal opinion, or any issues you do not want to share with the authors. Your peer review will go into the section labeled “comments to the authors” – often by simply copy/pasting it into the appropriate box. Even if I think a paper should be rejected, I like to share my thoughts about the manuscript here, so that the authors can improve their paper before submitting it somewhere else.
Managing peer review requests
Which and how many peer reviews should you perform? Most junior scientists will start to get peer review requests after publishing their first paper as a first author. Often, you will get asked if one of your papers is listed in the references of the manuscript, so the new study will be in your field. You should only accept the peer review if you feel you have expertise in the topic of the paper, even if you are not an expert in all the techniques used. Typically, a manuscript will be sent out to about 3 reviewers, so as a rule-of–thumb you should perform three times more reviews than the amount of manuscripts you typically submit per year. More is better!
Once you have peer-reviewed for a journal, the journal will ask you again, but usually only once or twice a year. The more papers you have published, the more requests you will get. You do not have to accept all of them, though. I’ll try to only have two ongoing peer reviews at the time; if I get more requests, I will turn them down until I have finished the previous ones. However, I will always accept requests to re-review a manuscript for the second time, and give priority to requests from journals where I serve in the editorial board.
Publons has been a great resource for me to keep track of all my peer reviews. It is a nice way to get recognition for all the work we peer reviewers do, mostly anonymously. In addition, it is great to compare my acceptance rate and length of peer reviews to that of others, and to have a feel for how many reviews other scientists perform, and for which journals. I can highly recommend signing up!
Peer review is a great way to become a better scientist. By regularly doing peer reviews, you will start to think like the peer reviewer who will criticize your paper, so it will make you a better writer. It is an essential part of being a scientist. And, to be honest: it can feel really great to be in the power seat sometimes! But, remember to always stay friendly and polite.
