NIH: Exploring the Publication Gap Between Social/Behavioral and Biomedical Research

[William T. Riley is the director of the National Institutes of Health’s Office of Behavioral and Social Science Research. The article in PLOS One was co-written with Katrina Bibb, Sara Hargrave and Paula Fearon.]

In November 2020, we published an article in PLOS ONE on “Publication rates from biomedical and behavioral and social science R01s funded by the National Institutes of Health.” In that article, we reported that the percent of R01s – a standard grant mechanism in NIH — with zero publications five years from the project start date was low overall (2.4 percent) but higher for behavioral and social sciences research (BSSR) grants (4.6 percent) than for non-BSSR grants (1.9 percent). We also reported that the time to first publication was slower for BSSR than for non-BSSR grants. Differences remained even after controlling for factors such as conducting human research or clinical research. 

We did not generate this research question about publication differences between BSSR and non-BSSR R01 grantees de novo. NIH has been concerned about publication productivity, particularly for clinical trials, for some time, and the NIH clinical trials policies were developed in part to ensure that results reporting occurred, regardless of null results or publication bias. More recently, NIH is performing an ongoing quality review of its Center for Scientific Review study sections. This review, called ENQUIRE, includes, among the metrics considered, the productivity and impact of the research published from the grants reviewed and funded by each study section, specifically the number of grants within a study section that fail to produce a single publication in a timely manner.

Early analyses of some of these study sections showed a greater number of grants with zero publications among study sections that review mostly BSSR grant applications relative to other study sections. We hypothesized at the time that BSSR grantees would understandably take longer to publish given the large proportion of human research and longitudinal clinical or applied research relative to our biomedical colleagues. That hypothesis, however, begged for research to test this hypothesis, leading us to analyze the publication productivity of grants funded by the NIH from 2008 through 2014.

We chose to focus specifically on R01 grants, the most common grant mechanism of the NIH, because, given the preliminary data required and the level of investment provided by this award, the likelihood of failing to produce a single publication should be quite low. As expected, the rate of zero publications from a grant within five years from the project start date was low (2.4 percent). This rate is much lower than found in prior clinical trials publication research, but clinical trials research, by its human research and longitudinal nature expected to take longer to publish, and the outcome variable in most of these prior studies with the primary outcome paper, not any paper associated with the grant as it was in this analysis. While we do not know what the optimal rate of zero publications within this time period should be, for the $3 million to $4 million investment of the typical R01, it is reasonable for the taxpayers and their representatives to question why any R01 grant does not produce a single scientific publication within five years of funding. 

Of course, there are many legitimate reasons why this might be the case, and why the rate of zero publications and the lag to first publication is slower for BSSR vs. non-BSSR grants. These include:

• Some research does not produce publishable findings. Research plans do not usually go as planned. Risky research may not be feasible, and publication bias against null findings remains despite efforts to reduce such bias. Yet even “failed” projects provide useful insights for the field, hence the NIH efforts toward required registration and results reporting. 
• Some research takes longer than others to complete and publish. Research involving humans takes longer than research involving cells, and longitudinal studies take longer than cross-sectional studies. Our analyses could account for only a few of these factors (e.g., clinical trial or not, human research or not, child research or not) due the constraints of the data available in the NIH grant database, and these factors were shown to be important contributors to the time lag to first publication, but they did not fully explain the difference in publication lag times between BSSR and non-BSSR grants.
• Scientific disciplines have different publication “cultures”. Economics and social sciences tend to publish less frequently but with lengthier publications than their biomedical colleagues. Computer scientists place a higher value on scientific communication via conference presentations than via journal publications. 
• Pubmed Central, the full-text archive operated by the NIH’s National Library of Medicine, may not be an accurate source of publications associated with a grant. Biomedical journals routinely ensure that their publications, and their associated grant funding information, are included in Pubmed Central, but is unclear if social and behavioral science journals, especially ones without a health-relevant scope, do so routinely. Since NIH relies on Pubmed Central for evaluating the publishing productivity of the grants it funds, it is incumbent on social and behavioral science journals and the authors of articles published in these journals to ensure that the publication record is included in Pubmed Central and that the grants that supported the work described in the publication are appropriately associated with the grants. 

There are many potential reasons for the findings from this study regarding publication lag times, and we plan on reanalyzing the publications from these grants in a couple of years to see if with more time, more research from these grants was published. But the take-home message of this study is while there are many factors that contribute to longer times to publication and failure to publish in a few situations, it is the responsibility and obligation of NIH-funded investigators to the research field, to the taxpayers who support their research, and to their study participants to publish or make public in some form the results of their research in a timely manner. 

This post was originally titled, “Slower Publication Rates from Social and Behavioral Researchers Compared to the Biomedical Researchers Supported by NIH.”