A Cautionary Tale in the Quest for Novelty
There is growing concern that funding agencies which support scientific research are increasingly risk-averse and that their competitive selection procedures encourage relatively safe and exploitative projects at the expense of novel projects exploring untested approaches. At the same time, funding agencies increasingly rely on bibliometric indicators to aid in funding decisions.
Research underpinning scientific breakthroughs often is driven by taking a novel approach, which has a higher probability of major impact but also faces a higher risk of failure. It may also take longer for novel research to make a mark, either because of resistance from incumbent scientific paradigms or because it takes longer to incorporate novel contributions into follow-on research. The “high risk/high gain” nature of novel research makes it particularly appropriate for public support. If novel research, however, has an impact profile distinct from non-novel research, the increasing use of bibliometric indicators for evaluation may bias funding decisions against novelty.
This article by Jian Wang, Reinhilde Veugelers and Paula Stephan originally appeared on the LSE Impact of Social Sciences blog as “Bias against novelty in science: A cautionary tale for users of bibliometric indicators” and is reposted under the Creative Commons license (CC BY 3.0).
In Wang, Veugelers, and Stephan (2016), we examine the complex relationship between novelty and impact. To do this, we develop a measure of novelty for individual scientific publications , using the life-time citation trajectories of 1,056,936 research articles published in 2001 across all disciplines in the Web of Science (WoS).
To assess the novelty of a research article, we examine the extent to which it makes novel combinations of prior knowledge components. Specifically, we examine whether an article makes new combinations of referenced journals that have never appeared in prior literature, taking into account the knowledge distance between the newly-combined journals based on their co-cited journal profiles (i.e., “common friends”). Technically, we measure the novelty of a paper as the number of new referenced journal pairs weighted by the cosine similarity between the newly-paired journals.
(A) Estimated dispersion of citations (14-year). (B) Estimated probability of being among the top 1 percent cited articles in the same WoS subject category and publication year, based on 14-year citations. (C) Estimated probability of being cited by big hits. (D) Estimated probability of being among the top 1 percent cited articles in the same WoS subject category and publication year, based on 14-year home and foreign field citations separately. (E) Estimated probability of big hits, using 14 consecutive time windows to dynamically identify big hits. As an example, big hits in year 3 are identified as the top 1 percent highly cited papers based on their cumulative citations in a 3-year time window, i.e., from 2001 to 2003. (F) Estimated Journal Impact Factor. All regression models control for field (i.e., WoS subject category) fixed effects, the number of references and authors, and whether internationally coauthored, (C) additionally controls for the number of citations. All estimated values are for an average paper (i.e., in the biggest WoS subject category, not internationally coauthored, and with all other covariates at their means) in different novel classes. The vertical bars represent the 95 percent confidence interval. Data consist of 1,056,936 WoS articles published in 2001 and are sourced from Thomson Reuters Web of Science Core Collection.
Applying this newly-minted indicator of novelty, we find that novel research is relatively rare: Only 11 percent of all papers make new combinations of referenced journals, and the degree of novelty is also highly skewed. To work with this skewedness, we classify papers into three categories: (i) non-novel, if a paper has no new journal combinations, (ii) moderately novel, if a paper makes new combinations but has a novelty score lower than the top 1 percent of its field, and (iii) highly novel, if a paper has a novelty score among the top 1 percent of its field.
In view of the risky nature of novel research, we expect novel papers to have a higher variance in their citation impact. We estimate the the mean and dispersion of received citations in a 14-year time window for novel papers compared to non-novel ones, controlling for other potentially confounding factors, such as the scientific field of the articles. We find that the dispersion of the citation distribution is 15 percent higher for highly novel papers than non-novel papers, confirming their higher risk profile.
We also find that novel research papers are more likely to become “big hits,” i.e., being top 1 percent highly cited among the articles in the same field and publication year: The odds of being a big hit are 40 percent higher for highly novel papers than comparable non-novel papers. Novel research is not only more likely to become a big hit itself but also more likely to stimulate follow-on research which generates major impact: The odds of being cited by other big hits are 26 percent higher for highly novel papers than for comparable non-novel papers receiving the same number of citations.
Read the Paper
This post is based on Wang J, Veugelers R, Stephan P (2016) Bias against novelty in science: A cautionary tale for users of bibliometric indicators. CEPR Discussion Paper No. DP11228; NBER Working Paper No. 22180.
Examining the disciplinary profile of impact reveals that, compared with an equally cited non-novel paper, highly novel papers are cited by more fields and have a larger ratio of citations from other-than-own fields, suggesting a greater cross-disciplinary impact of novel research. Furthermore, we find that novel papers are more likely to become highly cited only in foreign fields but not necessarily in their home field, calling to mind the passage from Luke 4:24: “Verily I say unto you, No prophet is accepted in his own country.”
We find that novel papers have a delayed recognition. Highly novel papers are less likely to be top cited in the first 3 years after publication but significantly more likely from the fourth year onward, and their advantage over non-novel papers further increases over time. Moderately novel papers suffer even more from delayed recognition: They only have a significantly higher chance of being top cited with time windows of at least nine years. The finding of delayed recognition for novel research suggests that standard bibliometric indicators which use short citation time-windows (typically two or three years) are biased against novelty.
Assessing the Impact Factor of the journal in which the paper is published, we find that of highly-novel papers to be 18 percent lower than comparable non-novel papers , suggesting that novel papers encounter obstacles in being accepted by journals holding central positions in science.
Taken together, our results suggest that standard bibliometric measures are biased against novel research. Consequently, they may fail to identify papers and individuals doing novel research. This bias against novelty matters, because novel research, as we have shown, is much more likely to become a big hit in the long run, as well as to stimulate follow-up big hits.
The bias against novelty is of particular concern given the increased reliance funding agencies place on classic bibliometric indicators in making funding and evaluation decisions. This bias against novelty applies not only to funding decisions but to science policy more generally. The prevailing use in various science decisions of standard bibliometric indicators which rely on short citation time windows and Journal Impact Factors is likely to disincentivize novel research.
In addition, the finding that novel papers are highly cited in foreign fields highlights the importance of avoiding a monodisciplinary approach in peer review (see Gabriele Bammer’s recent piece for more on this). Peer review is widely used in science decision-making and is typically organized along disciplinary lines. Studies of interdisciplinary research demonstrate that a discipline-based science evaluation system is detrimental to the advancement and societal accountability of science. Our findings suggest that peer review which is bounded by disciplinary boarders may also fail to recognize the full value of novel research which typically lies outside its own field.
If we want to encourage risk taking, we need to be aware of the inherent bias in bibliometric measures against novel research. More importantly, we advocate, when bibliometric indicators are used as part of the evaluation process, a wider portfolio of indicators and of time windows beyond the standard window of two or three years.
