The Pernicious Problem of Those FIRST Steps
There is a joy in being FIRST across the line, but it’s the input of all the other runners who make it scientifically valid.
A few weeks ago, I penned an op-ed for the Boston Globe alerting colleagues and the general public about impending legislation that could have severe negative implications for those engaged in social science inquiry. Called the Frontiers in Innovation, Research, Science, and Technology Act, or FIRST Act, the new law, if enacted, would limit funding directed to American social science projects, while empowering politicians, at the expense of scientists, to define our nation’s priorities in research.
It’s probable that FIRST will come out of committee and before our Congress in the coming weeks, and that, as is often the case, a potential vote would be determined according to party lines. By and large, Republicans would support the legislation, using fiscal austerity as a catch phrase to gather more control on the kinds of science they’d like to fund (“areas that promote economic growth,” according to one prominent Republican congressman), and away from the kind of science they’d prefer not to fund (such as climate change). Most Democrats, on the other hand, would be likely to vote against the bill, supporting the idea that funding decisions should be left to those who have devoted their life’s work to research.
This piece by David T. Takeuchi originally appeared on the LSE Impact of Social Sciences blog as “The FIRST Act’s demand for relevance at the expense of replication puts the entire scientific enterprise at risk ” and is reposted under the Creative Commons license (CC BY 3.0).
Currently, the U.S. Congress earmarks about $7.2 billion to the National Science Foundation, or NSF, an organization that funds nearly a quarter of all “federally supported basic research conducted by America’s colleges and universities.” Within the NSF, review committees made up of accomplished researchers in appropriate fields evaluate the merits of proposed studies. FIRST, however, allows politicians more say as to what they’d like to see funded. The bill’s authors have already said that the social and behavioral sciences are not among those preferred fields of research, and therefore, monies directed to these areas would be slashed (early estimates were by as much as 40 percent).
One tracker of congressional bills suggests that FIRST has only a 29 percent chance of being enacted. So, why be concerned about this bill? Even if FIRST were to fail, politicians will almost certainly continue to pursue legislation giving them more say in choosing the types of science that’s funded by federal sources. And there’s more going on here — at risk is the very fabric of scientific enterprise. Once we’re forced to cater our research to the whims and fancies of politicians, we also become more likely to forget to maintain our integrity. When this happens, science suffers, and on a disconcertingly broad level. Certainly, the reach of politics into science goes far beyond U.S. borders.
More and more in the current academic landscape, we feel the pressure to make our research more relevant to the general public. While this can be a good thing, there can be a rush to report findings and we forget that initial outcomes aren’t evidence, and, particularly where people are affected, that interventions should only be introduced following substantive, verifiable inquiry.
I’m reminded of an important article that ran in the Economist last fall about the “culture of novel findings” which exposed the fact that “science’s self-correcting mechanism,” namely replication, is no longer in “working order.”
“Replication is hard and thankless,” wrote the magazine. “Journals, thirsty for novelty, show little interest in it; though minimum-threshold journals could change this, they have yet to do so in a big way. Most academic researchers would rather spend time on work that is more likely to enhance their careers.”
Even if FIRST were to fail, politicians will almost certainly continue to pursue legislation giving them more say in choosing the types of science that’s funded by federal sources
This is, of course, all too true. Researchers, particularly those on the tenure track, are evaluated based on their published research, the reputation of the journals that accept their manuscripts, and how often their peers cite their work. There’s nothing wrong with that, but we shouldn’t have to sacrifice our scientific integrity to get ahead. Replication of novel findings matters deeply — without it, budding science can’t effect real change. But popular (i.e. political) oversight of research (such as is being proposed through FIRST) would only further ensure that the kind of scientific scrutiny that leads to real progress becomes less and less valued.
Take, for example, all of the excitement surrounding Google Flu Trends. Back in 2008, the Internet search giant successfully predicted flu outbreak prevalence based on search results on its site. The American Centers for Disease Control (CDC) hailed the technology and heaps of positive press followed suit. But since then, Google Flu Trends has failed miserably; in 2013 Nature found it was predicting more than 50 percent as many physician visits than were reported by the CDC.
What happened? In short: Google’s algorithm incorporating data from the Web and social media needed some serious tweaking before it should have been accepted as a viable means for tracking the flu. But, as is increasingly the case in science, the cart came before the proverbial horse, and the technology was hailed as a next, great innovation. For all the data our tech giants can aggregate, we’re reminded that crowdsourcing should not be confused with scientific inquiry.
The mistakes made with Google Flu Trends should serve as a cautionary tale, and above all, as a reminder to all of us in the field of research that replication matters. One set of findings does not make a trend. Politicians build careers upon “what have you done for me latelys,” and their stances are often manipulated by the tenor of a specific moment in history. Science, on the other hand, doesn’t work this way. We have a process, and if this process is threatened, we risk losing more than just our jobs. Without establishing foundational science, effective interventions to solving problems won’t follow. This will not only affect science, and the careers of scientists, but it could cost lives.
