Why Social Science? Because It Makes an Outsized Impact on Policy
We know that social and behavioral science (SBS) has a hold on the conversation when institutional and government decision-makers parley over what goes into ‘policy.’ But oftentimes the SBS researchers whose own work goes into policy are unaware that they’re making an impact in the real world and are stymied from amplifying their findings or using them to advance their careers or fields.
With this in mind, Sage partnered with Overton to create Sage Policy Profiles, a free-to-use tool that enables researchers to discover the real-world impact – drawing from a pool of more than 10 million policy documents – of their work on policy, visualize, export, and share what they find. [Sage is the parent of Social Science Space, while access to Sage Policy Profiles is provided via Social Science Space.]
In light of this launch, I sat down with Euan Adie, founder of Altmetric and Overton and currently Overton’s managing director, to learn more about the outsized impact that SBS makes on policy and his work creating tools to connect the scholarly and policy worlds.
What inspired you to found Overton? What was your ultimate goal?
I was always really interested in what goes into policy: How many decisions are backed by evidence? Who is influencing how that research is found and interpreted? We might imagine government experts, think tanks and corporate lobbyists are all interacting somehow – but how do you get involved as an academic? So one goal behind Overton is to make it a bit more transparent.
If you ask – as a researcher – how you can best engage with policymakers, you’ll get a very unsatisfactory answer about having to be very patient, but also ready to suddenly drop everything and put in lots of effort for potentially little return. All of which is correct but another goal at Overton is to make this process a little less painful for both sides.
I love that it’s an area full of tricky challenges. COVID showed most people that it was useful to have relevant experts contributing to big policy decisions, and laws like the Evidence Act in the US show that lawmakers want that, too. But how do you translate the nuanced answers we’re trained to give as academics into something more pragmatic that can actually be used by a policymaker? That’s an exciting question.
Do you see disciplinary differences between how research impacts policy? What does that look like?
The biggest difference that jumps out at you is that the natural and physical sciences are rarely cited in policy while the social sciences are everywhere.
Can you share some data behind this?
Yes. To get some broad-brush numbers for this we took a random sample of ~ 1 million journal article records published between 2011 and 2020 from the OpenAlex database and then joined them with data from Overton to see which were cited by policy and how often. Fifty-eight percent of the records in our set had scholarly citations and 4 percent have policy citations.
Those policy citations weren’t evenly distributed across subjects. We used OpenAlex’s “concepts” taxonomy to group articles into different subject areas.
In all of the areas when you compare mean scholarly citations to mean policy citations the scholarly citations mean is higher, which makes sense: articles get cited by other scholars far more than they do in policy.
But the ratio between the two changes from one subject to another. Of the areas we looked at, Chemistry, Materials Science, and Physics publications have the highest average number of scholarly citations (relative to other subjects) but the lowest average number of policy citations.
In Business & Economics things are flipped: the subject has few scholarly citations (generally, and relative to other subjects) while receiving many more policy citations. The same holds true for Political Science, Psychology & Sociology.
We found something similar when looking at the Scopus journal categories of all articles cited in policy as part of a broader analysis that was published in QSS.
Why do you think this is?
There may simply be more policy about education, social care, prison welfare or macroeconomics than there is about organic chemistry or quantum physics.
Anecdotally for lots of different reasons some policy areas are better at finding, interpreting, using and citing evidence than others – public health or food safety, for example, often have associated technical agencies and policymakers who are themselves experts in the field.
Other areas are broader and you might have policymakers who are generalists trying to collect information as best they can on a tight schedule.
And that said…. there are biases and caveats in the data too. We’re only looking at papers explicitly cited in policy documents, so have to acknowledge that the numbers will be biased towards subject areas where it’s made more clear where the evidence is coming from, where there’s more transparency about what evidence was used in each document.
According to your data, what policy sources across the globe most use SBS research? What about the most US-based sources?
Intergovernmental organizations (IGOs) are big users of evidence, and very important as knowledge brokers sitting in-between the academic and policy worlds. The OECD, World Bank, UNESCO and WHO are all big users of social sciences research.
Federal government aside, two of the biggest users in the US are actually think tanks: RAND and Brookings. Social sciences research also gets picked up as evidence by the Congressional Research Service or used by House and Senate committees.
Can you break down your data by topic area? What topics are most cited according to your data?
We can! In terms of which topics are most cited by policy documents, when we look at Scopus journal subjects, the top ones are:
- Medicine — Public Health, Environmental and Occupational Health
- Economics — Economics and Econometrics
- Life Sciences — Ecology, Evolution, Behavior and Systematics
- Social Sciences — Sociology and Political Science
- Medicine — Psychiatry and Mental Health
To some degree that fits with the topics of the policy documents themselves. We can get a feel for that by looking at what proportion of policy documents fit into the different UN Sustainable Development Goals (SDGs) with the caveat that they are development-focused. The three most common SDGs are:
- Good Health and Well-being
- Decent Work and Economics Growth
- Quality Education
How would you say Sage Policy Profiles differs from other products you have put out?
We spent much more time on this app thinking about the user experience for individual researchers, partly after seeing how much Sage cares about the same thing, which I think has really paid off.
Being focused on one person’s data at a time was also new and opened up some opportunities that we couldn’t do otherwise, such as letting the user curate their citations and mentions.
Now looking beyond these tools, what do you think is needed to strengthen the ties between researchers, academic institutions, and policymakers?
There’s really interesting work happening on the demand side of evidence-based policy that I think has the potential to do a lot of good.
Simple sounding things like making sure academics and funders know what topics their government is actually looking for help on: Learning Agendas in the US and the Areas of Research Interest in the UK are two good examples of this, and all the work Transforming Evidence has done on the latter is a highlight.
Policymakers and academics work on different timescales and have different incentives, and that’s something else that I think needs addressing. Arguably the current system we have for assessing researchers is pretty broken, but we don’t need to overhaul the whole thing to make meaningful improvements to it for people who want their work to be useful: there are incremental improvements we might make.
