How Three False Starts Stifle Open Social Science
Open social science is new, and like any beginner is still finding its way. However, to a large extent we are still operating in the shadow of open science (OS) in the Science, technology, engineering, mathematics, and medicine, or STEMM, disciplines. Nearly a decade ago an influential Royal Society report argued:
‘Open science is often effective in stimulating scientific discovery, [and] it may also help to deter, detect and stamp out bad science. Openness facilitates a systemic integrity that is conducive to early identification of error, malpractice and fraud, and therefore deters them. But this kind of transparency only works when openness meets standards of intelligibility and assessability – where there is intelligent openness’.
More recently, the Turing Way project defined open science far more broadly as a range of measures encouraging reproducibility, replication, robustness, and the generalisability of research. Alongside CIVICA researchers we have put forward an agenda for progressing open social science in line with these ambitions. Yet for open social science to take root it must develop an ‘intelligent’ concept of openness, one that is adapted to the wide range of concerns that our discipline group addresses, and is appropriate for the sharply varying conditions in which social research must be carried out.
This task has been made more difficult by a number of premature and partial efforts to ‘graft’ an ‘open science’ concept from STEMM disciplines onto the social sciences. Three false starts have already been made and have created misconceptions about open social science. Below, I want to show how each of the strategies may actually work to obstruct the wider development of open social science.
Bricolage – Reading across directly from STEMM
This approach sees open social science as just about picking up (not quite at random) the best-known or most discussed individual components of open science in STEMM disciplines – focusing on specific things like open access publishing, the FAIR principles for data management, replication studies, or the pre-registration of hypotheses. Important though they are, adoption of these open components doesn’t reveal any deep commitment to openness or innovative ways of working.
This is highlighted by Christensen et al’s fascinating 2020 research on American social science authors ‘open’ practices (see figure). They found that publishing at least something open access (once) is now near universal, and that depositing data at least once is also common in experimental fields, less so in quantitative work. Differences in posting the research instruments from studies online are sharper and pre-registering hypotheses has yet to take off much in non-experimental work. The results also unequivocally show substantial differences in their use of ‘open’ between authors in experimental fields (who came out top), those using non-experimental quantitative work (who were in the middle), and theoretical or qualitative fields (whose use of ‘open’ was lagging).
How Christensen et al. chart the development of three open science practices amongst U.S. social scientists over time
The questionnaire only asked if the academics have done something ‘open’ at least once, so it may considerably overstate the extent at which open practices are being used at any one time. However, Christensen et al also asked all the authors (taken together) to give their perceptions of how much progress ‘open’ was making in their field, versus how much they had used open working themselves (their Figure 6). Respondents generally estimated open science progress in their subject field more pessimistically than changes in their own practices. Yet, the survey highlights how this can lead to isolated sites of open practice, especially when open methods are not situated within an overall approach to open science.
Of course, it is useful for social scientists to consider adopting particular practices that STEMM scientists have found useful and developed concrete protocols for, and in some fields (like cross-national questionnaire research) this approach is already almost inevitable. However, it may also be wise to remain somewhat skeptical about how far ‘open’ behaviors pledged by authors are actually carried through. Some recent studies in medicine and other subjects show that where data for an article is supposedly available ‘on demand,’ the researchers involved overwhelmingly do not respond to requests for access.
‘Open’ as bureaucracy and neo-liberal surveillance
A second, more organized view misrepresents the cultural shift to ‘open’ as being primarily about more external surveillance of academia. ‘Open’ ways of working can easily come across as just another set of burdensome and extraneous bureaucratic hoops through which academics must jump. Especially when allied with the bricolage approach above, it is very easy for a ‘research briefing’ by well-intentioned university support staff to end up as just a lengthy Powerpoint presentation detailing the varying ‘open’ requirements of funding bodies, or extensive data management principles. This burden-boosting account is particularly off-putting in the social sciences, because only 9 percent of UK social scientists are full-time researchers with no teaching commitments, and thus able to specialize in particular aspects of grant seeking and meeting open science requirements. (By contrast, in the STEMM sciences 35 percent of staff are research-only). Busy researchers-plus-teachers have a lot on their plate, so more time-consuming bureaucracy is doubly unwelcome.
In addition, many university staff dislike recent trends toward greater use of metrics in research assessment and are understandably critical of further (allegedly ‘neo-liberal’) surveillance. Even amongst STEMM sciences some authors have extended the critique of metricization and ‘surveillance’ to apply to open science. For example, Philip Mirowsksi argues: “The [open access] agenda is effectively to re-engineer science along the lines of platform capitalism, under the misleading banner of opening up science to the masses.” While much of this commentary is wildly overstated, and ignores both the many autonomous sources of the impact agenda within academia and the research benefits of systematic search, this reaction is still widespread and deeply felt in parts of the social sciences.
These problems are made worse by justifications of open social science (OSS) that dwell on counteracting problematic practices like p-hacking (massaging model regression analyses over particular ‘statistical significance’ levels) to push work into high-status journals, selective publication of positive results and non-publication of negative results, and ‘discoveries’ by one researcher that cannot be replicated. These are live problems, and many components of open social science may help to mitigate them, but this is not what OSS is centrally about. Social scientists cannot assume that they are studying an invariant entity (‘nature’), with unchanging law-like (if very complex) mechanisms at work. Instead, they must deal with a constantly changing and reflexive human and societal capability to absorb knowledge and do things differently as a result. Open social science is a cultural change that aims to do this more successfully and reliably than before, a goal that could not be more integral to the whole purpose and rationale of good research from its earliest beginnings through to its publication and discussion by a professional or wider audience.
Reawakening methodological sectarianism
Many social science disciplines have recently had (or still have) acute ‘methodology wars’ between exponents of formal theories and highly developed quantitative and mathematized research approaches on the one hand, and on the other hand social scientists using more qualitative methods. Furthermore, in all the social sciences a great deal of scholarship involves updating ‘first draft of history’ accounts of recent societal and economic changes, using more discursive and descriptive approaches.
These conflicts have declined from their peak intensity, notably in political science where the American Political Science Review editors recently welcomed robust qualitative work, a radical change of mood from the perestroika era conflicts at American Political Science Association’s 2000 conference in San Francisco. It would be all too easy, however, for those pushing a premature STEMM-only notion of ‘open social science’ to reopen past wounds and reawaken the past alarms of qualitative scholars about being ‘squeezed out’ of their disciplines. This is particularly notable, where an ‘open’ effort is over-linked to an alleged ‘replication crises’, or becomes associated with attempts to rewrite research integrity rules in crude ways that actually are only practicable in (some kinds of) quantitative work.
For instance, the 2019 book, Transparent and Reproducible Social Science Research: How to Do Open Science, falls headlong into this trap. It claims to be “The first book to summarize and synthesize new approaches to combat false positives and non-reproducible findings in social science research, document the underlying problems in research practices, and teach a new generation of students and scholars how to overcome them.” Yet, in practice, this volume covers only disciplines and areas of work with the most developed quantitative methods, and stresses only the replication/research integrity arguments linked to quantitative methods. No effort is made to consider how open science approaches might be involved in movements toward an inclusive open social science, or how ‘open’ ways of working might be developed for other kinds of social science work. Indeed, remarkably, the book’s nearly 200 pages manage never to mention qualitative research at all. Searching the institutional website behind the book (Berkeley’s ITSS) reveals only a few scattered examples of any effort to address qualitative research as open social science.
Other US-based promoters of open science, like the Center for Open Science, also cover mainly the quantitative disciplines in social sciences (like psychology and health studies). Yet COS’s somewhat disturbing strategy for accelerating cultural change towards ‘open’ starts with ‘Make it Easy,’ goes through ‘Make it Normative (sic)’ and then ‘Make it Rewarding,’ but ends up with ‘Make it Required’. No apparent explanation is given of how that last stage would work outside of laboratory or experimental work, some kinds of randomized control trials, or purely computational research. A careful analysis of many psychology journals’ policies by Prosser et al suggests that ‘open science risks becoming a closed door’ for qualitative researchers in the discipline.
Doing open social science right
Valuable as the insights about open science in STEMM disciplines are, a broader and more inclusive approach is needed if ‘open’ is to develop fully across the social sciences – in the process hopefully reshaping not just into crossover disciplines with the humanities (like law, social and political philosophy, contemporary history and the digital humanities), but also spilling over into a wider range of humanities subjects (like older history, philosophy and literature studies). Open social science is not about science-envy, or yet more research bureaucracy, or strengthening top-down surveillance within academia, or a new episode of past destructive and misguided methodology wars. It is instead about moving all social science research forward towards being more soundly based, more coherent and generalizable, and more open for citizens to access and understand.
