Altmetrics Aren’t Always So ‘Alt’ – Ask the Developing World

A significant group of scholars from around the world love to hate the Journal Impact Factor (JIF). An incredible amount of ink has been spilled on describing its methodological limitations, its abuse and misuse, and its pervasive effects on “science.” But, while the loathing of the JIF (I hazard to guess) is distributed fairly equally around the world, the scholars who are affected by its use are not. It is scholars from developing regions who suffer the most egregious consequences.

This piece by Juan Pablo Alperin originally appeared on the LSE Impact of Social Sciences blog as “Altmetrics could enable scholarship from developing countries to receive due recognition,” and is reposted under the Creative Commons license (CC BY 3.0).

The problems for developing regions stem from the under-representation of developing world research in Thomson Reuters’ Web of Science (WoS), from which the JIF is calculated. In a seminal piece from 15 years ago, Cetto & Alonso-Gamboa (1998) laid out the disheartening situation of Latin American journals in international information systems such as the WoS. As can be seen from the figure below, which shows the relative number of works authored by scholars from around the world in WoS, this situation has not significantly changed over time.

The shortage of research from developing regions is not for a lack of research. In Latin America, to draw on the region I am most familiar with, in 2012, only 4 percent of Latin American peer reviewed journals were included in WoS (242 out of over 5,000) (see them in the Latindex Catalog). To give another example, two initiatives, SciELO and RedALyC, working with only a subset of these 5000+ journals, have indexed over half a million articles in regional journals, primarily from Latin American authors. Thomson Reuters recently announced a partnership with SciELO, whereby journals in SciELO will be indexed and appear in the Web of Knowledge. SciELO also calculates an Impact Factor based on its collection of over 1,100 journals. However, even with SciELO, only a fraction of Latin America’s research can receive an Impact Factor. There is an abundance of locally produced and published research in developing regions, just not in WoS.

See here, for an interactive map going back to 1990.

The argument for this bias has always been that research from the developing world does not form part of “mainstream” or “international” science. Although I take exception to this argument, regardless of its rationale, the end result is the same: the WoS is an inadequate dataset to understand the impact of scholarly communications from developing regions, or to otherwise study it. To serve scholars from developing regions, it is imperative to find an alternative.

The proposal

The scholarly community is abuzz with altmetrics and the related (but different) term Article Level Metrics. Ian Mulvany, head of technology for eLife, drew a nice Venn diagram depicting the distinction between the two. These metrics, derived primarily from “the social Web,” have been purposely constructed to be alternatives to the JIF. Since the drafting of the altmetrics manifesto, there has been a special issue, a PLoS collection, a Mendeley group, several annual workshops, an increasing number of research papers, and several altmetric start-ups. During a few months in the last 12, the term altmetrics has even been more popular on Google than bibliometrics and bibliometric (although the term Journal Impact Factor still dwarfs both).

All of these signs indicate that altmetrics may not remain alternative for long. Whether they supplant or complement the JIF, they bring with them a promise, but no guarantees, for developing regions.

Or see what this looks like today

The promise

Altmetrics are captured from the Web (i.e., social media, blogs, Wikipedia), and thus are (somewhat) more democratic – one reader, one vote. More precisely: one reader, several potential votes. Unlike citations, which can only be counted if the citing document is in a select group of journals, altmetrics are counted regardless of where in the world they are originated, with one important consequence: they open the possibility of tracking impact in new segments, both within and beyond the academy.

The corollary to this consequence is that altmetrics enable scholars to be incentivized and rewarded for impact in these new audiences. The JIF only captured impact from one audience, those publishing in WoS journals; altmetrics, on the other hand, can capture “diverse audiences including scholars but also practitioners, clinicians, educators and the general public” (Piwowar, 2013) And, given that developing world scholars were systematically under-represented in the WoS, new audiences could mean into developing world audiences, interested in local and regional academic or public interests. Of all the potential benefits of altmetrics, this is the true promise for the developing world: an opportunity to redirect incentive structures towards problems that contribute to development, or at least to local priorities, be it through academic, policy, personal, or professional-practice impact.

The problem

This promise, however, is not by any means guaranteed. To realize the promise, the altmetrics community needs to actively engage with scholars from developing regions. As the field advances, it will become essential to further understand the ways in which altmetrics are different from citations, and how these new metrics might shape research agendas. So far, much of the research has found that these new metrics capture a different dimension, flavor or type of impact than citations (Torres-Salinas et al., 2013; Costas et al., 2014; Haustein & Peters, 2013; Eysenbach, 2011), but it has not theorized or explored what else they might mean. There is reason to be optimistic – most altmetric research ends with a call for further study of the reliability, validity, and context of the available metrics, but there is also a risk that without this understanding, altmetrics will only be used as a proxy for traditional citation impact.

There is also a risk of altmetrics becoming yet another method for ranking scholars. If this happens, then it will once again turn attention of developing world scholars to the audiences in the United States and Europe for this is where social media has most deeply penetrated. At the ALM workshop in October 2013, I presented a series of maps showing the varying levels of penetration of Internet, Twitter, Facebook, and Mendeley (all common altmetric sources). A new map from the Oxford Internet Institute now shows us the uneven geography of Wikipedia. If the name of the game becomes increasing altmetric scores, it will still be a better strategy to cater research to places where the sources of altmetrics are more heavily used (read: not in the developing world).

Read more here

The prognosis

The current focus on assessment (and to a lesser degree filtering and discovery) will give authors a new tool for demonstrating impact beyond citations, and it may help connect researchers with research. It is an important and necessary first step for the reasons I note above. If the first years of altmetrics/ALMs saw the shift from “WHAT (are ALMs and should we care) to HOW”, then beneficial directions for developing regions would be to ask WHO (is behind the metrics) or to again ask WHAT, but this time, WHAT (do we need metrics of). The WHAT should be to expand the offerings of altmetrics to the non-traditional research products of scholars from the developing world, like those proposed by Piwowar (2013), but also arbitrary URLs of reports, policy briefs, program evaluations, and to the many journal articles published without DOIs. The WHO would necessitate building tools that help scholars identify and connect with their audiences, which should create a feedback loop that encourages authors to direct their research to areas of relevance to their primary audience, be it academic or public.

It is early days and these are only early ideas. Further research on altmetrics, especially in contexts beyond well-known and well-established journals (i.e., Science and Nature), and outside of the in so-called global North (i.e., the United States or Western Europe) is desperately needed. However, this alone is not enough. The field and the tools of altmetrics must be crafted with the participation of a diverse group scholars, so that their development can be inclusive of multiple perspectives and needs.

In this direction, the Public Knowledge Project (where I am a researcher) is now working with PLoS’ ALM application to provide a free altmetrics service to journals using the OJS platform (primarily in developing regions, see our OJS Map), and working to explore available indicators of Open Access in developing regions (project report forthcoming). The idea is to bring altmetrics to developing regions now, not after scholars elsewhere have established the methods and norms of how altmetrics will be used.

As the field begins starting to consolidate, I remain optimistically pro-altmetrics for developing regions, and I have faith in the altmetrics community to serve all scholars. Which directions altmetrics should go, how they should be used, or how the tools should be implemented is not for me to prescribe, but if we exclude (or do not seek to include) scholars from developing regions, altmetrics will become another measure from the North, for the North. And we already know that story.