Impact

Criticisms of the Citation System, and Google Scholar in Particular

March 20, 2019 4352

In the last post I summarised what Google Scholar is and its benefits, in this article I’ll explore some of the common criticisms levelled against the research citation system, and Scholar’s part in that in particular. Of course, some of these issues could be levelled at other services too.

Scholar is bigger and more inclusive than its competitors because it is less curated and more automated, but it has been found to be less accurate. It has also been blamed for not vetting sources and including predatory journals. Search results can be inconsistent and hard to replicate. Some librarians, bibliometricians and university administrators do not recommend using Scholar as the data is deemed “not valid”, and usage guides sometimes come with warnings.

It has been shown it’s possible to spoof Scholar, with fake articles of randomly-generated words accepted, and even a fake author who became the world’s 21st most-highly-cited scientist. Self-citation and routine group citation can also help game metrics, including in Scholar.

Author order is also not currently accounted for in Scholar, a growing issue with many papers now involving high numbers of collaborators. Being a first or sole author is somewhat different to being the 100th, yet Scholar currently gives all three the same weight.

James Wilsdon, of the University of Sheffield, highlighted to me that Scholar, like other citation databases, suffers from an inability to discriminate between different types of citation e.g. this article is completely wrong vs my entire argument is built on this article. Currently both would be treated the same..

Scholar is also criticized for not providing an open application programming interface (API) to other services, and for not sharing its sources, ranking approach or algorithms. The latter practice is still standard in Silicon Valley but completely counter to typical scholarly behavior. In some ways, Scholar is aligned to the Open Access movement, but not quite.

Although Google Scholar metrics can allow comparisons between authors without some biases (e.g. when hiring), it can introduce others – such as between disciplines, where different conventions can result in wildly different metric ‘scores’. You should also be very wary comparing metrics (such as an author’s h-index) between services. Google Scholar and each of its competitors have different coverage, and so give different metric “scores” for authors and journals. Those differences can be patterned in ways relevant to particular disciplines, for instance with Scholar being found to cover fewer pre-1990 publications.

It can be useful for metrics to allow non-experts to evaluate an author’s influence in an academic field, but they can do so without requiring any appreciation of the actual ideas or content of that research area. The inexorable move to quantitative proxies for measuring research success, or financial proxies for real-world impact (consultancy revenues, patents or spin-out companies etc.) can also mask the genuine, often messy, impact of an author’s research on other authors, practitioners or policymakers – but that’s for an article later in the series.

The third and final article on the topic of Google Scholar will look at how the citation system, and Scholar specifically, could be improved.


Louis Coiffait is a commentator, researcher, speaker, and adviser focused on higher education policy, with a particular interest in impact and knowledge exchange He has worked with Pearson, Taylor & Francis, SAGE Publishing, Wonkhe, think tanks, the Higher Education Academy, the National Foundation for Educational Research, the National Association of Head Teachers, the Teacher Training Agency, an MP, and a Minister. He has led projects on how publishers can support the impact agenda, the future of higher education (the Blue Skies series at Pearson), access to elite universities, careers guidance, enterprise education, and the national STEM skills pipeline (for the National Audit Office). He is also committed to volunteering, including over a decade as a school governor and chair of an eight-school federation in Hackney in East London, and recently as vice-chair of a school in Tower Hamlets. He spent three years as chair of Westminster Students’ Union. He studied at York, UCLA and Cambridge. Louis is an RSA Fellow, amateur photographer, “enthusiastic” sportsman, proud East London citizen and Yorkshireman (really).

View all posts by Louis Coiffait

Related Articles

Three Decades of Rural Health Research and a Bumper Crop of Insights from South Africa
Impact
March 27, 2024

Three Decades of Rural Health Research and a Bumper Crop of Insights from South Africa

Read Now
Using Translational Research as a Model for Long-Term Impact
Impact
March 21, 2024

Using Translational Research as a Model for Long-Term Impact

Read Now
Coping with Institutional Complexity and Voids: An Organization Design Perspective for Transnational Interorganizational Projects
Research
March 19, 2024

Coping with Institutional Complexity and Voids: An Organization Design Perspective for Transnational Interorganizational Projects

Read Now
2024 Holberg Prize Goes to Political Theorist Achille Mbembe
News
March 14, 2024

2024 Holberg Prize Goes to Political Theorist Achille Mbembe

Read Now
AAPSS Names Eight as 2024 Fellows

AAPSS Names Eight as 2024 Fellows

The American Academy of Political and Social Science today named seven scholars and one journalist as its 2024 fellows class.

Read Now
Britain’s Academy of Social Sciences Names Spring 2024 Fellows

Britain’s Academy of Social Sciences Names Spring 2024 Fellows

Forty-one leading social scientists have been named to the Spring 2024 cohort of fellows for Britain’s Academy of Social Sciences.

Read Now
Norman B. Anderson, 1955-2024: Pioneering Psychologist and First Director of OBSSR

Norman B. Anderson, 1955-2024: Pioneering Psychologist and First Director of OBSSR

Norman B. Anderson, a clinical psychologist whose work as both a researcher and an administrator saw him serve as the inaugural director of the U.S. National Institute of Health’s Office of Behavioral and Social Sciences Research and as chief executive officer of the American Psychological Association, died on March 1.

Read Now
5 1 vote
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments