Proof Over Promise: Finding An Impact Factor for Your Career
If only there were some cheap, easy-to-find and easy-to-use database out there …
In this blog we explain the added value of a career-impact focus and envision some practical implications for business schools. We will show that an impact driven approach using Google Scholar creates a more democratic and inclusive assessment.
Proof (impact) and Promise (journals)
On average, publications in high-impact journals by definition get cited more frequently than publications in low-impact journals as the journal impact factor is based on average citations. This still is the fundamental logic behind the creation and maintenance of journals list by business schools and associations such as the Association of Business Schools. We call this principle “promise”, i.e. publishing in a high-impact journal carries the implicit promise that the article will also be highly cited. However, not all individual papers published in these high-impact journals will fulfill this promise. In our paper “Proof over promise: Towards a more inclusive ranking of Dutch academics in Economics & Business,” we therefore set out to create a ranking based on “proof,” i.e. rather than looking at the promised number of citations implied by the journal impact factor or article influence score, we look at actual citations to an author’s work.
This article by Anne-Wil Harzing and Wilfred Mijnhardt originally appeared on the LSE Impact of Social Sciences blog as “Proof over promise: Moving citation metric systems beyond journal impact towards a career impact approach” and is reposted under the Creative Commons license (CC BY 3.0).
Introducing two new impact metrics with individualized career focus
In our paper we use two relatively new metrics, the citations per author per year (CAY) metric and the individual annual h-index (hIa, see Harzing, Alakangas & Adams, 2014 for details). These two metrics are the individualized and annualized equivalents of respectively total citations and the traditional h-index, two important bibliometric measures. We correct our metrics for the number of co-authors as the number of co-authors positively influences both the number of publications an academic is able to publish and the number of citations. A correction for the number of years an academic has been active is important as citations continue to increase over an academic career.
hIa career impact: field independent focus on contribution
In this blog, we focus on the hIa index, which is calculated by dividing the individual h-index (an h-index corrected for the number of co-authors) by the number of years an academic has been active, i.e. the number of years that have lapsed since their first publication. The metric thus represents the average number of single-author-equivalent “impactful” articles that an academic has published per year and hence permits an intuitive interpretation. Based on an empirical example of 146 academics in five major disciplines at different career stages, Harzing, Alakangas & Adams (2014) showed that the hIa index attenuates h-index differences that are purely attributable to (disciplinary) co-authorship practices and career lengths. As such, this metric provides a more reliable comparison between academics in different disciplines and at different career stages than the h-index.
Further details
Harzing, A.W.; Mijnhardt, W. Proof over promise: Towards a more inclusive ranking of Dutch academics in Economics & Business, in press for Scientometrics, DOI: 10.1007/s11192-014-1370-z, prepublication version available at http://www.harzing.com/papers.htm#top40
A second aspect of our focus on proof over promise is the use of Google Scholar rather than ISI as a data source. Although ISI listing is seen by many to imply a quality stamp, in our view it should not matter where research is published. If a particular research output is highly cited, it clearly influences the field and that should be more important than the journal in which it is published. As Google Scholar on its own is not very suitable for bibliometric analyses, Publish or Perish (Harzing 2007) was used to collect citation data from Google Scholar. There are now more than 500 published articles referring to the Publish or Perish program. This indicates that—in spite of its limitations—Google Scholar is perceived to be a useful source of bibliometric data.
Resulting ranking for economics & business in the Netherlands
The resulting hIa-based ranking of academics in economics & business in the Netherlands (see Harzing & Mijnhardt, in press for details) is substantially different from the original ranking based on publications and is more inclusive in terms of disciplines, age and affiliation.
- Whereas in the original publication-based Economics top-40 more than two thirds of the listed academics works in Economics or Management Science, this proportion is reduced to 43 percent in the hIa ranking. The change in disciplinary composition is particularly striking in the top 20. In the publication-based Economics Top 40, three quarters of the academics in the top 20 were economists, in the PoP hIa top 40 this is reduced to just over a third.
- At 47, the average age in the hIa top 40 is lower than in the original ranking (50), but most noticeable is the addition an additional five academics under 40 and another five aged between 40 and 45. In fact, all but three of the newly listed academics in this top 40 are aged 45 or under.
- In the original top 40 academics affiliated with Erasmus and Tilburg made up just over or just under half of the list, in the hIa top 40, this is reduced to a third. Apart from the University of Utrecht, every university is represented in the hIa top 40. Maastricht, Eindhoven and the UvA (University of Amsterdam) in particular do much better in citation-based rankings than in the original publication-based top 40.
We argue that our “proof over promise” approach is more “democratic”/inclusive than the original Economics Top 40. First, by expanding the type of research outputs considered beyond the narrow scope of publications in ISI-listed journals, we remove the disciplinary bias against management, marketing and accounting & finance, disciplines in which a smaller proportion of high-quality journals is ISI-listed than in economics and management science (Harzing & van der Wal, 2009).
Second, citation-based performance metrics can be argued to be more democratic as their “verdict” is based on the reception of the paper by the academic community as a whole, whereas acceptance in a high-impact journal is dependent on only a handful of gatekeepers (the editor and reviewers).
Third, our ranking was conducted with a free software program (Publish or Perish) and a publicly available database (Google Scholar). Hence, any reader can easily replicate the ranking without the need for subscription-based databases. This also means that any academic can look up their own citation record and easily find out where they score in the current ranking
Fourth, citation-based rankings and in particular the hIa ranking are likely to provide more dynamic rankings in terms of changes over the years. Younger academics can more easily enter into the hIa ranking if they perform well relative to their career length in terms of single-author equivalent impactful papers.
As for other research metrics, the CAY or hIa-index should never be used as the sole criterion to evaluate academics. Another crucial question that should always be asked is: “Has the scholar asked an important question and investigated it in such a way that it has the potential to advance societal understanding and well-being?” (see e.g. Adler and Harzing, 2009). However, we argue that the hIa-index provides an important additional perspective over and above a ranking based purely on publications in high impact journals alone.
Implications for business schools
Using metrics such as the CAY and hIa, business schools can move away from the journal-based logic and start thinking from an impact -driven logic. This could have important consequences for incentive policies, mainly for senior faculty. We advice deans of business schools to start assessing their senior faculty from an impact perspective, instead of the narrow journal-based assessment. This approach is not only more inclusive, it is also more democratic and gives more opportunity for diversity to flourish, especially in business schools with a diverse social sciences background.
