Prone to Pressure: How Stakeholders Endanger the Independence of Evaluations and What We Can Do Against it
Pleger, L., Sager, F., Morris, M., Meyer, W., & Stockmann, R. (2016). “Are Some Countries More Prone to Pressure Evaluators Than Others? Comparing Findings From the United States, United Kingdom, Germany, and Switzerland.” American Journal of Evaluation, Published online before print, August 22, 2016, doi: 1098214016662907.
How important is independence of research in terms of a focus on facts only and an absence of any misrepresentation of results? It might be rather indisputable to argue that the answer to this question should be “very important.”
All the more worrying therefore are results from several studies from different countries during the last years which dealt with the independence of contract research and revealed that this requirement is often not met in practice. To be more precise, the studies focused on ethical challenges of evaluators during their evaluation work. A recent article compared the findings of the different surveys. Our article in the American Journal of Evaluation, “Are Some Countries More Prone to Pressure Evaluators than Others? Comparing Findings from the USA, UK, Germany and Switzerland,” addresses the independence of evaluations by focusing on pressure on evaluators by different stakeholders.
The article compares surveys from four different countries that analyze independence of evaluations. Studies from the UK, US, Germany and Switzerland find evidence that evaluators are often confronted with influence by different stakeholders on their evaluation work. All studies identify the person who hires the evaluator as the stakeholder who most often tried to exert influence.
Source: Pleger, et al. (2016)
While the scope of pressure covers a broad spectrum and occurs in in different manners, it nonetheless appears to be very similar in all countries. Pressure on evaluators varies from requests by the commissioner to change the wording (often with the intent to formulate something more positive) to requests to misinterpret results. In addition, a comparison of the frequency indicates that many evaluators are regularly confronted with influence attempts:
“Regarding the frequency of experienced pressure to misrepresent, 70% of the pressured subgroup of [American Evaluation Association] members and 49% of DeGEval [Gesellschaft für Evaluation e.V., Germany’s Evaluation Society] members have experienced influence in more than one evaluation. Moreover, pressure has not been an isolated incident for 90% of the [Swiss Evaluation Society] members” (Pleger et al. 2016).
The paper highlights the similarities and differences of findings for the four countries and then turns to the question of how evaluators can deal with attempts to influence and discusses potential preventive strategies. Within the American and Swiss studies, the question for preventive action is explicitly addressed and leads to similar results: The most common response given by Swiss and American evaluators for potential preventive action was to “[d]evelop [a] fuller understanding of goals/purposes/roles.”
How can a mutual understanding between evaluators and the commissioner be achieved? This question is especially relevant because such strategies are not limited for the evaluation area only but could be applied to other research areas as well. A mutual understanding of goals, purposes and roles assumes a definition and understanding of them in a first step. Thus a transfer of the one party’s goals, purposes and roles then can occur in a second step. From the evaluators’ perspective, the goals, purposes and roles are defined and specified by the evaluation standards or guiding principles by the evaluation societies and which are “intended to serve as a support for professional conduct of evaluations.” The comparative article investigates the evaluators’ familiarity with these evaluation standards and reveals that most evaluators are not only conversant with the guiding principles but also try to follow them.
In contrast, we lack research addressing the goals, purposes and roles of the other party — the person who hires the evaluator. One conclusion of the article consists in the appeal that “[f]uture research should look into the drivers of such a mutual understanding” which implies that an understanding of both perspectives should exist. Without understanding the commissioners’ perspective, no strategies can be developed which foster such a mutual understanding. For the sensitive area of contract research, an initial inventory of the commissioners’ perspective is therefore required. That would then allow a comparison of the evaluators’ with the commissioners’ perspectives which, in turn, might shed light on potential causes for pressure or influence.
In this context, another aspect is relevant. The terms ‘influence’ or ‘pressure’ are characterized by a subjective assessment. It is likely that evaluators and commissioners do not share the same understanding of these terms. Therefore, future research also needs to approach these central concepts in more detail. It is important to emphasize that it is not sufficient for independent evaluations to prevent influence under any circumstances. Due to the nature of contract research, cooperation always implies a mutual exchange. The differentiation between different types of influence, between constructive cooperation and destructive misrepresentation, is crucial for approaching the evaluators’ and commissioners’ perspectives on independence of evaluations.
The key messages for future research drawn from the article’s results are twofold and can be summarized as follows: First, research on independence of evaluations has been mainly one-sided when considering influence and pressure on evaluators. The commissioners’ perspective is almost entirely missing. Only by including both perspectives, commissioner and evaluator, can causes and solutions for influence be identified and independence of evaluation guaranteed. Second, a deeper look into the central concepts of pressure and influence is required to meet the distinctive features of mutual cooperation within contract research. It is important to be able to distinguish between different types of pressure and influence. The underlying goal, however, must be to ensure that the integrity and credibility of research is not compromised by the evaluator-stakeholder cooperation.
