Pub-Test Folly: Why We Quit Australia’s ARC College of Experts

On Christmas Eve 2021, the pub-test folly struck again. The two of us found ourselves, angry and heartsore, resigning from the Australian Research Council’s (ARC) highly respected College of Experts in protest at the minister’s rejection of grant funding recommendations.

This was not a comment on the college, a laudable body of experienced research leaders committed to supporting the best and most worthwhile research. Nor on the ARC, whose dedicated, knowledgeable staff operate on a shoestring to maximise how much of the organisation’s limited funding is spent on research.

We were prompted by the acting minister for education and youth disregarding the expertise of Australia’s best by blocking six grants they had recommended for funding. The explanation? Unsupported statements about “value for taxpayers’ money,” and “the national interest.” That is, a pub test: if the imagined average punter can’t immediately spot its value from a potted summary, then it’s not in the national interest.

You can’t pick good-value research with a pub test

Deciding what research to support is hard. As argued previously, it is difficult, maybe impossible, to predict what lines of inquiry will bear the best fruit – or even what fruit to grow. As is generally attributed to Oren Harari:

“The electric light did not come from the continuous improvement of candles.”

It is only obvious in hindsight that understanding electricity represented “value for money”. Likewise, as Ofer Gal explains, the national interest in understanding history and culture may only become visible after the fact, through the tragic consequences of ignorance.

In an ideal world, we could just do all the research. But research costs money: for equipment, lab space, consumables, travel to collaborate with experts elsewhere, and capacity, typically in the form of postdoctoral researchers. The investment repays itself many times over in future economic activity, but we must live within our means. So we must choose.

And there is much to choose from. How do we fight COVID-19? Research. How can we achieve a carbon-free future? Research. What lifestyle choices maximize health in old age? What factors led to the emergence of the modern state of China? Research, and more research.

Sometimes only experts can understand even the questions. How can we construct symmetric informationally complete positive operator valued measures in arbitrary dimensions? It sounds abstruse, but this research could enable reliable error correction in quantum computing.

How are grant applications assessed?

Of course, government should be involved in setting strategic research funding directions. It should determine funding parameters and areas of immediate priority, and clear rules, procedures and criteria. For example, the research should be:

• original – don’t re-invent the wheel
• significant – not just minor tweaks to existing understandings
• feasible – anyone can make grandiose claims, but funding requires a reasonable expectation of results
• of benefit – a positive impact on the field or society.

These criteria have been at the core of ARC funding decisions for decades.

But assessing these criteria is wickedly difficult. In particular, assessing value for money requires expertise: the expected benefit of research can be deep and very real, without being superficially visible. The ARC’s College of Experts provides, and facilitates, this expertise.

At least two college members assess each proposal, running to 50-100 pages, in detail. They read every word.

College members also select four subject experts to assess each proposal. The members then meet over multiple days to discuss the applications in detail and make funding recommendations.

By and large this arduous process, though imperfect, works. It taps both the expertise of college members – in assessing grants and in selecting detailed assessors – and of those assessors. The resulting funding recommendations represent the collective best judgment of world-leading minds and experience that Australia has proudly cultivated over generations.

Political meddling does lasting damage

The minister spurned this in favor of a pub test. It’s already been argued strongly that ministerial veto compromises academic freedom. But it also betrays ignorance of the complexity of assessing cutting-edge research and shows contempt for the expertise, time and diligent effort embodied in the college’s recommendations.

Further, it compromises our capacity to assess in future. Will international leaders in their fields continue to give their time to assess applications knowing their recommendations may later be overturned on a ministerial whim?

The damage to our international reputation is apparent. The minister’s decision has been condemned by international voices and numerous Australian bodies: the Australian Mathematical Society, members of the ARC College of Experts, Australian Laureate Fellows, the Australian Academy of Arts and Humanities, and more.

Of course researchers must communicate the goals and value of publicly funded research to the public who fund it. The ARC has long published such benefit statements. But these statements, divorced from the nuance and detail in the applications, and from the expertise needed to understand their implications, cannot be the test for funding.

Such meddling is unheard of in comparable democracies (like Canada, New Zealand, the UK, the US). Per Britain’s Haldane Principle, once funding parameters, rules and assessment processes are set, the complex and wickedly hard decision as to which research represents the best mixture of originality, significance, feasibility and, yes, benefit should be left where it belongs: in the hands of experts.

As mathematicians, we are not experts in the areas of the vetoed grants – we are the mythical “pub-goers.” So we trust the expertise of those who assessed them. We resigned from the College of Experts because we could not be complicit in a process that does otherwise.

Andrew Francis served on the Australian Research Council’s College of Experts from January 1, 2018 to December 26, 2021, during which time he served on several assessment panels for grant schemes. He also served on Research Evaluation Committees for the ARC during the Excellence in Research for Australia exercises in 2015 and 2018. His university was financially compensated for his time performing duties for the ARC, and he received some of that compensation as a salary loading. He has received competitive research funding from the Australian Research Council to support his research projects.

Aidan Sims served on the Australian Research Council’s College of Experts from January 1, 2019 to December 29, 2021, during which time he served on several assessment panels for grant schemes. His university was financially compensated for his time performing duties for the ARC, and the university made these funds available to him in the form of research-support funding. He has received competitive research funding from the ARC to support his research projects.