What do MPs Think of Randomised Controlled Trials?

A survey of MPs’ attitudes has found unexpected support for using randomised controlled trials (RCTs) to test social policy. It also found tensions over fairness, and a preference for personal stories when talking to the public.

Read Tracey Brown’s blog on the survey results If we want evidence-accountable policies, stop bashing MPs

The report, launched on 13th April 2015, from a face-to-face survey of MPs conducted by Ipsos MORI for Sense About Science, shows that MPs:

– Support the use of controlled trials to design and test social policy

• 67% either tend to support or strongly support.

– Expect that the use of trials in policy will increase

• 40% agree that more trials of social policy in the next few years are inevitable, only 22% disagree.

– Don’t consider the cost of running policy trials a barrier

• Fewer than one-in-ten (9%) agreed that “controlled trials are too expensive as ways of designing and testing social policies”, whereas 47% of MPs, from all parties, disagreed with this statement. 

– Show some confusion about the purpose of control groups

• 35% think that randomly choosing who gets a policy intervention is unfair. Even 26% of MPs who support the use of trials agreed with this statement.
• 64% were in favour of using pilot schemes without control groups, though this might show that they considered them better than nothing
• The absence of strong views on most questions may indicate weak understanding and engagement with the benefits of policy trials

– Strongly emphasise anecdote and personal experience to justify policies to voters

• More than two-thirds of MPs questioned say they’ve used personal experiences (70%) and their own principles (73%) to justify policies, but very few (8% personal experiences and 34% own principles) believed these to be among the things that politicians should pay most attention to. In fact, taken together these responses suggest sensitivity to expertise and evidence, and to the human terms in which social issues and decisions need to be communicated. (The public’s preference for research to be communicated with vivid stories emerged in a study in 2014 by Ipsos MORI and the Royal Statistical Society.)

Prateek Buch, Policy Director of the ‘Evidence Matters’ campaign at Sense About Science, said, “One of the most surprising findings, during an election fought on priorities for state spending, is that few MPs object to the costs of running randomised controlled trials. We hope it is because MPs expect more effective policies to emerge from them. Of course it will only be maintained with good judgement on where RCTs would be most useful, but such broad support for policy experiments will be welcomed by the many bodies who have been pressing for better use of evidence in policy, in areas from nutrition to education to prison reform. ”

Sense About Science has concluded that advocates of evidence in policy now need to stop moaning about MPs, appreciate their skill at communicating, and work with MPs to help them develop a better understanding of randomised controlled trials.

Following the election on 7th May there may be fewer MPs with a background in scientific research, statistics or medicine especially as some champions of science such as Andrew Miller and David Willetts are stepping down. While it is not necessary to have a background in research to interrogate policies or to press for sound evidence it does make the task of improving the understanding of randomised controlled trials among MPs even more important.

Tracey Brown, Director of Sense About Science, said, “It’s clear that the research community needs to explain the benefits of randomised controls more clearly. They are the closest thing a politician will ever have to a magic wand, but not easy to explain to the public. MPs are adept at discussing difficult subjects in human terms, something their critics might forget. So it’s important that they understand well why randomisation is so effective and can communicate this to constituents when it might seem ‘unfair’. We will now involve them in developing our Ask for Evidence public help centre, but would like to see research bodies take up the challenge too.”

Bobby Duffy, Managing Director of the Ipsos MORI Social Research Institute, said, “We have seen rapidly growing discussion of the use of randomised trials in social policy in recent years and we were keen to explore this in our MPs survey, the first time such questions have been put to MPs. Looking across the survey as a whole, it seems around 10% of MPs are firmly in favour of trials, 10% firmly against – with the large majority in the middle, but on balance seemingly open to their benefits. So there is a lot to play for – which makes a proper discussion of the advantages of trials even more important.”

Ben Goldacre, Doctor, Author, and Senior Clinical Research Fellow at the Centre for Evidence Based Medicine in the University of Oxford, said, “It’s great that so many MPs are enthusiastic about getting good evidence on what works in government, and we need to get better at helping them to learn more about how RCTs work. There are some very worrying knowledge gaps here. Many MPs say they’re worried that RCTs are “unfair”, because people are chosen at random to receive a new policy intervention: this is exactly what already happens with “pilot studies”, which have the added disadvantage of failing to produce good quality evidence on what works, and what does harm. ”

Hetan Shah, Executive Director, The Royal Statistical Society, said, “We are pleased to see so many MPs indicating that they see the value of evidence to inform policymaking. It’s now time they acted on their impulse and made the UK the world leader in data-informed-government.”

Andrew Miller, former chair of the House of Commons Science & Technology Committee, said, “Support amongst MPs for RCTs needs to translate into a truly experimental approach to government. We need RCTs designed to help create a better society, such as testing how a delivery mechanism works for a social security benefit. MPs are already getting better at talking to the science community, and the quality of information available to them is increasing. Parliament has recently addressed scientific issues such as mitochondrial donation, climate change and fracking, and in those debates more than ever before good evidence has been at the forefront. We now need to see this progress in social policy issues like crime, education and welfare.”

Julian Huppert, Standing for re-election as Member of Parliament for Cambridge (Liberal Democrat),said, “It is encouraging that so many MPs say they support randomised control trials, but it is clear that there is a long way to go before they fully understand them. It’s a very important issue though – lots of money is, I’m sure, being spent on things that sound good but don’t actually work.”

MPs who want to learn more about RCTs in public policy should watch this short video by Ben Goldacre  and read the paper he wrote with civil servants, called “Test, Learn, Adapt”

Research methodology: Interviews were undertaken as part of the Winter 2014 study of Britain’s Members of Parliament, part of Ipsos MORI’s programme of regular multi-sponsored studies among key audiences. Interviews took place 4th November – 19th December

See the full survey results including graphics here.