Andrew Lansley and his imaginary evidence

Ben Goldacre, The Guardian, Saturday 5 February 2011

I have never heard one politician use the word “evidence” so persistently, and so misleadingly, as Andrew Lansley defending his NHS reforms. Since he repeatedly claims that the evidence supports his plan, let’s skim through what we can find on whether GP consortiums work, the benefits of competition, and the failures of the NHS.

Are GP consortiums better than PCTs for commissioning? There have been 15 major reorganisations of the NHS in 30 years. We’ve had GP fundholders, GP multifunds, primary care groups, primary care trusts, family practitioner committees, purchasing consortiums, and more. After all this change, lots of data should have been gathered on the impact of specific strategies.

In reality, few were properly studied. Here are 4 papers on GP fundholding, which is broadly similar to Lansley’s GP Consortiums. Kay in 2002 found it was introduced and then abolished without any evidence of its effects. In 2006 Greener and Mannion found a mix of good and bad but no evidence that it improved patient care. In 1995 Coulter found nothing but gaps in the evidence and no evidence of any improvement in efficiency, responsiveness, or quality.  Petchley found there was insufficient data to make any judgement. Lansley says he is following the evidence. I see no evidence to follow here.

Next, competition. Andrew Lansley has repeatedly denied that he is introducing competition on price. This is disturbing behaviour: his bill explicitly introduces price-based competition, it’s in paragraph 5:43 of his NHS Operating Framework.

Does variable-price competition work in healthcare markets? It’s hard to measure, but the evidence even on fixed-price competition – where you compete on quality – is mixed. There are various ways to assess it: often people choose an outcome – like the number of people who survive a heart attack – and compare this outcome in areas of more intense or less intense competition. Sometimes competition makes things worse, sometimes better.

Working from first principles, markets where people compete on price as well as quality will probably make quality worse, because prices are easy to measure, while quality is not. The evidence seems to support this theory. The introduction of variable price competition in New Jersey in the 1990s was associated with a worsening death rate from heart attacks, while in the UK, stopping variable price competition was associated with improvement. It’s hard to measure either way, but despite his using the word repeatedly, again, the “evidence” does not support Lansley here.

Lastly, there is the justification for reform. Both Lansley and Cameron overstate our mortality figures to claim that the NHS is failing. Everyone wants more improvement, but money does not produce an immediate and visible reduction in mortality from one thing. Interventions take time to have an impact, especially on things that kill you slowly, and treatment isn’t the only factor affecting how many people die of something. But to take just two things, mortality from cancer has fallen every year since 1995, and heart attack deaths have halved since 1997.

The government claims that our rate of death from heart attacks is double that in France, even though we spend the same on health. Health economist John Appleby instantly debunked this claim in the BMJ, and his piece will become a citation classic. From static 2006 figures in isolation the government is right: but the trajectory of improvement in the UK is so phenomenal that if the straight line continues – as it has done for 30 years – we will be better than France by 2012.

I’m not in favour of, or against, anything here: all health service administrative models bore me equally. But when Andrew Lansley says all the evidence supports his interventions, as he has done repeatedly, he is simply wrong. His wrongness is not a matter of opinion, it is a fact, and his pretence at data-driven faux neutrality is not just irritating, it’s also hard to admire. There’s no need to hide behind a cloak of scientific authority, murmuring the word “evidence” into microphones. If your reforms are a matter of ideology, legacy, whim, and faith, then like many of your predecessors, you could simply say so, and leave “evidence” to people who mean it.

View the original article here

Why is evidence so hard for politicians?

Ben Goldacre, The Guardian, Saturday 12 February 2011

One thing you hope for, with politicians, is that they won’t make the same mistakes over and over again.

Last week we saw that the government has overstated the problems in the NHS by using dodgy figures (to be precise, they used misleading static figures instead of time trends). We saw that Andrew Lansley’s repeated claim that his reforms are justified by evidence was untrue: the evidence doesn’t show that his price-based competition improves outcomes (if anything it makes things worse); and the evidence also doesn’t show that GP consortia improve outcomes (unless you cherry pick only the positive findings). It’s okay if your reforms aren’t supported by existing evidence: you just shouldn’t claim that they are.

Now Lansley’s junior minister Paul Burstow MP has kindly responded, repeating the exact same mistakes again, only more clumsily. I find this, in all seriousness, genuinely frightening from a minister, so I’ll explain how he does it.

The government initially claimed that UK heart attack death rates were twice as bad as France. This was an overstatement: they are, but following recent interventions the gap is closing so rapidly that on current trends it will have disappeared entirely by 2012. In response, Burstow cites a 2008 paper by McKee and Nolte which he says “concluded that the UK had one of the worst rates of mortality amenable to healthcare among rich nations”.

Burstow either misunderstands or misrepresents this very simple and brief paper. It is a study explicitly looking at time trends, not static figures, and it once again finds that comparing 2003 with 1998, the UK still had fairly high rates of avoidable mortality, but these were falling faster than in all but one of the other 18 industrialised countries they examined (meanwhile in the US, avoidable mortality improved at a disastrously slow pace, although they spent more money).

This is a paper showing the success of the NHS, and the fact that we are discussing such a massive improvement in avoidable mortality from Labour’s first term in government is not my choosing: this is the paper that was cited by the Tory minister as evidence, bizarrely, of the NHS’s recent failures.

Next Burstow says I “overlooked the impact assessment we published alongside the health and social care bill, where we present a thorough analysis of the evidence for and against our plans… studies show that GP fundholding and practice-based commissioning delivered shorter waits and fewer referrals to hospitals for patients.”

In its section on GP fundholding, this “thorough analysis” ignores the four peer-reviewed academic papers I described last week, which sadly found no evidence of an overall benefit from GP fundholding. It makes a series of 5 assertions about outcomes, though these are unreferenced to any paper at all.

I contacted the Department of Health, who ferreted out the sources: there was just one, a document from the King’s Fund. It’s not a peer-reviewed academic journal article, but the King’s Fund are pretty good, in my view. If you read this document, it too finds that the results of GP Fundholding were mixed: some things got better, some things got worse.

So the Minister has cherry picked only the good findings, from only one report, while ignoring the peer-reviewed literature. Most crucially, he cherry-picks findings he likes whilst explicitly claiming that he is fairly citing the totality of the evidence from a thorough analysis. I can produce good evidence that I have a magical two-headed coin, if I simply disregard all the throws where it comes out tails.

Here is what politicians apparently cannot understand: it’s fine to make policy based on ideology, whim, faith, principles, and all the other things we’re used to. It’s also fine for evidence to be mixed. And it’s absolutely fine if your reforms aren’t supported by existing evidence: you just shouldn’t claim that they are.

View the original article here