If the prime minister David Cameron is to be believed, his flagship troubled families programme (TFP) has been a miraculous success, having saved the taxpayer more than a billion pounds.

The trouble with Cameron’s bold assertion is that, while there is some qualitative evidence that TFP has enabled some improvements in professional practice, there appears to be no solid data to underpin his wider claims.

The seemingly runaway success of TFP was just one of several bold declarations made by Cameron in his welfare speech in Runcorn on Monday morning. Introduced under the last government, TFP identifies (in theory) the most vulnerable families and prepares innovative, focused interventions to help them.

These intensive interventions, the prime minister said, had “changed lives” by tackling seemingly intractable problems of worklessness, addiction, truancy and anti-social behaviour:

I can announce today that almost all of the 117,000 families which the programme started working with have now been turned around – in terms of either school attendance or getting a job or both. This has saved as much as £1.2bn in the process.

Such is its success, said Cameron, that TFP is to be expanded to 400,000 more families by the end of this parliament, and the principle behind it – the so-called Payment by Results (PBR) approach – will be applied to those other parts of the public sector “where state institutions have all too often failed”.

But is the prime minister correct about the achievements of TFP?

Jonathan Portes, the head of the National Institute of Social and Economic Research, had a view of its success back in March, when very similar claims for the TFP were made by the then communities secretary Eric Pickles:

We have, as of now, absolutely no idea whether the Troubled Families Programme has saved taxpayers anything at all; and if it has, how much. The £1.2bn is pure, unadulterated fiction.

Portes, who repeated his criticisms of TFP on Monday morning, points out that although an evaluation has been commissioned of the TFP, its findings have yet to be published. The £1.2bn savings figures is based on rough, self-reported estimates by councils who (in theory) have a financial interest in presenting optimistic assessments of their own TFP performances (they get paid by the number of families they “turn around”).

Moreover, the TFP data, Portes pointed out, is not properly tested. There has been no assessment of what would have happened to these families had they not been subject to the programme. There is no counterfactual, and therefore no legitimate basis on which to make a savings estimate.

Portes added:

Until the results of the evaluation are in, we know nothing about whether there any savings, let alone what they are.

The government confirmed on Monday afternoon that the results of the formal evaluation had not yet been published.



The Durham university researcher Stephen Crossley also looked at these figures on their first official airing in March, and noted the staggeringly unlikely assumptions on which the data Cameron uses are based:

The TFP has apparently achieved a 100% success rate in local authorities identifying exactly the same amount of troubled families in their area as the indicative number provided by the government – the total number of families identified and worked with by local authorities comes to 117,910. Every single local authority has identified every single ‘troubled family’ in its area and every single local authority has begun working with them. The practice on the ground, therefore, represents a perfect alignment with the information provided by politicians and civil servants in Whitehall. This is despite the data used to identify the number of ‘troubled families’ being six years out of date and compiled for completely different purposes and identifying potentially very different families.

Crossley, who has made other detailed criticisms of TFP, points out that unlikely savings aside, there is an air of unreality about the statistical claims for the success of the on-the-ground work with familes:

It is ... staggering that work with some of the most disadvantaged families who have allegedly been immune to all previous policy interventions and whose ‘troubles’ have existed ‘for generations’ has been so successful at a time of wide-ranging and deep cuts to local authorities and others’ public services.

Portes remained similarly incredulous:

In other words, CLG (Department for Communities and Local Government) told Manchester that it had precisely 2,385 troubled families, and that it was expected to find them and “turn them around”; in return, it would be paid £4,000 per family for doing so. Amazingly, Manchester did precisely that. Ditto Leeds. And Liverpool. And so on. And CLG is publishing these figures as fact. I doubt the North Korean statistical office would have the cheek.

Cameron nonetheless heralded Payment by Results as an approach which should be rolled out to other services. His enthusiasm is not shared by the National Audit Office (NAO) however. Only last week, it urged caution on PBR, not least because there was so far no evidence that it offered value for money, and no sign that Whitehall was monitoring how well PBR was working.

The head of the NAO Amayas Morse concluded:

Payment by Results potentially offers benefits such as innovative solutions to intractable problems. If it can deliver these benefits, then the increased risk and cost may be justified, but this requires credible evidence. Without such evidence, commissioners may be using this mechanism in circumstances to which it is ill-suited, to the detriment of value for money.

There is support among some professionals working with troubled families for the changes in practice that the programme enables. The family work is as a result often more sustained, and the problems addressed more wholistically, with agencies working together. Anecdotally, results with individual families have been positive.

But even its supporters have acknowledged that its benefits, like those of many preventative interventions, are often hard to quantify; and that there is a risk that these elements of good practice have been lost in the political imperative to over-claim for what Crossley called, with withering sarcasm: