Are evidence-based programmes worth the investment?

Birmingham City Council’s Brighter Futures strategy led to unprecedented investment in evidence-based early intervention and prevention. Louise Morpeth and Michael Little explain more

PUPILS IN BIRMINGHAM WILL PROBABLY be a little happier, smarter, and less badly behaved as a result of the Brighter Futures strategy introduced by the City Council in2007. It focuses on prevention and early intervention strategies, with the aim of improving outcomes for the city’s children and young people. Currently, several programmes are being piloted and, if they prove successful, decisions will be made on whether they will be rolled out across Birmingham.

What we know
● When investment is planned, it makes sense to invest in tested and effective programmes and practices.
● Evidence-based programmes are part of the solution to many social problems, and can therefore be worth the investment.
● Programmes must be implemented with fidelity if they are to be effective.

The Brighter Futures strategy is unusual in two ways: first, for the collaborative way it was written; and, second, for its approach to evidence. The most senior staff in children’s services gave up five days to work co-operatively to draft the strategy, and they were heavily influenced by evidence about “what works”.

Birmingham had great data on children. A reliable picture of the wellbeing of all children aged 0 to 16 had been assembled through a questionnaire administered to large, representative samples of children from across the city. The wellbeing data shaped the strategy, including the outcomes the city would achieve, the activities that would secure these outcomes, and the necessary investments.

Although only tested and effective programmes and practices made it onto the menu of activities, there was concern that the programmes might not work in Birmingham. As a result, every programme on the menu was to be rigorously evaluated in context. Initially, these were the Incredible Years parenting programme, Promoting Alternative Thinking Strategies (PATHS), Triple P parenting programme, and Family Nurse Partnership (FNP). The case for investing in this menu was compelling. Drawing on economic analysis from the Washington State Institute and other reliable sources, the city committed £42 million over five years on the expectation that it would secure £101 million in savings over 15 years.

So, what happened?

The city embraced the strategy, set up an ambitious transformation programme, commissioned three randomised controlled trials, and set about implementation.

After a rocky start, the Incredible Years programme went from strength to strength. Seeking to improve behaviour by fostering positive parenting behaviours, it was targeted at parents of children aged three to six with a probable conduct disorder. Lots of practical help was offered to make the programme accessible, including free taxis, a crèche, and meals.

All the hard work paid off. The evaluation found that the programme had the same positive impact on children’s behaviour in Birmingham as it had in the US and in Wales, where similar evaluations had been conducted.

Unfortunately, this was not the case for Triple P. It was offered to parents of children in primary school who had problematic behaviour and, despite similar concerted efforts to make the programme accessible, it did not make an impact. This result is not a reason to write off Triple P; rather it looks like extra attention needs to be paid to fidelity (the delivery of the programme as it is designed), since this was very variable.

The jury is still out on the other two programmes. The final wave of data from the evaluation of PATHS is currently being analysed. The national impact evaluation of FNP is not due to report until 2013. The process evaluation bodes well, as it has found the programme was delivered with high fidelity.

Data aside, one thing is clear, parents and teachers speak very warmly of these programmes. Mothers talk of how Incredible Years has transformed their relationship with their children. Head teachers’ enthusiasm for PATHS has persuaded many of their peers to take the programme on.

Sadly, amidst all the energy and hard work, two shadows loomed large over the city: a tragic and preventable child death and the economic recession. Both threatened Brighter Futures. The cuts to public spending were so severe that any potential savings created by the programmes were wiped out. Politicians had no choice but to bolster the social work service to better safeguard children.

What has Brighter Futures taught us?

By anyone’s standards the £42 million investment was significant, but it was dwarfed next to the £1.3 billion spent by all the constituents of children’s services. Having “new money” certainly helped, but it did not lever significant change in business as usual. If anything, Brighter Futures was treated as something separate and on the edge.

This was best illustrated by the struggle to demonstrate the value of Incredible Years and FNP to hard-pressed social care staff. They did not see these programmes as being relevant or helpful to the families they were working with. It is understandable that early intervention might seem nice in theory but no use in practice, but it is ironic that FNP is one of the only interventions proven to reduce maltreatment.

The investment in Brighter Futures highlights the difference between a theoretical and actual saving and the need to be able to “realise” benefits. In the case of Incredible Years, economic evaluations suggest that every £1 spent on the programme yields £4 of benefit.

On the face of it this is a great investment. The benefit comes from improved educational attainment, reduced chance of contact with the police, and a modest increase in overall earnings. But how is this translated into hard cash for the investor? Reducing the number of police officers, perhaps?

The original vision for Brighter Futures was for all programmes to be taken to scale, if the evaluation demonstrated their impact. In the case of PATHS, going to scale means delivery in the 260 or so primary schools in the city. So far, PATHS has been delivered in 27 schools with an expansion to 51 under way this year. This is a great achievement, but a long way from scale.

What does this all mean for schools?

1. Schools are a good setting for evidence-based programmes. There are several that can be incorporated into school routines and curricula; PATHS is just one. Since schools are central to many communities and have relationships with large numbers of parents, they are also well placed to offer programmes that help foster positive parenting practices.

2. While not a panacea, evidence-based programmes are part of the solution to many social problems like anti-social behaviour, alcohol and drug misuse, and social illiteracy. Any school concerned with these issues would not be taking a risk if they implemented one or two proven programmes.

3. Good implementation takes effort. A poorly implemented programme gets the same results as not implementing a programme at all, so it is worth going the extra mile and striving for fidelity. Strong leadership, careful selection of staff, good training, regular supervision, and data on implementation are some of the ingredients that have been found to improve implementation.

4. There are economies of scale. There is a greater chance of success, and savings to be made, if schools adopt a programme as a consortium. The costs of training and materials can be shared. Staff can exchange best practice and problem-solve together

5. Probably most importantly, the social benefits of nearly all of these programmes translate into educational benefit. Improving children’s social and emotional literacy has been found to have a direct effect on their educational attainment.

So, in response to the original question, “are evidence-based programmes worth the investment?”, the answer now is a tentative “yes, so long as you do them well”.

Over the next two years, as the results of more UK trials are published and we better understand the costs and benefits of programmes, we will be able to say so much more.

About the authors

Louise Morpeth and Michael Little are co-directors of the Social Research Unit in Dartington. The unit strives to improve outcomes for children by improving the links between research, policy, and practice.

Further reading

Birmingham City Council (2007), A Brighter Future for Children and Young People: The Birmingham Strategy.