Rolling out evidence-based programmes in schools

Judy Hutchings and Suzy Clarkson consider the lessons learned from rolling out and evaluating KiVa, an anti-bullying programme, in the UK

There are challenges to implementing and evaluating any evidence-based programme under real-world conditions. Without attention to fidelity it is unlikely that the outcomes from coal-face replications (effectiveness trials) will match those of the research trials (efficacy trials). This is the main challenge of our work at the Centre for Evidence Based Early Intervention (CEBEI) and the focus of much attention across the field of psychosocial interventions.

What we know
● When rolling out an evidence-based programme, there are challenges for schools, implementers, and researchers.
● Time, communication, and technology are common themes.
● Overcoming these challenges is a continuing process.

KiVA, a school-based anti-bullying programme, originated in Finland. There, even with government funding, the results from the large-scale roll-out, although still very positive, were not as good as those from the original randomised controlled trial (RCT) (www.kivaprogram.net). The question with any multi-component intervention is “what needs to be done to achieve results that match those of the efficacy trials?” When the task is taking an evidence-based intervention and delivering it in another country/cultural context there are added challenges.

Our first potential barrier in bringing the programme to Wales was easily overcome. A brief presentation at a Welsh Government meeting of school improvement officers resulted in the inclusion of KiVa on a list of eligible evidence-based programmes for a Welsh Government School Improvement Grant.

The School Improvement Grant funded training and resources for 14 pilot schools in Wales and a visit by Prof Salmivalli, the KiVa programme developer, to deliver the initial training. At the same time, CEBEI staff were trained to support implementation and subsequently trained in Finland to be trainers.

We next obtained European Knowledge Economy Skills Scholarship (KESS) funding for a Master’s project to evaluate the pilot trial in Wales. Survey results from this project were good, showing significant reductions in both bullying and victimisation, and teacher feedback was extremely positive, reporting that the lesson plans were easy to use and had a positive impact on child well-being and behaviour. A follow-up KESS-funded Master’s project reported that the reduced bullying and victimisation of children after one year had been maintained for those children who had received the programme for two years.

The results of the pilot trial led to the BIG Lottery Fund supporting an on-going small effectiveness RCT in 20 schools. This project funded participating schools for two years with all training, resources and registration costs. In addition, it overcame another implementation barrier for Wales by paying for all child and parent material to be translated into Welsh.

Despite the funding that enabled the initial introduction of KiVa into Wales, there have been a number of challenges for schools and for us at CEBEI, as both implementers and researchers.

Challenges for schools

  • The main challenge for schools has been cost. This was not a problem for the pilot trial and RCT, where the costs were grant funded. However, once these trials had ended and schools had to pay the annual registration fee of £2.50 per child per year, several opted out. All declared their intention to continue with lessons, but they are deprived of ongoing support and access to material, such as annual survey feedback and online games for children that can be played at home.
  • Turnover of staff. Schools must identify and train a school lead and a KiVa team lead. The role of the school lead is to support staff in all aspects of the programme, ensuring that teachers are trained, that resources are available, and that the annual online child survey is completed. Significant changes in staff in primary schools have meant that some schools have lost their KiVa leads and/or the lead has had too many other responsibilities.
  • Creating a consistent understanding of bullying across the parents/community.
  • IT challenges. Most primary schools in Wales do not have the IT resources to sit a whole class at the computer to take the annual on-line survey. It takes time and planning to implement this.
  • Other IT challenges. Teachers have made errors when logging in to the KiVa site and/ or losing passwords.

Solutions for schools

Our support and feedback meetings have identified the following tips on successful implementation:

  • It must be a whole-school approach and be reviewed regularly in staff meetings.
  • It must have the full commitment of the school head teacher, although it is best if the head is not the KiVa school lead.
  • The KiVa school lead must have time to take on this role and oversee the IT issues, or work with a designated IT person.
  • The forms in the teacher manual to screen potential bullying incidents are very helpful. This also aids record keeping and several schools have found this to be helpful in school inspections and as a record when faced with challenging parents.
  • Creating a link to the KiVa website and resources through the school website enables parents and children to access material. Several schools report having found it very helpful to be able to direct parents to the KiVa parent website and to explain the KiVa definition of bullying, process and approach to bullying.
  • The KiVa school lead must ensure that teachers deliver lessons and complete lesson evaluations as soon after the lesson as possible. Some schools have designated a specific time on the timetable when every Key Stage 2 (age 7-11) class will be receiving a KiVa lesson.
  • Some schools have trained one staff member, often a support staff member, to take the survey with the whole of Key Stage 2 (age 7-11) and this has ensured consistency and been time effective.

Challenges for implementers

  • Communication. Schools receive so much spam, and both emails and letters can be overlooked.
  • Funding for local authorities has been reduced. We have to target schools directly and school staff are not always familiar with evidence and how to interpret it. There are a lot of non-evidence-based programmes available to schools.
  • School inspectors in Wales will not identify individual programmes so, even when they give a glowing report on the implementation of KiVa in a school, the programme itself is not identified.
  • There can be a high turnover of primary school staff.
  • Some schools do not recognise a bullying problem; there is still a significant mismatch between child-reported and school-identified levels of bullying.
  • It can be hard to ensure the availability of training, resources, and staff to be able to follow up expressions of interest.

Solutions for implementers

  • Support existing schools in disseminating information on their successes to other schools.
  • Include presentations at conferences.
  • Write in professional journals and target newspaper and television coverage.
  • Place information on key websites.
  • Provide support. We have continued to provide termly support meetings in North and South Wales, and advice and support on-line from CEBEI.
  • Provide local support. For successful implementation, KiVa needs to be supported by a local trainer. CEBEI is the approved training centre for the UK, but our goal is to train and support a network of trainers across the UK.

Challenges for researchers

Some challenges have arisen from our determination to establish an evidence base for KiVa in the UK. They include:

  • Asking for additional data from schools.
  • A lack of understanding of RCTs, particularly for control schools, which, despite clear contracts, failed to understand the data collection requirements because schools did not pass on emails.
  • Challenges in obtaining ethical approval/ parental consent for access to child data.
  • Not all school staff have received the training that the KiVa lead should deliver within the school, making it hard to ensure the quality of lesson delivery and to collect evidence on implementation quality. This is very dependent on the enthusiasm of the KiVa lead and/or head teacher.

Solutions for researchers

  • Continue to seek funds for both RCTs and other kinds of evaluations. KESS European funding has enabled two years of follow-up in the pilot schools.
  • Continue to support schools in dealing with IT issues through termly consultation days and email and phone support.

Conclusion

Children deserve evidence-based education and, with the support of schools, we are working to overcome barriers to the evaluation and dissemination of KiVa across the UK, hopefully in ways that will ensure effective delivery.

About the author

Judy Hutchings leads the Bangor University Centre for Evidence Based Early Intervention. She has undertaken trials and published results from more than 30 years of research into parenting and school-based programmes to prevent and reduce violence. Judy led the pilot KiVa trial.

Suzy Clarkson is a research project support officer for the KiVa RCT.

Further reading

Hutchings J, Bywater T, and Daley D (2007), Early Prevention of Conduct Disorder: How and Why Did the North and Mid Wales Sure Start Study Work? Journal of Children’s Services, 2(2), 4–14. doi:10.1108/ 17466660200700012.

J (2012), Introducing, Researching and Disseminating the Incredible Years Programmes in Wales. International Journal of Conflict and Violence, 6(2), 225–233.

Hutchings J and Clarkson S (2015), Introducing and Piloting the KiVa Bullying Prevention Programme in the UK. Educational and Child Psychology 32 (1), 49–61.

This article is available to subscribers only. If you are an existing subscriber, please login. New subscribers may register below.

Existing Users Log In
   

Published

November 2015