Reflections of an RCT rookie

Gillian Hampden-Thompson explores the challenges in conducting randomised controlled trial evaluations in education

In this article, I highlight four challenges that need overcoming in order to grow a broader acceptance of randomised control trials (RCTs), as an evaluation approach, in education. I call these challenges

  • RCT infancy;
  • The medical burden;
  • Ethical transparency; and
  • Infidelity.

I think it is important to acknowledge, while I am involved in two RCT evaluations, my comments and reflections are based on a much broader set of experiences.

RCT infancy

Contrary to much popular belief, RCTs in education are still relatively new. For example in the early 2000s in the United States, when you combined the articles published across the American Educational Research Association (AERA) suite of journals, only 6% used an RCT as the primary research method. This was in contrast to nearly 40% of qualitative studies and nearly 50% that were correlational. In 2003, Grover Whitehurst, head of the Institute of Education Sciences (IES), which is part of the US Department of Education, caused quite a stir when he addressed the AERA delegates in 2003 by saying that: “The people on the front lines of education want research to help them make better decisions in those areas in which they have choices to make, such as curriculum, teacher professional development, assessment, technology, and management. These are questions of what works best for whom under what circumstances. These are questions that are best answered by randomised trials of interventions and approaches brought to scale. These are questions and methods and development efforts with which relatively few in the education research community have been engaged.”

What we know
Four challenges need to be overcome to allow a broader acceptance of RCT:
● RCTs in education are still relatively new.
● Making comparisons with medicine is often not helpful.
● Ethical issues need to be shared.
● Staying faithful can be a problem.

A similar dearth of RCTs existed in the United Kingdom when it came to education research. This was especially true of large-scale RCTs. However, since then there has been a significant sea change. For example, since its inception, more than 100 projects have been funded by the Education Endowment Foundation (EEF) and many have used an RCT to evaluate their effectiveness.

The important point here is that RCTs are new in education. This is in stark contrast to health and medicine, in which evidence of clinical trials dates back to 600 BC. The infancy of RCTs is a challenge and one that needs addressing. Many researchers are like me, in that they have only been involved in large-scale RCT evaluations for a short period of time. Others have no experience at all. This is problematic if RCTs are considered to be an appropriate and necessary method for evaluating educational interventions.

To face the challenge of RCTs being new to education research it is important to cascade the knowledge. More formal training opportunities need to be made available to teachers, education researchers, undergraduates, and postgraduates. In many ways, informal training is taking place as more and more schools become involved in EEF-funded trials. However, this only consists of exposure to RCTs, with only a few teachers actually being involved in conducting RCTs. Maybe there is a role here for the EEF to build in a training element to some of its projects.

Universities, of course, could do more in this regard. How many undergraduate and postgraduate education research methods modules include training in experimental design and RCTs? Anecdotally, I think, probably very few do provide training in RCTs and this is something we have started to address in the School of Education and Social Work at the University of Sussex. More widely this may also be changing through the undergraduate quantitative methods (QM) teaching initiative. At the postgraduate and research level, Doctoral Training Colleges and the National Centre for Research Methods (NCRM) could look to offer more training in this area. In a recent search of the upcoming training programmes available through the NCRM database, no RCT courses or similar were being offered over the next six-month period.

The medical burden

The second challenge I see is what I term the “medical burden”. As we know, in medical research, the RCT is often referred to as the “gold standard” for a clinical trial. The RCT and its gold standard label has been co-opted from the world of medical research and bandied around in education research circles. In many ways, this has not been helpful. At worst, it makes the suggestion that all other research approaches lack rigour and, of course, this is neither fair nor accurate.

In many ways, I would welcome a situation in which researchers who do RCTs in education refrain from drawing parallels with health and medicine and avoid using examples from these fields. In my opinion, the use of clinical trial examples does not always resonate with audiences of educators and social scientists. It is more helpful to try and use education examples when explaining RCTs rather than discussing trials that involve hand washing in hospitals.

Ethical transparency

The third challenge I have found myself grappling with are issues around ethics and RCTs. I am not suggesting that there is a lack of ethical transparency. Instead, I feel that researchers involved in RCTs need to share the challenges they face when conducting RCTs. While avoiding specifics, the two RCTs I have been involved in directly have had their fair share of ethical dilemmas, puzzles, and issues, all of which would benefit from being shared more broadly with the research community and discussed in order to find appropriate solutions.

Related to this, a wider discussion of the ethical dilemmas may benefit those social scientists who serve on university ethics committees. It may be the case that ethics applications are being reviewed by education specialists, but not ones familiar with RCTs. Ethical transparency, in terms of frank and open discussions, would greatly assist these committees in fulfilling their roles.

To somewhat flip the ethical question on its head, we should also be discussing within our field whether it is unethical not to conduct RCTs. There are inventions that need robust evaluations because they have the real potential to make a difference in people’s lives. As such, it is incumbent on us to “speak back” and make the case from an ethical viewpoint for the use of RCTs in education research.

Infidelity

Ensuring that the intervention is delivered consistently and as intended is a real challenge in conducting RCTs. Even with my limited experience, the nature of this issue has become clear to me. Of course, interventions in real-life settings will always be the subject of variability. In a year-long RCT I have been involved in recently, we noticed from the data that students missed some intervention sessions in December. What happened in December? Well, it is the holiday season with schools involved in activities and plays and other activities that result in the regular schedule of lessons being changed. Some of these students had made up sessions of the intervention and others did not. At first blush, I was obviously concerned about this because the students did not get the exposure that was intended. However, then you realise that this is how schools are. They are not controlled environments and if the intervention is effective then this will be the reality of its use in schools. Pupils will miss some sessions because of illness or sports day or a hundred other reasons. While fidelity is a real issue that needs our attention it is vital that we acknowledge these issues and are transparent in our reporting. The process evaluation is, in my opinion, an essential part of an RCT evaluation and is helpful in highlighting issues concerning fidelity.

Clearly, over the past decade RCTs have played a larger role in education research in the United Kingdom and the United States. However, they are still in their infancy and knowledge and experience of RCTs among education researchers and professionals is not widespread. In order to engage the field we need to provide training opportunities, shift our vocabulary away from the medical field and be more open and transparent about issues concerning ethics and implementation. Rigorous large-scale evaluations of interventions are imperative and there is more we can do to meet the challenges laid out here.

About the author

Gillian Hampden-Thompson is Professor of Education at the University of Sussex, where she leads the Centre for Teaching and Learning Research. Working across a broad range of educational projects, her research broadly addresses issues surrounding social justice.

This article is available to subscribers only. If you are an existing subscriber, please login. New subscribers may register below.

Existing Users Log In
   

Published

November 2015