Before choosing, ask three questions

Steve Fleischman explains the difficulties in making use of research, and offers some guidance

YOU MAY BE FAMILIAR WITH RECENT NEWS that the Federal Trade Commission (FTC) fined Reebok $25 million in September 2011 for making unsupported claims regarding its EasyTone line of athletic shoes. According to the UK’s Daily Telegraph newspaper: “Fitness and toning shoes have been claiming to work leg-sculpting miracles for several years now.” However, the FTC found these claims to be unsubstantiated. Reebok may no longer assert that the shoes tone leg muscles or produce firmer, shapelier buttocks. As the director of the FTC’s Bureau of Consumer Protection stated: “Consumers expect to get a workout, not to get worked over.”

What we know
● Making choices is a complex and challenging process.
● Providing educators with reliable and useful evidence can support effective choices.
● Sound judgment, driven by critical questions, can improve decision-making. What is the recipe for using evidence successfully?

Unfortunately, as educators we are all too familiar with claims that turn out to be too good to be true. Take the concept of “learning styles.” Perhaps you have used the concept to guide your classroom instruction? After all, the concept and practice have been widely shared for nearly two decades. Yet, you may be surprised to learn the conclusion reached in December 2008 by experts who reviewed the research literature on the subject and found “at present, there is no adequate evidence base to justify incorporating learning-styles assessments into general educational practice.”

The team wrote: [We] found virtually no evidence … validating the educational applications of learning styles. Although the literature on learning styles is enormous, very few studies have even used an experimental methodology capable of testing the validity of learning styles applied to education. Moreover, of those that did use an appropriate method, several found results that flatly contradict the popular … hypothesis.

The review cautioned against the practice until better evidence is provided. However, this has not stopped publishers and developers from making large amounts of money in promoting the use of “learning styles” in schools.

Thus, while teachers may be protected when it comes to which shoes they wear to tone their muscles, they have little protection against choosing to use what may be ineffective practices in their classrooms.

Why is choice so difficult?

Experts in the field note that making choices is a complex, challenging, and often irrational process. In The Art of Choosing, Sheena Iyengar, a business professor at New York’s Columbia University, points to psychological, sociological, and cultural factors that shape views, attitudes, and actions related to choice. She cites examples demonstrating that in blind tests of products such as water or wine, novices often fail to distinguish the most expensive or “best” from the rest. As Iyengar points out, “Most of us need to rely on external information in order to choose well.”

In another example, Iyengar reports on an experiment she conducted that demonstrates that “too much” choice can lead to confusion and a decrease in choice making. With the cooperation of a large supermarket, she set up a booth that at different times offered tastings of either 24 or 6 of the jams produced by Wilkin & Sons. Customers who tasted the jams received a coupon allowing them to purchase their preference at a discount. Iyengar reports that “30% of the people who had seen the small assortment decided to buy jam, but only 3% bought a jar after seeing the large assortment.” Thus, having more choice can inhibit choice!

In How We Decide, science journalist Jonah Lehrer provides further examples of how our choices are often irrational or poorly understood. Lehrer points to a particularly modern challenge – having access to too much information – which he argues “can actually interfere with understanding.” He writes that when there is too much information, one’s brain is overwhelmed and “a person can no longer make sense of the situation. Correlation is confused with causation, and people make theories out of coincidences.”

What type of information do “education consumers” want?

If making choices is a difficult process, which often requires outside guidance, then what type of help would education decision makers like to receive? My colleagues at Education Northwest explored this question in a 2009 study on evidence use in education. They reported a gulf between research design and “real world” practice, which often results in findings with limited applicability. Furthermore, they found that educators struggle to acquire, interpret, or apply research because of their own lack of knowledge, skills, and time.

Despite these challenges, the study points to three principles which, if followed, might support improved use of evidence in education:

1. Research should be contextualized. Participants expressed strong preferences for research evidence linked to local contexts.

2. Research should be easy to read, absorb, and apply. Participants asked that research evidence be presented in brief reports, written in non-technical language.

3. Research should be “translated” and “transmitted” by intermediaries. Participants defined intermediaries as trusted organizations and individuals that help to locate, sort, and prioritize available evidence.

Evidence isn’t everything: Three critical questions

The past decade has seen an increase of evidence in education. Today, “consumer-reporting” projects such as the Best Evidence Encyclopedia (UK and U.S.), EPPI-Centre (UK), and What Works Clearinghouse (U.S.) provide evidence reviews of leading programs and practices. However, having better evidence is not enough. Ultimately, we must become more sophisticated consumers of information.

My experience as a teacher, researcher, and technical assistance provider has convinced me that making sound choices in education depends on asking three fundamental questions consistently: What works? How do you know? So what?

  • What works? Our actions should be guided by the principle that the best approaches to use in education are those that are most likely to get the desired results. Asking this question compels us to reject opinion, ideology, marketing, and popularity and instead ask whether the proposed educational approach works. Improvement approaches should not simply sound good, but rather, should work well.
  • How do you know? In the U.S. it is rare to encounter an education program or practice that does not claim to be “research based.” We must go beyond these claims and, in a spirit of healthy skepticism, demand the compelling evidence that a program does work. The “consumer-reporting” projects mentioned before can provide support in answering this question.
  • So what? This question is focused on significance. It suggests a series of critical questions that go beyond looking for generalized demonstrations of the effectiveness of a program or practice. Questions of this type include: Will the program be effective with my students or work in my context? Given the level of demonstrated effectiveness, is this program or practice worth the resources required to adopt and implement it? Are there better alternatives? What kind of help will I receive so that this program gets the same effects in my setting as it has elsewhere?

Iyengar concludes her book by observing, “Science can assist us in becoming more skillful choosers, but at its core, choice remains an art.” As with all else in education, the best results for students will emerge when educators combine the best available evidence with keenly honed professional insights and judgments, and a set of questions that help them to avoid their own decision-making biases.

About the author

Steve Fleischman is deputy executive officer of Education Northwest (, a non-profit organization based in Portland, Oregon. The organization conducts research, evaluation, technical assistance, training, and strategic communications activities to promote evidence-informed education policy and practice.

Further reading

Iyengar S (2010), The Art of Choosing. New York, NY: Hachette Book. Lehrer J (2009), How We Decide. Boston, MA: Houghton Mifflin.

Lockwood AT, and Fleischman S (2010), Choosing a School Turnaround Provider. Lessons Learned, 1(3), 1–4. Portland, OR: Education Northwest.

Nelson SR, Leffler JC, and Hansen BA (2009), Toward a Research Agenda for Understanding and Improving the Use of Research Evidence. Portland, OR: Northwest Regional Educational Laboratory.

This article is available to subscribers only. If you are an existing subscriber, please login. New subscribers may register below.

Existing Users Log In


February 2012