Science textbooks are an important source of information in science classrooms, but the quality of these resources is often poor. Jo Ellen Roseman explains the evidence
CURRENT EFFORTS to improve science education are based on the premise that school leavers need an understanding of the natural world that is richly interconnected. Rather than knowing isolated pieces of information, pupils should appreciate how the most important ideas fit together and how to apply them in a variety of contexts. The way in which these ideas are presented in textbooks is very important, as gaps in textbooks may translate into gaps in pupils’ learning.
To find out what pupils do and do not know, Project 2061 has been gathering data in the US to help clarify how gaps in textbooks relate to what pupils know. Founded in 1985, Project 2061 is a long-term initiative of the AAAS (American Association for the Advancement of Science) which aims to help all Americans become literate in science, mathematics and technology. We have been assessing elements of US national and state science curricula through tests that are carefully aligned to each idea. Findings of these studies have implications for the design and use of curriculum materials, and for the preparation and practice of science teachers.
Gaps in the science story
Studies have shown that textbooks are important in teaching and learning. With this in mind, we conducted a series of evaluation studies on US middle school (age 11-14) and high school textbooks that were in use from 1997 to 2000 to see how effective they might be in helping pupils achieve national and state science standards.
For the nine high school biology textbooks included in our evaluation, we looked at how the textbooks treated several topics – from cell structure and function to heredity, natural selection and evolution. We considered whether each textbook presented the relevant ideas coherently and provided adequate support for teaching and learning.
We began our evaluation by first defining ideas that would be essential to understanding each topic. We expected to see those ideas presented as a coherent “story” in the textbooks and at a level of detail and sophistication that would be appropriate for early high school pupils. For example, in presenting the story of matter and energy transformations in living systems, we expected textbooks to go beyond verbal descriptions of the general principles, giving examples of observable phenomena involving transformations of substances made from carbon-containing molecules.
We also expected textbooks to engage pupils using models to relate their observations about the disappearance of the starting substances and the appearance of new substances to the underlying molecular changes. We were not looking for the details of biochemical reactions of photosynthesis or cellular respiration, but we did expect to see ideas presented so that pupils could develop a Lego-like mental model for visualising the conservation of carbon, hydrogen, and oxygen atoms, even though the molecules built from those atoms were different. We also looked to see whether textbooks made important connections between the ideas – linking prerequisite ideas to key ideas, for example – and the degree to which the textbooks were explicit in describing how one idea related to another.
Our findings showed that, overall, the biology textbooks being used at the time of the evaluation failed to provide a coherent account of the set of ideas or to provide pupils with opportunities to make sense of them. Nonetheless, there was considerable variation across the nine textbooks in how many and which of the ideas were treated. With this data in hand, we then wanted to find out whether there was a relationship between the number of textbooks that covered an idea, and what pupils who were likely to have used those books had learned.
Measuring pupils’ knowledge
Because of the central importance of ideas about matter and energy transformations to an understanding of the relationship between living things and their environment, Project 2061 has also been collecting data on what pupils do and do not know about them. As part of an NSF-funded research and development effort focused on science assessment, we have used a rigorous process to develop, test, revise, and retest questions that are closely aligned with important ideas found in US national and state standards. Each of our questions has been given to approximately 2,000 middle and early high school pupils, and in some cases, high school graduates and university undergraduates.
To examine our textbook evaluation findings in light of our data on pupils’ science knowledge, we used the results from a set of 11 questions we designed to test whether pupils could use a molecular model to make sense of ideas about matter transformation in living systems. We were not surprised to find that fewer than half of the 2,000 or so middle school pupils who answered the questions were correct; most pupils are not taught to think in terms of a molecular model until high school. We then used the same items to measure the knowledge of 200 university students enrolled in an introductory biology course. All of these pupils had taken both biology and chemistry courses in high school and so were expected to have learned about the molecular model for matter transformations.
We found that the high school graduates who were tested were knowledgeable about some of the ideas – that plants make sugar molecules from CO2 and H2O molecules and that animals and plants store food molecules for later use – but much less so about other ideas. By comparing the findings from our biology textbook evaluation with these assessment findings, we see in Table 1 the relationship between the number of biology textbooks presenting an idea and the percentage of pupils that appear to have learned it. The greater the number of textbooks covering an idea, the better pupils performed on the relevant assessment items.
|Idea||Number of textbooks (out of 9)||% Students getting item correct|
|(Idea A) What food is||5||50% (2)|
|(Idea B) Plant food making||9||74% (1)|
|(Idea C) Converting food into body structures of plants||2||21% (1)|
|(Idea E) Converting food into body structures of animals||2||25% (3)|
|(Idea G) Storing food for later use||7||63% (4)|
Our results suggest what many science educators have argued – that mere presentation of ideas, even if the presentation is coherent – is not sufficient to promote deep understanding. Even when all of the textbooks covered an idea, for example, that plants make their own food, only 74% of pupils who had already taken chemistry and biology courses responded correctly. Pupils need to be actively engaged in sense making, which depends on the quality of curriculum and teaching.
While we were not able to examine the quality of teaching pupils encountered, we did have a good sense of the quality of support provided by the textbooks for teaching and learning. Not surprisingly, Project 2061’s biology textbook evaluations showed that to be poor across books. Given the picture that emerges from our work, there is a need for more detailed and systematic study to determine the precise nature of the relationship between textbooks’ treatment of particular ideas and pupils’ learning of them. It is also clear that the poor quality of textbooks can place a burden on teachers to fill in the gaps. Preparation and ongoing professional development can help teachers develop a coherent understanding of the science ideas their pupils are expected to learn, to evaluate the strengths and weaknesses of the materials they are using, and to select and use teaching strategies that are most likely to be effective.
About the author
Jo Ellen Roseman is Director of Project 2061. In this capacity she is responsible for overseeing all of the project’s programmes and activities in the areas of curriculum, teaching, and assessment. Roseman also serves as Director of the Center for Curriculum Materials in Science (CCMS), a collaboration of Project 2061, Michigan State University, Northwestern University and the University of Michigan.
American Association for the Advancement of Science (1989), Science For All Americans. New York: Oxford University Press.
American Association for the Advancement of Science (1993), Benchmarks for Science Literacy. New York: Oxford University Press.
American Association for the Advancement of Science (2005), High School Biology Textbooks: A Benchmarks-based Evaluation. Retrieved on September 27, 2009, from www.project2061.org/publications/textbook/hsbio/report/default.htm
DeBoer GE, Herrmann-Abell CF, Gogos A, Michiels A, Regan T, & Wilson P (2008), Assessment Linked to Science Learning Goals: Probing Student Thinking Through Assessment. In J Coffey R Douglas, & C Stearns (Eds.), Assessing Student Learning: Perspectives from Research and Practice (pp. 231–252). Arlington, VA: NSTA Press.
DeBoer GE, Herrmann-Abell CF, Wertheim J, & Roseman JE (2009, April), Assessment Linked to Middle School Science Learning Goals: A Report on Field Test Results for Four Middle School Science Topics. Paper presented at the conference of the National Association for Research in Science Teaching (NARST), Garden Grove, CA.
Fulp SL (2002), 2000 National Survey of Science and Mathematics Education: Status of Middle School Science Teaching [Electronic copy]. Retrieved March 23, 2010, from http://2000survey.horizon-research.com/reports/mid_science/mid_science.pdf.
National Center for Education Statistics. (2008). Science Framework for the 2009 National Assessment Of Educational Progress (NAEP) http://www.nagb.org/content/nagb/assets/documents/publications/frameworks/science-09.pdf.
National Research Council (1996), National Science Education Standards. Washington, DC: National Academy Press.
Schmidt WH, McKnight CC, Houang RT, Wang H, Wiley DE, Cogan LS, & Wolfe RG (2001), Why Schools Matter: A Cross-national Comparison of Curriculum and Learning. San Francisco: Jossey-Bass.