mathematizing4all

“The design flaws alone [of Project Follow Through] have caused the study to be called a good example of a bad evaluation” (Kennedy, 1978, p. 11)”

“So the ultimate defense of the evaluation [of Project Follow Through] is its objectivity, but its objectivity melts away upon inspection. The measures turn out to be reliable but of questionable validity. (Some are also unreliable.) The statistics are more or less proper but sometimes inappropriate, the interpretation from data questionable. The evaluation is replicable, reliable, and invalid.” (House, 1979, p. 38)

When I found out in the middle of October that the new season of Sold A Story was about Project Follow Through and Direct Instruction, I was surprised, mostly because I had just spent about a week researching and writing about the same fairly obscure research study from the 1960s. The book, which I am writing with Maisie Gholson and Chuck Munter, is about research in mathematics teaching and learning with a goal to provide math leaders a deeper understanding of the complexity of research in order to help us all focus on what matters and resist oversimplified narratives. Right now, I have only seen the trailer. This blog post, hopefully one in a series, is about Project Follow Through, not about the podcast or how it covers the story.

While Project Follow Through is probably unfamiliar to math educators of today, it is not new to me, or other special education researchers. According to the narrative I have heard from advocates of Direct Instruction, Project Follow Through was the most massive educational study in US history, and Direct Instruction “won” and the education establishment buried this story to promote other, failed pedagogies. This is not a new gripe, not a new concern, not a new story to me. However, this narrative of grievance appears to be new to many in education outside special education.

But I actually wasn’t writing about this angle in the book. Instead, I was using Project Follow Through to explain something more general about quantitative educational research. Policymakers, superintendents, certain researchers etc. too often paint quantitative research as the only kind of research that can answer the questions that matter. Well- designed quantitative research CAN help us understand certain questions, and critically, can uniquely help us understand whether or not something caused something else. However, quantitative research is best understood by the maxim of Spiderman (or Spiderman’s uncle?) – with great power comes great responsibility. Because stakeholders tend to trust quantitative findings, we need to be sure that the studies are well-designed, that measures are accurately measuring what we want to measure, and that our models do not over or under estimate effects too much. Because these design flaws can provide results that look true, but are not. That is the other story of Project Follow Through. For most experts in educational research, the story of Project Follow Through is one of a massive failed quantitative experiment, one in which mistakes in methodology were made at multiple levels over many years. The take-away point is not that there was a vast conspiracy to hide the outcomes (as advocates for Direct Instruction argue, and the Sold a Story folks seem to have bought hook, line and sinker), but that this was basically a series of missteps methodologically that mean that the research is highly problematic, whatever it found. The main finding that could be mathematically supported is that educational research depends more on the implementation than on the pedagogy.

Project Follow Through emerged in the context of the late 1960s, where there was a unique moment in which the federal government used resources to attempt to understand and eradicate poverty, the War on Poverty (House, 1979). Project Follow Through was designed as a service program first, experimentally extending Head Start into elementary school up to Grade 3, and a research project only secondarily. For many involved, the most important aspect was a massive investment in early childhood education through multiple programs. The problem was that there were not models of early childhood that they could agree were effective, and that political pressure from parents and communities at the beginning of the study demanded local control. The study used a new methodology which they called a planned variation study, meaning that it was understood that there were not already in place sufficient models of early childhood pedagogy to be compared. Instead, a variety of groups of early childhood educators were funded from the late 1960s until the early 1970s to create new, more well -defined early childhood pedagogical models and then implement them in multiple sites across the country. The planned variation was in the multiple models of early childhood teaching. It was then, by design, a study of emergent models, ones that would be carefully studied even as they were in development and being rapidly expanded. Thus, the project began more as a way to stimulate innovation in early childhood than to systematically study it. Asa Hillard (1978) describes the entire project as deeply flawed from the beginning because the study was not evaluating known models, but developing new models and researching them as they were built;

When the Office of Education asks the question, “Does Follow-Through work?” it sounds as if it is an evaluation question. In reality, it is not. It is a research question, for the simple reason that, at the outset, neither Follow-Through itself nor the planned variations were programs of known instructional quality.” (Hilliard, 1978, p. 24)

In this way of thinking, evaluation is reserved for studying known quantities, not to study emergent change.

Evaluation of such a complex and large project was necessarily complex, particularly with the methodological tools of the time and the multiple objectives of the project. Mary Kennedy from the US Office of Education, whose office funded the project, described it “not a true experiment, and not even a well-designed quasi-experiment”(Kennedy, 1978, p. 3). School districts were invited to apply, and then chose the model they would implement, as well as the classrooms, teachers and students who would participate. Comparison groups of students were chosen only afterwards. Kennedy notes that the control groups of students tended to be higher income than the groups that the district chose to put in Follow Through classrooms, as the districts’ goal tended to be to most impact student achievement by including the students who needed the most support. This design is not a randomized-controlled experiment, then, as who participates in what is not random. Giving school districts choice in what they implemented and how was probably extremely useful in getting additional participation, however. There was also wide variation in the implementation of the models, something that was not measured in a systematic way (Kennedy, 1978). These models, highly variable and quite different in their design and professional learning expectations, were implemented by project sponsors with little experience implementing their models (many of which were still being designed as they were rolled out) across the models. As Kennedy notes, it is probable that those models that were implemented with more fidelity were those who were easier to learn and more insistent on being implemented in certain ways. These are all issues in which quantitative analysis rests upon- issues with control groups, fidelity of implementation and research design are considered critical to being able to trust that the results of quantitative analysis are actually telling a story we should believe in.

This large study included different approaches to early childhood. It included models of early childhood education that were more progressive, such as one from Bank Street, where children were guided to develop concepts and skills through free play, as well as Direct Instruction, in which the young learners are expected to sit down and learn basic skills from scripted teacher modeling and demonstration. Some programs were more focused on affective and emotional development of the young students, other programs on skill-building. By the way, Direct Instruction (what we in the special ed business call “capital D capital I Direct Instruction”) is not the same as lowercase “direct instruction.” The latter is a general term for teacher-directed instruction that includes elements of teacher modeling and student response, with guided release of responsibility. Direct Instruction is a very specific pedagogy that follows a script that is carefully designed to remove all ambiguity from language. The theory is that any topic can be explained in a way that makes it understandable (I highly recommend reading Engelman’s book if you want to understand this). It includes choral response from students. It is very much a particular thing, not a general theory of education. Just telling kids something is NOT Direct Instruction (TM).

These models of education were quite different, and the project sorted them into three categories: Basic Skills, Cognitive-Conceptual, and Affective-Cognitive. Already, we can see another issue arising. How would one design an evaluation to compare  preschool pedagogical methods with such different goals? It would be all too easy to pick an assessment that would favor one group or another. Perhaps measures of emotional development would favor Affective-Cognitive. An assessment of basic skills might favor approaches, like Direct Instruction, in which students are spending their time learning those skills. Thus, to actually compare, one would want a healthy group of measures to assess the young student’s development and learning in different ways.

As early childhood models were recruited, the project assured them that there would be multiple measures to assess the different focuses of the programs. Multiple evaluators worked on the issue of measures. The initial data set included mixed methods, or both quantitative and qualitative analysis. For example, over 60,000 parents were interviewed, for example, data which never appeared in the final evaluation (House 1979).

One more note about models: this study did not compare Direct Instruction to what we would consider standards-based math instruction of today (or inquiry or guided inquiry etc). This study compares emerging models of early childhood in the late 1960s including Direct Instruction, which developed rapidly as it expanded through the funding Follow Through provided (Engelmann, 2007). I know from my 31 year history as an educator that much has changed in models of early childhood since I started, let alone since 1968. If Direct Instruction hasn’t changed since then, that is an interesting question to discuss for another blog post. Think for example how our understanding of Piaget’s work has shifted over time. I would certainly not assume the constructivism of the late 1960s was the same as early childhood constructivism now.

After years of arguing about measures, a set was chosen that seemed to make no one happy (House et al., 1978; Hilliard, 1978; Kennedy, 1978). One assessment was the MAP test, a basic test of mathematics and reading, which was normed and validated for this use.

Another test was the Raven’s Progressive Matrices Test (recent version here), which is a test of nonverbal problem solving, and has been used as an IQ test (Hilliard 1978).  This test was described as being aligned with the cognitive-conceptual group. It is a major design flaw to use an IQ test as the primary measure of an educational intervention (Hilliard 1978). While we may currently have more flexible understandings of IQ as being malleable, at the time, IQ was understood as something that would not change over the lifetime. So as an assessment of conceptual development, this is problematic. At the same time (reflecting the messiness of this as an assessment item), there was evidence that this test was teachable, or that one could just give these kinds of items to students and they would improve (Hilliard, 1978). In other words, this was not a great assessment choice to reliably and validly assess the learning in early childhood models.

The measures of affective growth were the most problematic of all (House 1979). There were also measures of self-esteem and independence, but there were significant reported difficulties with the alignment and delivery of these scales, as the items were delivered whole group to young children (House et al., 1978). Some who analyzed the study found the measures of affective growth to be so problematic as to be useless (Kennedy, 1978). Thus, while these very different early childhood programs were told that they would be assessed by a battery of assessments that would be sensitive to the growth they were designed for, only one measurement tool was considered valid and reliable- the assessment of basic skills. These issues were all reported in the original evaluation (Stebbins et al., 1977) and focused on in a critique of that evaluation (House et al., 1977). As Hilliard writes of the “compromise” that resulted in these four assessments:

It is a virtual surrender to triviality. There simply must be some point beyond which essential meaning for evaluation has been compromised away . . . when the mismatch between the content of instruments and program descriptions is as gross as appears to be the case in Follow-Through (p. 20)

This issue of measures was a political one as well. Early in the project the community collaborators were promised that the project would evaluate a broad set of goals: “the impact of Follow Through on pupils, parents, school personnel, members of the community, and the school as a social institution in those settings where Follow Through programs are initiated”(House, 1979) as well as provide detailed descriptions of the emerging models. However, when assessment began, it was limited to the cognitive measures. This led to a protest in the offices of Project Follow Through in 1968, which seemed to result in proposed changes but no material shift in evaluation. Community members knew that the measurement mattered, and they wanted measurement of what mattered most to them. There is a lot of history here that connects to the civil rights moment, to Black Power, and to how students in poverty were framed as the problem that made the program fail (Hilliard 1978; House 1979). The development of Direct Instruction prompts similar questions about power (Engelmann, 2007). I hope to expand this analysis in a future (perhaps collaborative) post.

The main finding of the original report was the variability across sites (Stebbins et al., 1977). There was wide variability in the effectiveness of (as measured by these tools) all the approaches. Overall, this reminds us that there is no perfect tool, and it is important to focus on how well we implement any educational curriculum. The original report goes into detail into this finding. This graph, from an article written by one of the evaluation team (Andersen, 1977), shows this variability. You can see each model (such as Direct Instruction)’s average test scores for individual sites, as well as pooled across sites.

The second finding of the original report was that when the models were compared, the Direct Instruction model was the most successful on several measures. I cannot overestimate how much argument and articles were published critiquing every element of this evaluation, with pushback coming from multiple directions. Some felt that the findings overemphasized the scores of Direct Instruction, others that it minimized the scores of Direct Instruction. I chose this image because it shows the top 2 findings of the report: 1) variability across sites for all models and 2) the higher averages of Direct Instruction in basic skills.

Even posting this graph to those who follow my work (mostly math educators) probably means that some readers are jumping into the narrative of “who won.” But remember what I already wrote about even just some (yes, I did not detail ALL the critiques of methodology I found) critiques of this study. The issues with the design of this project, combined with a critique of measures that were invalid (did not measure what they were supposed to be measuring) means that any result about what is worse or better is a finding, but one with so many limitations. If you dislike Direct Instruction, and feel compelled to argue that Direct Instruction did not “win” in the category of Cognitive Conceptional Skills, I will remind you that the measure for that analysis was a test of manipulation of visual patterns. Indeed, the only section with a measure that was described by experts consistently as both valid and reliable is in the domain of Basic Skills. This is not a well designed study that gives us tons of actionable information. It is a 60 year old study that is well known as a project that was way too big with way too many cooks in the kitchen, one that serves as a reminder on how NOT to do educational research. Based on that, the only finding about models that seems justifiable is that Direct Instruction was more successful overall at raising scores on basic skills. As House noted, that test focused on punctuation, and so did Direct Instruction. That’s not a dig at Direct Instruction, it is an observation that measurement tools affect outcomes.

Yet the first finding of the evaluation report, the one that is stressed again and again in subsequent analyses, is that sites were more important than models (Anderson et al., 1978). No one way of teaching worked everywhere. Sites and implementation matter. This finding of variability across sites has been an important legacy of the study in educational research, and something that has proven to be basically true in education. There is NO silver bullet pedagogy that can be thrown at teachers and implemented effortlessly. When educational researcher David Berliner (2002) wrote of Project Followthru in an essay called “Educational Research; The Hardest Science of All”, it was to stress the finding of variability across settings. In the essay, Berliner describes how complex it is to study learning across settings, with such difference across populations and settings to make massive generalizations scientifically inadvisable, noting that educational science is far more complex than the “hard” sciences, which do not face the same messiness of humans. He writes,

The decades old Follow-Through study should have taught us about the problems of replication in education (House, Glass, McLean, & Walker, 1978). In that study, over a dozen philosophically different instructional models of early childhood education were implemented in multiple sites over a considerable period of time. Those models were then evaluated for their effects on student achievement. It was found that the variance in student achievement was larger within programs than it was between programs. No program could produce consistency of effects across sites. Each local context was different, requiring differences in programs, personnel, teaching methods, budgets, leadership, and kinds of community support. (Berliner, 2002, p. 19)

The study is understood as a cautionary tale of educational research, vast time and resources applied to compare different models, only to find that variability is the fundamental lesson. Berliner goes on:

These huge context effects cause scientists great trouble in trying to understand school life. It is the reason that qualitative inquiry has become so important in educational research. In this hardest-to-do science, educators often need knowledge of the particular-the local-while in the easier-to-do sciences the aim is for more general knowledge. A science that must always be sure the myriad particulars are well understood is harder to build than a science that can focus on the regularities of nature across contexts. The latter kinds of science will always have a better chance to understand, predict, and control the phenomena they study.” (Berliner, 2002, p. 19)

These calls for increased contextual understanding of quantitative findings is not meant to replace quantitative studies. Berliner is writing in the context of the research policies of the No Child Left Behind era, in which randomized controlled studies were codified in federal law as the only research that could be used to determine effectiveness. Berliner describes this as unscientific, to place your trust in a method, rather than the rigor and appropriateness of that method. Without careful methods, randomized controlled studies are not only untrustworthy, but more concerning, have the veneer of infallibility. This is when we can go seriously astray- when policy makers, politicians and superintendents believe in the method rather than the rigor of the method.

This history contains a warning for the media. The evaluators, in their response to the article by House et al., 1978), describe how the media ignored their primary finding (variability across sites) to focus on the “which one works” secondary finding because it was easier to sell in the paper. They note that Kilpatrick wrote an article “basics are better” about questions “we were careful to point out, we could not answer”(Anderson et al., 1978, p. 1964). I hope that Sold A Story is not repeating these mistakes, oversimplifying one of the most complex and contested educational studies ever to cost $50 million dollars.

If Sold a Story 2 wants to argue that Direct Instruction is the instructional approach we should be using now, perhaps the focus should NOT be a study that is older than I am at 53. There is certainly more recent, more well designed research to make their point (such as Stockard et al., 2018), as well as more recent, well designed research to argue for guided inquiry in mathematics and other less rigid forms of instruction (Alfieri et al., 2011; de Jong et al., 2023). Anyone who argues that one kind of instruction is always more effective is selling a product, not presenting a accurate view of research. As both articles I just cited above make clear, different kinds of pedagogy are effective for different kinds of knowledge, with neither the clear winner. However, overall, the best bet appears to be guided inquiry with explicit instruction as needed (Darling-Hammond et al., 2020).

I may love me some historical analysis of education, but it appears that all of this is smoke and mirrors for a old, old narrative that tells an oversimplified story. It is not just that the study is old, but that it is infamous for its lack of clear methodology. If the argument is that Direct Instruction is effective, why focus on such a dated, problematic study? If your argument is that progressive pedagogy has taken over education, maybe take a look at what is actually happening in schools, which is mostly what teachers would call direct instruction (excellent analysis of this by Michael Pershan here).

If you need access to these sources, please email me at rlambert@ucsb.edu, including a quick note about who you are and why you are interested. I would love to help out education reports, and classroom teachers, and actually anyone who wants to do some digging themselves.

Original report (1977)

• Stebbins, L. B., & Others, A. (1977). Education as Experimentation: A Planned Variation Model. Volume IV-A: An Evaluation of Follow Through. Abt Publications
  • This is the full report from the final evaluation team (Abt). Robert Kaplinsky tracked down the whole version here.

Contemporary commentary/analysis of the original report (1977- 1978)

• Anderson, R. B. (1977). The Effectiveness of Follow Through: Evidence from the National Analysis. Curriculum Inquiry, 7(3), 209–226. https://doi.org/10.2307/1179724
  • This article, written by a member of the Abt Evaluation team, goes point by point into the critiques of the study design, measures etc., and presents the findings of the team that also connects to the critics.
  • This special issue of Curriculum Inquiry is all about the report, and has other articles of note in it.
• House, E., Glass, G., McLean, L., & Walker, D. (1978). No Simple Answer: Critique of the Follow Through Evaluation. Harvard Educational Review, 48(2), 128–160. https://doi.org/10.17763/haer.48.2.j2167r4594027x87
  • This is the most often discussed response to the evaluation, published the next year, by a group of scholars in evaluation who were commissioned to write a report by the Ford Foundation. This is part of a special issue of Harvard Educational Review that is focused on additional analysis of the evaluation.
• Anderson, R., Pierre, R. S., Proper, E., & Stebbins, L. (1978). Pardon Us, But What Was the Question Again?: A Response to the Critique of the Follow Through Evaluation. Harvard Educational Review, 48(2), 161–170. https://doi.org/10.17763/haer.48.2.6160014378p45183
  • This is the response to the the response- what Anderson and colleagues (from Abt) wrote in response to House et al. 1978. Also part of the special issue in Harvard Educational Review.
• Kennedy, M. M. (1978). Findings from the Follow through Planned Variation Study. Educational Researcher, 7(6), 3–11. https://doi.org/10.2307/1175762
  • This is written by an official at the US Department of Education.
• House, E. R. (1979). The Objectivity, Fairness, and Justice of Federal Evaluation Policy as Reflected in the Follow Through Evaluation. Educational Evaluation and Policy Analysis, 1(1), 28–42. https://doi.org/10.3102/01623737001001028
  • This is one of my favorite articles because it gives a lot of tea about the history of the evaluation.

More recent commentary/analysis of the original report (2002- current)

• Berliner, D. C. (2002). Comment: Educational Research: The Hardest Science of All. Educational Researcher, 31(8), 18–20.
  • This commentary from prominent researcher Berliner discusses Project Follow Through as an example of an unsuccessful quantitative study.
• Engelmann, S. (2007). Teaching needy kids in our backward system: 42 years of trying. ADI Press.
  • This is a LONG book that gives lots of insight into the development of Direct Instruction. This goes deeply into the ire of Direct Instruction advocates that the success of Direct Instruction was ignored by the educational elite. Expect published versions of letters back and forth between major players. If you want to learn about Direct Instruction, I don’t think there is a better source than this book. You might like it, you might hate it, but at least you are getting the information direct from the source.
• Woodward, J. (2004). Mathematics education in the United States: Past to present. Journal of Learning Disabilities, 37(1), 16–31.
  • This is a history of mathematics education that includes special education research. Highly recommended for a historical analysis of this intersection, and mentions Project Follow Through.
• Zhao, Y. (2018). What works may hurt: Side effects in education. Teachers College Press.
  • I love the premise of this book from an international education policy expert: we should not just be researching the effects of educational interventions, but the unintentional side effects. The chapter on side effects of Direct Instruction includes analysis of Project Follow Through.

Other citations

Alfieri, L., Brooks, P. J., Aldrich, N. J., & Tenenbaum, H. R. (2011). Does discovery-based instruction enhance learning? Journal of Educational Psychology, 103(1), 1–18. https://doi.org/10.1037/a0021017

de Jong, T., Lazonder, A. W., Chinn, C. A., Fischer, F., Gobert, J., Hmelo-Silver, C. E., Koedinger, K. R., Krajcik, J. S., Kyza, E. A., Linn, M. C., Pedaste, M., Scheiter, K., & Zacharia, Z. C. (2023). Let’s talk evidence – The case for combining inquiry-based and direct instruction. Educational Research Review, 39, 100536. https://doi.org/10.1016/j.edurev.2023.100536

Stockard, J., Wood, T. W., Coughlin, C., & Rasplica Khoury, C. (2018). The Effectiveness of Direct Instruction Curricula: A Meta-Analysis of a Half Century of Research. Review of Educational Research, 88(4), 479–507. https://doi.org/10.3102/0034654317751919