Formative Ed Research, Part 1. The Coleman Rorschach
Ed’s biggest study is ignored by far too many.
The most amazing thing about ‘ed innovation’ is how many times we try the same thing again. Oddly, we tend to get similar results. Too often, we think that despite similar inputs, we’ll get different outputs if we only focus on the process. There are a handful of key reports and studies that everyone in ‘ed innovation’ should study. So, here’s a series of mostly summaries of some key studies and reports. First up, Coleman.
The Coleman report (formally titled “Equality of Educational Opportunity”) is usually cited as the largest education study ever as it studied 645,000 U.S. students.[1] Hanushek writes that it’s “the fountainhead for those committed to evidence- based education policy. Remarkably, this 737-page tome, prepared 50 years ago by seven authors under the leadership of James S. Coleman, still gets a steady 600 Google Scholar citations per year. … breathtakingly innovative, the foundation for decades of ever-improving inquiry into the design and impact of the U.S. education system.”[2]
Coleman essentially invented the idea of measuring outputs against inputs (previously, people just measure inputs … i.e. we spent more money so that must improve outputs, whatever those may be). Coleman further established the idea that measuring broader environmental inputs is necessary for assessing outputs. Coleman’s conclusions were startling:
Stuff that doesn’t matter (no statistically significant correlation): credentials of teachers, curricular variation, school facilities, funding per student.
Stuff that matters: your parents (mostly); your peers (somewhat).
Perhaps more shocking, Coleman has held up under scrutiny (except for the integration angle).
While Coleman collected massive data on individuals, it clustered the data for comparative purposes. One conclusion is that variation among outcomes occurs not between schools but within them. An extension of that (largely confirmed) observation is that variation among teachers also occurs not between schools but within them. (Coleman found minimal variation between schools.)
To this day, there are papers, conferences, etc on Coleman. Two recent papers, one a summary and suggested areas of research by Eric Hanushek (the dean of ed econ) and the other is a similar paper by Alexander and Morgan, are both good places to start learning about Coleman. (I’d suggest, though, that Morgan’s paper is a good example of totally missing the point.[3]) These and similar papers from all sides are worth reviewing as Coleman is essentially the beginning of large-scale data-driven ed research.
Coleman is fascinating because it often acts as a Rorschach test — what one extracts from the report is often a reflection of one’s own interests and values. An example of that is while Coleman has been largely supported by additional evidence and further studies, the policies (and politics) of ed reform have often ignored Coleman. It seems that Coleman supports the idea that culture (at home and school) is remarkably important (i.e. a culture of expectations, of acceptable social behavior, etc). But visit an urban high school and one wonders whether the district was intentionally attempting to eradicate such positive cultural attributes that Coleman found vital to success. (Contra that, many charter schools have aggressive programs to establish and perpetuate cultural norms with the school.) Further, look at the conversations around funding. Coleman found that funding isn’t a primary determining factor in outcomes — tethering funding directly to outcomes is slipping on a correlation banana. That was established in the 1960s and yet there are still never-ending arguments that go in circles the objective of which seems to be to eradicate Coleman’s conclusions about funding.
And witness the perpetual conversations around curriculum reform (not to mention ‘personalized learning’). How many nuevo-curriculum spasms has U.S. ed suffered since the 1960s despite the fact that Coleman suggests that such a focus is misguided.
My one knock on Hanushek is his assertion that others claim that Coleman demonstrates that “money doesn’t matter.” That’s not really what any informed person argues; rather, Coleman shows that there’s no causation between funding and outcomes. (Hanushek concurs on that point … poverty is an indicator, not a cause.) The logical conclusion is that increasing funding doesn’t necessarily have any effect on outcomes. Now, walk into a school board and let them know (you’ll be hammered out of order).
On what makes a good teacher, Hanushek writes “Scholars remain in the dark even today as to exactly why some teachers are effective (that is, why some teachers, year after year, have strong positive impacts on the learning of their pupils) while others are not. In short, it is easier to pick out good teachers once they have begun to teach than it is to train them or figure out exactly the secret sauce of classroom success.” Teacher variation can be massive, but it occurs within schools (not between them) and the secret sauce of effective teaching still remains to be distilled. Imagine informing schools (and teachers’ colleges) of that.
The amazing final question that Coleman leaves us with is: how does one impact outcomes? Coleman provides no easy answers but its massive data leaves one thinking that so much of the ed reform world ignores what we’ve known since the 1960s.
Next we’ll look at a few other studies, then we’ll circle back to Coleman.
About Nathan Allen
Formerly of Xio Research, an A.I. company. Previously a strategy and development leader at IBM. His views do not necessarily reflect anyone’s, including his own. (What.) Nathan’s academic training is in intellectual history; his next book, Weapon of Choice, examines the creation of American identity and modern Western power. Don’t get too excited, Weapon of Choice isn’t about wars but rather more about the seeming ex nihilo development of individual agency … which doesn’t really seem sexy until you consider that individual agency covers everything from voting rights to the cash in your wallet to the reason mass communication even makes sense….
