Houston ISD Imploding
Junk Data Science Partially to Blame
Houston Independent School District, one of the largest in the country, is on the brink of being taken over by the state. The school board will be dissolved and the superintendent terminated.
This is major news in the sense that major districts with major budgets usually find out ways to solve their problems. Houston’s annual budget is about $2 billion. You can hire a lot of consultants and change-agents for that. Probably could have called up the Gates Foundation and bought some time (in the end, they would have made things worse). But after over a year of warnings, things at Houston ISD have gone from weird to bad.
Yes, I have done some work with Houston ISD (two superintendents ago), so my interest is probably higher than usual. And no, this problem isn’t my fault.
The bigger issue is the junk science that partially justifies commandeering the district.
According to the Texas Education Agency (TEA), there are two primary issues: (1) the school board’s “failure of governance” and (2) the repeated low academic performance of Wheatley High School, which received its seventh failing rating this year.[1]
The “failure of governance” charge may be accurate; I don’t know but there’s some substantial evidence to corroborate it.[2] The board terminated then unterminated the superintendent in less time than it takes to boil an egg. Then one board member was filmed physically threatening another (at a leadership workshop, no less). And in between bouts of seeming incompetence, the board accomplished little else.
But Wheatley’s seven years of failure strikes me as problematic, and judging success or failure based on longitudinal test scores from dynamic populations is pure junk science.
Wheatley’s population, in general, is not static over seven years. Wheatley is largely a minority school in a historically black ward (and fyi, I’ve actually been to the school and worked with their teachers). The only way you can longitudinally compare, for example, dynamic subpopulations of blacks is to assume that “blacks” compose some static group. Which is to say: you must implicitly assume that all blacks of roughly the same age and geography are the same. Of course, you’re also assuming that similarly situated Hispanics are the same, and Asians, etc. The TEA has no data — none — that permits such comparisons. (What? They’re all poor? So all poor people are the same? You can mutate the racism and bigotry, but it’s still racism and bigotry.)
The only way to determine success or failure from such metrics is to assume your inputs are static (and thus can isolate process variation). And the data — those inputs — are humans in groups, which is a complex system in a complex system.
The second problem is this belief that such assessed “improvement” or lack thereof is meaningful. Improving a student from, for example, the 20th percentile in reading to the 25th percentile — which most educators would tout as significant — actually has no real-world impact on the student. None. So we spend all this money and cause all this angst over an artificial metric that is meaningless in the real world. We’re terrorizing these kids with meaningless data borne from junk science.
I’m not against trying to determine whether educators are effective (whatever that means), but your metrics are just an illusion of knowledge and control.
See also:
Why you can’t compare data of dynamic subpopulations. Interpretive Frameworks, Fragile Empiricisms, and Data
Why your ed-data obsession is psychological warfare on kids. Gatesian Utopia
No, you can’t predict much about complex systems in complex systems. A.I., Education Con Jobs, and Cats
About Nathan Allen
Founder of Xio Research (A.I.), Applied Magic (A.I.), and Andover (data). A.I. strategy and development leader at IBM. Academic training is in intellectual history; his most recent book, Weapon of Choice, examines the creation of American identity and modern Western power. Don’t get too excited, Weapon of Choice isn’t about wars but rather more about the seeming ex nihilo development of individual agency … which doesn’t really seem sexy until you consider that individual agency covers everything from voting rights to the cash in your wallet to the reason mass communication even makes sense…. Lectures on historical aspects of media, privacy/law, and power structures (mostly). Previous book: Arsonist.
