Education Reform Is Slow

Kevin Drum notes a growing backlash against education reform, citing Diane Ravitch, Emily Yoffe and this Newsweek (which is really this private foundation report in disguise) as examples. The last of these, about the failed attempts of several billionaires to improve education through foundation grants, is really kind of maddening. It makes the billionaires in question (Bill Gates, Michael Dell, Eli Broad, and the Wal-Mart Waltons) sound like feckless idiots, but I can't tell if that's just bad writing.

The core of the piece is the finding that the districts these guys put money into haven't made the dramatic improvements they hoped for:

In a first-of-its-kind computer analysis, iWatch News analyzed the graduation rates and test scores in 10 major urban districts -- from New York City to Oakland -- which collectively took in almost one-fourth of the total money poured in by these top four education philanthropists.

The results, though mixed, provide dispiriting proof that the billionaires have not found a one-size-fits-all solution to education reform and that money alone can't repair the desperate state of urban education.

For all the millions spent on reforms, nine of the 10 school districts studied substantially trailed their state's proficiency and graduation rates -- often by 10 points or more, the analysis found. And while the urban districts made some gains, they managed only 60 percent of the time to improve at a rate faster than their states. Those spikes weren't enough to materially reduce persistent gulfs between poor, inner-city schools, where the big givers focused, and their suburban and rural counterparts.

That sounds damning, but not necessarily of the billionaires' efforts. The way a lot of this is phrased makes it sound like they did really superficial and inadequate comparisons, and the "methodology" notes in the sidebar they give don't have enough detail to tell whether they really did the comparison stupidly, or just wrote about it badly. Of course, they also make a point of noting that the metrics they used are the same used by some of the wealthy foundations involved, which may make the exact nature of the failure kind of a moot point.

The core problem, as with most educational policy issues, is a failure to control for (or report that they controlled for) external socioeconomic factors. Comparing schools in poor inner-city districts directly to schools in wealthy suburban districts is foolish-- the two groups of students live in vastly different environments. Even if you dump money into the inner-city schools, you're not likely to change the home lives of those students to make them more like the suburban experience. If you want to see what effect money has, you need to compare urban districts receiving funding to urban districts with similar demographics (preferably in the same region of the country) who aren't receiving funding.

Moreover, you need to track this over a long period of time-- the five-ish years they talk about is the absolute minimum, particularly if you're talking about something like graduation rates. It takes four or five years just to get rid of students who came up through the pre-reform system, plus it almost certainly takes 2-3 years to get the kinks out of a new curriculum. Anybody who has taught can tell you that-- we've changed the intro course twice since I've been here, and both times it's taken me at least two passes through the new curriculum to get a good (I think) feel for how to best use the new material. I'm still getting some bugs out of the latest version, and next fall will be my fourth time teaching that class with the new book.

If you want to know whether an educational reform is really making a difference, you need to look at at least 6-8 years of data, because it takes time for reforms to propagate through the system, and because there are large year-to-year variations in the students, depending on the specific mix of backgrounds and local economic conditions represented in a particular class year. Any shorter than that, and you risk making decisions based only on statistical noise.

And that's just if you're changing high schools and looking at graduation rates. If you want to change elementary education (which a lot of things suggest is the place where the biggest difference can be made), you need ten years just to start seeing kids from the new system graduate.

Going into education reform expecting instant change is a fool's game. And, unfortunately, the article makes it sound like the billionaires they track are fools-- there's lots of talk about the Gates Foundation, for example, scrapping a big program and starting over with something new (possibly scrapping two different programs and starting over; it's a little difficult to tell). There are very few firm dates given, but the Gates thing is specifically cited as being dumped after eight years, which is just barely enough, and other programs sound even worse-- Broad is cited as trying at least three different reform approaches in the 12 years he's been in the foundation game, which is just crazy.

Overall, the article paint a pretty disheartening picture. While the idea of philanthropy as a source for funding to try new techniques sounds really promising, on the whole, the people doing the funding come off sounding pretty bad. They're portrayed as basically coming in full of ideas they were sure would work, on the basis of no particular evidence, then abandoning them before there's time to properly test their effect. Some of those abandonments, such as the "CEO-style principal" model, I'm not sorry to see go, but it seems like this money could've been used a lot more effectively.

Really, I think the core lesson that needs to be taken from this is more or less the same as what Timothy Burke wrote about international development in the wake of the Greg Mortenson thing:

If I gave you an unlimited line of credit and carte blanche to run everything your way, do you think you could make a single secondary school work? I mean, really work so it was beyond reproach, was by almost any measure superior in outcomes and character and ethos to any alternative? Now what if I took away from you the choice of where your school was located and restricted you to pupils who lived within 30 miles of your school? Now what if I required you to obey all relevant national and local laws addressing education? Still confident? Now what if I made you operate within a budgetary limit that was generous by local and national standards but not unlimited? Getting harder yet? Now what if I put your school in a location with very little infrastructure and serious structural poverty?

The point here is that when one crucial task like that is hard enough, we should be deliriously happy to see a person dedicate their life and money and effort to make that task work. One. When we keep our checkbooks closed and our frowny-faces on because that's not enough, not nearly enough, we create a situation where development messianism is inevitable. We invite not mission creep but mission gallop: make a hundred schools! change gender ideology! eliminate poverty! Under the circumstances, looking back, you have to ask how that was ever creditable, why anyone cheered and hoped and wrote checks.

The processes involved in reforming education are complex and ponderously slow. They don't work like dotcom start-ups, where the time from initial idea to first implementation to obscene overvaluation to inevitable crash is about a year and a half. You're not going to swoop in and fix everything overnight, no matter how much money you lucked into on the stock market.

If you're going to go into the reform business, like the international development business Burke writes about, it needs to involve a long-term commitment, and a willingness to let things run their course so you can do proper evaluation of the changes (meaning things like fair statistical and demographic comparisons, as well as just letting a few classes graduate before you declare victory or defeat). Anything else is ultimately a waste of time and money.