If You're Going to Glorify Education Statistics, You Have to Get the Numbers Right: The Bill Gates Edition

He might be really good at designing operating systems (or not), but Bill Gates has a slight data problem. In an op-ed arguing that class size is unimportant and that teacher evaluation is crucial--and should be combined with merit pay, Gates makes this blunder (italics mine):

Perhaps the most expensive assumption embedded in school budgets - and one of the most unchallenged - is the view that reducing class size is the best way to improve student achievement. This belief has driven school budget increases for more than 50 years. U.S. schools have almost twice as many teachers per student as they did in 1960, yet achievement is roughly the same.

What should policymakers do? One approach is to get more students in front of top teachers by identifying the top 25 percent of teachers and asking them to take on four or five more students. Part of the savings could then be used to give the top teachers a raise. (In a 2008 survey funded by the Gates Foundation, 83 percent of teachers said they would be happy to teach more students for more pay.) The rest of the savings could go toward improving teacher support and evaluation systems, to help more teachers become great.

Let's leave aside that adding a few students, in some districts, could raise class sizes to ridiculous, Detroit-like levels. The howler is Gates' statement that "achievement is roughly the same--it's not:

Regarding long-term trends, according to the NAEP, African-American students have increased reading test scores by the rough equivalent of three grade levels during the period of 1971-2008...

White students haven't stagnated either, with a grade level increase. Yes, there are states and localities that are failing (and, oddly, when it comes to educational reform, we only dwell on cities, not underperforming states. I'm sure that has nothing to do with the majority-white racial composition of states versus a large non-white urban population. Nothing at all). But there are states which do very well, and, nationally, we have made impressive gains.

When someone is arguing that we need to be data-driven, yet he doesn't understand the data, it's foolish to take him seriously. We should always strive to do better, but if we don't define the problem (or problems) accurately, we won't fix (or mitigate) them.

Gates is obviously very intelligent: a lack of comprehension isn't what's going on here. This is willful ignorance that rivals that of creationism. Would you think twice before aligning yourself with a creationist--on any issue? I would. You should too.

More like this

I write about education and educational data a lot, and I'm always struck by the insistence that the U.S. K-12 educational system is DOOMED! This is a staggering display of willful ignorance that rivals creationism (and, arguably, is more pernicious). Without going through the entire backstory (…
Given the fundamental problems that New York City's 'proficiency growth' evaluations of teachers have, it's absolutely unclear why Massachusetts, which leads the nation according to the gold-standard NAEP, would want to adopt them (we'll return to this point later). Yet the contagion of stupidity…
The Times last weekend had a big article on the "achievement gap" in education, where poor and minority students are found to lag behind upper- and middle-class white students in many subjects. The author looks at a number of innovative shools that are producing good results with students from the…
One of the supposed key innovations in educational 'reform' is the adoption of value added testing. Basically, students are tested at the start of the school year (or at the end of the previous year) and then at the end of the year. The improvement in scores is supposed to reflect the effect of…

But even if you do get the numbers right, isn't it important not to cherry pick the data? Public school education is a ~13 year process (ideally), and If you are going to expound on the state of education as a whole, it makes more sense to take the data from the end of that process instead of cherry picking from intermediate grade levels. Gates made it clear he was addressing k-12 for the country as a whole, not primary or middle school.
From the NAEP long term trend section:

"At age 17, the average score in mathematics in 2008 was not significantly different from the scores in 2004 and 1973."

"At age 17, the average reading score in 2008 was higher than in 2004 but not significantly different from that in 1971."

hibob: "If you are going to expound on the state of education as a whole, it makes more sense to take the data from the end of that process instead of cherry picking from intermediate grade levels."

It is hardly cherry picking. The front lines of education are the middle schools. That is where students are, by and large, divided into winners and losers. The losers tend to drop out of high school or not attend. They are not around to be tested with the 17 year old students. So you get a selection effect with the 17 year olds that you do not get with the younger students.

@Min: I don't disagree that results at the middle school level are predictive of improved results at the high school level; I'd also be the first to admit that schools (and students!) face a lot more challenges than they used to. Be that as it may: the goal of the system isn't to produce middle school graduates, the goal of the system is to produce high school graduates. If you want to know how the system as a whole is doing, you don't look at a part from the middle and extrapolate results from there, you take your data from the end results. The high school graduation rate over the past 40 years has remained flat (77%). The rate has improved since 2000, and overall it has gone up since 1960 - so a partial point against Bill.