A Case Study in Bias

Two studies came out comparing the performance of schools or teachers. In the first case, Raj Chetty, John Friedman, and Jonah Rockoff came up with just about the most extensive and sophisticated study of teachers' value-added that I've ever seen. As highlighted in the New York Times, the study includes estimates for how much high-quality teachers improve their students' income years later, and also (see pp. 29 ff.) includes a new way to check for bias by looking at how cohorts of students change performance when a high or low value-added teacher arrives from somewhere else. Very cool.

But such a study, implying that some teachers are better than others, and that teacher quality can be revealed by how well their students do on tests (conditioning on prior achievement and student demographics), is disturbing to some people. Diane Ravitch tweeted at least 67 times the day the study came out, trying to undermine the study by questioning its lack of peer review (so far), the way in which it was conducted, and the very project of looking at test scores in the first place.

 In the second case, there's a group called Educate Now in Louisiana that released a PDF chart (available here) that merely lists the schools in New Orleans identified by whether they are Recovery School District schools or voucher-accepting private schools, and then listing what percentage of students score above basic on English and Math in grades 3-5. That's all. No attempt to control for the individual students' prior achievement, no attempt to control for any student demographic variables such as poverty, no attempt to control for the fact that students are eligible for vouchers only if they had been attending a failing public school, no statistical analysis whatsoever.

This is as primitive as it gets, and is a horrible way to judge the merit of voucher schools (as I explained here).

 Did Diane Ravitch tweet 67 times criticizing this purported attempt to compare voucher schools to public schools? No: right in the midst of her incessant criticism of an immeasurably superior study, she sent out one tweet that said, "How did voucher schools in New Orleans do?" followed by a link.

Ravitch here displays the worst sort of intellectual bias: when what looks like one of the best studies out there doesn't fit her ideology, she acts as if it is far more questionable than the baloney that she otherwise is happy to plug. To be sure, it's OK to ask questions about the new value-added study, what it means, how it was done, and whether it was oversold in the media. But it's not OK to pass along a worthless analysis of the merits of vouchers.

Anti-reformers need to think a bit more carefully about whether they want someone as their standard-bearer who doesn't know the difference between good and bad research (or, worse, who doesn't care).