Charter Schools and Averages

We often hear that something has no overall effect on something else -- education, health, whatever -- but it seems to me that in a world with widely varying individuals and situations, the overall mean effect isn't very interesting or useful. At least not as often as some people seem to think.

For example, we might hear that a certain kind of medicine has no overall survival benefit, which is the reason that the FDA moved to block to approval of Avastin to treat metastatic breast cancer. But even if there is little overall mean effect, Avastin could still cause a remission in a few people:

Christi Turnage of Madison, Miss., said her cancer has been undetectable for more than two years since starting therapy with Avastin. She was diagnosed with breast cancer in June 2006 and began taking the drug in 2008 after the tumors spread, or metastized, to her lungs. Breast cancer that spreads to other parts of the body is generally considered incurable.

If it turns out that you're one of the few people for whom the medicine works, it's no comfort to be told that you can't take it because not enough other people would benefit if they took the medicine.

The same is true for charter schools. We've heard about several studies indicating that charter schools don't have a higher average effect than regular public schools. Take, for example, this Mathematica study of charter schools in 15 different states. It found no overall impact on the students. But this masks an important variation in who benefited and who didn't:

We found that study charter schools serving more low income or low achieving students had statistically significant positive effects on math test scores, while charter schools serving more advantaged students—those with higher income and prior achievement—had significant negative effects on math test scores.

In other words, charter schools were doing very different things for different students — raising up low income and poor-scoring students while actually harming richer and higher-scoring students' test scores.

Average them all together, and you find no effect. But if you want to expand charter schools in impoverished urban areas, the “no overall average benefit” finding would be completely beside the point.

Consider as well this very recent study of charter schools in Milwaukee. The author (Hiren Nisar) says that "charter schools on average have no significant effect on student achievement." An opponent of charter schools (say, a Diane Ravitch) would cite that finding as if it represented the entirety of the study.

But Nisar goes on to find that the overall average is hiding a critically important distinction:

Charter schools with higher level of autonomy from the district in terms of financial budget, academic program, and hiring decisions, are effective. I show that students in these charter schools would read at a grade level higher than similar students who attend a traditional public school in three years. Irrespective of the type and the age of the charter school, race of the student, or grade level, attending a charter school has a positive effect on low achieving students. I show that these effects on low achieving students are substantial and are more than enough to eliminate the achievement gap in two years.

Once again, the overall average is completely meaningless if you are interested in expanding the very charter schools that are most likely to work, i.e., the ones that serve low-achieving students and that have more autonomy from their competition (the school district).