Value-Added Has Errors . . . Which Means What?

A recent New York Times article profiled a dedicated and well-reviewed middle school teacher who may not get tenure thanks to the New York City value-added model that places her at the 7th percentile (judging by how much growth her students showed on tests).

Assuming that the article is correct, and that the value-added model here is being unfairly applied for any number of reasons, that means that value-added models can make Type II errors (false negatives).

OK, so we should ditch value-added models and keep on with the "last in, first out" system in which the newest teachers get laid off first (in the case of layoffs). No messing around with incomprehensible math that could be wrong.

But wait, the "last in, first out" model can make Type II errors too. Look at what happened last year in Wisconsin, where the zero-deductible health insurance plan was so expensive (note: this is one reason why Governor Walker wanted to strip collective bargaining over such benefits) that it required Milwaukee to lay off some new teachers acknowledged as good:

Megan Sampson was named outstanding first-year teacher by the Wisconsin Council of Teachers of English last week.

Second-year social studies teacher Kevin Condon, also at Bradley Tech High School, has four licenses and can command the attention of 40 students in an open-concept classroom.

Both are among 482 educators - more than 12% of the full-time teachers in the district - who have received layoff notices from Milwaukee Public Schools.

On Monday - the last day of the year for schools in MPS and the first day teachers reunited after hearing the news of the layoffs - some teachers expressed frustration at losing their jobs because of experience, not performance. . . .

Meanwhile, the Milwaukee School Board president and the Milwaukee Teachers' Education Association leadership continued to disagree on how to handle the tightened $1.3 billion district budget, and whether teachers should accept a lower-cost health-care plan to avoid layoffs. . . .

Bonds said if all teachers switched to the lower-cost plan, about $48 million could be saved, enough to pay for 480 educators.

"I'm not aware of any place in the nation that pays 100% of teachers' health-care benefits and doesn't require a contribution from those who choose to take a more expensive plan," Bonds said.
. . .

Sampson and her laid-off colleagues, all who have less than three years of teaching experience, also expressed frustration that their jobs would be filled by more veteran, but not necessarily better, educators.

Sampson and Emily Kaphaem, a world geography and citizenship teacher at Tech, said they have received exemplary performance reviews.

"I feel kind of let down by my city today," said Kaphaem, 25, as she lost the fight to hold back tears in Principal Ed Kupka's office.

Kupka is equally frustrated. He hand-selected the new teachers because of their talent and enthusiasm for turning around Tech, recently designated as one of the worst-performing high schools in the state.

"Based on the pressures we're under as a low-performing school, I absolutely would have chosen a different nine (for layoffs)," Kupka said.

So now what? Well, for one thing, it's silly for ideological advocates on either side to act as if they have a perfect system that will preserve the best teachers, either by 1) improperly equating quality with years of experience or 2) improperly assuming that each teacher can be given an exact number that represents her quality. It's equally silly for ideological advocates to act as if they've proven the opposite side's system to be some horrible inequity just because it results in an occasional good teacher getting fired. Unless God is running the system, there are going to be errors.

What we need to know is which system makes more errors. But what constitutes an "error"? One way to think about this would be to assume, for the sake of argument, that teachers really are good teachers when they have received outstanding (not merely adequate) performance reviews from their principals. So then, if layoffs are going to occur, would more teachers deemed outstanding by their principals be laid off under 1) a value-added model, or 2) the "last in, first out" standard?

Charter Schools and Averages

We often hear that something has no overall effect on something else -- education, health, whatever -- but it seems to me that in a world with widely varying individuals and situations, the overall mean effect isn't very interesting or useful. At least not as often as some people seem to think.

For example, we might hear that a certain kind of medicine has no overall survival benefit, which is the reason that the FDA moved to block to approval of Avastin to treat metastatic breast cancer. But even if there is little overall mean effect, Avastin could still cause a remission in a few people:

Christi Turnage of Madison, Miss., said her cancer has been undetectable for more than two years since starting therapy with Avastin. She was diagnosed with breast cancer in June 2006 and began taking the drug in 2008 after the tumors spread, or metastized, to her lungs. Breast cancer that spreads to other parts of the body is generally considered incurable.

If it turns out that you're one of the few people for whom the medicine works, it's no comfort to be told that you can't take it because not enough other people would benefit if they took the medicine.

The same is true for charter schools. We've heard about several studies indicating that charter schools don't have a higher average effect than regular public schools. Take, for example, this Mathematica study of charter schools in 15 different states. It found no overall impact on the students. But this masks an important variation in who benefited and who didn't:

We found that study charter schools serving more low income or low achieving students had statistically significant positive effects on math test scores, while charter schools serving more advantaged students—those with higher income and prior achievement—had significant negative effects on math test scores.

In other words, charter schools were doing very different things for different students — raising up low income and poor-scoring students while actually harming richer and higher-scoring students' test scores.

Average them all together, and you find no effect. But if you want to expand charter schools in impoverished urban areas, the “no overall average benefit” finding would be completely beside the point.

Consider as well this very recent study of charter schools in Milwaukee. The author (Hiren Nisar) says that "charter schools on average have no significant effect on student achievement." An opponent of charter schools (say, a Diane Ravitch) would cite that finding as if it represented the entirety of the study.

But Nisar goes on to find that the overall average is hiding a critically important distinction:

Charter schools with higher level of autonomy from the district in terms of financial budget, academic program, and hiring decisions, are effective. I show that students in these charter schools would read at a grade level higher than similar students who attend a traditional public school in three years. Irrespective of the type and the age of the charter school, race of the student, or grade level, attending a charter school has a positive effect on low achieving students. I show that these effects on low achieving students are substantial and are more than enough to eliminate the achievement gap in two years.

Once again, the overall average is completely meaningless if you are interested in expanding the very charter schools that are most likely to work, i.e., the ones that serve low-achieving students and that have more autonomy from their competition (the school district).