Should a Teacher's Value-Added Last Forever?
In the middle of a review of a recent Gates Foundation report on teacher value-added scores, economist Jesse Rothstein writes this:
An extensive literature makes clear that students assigned to high-value-added teachers see higher test scores in the year of that assignment, but that this benefit evaporates very quickly.
He cites, among other things, a Jacob/Lefgren/Sims paper arguing that:
But to what other realm of human endeavor would such a standard be applied?
Consider another aspect of human development: physical fitness. Would anyone suggest that the value of vigorous exercise is somehow discredited because a vigorous exercise program has the most effect only in the year that it is actually followed, but its effects mostly disappear after the individual stops exercising? Of course not. Everyone who has ever exercised knows that if you stop for even a few months, you will lose much or most of your conditioning. Does that mean that vigorous exercise wasn't effective? No: it was effective as long as you kept exercising.
The same is true for medicines and vitamins. Would anyone find it important to observe that the effectiveness of medicines and vitamins disappears after you stop taking them for a year or more?
Or take weight loss. Would anyone be surprised to find that a healthy diet enabled people to lose weight only when they actually followed the diet, but didn't have permanent effects that allowed people to quit the diet and eat whatever they wanted therafter?
Shouldn't the same logic apply to teachers? A great teacher can have a powerful effect on your learning of a specific subject. If the effect of good teaching fades out over time, that may simply mean that teaching is like nearly everything else in life: it doesn't magically guarantee permanent improvement.
But it can still be important to figure out what good teachers are doing and who the good teachers are, just as it's important to know which weightlifting coaches produce Olympic champions and which ones are incompetent -- even if the effect of good teaching in either field fades out once an individual moves on and does other things.
Indeed, the fadeout of high value-added teachers may actually mean that it's all the more important to implement well-thought-out policies that encourage such teachers. Why would a 5th grade math teacher's effect fade out in 6th and 7th grade? Yes, it could be because the 5th grade teacher was artificially gaming her scores by teaching topics or problems that are completely idiosyncratic to the 5th grade math test in a particular state (although the blame really would lie with the state standards and tests in such a case). But it could well be because a great 5th grade math teacher was followed by mediocre 6th and 7th grade math teachers who failed to capitalize on the 5th grade teacher's accomplishments, and students' performance -- no surprise! -- wasn't magically and permanently raised.
In other words, the fadeout of high-value added teachers might just mean that we need to be finding the mediocre 6th and 7th grade teachers who are managing to erase their students' prior knowledge and achievement, and either helping those teachers to improve or else gently moving them into other professions for which they are better suited.
UPDATE: I'd apply the same logic to all of the other educational interventions whose effect fades out or disappears over time -- Head Start, class size reduction, rewarding students for reading books (Roland Fryer's experiment). In all these cases, why would anyone ever have expected a one-time treatment to have a permanent effect? Nothing about our bodies or minds works that way.
An extensive literature makes clear that students assigned to high-value-added teachers see higher test scores in the year of that assignment, but that this benefit evaporates very quickly.
He cites, among other things, a Jacob/Lefgren/Sims paper arguing that:
the vast majority of the contemporary test score effect attributed to teacher value-added is transitory. This suggests that the teacher value-added literature overstates the effect of teachers on longrun learning and, therefore, the ability of policies that target teacher value-added to change ultimate student outcomes.For some reason, the short-lived effect of teacher value-added scores is seen as somehow disproving their worth, and the worth of policies that would "target teacher value-added."
But to what other realm of human endeavor would such a standard be applied?
Consider another aspect of human development: physical fitness. Would anyone suggest that the value of vigorous exercise is somehow discredited because a vigorous exercise program has the most effect only in the year that it is actually followed, but its effects mostly disappear after the individual stops exercising? Of course not. Everyone who has ever exercised knows that if you stop for even a few months, you will lose much or most of your conditioning. Does that mean that vigorous exercise wasn't effective? No: it was effective as long as you kept exercising.
The same is true for medicines and vitamins. Would anyone find it important to observe that the effectiveness of medicines and vitamins disappears after you stop taking them for a year or more?
Or take weight loss. Would anyone be surprised to find that a healthy diet enabled people to lose weight only when they actually followed the diet, but didn't have permanent effects that allowed people to quit the diet and eat whatever they wanted therafter?
Shouldn't the same logic apply to teachers? A great teacher can have a powerful effect on your learning of a specific subject. If the effect of good teaching fades out over time, that may simply mean that teaching is like nearly everything else in life: it doesn't magically guarantee permanent improvement.
But it can still be important to figure out what good teachers are doing and who the good teachers are, just as it's important to know which weightlifting coaches produce Olympic champions and which ones are incompetent -- even if the effect of good teaching in either field fades out once an individual moves on and does other things.
Indeed, the fadeout of high value-added teachers may actually mean that it's all the more important to implement well-thought-out policies that encourage such teachers. Why would a 5th grade math teacher's effect fade out in 6th and 7th grade? Yes, it could be because the 5th grade teacher was artificially gaming her scores by teaching topics or problems that are completely idiosyncratic to the 5th grade math test in a particular state (although the blame really would lie with the state standards and tests in such a case). But it could well be because a great 5th grade math teacher was followed by mediocre 6th and 7th grade math teachers who failed to capitalize on the 5th grade teacher's accomplishments, and students' performance -- no surprise! -- wasn't magically and permanently raised.
In other words, the fadeout of high-value added teachers might just mean that we need to be finding the mediocre 6th and 7th grade teachers who are managing to erase their students' prior knowledge and achievement, and either helping those teachers to improve or else gently moving them into other professions for which they are better suited.
UPDATE: I'd apply the same logic to all of the other educational interventions whose effect fades out or disappears over time -- Head Start, class size reduction, rewarding students for reading books (Roland Fryer's experiment). In all these cases, why would anyone ever have expected a one-time treatment to have a permanent effect? Nothing about our bodies or minds works that way.
Labels: education
posted by Stuart Buck at
9:45 AM
5 Comments:
Wouldn't the appropriate comparison be to a personal trainer rather than a course of exercise. Ideally a trainer will teach you to exercise then you can do it on your own and the value added would last. The analogy to teaching would be to have teachers that make you a better learner rather than being effective conduits for knowledge. Truly some teachers are very good at making difficult subjects easier to learn but that does not translate into making students better at learning difficult subjects or learning in general.
A personal trainer would be a more apt analogy. Ideally they would teach you how to exercise then you could exercise on your own and the value added would have a long term effect. If a teacher makes you better at learning then the value added will last. However, if the teacher makes a difficult subject easier to learn it does not make the student better at learning and the value added will be gone once they leave the class.
OK, using the personal trainer analogy: Suppose that one year you have a personal trainer who makes you run sprints, do heavy squats and deadlifts, etc., and you end the year much stronger and faster. Then you're assigned to a different personal trainer who cuts out all the heavy lifting, and who has you doing 5-pound curls while standing on one foot on a Bosu ball. At the end of the year, you're weaker and slower and less fit in every possible way.
Does that prove that it's pointless to try to measure the value of personal trainers, because the "good" personal trainer didn't have a permanent effect?
As a high school teacher, my experience suggests an alternative explanation for "fade out."
In my classes, I have a tremendous variation in preparation, motivation, and raw smarts. Some students "get it" right away and some need lots and lots of time and practice.
So what do I do? Do I teach to the ones who pick things up quickly and leave the rest in the dust? Hardly seems fair. Do I wait for everyone to get it ("learn to mastery") and then go on? If I did, I might get through half the syllabus.
A department head once said, "Teach to the 25th percentile." By which he meant, "Move on, but err on the side of the slow ones."
Imagine a third grade class with a great teacher. The students make great gains in reading and math (the Jacob, Lefgren, and Sims paper is about elementary math and reading.).
The next year, they are randomly assigned to classes. Most of their classmates won't have shot ahead like they did. The teacher teaches to the 25th percentile and the ones who had made so much progress the previous year don't make nearly so much this year. They are then randomly assigned to fifth grade classes and the same thing happens. Ditto sixth. Surprise! Surprise! Nearly all of their gains have "faded out."
In a system that randomly assigns students regardless of previous achievement, "regression to the mean" is pretty much inevitable.
Did the school system that the researchers investigated randomly assign people? The authors say nothing about reading but they do say, "Because this district uses tracking by ability groups for some mathematics instruction, we restrict math scores to untracked classrooms." p. 16
The authors are very concerned that some of what seems to be persistence in the value-added effect of a previous teacher may actually be caused by students who have done well being non-randomly assigned in later years to good teachers. But if good students are non-randomly assigned to good teachers, then they are in a class with better students, which means they can do more and do it faster--no matter who the teacher is. What the researchers may be seeing is persistence because the students have been put in a situation where persistence is more likely.
The authors do sort of realize that random assignment will slow down the people who have previously moved ahead at a greater than average rate: "Alternatively, low persistence in teacher value added effects could be due to some sort of compensatory teaching, whereby later teachers change their curriculum to address the students who learned less in previous years because they had low value added teachers." p. 24
In a system of random assignment, this is not just possible but inevitable. No teacher wants to see lots of students fail.
The authors state near the end, "Previous researchers have referenced a counterfactual world where a series of high value added effects for a hypothetical student with a string of good teachers may simply be added together." p. 25
This might be possible if the hypothetical student was, in the next year, put with other students who had achieved to her level, and then challenged to continue her improvement.
It is impossible when students are assigned on the basis of nothing but age.
One-time intellectual treatments do have persistent effects, unlike physical treatments.
I will get fat again if I stop my physical training for a year. But my Ph.D. and your J.D. does not become worthless a year after we leave school. What we truly learn (as opposed, say, to calculating tricks that help on tests) stays with us.
Post a Comment
Subscribe to Post Comments [Atom]
<< Home