The 22 Percent Solution
December 13, 2006 10:15 AM
I didn't make it to the panel at the Center for American Progress (CAP) on teacher pay reforms earlier this week, but I heard the NPR story this morning on the new paper by Dan Goldhaber that was released at the CAP event. The paper's main conclusions are that 1) we don't really know how to identify effective teaching; and 2) we need more experimentation and research around the use of "student achievement tests as a gauge of teacher effectiveness." (On the second point, we should learn something from the Teacher Incentive Fund program grants that were recently distributed to several urban districts across the country. The grant programs that were developed with the input of teachers and their unions look the most promising, such as those in Ohio.)
One of the things that tends to get overlooked in discussions about using student test scores to measure teacher effectiveness--usually through a value added model--is that data is only available for those who teach in tested subjects and grades. My colleague Howarad Nelson has estimated that such value added teacher evaluation systems would only apply to about 22 percent of teachers in most states. Nelson was at the CAP event and tried to press Goldhaber on this point, but didn't get much of an answer.
So what to do about the other 78 percent of teachers? Gordon, Kane and Staiger propose using peer, principal and parent evaluation to judge the effectiveness of teachers. While there is certainly evidence of successful peer assistance and review programs, principals don't seem to have a very good record of judging teacher performance (isn't that one of the primary complaints from various quarters--that too few "bad" teachers are dismissed), and anyone who has ever taught school knows that parent evaluations are fraught with problems.
Given these limitations, it's not surprising that the signature study by RAND on the use of value added models to evaluate teachers found they weren't ready for prime time.
Comments
On a tangent for technocratic solutions, Massachusett's Board of Ed, just before Governor Romney's hand picked successor was trounced in the race for Governor, voted to eliminate a six month "fact finding" period that follows a negative AYP determination, and go directly to the next restructuring step.
They also voted to streamline restructuring, in what looks to me essentially like a recommendation for schools labeled as failures to hire an outside manager or face state takeover.
I thought it might be worth pointing out that it takes the state five months to return test results to parents, and that is occuring well into the next school year for kids who missed the opportunity for tutoring or parental intervention up to a year earlier, missed the opportunity for summer school, and missed the opportunity to re-evaluate the best school placement for their child in August.
The whole premise of state-supervised annual comprehensive testing is limited in its ability to help any individual child.
If Mass wants to expedite making use of standardized test results, there's four months of room for technocrats to get test results back earlier. I'd suggest a goal that multiple choice tests should be returned within a week, and open response tests returned within a month, of test taking.
But a better use of technology might be to deduce that we've gotten the low hanging fruit from state supervised comprehensive testing, and see what can be done by applying technology to formative testing supervised locally.
That is, instead of teaching to the test and waiting until next year to find out what we already knew four years ago - which schools were labelled as "failing" - focus for a while on formative testing and get appropriate feedback to parents, teachers, and local administrators every week.
Posted by: MassParent | December 13, 2006 01:27 PM
Trying to judge a teacher's worth through student test scores is like trying to judge a track coach's worth by looking at the times of track team members.
For example, suppose a cross-country runner reduces his time by 2 minutes. There may be many reasons for that success, including that the runner:
a) had great coaching
b) had a parent who pushed him to practice more
c) went on a new diet
d) grew to become stronger and faster naturally
e) read an inspirational book about running
f) tried extra hard in a particular race because he wanted to impress his girlfriend
In addition to a-f above, there are likely countless other variables that we can't even possibly imagine.
So, how do you isolate just variable A, so you can attribute proper credit/blame to the coach? And, assuming we can do that, does that mean we should develop a system whereby we hire, fire or otherwise reward/punish coaches based on the times of track team members?
Posted by: Dan at AFT | December 13, 2006 02:25 PM