Testing, Testing--One, Two, Three
October 11, 2007 09:11 AM

I agree with one aspect of what Kevin Carey at The Quick and the Ed said yesterday about Linda Perlstein's new book, Tested--that education journalists often overreach, taking what they have observed in one classroom or one school and then making sweeping generalizations. However, I take great exception to his second post on the book. Carey writes:
Teaching students material that's way above their heads is bad educational practice. And you can't say the incentives built into NCLB leave educators with no other option, because--as Perlstein makes clear--it's also futile. Even under the current, no-growth-model law, schools teaching students like Whitney have two choices: inappropriately teach at grade level, in which case Whitney fails the test this year and every year after that, or teach at the right level, in which case Whitney fails the test this year but catches up and passes tests in the future.
Wait--so Carey thinks it is educationally inappropriate to teach students above grade level but it's just dandy to test them every year knowing they will fail, because some day they will catch up to grade level and pass such tests? Maybe I am functioning below grade level, because I fail to see Carey's logic.
And, as much as some would like to continue to deny it, it sounds as though Perlstein's book is part of the mounting evidence that NCLB has led schools to focus more instructional time on testing and, in some places, caused a narrowing of the curriculum. Anyone who talks to teachers on a regular basis hears it too often from too many quarters to dismiss those who protest as nee-sayers, uh, I mean, naysayers. Can you hear me now? Good.
Update: Carey responds here. I think the nee-sayers would argue that using an out-of-level assessment is the best way to measure what students who are not performing at grade level know and are able to do, whereas using a grade level assessment would not provide useful information.



Comments
But the question remains why didn't Whitney learn how to read by first grade in the first place? Why did she ever fall behind?
If Whitney's teachers had used one of the AFT's recommended reading programs she should have been able to read on time. Teh creator of the program has repeatedly claimed that "[We have] consistently demonstrated that if a reading sequence is properly implemented in kindergarten, virtually all at-risk students with the exception of the profoundly retarded and the very frequently absent will read by the end of the year."
Whitney is neither profoundly retareded nor habitually absent, so why should we hold her to a lesser standard than NCLB's current standard?
I've yet to hear a compelling reason.
Posted by: KDeRosa | October 11, 2007 02:49 PM
The real problem in this discussion is the misrepresentation of what a growth model will measure or do. George Miller commented at one of the hearings months ago that many people believe growth models will be a "silver bullet" to fix the multiple measurement problems associated with NCLB (http://edworkforce.house.gov/hearings/fc032107.shtml). Miller, himself, said that growth models were not a silver bullet. When he asked a panel of experts from several states and even some statisticians if growth models were a silver bullet, they all said, "No."
If you want proof, look to the states that have already implemented growth models. You might ask: How many schools or districts had their AYP status changed when the models were run? Or, even more to the point, how many inappropriate classifications were corrected by the revised calculations? After all, if a growth model offers a clearer picture of student achievement, shouldn't it improve or correct the previous picture? If the classifications were corrected, how did people know the growth model answer was more accurate and not just different?
Back when NCLB started, many people with no understanding of testing drank the sweet cherry Kool-Aid: Proficiency testing at every grade would present an accurate, precise measure of whether students were learning what they should learn. We now have five years of evidence to know that it is not so simple, from cheating in Texas to grade-inflated scoring of open-ended responses in Florida to games played with cell sizes, student assignments, and cut scores. Those mechanical problems don't even get to the larger questions of curriculum alignment, limited content scope of a single test, and other questions of validity. We know without a doubt that the NCLB system is simply too weak to support the kinds of weighty conclusions resting on it. Now, we are being sold growth models as the fix for everything wrong with NCLB. Some of the same people who sold us the first cup are saying, "We know the Cherry Kool-Aid made you sick to your stomach, but you'll love the new Tropical Punch." Well, many of us who had a cup the first time simply aren't interested in more.
There is only one thing that clear-minded, thoughtful educators can do. Whenever anyone says what a growth model will do or explains how a growth model will fix a certain problem with the current accountability system, ask them to be as precise as possible. Tell us exactly how, using a real, approved growth model, under what circumstances a school would get credit for that kind of change in performance and how that change would be reflected in a real accountability measure: Student report, AYP measure, whatever -- some REAL measure. Ask if the model they are describing is one that is already in use. If it is, ask for the data on how it changed the measures they cited. Real examples, not hypotheticals.
For example, is there any growth model currently approved in which a school can get credit for taking a student from, say, 2 points below 4th grade Proficient to 2 points below 5th grade proficient? Assuming the tests are stable, that student has made a year's worth of growth, but is no closer to proficient. Which model gives credit for that and how is that credit reflected in a student or school score? Which model gives a school credit for a student who moves from just barely proficient to the high end of proficient or better? How does the model handle the fact that, as several recent reports have indicated, proficiency cuts are often not even internally aligned from grade to grade, so that talking about whether a fifth grader will be proficient in two or three years is statistically like hitting a hummingbird with a water balloon?
If you want to see if you are getting a complete and honest answer, ask the person which problems in the NCLB accountability system ? the measurement, analysis, and reporting of data requirements -- that a growth model will not fix. If they cannot name a specific weakness that a growth model will not fix, they are either too ignorant or too ?invested? to be speaking on the subject. The researchers at Miller?s March 21st hearing weren?t afraid to answer.
We will very soon have data from many states that tell us exactly what changes if any a growth model will make in improving the truthfulness, or truthiness, of accountability classifications. Shouldn?t we be able to take a look at that data before we decide what those models will and won?t do in terms of fixing NCLB?s fundamental flaws? Wasn?t that the point of having pilots? To answer that question before we embark on a whole new, potentially equally flawed system? Many people have gut feeling based on a rumor that a growth model will fix this broken system. Whatever happened to using data to make decisions?
On the other hand, if NCLB?s accountability system is so seriously flawed that we cannot afford to wait to see the effects of growth models in practice, maybe what we really need is a one or two year testing moratorium. We could take the money saved and see if we can make sense of all of the data we have already collected, using the raw data to try to understand and improve education, not just punish or reward schools. What if we could run five different growth models on 50 states? worth of data without any state having to be afraid of the political consequences? But I digress?
Posted by: Current no-growth-model Law? | October 13, 2007 06:35 AM