Education Secretary Margaret Spellings announced a differentiated school accountability pilot program yesterday.  But Ed Week's David Hoff threw a flag, noting that the announcement was delivered in Minnesota, a state that is ineligible for the pilot program.

(Feel like an instant replay?  On Secretary Spellings' 2005 trip to Florida, she praised Robles Elementary School for, as St. Petersburg Times reporter Ron Matus put it, "ris[ing] above poverty and meet[ing] No Child's definition of success."  Matus threw the flag back then, observing that Robles hadn't made No Child Left Behind's adequate yearly progress requirement.  Oops.)

AFT's Executive VP Toni Cortese threw another flag on yesterday's announcement, saying it was a false start:

This pilot program would not even begin to address the major problems with the No Child Left Behind Act (NCLB).

The proposal does nothing to fix NCLB’s adequate yearly progress formula, a poor measure of school quality. Nor does it change NCLB’s wrong-headed approach—providing punishment instead of help—to schools and students that are struggling.

NCLB is in need of a dramatic overhaul and cannot be patched up with Band-Aids and pilot programs.

But wait, there's more.  Eduflack also called a penalty, saying the Department didn't spin the announcement the right way (Eight Dollar Words).

You know it's a bad game when even your cheerleaders get flagged.

(Photo by Flickr user Michael (mx5tx) used under a Creative Commons license.)

Posted by John at 03:52 PM | Permalink | Comments (1)

I don't know Jackie Bennett, who writes for our sister-blog Edwize, but I probably should.  Here's a lovely mix of elegance and eloquence:

Now here is the problem with focusing on outputs: there’s really no such thing. I mean, yes, there is such a thing as “outputs” (though the word is repugnant to me when applied to educating children), but there is no such thing as focusing on them. After all, how long can you look at a test score? Sooner or later, you have to look at the things that you think might influence that test score, and that’s inputs — that is, the nature of instruction and the quality of conditions at the school.Since there’s no escaping inputs, what outputists really do is simply replace one set of inputs with another.

Read the whole enchilada here. 

Posted by John at 03:07 PM | Permalink | Comments (0)

Education types, at least those who live and work inside the beltway, have been buzzing about growth models for quite some time, and one type of growth model, value-added, has been especially intriguing to policymakers.

As a 2004 Rand report put it:

Policymakers see VAM as a possible component of education reform through improved teacher evaluations or as part of test-based accountability. They are particularly intrigued by VAM because of the view that its complex statistical techniques can provide estimates of the effects of teachers and schools that are not distorted by the powerful effects of such noneducational factors as family background. 

The RAND report went on to caution that VAMs were a little rough around the edges, and economist Dale Ballou noted VAMs' limitations and problems way back in 2002.  

Recent work by Princeton researcher Jesse Rothstein casts even more doubt, adding an element of the absurd.  Using an ingenious approach, Rothstein argues that supposedly reliable value-added models can result in impossible, time-warping correlations. (To my mind, this is akin to the indirect proofs I learned --and later taught -- in geometry class. Start with an assumption; take it where it leads you; if it leads to a contradiction, the assumption is false.)

Rothstein writes: 

My tests are based on a simple falsification exercise: Future treatments cannot have causal effects on current outcomes, and models that indicate such effects must be misspecified.  I demonstrate that a simple VAM of the form typically used in the literature indicates large effects of current teachers on past achievement:  A student's 5th grade teacher has nearly as large an effect on her 4th grade achievement growth as on her learning during 5th grade.  This is direct evidence of non-random assignment.

If 5th grade teachers have an affect on 4th-graders' performance, then either time travels backward or there's something wrong with the model.

Or maybe there's something wrong with Rothstein's work.  If so, he'll hear about it.  He's inviting comments on his findings, currently in the form of a working paper.

Via Eduwonkette. 

*The Center for Public Education has a handy "At a Glance" piece on growth models here.

Posted by John at 04:29 PM | Permalink | Comments (0)

From Ed Daily, Jay Matthews on the value of the Newsweek high school rating system, the Challenge Index, which looks at the rate of participation in AP, IB and other college level test programs:

The more conventional method of using state achievement test results to assess a high school’s performance is more a reflection of the school community’s socioeconomic status than of its success in advancing student achievement, he said.

I guess making AYP must not mean much either.

Posted by Michele at 09:46 AM | Permalink | Comments (0)

Last week, after U.S. News followed Newsweek's lead and rated the nation's top high schools, Alexander Russo of This Week in Education said he really wanted to see "a comparison between the two magazines and AYP." 

Hey, Alexander, that's way too much for me to do, but here's a quick look at the three results for Athens High School in Troy, Michigan, which a school official describes as one "of the strongest high schools in the nation."

Made Jay Mathews' Newsweek list of Best High Schools.

Didn't make the U.S. News list.

Didn't make AYP because only 28 of the school's 32 special education students took the NCLB-required state test.

One interpretation is that Santa thinks this school is mostly nice but a little naughty, so, while Athens won't get a great present this year it probably will most other years.  Another view is that Santa's list is too random.

Sorry, with two kids under 5, I've overdosed on Santa already.  What I meant to say is that an NCLB-charitable interpretation would be that Athens is a good school that just needs to include more of its special education students in its assessments.  A less charitable view is that NCLB's accountability formula doesn't distinguish between excellent schools and terrible ones, or, as a former teacher of mine would put it, "doesn't know beans from apple butter.")

(Link to the Detroit Free Press article via Swift and Changeable; photo from BranchDesign via Flickr

Posted by John at 12:21 PM | Permalink | Comments (0)

Anyone who doesn't think that NCLB stacks the deck against diverse, urban schools should take a look at the recently-released NCLB implementation report. According to data for the 2003-04 school year, researchers at AIR found that:

• Even after controlling for poverty, 39% of schools with six or more subgroups did not make AYP, compared to 10% of schools with only one subgroup
• At every level of poverty, schools with six or more subgroups made AYP at a rate at least 30% lower than those with only one subgroup

Gee, I wonder why some states are "gaming" the system? 

Posted by Michele at 10:42 AM | Permalink | Comments (0)

I started watching The Wire through Netflix, so my inital reaction to Ed Sector's new report on how states "game" the NCLB accountability system is--as Avon Barksdale would say--"Play or get played." It's no surprise that in the face of federal requirements that they view as unfair and punitive, states look for ways to make their state accountability systems look good. That said, whether everything that states are doing constitutes "gaming" is subject to debate.

I strongly agree with one of the report's recommendations: that the U.S. Department of Education needs to improve the peer review process. States were clearly told different things by ED, and they quickly learned to ask for what other states had gotten. You feel me?

Posted by Michele at 03:52 PM | Permalink | Comments (5)

The Daily Herald, a suburban Chicago paper has a thoughtful, readable piece on what NCLB 2.0 might mean for Illinoisians.  The article also includes a quote about growth models that exceeds, in its hyperbole, any NCLB rhetoric I've seen so far:

Ed DeYoung, Elgin Area School District U-46's test score guru, is confident the nation will move to a system that looks at student growth rather than insisting all kids jump over the same bar.

"The alternative," DeYoung cautions, "is to self-destruct as a nation [emphasis added]."

In addition to this word-bomb, the piece hits on many of the big issues for reauthorization:  local assessments, multiple indicators, getting testing under control, uniform N sizes, and distinguishing between schools that miss AYP by an inch and those that are truly struggling.

Meanwhile, Steve Sawchuk of Education Daily ($) takes a more analytical approach to growth models and finds they have little impact and vary from state to state.  Looking at five states with growth models approved for a U.S. Department of Education pilot program, Ed Daily found that three -- in Florida, Iowa, and Arkansas -- increased the number of schools passing AYP by about 10 percent.  In two others, Tennessee and Alaska, growth models had little effect. 

Left unanswered -- and this is the $64 question:  Do real-world growth models do a better job of isolating the school effect? 

(Also left unanswered is whether Canada will be left unscathed if growth models are left out of NCLB 2.0 and the U.S. self-destructs. For an answer to that question, keep your eyes on the only news outlet that dares to explore such possibilities.)

Posted by John at 10:26 AM | Permalink | Comments (0)

I agree with one aspect of what Kevin Carey at The Quick and the Ed said yesterday about Linda Perlstein's new book, Tested--that education journalists often overreach, taking what they have observed in one classroom or one school and then making sweeping generalizations.  However, I take great exception to his second post on the book.  Carey writes:

Teaching students material that's way above their heads is bad educational practice. And you can't say the incentives built into NCLB leave educators with no other option, because--as Perlstein makes clear--it's also futile.  Even under the current, no-growth-model law, schools teaching students like Whitney have two choices: inappropriately teach at grade level, in which case Whitney fails the test this year and every year after that, or teach at the right level, in which case Whitney fails the test this year but catches up and passes tests in the future.

Wait--so Carey thinks it is educationally inappropriate to teach students above grade level but it's just dandy to test them every year knowing they will fail, because some day they will catch up to grade level and pass such tests? Maybe I am functioning below grade level, because I fail to see Carey's logic.

And, as much as some would like to continue to deny it, it sounds as though Perlstein's book is part of the mounting evidence that NCLB has led schools to focus more instructional time on testing and, in some places, caused a narrowing of the curriculum.  Anyone who talks to teachers on a regular basis hears it too often from too many quarters to dismiss those who protest as nee-sayers, uh, I mean, naysayers.  Can you hear me now? Good.

Update: Carey responds here. I think the nee-sayers would argue that using an out-of-level assessment is the best way to measure what students who are not performing at grade level know and are able to do, whereas using a grade level assessment would not provide useful information.

Posted by Michele at 09:11 AM | Permalink | Comments (2)

It seems that NCLB 2.0 is adhering to the principles of evolution--it's getting more complex, but I'm not sure it's a higher level organism. The Miller-McKeon discussion draft includes called for changes to AYP, allowing the use of growth models, performance indices and multiple measures.  However, the way these changes are incorporated renders the law's accountability provisions indecipherable, except perhaps to psychometricians. And, it's not clear that doing the Miller-McKeon accountability kabuki is worth it, that the ultimate judgment rendered about a school--either yea or nea--will be more accurate.  Don't get me wrong--I think that a fair accountability system must count student growth, but it should be more transparent and (gasp!) allow states more flexibility.

The deck is really stacked for high schools.  Current law requires that states count growth on a school's overall graduation rate to make AYP. The Miller-McKeon draft now requires that the graduation rate grow by 2.5 or 3 percent annually--overall and by every subgroup. So, a high school that shows growth in student test scores under a new growth model, for example, but does not improve its graduation rate for Hispanic students by 2.5 percent, would not make AYP. Should the graduation rate for poor and minority students be higher than it is?  Absolutely.  But changing how graduation rates are calculated for AYP purposes goes beyond this worthy objective and changes the accountability system in such a way that it is unlikely to be viewed as an improved measure of school performance.

Posted by Michele at 10:20 AM | Permalink

I am currently reading the new Shanker biography the Washington Way--I am looking up names in the index and then seeing what people said, or what is said about them, in the text.  (C'mon, I work here, it's all part of the intrigue.)  So, who did I start with?  Bella Rosenberg. And, in the short section on NCLB, Bella says:

"Al believed in eradicating achievement gaps, group distinctions . . . But Al also knew that since the beginning of time, there had been individual variability," so a performance standard which requires 100 percent proficiency by a certain date "is just a human impossibility."

In reading over that section, I thought, gee that sounds familiar. Who was it, who was it, oh yeah, it was Amy Wilkins at Ed Trust who recently said in press release a on the Miller-McKeon bill:

"The 2013-14 deadline for proficiency is a powerful disincentive to raising standards. If we are going to ask states – and students – to climb a higher mountain, we need to give them more time to get there . . ."

So, it only took Wilkins five years and half years of NCLB to realize what Shanker knew intuitively over ten years ago. He was a man ahead of his times.  To read some of the reviews of his biography, click here.

Oh, and if you really want to understand what makes the AFT tick, read the sections on the Social Democrats with care.  I, myself, never realized that Yetta Barsh, Shanker's assistant, was married to Max Shachtman.

Posted by Michele at 12:28 PM | Permalink | Comments (0)

September 5, 2007

Dear Chairman Miller and Ranking Member McKeon:

On behalf of the more than 1.4 million members of the American Federation of Teachers (AFT), I am writing to offer our comments on the Aug. 28, 2007, Title I discussion draft of the Elementary and Secondary Education Act reauthorization, No Child Left Behind (NCLB).

We understand that this is a “draft” document and as such is not meant to represent the House Education and Labor Committee’s final resolution of the important issues addressed in the document. We have approached it as such, and view it as the beginning of what we hope will be productive discussions leading to the passage of a bill that will provide our students and schools with the support they need to be successful.

Parents, teachers, elected officials and others have called for substantive changes to NCLB. However, the suggested revisions do not appear to adequately address their concerns and need further thought and substantially more clarity. We also would like to note that our comments are somewhat preliminary and may need further revision; we have yet to see the other titles of the bill and therefore cannot make an informed judgment as to how all the pieces fit together.

Five years of experience with NCLB have taught us many things. Chief among them is the need to take time and care to ensure—as nearly as possible—that what is enacted in Washington will work in our nation’s classrooms. We cannot achieve the law’s admirable goals if we do otherwise.

The AFT is particularly concerned with the following:

Continue reading "AFT Letter on Miller-McKeon Draft" »

Posted by Michele at 03:02 PM | Permalink | Comments (1)

(From AFT intern Justin)

In a fairly dramatic policy change, Nebraska Gov. Dave Heineman has authorized statewide reading and math exams with the passage of LB 653. EdWeek has the story here. Nebraska’s existing statewide accountability system, STARS, is cited as an innovative example of how a state’s accountability system can be built upon local expertise. The STARS name alone, “School-based Teacher-led Assessment and Reporting System,” implies the magnitude of district and teacher involvement in statewide accountability. On paper, STARS looks very innovative.

The statewide tests may salve any of ED’s residual concerns over Nebraska’s multifaceted, multi-textured assessment landscape. But introducing another assessment to an already assessment-heavy policy mix may incite controversy analogous to what we see with NAEP and state tests, i.e. are NAEP scores more valid and reliable, or are state tests? You only have to look back as far as a few days ago to the CEP NCLB results.

Assuming Nebraska keeps the STARS policy intact, it appears students will be assessed twice in math and reading: once locally and once at the state level in math and reading. Not only will students be twice-assessed, presumably in very different ways (local assessments are criterion-referenced and typically classroom-based), but Nebraska students could be tested on two different sets of standards. Since Nebraska districts adopt their own content standards (state minimum guidelines apply), students taking tests could potentially have to master two sets of standards.

There are any number of important questions here. Here are my top three:

• What incentives will districts have to maintain their own local content standards if a statewide test will test statewide standards?
• With so much local agency and teacher buy-in already committed to the current STARS system, what message does this send to teachers in the state? Seems like a total bummer from my vantage.
• How will the state deal with the inevitable NAEP-like bickering—which test is more reliable and valid, state or local?

LB 653 is bold. Redundancy issues aside, with the passage of this bill, Gov. Heineman has introduced statewide tests in a state that operates off of guiding principles like local control and teacher agency. He’s also created the impetus for the unraveling of local assessments. Doubling back on the teachers and districts that have invested so much into STARS, at this juncture, seems like a huge waste. Not to mention the Nebraska Education Department’s ongoing efforts to make STARS EDworthy. Let’s hope Nebraska didn’t just toss the baby out with the bathwater.

Posted by Guest at 02:44 PM | Permalink | Comments (1)

Bill Ratliff was the Texas state senate education chair back in the days when that state’s accountability regime was being constructed.  As we all know, that regime became a model for NCLB.  He is one of the main architects of the Texas approach to school accountability. This from a 2001 NCSL magazine profile:

In 1995, on his notebook computer, Ratliff rewrote all 1,100 pages of the state's education code except the financing section he'd written in 1993. He kept a no-pass, no-play requirement for school extracurricular activities, and a requirement the state maintain class sizes no larger than 22 students per teacher, but threw out most requirements about how teachers should teach. He replaced them with tests designed to see what students had learned.

"We removed all the methodology dictates and said, 'You go teach,'" Ratliff says. "All we want to do is say, 'This is what they need to know, and we're going to check to see if they know it.'"

 A few days back Ratliff testified in front of the Texas house education committee.  He said that the system has become so complex and unfair that Texas should "not try to fix it one piece at a time" but instead should "wipe the slate clean and start over."  According to my friends in Texas, the goal, he said, should be a system that is "understandable to parents, meaningful to educators, and consistent with national requirements."  Ratliff stressed that the proposed overhaul is "not about dismantling accountability" but about undoing the contradictions and complexities in the state system that have resulted from add-ons and from the additional layering of “NCLB on top of it."  Ratliff actually described the system as having  "gone berserk." He noted "as the father of the system," that "it was designed as a pretty simple system that tested math, English, and reading.  It was mostly diagnostic and served as a report card to the community as to how schools were performing."  In his testimony Ratliff decried a system that has gone from diagnostic to “"too complex for parents and educators to understand and too punitive to produce constructive results." The Austin American Statesman took up his call.  Ratliff is working with a group of business leaders under banner of Raise Your Hand Texas. Their goal is to repeal the current accountability system and start over.    

Posted by Ed at AFT at 10:58 AM | Permalink | Comments (1)

"In 22 schools, 100 percent of math and reading exams were discounted."  Assessments for transient students is a real issue, but this is not the way to fix NCLB.

Posted by John at 02:55 PM | Permalink | Comments (0)

NCLB is all about accountability, unless you are responsible for releasing test results. That is the lesson from Illinois, where they’re just getting around to releasing 2005-06 school year test results.  Problematic?  Only if you are trying to plan instruction or care about how improving upon test scores from last year. 

Luckily, NCLB’s AYP doesn’t measure progress, so schools in IL don’t have to be concerned with how they did last year or try to improve this year.

Is there blame to go around? You betcha. From the test administrators to the state data system managers, everyone had something to contribute to this mess.

The punchline: everyone promises to do better this year, when testing begins March 12.

Posted by Beth at 12:33 PM | Permalink | Comments (1)

Posted by Dan at AFT

In my first post in this series, I examined the claim that student achievement is declining, a common refrain of public school critics.  To counter this claim, I highlighted the impressive rise in test scores of 9-year-old African-Americans on the NAEP long-term trend assessment, widely considered the most reliable gauge of student progress.  (Click here to learn about the major differences between the NAEP long-term trend assessment and the NAEP main assessment.)
 
But what about other ages and subgroups?  Are they showing the same sharp rise in NAEP long-term trend scores? You can see the data for yourself here or read the highlights below. Remember: 10 points on NAEP equals about one year of learning.

Age 9

• Math (1973 to 2004):  African-Americans are up 34 points, Latinos are up 28 points, and Whites are up 22 points.
• Reading (1971 to 2004):  African-Americans are up 30 points, Latinos* are up 22 points, and Whites are up 12 points.

Age 13

• Math (1973 to 2004):  African-Americans are up 34 points, Latinos are up 26 points, and Whites are up 14 points.
• Reading (1971 to 2004):  African-Americans are up 22 points, Latinos* are up 10 points, and Whites are up 5 points.

Age  17**

• Math (1973 to 2004):  African-Americans are up 15 points, Latinos are up 12 points, and Whites are up 3 points.
• Reading (1971 to 2004):  African-Americans are up 25 points, Latinos* are up 12 points, and Whites are up 2 points

With the exception of the reading scores of 17-year-old Whites, all the gains are statistically significant. There are, of course, still sizable gaps in achievement between Whites and African-Americans, and Whites and Latinos, and we must continue to narrow them.  But the facts are undeniable:  Real and substantial progress has been made by all subgroups, particularly in math (a subject that is more easily influenced by in-school factors than reading) and in the earlier grades (when kids are more likely to take the tests seriously).

In my next post, we’ll take a look at why overall average NAEP scores (which public school critics like to seize on) mask the quite impressive gains made by student subgroups.

*In reading, NAEP didn’t treat Latinos as a separate category until 1975.

**Tests for this age group are notoriously unreliable so results should be interpreted with caution.

Posted by Guest at 07:15 AM | Permalink | Comments (2)

Posted by Beth 

The latest report from the Center for Education Policy shows that many states delayed getting their AYP results to schools this past school year, due to the ramp-up of testing to grades 3-8. 

One implication is that schools can’t start supplemental educational services (SES) and public school choice programs until they receive their AYP results. The vocal contingent that’s outraged that schools and districts drag their feet to implement SES and choice also happens to include the folks responsible for the problem. 

Here’s the ugly cycle: Rich, mega-education companies that run SES programs complain that schools are obstructing them from getting into schools early in the school year to run SES programs. The schools rightly say that they can’t start SES programs because they haven’t gotten AYP and school improvement information from states. States pass the blame back to the mega-education companies that, in addition to running SES programs, also own and score the tests that determine AYP.

Posted by Guest at 07:41 PM | Permalink | Comments (1)

NCLBlog welcomes Matt at AFT to our humble blog. 

"Accurate, but misleading." Those were the words used by the National School Boards Association’s Patte Barth to describe how the Washington Post chose to characterize--rather, mischaracterize--the performance of U.S. students on an international math assessment.

In a letter to the editor in Saturday’s Post, Barth writes:

The table that compared the United States with other countries in performance in fourth-grade math gave the false impression that the United States ranked 12th out of 13 countries, and it suggested that the only country the United States outperformed was Cyprus.

Not shown were 12 countries, including Italy, Australia and Norway, that were outperformed by the United States, leaving the impression that U.S. fourth-graders were next to last in the world. In reality, the United States is among neither the highest-performing nor the lowest-performing countries in fourth-grade math.

A fuller picture would also reveal that U.S. performance varies by subject and grade level. For example, U.S. fourth-graders performed well in reading; they were outperformed by their peers in only three of the 34 countries that took that test, and the Americans were above the international average.

In contrast, U.S. high school students were outperformed by their counterparts in 20 of 30 countries in math, a performance that was below the international average.

Posted by Guest at 10:09 AM | Permalink | Comments (0)

Posted by Beth 

Maybe it is because I am a big nerd but this may be my favorite annual NCLB-related report. Unlike other "reports," this one lacks opinionated bluster, and contains important data on AYP determinations that can only be gathered by someone with access to state education departments. There is a newsworthy story on every page, but the minutiae of accountability plan changes are rarely reported by any media outlets. And, the report always contains information that surprises me.

For example:

• Although states are required to test in each of grades 3-8 during the 2005-06 school year, many applied for and were granted permission to delay inclusion of results from newly tested grades into AYP calculations for this year.
• One state applied to exclude English language learners and students with disabilities from its low-income and racial/ethnic subgroups. ED has not yet responded, though an emphatic rejection seems like a complete no-brainer to me.
• Regarding the 95 percent participation rate requirement, many states applied for and were approved to count students who attempted a test but whose score was invalidated as a non-participant.
• States have been allowed to recalculate AYP for students who reach proficiency on a retest given after summer school.

And the report illuminates new wrinkles to NCLB accountability, as in the case of the state that pointed out that allowing students to transfer schools under the choice provision would mean that most of these transferring students would not count in AYP calculations for that year because they would not have been in the transferred-to school for the required full academic year.

Bottom line: Despite ED’s constant press releases and speeches declaring that NCLB is responsible for raising student achievement, it remains nearly impossible uncover the truth because factors like who is included in the different subgroups and the definition of proficiency are constantly changing.

Posted by Ed at AFT at 01:14 PM | Permalink | Comments (0)

Update: More from Mehta. 

First, welcome to Eduwonk Guestblogger Jal Mehta who has jumped fearlessly into the fray with a post on Sam Dillon's New York Times piece about whether NCLB is closing the achievement gap.  After reading the article, and Mehta and Alexander Russo's posts, I was reminded of an earlier post I had written, The Case for Being Mean. If the gap is not beginning to narrow, it could be argued that NCLB's "mean" accountability measures are not working, and that what struggling schools really need is assistance in how to improve. And, though the AFT has recommended that NCLB sanctions be changed to be more like interventions, I will instead pimp--as Mehta calls it--for Dick Elmore, who argues in this American Educator piece that,

Schools don’t suddenly “get better” and meet their performance targets. Improvement is a process, not an event. Schools build capacity by generating internal accountability—greater agreement and coherence on expectations for teachers and students—and then by working their way through problems of instructional practice at ever-increasing levels of complexity and demand. Right now, virtually no infrastructure exists to provide continuous support to failing schools.

Building capacity in failing schools is going to require a lot of feet on the ground—people who know something about school improvement and who know what they don’t know . . . I would look for these people in what I have called “improving” schools . . . where faculty and school leaders have worked through several stages of improvement, and in improving school districts, where district-level personnel have gained real knowledge about the kind of support and resources schools need to improve.

Posted by Michele at 11:56 AM | Permalink | Comments (0)

Posted by Beth 

Which is the more difficult of NCLB’s twin goals—closing the achievement gap or attaining proficiency for all? According to Rothstein and friends in this new piece, it is the latter, and setting a high bar for proficiency for all (like NAEP does) sets up a task that no nation is anywhere near achieving. Setting a low bar for proficiency for all (like many states have done) doesn’t mean a whole lot in terms of educational goals because the point is that almost everyone can and should pass without a lot of effort (think driver’s license exam).

Bonus material: a lengthy discussion of the long, sordid history of how the NAEP bar got so high.

Posted by Guest at 03:20 PM | Permalink | Comments (0)

Update: Ed Daily($) reports that the Council did formally announce its support of national standards at the release of the urban NAEP science scores yesterday.  The article also quotes Mike Petrilli of the Fordham Foundation as saying that the Council, in addition, supports national testing, but I am not sure that is the case.

From the Council of Great City School's press release on the urban NAEP results in science, it sounds like the organization is supporting a move to national standards.

Posted by Michele at 12:00 PM | Permalink | Comments (0)

Posted by Beth 

In response to criticism and more criticism of this post on growth models, my point was not that growth models should be a great big loophole for schools that don’t make "real" AYP.

The central criticism of NCLB’s current AYP system is that it does not give credit to schools that are making meaningful progress with students who start furthest behind academically. A growth model that only gives credit to those "on track to proficiency within three years" does not seem like it addresses this main criticism. The U.S. Department of Education is calling this a pilot program. As such, I believe they should allow a broad range of models, and see what happens. What they did, at least with the first two acceptances, was set up such a narrow definition of growth that in a few years, folks can sigh and say, "Too bad that didn’t work. I guess bad schools are bad schools no matter how you measure them. The solution has to be vouchers."

Posted by Guest at 01:50 PM | Permalink | Comments (1)

Posted by Beth 

Delaware, Arkansas, and Florida got the nod from ED to use growth models to calculate AYP, joining North Carolina and Tennessee.  Two questions to ponder:

• Do the models for these three states differ significantly from the models for NC and TN? When those states were approved, there were grumblings that ED’s narrow definition of “growth model” should be expanded to allow for some diversity.  We’ll see—ED is supposed to post the three new models within the week.
• What’s the point? Apparently, in NC, no schools made AYP just because of their growth model.  It’s a lot of work for states and the feds to submit and analyze plans.  If it is not helping, why continue with the process?
  

Posted by Guest at 08:45 AM | Permalink | Comments (2)

During the recent AFT education policy forum, Paul Barton made reference to a piece in the newest edition of Education Next, where Paul Peterson argues that NCLB's AYP formula is as likely to identify a school as failing as is flipping a coin.  I finally took a look at the piece, and here is what Peterson says specifically:

Still another way of thinking about the accuracy of the NCLB yardstick is to calculate the probability that AYP identifies correctly the higher-performing of any two schools being compared. Of course, any two-category classification system will get it right 50 percent of the time, by chance alone, just as one can guess correctly half of the time which way a coin will flip.

How much better than chance did the NCLB grading system do in Florida in 2004? In math, a school that made AYP outperformed a random non-AYP school 71 percent of the time. In other words, 29 percent of the time the school in which students are making smaller gains is the one that passed AYP, a pretty hefty error rate. In reading, that error rate was 28 percent. To be wrong nearly three times out of ten does not inspire confidence—especially when one can get it right half the time simply by random guessing.

NCLB seems to make for strange bedfellows. (Now who will be the first blogger to make a ribald joke--Eduwonk?)

Posted by Michele at 01:05 PM | Permalink | Comments (2)

Nothing, but Andy Rotherham managed to invoke both in his comments at the AFT's education forum on school accountability earlier this week.  Barton's proposal, which calls for pre- and post-testing of students and moves away from annual testing, was praised by by the panelists as thoughtful, but received an equally thoughtful critique.  To read coverage of the forum, check out the Ed Daily article here.  And, thanks to all the bloggers who submitted questions, but the forum did not get to the issue of value-added systems.

Posted by Michele at 09:57 AM | Permalink

UPDATE: Howard at AFT responds to MassParent.

You must be one of those MIT professors living In Lexington. You have stumbled upon one of two “secrets” that lets Massachusetts look good even when it has allegedly high standards.

Here is the one point you missed. Proficiency is measured by the Composite Proficiency Index (CPI), which gives partial credit for proficiency to many students who scored at “basic” and even “failing” levels. Personally, I think that these proficiency indexes should be used by all states (only a few have been allowed to do so).  But politicians should stop talking about high standards when partial progress is rewarded.

This is an elaboration of the point that you made: Massachusetts has been using a growth model with error bands called “Alternate Safe Harbor” Schools in Massachusetts can make AYP either by meeting state proficiency targets (the ones that give credit for partial progress); by reaching improvement targets that are unique to each school and subgroup (described by the state as an alternate version of Safe Harbor); or by meeting the Safe Harbor provision as it was specifically defined in the No Child Left Behind (NCLB) legislation. Massachusetts also applies confidence intervals (error bands) to the improvement targets in the alternate AYP calculation (but not to the proficiency targets).

Massachusetts’s Alternate Safe Harbor includes many features that the U.S. Department of Education rejected in 2006 when it allowed other states to propose a growth model such as growth trajectories based on the year 2014, counting subgroup growth rather than individual student growth, and the application of confidence intervals to the growth rate.

If you really want to get into the details and the implications of the two AYP secrets used by Massachusetts, see:

http://www.aft.org/topics/nclb/downloads/MASS.pdf

Not to sound like a spoilsport but . . . the Boston Public Schools recently won the Broad Prize for, among other reasons, "consistent high performance while reducing achievement gaps across ethnic groups," yet 65% of its schools failed to make AYP in 2006.  What's up with that?  Seems either the Broad are pretty loose in awarding half a million dollars, or AYP is not an accurate measure of the progress achieved in urban school districts.

Posted by Michele at 07:40 AM | Permalink | Comments (2)

For those coming to today's Education Policy Forum at the AFT, be sure to give Paul Barton's new paper a quick read ahead of time.  Heck, read it even if you aren't coming--and if you have a question, submit it here in the comments section or to michele@letsgetitright.org.

Posted by Michele at 07:36 AM | Permalink | Comments (2)

Posted by Amy 

On Tuesday October 10th at 3 pm, the AFT will launch its new Education Policy Forum series with a debut of Paul Barton's AFT-published report called "Failing" or "Succeeding" Schools: How Can We Tell? Panelists Paul Barton, Jack Jennings, Raul Gonzalez, and Andy Rotherham will duke it out over the ideas raised in Paul's paper, which details what Paul calls a "morphing" of the standards movement into "assessment as the treatment." Barton's solution? Get back to content and curriculum, and figure out how to measure what students have learned in a school year.

Barton suggests pre/post school year testing as the best way to measure student learning gains. And yes, he admits that under the current NCLB model, his approach would require more testing. But if we look beyond the NCLB mindset of grade 3 through 8 plus high school testing each and every year, in math, reading, and (soon) science, and if we realize that not all tests are created equal, Barton's model begins to really sing. Here are some of the questions Barton, himself, asks us to consider:

Is it really necessary to measure quality of school performance in every year, in every subject, in every grade to promote school improvement?

Can some grades be tested in some years, on a rotating basis?

Can schools and subjects be "sampled" for testing, rather than applying 100 percent testing annually?

Do schools change that quickly to require measuring every year?

Pre/post testing would mean more testing, yes. But if testing for NCLB school accountability purposes is less frequent, this creates space for more frequent testing for diagnostic purposes, and it's these kind of assessments that most help teachers target their instruction to increase student learning.

Is Barton's proposal politically sailable or at all realistic in today's high-stakes accountability era? Come and participate in next week's discussion--you can pre-register for the event through today: Or, submit your questions ahead of time by commenting on this blog.

Posted by Guest at 07:50 PM | Permalink | Comments (0)

Posted by Beth 

So, what to do if you’re a good Republican and having a very bad week? Rally the forces around good ol’ Republican values? Uh, that’s not going to work this week. Plan B: Send the president into a charter school with the message, "Smile, No Child Left Behind is Working!!!" Secondary messages: there is not too much testing, there’s plenty of money to get the job done, and let’s expand that voucher program.

Posted by Guest at 02:10 PM | Permalink | Comments (0)

Posted by Beth 

Once again, ED has states in a bind. (A version of this post has been written so many times it could write itself.)

Rounds two and three of the review of growth model proposals are scheduled to begin in mid-October. One criterion for acceptance is a fully-approved standards and assessment system. Here’s the catch: only four states currently have a fully-approved system, and two of the four are TN and NC, the two states currently allowed to implement growth to calculate AYP. The other two, AK and DE applied, but so did FL, OR, and AR, states slapped with a "mandatory oversight" label on their standards and assessment system.

Funny how ED says they want to see a diverse range of growth proposals, but then creates rules that limit the field to four. So funny you’d think they were being insincere.

Posted by Guest at 08:44 AM | Permalink | Comments (0)

Posted by Beth 

I’ve been wondering what exactly folks mean when they say that NCLB should expand into high schools. Ray Simon said it here, but his explanation ("increasing the role of the Title I program in our high-poverty high schools") didn’t clarify precisely what this Administration would like to do.

I’ve heard groups talk about how they would like to increase testing in high school, tighten up the graduation provision within AYP, and expand programs geared toward high school (like Striving Readers).

And there is the funding issue. The Alliance for Excellent Education provides this helpful stat:

"Secondary schools are less likely than elementary schools to be targeted for Title I funds, and even when they are targeted, they receive fewer dollars. On average, elementary schools average $495 per student compared to only $372 for middle school and high school students, a $123 difference per child."

Does this Administration intend to even out the distribution of Title I funds between elementary and secondary schools? If so, I am interested in hearing the "how" and "why" of their proposal. I'm also interested in seeing how it plays with elementary school advocates who, without a significant increase in overall Title I funding, would lose funds in the evening out process.

Posted by Guest at 12:36 PM | Permalink | Comments (0)

Posted by Beth 

Number one on just about everyone’s (including ours) list of suggested fixes to NCLB is the AYP system. Schools should get credit for meaningful progress not just an arbitrary proficiency level.  It seems fair and it  means more schools would be faring better, right?

Except that we’re getting evidence that schools don’t fare much better under growth. Education Week reports that in TN, one of the two states allowed to use a growth model to calculate AYP last school year, only eight schools made AYP through their growth model. (Results from NC, the other state allowed to use a growth model this year, are not yet in.) Kati Haycock, in her July 2006 testimony to the House Education committee warned that in NC, if accountability were not based on proficiency, but based solely on growth, fewer schools would have made AYP than currently do.

Just because it is not going to get most schools out from under NCLB’s accountability hammer shouldn’t be the sole reason for supporting growth as an alternative or addition to the current AYP model. It is still the right thing to do, but just adding growth as a way to calculate AYP is not going to “fix” NCLB.

Posted by Guest at 01:19 PM | Permalink | Comments (2)

So, I went to the Business Roundtable forum on NCLB yesterday, and the Ed Sector sent an intern--not sure what that says about my status at the AFT! :-) I guess Ed Sector figured not much would be said--as intern Alex Redfield describes the event in his post, "Everyone was utterly predictable and on their best behavior." Let's see: who do you think, gave NCLB a grade of "F"?

a. David Dunn, Acting Education Department Under Secretary and Chief of Staff

b. Howard "Buck" McKeon, House Education and the Workforce Committee Chair

c. George Miller, Ranking Democrat on the House Education and the Workforce Committee

d. Reg Weaver, President of the NEA

If you said David Dunn, do not pass go or collect $200 dollars.  In fact, Dunn exhibited his usual boyish charm.  (I recall him presenting to my public accounting class at the LBJ School, taught by the scion of the Texas state budget, Tom Keel.  Dunn was quite charming and even more boyish back then.)  The only time Dunn went off message was when responding to a commenter who said all three of his sisters are teachers and hate NCLB.  Dunn replied, "Yeah, my sister is a teacher and she hates it too." Wonder if she's a teacher in Texas? Could be an AFT member.  Hmm.

Another moment of stark honesty came from McKeon who, when asked whether vouchers would figure in the next reauthorization, responded, "As far as I'm concerned, it's a dead issue."

Posted by Michele at 01:51 PM | Permalink

So, one day after House Committe on Education chairman Buck McKeon and ranking committee Democrat George Miller declare national tests and standards D.O.A. at a Business Round Table (BRT) event on No Child Left Behind, former U.S. education secretary Bill Bennett and Rod Paige renew the call for both in today's WaPo.  Great timing. I think McKeon's exact words were that he and Miller would be "hung" if they proposed national standards and tests.  Doesn't sound too promising.

What does sound interesting is this part of the op-ed: "As both of us have long argued, Washington should set sound national standards and administer a high-quality national test.  Publicize everybody's results, right down to the school level.  Then Washington should butt out."  Really?  So, districts would then be free to work with low-performing schools to tailor the interventions needed to turn them around, instead of following the lock-step order of NCLB sanctions leading from public school choice to stake takeover or privatization?  OK, now I'm listening.

Posted by Michele at 06:36 AM | Permalink

Bob Linn from CRESST wrote an interesting paper back in June discussing whether tests should be used to make causal inferences about school effectiveness.  Linn believes that such practice is "not scientifically defensible."  He instead argues for, "Treating the [test] results as descriptive information and for identification of schools that require more intensive investigation of organizational and instructional practice."  Given policymakers' infatuation with tests, his argument is unlikely to persuade many, but if adopted, it would allow states and districts to focus school improvement efforts on those schools that need the most attention, rather than spreading limited resources thinly across schools.

Posted by Michele at 09:50 AM | Permalink | Comments (0)

The AFT is launching the first of a new series of policy forums, beginning October 10th with the release of the Paul Barton paper, "Failing" or "Succeeding" Schools: How Can We Tell? The forum will be held from 3:00-4:30 in the 4th Floor Conference Room at the AFT, located at 555 New Jersey Avenue, NW in Washington, D.C.

Barton is an education writer and consultant, and the former Director of the Educational Testing Services' Policy Information Center. The paper will be the subject of a lively debate by the author and his co-panelists: Jack Jennings, President and CEO, Center on Education Policy; Delia Pompa, Vice-President, National Council of La Raza; and Andrew Rotherham, Co-Founder and Co-Director of Education Sector.  AFT Executive Vice-President Antonia Cortese will moderate the panel.

The event is free and open to the public, but pre-registration is required. Click here to register.

Posted by Michele at 04:17 PM | Permalink | Comments (0)

The AFT has developed some online resources to help folks sort through the various graduation rate studies that have recently been released, including a summary of the research, information on how states calculate graduation rates, and a refresher on what NCLB says about graduation rates.

Posted by Michele at 01:12 PM | Permalink | Comments (0)

Secretary Spellings seems to think anyone who doesn't want schools held accountable for teaching spelling is a crybaby, but there are some non-whiny reasons why -- leaving the issue of spelling aside for the moment -- it might be unwise to add more NCLB tests.

NCLB, in broad terms, seeks to measure the effectiveness of schools and use federal dollars to change those that are ineffective.  Unfortunately, adequate yearly progress (AYP), the law's formula for measuring school effectiveness, fails to give credit for the progress of individual students.  This means schools that do a good job with students who start out far behind (often economically disadvantaged students) are deemed "in need of improvement" while schools that do a mediocre job with relatively affluent students are given a pass. (AFT has recommendations to improve AYP by addressing this and other problems.)

In part because of NCLB's AYP formula, the law is perceived (by parents and teachers) to have contributed to a phenomenon sometimes called narrowing of the curriculum -- increased attention to math and reading, less attention to everything else.

One way to address this concern would be to change AYP so that it factored in student test scores for every subject we think is important.  (Parents seem receptive to this.)  That might solve the narrowing of the curriculum problem, but it would create other, greater problems.

It's not simply that more tests would mean more opportunities for students and schools to fail.  If more testing meant more accurate and meaningful information about schools, that would counter many of the arguments.  But it makes no sense to demand more information right now from a broken AYP formula and an uneven testing regimen. 

With science tests already in the pipeline, that subject would be the most likely to candidate for inclusion in AYP.  But Beth has written about why that idea is not ready for prime time. 

Trying to do too much may lead to a backlash against NCLB that would threaten not only the law's current accountability system but also the funding NCLB brings to disadvantaged students and the schools that serve them.

Posted by John at 01:52 PM | Permalink | Comments (0)

Posted by Beth at AFT

Education Week reports that many states won’t have last school year’s AYP results ready until well after this school year has begun. Is this a problem?

Yes, if these states care to comply with U.S. Department of Education guidance on the subject, which says "The SEA [state education agency], in conjunction with the LEA [local education agency], must conduct its review of school progress (AYP) annually, in the period of time between the release of student results on the State academic assessments and the start of the school year following the administration of the assessments."

Yes, if you're a parent trying to make an informed decision about your child's schooling, or a teacher trying to plan instruction.

No, if you think that current sanctions for not making AYP (choice and SES, in particular) aren’t actually beneficial to students or schools, so what’s the hurry in making these options available at the risk of rushing the data and making mistakes.

Maybe so, if you’re the director of this Tennessee charter school, which, apparently, has a very vested interest in seeing her local district public schools fail to make AYP. School Director Smithson says, "We recruit students only from schools that have had low TCAP scores. But TCAP scores are released just a few weeks before school starts. We end up with an unfair disadvantage of not being able to find enough students to enroll."

Posted by Guest at 03:24 PM | Permalink | Comments (0)

Posted by Beth at AFT 

Remember how Connecticut challenged NCLB's requirement that states test in every grade 3 through 8?  CT contended that they weren't going to get any new information from testing in every grade beyond what they got from testing in every other grade, and that they didn't have the funds necessary to implement the testing. Well, CT's AYP results are in, and the number of schools that did not make AYP in 2005-06 exactly doubled to 290 schools (from 145 schools in 2004-05).  

So, was CT right? One of the promises of NCLB is that more and disaggregated test data will assist a state in focusing attention on schools that need the most assistance.  When doubling the grades tested leads to double the number of schools not making AYP, has a state learned anything more about how to focus its resources?

CT has argued that they don’t have the capacity and financial resources to implement NCLB’s technical assistance requirements. Well, now the Constitution State has identified 290 schools needing technical assistance rather than 145.

Posted by Guest at 10:15 AM | Permalink | Comments (0)

Posted by Beth 

Title of this week's press release from the Texas Education Agency: Number of Exemplary and Recognized Districts and Campuses Increases Dramatically.

Paragraph 5: "Although there was also a slight increase in the number of Academically Unacceptable districts and campuses…"

Paragraph 21: "Schools in counties declared a federal disaster area due to Hurricane Rita that were closed for 10 or more instructional days because of this storm were given a rating based on 2006 ratings data only if their rating would be equivalent to or better than their 2005 rating."

Not included in this press release: These results have nothing to do with AYP under NCLB, but are the state’s own accountability ratings.

Also not included in this press release: In 2005, Texas was fined by the U.S. Department of Education for not notifying districts of schools’ AYP status until September 27, more than a month after most schools had started their school year. In Houston, school year 2006-07 starts on August 14.

I’m not falling for a Texas miracle anytime soon. In the words of the governor who presided over the