In the short life of NCLBlog, we've discussed education studies on a range of topics, including teacher certification, testing, AP course-taking, student achievement in private and public schools, and charter school authorization.  NCLB is supposed to encourage research-based programs and direct millions in federal dollars to such programs, so it's worth taking a little time to sort out how studies and statistics are used and abused by people advocating a particular position.

A quick look around suggests that stat abuse is all too common.  For example, a recent issue of "Education Gadfly," an electronic newsletter from the Fordham Foundation, ignores "statistical signficance," the test by which statisticians determine whether a finding is meaningful or just a numerical quirk.  Gadfly reviews a report that finds a statistically significant result:  public school students outperform similar charter school students on 4th grade NAEP math tests.  And Gadfly equates that finding with one of the report's statistically insignificant findings about 8th graders' test results.  Of course, the statistically insignificant finding to which Gadfly clings just happens to support the Gadfly's blinded-by-love approach to charter schools.  Worse still, it was just 18 months ago that the Gadfly tried to knock down data unearthed by NAEP AFT researchers because, while the overall results were statistically signficant, some subgroup findings were not.

Researcher Caroline Hoxby attempted to dismiss a statistical technique used in report about student achievement, saying "Only methods that guarantee apples-to-apples comparisons, such as randomization, produce results that can be taken seriously."  Hoxby sets the bar high -- so high, in fact, that she is arguing, essentially, that her research on the effects of school competition (her "rivers and streams" study), which is based on econometric techniques rather than randomization, should be questioned.  The same is true of her 2004 report on charter school achievement.

Most of us can't "do the math" when it comes to evaluating sophisticated studies, but the miracle that is Google allows us to see when Hoxby, the Gadfly and others bend over backwards to tout research that supports their views and attack research that doesn't.