Answering the Question That Matters Most: Has Student Achievement Increased Since No Child Left Behind?

CLAIMS

Methodology is too weak to support finding that student

achievement has increased since passage of NCLB

EAST LANSING, Mich. — A report released last month by

the Center on Education Policy (CEP) is being used to argue that student

achievement has increased since the implementation of the federal No Child Left

Behind law. A review of this report finds that it suffers from important

weaknesses and that the wording of numerous findings and key conclusions imply a

much stronger connection between NCLB and increased achievement than can be

substantiated by the data.

The report, Answering the Question That Matters Most:

Has Student Achievement Increased Since No Child Left Behind was reviewed

for the Think Twice think tank review project by John T.Yun, a professor at the

University of California at Santa Barbara.

The CEP’s report has already

received widespread attention from the news media, including front-page coverage

in the Washington Post . The U.S. Secretary of Education immediately

pointed to the report as confirming NCLB’s success. As reviewer Yun notes, the

report “is likely to be cited often in the upcoming debate on the

reauthorization of the No Child Left Behind Act (NCLB).”

While Yun credits the report

with attempting “to carefully analyze the complex issue of test score

improvement before and after the implementation of NCLB in 2002,” he describes

how shortcomings in the data and analyses may have “resulted in a much more

optimistic picture of the impact of the legislation than the data warrant.”

Additionally, Yun says, while the report’s title may convey the impression that

it seeks to examine the direct impact of NCLB on student achievement, the report

itself acknowledges that an analysis accomplishing this goal may be impossible.

The possible effects of NCLB cannot be disentangled from the possible effects of

the large number of other state and local policies aimed at raising achievement

during the same period of time.

The review from Professor Yun contends that the most useful

and important finding in the new report is its explanation of current weaknesses

in state data availability. The authors had great difficulty in obtaining and

analyzing state-level achievement data that should be readily available; until

this situation is improved, researchers such as those at CEP will be faced with

many of the same obstacles encountered here. But Yun stresses that this finding

was given far too little attention and should not have been overshadowed by the

problematic analyses of student achievement.

Regarding the student

achievement analyses, Yun does credit the report with offering thoughtful

approaches and concludes that the report does represent progress toward more

comprehensive examination of the outcomes of the law. He also commends the

report for cautionary notes that help readers understand some of its

limitations—yet not the three that he identifies as most serious:

The report’s look at

whether achievement scores have increased since 2002 (its ‘trend analysis’)

used an approach that had a likely unintended effect of analyzing a sub-sample

of states that was biased toward those that were most likely to have inflated

test scores.

The report’s finding of narrowing achievement gaps

between groups of students suffered from the same weaknesses as its trend

analysis, and also suffered from the problem of small sample sizes among some

of the studied racial and ethnic groups, meaning that each of the

percent-proficient estimates were effectively unreliable.

Selection bias is also likely to have “seriously

damaged” the value of the report’s analysis of pre- and post-NCLB outcomes,

which found yearly gains in test scores greater after NCLB took effect rather

than before. In fact, Yun points out that the authors selected for “exactly

the wrong group” of states to consider if the goal of the analysis was to

examine the impact of NCLB, because the approach used had the effect of

screening out those states that changed their approaches following NCLB’s

passage.

These limitations seem particularly salient given the

report’s prompt interjection into the public debate. U.S. Secretary of Education

of Education Margaret Spellings, for instance, used the report to argue for NCLB

reauthorization. Her official statement said, “This study confirms that [NCLB]

has struck a chord of success with our nation’s schools and students. … We know

the law is working, so now is the time to reauthorize [it].”

Commenting on this new review in light of such reaction the

report, Professor Kevin Welner of the University of Colorado at Boulder,

co-director of the Think Twice project, focused on the issue of publicity. “The

methodological problems pointed out by Professor Yun are important and should be

carefully considered by any policy maker or researcher who makes use of the

study. But the bigger problem here seems to be in the packaging and subsequent

publicity. Neither the data nor the analyses in the report are anywhere near

strong enough to meaningfully support the report’s title, ‘Answering the

question that matters most,’ nor can the report support the sort of puffery we

see from Secretary Spellings.”

Find the complete review by John Yun as well as a link to

the Center on Education

Policy report at: http://www.greatlakescenter.org .

About Think Twice

The Think Twice project provides the public, policy makers

and the press with timely, academically sound reviews of selected think tank

publications. It is a collaboration of the Education Policy Studies Laboratory

at Arizona State University and the Education and the Public Interest Center at

the University of Colorado at Boulder and is funded by the Great Lakes Center

for Education Research and Practice.

The mission of

the Great Lakes Center for Education Research and Practice is to identify,

develop, support, publish and widely disseminate empirically sound

research on education policy and practices designed to improve the quality

of public education for all students within the Great Lakes Region.

Visit the Great Lakes

Center website at: http://www.greatlakescenter.or g

Contact:
John T. Yun (805) 893-2342
(email) jyun@education.ucsb.edu
Kevin Welner (303) 492-8370
(email) kevin.welner@gmail.com