Pieces of the Puzzle Table 4a.17 Comparison of characteristics of NAEP and state reading assessments in grades 4 and 8, by state, 2007 Factor
NAEP
Georgia
Massachusetts
North Carolina
Ohio
Focus
Reading comprehension
Reading and English language arts
Reading and Composition
Reading
Reading
100% of test
Majority of test*
100% of test
No
Yes (minimal)
No
Majority of test* Yes (minimal)
Could not determine** Could not determine** ~ 200 – 400*
~ 60%
Could not determine** Could not determine** Could not determine**
Balance of Item Types MultipleFewer than 50% choice at each grade Constructed At least 50% at Response each grade Balance of Text Types At least 55% of Literary passages Informational/ Procedural Passage Length, in words
At least 45% 200 – 800 at fourth grade 700 – 1000 at eighth grade
~ 40% 640 – 885 at fourth grade* 440 – 1300 at eighth grade*
~ 40% - 50% ~ 40 – 50% 300 – 700 at fourth grade 450 – 1000 at eighth grade
Notes: Information about the 2003 NAEP Reading Assessment was taken from the Reading Framework. Information on state reading assessments at grades 4 and 8 was taken from state education agencies’ public Websites. In some cases, information on the assessments for years of interest in this study was limited or not available. Entries marked by an asterisk (*) were estimated using material on state education agencies’ public Websites. Information for entries marked by a double asterisk (**) could not be found on state educational agencies public Websites. Information about the Ohio tests was available from AIR’s Assessment Program, which develops the tests.
One should be cautious about drawing conclusions about the information in table 4a.17 that compares NAEP and the state reading tests, however. The cognitive demand on NAEP reading may best be judged by examining actual ―item sets,‖ that is, the passages students must read and their accompanying multiple-choice and constructed-response items. NAEP reading passages are usually followed by eight to 10 questions or items. The cognitive demand of the proxy specifications, which were used for comparative purposes, provides only partial information about how demanding the assessments actually were. Longer passages were often more cognitively demanding to read because more information was included and because questions often required test takers to make connections across longer stretches of text. In addition, constructed-response items can be more demanding than multiple-choice items because test takers may be required to synthesize or analyze information or stand back from what they have read and express their own evaluations or judgments in their writing. This is especially true for extended constructed response items that require students to write three or more connected sentences that are scored on a four-point scale. Summary of Analysis of Reading Standards Alignment and NAEP Results Our analysis showed that content and cognitive-demand alignment was not high between NAEP reading specifications in grades four and eight and state and district standards in Atlanta, Boston, Charlotte, and Cleveland. 81 Council of the Great City Schools and the American Institutes for Research
Council of the Great City Schools • American Institutes for Research • Fall 2011
101