3
METHODOLOGY AND ANALYSIS OF TUDA DATA CONT’D Pieces of the Puzzle
District that consistently failed to make gains or posted losses: o
Cleveland
o
Charlotte
District that outperformed other districts adjusting for background characteristics: These four districts constituted our sample of districts for the alignment analyses and case studies of instructional programs and other contextual factors that may be associated with TUDA performance on NAEP.
Part 2. Alignment and Subscale Analyses and Site Visits This section describes the methodology for analyzing the alignment between NAEP and the state and/or district standards in the four selected jurisdictions and for conducting the site visits to determine the nature of the instructional programs in the districts. Alignment Analysis The guiding research questions for this component of the project were as follows: In the four selected districts, to what degree were the state and district standards in place in 2006–2007 for reading and math in grades four and eight aligned with NAEP specifications in terms of content, sequence, and depth of cognitive demand? And to what degree were the state and district standards in place in 2004-2005 for science aligned with NAEP specifications in terms of content, sequence, and depth of cognitive demand? This section describes the processes used to answer these questions, including the training, recording, and validation procedures used to match subject matter content by grade for states and districts, code the matched content, and summarize the codes. Materials collected To conduct the reading, mathematics, and science alignment analyses, the project team first assembled NAEP content specifications in place for 2003 through 2007. The specifications for NAEP reading in 1992 through 2003 were not public, so the core material for conducting the alignment included The Reading Framework for the 2003 National Assessment of Educational Progress, published by the National Assessment Governing Board (NAGB)9; its predecessor document, Reading Framework for the National Assessment of Educational Progress: 1992-1998 (NAGB, no date); and internal documents provided by the Federal Statistics Program (FSP) housed at the American Institutes for Research (AIR). Of particular use was the 2003 Reading Framework, which provided examples of how the assessment had defined and measured ―aspects of reading‖ within each ―context for reading.‖ This source provided material from which proxies for reading specification statements could be extrapolated. The team then developed most proxy specifications, with a brief note about the reading behaviors they covered.10
9 10
Available at http://www.nagb.org/publications/frameworks/r_framework_05/toc.html The content leader for the reading component of the alignment work had coordinated the development of the 1990 NAEP reading assessment, served for four years on the technical review team for the 1992 NAEP reading assessment, and served at AIR, directing the project that developed The Framework for the 2009
45 Council of the Great City Schools and the American Institutes for Research 64
PIECES OF THE PUZZLE: FACTORS IN THE IMPROVEMENT OF URBAN SCHOOL DISTRICTS ON THE NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS