3
METHODOLOGY AND ANALYSIS OF TUDA DATA CONT’D Pieces of the Puzzle
Additional information on the FPE methodology is provided in appendix C. Where possible, our analyses were performed using both the reporting sample and the FPEs, but there were two notable exceptions: 1. Analyses controlling for student background characteristics. Some covariates for these analyses are derived from the NAEP student background questionnaire. Data on background characteristics for students not participating in NAEP are not available. Therefore, the calculation of scale scores controlling for background characteristics was conducted on the reporting sample only, not the FPE. 2. Subscale analyses. Imputed composite scale scores are estimated for students who do not participate in NAEP, but FPEs for subscale scores are not available. Therefore, subscale analyses were performed using only the reporting sample, not the FPE. Quintile Scores We also examined trends over time by looking at various points in the distribution of achievement scale scores and calculating average or mean scores and gains at each quintile. Change in achievement is often measured by comparing the overall differences in average scale scores between two periods. Instead of aggregating all students into one average scale score, however, we disaggregated the data into equally weighted quintiles, which yielded five separate achievement groups. We then calculated average scale scores and gains for each quintile using the following procedures. 1. For each group in each time period, we ranked student scale scores from lowest to highest. 2. For each group, we partitioned students into five equally weighted quintiles with the lowestscoring students in the lowest quintile, the second lowest-scoring student group in the next quintile, and so on. 3. We computed the average scale score for students in each weighted quintile for each time period—2003, 2005, and 2007.
Process for District Selection This section describes the methodology used to narrow the number of TUDA districts for more in-depth study. Selection of Districts Based on Gains and Losses Across Years Defining what it meant for a district to significantly and consistently make gains on NAEP was of critical importance to the study. Accordingly, we measured gains using changes in scale scores. We documented gains in reading and mathematics at grades four and eight using changes both in scale-score averages for the overall district sample and at each quintile for the time periods 2003 to 2005, 2005 to 2007, and 2003 to 2007. This method yielded 792 statistics.2 To make the district site selections for the subsequent case studies, we implemented the following steps: 2
Six statistics for each of the 11 districts in the three time periods in each subject-grade combination, except for one district (Austin), which has data on only time period 2005 to 2007.
33 Council of the Great City Schools and the American Institutes for Research 52
PIECES OF THE PUZZLE: FACTORS IN THE IMPROVEMENT OF URBAN SCHOOL DISTRICTS ON THE NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS