What Are the Critical Steps in the Analysis of Large-Scale Assessment Data?
TABLE 6.1. Percentage of Correct Answers on Ghana’s National Large-Scale Assessment According to Sex Assessment
Male (percent)
Female (percent)
Mathematics grade 4
41.9
41.5
Mathematics grade 6
44.9
42.8
English grade 4
49.8
52.0
English grade 6
47.6
48.1
Source: Ministry of Education, Ghana Education Service, and National Education Assessment Unit 2016. Note: Results are presented as percentage of items correctly answered in each assessment.
NORM-REFERENCED ASSESSMENT REPORTING
Norm-referenced reporting involves describing student performance in terms of the characteristics of the statistical distribution of student test scores. The results for each student are compared with the average or typical proficiency of all tested students. Typical student performance might be captured using the mean or arithmetic average, the median or midpoint of the score distribution, or the mode or most commonly achieved score. Most national large-scale assessments use the mean or arithmetic average. In addition to average levels of achievement, score reporting should include information about variation in student test scores, such as the possible and observed upper and lower range of scores. Including measures of variability in student performance provides stakeholders with a more complete picture of student achievement and prevents overinterpretation of differences in average performance (Shiel and Cartwright 2015). Figure 6.1 summarizes the results for the language and communication and mathematics sections of Chile’s national large-scale assessment. The two bars on the left side of the graph represent the overall national average in these two subjects. Regional average scores are represented in the subsequent bars, with the number at the top of each bar being the region’s average score. The regional scores in red are significantly different from the national average scores. Norm-referenced reporting of test results can support the comparison of subgroups in a sample of student performance over time, although this form of reporting does not map directly to test content. If stakeholders are interested in understanding the students’ level of mastery of a domain or which skills they have acquired, they can better do so using a standards-referenced approach. STANDARDS-REFERENCED REPORTING
National large-scale assessments are increasingly using a standards-referenced approach to report results. Standards-referenced reporting involves describing student performance in terms of what they know and can do in relation to domain-specific achievement standards. Instead of numeric scores, results are reported using descriptions of the tasks that students are able to perform, such as reading “at a basic level of comprehension” or performing “advanced mathematical operations.” Defining and setting standards is a complex task that requires the involvement of curriculum experts and statistical analysts; readers
75
