6.2 Scoring Rubrics

Open-ended and short-response items require written rubrics to evaluate student responses, which should be included in the codebook (box 6.2). Enumerators should have clear scoring rules accompanied by examples showing how illegible or unclear responses must be scored. Anderson and Morgan (2008)

BOX 6.2. Scoring Rubrics

A scoring rubric is a set of scoring guidelines with examples or descriptors of the potential range of student responses that facilitates the reliable scoring of open-ended responses. It is essential for the assessment team to confirm that the scoring guidelines are appropriate for all planned analyses. An example of a scoring rubric from the regional large-scale assessment for Latin America, Laboratorio Latinoamericano de Evaluación de la Calidad de la Educación (LLECE), is shown below. LLECE is reviewed in detail in chapter 9. The LLECE language assessment administered in 2013 includes a writing task that is scored using a rubric with eight indicators. Each indicator has four levels of performance. The indicator Genre is shown below. The task for the student is to write a letter to a friend; this indicator scores whether the authored letter contains all of the formal elements it would be expected to include. As shown, scoring guidelines should describe the characteristics of student responses across the full range of student proficiency to help raters score answers reliably. In addition, the guidelines should be written to ensure a high degree of interrater reliability, which means that multiple raters who read the same response to an open-ended item would score it similarly. Pilot studies offer valuable opportunities to gather evidence on any revisions that should be made to scoring rubrics. The range of student responses received during a pilot can inform revision of these scoring guides and associated examples to ensure that they reflect actual rather than idealized ranges of student performance.

Writing rubric indicator and dimensions in LLECE 2013

Indicator 1b. Genre. This indicator measures the ability to act based on a socially established text model considered appropriate to solving a communication problem. In this case, genre is understood as the prototypical, relatively stable, socially acceptable way in which texts are used in society. The purpose is not only to assess the formal aspects of text genres in terms of knowledge, but also to characterize the use of discourse markers for a given communicative purpose (for example, presence of greeting, orientation to a recipient).

Proficiency levels and descriptors

Level 1. The written text is not a letter, but something that belongs to a different genre (for example, dialogue or short story). Level 2. The written text is a letter including only the body of the letter but without a clear message targeted to the recipient. Level 3. The written text includes the body of the letter in addition to a greeting at the beginning or a goodbye message to the recipient at the end. Level 4. The written text includes a formal greeting at the beginning, the body of the letter, and a goodbye message to the recipient at the end.

Source: Adapted from Flotts et al. 2016.