4.6 Advantages and risks of participating in an international assessment

TABLE 4.6 Advantages and risks of participating in an international assessment

ADVANTAGE

Adherence to high technical standards of assessment design, instrumentation, sampling, administration, analysis, and reporting Development of indigenous capacity to meet international standards of assessment practice High degree of transparency in dissemination of the results and political gains if performance is found to be relatively good compared with that of peers Positive effects of driving up performance from diagnostic application of results, exposing the educational system to external scrutiny, and tracking the impact of certain interventions and reforms over time

Opportunity to adapt (“version”) survey instruments that meet international standards to the national language and context

Source: UK DFID 2009. RISK

Criticism of the cost of participation, particularly in view of the need to commit to successive rounds if the initial investment is to be worthwhile

Disaffection with the international exercise if its assessment framework is of limited relevance

Unfavorable comparison of the results of neighbors and peers— with the attendant political consequences

Unreasonable expectation that international assessment exercises will deliver the accountability outcomes that national census assessment exercises provide

Failure to fully adapt survey instruments to the national context. “Versioning” goes beyond translation to ensure that literacy texts (in particular) are suited to children’s educational experience and sociolinguistic background

not a means of screening individual students, modified versions of the test can be used as a screening tool to identify individual students with needs (RTi 2016). Like national and international assessments, these assessments are designed to provide policy makers with information on the quality of early-grade instruction, with a view toward addressing system needs. eGRA tests for letter recognition, phonemic awareness, reading simple words, and listening comprehension, whereas eGMA tests number recognition, comparisons, and ordering sets of objects. eGRA has been used in about 70 countries to provide national- or system-level diagnostic information about children’s early learning, as well as to support classroom-based assessment and program evaluations (uk DFiD 2009).

Sri Lanka would benefit from participating in international assessments. Most high- and upper-middle-income countries participate in a range of international assessments, such as the Progress in international Reading Literacy Study (PiRLS), TiMSS, and PiSA. Developing countries are also stepping up efforts to participate in these tests because they help policy makers benchmark learning outcomes against international standards. Many countries have used international assessments to accelerate educational reforms and ultimately improve educational outcomes. For example, TiMSS results in Japan spurred parliamentary discussions about planned changes in education policy. in new Zealand, a task force on mathematics and science was established in response to the TiMSS results. in Canada, the results led to the development of instructional materials based on an analysis of the common misconceptions and errors of students in their response to TiMSS tasks. in Canada and Singapore, participation accelerated changes in the revision of curricula (Greaney and kellaghan 2008). Box 4.11 also provides the example of Poland, a country that successfully utilized PiSA to evaluate the efficacy of educational reforms over time. Sri Lanka would benefit from participation in international assessments, particularly as it undertakes reforms to improve the quality of its educational system. Sri Lanka’s participation would help place its students’ achievement in the global context and may also prompt the reforms needed to spur further improvements in learning outcomes.

TABLE 4.7 Comparison of PISA and TIMSS

Purposes

TIMSS 2003

• To provide comparative evidence on the extent to which students have mastered official school curriculum content in mathematics and science, which is common across a range of countries • To monitor changes in achievement levels over time • To monitor students’ attitudes toward mathematics and science

• To examine the relationship between a range of instructional and school factors and achievement (reading is covered in a separate PIRLS assessment)

Framework

Developed by content experts Target population Grades 4 and 8 Curriculum appropriateness

Designed to assess official curriculum organized around recognized curriculum areas common to participating countries

Item content differences (mathematics, grade 8) Grade 8, item distribution: • Numbers, 30% • Algebra, 25% • Data, 15% • Geometry, 15% • Measurement, 15%

PISA 2003

• To provide comparative evidence on the "yield" of the school system in the principal industrial countries and to assess whether students can apply their knowledge and competencies in reading, mathematics, and science to real-world situations

• To monitor changes in achievement levels and equity of learning outcomes over time • To monitor student approaches to learning and attitudes to mathematics, science, and reading • To provide a database for policy development Developed by content experts 15-year-olds Designed to cover knowledge acquired both in and out of school, defined in terms of overarching ideas and competencies applied to personal, educational, occupational, public, and scientific situations

Mathematics, overarching ideas:

• Quantity • Space and shape • Change and relationships • Uncertainty Item distribution:

• Numbers, 31.8% • Geometry, 21.2% • Statistics, 21.2% • Functions, 10.6% • Discrete math, 5.9% • Probability, 5.9% • Algebra, 3.5%

Cognitive processes Grade 8: • Solving routine problems, 40% • Using concepts, 20% • Knowing facts and procedures, 15% • Reasoning, 25% Item distribution:

• Connection, 47% • Reproduction, 31% • Reflection, 22%

Item types (mathematics) About two-thirds are multiple-choice items, and the remainder are constructed response or open-ended items About one-third are multiple-choice items, with the remainder generally being closed (one possible correct response) or open (more than one possible correct response) constructed-response items

Frequency

Geographical coverage Every four years—equal emphasis on mathematics and science in each cycle

48 countries: 20 high-income, 26 middle-income, and 2 low-income Every three years—extensive coverage of one domain (subject) every nine years (reading in 2000, mathematics in 2003, and science in 2006), plus less extensive coverage of the other two every three years 30 OECD countries as well as 11 other countries

Analysis Four benchmark levels and a mean score, which are based on all participating countries Seven mathematics proficiency levels and a mean score, which are based on OECD countries

Source: Greaney and Kellaghan 2008. Note: OECD = Organisation for Economic Co-operation and Development; PIRLS = Progress in International Reading Literacy Study; PISA = Program for International Student Assessment; TIMSS = Trends in International Mathematics and Science Study.