3.1 How does language of instruction affect test scores?

countries (OECD 2018) or the difference between Singapore, one of the world’s top-ranked countries in math, and Serbia, ranked 46 (OECD 2018).

Differences across and within countries coexist with large within-school differences in learning. A simple variance decomposition reveals that, across SDI countries, on average, about 25 percent of the variation in student test scores comes from between-school variation, which is in line with the Organisation for Economic Co-operation and Development (OECD) Program for International Student Assessment (PISA) average of 33.6 and 37 percent in 2004 and 2012, respectively (OECD 2004, 2013), as well as with the 34 percent estimated variation obtained using the World Bank Global Learning Assessment Database (GLAD) (Azevedo and Goldemberg 2020). Although it is possible for students to have different experiences in the same school because of factors such as teacher biases, the large amount of unexplained variation suggests that factors beyond the school level might also be influencing learning. Unfortunately, the information available on students’ characteristics and home environment is limited in the SDI sample analyzed here.10 More recent SDI surveys are attempting to fill this gap (see chapter 4). The language of instruction may also affect student scores, as detailed in box 3.1.

BOX 3.1 How does language of instruction affect test scores?

Students are normally tested in the official language of instruction, especially for international and national large-scale assessments. This practice works well in monolingual contexts but creates numerous problems in multilingual ones. Students for whom the language of instruction is not their mother tongue (L1) systematically score lower than students for whom it is.

The literature documenting lower scores is voluminous and consistent. Mullis et al. (2017) find that 92 percent of students from the 48 countries tested in the 2011 Progress in International Reading Literacy Study (PIRLS) assessment spoke the language of the test at home. Those not tested in their L1 scored significantly lower, by more than one-third of a standard deviation. Glewwe, Chen, and Katare (2012) find that linguistic-minority students fall behind very early in their school experience and have a hard time catching up. Other authors have documented the strong correlation between being taught in one’s L1 and continuing in primary school (Ramachandran 2012).

Do these persistently lower scores indicate lower achievement, test bias, or both? If there is test bias, is it due to poor translation or to deeper “configural problems,” when the constructs themselves fundamentally differ between languages? Translation problems are relatively easy to spot and fix with enough resources, but configural problems pose a greater challenge.

At the heart of the problem is the fact that tests in a single language cannot distinguish students who answer incorrectly because they truly do not know the construct and those who could answer correctly if the question were asked in their mother tongue. When students are grouped and tested in their L1, analyses can, in theory, estimate the extent of bias between versions of the test. In practice, this way of testing and analyzing results is rare outside of the main languages spoken in Organisation for Economic Co-operation and Development (OECD) countries (Ramachandran 2012). One part of the story is clear: the problem is not the inability of bilingual or multilingual students to achieve as well as or better than monolingual students. Collier and Thomas (2017) find that, when students receive enough high-quality instruction in both

(Continued)