4 minute read

Methodological Challenges

Next Article
References

References

quality. Internationally, several scholars assert the need for subject specificity when analyzing the qualities of classroom teaching and learning. Hill and Grossman (2013) argue that if classroom analyses are to achieve the goal of supporting teachers in improving their teaching, these frameworks must be subject-specific and involve content expertise. This will enable teachers to provide information that is relevant for situation-specific teaching objectives, regardless of whether this is student engagement, group problem solving or algebra learning. Blöemeke et al. (2015) show how a combination of generic factors and subject-specific factors (in their case, regarding mathematics) is required for producing valid knowledge on how different teaching factors contribute to student learning. Klette et al., (2017) use the PLATO framework (targeted for language subjects) to capture both subject-specific and generic aspects when analyzing the features of Norwegian instruction in languages and mathematics. The MET study (Kane et al., 2012) argues that there were no big differences across the different frameworks deployed when trying to measure teaching quality in 3000 US classrooms using five different frameworks – three subject specific and two generic. There is probably not one right solution to the question of whether to use generic or subject-specific observation frameworks. Instead, the answer to this question will depend on the purpose of the study, be it strengthening student engagement and participation, classroom discussion or content-specific teaching.

METHODOLOGICAL CHALLENGES

Different methodological traditions contribute to the aggregated available knowledge on effective teaching and high instructional quality. While intervention studies have provided a growing body of empirical evidence of instructional strategies that are effective with regard to student learning, observation studies have often found that such practices (for example, the explicit instruction of learning strategies or the scaffolding of performance through effective feedback), are scarce in today’s classrooms (Elbers et al., 2008; Klette et al., 2017; Magnusson et al., 2019). This means that even though a significant number of studies define what highquality instruction would look like, few studies are able to locate these practices in naturally occurring, non-intervention classroom studies.

Although research studies of teaching quality take stock of measures of student achievement over time in order to identify indicators of quality, few studies have yet investigated the extent to which such indicators might have differential effects on students with regard to achievement levels, gender, student motivation, or language proficiency. Such effects have been identified in intervention studies target-

ing well-defined ingredients of teaching such as questioning strategies, self-regulation, or strategic group work (Schünemann et al., 2013; Taboada et al., 2012), but rarely in relation to broader categories of teaching qualities. We therefore need studies that specifically examine how different indicators of classroom quality affect different learners, as students of various backgrounds and capabilities may benefit from different kinds of instruction and need different forms of support.

A major concern when advocating additional teaching quality research is the acute awareness of its methodological challenges. Measuring teacher performance by global observation systems on high-inference variables prompts a range of reliability issues. Indeed, the requirements of reliability and validity may depend to a certain extent on the proposed use of the research findings. But, given the farreaching social, cultural, political, and professional implications of scientifically defining a phenomenon such as teaching quality, it is necessary that the measures we use are accurate, reliable, and relevant. This is certainly one of the reasons why instructional quality and assessments of individual teachers are so contested. No measurement tool is likely to capture the complex interaction between teachers, students, and content that teaching entails, and, as Archer et al. (2015) emphasize:

The list of things teachers do that may have a significant impact on student learning is extensive. Ensuring accuracy in the face of such complexity poses a major challenge to the design and implementation of tools for measuring effective teaching. (p. 1)

And, as pointed out by Raudenbush & Jean (2015), many indicators of classroom quality are highly correlated, making it challenging to make informed decisions about what kinds of data provide the most useful information, and how predictive of instructional quality different data can be. Further, the cumulative nature of education makes it hard to measure accurately, as we often lack longitudinal data and must rely solely on current input measures, which often leads to “analytical errors and biased estimates of specific inputs” (Hanushek, 2020, p. 165).

A distinct kind of measurement issue, one that is certainly less thoroughly examined, is the student outcome against which teaching quality indicators are typically validated. In large-scale observations, student learning over time is commonly assessed either by standardized tests, or, if the teaching observed is restricted to a specific domain, by tests specifically designed to target those domains of knowledge (Fauth et al., 2014; OECD, 2010; 2016). Standardized tests, of course, refer to a range of different measures, including state exams, national tests, various commercial assessment products etc. However, in the context of teaching quality studies, they generally involve the measurement of year-to-year progress in either reading, math-

This article is from: