Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

Date: April 2019 Team: Academics, NewGlobe Acknowledgments: Special thanks to:

Leaders in Learning. https://newglobe.education/

Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

2

Contents 1.

2.

3.

4.

5.

6.

7.

Abstract Introduction Review of the Literature Methods Results Discussion References

Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

Page 04 Page 05 Page 06 Page 08 Page 12 Page 14 Page 15

3

1. Abstract Within-class ability grouping sorts pupils into homogeneous learning groups within a single classroom.

This system of grouping presents an alternative to tech-enabled or school-wide efforts to promote personalised learning. Some literature suggests that within-class ability grouping results in positive effects, even compared to across-grade ability grouping. This study explores the impact of withinclass ability grouping among Class 4 pupils attending 202 schools in Kenya. Pupils in the control group were grouped heterogeneously and completed a universal problem set each day. Pupils in the treatment group were grouped homogeneously and completed levelled problems aligned to their ability levels. Because lesson guides are digitally synced to individual teacher computers, this study offers a unique analysis of the impact of an instructional intervention across comparable instructional settings within a single school network, Bridge International Academies. At the midline of the study, we find no positive effects of homogeneous ability grouping on content knowledge. Some early evidence suggests a positive impact among the lowest-performing pupils. Qualitative field data identifies several challenging areas complicating the design and implementation of homogeneous ability grouping and the accompanying instructional materials.

Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

4

2. Introduction Learning technologies have played an increasingly central role in schools and classrooms.

A growing portfolio of computer programmes and learning apps have enabled teachers to implement personalised learning approaches in the classroom. These programmes typically offer an adaptable, responsive learning experience in which pupils complete learning activities and receive questions or problems aligned to their literacy or numeracy levels. Yet the use of these personalised learning platforms requires access devices such as laptops, tablets, or mobile phones. While technologyenabled classrooms may be a reality in developed countries, the vast majority of schools in developing countries still lack access to these devices that make such personalised learning possible. Schools in developing countries often lack the funding to invest in technology. This challenge is compounded by lack of internet connectivity, electricity for charging, or secure storage spaces for devices. It is imperative to explore low-tech, scalable opportunities for personalised learning. This will not only offer a blueprint for schools in low-resource settings to adopt personalised learning methodologies; it will also advance our collective understanding of the basic principles of personalised learning and how this approach can accelerate learning gains in all classrooms.

A growing body of literature supports across-grade ability-grouping programmes such as Teaching at the Right Level. These programmes group pupils according to literacy or numeracy levels. Pupils then attend lessons alongside pupils with a similar learning level. More importantly, pupils receive instruction narrowly aligned to their learning levels. Programmes such as Teaching at the Right Level have achieved remarkable results in both developed and developing contexts. But they are Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

5

challenging to implement. Across-grade ability grouping programmes require strong school leadership, reliable assessment data that can be used to group pupils, and a group of teachers across multiple grades committed to school-based reform. An alternative, albeit less robust, stream in the literature suggests that within-class ability grouping can produce similar learning gains. Within-class ability grouping forms distinct groups with similar ability levels within the same classroom and assigns practice opportunities aligned to the ability levels of each group. Currently, very little randomised evidence exists to support within-class ability grouping. But this approach offers a more accessible, classroom-based approach to low-tech personalised learning or school-wide sorting.

The purpose of this study is to analyse the impact of homogeneous ability grouping on termly learning gains compared with heterogeneous grouping. This study is being conducted among schools operated by Bridge International Academies in Kenya. The study focuses on pupils enrolled in Class 4 Mathematics. Pupils in the treatment group are assigned to a lower-performing group or a higher-performing group within the same class. Pupils in each ability group then complete problems that are narrowly aligned to their numeracy levels. Pupils in the control group remained grouped heterogeneously and continue to complete problems aligned to the median learning level in the classroom. This research explores a vital alternative to tech-enabled personalised learning and school-wide ability grouping. It also examines the impact of an approach that is broadly accessible to teachers in all settings and of all ability levels. The research supporting personalised learning is clear. But it is imperative that we in the educational community seek to democratise pathways to personalised learning through innovative, scalable, accessible solutions.A growing body of literature supports across-grade ability-grouping programmes such as Teaching at the Right Level. These programmes group pupils according to literacy or numeracy levels. Pupils then attend lessons alongside pupils with a similar learning level. More importantly, pupils receive instruction narrowly aligned to their learning levels. Programmes such as Teaching at the Right Level have achieved remarkable results in both developed and developing contexts. But they are challenging to implement.

Across-grade ability grouping programmes require strong school leadership, reliable assessment data that can be used to group pupils, and a group of teachers across multiple grades committed to school-based reform. An alternative, albeit less robust, stream in the literature suggests that within-class ability grouping can produce similar learning gains. Within-class ability grouping forms distinct groups with similar ability levels within the same classroom and assigns practice opportunities aligned to the ability levels of each group. Currently, very little randomised evidence exists to support within-class ability grouping. But this approach offers a more accessible, classroom-based approach to low-tech personalised learning or school-wide sorting.

3. Review of the Literature Within-class, across-class, and across-grade tracking While some literature suggests that within-class ability grouping presents a viable alternative to across-grade ability grouping, randomised research is somewhat limited.

Slavin (1987) conducted a meta-analysis of 14 studies. He included only studies of randomised research exploring the impact of ability-grouping in elementary school settings in the United States and the United Kingdom. The meta-analysis sought to synthesise and analyse the evidence regarding the impact of within-grade and across-grade ability grouping on pupil achievement.

He found that within-class ability grouping in reading lessons has a median effect size of 0.34.

Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

6

Overall, ability grouping was most successful when it was used for 1-2 subjects rather than for an entire course of studies.

Building on Slavin’s earlier analysis, Kulik and Kulik (1992) conducted a meta-analysis of 14 studies.

These studies included randomised and correlational research exploring the effects of ability-grouping in elementary settings. Like Slavin (1987), they analysed the impact of within-class and across-grade ability grouping on learning outcomes. The authors found that within-class ability grouping had a small but significant effect size, raising achievement levels by 0.25 standard deviations. Nine out of eleven studies that included within-grade ability grouping demonstrated significant effects. The average effect size was 0.30 for higher-ability pupils, 0.18 for middle-ability pupils, and 0.16 for lower-ability pupils.

Two additional studies used a small-scale, randomised design to investigate the impact of withingrade ability grouping. Adodo and Adbayewa (2011) used a within-class ability grouping approach in a study that included 60 pupils attending junior secondary school in Nigeria. In the control group, pupils were purposively selected, assessed on numerical and word knowledge, and assigned to heterogeneous learning groups in Science. Inversely, in the treatment group, pupils were selected, assessed, and assigned to homogenous learning groups in Science. In the study, they explored the learning gains associated with pupils who were grouped in homogeneous and heterogeneous ability groups. The authors found that pupils taught in homogenous ability groups learned significantly more than pupils in heterogeneous ability groups.

Collins and Gan (2013) conducted analysis of ability-grouping using a sorting index measuring the homogeneity according to ability levels. The model also considered incoming ability, classroom, and pupil characteristics. The study included 9235 4th grade pupils enrolled in 135 schools in Dallas Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

7

Independent School District during 2003-2004. The authors analysed the net effect for lower performing students grouped according to ability. The study found that sorting homogeneously by previous performance significantly improved students’ math and reading scores. The effect was present for all students across the score distribution, demonstrating benefit for high performing and low performing students.

Some counter-evidence does call into question the impact and logic of ability grouping. Naomi (2009) found that schools that use ability grouping are likely to be public, low-performing, low-SES, highminority schools. Kaya (2015) formally compared instruction in homogenous groups to instruction in heterogeneous groups and found no significant difference between the two classrooms.

While there is certainly literature to suggest the potential impact of within-grade ability grouping, several gaps in the literature remain. First, the vast majority of the randomised research on withinclass ability grouping has been conducted in western, developed contexts. Second, no large-scale, randomised evaluation of within-class ability grouping has been conducted among schools in a comparable setting. Finally, the approaches to ability-grouping were fundamentally structural, not instructional. Teachers created homogenous groups within a classroom. But little attention was paid to the type of diversified instruction that accompanied the creation of homogenous learning groups.

This study addresses each of these three gaps. First, it takes place in Kenya, a developing context with a drastically different educational landscape from the United States or United Kingdom. Second, it includes 202 schools randomly assigned to homogeneous or heterogeneous learning groups. Third, it uses a clear, scalable approach to the design of instructional materials that support heterogeneous or homogeneous learning groups at the classroom level.

4. Methods Participants This study was conducted among schools in Kenya operated by Bridge International Academies (Bridge). Bridge was founded in 2009 and since then, has educated over 500,000 children at more than 1000 schools across Africa and Asia. Initially, Bridge operated community-based private schools.

More recently, Bridge has begun to partner with governments to provide technical support and expertise in instructional design, teacher training, and programme evaluation in government schools.

In both private and public schools, Bridge provides teacher training, teacher technology, lesson guides aligned to the national syllabus, and data-driven professional development.

Bridge operates 297 community-based private schools in Kenya. This study include 202 of those schools. Those 202 schools were purposively selected based on stream availability. The selected 202 schools offer Class 1 through Class 8. When a school offers fewer than 8 classes, it is digitally programmed differently and thus excluded from Bridge’s testing platform. All 202 available schools were selected to maximise statistical power in the pilot. Specifically, the study focused on Class 4 pupils. 3559 Class 4 pupils attended those 202 schools.

In this study, the unit of randomisation was the academy. Approximately half of schools were allocated to the treatment group, while approximately half were assigned to the control group.

Stratified randomisation was used to increase power and ensure balance between the control and treatment groups. Strata were constructed using baseline pupil performance on internal assessments Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

8

(historical midterm and endterm exams). Randomisation was conducted by external partners from Harvard University.

Procedure All teachers at Bridge are equipped with a personal teacher tablet. Daily lesson guides are uploaded via mobile network to each school leader’s smartphone. Each teacher syncs their tablet with the smartphone each morning, which allows teachers direct access to each day’s lessons.

The lesson guides themselves are designed by Bridge instructional design experts based in Lagos, Kenya, and the United States. Each lesson is aligned to the national curriculum and maps back from daily objectives.

Because digital lessons are synced to individual teacher tablets, Bridge is able to design alternative streams of lesson guides using a new instructional approach. Original, incumbent lesson guides can be programmed to sync to the tablets of control group teachers, while treatment group lessons using the new intervention can be programmed to sync to the tablets of treatment group teachers.

This ensures that lessons are identical except for the instructional approach being evaluated.

In addition, all other aspects of a teacher’s day (timetable, content and structure of other lessons, training and professional development, etc.) are held constant. This allows for a unique opportunity to isolate the impact of a very specific approach to instructional design on learning outcomes.

The control group lessons are comprised of Bridge’s incumbent approach to Maths instruction.

In the control group, pupils are grouped heterogeneously in each classroom. Each lesson begins with a race, in which pupils solve simple numeracy equations for 1 minute. Following the race, the teacher demonstrates two problems on the board for 10 minutes. Following the demonstrations, pupils complete independent practice for 15 minutes. In the independent practice, the teacher assigns a single group of problems to all pupils in the classroom. Problems are aligned to the ability-level of the median pupil in the classroom. Typically, the problem set includes 10-12 problems. During this time, the teacher provides individualised feedback to pupils working on problem sets. The lesson concludes with a 5 minute revision section where the teacher reviews how to solve a question that was challenging for pupils.

The treatment group lessons are identical to control group lessons in many key ways. In the treatment lessons, the teacher leads the class in a race, demonstrates two problems on the board, assigns a problem set for independent practice, and revises a challenging problem. The problems in the race and the problems demonstrated on the board are identical to the problems in the control group. The teacher instructions to deliver the lesson are identical in the control and treatment group. This ensures strong comparability between the control and treatment group.

There are two fundamental differences between the control and treatment group. First, pupils are grouped into homogeneous ability groups at the outset of the independent practice session. The teacher uses her or his specific knowledge of pupil ability levels to group pupils into two groups: lower-performing (Group 1) and higher-performing (Group 2). The teacher assigns pupils to their groups at the beginning of the study, but retains the opportunity to adjust pupil group assignments at any time over the course of the three-term study. This could be daily, after each unit, or termly at the discretion of the teacher. While leaving grouping at the discretion of the teacher creates a vulnerability in terms of accurate groupings, it does allow for increased flexibility compared with an approach that sorts pupils a single time based on baseline assessment results. Second, pupils Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

9

received levelled problem sets aligned to the learning levels of their homogeneous ability-group.

The problems assigned to the lower-performing group are aligned to the ability level of the median pupil in the lower-performing group. The problems assigned to the higher-performing group are aligned to the ability level of the median pupil in the higher-performing group. The teacher provides individualised feedback to pupils in both groups, and reviews the answers separately at the end of the independent practice session.

Baseline learning was assessed using an internally-designed pretest. Pretests were administered at the start of each term. Both control and treatment pupils took the same assessment. Each pretest included 30 questions and previews the upcoming content of the term. All questions were in equation format. The response-format was open-response solutions, where pupils solve the equations and write their answers in free-form. Pupils wrote their answers on a blank sheet of paper torn from their exercise books. Pupils received 70 minutes to complete the assessment. Pretests were delivered to teachers digitally by uploading them to each teacher’s teacher computer. Answer keys are also digitally synced to teacher computers. On the following day, teachers receive time to mark answers.

After marking all answers, teachers enter each pupil’s score into their teacher tablet. After syncing with the school leader’s smartphone, these scores are automatically uploaded into a report on Bridge’s reports server, where they are accessible for download.

Posttests were administered at the end of each term. Posttests are designed as mirror-assessments.

The order of questions from the pretest is jumbled. The numbers used in each problem are also adjusted. But the content-focus of each question is the same. This ensures that test familiarity does not inflate posttest scores, but also ensures that differences between pretest and posttest represents real growth on termly content. The process for administering posttests is exactly the same as for administering pretests.

Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

10

After each assessment, Bridge staff conducted SMS and call campaigns to encourage all teachers to enter scores into their teacher tablet. Approximately 85% of teachers entered scores into their teacher tablets. In some instances, teachers were unable to enter scores into their teacher tablet due to software or technology issues. In other instances, teachers reported entering scores into their teacher tablet, but the scores did not appear in the report. In these cases, school leaders were instructed to photograph the physical score sheet and send to the central support office.

These scores were manually entered into the reports server by Bridge staff. At the end of the study, external auditors will evaluate the extent to which the scores on these photographs match the scores entered into the report. This process will ensure that scores were entered accurately and honestly

by Bridge staff.

In this paper, I report the outcomes of the first term of the study. The first term included 38 instructional days. 4 of these days were used to administer and mark pretests and posttests.

The study, however, is ongoing and will continue through the end of term 2, 2019. During the final 2 terms, the study follows pupils who have matriculated into Class 5. For each additional term, pupils will receive a consistent approach to independent practice (heterogeneous vs. homogeneous ability grouping). For each term, outcomes will be measured using a pretest/posttest assessment design.

Analytic strategy We estimate the impact of levelled problem sets using an ordinary least squares regression, controlling for baseline assessment results, teacher quality (as measured by lesson completion %), and pupil attendance. Lesson completion percentage is measured as each teacher’s delivery of at least 80% of the lesson guide on the day during which the guide was assigned. Effective teachers in the Bridge system consistently deliver each lesson and are able to deliver the lesson within the time allocated for the lesson. Each teacher receives an overall lesson completion percentage over the course of the experiment duration. Pupil attendance is calculated by averaging the overall attendance for a classroom over time.

Our model estimates endline assessment scores for student i in school j at time period t as a function of baseline assessment scores (Yi,j,t-1), lesson completion (Completion,j), attendance rates (Attendance,j), and an error term (εi,j).

Yi,j,t = 𝛽0 + 𝛽1 Treatmentj,t + 𝛽2 Yi,j,t-1 + 𝛽3 Completioni,j + 𝛽4 Attendancei,j + εi,j

Our coefficient of interest is 𝛽1, which provides an estimate for the impact of the levelled problem sets on student performance on termly content knowledge. All estimates cluster standard errors at the academy level.

Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

11

5. Results Impact on pupil achievement A restricted sample was used in this analysis. Data entry was relatively low for the term 3 pretests, which limited the total restricted sample. No significant differences were found between baseline characteristics, including baseline assessments, lesson completion, or pupil attendance.

Below, I present a table of descriptive statistics comparing pretest and posttest outcomes (in terms of raw scores on a 30-item assessment) among control group and treatment group pupils. There was a significant difference among pretest scores between control group and treatment group pupils.

The difference, 0.90, was significant at the 0.01 level. There were no significant differences among posttest scores between control and treatment group pupils.

Because treatment group pupils had significantly lower pretest scores at the baseline, I conduct a simple gains score analysis. Below, I present the results (in terms of raw gains scores between pretest and posttest, each a 30-item assessment). While treatment group pupils made larger gains between baseline and endline (0.44), those gains were not significant.

I present the results of the ordinary least squares regression below. In each stage of the model, treatment assignment does not predict a significant amount of the variance in posttest scores, controlling for pretest scores, lesson completion, and pupil attendance.

Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

12

The limited duration of the intervention should moderate any initial judgements about the efficacy of levelled problem sets. Yet the early data certainly suggests that homogeneous group had no effect on performance. Interestingly, we do find that the lowest performing pupils in the treatment group (scoring 0-4 on the baseline assessment) made significantly larger gains on the endline assessment (p<0.05).

Non-experimental evidence from lesson observations In order to monitor the fidelity of implementation of the homogeneous grouping and assignment of levelled problem sets, Bridge relies on academic field officers. Academic field officers regularly observe lessons at different academies within a particular region. These lesson observations capture granular details about the quality of the lesson, such as lesson quality, the extent to which pupils achieved the academic objective of the lesson, the precision of section-by-section timing, the clarity and accuracy of the lesson guide, and the ratio of independent practice to total classroom time. An academic field officer observed 14 different control and treatment group lessons throughout the term.

Qualitative lesson observations indicate that the quality of teachers, timing difference, number of errors, completion rate, and ratio of independent practice to total time did not significantly differ. But lesson observations show that there were more confusing lines encountered among teachers of treatment group lessons compared with control group lessons. In addition, pupils achieved a lower academic objective rating in treatment group lessons compared with control group lessons.

Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

13

Lesson observations provide an invaluable glimpse into the challenging elements of homogeneous ability grouping. Reports suggest that teachers struggled to subjectively group pupils according to ability. In some cases, pupils were incorrectly or arbitrarily grouped. In other cases, teachers struggled to deliver the diversified instruction targeted to each individual ability group.

6. Discussion The purpose of the research is to explore the impact of levelled problem sets on pupil achievement compared with single problem sets. The underlying question is whether homogeneous ability-grouping and the corresponding instructional materials can produce significant gains, particularly among pupils at the lower and upper end of the distribution. We found no significant gains among pupils completing levelled problem sets. There are some early indicators that suggest that the lowest-performing pupils gained more when assigned levelled problem sets compared with single problem sets.

These results should be considered cautiously for several reasons. First, this paper presents midline data for an ongoing study. There is a reasonable likelihood that pupils simply have not received enough time and exposure to derive any benefits from homogeneous ability grouping. Teachers, as well, may require additional time to get accustomed to the new lesson materials. Second, qualitative lesson observations highlight implementation challenges related to the grouping of pupils.

In some cases, teachers grouped pupils incorrectly. While this approach was piloted successfully among several teachers, it proved more challenging when being administered at scale. Broadly, the instruction to subjectively group pupils could have led to inaccurate groupings of pupils.

Future interventions should provide more clarity on grouping procedures according to baseline assessment scores.

Other possible explanations exist to explain the null effects. It was assumed that the original problem set was aligned to the median pupil. But if this problem set was slightly harder or easier than the median ability level, the levelled problem sets may also have been incorrectly aligned to the formulated ability groups.

The possible explanations illustrate the challenges of homogeneous ability grouping. First, without a clear cutoff score precisely aligned to the rigour of the corresponding instructional materials, there is a risk of pupils learning at the wrong level, even in homogeneous learning groups. Second, subjectively assigning pupils to their groups without mapping back from concrete assessment data could lead to incorrect grouping of pupils. Third, supporting teachers to deliver diversified instruction leads to a less clear, more challenging lesson guide. It is crucial to ensure that achievement gains outweigh any sacrifice in lesson clarity.

It is too early to tell how this research will contribute to the literature on homogeneous grouping.

In addition, Bridge will await additional results before making a decision to scale, retest, or abandon the idea of homogeneous ability grouping. But early data highlights clear and concrete ways in which within-class ability grouping can be executed more effectively at scale across a large network of schools. Regardless of the outcomes, future research should consider the impact of within-class ability grouping for younger or older pupils. Future research might also explore the differential impact of ability grouping across different subjects.

Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

14

7. References Adodo, S. O. and J. O. Agbayewa (2011). “Effect of Homogeneous and Heterogeneous Ability Grouping Class Teaching on Student’s Interest, Attitude and Achievement in Integrated Science.” International Journal of Psychology and Counseling 3 (3): 48-54.

Collins, Courtney A. and Li Gan (2013). Does Sorting Students Improve Scores? An Analysis of Class Composition. Cambridge: National Bureau of Economic Research.

Kuli, James A. and Chen-Lin C. Kulik (1992). “Meta-analytic Findings on Grouping Programs.” Gifted Child Quarterly 36 (2): 73-77.

Slavin, Robert E. (1987). “Ability Grouping and Student Achievement in Elementary Schools: A Best-Evidence Synthesis.” Review of Educational Research 57 (3): 293-336.

Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

15

Leaders in Learning https://newglobe.education/

Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

Leveled problem sets: A light-touch approach to addressing heterogeneity within the classroom

16