Inves&ga&ng Summer Learning & Middle School Student Achievement WHY SUMMER LEARNING
STUDY DESIGN & METHODOLOGY
Children’s summertime experiences are closely connected to their future success in school and in life. When children participate in camps, travel, and play sports, they are at the same time strengthening their academic achievements, self-confidence, socialemotional development, physical health & well-being. When they don’t participate in summer learning, their skills erode and their progress is stalled.
In 2012, BELL and MDRC completed a randomized control trial (RCT) of BELL’s middle school summer learning model. The report, An Analysis of the Effects of an Academic Summer Program for Middle School Students, aimed to achieve the following goals:
For children in low-income communities, access to quality summer learning opportunities is limited. Schools, communities, nonprofits, and others are working together to expand access to summer learning programs for children whose families cannot afford and/ or access them. This investigation seeks to yield insights into what works so that schools and communities can best support student achievement and deliver the greatest return on investment in summer programs.
WHY MIDDLE SCHOOL Success in grades six to eight is closely tied to children’s on-time graduation from high school & long-term achievement. Yet, patterns of academic performance show that the middle school years are the low point in a student’s academic career. According to one measure of student achievement – test scores - only 35% of 8th graders across the country are proficient in math and 36% are proficient in reading. While schools and nonprofits expand learning time to boost middle school students achievement, there is little evidence on the effectiveness of summer academic programs for middle school students to guide practice, especially programs in which participation is voluntary. Most research on summer learning has been focused on the elementary grades – including evidence from an earlier study that found BELL’s summer model increased reading achievement in grades K-5.
à Build on the knowledge base about how time for learning can impact middle school student achievement. à Determine if and how BELL and its school partners replicated the summer learning model with consistency and quality. à Better understand the context in which these summer programs are implemented. The RCT included several programs delivered in 2012 through new partnerships with schools and school districts. It included 1,032 total study participants who completed grades 5, 6, or 7 in the spring of 2012 and who were advancing to grades 6, 7, or 8. Approximately half of the students were randomly assigned to participate in BELL’s summer learning programs (the treatment group) and half were not (the control group). The full report is available at mdrc.org. BELL’s middle school program serves rising sixth- through eighth-grade students, most of whom are underperforming in school. The goals of the program are to increase student’s literacy and math skills and to enhance their self-confidence, social skills, and growth mindset. Working in partnership with schools, BELL’s model provides students with 6.5 hours of daily programming for approximately five weeks, five days per week. Several types of activities are provided: academic instruction in math and English Language Arts, social and academic enrichment activities, field trips, guest speakers, and community service.
experienceBELL.org
FINDINGS
STUDY LIMITATIONS
à Academic Achievement
The study’s limitations mean that the findings should be considered preliminary and suggestive, and are not generalizable to BELL’s model as it exists today. They should be used to generate hypotheses that could be more definitively tested in a future evaluation, and applied to BELL’s continuous improvement cycle. Limitations include:
Math: The pattern of math impacts suggests that upon returning to school in the fall of 2012, BELL students had stronger math skills than the control group — equivalent to one month’s academic learning compared to the control group. This impact is equivalent to what one would expect from a five-week program. While the study was not sufficiently powered to determine statistical significance, the math findings are promising given patterns of student underperformance in middle school. Reading: In the average study district in fall of 2012, BELL students performed the same as controls on reading achievement tests.
à Program Implementation The BELL Summer program was well implemented relative to the nonprofit’s middle school model and national program quality standards, including those recommended by RAND Corporation and the National Summer Learning Association. The teaching staff was highly qualified: 70% of teachers had a master’s or doctorate degree; 89% have five or more years of teaching experience.
à Student Engagement Students attended the full-day, five-week program at a high rate even though the program was voluntary. In the average study district in Summer 2012, the attendance rate among program participants was 82 percent, above the program’s target of 80 percent. Positive math impacts correlated with increased time engaging in math activities. Surprisingly, the control group reported engaging in reading and writing as frequently as the treatment group.
à Limited Sample Size
The study was designed for a sample size of 3,000 total participants, but only included 1,032. This greatly diminishes the power of the study to detect realistic impacts of a 5-week summer program.
à New Partnerships Participating districts were new partners in 2012. Thus, the findings may not be representative of the effect of summer learning programs that are delivered in schools and school districts with previous experience running such programs.
à Voluntary Participation Participation in the BELL program was voluntary. Therefore, the findings may not be representative of the effect of summer programs for which participation is mandatory for on-time promotion, as many school-based summer learning programs are.
à Post-Test Only No pre-test was administered at the start of the program, so the study was not able to detect growth in BELL’s scholar population during the program or rule out possible summer learning among the non-BELL group.
“To date, there has been very little evidence on the effectiveness of summer academic programs for middle school students, especially programs in which participation is voluntary. Thus, the present study is important not only to BELL but also to leaders of other middle school summer programs.” - Gordon L. Berlin
“I think these teachers are absolutely wonderful. Some of the things that they have been doing in the classroom have been phenomenal…when you go to the classrooms you just see a lot of good things happening, and the kids seem to be excited about what they’re doing.” - School Leader &
President, MDRC
Summer Program Manager
LEARNING & CONTINUOUS IMPROVEMENT The findings from MDRC’s study informed BELL’s ongoing efforts to continuously improve program quality and scholar outcomes. In addition to the RCT conducted in 2012, BELL also collected data from pre- and post-program formative assessments, scholar, teacher, and parent surveys, engagement indicators such as scholar attendance and retention, and quality metrics such as classroom size and teacher engagement. Together, data suggests that while summer learning can have positive impacts on student achievement, reading impact may be a more challenging task that requires relevant, interactive, and hands-on programming. Some of the enhancements made to BELL’s middle school model in the last few years include:
à Summer Staff Training BELL has learned to better harness schools’ talent in customizing training and providing ongoing coaching and support to improve the quality of instruction. By shifting to a “train the trainer” model driven by school and district partners, BELL has increased the alignment between summer programs and school culture & priorities, while creating leadership and professional development opportunities for teachers.
à Curriculum BELL and its school partners transitioned to Common Core-aligned reading and math curricula. The benefits of creating extra time and space in the summer for teachers to plan, collaborate, and experiment has paid dividends in terms of increased comfort and familiarity with CCSS standards and assessments, leading to increased quality of instruction.
Upgrading curriculum has yielded another benefit: it is fully consumable. Students can bring books and other materials home at the end of the program to read and share with siblings and friends.
à Assessment To help teachers use time strategically to teach the skills scholars need most, BELL and its school partners began utilizing computer-adaptive assessments aligned with Common Core State Standards. These assessments can be administered quickly and easily via tablet or laptop computer, minimizing the time required for assessment and improving the quality of insight into scholars’ learning needs.
à Learning Agenda BELL continues to work with MDRC, its Evaluation Advisory Board, its new Middle School Advisory Board, and experts in the field to advance an ongoing learning agenda over the course of the next three years. Emerging questions include: - What is the counterfactual experience of disadvantaged middle school youth that do not participate in BELL Summer? - Which outcomes and what measures best capture the benefits of BELL’s intervention during the summer months? - How can promising contextual adaptations to our model be replicated? - How can a five or six-week program be optimized for the greatest level of engagement among this group and the greatest impact?
FINDINGS IN CONTEXT The pattern of findings from recent research supports the hypothesis that expanding quality learning time for struggling students can increase their academic achievement. A meta-analysis identified elements of successful practice that are consistent with BELL’s Core Essentials, including (1) use of certified teachers; (2) traditional instructional strategies such as clear learning objectives and focused lesson plans; and (3) targeting specific subgroups of students. Initial findings from an RCT that focused on large-scale, voluntary, district-run summer learning programs showed a positive impact on mathematics achievement, but no difference in the reading achievement between treatment and control groups. Math impacts correlated with strong attendance and academic dosage; while instructional quality, teacher grade-level experience, and site orderliness were key to boosting literacy skills.
EVALUATION ADVISORY BOARD BELL’s assessment and evaluation activities are guided, advised, and endorsed by an external, interdisciplinary group, BELL's national Evaluation Advisory Board. The Board oversees the application of principles and standards established in the educational evaluation and assessment fields. Duncan Chaplin, PhD, Senior Researcher Mathematica Policy Research, Inc. Christian A. Friend, PhD, President C.A. Friend Consulting Leslie Goodyear, PhD, Senior Research Associate Educational Development Center, Inc.
ABOUT BELL BELL is one of the nation’s leading nonprofit providers of quality expanded learning programs for children in grades K-8. Its mission is to transform the academic achievement, self-confidence and life trajectories of children living in lowincome, urban communities. BELL serves more than 12,000 students in public schools across the country through its summer and afterschool programs. This evaluation is BELL’s second randomized controlled trial. Earlier, the Urban Institute completed a two-year study of the BELL Summer program’s impact on elementary school students’ reading achievement and parental involvement. It found statistically significant evidence that BELL’s model improves the reading skills of low-performing children, as well as positive impacts on the degree to which parents encouraged their children to read and the degree to which they read with their children.
Priscilla Little, Research & Evaluation Consultant (Chair) Beth Miller, PhD, Director of Research & Evaluation, Nellie Mae Education Foundation Dr. Lauren Smith, MD, MPH Boston University School of Medicine
experienceBELL.org