Page 1

Best Practices in Out of School Time (OST) for Middle School Aged Children: Literature Review Matrix Compiled by Laura Gogia, January 2013 Author(s), Year Bodilly, McCombs, Orr, Scherer, Constant, Gershwin (2010)

Article Title and Journal Hours of Opportunity, Volume 1 www.wallacefoundation.org

Participants and Research Design Wallace Foundationfunded programs in Providence, NYC, Boston, Washington DC, and Chicago Qualitative, replicated case-study design involving interviews, business plans, other documents Program objectives being reviewed: 1. Increase access and participation 2. Improve program quality 3. Develop information systems to aid in decision making 4. Plan for financial stability

Major Findings

Other notes

1. Early planning (needs National best assessment, concerns and practices desires of parents and children, barriers to participation like safety or transportation) 2. MI system (enrollment, attendance, demographics) helps with future planning and obtaining funding 3. Adequate recruitment and advertising techniques, especially: a. program locators (web-or paper-based frequently distributed through schools) were directly linked to enrollment rates b. program site coordinators who worked with schools to encourage enrollment and regular attendance 4. City-wide collaboration— key components identified include: a. Mayoral support in restructuring agencies, increasing funding, demanding


progress reports, appointing a special advisor with power and authority to ensure interagency cooperation like establishing memos of understanding (MOUs) to document sharing of resources and interpretation of policy Yohalem, Devaney, Smith and WilsonAhlstrom (2012)

Donner (2012)

BUILDING CITYWIDE SYSTEMS FOR QUALITY: A Guide and Case Studies for Afterschool Leaders http://www.wallacefoundation.org /knowledge-center/after-school/ coordinating-after-schoolresources/ Documents/Building-Citywide-Systems-for-Quality-AGuide-and-Case-Studies-for-Afterschool-Leaders.pdf

Making the Connections: A Report on the First National Survey of Out-of-School Time Intermediary Organizations http://www.wallacefoundation.org/knowledge-

Grounded theory approach to create theoretical framework of QIS, then case studies and local expert opinion of ASPs in Atlanta, Austin, Chicago, NYC, Palm Beach Fl, and Hampden Co MA to adjust framework. Framework is couched in a continuous improvement approach— organizations regularly measure against a standard, then develop, implement improvement plans and then begin cycle again Online survey of 212 OST nonprofit coordinating orgs

Recommendations for citywide collaboration: 1. Shared definitions of quality and standards amongst stakeholders. 2. Clear leader organization 3. Engaged stakeholders 4. Continuous improvement model including: standards for high quality performance, assessment tool, and aligned improvement supports (planning, coaching, training) 5. Management Information System (to collect data on programs—attendance, etc.) 6. Clear guidelines and incentives for participation 7. Adequate resources Defines OST intermediary: 1. Connects public and private funders with direct service providers 2. Provides technical

National best practices

National best practices and indicators for system design


center/after-school/coordinating-after-schoolresources/Documents/Making-the-Connections-ReportFirst-National-Survey-of-OST.pdf

assistance and other supports to direct service providers OST intermediaries need 3 years to show a positive impact in areas of building data systems, increasing investment in quality standards and tools, increasing kids’ access to programs

LaFleur Russell Low Romash (2011)

THE BEACON COMMUNITY CENTERS MIDDLE SCHOOL INITIATIVE: Final Report on Implementation and Youth Experience in the Initiative http://www.wallacefoundation.org/knowledgecenter/after-school/evaluations/Documents/BeaconCommunity-Center-Middle-School-Initiative-FinalReport-on-Implementation-and-Youth-Experience.pdf

80 Beacon Community Centers in NYC

60% of OST intermediaries focus on increasing access to affordable, high quality OST programs for underserved kids Community-based, located in Exemplary selected public schools, Program serving youth in evenings, weekends, holidays, and summers. They do have Youth Councils to provide youth input into programming Provide structured programming for grades 5-8 in academics, life skills, career awareness, civic engagement, physical health, arts, and culture

Strategies

Middle school initiative asked for target enrollment of 200 middle schoolers per center (each center average total student enrollment of 1200)

Performance measures: Program Enrollment and Attendance


For 2009-2010, set program level goals of 216 hours of programming for middle schoolers (avg of 72 days a year, 3 hours a day participation) Saw slight increase in ELA (English Language Arts) and standardized math scores for program participants compared to those who did not participate

Community Indicator

Programs housed in middle schools had the highest middle school enrollment

Grossman, Campbell, & Raley, (2007)

Quality time after school: What instructors can do to increase learning

Philadelphia Beaconbased; not designed to measure impact but rather http://www.pop.givewell.org/files what affects student /Cause4/East%20Harlem%20Tutorial%20Organization/ engagement/enjoyment PPV213_publication.pdf 1. Surveys of youth on the perceptions of activities 2. Surveys of staff for training profiles 3. Observations focused on adult/youth and peer relationships, teaching methods, behavior management, youth decision-making

Interaction with family was important to promote ongoing youth participation 1. Project based learning 2. The most important ways to ensure youth engagement involves group management: a. setting reasonable ground rules b. providing positive reinforcement c. being consistent and fair in reinforcing expectations d. remaining firm, but not harsh, when ground rules were broken. Quantitative analysis did not

Strategy and best practices


and youth input to further describe what occurred in each activity.

Kauh (2011)

AfterZone: Outcomes for Youth Participating in Providence’s Citywide After-School System

Quasi-experimental design

http://www.wallacefoundation.org/knowledgecenter/after-school/evaluations/Documents/AfterZoneOutcomes-Youth-Participating-Providences-CitywideAfter-School-System.pdf

768 6th graders, 50% enrolled in Afterzone and 50% not enrolled Data collected from: 1. Youth surveys administered at the beginning and end of 6th grade and end of 7th grade 2. Administrative school records 3. PASA MIS Did NOT look at the quality of the Afterzone programs

find a direct link between peer affiliation or cooperative peer learning and participants’ level of engagement or their perceived level of learning. But the more participants reported that staff encouraged them to work together, the more youth enjoyed the activity and the more they wanted to return. Offers programming in sports, skills, and arts

Exemplar Program

3 distinct sessions a year (Fall, Spring, Summer) 2.5 hours a day, 4 days a week, for 6-8 grade 4 key model features: 1. Single set of quality standards with training/support provided for providers 2. Neighborhood campuses—multiple sites in geographically clustered areas anchored by 1 or 2 middle schools where the day begins and ends 3. Developmentally appropriate programming (i.e. encouraging independence and exposing to new experiences) 4. Check-in and check-out process, so student presence monitored at all times; safe

Strategies


transport provided between all sites Benefits: 1. Strikingly higher school attendance (but only in the first year of participation) 2. Improved attitudes towards community resources, social skills, increased connection to school 3. In seventh grade, higher grades in Math

Kotloff and KoromDjakovic (2010)

AfterZones: Creating a Citywide System to Support and Sustain High-Quality After-School Programs http://www.wallacefoundation.org/knowledgecenter/after-school/coordinating-after-schoolresources/Documents/Afterzones-Creating-CitywideSystem-to-Support-and-Sustain-High-Quality-AfterSchool-Programs.pdf

Durlak, Weissburg, Pachan (2010)

A Meta-Analysis of After-School Programs That Seek to Promote Personal and Social Skills in Children and Adolescents

Mixed methods study to examine how Afterzones: engaged and retained youth, promoted positive youth development, ensured high quality programming Methods 1. Site visits (interviewing staff and stakeholders) 2. Program observations using quality assessment tool RIPQA 3. Youth and Instructor feedback surveys 4. Program documents Meta-analysis of 69 ASPs that foster personal and social skills as main objectives. Those

Performance Measures

There was a dosage effect to benefits. To track students through the Exemplar zone and collect participation program— data, AfterZones uses system design youthservices.net 4 dimensions of quality measured for programs: 1. adult support 2. physical/emotional safety of the environment 3. quality of adult and peer interactions 4. ability for the youth to make choices and plan

Evidence for following SAFE technique for skills training: 1. Sequenced

Recommended strategies and performance measures


Am J Community Psychol (2010) 45:294–309 DOI 10.1007/s10464-010-9300-6

Duffett, Johnson, Farkas, Kung, Ott (2004)

All Work and No Play? Listening to What Kids and Parents Really Want from Out-of-School Time http://www.publicagenda.org/files/pdf/ all_work_no_play.pdf

focused entirely on academic outcomes were excluded

2. Active 3. Focused 4. Explicit

2 national random sample surveys, one with 609 middle and high school students and another with 1003 parents of schoolage children

Noted increases in selfperception, positive social behaviors, grades, and reduction in problem behaviors Cost, transportation issues, and absence of the programs within neighborhoods are all barriers to low-income family involvement in OST programs

Research supporting need for early planning/needs assessment

Commissioned by Wallace Foundation

Grossman, Price, Fellerath, Jucovy, Kotloff, Raley, Walker (2002)

Multiple Choices after school: Findings from the Extended-Service Schools Initiative http://www.issuelab.org/resource/multiple_ choices_after_school_findings_from_the_ extended_service_schools_initiative

Multi-method of OST programs in 10 cities: Site coordinator filled out annual organizational survey and in-depth site visits with interviews of staff, students, parents, other stakeholders, parent telephone survey, and observations of OST activities

Parents are not necessarily looking for OST to be academically focused; 42% would like it to help their children develop interests/hobbies 4 nationally recognized models for OST programs: Beacon, Bridges to Success, Community Schools, West Philadelphia Improvement Corporation.

Exemplary programs?

They all: Strategies and 1. Operate out of schools programs 2. Involve collaboration with community-based organizations and/or Research questions: universities • Which youth came to 3. Offer a range of activities: the afterschool academic, sports/rec, Programs and Why? Were enrichment


the programs attracting the young people who could benefit most? • What were the characteristics of highquality programs? • What were the benefits of participation? • Cost to operate and ways to finance

4. Finances are controlled by partnering organization, not Performance the school Measures Youth who attended reported: 1. that they used less alcohol 2. handled anger in socially appropriate ways 3. less likely to skip school Parents in the phone survey: 1. felt the kids were getting along better with peers 2. less likely to get in trouble 3. more self-confident 4. to have a better attitude towards school Authors concluded that enrichment programs were most beneficial, offering more opportunities for fostering strong adult-child relationships, peer collaboration, decisionmaking, and leadership skills Locating programs in schools serving low-income communities was an effective way of targeting low-income students, but additional resources were required to target older youth and the most high needs students

Performance Measures

Evidence supporting needs assessment and meaningful recruitment


Bodilly and Beckett (2005)

Making Out of School Time Matter: Evidence for an action agenda http://www.rand.org/content/dam/rand/pubs /monographs/2005/RAND_MG242.pdf

Narrative Review

Program objectives/activities: 1. To provide safe environment 2. Change attitudes towards academic achievement, achievement (test scores), or level of attainment (continuation to next grade, high school graduation, jobs) through: a. Tutoring b. Homework assistance c. Small group learning d. Homework assistance e. Field trips/College trips f. Career exploration 3. Change social/health behaviors through: a. drug/violence counseling or prevention b. health education 4. Change social interactions a. conflictresolution,anger management b. peer discussion groups c. parent support groups 5. Cultural enrichment (drama, crafts, music)— although never assessed as an outcome

Potential performance measures and indicators

Programs/ Activities

Research questions remaining

Problems with current research


Impact of OST programs on community beyond the youth (ie crime reduction) has not been measured convincingly to date. Current program evaluation plagued with selection bias and lack of control for participation after enrollment In general, OSTs program eval shows very little impact on test scores Although no rigorouslyobtained data available, a convergence of expert opinion from multiple sources suggest the following lead to positive youth outcomes: 1. clear mission 2. high expectations and positive social norms 3. safe/healthy environment 4. supportive emotional climate 5. small total enrollment 6. stable, trained personnel 7. appropriate content and pedagogy relative to the children’s needs 8. opportunities to engage integrated family and community partners 9. frequent assessment.

Problems with test scores as an indicator National best practices


Lauer, Out-of-School-Time Programs: A Meta-Analysis of the Akiba, Effects for At-Risk Students Wilkerson, Review of Educational Research 76 (2), 275-313 Apthorp, Snow, MartinGlenn (2006)

Meta-Analysis of 35 OST quantitative studies employing control or comparison groups, published in peer-review journals, reporting effect sizes based on gain scores between pre- and posttests or post-test scores between comparison groups. Included K-12 programming

Cooper, Charlton, Valentine, Muhlenbruck (2000)

Meta-Analysis and narrative review of summer school research

Making the most of summer school: A meta-analystic and narrative review. Monographs of the society for Research in child development (Serial No. 260), 65(1), 1-118

Empirical evidence to support: 1. Careful needs assessment and market targeting, providing ample program advertising and recruitment 2. Monitoring enrollment and attendance, follow-up on absences, incentive programs for consistant attendance Indicators used: Math and Reading standardized test scores OR grades OR classroom assessments 1. Small but significant effects on reading and math student achievement 2. Large positive effects for programs with specific characteristics such as tutoring 3. After school versus summer did not make a difference 4. OST programming does not have to focus entirely on academics to make a difference in indicators 1. Positive academic effects of summer school for both low-income and middleincome students 2. More positive results for programs for smaller groups of students involving individualized and small group instruction

Indicators

Support for relationship and strategies

Support for relationship between OST and achievement Support for strategy


3. Early elementary school students benefitted more than older elementary school students

Fashola (1998)

Review of extended-day and after-school programs and their effectiveness (Report No. 24). Baltimore, Center for Research on the Education of students. **Have not been able to locate. This info is derived from secondary sources.

Redd, Cochran, Hair, Moore (2002)

Academic Achievement Programs and Youth Development: A Synthesis **Info based on abstract in ERIC

Review of 34 extendedday or after-school programs A. Programs studied fell into following categories: 1. Language arts afterschool programs, 2. Study skills programs, 3. Academic programs in other curriculum areas, 4. Tutoring programs for reading 5. Community-based programs. Review of 12 academically oriented experimental or quasiexperimental studies of OST programs for adolescents

Notes general paucity of data on OST program evaluation. For programs intending to increase academic achievement there is some evidence of effectiveness of those that provide: 1. Greater structure/predictable routine 2. Link to the school-day curriculum 3. Well-trained staff 4. Individual tutoring Noted considerable variability in outcomes.

Questions remain in research Support for national best practices

Inconsistency in research

Most programs had academic Program achievement as only one objectives program objective. Other objectives included health and social wellbeing, selfsufficiency. Relationship Programs that focused on an between objective other than strategy and academic achievement (but outcome included an academic component) tended to do better at achieving that outcome than programs that


focused entirely on academics. Quasi-experimental data suggests participation for longer and more intensive durations had a more positive effect on outcomes Miller (2003) Critical hours: Afterschool programs and educational success, Quincy, MA: Nellie Mae Education Foundation.

McComb & Scott-Little (2003)

http://www.nmefoundation.org/ getmedia/08b6e87b-69ff-4865-b44ead42f2596381/Critical-Hours?ext=.pdf After-school programs: Evaluations and Outcomes. Greensboro, NC: SERVE

Comprehensive narrative review of middle school after school programs in promoting academic success and positive adolescent development Students who attend more, benefit more

Barley et al (2002)

Tutoring and peer tutoring can be effective strategies for improving achievement during the school day

Policy Studies Associates (1995)

Key is to engage student attention

Strategy


Hobbs, (2012)

Niehaus, Rudasill, Adelson (2012)

Effects of an afterschool program on elementary and ANCOVA comparison of middle school math achievement in Georgia Schools: A a sample of 180 at-risk Dissertation upper elementary and middle school students http://digitalcommons.liberty.edu/cgi/ who did and did not viewcontent.cgi?article=1545&context=doctoral participate in OST program

Self-Efficacy, intrinsic motivation, and academic outcomes among latino middle school students participating in an After-School program Hispanic Journal of Behavioral Sciences 34, 118

47 Latino 6-8 grade students in an afterschool program specifically for Latino students 1 year longitudinal study

Indicator for study: Georgia Criterion Referenced Competency Test (GCRT) Indicator for program (21st Century Community Learning Center): Maintaining student enrollment and attendance, math and reading classroom grades, improving behavior, completing homework, involving parents Findings: No significant difference in test scores 1. Intrinsic motivation positively correlated with GPA

Measurement/ Outcome Relationships

2. Self-efficacy was a positive predictor for school attendance and standardized math achievement scores 3. Program attendance contributed positively to math achievement 4. Multilevel growth modeling showed that selfefficacy and intrinsic motivation remained stable across school year, unrelated to degree of participation in the program 5. Incorporated Bandura (mastery experiences, vicarious learning, social

Program strategy


Valentine, Cooper, Patall, Tyson, Robinson (2010)

Little & Harris (2003)

persuasions) and opportunities to choose activities, enhanced interpersonal relationships by maintaining communication with students’ teachers and parents. 1. Many syntheses on ASPs (After school programs) aren’t published in traditional academic outlets—are better found by searching for nonprofits that evaluate ASPs, meeting notes, etc

A method for evaluating research synthesis: The quality, conclusions, and consensus of the synthesis of the effects of after-school programs Research Synthesis Methods, 1, 20-38

A Review of Out-of-School Time Program QuasiExperimental and Experimental evaluation results. Out-of-School Time Evaluation Snapshot Harvard Family Research Project

Review of 27 experimental and quasiexperimental OST program evaluations

2. Common indicators: Academic achievement, prevention, hobbies/interests, social-emotional development, safe environment OSTs linked to better attitudes towards school, higher educational aspirations, better school performance and attendance, less disciplinary action. Also decreased drug/alcohol use, sexual activity/pregnancy, and increased skills for coping with peer pressure. Positive links to these “youth development outcomes”: decreased behavior problems, improved social

Indicators

Research supporting programs/ potential performance measures


and communication skills, increased community involvement and broadened world view, increased selfconfidence and self-esteem. Little, Priscilla (2012)

A Field is Born: Reflections on a decade of afterschool http://www.hfrp.org/out-of-school-time/publicationsresources/a-field-is-born-reflections-on-a-decade-ofafterschool

Expert opinion

Family Engagement should be broadened to include: 1. Family involvement in programming or onsite 2. Family ensuring kids get to the program 3. Helping kids make informed choices regarding programming choices 4. Discussing kids’ progress with staff Six Strategies for Engaging Families 1.Have adequate and welcoming space to engage families 2.Establish policies and procedures to promote family engagement 3.Communicate and build trusting relationships 4.Be intentional about staff hiring and training to promote effective staff– family interactions 5.Support families and their basic needs 6.Connect families to each other, to the program staff, to schools, and to other community institutions

Best practice


Bethea (2012)

The Impact of Oakland Freedom School’s Summer Youth Program on the Psychosocial Development of African American Youth Black Psychology, 38, 442

JamesBurdumy, Dynarski, Moore, Deke, Mansfield Pistorino. (2005)

When Schools Stay Open Late: The National Evaluation of the 21st Century Community Learning Centers Program: Final Report. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, 2005. http://www.ed.gov/ies/ncee

**There is controversy regarding this study.

Program outcomes of Leadership excellence, Inc., Oakland Freedom Schools (6 week summer Language Arts enrichment course for 514 yos) 79 students (20 were middle schoolers) completed pre- and postERAS for attitudes towards reading, PRSCS or Student Self Concept Survey, WALLY for problem solving, SSRS for social behaviors, Adolescent Survey of Black Life for racial identity, Social Action Questionnaire 26 21st CCLC centers in 12 school districts Synthesis of two previous reports plus two years of follow-up on elementary school students Randomly controlled field trial Data on students’ supervision after school,

Program positively influenced racial identity and youth views toward AA culture and precepts Increase in pretest to posttest scores on social skills strategies and future commitment to social action No significant increases in attitudes toward reading or self-concept.

Performance measures

Program strategies—books, field trips, games reinforce each other in context of AA history/culture through the eyes of children. Emphasis on AAs engaging in social action or community service. No testing or entrance requirements for students

Program strategies

Typical program activities: homework sessions, academic activities, enrichment activities like art, drama, music, and recreation activities.

Activities

Treatment subjects felt safer after school but had no improvement in academic achievement (test scores in math, reading, science, social studies) and an increase in

Outcomes


See Mahoney & Zigler (2006). Translating science to policy under the No Child Left Behind Act of 2001: Lessons from the national evaluation of the 21st century community learning centers. J of Applied Developmental Psychology, 27(4) 282-294.

House, 1992

The relationship between perceived task competence, achievement expectancies, and school withdrawal of academically unprepared adolescents. Child Study Journal 24(4), 280-299

academic achievement, behavior, developmental outcomes, and feelings of safety after school were collected from parents, teachers, students, and school records in fall 2000 (baseline), spring 2001 (first followup), and spring 2002 (second followup) for the first cohort of students, and one year later for students who applied to centers in fall 2001. The Stanford Achievement Test in reading was administered at baseline and followup. Implementation data from program staff and principals and two sight visits to each site also performed. 4 year, prospective, longitudinal study of 378 academically underprepared students Academic self-concept and achievement expectancy measured

negative behaviors (measured by suspensions, calls home from teachers, teacher disciplinary action), mixed results in developmental indicators **Parental involvement was measured by asking parents to report how often they helped their children with homework. Parents with elementary school children in the program averaged helping their children with homework 3 times a week, significantly higher than those not in the program

Students’ educational expectations are very strongly related to eventual decision to withdrawal from school

Relationships between potential measurements and indicators


National League of Cities, 2011

City Strategies to Engage Older Youth in Afterschool Programs http://www.nlc.org/Documents/Find% 20City%20Solutions/IYEF/Afterschool/ city-strategies-to-engage-older-youth-in-afterschoolprograms-oct-12.pdf (The Wallace Foundation)

Deschenes, Arbreton, Little, Herrera, Grossman, Weiss, Lee (2010)

Strategy guide geared towards helping large or mid-size city programs attract youths age 11-18 Based heavily on Deschenes, Arbreton, Little, Herrera, Grossman, Weiss, Lee (2009). Engaging Older Youth: Program and City-level Strategies to Support Sustained Participation in Out-of-School Time Cambridge, MA: Harvard Family Research Project.

Engaging Older Youth: Program and City-Level Mixed methods—large Strategies to Support Sustained Participation in Out-Of- sample survey and inSchool Time depth interviews from 6 cities (Chicago, http://www.hfrp.org/out-of-school-time/publicationsCincinnati, NYC, resources/engaging-older-youth-program-and-cityProvidence, SF, Wash level-strategies-to-support-sustained-participation-inDC) out-of-school-time Research Questions Commissioned by The Wallace Foundation 1. What are the characteristics of highparticipation OST programs that

Recommendations: 1. Coordinate systems to support effective service delivery (meaning strong needs assessment, making sure programs are accessible and support marketing/recruitment, track participation and impact) 2. Ensure quality programs (meaning well-trained staff, input from youth on types of activities, setting clear standards, providing staff professional development) 3. Offer relevant programming (older youth gravitate toward academic, arts, recreation, and science programs as well as skill acquisition for reaching long term goals) 4. Promote college attendance and workplace readiness Middle schoolers need programs focusing on choice, leadership opportunities, cultural enrichment, health and wellness, community service Successful in supporting high retention: 1. Provide leadership opportunities for youth (eg volunteer/community service activities, youth councils, opportunities to design activities for peers or younger youth, “officer” roles, paid staff positions) 2. Having staff keep informed about what the youth are doing outside of programs (eg collecting report cards and

National best practices

Research to support national best practices


support sustained participation as measured by retention? 2. How do these characteristics differ for middle school and high school youth? 3. What strategies are city initiatives implementing to support access to programs and sustained participation, and how do OST programs perceive the usefulness of city-level strategies for achieving their participation goals?

Harris (2011) Afterschool Evaluation 101: How to Evaluate an Expanded Learning Program http://www.hfrp.org/out-of-school-time/publicationsresources/afterschool-evaluation-101-how-to-evaluatean-expanded-learning-program

A toolkit for OST program directors for designing and implementing an evaluation strategy

contacting parents regularly, publically recognize accomplishments outside of program) 3. Being community based (rather than school based) 4. Enrolling 100 or more youth 5. Holding regular staff meetings Middle school programs reported that particularly around eight grade, youth stopped attending because they wanted a program that felt “older.� Consider programming for eight graders that includes more responsibility. High retention programs report using more strategies to engage families than low retention programs

Best Practices in Out of School Time Programming: A Literature Matrix  

A Literature Review Matrix compiled in January 2013 on best practices in Out of School Time (OST) programming for middle school students