Building Peace and Compassion Through Social and Emotional Learning by Hawaii Community Foundation

HAWAI‘I COMMUNITY FOUNDATION

Building Peace and Compassion Through Social and Emotional Learning VOLUME 1: Final Evaluation Report July 2020

Photo: St. Andrew’s Priory

IMPAQ International, Inc. Evaluation Team: Linda Toms Barker, Project Director Colleen McLelland, Research Analyst Nada Rayyes, Senior Research Associate Jackie Ng-Osorio, LEEAD Scholar Gail Dana-Sacco, LEEAD Scholar

Building Peace and Compassion Through Social and Emotional Learning: FINAL Evaluation Report About the Hawaiʻi Community Foundation The Hawaiʻi Community Foundation (HCF) is one of the oldest community foundations in the country, established in 1916. HCF manages a repository of more than 800 charitable funds that have been set up by generous individuals, families, and businesses across the state to benefit the people of Hawaiʻi. These funds reflect the charitable grantmaking interests of their donors and can vary from unrestricted grants to those for very specific areas such as scholarships or a geographical region. HCF distributes these charitable funds through a variety of grant programs. HCF also supports the advancement of knowledge, understanding, and effectiveness in the nonprofit sector by providing training opportunities and disseminating information through workshops, events, learning communities, studies, publications, and online resources. Hawai’i Community Foundation 827 Fort Street Mall Honolulu, HI 96813 (808) 537-6333 www.hawaiicommunityfoundation.org

About IMPAQ International IMPAQ International (IMPAQ) helps governments, businesses, foundations, non-profits, and universities evaluate and enhance their programs and policies through research studies, program evaluations, implementation assistance, surveys and data collection, technical solutions, and communications strategies. IMPAQ is headquartered in Columbia, MD, with offices in Washington, DC, Oakland, CA, Boston, MA, and Seattle, WA. Remote employees are located across the country and internationally. 10420 Little Patuxent Parkway Suite 300 Columbia, MD 21044 443.259.5500 http://www.impaqint.com

Acknowledgments The IMPAQ team would like to thank our project officer, Robbie Ann Kane, for her guidance and support in developing and implementing this evaluation. We would also like to thank the faculty and staff of the grantee schools who have been so gracious with their time and have shared with us their stories, their growing pains and their successes.

Table of Contents Table of Contents .......................................................................................... i Table of Exhibits ...........................................................................................iv Executive Summary ........................................................................................ i 1. Introduction ......................................................................................................... i 2. Program Implementation ......................................................................................... ii Description of Grantees’ SEL Programs ........................................................................ ii Changes or Adaptations from Grantees’ Original Design .................................................... ii Factors or Conditions that Support Effective Implementation ............................................ iii Cultural Considerations.......................................................................................... iii 3. Program Outcomes ................................................................................................ iv Summary of School Climate Findings ...........................................................................iv Summary of Student Behavior Findings ........................................................................vi Academic Achievement...........................................................................................vi Achievement Gap..................................................................................................vi Outcomes Summary .............................................................................................. vii 4. Lessons Learned .................................................................................................... vii Year 1 .............................................................................................................. vii Year 2 .............................................................................................................. vii Year 3 .............................................................................................................. vii Getting the Right Fit ............................................................................................ viii Cultural Relevance .............................................................................................. viii 5. Promising Practices .................................................................................................ix Use of Evaluation Results for Program Improvement ........................................................ x 6. Sustainability and Looking to the Future ....................................................................... x Institutionalize SEL ................................................................................................ x Continue to invest in professional development .............................................................xi Other sustainability strategies ..................................................................................xi

1. Introduction ............................................................................................ 1 What is the Hawai‘i Community Foundation’s Pillars of Peace SEL Hawai‘i Initiative? ..................... 1 What is Social and Emotional Learning? ............................................................................ 1 Which schools received PoPH SEL grants? .......................................................................... 2 How was the Initiative evaluated? .................................................................................. 3 Evaluation Questions.............................................................................................. 4 Data Sources ....................................................................................................... 6

2. Program Implementation ............................................................................10 2.1 Description of Grantees’ SEL Programs ....................................................................... 10 What SEL competencies, student behaviors and school climate concerns were being targeted by grantees?........................................................................................................... 10 What SEL programs were being implemented by grantees? ............................................... 11 IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

What were the grantees’ key activities? ..................................................................... 14 2.2 Changes or Adaptations from Grantees’ Original Design .................................................. 16 Changes to School Culture ...................................................................................... 16 Addition of More Intensive SEL Interventions ................................................................ 18 Changes to School Policies and Structures ................................................................... 18 Staffing Changes .................................................................................................. 18 Teacher Training and Professional Development............................................................ 19 2.3 Factors or Conditions That Supported Effective Implementation ........................................ 19 1. Time ............................................................................................................. 20 2. Champion ....................................................................................................... 20 3. Teamwork ...................................................................................................... 21 4. SEL Coordinator ............................................................................................... 21 5. Commitment ................................................................................................... 22 6. Professional Development (PD) ............................................................................. 22 7. Support ......................................................................................................... 23 8. Resources ....................................................................................................... 23 9. Careful Pacing ................................................................................................. 24 10. SEL for Adults ................................................................................................ 24 11. Reflection and Flexibility................................................................................... 24 12. Continuity of Staffing ....................................................................................... 25 13. Continuity of SEL Practice.................................................................................. 25 14. Sufficient “dosage” ......................................................................................... 25 15. Sufficient counseling resources ........................................................................... 25 2.4 Cultural Considerations.......................................................................................... 26 How appropriate were the SEL programs that grantees have been implementing for the local Hawaiian culture? ................................................................................................ 26 What efforts have grantees made to adapt these programs for greater cultural relevance? ........ 27

3. Program Outcomes....................................................................................29 3.1 School Climate .................................................................................................... 29 Tripod School Climate Survey .................................................................................. 29 Education for the Future (EFF) School Climate Survey ..................................................... 35 Teacher Ratings of School Climate ............................................................................ 36 Parent Ratings of School Climate .............................................................................. 40 Discussion of School Climate Findings......................................................................... 45 3.2 Student Behavior ................................................................................................. 47 School Attendance ............................................................................................... 47 Behavioral Incidents ............................................................................................. 50 Discussion of Behavioral Findings .............................................................................. 51 3.3 Academic Achievement.......................................................................................... 53 Achievement Gap................................................................................................. 56 Discussion of Academic Outcomes ............................................................................. 56 3.5 Other Outcome Measures ....................................................................................... 57 Student SEL Skills and Competencies ......................................................................... 57 Teacher Perceptions ............................................................................................. 58 3.6 Outcomes Summary .............................................................................................. 58 IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

4. Lessons Learned ......................................................................................60 Year 1 Lessons Learned .............................................................................................. 60 Year 2 Lessons Learned .............................................................................................. 61 Year 3 Lessons Learned .............................................................................................. 62 Getting the Right Fit ................................................................................................. 63 Cultural Relevance .................................................................................................... 63

5. Promising Practices ..................................................................................65 Use of evaluation results for program improvement ............................................................ 67

6. Sustainability and Looking to the Future ........................................................69 Institutionalize SEL ................................................................................................... 69 Continue to Invest in Professional Development ................................................................. 69 Use Data Effectively .................................................................................................. 70 Seek Additional Funding ............................................................................................. 70 Continue to Grow ..................................................................................................... 71 Continue Participating in the Larger SEL Community ........................................................... 71 Wishes for the Future ................................................................................................ 71 Sharing Stories ......................................................................................................... 72

IMPAQ International

iii

Pillars of Peace SEL Final Evaluation Report July 2020

Table of Exhibits Exhibit 1. Map of HCF PoPH SEL Grantees .............................................................................. 2 Exhibit 2. Types of Grantee Schools ..................................................................................... 3 Exhibit 3. Grade Levels of Participating Schools ...................................................................... 3 Exhibit 4. Conceptual Overview of the Evaluation .................................................................... 4 Exhibit 5. Matrix of Study Questions by Data Sources ................................................................ 5 Exhibit 6. Data Collected by Each School ............................................................................... 9 Exhibit 7. SEL Programs Implemented by Grantees .................................................................. 11 Exhibit 8. Change in Student Tripod Ratings by Domain Across Eight Schools ................................... 29 Exhibit 9. Summary of Significant Changes in Student Tripod Ratings Across Eight Schools................... 30 Exhibit 10. Change in Student Tripod School Climate Ratings by Grade Level Across Eight Schools ......... 31 Exhibit 11. Significant Changes in Student Tripod Ratings by Grade Level....................................... 32 Exhibit 12. Change in Student Tripod Ratings by School ............................................................ 33 Exhibit 13. Significant Changes in Student TRIPOD Ratings by School ............................................ 35 Exhibit 14. Change in Student EFF Ratings by School ................................................................ 35 Exhibit 15. Summary of Significant Changes in Student EFF School Climate Ratings ........................... 36 Exhibit 16. Hawai‘i STRIVE HI Data – Percentage of Students Who Feel Positively About Their School ..... 36 Exhibit 17. Change in Teacher School Climate Ratings Across All Schools by Domain .......................... 37 Exhibit 18: Change in Teacher School Climate Ratings by School ................................................. 38 Exhibit 19. Change in Parent Ratings Across All Schools by Domain ............................................... 41 Exhibit 20. Summary of Significant Changes in Parent EFF Ratings Across Four Schools ....................... 42 Exhibit 21. Change in Parent School Climate Ratings by School ................................................... 43 Exhibit 22. Average Daily Attendance by School ..................................................................... 48 Exhibit 23. Average Daily Absences by School ........................................................................ 48 Exhibit 24. Percentage of Students Who Missed 15 or More Days of Schoo*...................................... 49 Exhibit 25. Changes in Number of Reported Behavioral Incidents per 100 Students ........................... 51 Exhibit 26. Changes in Percentage of Public and Charter School Students Meeting Math, Language Arts and Science Standards .................................................................... 54 Exhibit 27. Reducing the Achievement Gap ........................................................................... 56

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Executive Summary 1. Introduction Beginning in Fall 2016, the Pillars of Peace Hawai‘i (PoPH) Social and Emotional Learning (SEL) grant program funded 11 public, charter, and private schools across the State of Hawaiʻi for three years to test and demonstrate the viability and efficacy of evidence-based, best practice SEL programs. Most of the grantees had at least some past experience with SEL, but only a few had significant SEL programming already in place. Most of the grantees are located on O‘ahu, with four located on neighbor islands. They include four regular public schools, four public charter schools, one Hawaiian immersion school, and two private schools. One of the charter schools offers both an English language program and a Hawaiian language immersion program. Some range from K-12 or pre-K–12, four are elementary schools, and one is a middle school. Grantee schools include: •

Kula Kaiapuni ‘o Ānuenue School (K-12)

•

‘Ele‘ele Elementary School (K-5)

•

‘Ewa Makai Middle School (7-8)

•

Honolulu Waldorf School (PreK-12)

•

Ka‘elepulu Elementary School (K-6)

•

Kualapu‘u Public Conversion Charter School (PreK-6)

•

Laupāhoehoe Community Public Charter School (PreK-12)

•

St. Andrewʻs Schools (PreK-12)

•

University Laboratory School (K-12)

•

Volcano School of Arts & Sciences Public Charter School (K-8)

•

Waikīkī Elementary School (K-5)

The Hawai‘i Community Foundation (HCF) contracted with IMPAQ International, LLC (the IMPAQ team) to conduct an evaluation. The data sources for the evaluation included document review, administrative data, school climate surveys, phone interviews, site visit interviews, observations, and implementation data from grantees. The IMPAQ team collected and analyzed primarily aggregate data rather than individual student-level data. That is, the outcomes analysis focuses on school-level data, with the school as the unit of analysis, with some sub-group disaggregation where possible, such as comparing elementary, middle and high school students, or comparing survey responses from students, teachers and parents. The major focus of this evaluation was not to address a common evaluation question of whether the intervention was effective. Since the grant required schools to implement evidence-based approaches, the research has already provided evidence that these SEL approaches are effective, at least in the contexts in which they were tested. Rather, this evaluation focused on topics such as how grantees implemented their programs, the kinds of resources needed for effective implementation, the challenges they encountered along the way, promising practices and lessons learned. This volume describes the evaluation and presents qualitative and quantitative findings. Volume II provides appendices with profiles of each of the grantees and additional exhibits to supplement the first volume. Volume III presents case studies of four of the grantees, telling their stories in more detail than is included in this first volume of the report.

IMPAQ International

Pillars of Peace SEL Evaluation – Executive Summary July 2020

2. Program Implementation This section describes the implementation of SEL programs across the 11 grantee schools. It begins with a description of grantees’ SEL programs, and then discusses a wide range of factors that affected program implementation in the schools. It also provides a summary of some of the cultural considerations schools encountered in adopting or adapting SEL programs here in Hawai‘i.

Description of Grantees’ SEL Programs Each grantee targeted different SEL competencies, student behaviors and/or school climate concerns with different reasons for seeking grant funds to develop or expand SEL programming at their school. Grantees described a variety of goals for their SEL programs including: • Building empathy in students • Reducing aggressive behaviors (including bullying) • Teaching conflict resolution and anger control • Building students’ responsible and ethical behavior skills • Developing compassion • Developing mindfulness • Building respect for each other and a sense of community • Reducing cyber bullying • Teaching self-advocacy skills • Preparing students to be contributing members of their communities Grantees implemented a variety of SEL programs. In many cases, the grant-funded programs built on SEL initiatives already in place at the schools. Most grantees implemented multiple programs. Some began with a single program and added others as additional needs became apparent. The overall goal for grantees was to create a safer and more positive learning environment through the SEL programs. Most grantees implemented school-wide programs, while some started with selected grade levels and/or teachers, and expanded programs to the rest of the school over time. Some targeted only a few grades and remained focused on those grades throughout. Using HCF grant funds, schools adopted a wide variety of evidence-based SEL programs.

Changes or Adaptations from Grantees’ Original Design Most grantees modified their SEL programs based on their implementation experience over the course of the grant period. As SEL started to take root at the schools, they tailored their programs to meet their own unique needs and culture. The following are examples of some of the changes schools made over the three years: •

Changes to School Culture - Several schools made school-wide changes in policies or procedures to support their work towards a positive learning environment and school culture. These included incorporating SEL into their schoolwide plan to ensure SEL would continue beyond the life of the grant, changing bell schedules to accommodate SEL instruction at regularly scheduled times each week, and setting up meditation rooms as part of a shift in discipline approach to one that was less punitive and more of a learning experience setting.

•

Addition of More Intensive SEL Interventions - Several schools realized that some of their students faced one or more significant life crises (homelessness, abuse, family death/incarceration) which limited their SEL growth. These schools sought out and embraced more intensive SEL

IMPAQ International

Pillars of Peace SEL Evaluation – Executive Summary July 2020

interventions, such as Trauma Informed Practices 1 and Resiliency Theory 2, to better reach these troubled students and provide additional emotional support. •

Staffing Changes - Some grantees experienced staffing changes with new administrators or key SEL players, which affected the implementation of their SEL programs. Where turnover and learning curve might have been somewhat disruptive, most schools shifted responsibilities or restructured their SEL implementation teams to address this challenge. Several schools also decided to shift from SEL lessons being taught by counselors or SEL coordinators to being taught by the teachers themselves. This was seen as important both for reinforcing the lessons and integrating SEL into the learning environment throughout the school day.

•

Teacher Training and Professional Development - Some schools identified the need for continued or even increased professional development beyond what was provided during initial implementation. This included expanding professional development beyond teachers to include the full faculty, as well as for onboarding new faculty and staff. One school decided to focus on expanding the role of its counseling staff to provide teacher training instead of continuing to invest in outside professional development.

Factors or Conditions that Support Effective Implementation During site visits and phone interviews, SEL coordinators, school principals, teachers and other staff identified a number of factors that support effective SEL program implementation. Conversely, a lack or limitation of these supports presents challenges to effective implementation. In collaboration with the grantees, we have identified fifteen of these conditions: 1. Time

9. Careful Pacing

2. An SEL champion

10. SEL for Adults

3. Teamwork among teachers and staff

11. Reflections and Flexibility

4. An effective SEL coordinator

12. Continuity of staffing

5. Commitment or “buy-in”

13. Continuity of SEL practice

6. Professional development

14. Sufficient “dosage”

7. Administrative support

15. Sufficient counseling staff

8. SEL program resources

Cultural Considerations Grantees reflected on how well the SEL programs they selected for their schools fit the cultural context of their students and community. For many schools, the key consideration was appropriateness of the program in the local Hawaiian and multi-cultural context. For some, the key consideration was how well the approaches used by the schools and the SEL domains they represent align with the Nā Hopena Aʻo

In a trauma-informed school, the adults in the school community are prepared to recognize and respond to those who have been impacted by traumatic stress, which can arise from bullying, dramatic weather events, school shootings, and the daily exposure to family trauma such as divorce, homelessness, incarceration, and suicide. 2 Instead of focusing on how to diagnose and fix children’s problems, Resiliency Theory identifies protective factors that enable students to overcome obstacles and grow into socially and emotionally healthy adults. An expansive body of psychological research identifies Caring Relationships, Community Belonging, and High Aspirations as vital protective factors. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1903337/ 1

IMPAQ International

iii

Pillars of Peace SEL Evaluation – Executive Summary July 2020

(HĀ) 3 framework. Some schools reflected on how well their SEL programs addressed the needs of other cultural groups as well. •

Volcano middle school implemented Hawaiian practices of Nā Kilo ʻĀina 4 and Piko Wehena. 5

•

In another Hawaiian immersion program, the teachers translated lessons into Hawaiian on the fly and delivered the lessons in Hawaiian. They translated some of the tools, such as cooperative learning goals, into Hawaiian and created a Hawaiian Peace Path.

•

After trying out Tribes and Second Step, the Hawaiian immersion school shifted to having their own teachers develop SEL lessons in the Hawaiian language. They focused not merely on translating SEL into a Hawaiian immersion setting, but also on the ways that Hawaiian spaces can cultivate social and emotional growth within the native context.

•

At one school a teacher developed lessons that looked at the characteristics of leaders in the Hawaiian community, including both the ali‘i 6 and the children’s own ancestors, to recognize what qualities they had, or wanted to build in themselves.

•

A couple of schools mentioned that the videos associated with the Second Step program were a little difficult for some of their students to relate to. The students in the videos did not look like them and their context, surroundings and even how they spoke seemed unfamiliar. Without the resources to make their own videos locally, teachers adapted by applying the lessons to their own lives and their students’ lives through storytelling and discussion. As they became more effective in encouraging students to tell their own stories, they no longer heard complaints from the students about the videos.

•

One school developed a bilingual Japanese-English Philosophy for Children (p4c) mentoring group, out of an awareness of the academic and social-emotional challenges faced by a large population of Japanese speaking students. Teachers reported a dramatic growth in the confidence of these students during their regular classroom p4c discussions.

3. Program Outcomes Outcome measures included school climate and student behavior measures for all grantees, student academic outcomes for all of the regular and charter public schools, and a variety of other measures specific to individual grantees.

Summary of School Climate Findings Because grantees used different school climate surveys with different rating scales, we grouped the questions on each survey into the school climate domains identified by the National School Climate Center

Nā Hopena A‘o or HĀ is a framework of the HIDOE to “develop the skills, behaviors and dispositions that are reminiscent of Hawai‘i’s unique context, and to honor the qualities and values of the indigenous language and culture of Hawai‘i.” It was adopted as policy E-3 by the Board of Education in June 2015. 4 Nā Kilo ʻĀina is a practice of close observation of the environment as “the places and things that feed us physically, spiritually, and emotionally”. Nā Kilo ʻĀina has been adopted at VSAS to incorporate acute awareness of outer (environmental) with inner (social-emotional) qualities to cultivate knowledge and understanding as well as personal connection and relationship to the land and environment. 5 Piko Wehena (“source opening”) is a school-wide daily morning protocol to start the day. 6 Polynesian chief, noble or king 3

IMPAQ International

Pillars of Peace SEL Evaluation – Executive Summary July 2020

and then computed the percentage of respondents with positive responses by domain. This approach had several limitations, including the fact that HIDOE changed the survey used statewide in Year 2 of the grants, leaving us with only two years of comparable data instead of three for eight of the schools and the fact that not every domain was addressed in every instrument each year. Here we summarize the key findings including both the percent positive ratings and significance testing to identify which of those changes are statistically significant. These include results for different instruments, by grade level and by school. •

Overall, students’ school climate ratings increased over the course of the grant. Student ratings on the Tripod survey increased both overall, and in every climate measurement domain from Year 2 to Year 3, with statistically significant increases in three of six domains.

•

Among the eight schools using the Tripod school climate survey, the climate domain that showed the largest increase in student ratings was Rules and Norms, which included questions about whether rules were clear and whether student behavior was consistent with school rules. Social Support-Adults (focusing on how teachers and administrators treat the students) and Support for Learning (focusing on how teachers support learning in the classroom and support individual students) also increased significantly.

•

Examining Tripod survey results by grade level revealed significant increases in students’ climate ratings for all three grade categories ̶ elementary, middle and high—with significant increases in almost every climate domain. The only decrease was in middle school ratings of Physical Security. Several middle schools reported outside events that affected students’ overall sense of physical security including local flooding, gunfire and the mistaken incoming ballistic missile alert in 2018 as possible contributing factors.

•

Analysis of the student Tripod ratings by school showed that the ratings increased to some extent for all the schools that administered this survey for the two years. ‘Ewa Makai, Ka‘elepulu and Waikīkī had the greatest increases.

•

Among the four schools using the EFF (Education for the Future) school climate survey, Volcano had significant increases in six domains and St. Andrew’s in four and both had a significant increase overall across all domains. Laupāhoehoe had significant increases in three domains, even though their increase across all domains was not significant. Honolulu Waldorf had no significant changes. Honolulu Waldorf saw a decrease in student ratings in Year 2, but the ratings increased again in Year 3. The Social-Emotional Security and Social Support – Adults domains increased significantly for three of the four schools.

•

According to HIDOE, the percentage of students reporting they “feel positively about their school” increased in six out of the eight public and charter schools who reported on that measure.

•

Teachers tended to give somewhat higher ratings than students.

•

Teacher ratings increased significantly in the Physical Security domain but decreased in the Rules and Norms and Support for Learning domains.

•

When examining teacher ratings by climate domain, none of the differences were statistically significant, perhaps because at the individual school level, the sample size of teachers was fairly small.

•

Parents tended to respond more positively than either students or teachers.

IMPAQ International

Pillars of Peace SEL Evaluation – Executive Summary July 2020

•

Perhaps because the ratings were already high in SY(school year)2016-17, there were very few changes in parent ratings over the grant period. The only changes were significant increases in the Social-Support-Students and Social-Emotional Security domains, and a decrease in the Connectedness/Engagement domain.

Summary of Student Behavior Findings Student behavior was documented through attendance, absences and behavioral incidents. Although the behavioral measures available for this evaluation were limited to those that schools were either already collecting or would be easy to report, the results were encouraging: •

Average daily attendance increased at Volcano, Laupāhoehoe and St. Andrew’s by approximately one percentage point. While there were no large across the board changes in attendance and absences, those schools that specifically targeted attendance as a concern were pleased with the results. For other schools, attendance was not identified as a major concern for the school or a focus for their SEL program.

•

In some schools, the number of reported incidents actually increased in Year 2 and then decreased again. These increases were expected and were attributed to changes in the ways that teachers and administrators reported and tracked incidents. Honolulu Waldorf, for example, had been tracking behavioral incidents through a strike system. Their strike rate actually went up in the context of their effort to use the strike system more proactively to more comprehensively identify incidents and problems as they arose, to make sure the school addressed them rather than allowing them to fester.

•

Other grantees worked towards decreases in behavioral incidents that they attributed to SEL practices. For example, Volcano implemented strategies to change the language used to describe incidents from labels such as “bullying” to descriptions of the actual behaviors that caused distress. They credit this for contributing to their reduction in behavioral referrals, along with expanding ‘Ike Hawai‘i from an elective to a required course for all students, and using long-term meaningful, place-based projects to develop a sense of responsibility for place through close observation, knowledge and stewardship.

Academic Achievement Although improving academic achievement was not generally a goal of grantees’ SEL programs, we analyzed student achievement data for the eight public and charter schools for which it was readily available through HIDOE. This included achievement data in three subject areas: English Language Arts, math, and science. •

All but the two highest performing schools saw an increase in percentage of students meeting academic standards in at least one subject area.

•

Three schools saw an increase of students meeting academic standards in all three subject areas.

•

An important observation when assessing academic outcomes is that schools clearly began at very different starting places, with a low of 4-21% of students meeting standards at one school to a high of 83-100% of students meeting standards at another.

Pillars of Peace SEL Evaluation – Executive Summary July 2020

students. Again, this was not an outcome measure that grantees chose, but one that we decided to look at, given the availability of data. Two of nine public and charter schools reduced the achievement gap between high needs and non-high needs students in English Language Arts and four reduced the achievement gap in math over the grant period.

Outcomes Summary All eleven grantees showed improvement in at least one of the three major areas of measurement — school climate measures, behavioral measures and academic achievement. All grantees showed improvement in at least one behavioral measure. Seven of the grantees showed improvement in all three measurement areas.

4. Lessons Learned

Improvement in at least one School Climate Measure

OUTCOMES SUMMARY



Ānuenue ‘Ele‘ele ‘Ewa Makai Honolulu Waldorf

      

Ka‘elepulu Kualapu‘u Laupāhoehoe St. Andrewʻs University Lab Volcano Waikīkī

Improvement in at least one Behavorial Measure

          

Improvement in at least one Academic Measure

  

N/A

   N/A

  

Grantees learned a great deal about implementing SEL programs, with some programs implemented as planned and others making mid-year course corrections. These lessons may also be useful to inform other schools and grantees of issues to be aware of when planning for or implementing SEL programs. Here we summarize lessons learned by the grantees in each of the three years of the grants. Lessons learned the previous year still remained relevant, but each year, grantees added new insights.

Year 1 1. There is no single best starting place. 2. SEL is a school-wide issue. 3. School climate is a primary focus. 4. SEL is about students AND adults. 5. Teacher and student commitment are key. 6. Schools can use data to get buy-in for SEL programs. 7. Invest in implementation and planning. 8. Keep it fresh for staff and students. Year 2 9. Planning needs to happen in the spring. 10. Leadership and teacher buy-in need to go hand in hand. 11. Sustaining focus and maintaining energy can be challenging. 12. SEL time needs to be structured into the school day. 13. Reflection is an important part of the process and worth the time and effort. 14. Students can be empowered to play a crucial role. Year 3 15. SEL programming can be a good fit with Hawaiian cultural values. 16. Build capacity to address the needs of students in crisis. IMPAQ International

vii

Pillars of Peace SEL Evaluation – Executive Summary July 2020

17. Strong leadership from the top is essential. 18. Start with the infrastructure. 19. Build capacity to analyze data. 20. Bring it back to the “why”. 21. SEL instruction is both explicit and implicit. 22. SEL can be transformative, but it does not happen overnight.

Getting the Right Fit Grantees offered advice on how to ensure the SEL programming chosen is a good fit for the school. Examples of key considerations include: •

Start with a target group of your highest needs or most at-risk group of students to help choose a program that has the best chances of reaching/relating to them.

•

What other schools are already doing it, and what have their successes been?

•

How much does it cost? And what are the professional development costs associated with implementing it?

•

Consideration of new personnel applies to fit. Fit includes: "Will this teacher/staff member 'fit' into our community's SEL culture?"

•

Is this a curriculum that needs class time? Who will deliver? Which classes? Is the program a preventative model, or is it an intervention? How does the training work? A few select advocates or whole faculty/staff? Is there an adult SEL piece? How much time for organization and prep needed? Will you have an SEL coordinator?

•

When choosing your SEL program, include staff, especially all teacher’s, ideas, outcome expectations, and concerns, and provide continued support.

•

Consider integrated support for adults within the program.

•

Does it help create a culture across campus?

Cultural Relevance From their experiences with adapting SEL programming to better suit both Native Hawaiian culture and the cultural diversity of their schools, grantees offered lessons on ensuring cultural relevance: •

Specifically, and explicitly plan for SEL and Hawaiian Values integration, using both Positive Behavioral Interventions and Supports (PBIS) 7 and the HĀ framework.

•

Choose Love - Choose Aloha. It fits into HĀ, but you need to make the connections.

•

We have found it helpful/necessary to add "other" practices/language that support cultural appropriateness to our Second Step SEL program.

•

We looked at our school's Hawaiian values stated in our mission/vision and made sure our SEL oals related and tie in.

•

Look for parallels between our local culture/community and the program's values/teachings. For example, Restorative Practices go hand in hand with ho‘oponopono 8.

Positive Behavioral Interventions and Supports (PBIS) is an approach to addressing student behavior through a three-tiered systems change framework for improving and integrating all of the data, systems, and practices affecting student outcomes. https://www.pbis.org/

The word “ho‘o” means “cause” in Hawaiian, while “ponopono” means “perfection”. The term “ho‘oponopono” can be translated as “correct a mistake” or “make it right”. This concept has much in common with the Western concept of “restorative justice.” 8

IMPAQ International

viii

Pillars of Peace SEL Evaluation – Executive Summary July 2020

•

Roots of Empathy specifically encourages the baby's family to bring in their language, songs, and customs. It acknowledges each student’s temperaments well.

5. Promising Practices These practices were identified by the IMPAQ team and grantees to have made a difference and to have promise as being transferable and useful to other schools implementing SEL programs. •

Ānuenue — School leadership found that increasing the amount of information shared with teachers about trauma-informed practices in their faculty meetings has helped increase empathy for students with significant behavioral issues and ensured everyone on campus feels a sense of urgency and responsibility for creating an inclusive environment.

•

‘Ele‘ele - The school used grant funds to support ten p4c teachers who meet as a group after school to talk about what has been working, challenges teachers face, and possible strategies to try. The teachers also received professional development during this time.

•

‘Ewa Makai — Middle School Student for a Day was a teacher/student shadowing activity. The teachers wore the uniform, carried a backpack, went from class to class, and students treated them like students. The teachers found it to be a very eye-opening exercise in empathy. Knowing that the adults cared enough to put themselves through the exercise made the students feel very heard and appreciated.

•

Honolulu Waldorf — The schools’ student support committee adopted a “three-legged-stool model”, supporting academic, behavioral, and SEL in their students with a point person for each “leg” at both the lower school and the high school. This resulted in a close look at behavioral supports and the schools’ current, punitive sounding, “strike” system. They moved to a new approach called TAB, or “take a breath”. Students used a reﬂection page to identify their behavior, how it affected others, identify their emotions and make a plan for the next time.

•

Ka‘elepulu — The school counselor delivered Choose Love lessons in the context of p4c. Students discussed the Choose Love concepts in a circle, in p4c’s proscribed, respectful manner. This approach extended the short Choose Love lessons to a full 30 minutes, allowing students ample time to explore the topics.

•

Kualapu‘u — A 6th grade teacher has implemented a system of Team Points, where Getting Along Together (GAT) teams in reading, writing, and social studies earned points for doing their work and could redeem their points for snacks. The teacher tracked individual and group points, including the reasons points were awarded or taken away.

•

Laupāhoehoe — The school changed their bell schedule to accommodate two periods of dedicated SEL instruction per week. All students received 50 minutes of SEL curriculum instruction twice a week. This dedicated and consistent SEL time was beneficial to both teachers and students as they practiced their SEL skills.

•

St. Andrew’s — The SEL team created a bimonthly, in-house, 2-page newsletter called The RULER Review. The newsletter kept faculty and staff up to date on which RULER tools to focus on for the month, and which to start getting ready to teach. The newsletters included links to pedagogical videos, reminders of upcoming RULER events, and photos of RULER in action in classrooms.

•

Volcano — The school implemented strategies to describe incidents not with labels such as “bullying”, but rather with descriptions of the actual behaviors that caused distress. This

IMPAQ International

Pillars of Peace SEL Evaluation – Executive Summary July 2020

improved communication and understanding and helped resolve conflicts. They credited this with contributing to their reduction in behavioral referrals. •

Waikīkī — The SEL program changed the format of their Art Mentoring Group to provide higher intensity mentoring for the students. In addition to weekly classes, the students attended public exhibitions at the Honolulu Museum of Art, two parent-child weekend classes, and two-night p4c and art classes for 4th and 5th graders, and middle and high school alumni of the program.

Use of Evaluation Results for Program Improvement Most grantees have used the data collected to measure the effectiveness of their SEL programs and other related school systems (i.e., discipline) in order to drive informed decisions for course correction or forward progress. Several schools were frustrated that they lacked the capacity to analyze their data to extract meaningful results in evaluating the success of their programs. Two schools hired outside consultants to help them make sense of their data. Schools described various ways of using data collection and analysis for program improvement. For example: •

Schools used data to ensure SEL programs were being implemented consistently.

•

Behavior data was being used to both diagnose issues needing attention and to evaluate the effectiveness of SEL strategies.

•

Schools came to recognize the value of capturing baseline data for comparison with postintervention data.

•

Several schools used school climate data to focus their efforts on specific school climate domains.

•

While schools were hungry for feedback, they realized they needed to balance the number of surveys implemented with limited teacher time and to consider student survey fatigue.

•

The data collected by some schools indicated a need for student trauma support, which led to their introducing ways to help faculty implement trauma-informed strategies.

•

Some schools collected large amounts of data but lacked the staff time and/or expertise to analyze the data for program improvement.

6. Sustainability and Looking to the Future Institutionalize SEL SEL strategies and activities as well as changes associated changes in school culture and climate can best be sustained if these are truly integrated throughout the school. SEL can be institutionalized by making explicit decisions to ensure these efforts become part of the fabric of the school itself. Here are some of the strategies the grantees are using to institutionalize SEL into their schools: •

Integrate SEL into school-wide plan/charter contract/school guiding documents.

•

Build SEL into existing structures in the school day to support explicit SEL instruction (advisory, guidance lessons, activity period, specialties).

•

Build a strong SEL team with representatives from lots of different areas within the school (administration, teachers, specialties). This way SEL is not dependent on just one or two champions.

•

Have teachers deliver their own SEL lessons rather than have a counselor or someone else deliver them. That way the teachers take ownership of SEL and reinforce it throughout the school day.

IMPAQ International

Pillars of Peace SEL Evaluation – Executive Summary July 2020

•

Many SEL tools are flexible enough to be integrated throughout the curriculum. Take the time to plan how to integrate tools such as p4c, Choose Love, and Second Step into core subject areas and ensure core teachers know how to do that well.

Continue to invest in professional development One of the benefits of the Pillars of Peace grants was providing resources for professional development (PD). Because PD time is limited and every year there are competing priorities, it is easy to let slide PD that is associated with work done under grants that have ended. Most schools have recognized that, especially given issues of staff turnover, it is essential to find ways to continue PD beyond the grant if SEL programming is to be sustained. Grantee suggestions include: •

Train teachers to be in-house "trainers" so that you don't need to pay for outside PD every year.

•

Make sure to provide PD beyond a small group of teachers. Get the other adults on campus trained as well. Then the adults can all support each other and continue to reinforce the learning.

•

Educate all new faculty and staff about the SEL program during their onboarding process. Require new faculty to take advantage of related online courses. Build in structures for teachers to cross teach each other.

•

In addition to more formal PD, create opportunities for the entire staff to come together and create measures, milestones and expected student learning outcomes. Integrating SEL specific PD days into the regular PD calendar gives time for the faculty and staff to work on their SEL goals and to continue to adapt or progress in their endeavors.

Other sustainability strategies •

Use data effectively — Grantees recommended a variety of strategies for using the data effectively, including administering climate surveys at the end of first semester to provide feedback during the year, using the climate surveys to target SEL efforts to domains that need improvement, and using evaluation results to document successes when seeking funding.

•

Seek additional funding — Most grantees are looking for additional funding to continue and expand their SEL practices. Even schools that have the materials and resources they need to continue their current SEL programming have identified the need for PD or for resources to expand their current efforts.

•

Continue to grow — Given that growth mindset and commitment to learning become part of a school’s culture, it is essential to continue to grow and expand their SEL practices over time.

•

Continue participating in the larger SEL community — Grantees greatly appreciated participating in the PoPH cohort gatherings, and the opportunity to discuss the SEL work they are doing and to learn from the experiences of others. Some even considered this to be one of the most valuable components of this grant program. They hope to continue to find ways to network with other likeminded schools.

IMPAQ International

Pillars of Peace SEL Evaluation – Executive Summary July 2020

1. Introduction This report describes the evaluation of the Hawai‘i Community Foundation’s Pillars of Peace Hawai‘i initiative conducted by IMPAQ International, LLC, focusing on the implementation of evidence-based social and emotional learning (SEL) programs in 11 schools across the state of Hawai‘i. The report is presented in three volumes. Volume I describes the evaluation and summarizes our findings. Volume II provides a brief profile of each of the participating schools as well as other appendices referred to in the first volume. Volume III presents case studies of four of the schools, providing additional detail about their implementation experience and evidence of success.

What is the Hawai‘i Community Foundation’s Pillars of Peace SEL Hawai‘i Initiative? “Peace does not mean an absence of conflicts; differences will always be there. Peace means solving these differences through peaceful means; through dialogue, education, knowledge, and through humane ways.” — His Holiness the Dalai Lama

This quote captures the challenges schools face in addressing social and emotional learning and exemplifies the core objectives of the Hawai‘i Community Foundation’s (HCF) Pillars of Peace Hawai‘i (PoPH) SEL initiative. Conflict and differences are inevitable in our school environments; how students and staff choose to approach these situations can directly affect the health and educational outcomes for all students and their communities. Beginning in Fall 2016, the PoPH Social and Emotional Learning (SEL) grant program funded 11 public, charter, and private schools across the State of Hawaiʻi for three years to: •

Test and demonstrate the viability and efficacy of evidence-based, best-practice SEL programs.

•

Provide financial and professional development (PD) support to educators, schools, and/or community organizations to implement quality SEL programs.

•

Teach SEL skills to all participating students to grow more compassionate and mindful youth and to reduce aggressive and negative behaviors in the schools; and

•

Create a safer and supportive school environment, and encourage learning, compassion, mindfulness and empathy. “…the process through which children and adults acquire and effectively apply the knowledge, attitudes, and skills necessary to understand and manage emotions, set and achieve positive goals, feel and show empathy for others, establish and maintain positive relationships, and make responsible decisions.”

What is Social and Emotional Learning? SEL is a broad term often used to describe non-cognitive skills that can ultimately influence academic achievement. It also encompasses and builds the concepts of character education, emotional intelligence, bullying prevention, and positive behavioral supports. The Collaborative for Academic, Social, and IMPAQ International

Pillars of Peace SEL Evaluation Report July 2020

Emotional Learning (CASEL) 1 defines SEL as including five components: (1) self-awareness, (2) selfmanagement, (3) social awareness, (4) relationship skills, and (5) responsible decision-making. When implemented well, evidence-based SEL programs have a documented record of significantly improving school climate as well as studentsâ&#x20AC;&#x2122; behavior and academic performance. While some SEL programs provide classroom curriculum that focuses on these five domains of SEL skills or competencies, others focus more directly on school climate, with interventions targeting the adults or the school as a whole, or specifically targeting students with identified needs.

Which schools received PoPH SEL grants? HCF awarded grants to 11 different schools to implement SEL programming. Most had at least some past experience with SEL, but only a few had significant SEL programming already in place. The map in Exhibit 1 shows the geographic distribution of the participating schools. Most of the grantees are located on Oâ&#x20AC;&#x2DC;ahu, with four located on neighbor islands.

Exhibit 1. Map of HCF PoPH SEL Grantees

Collaborative for Academic, Social and Emotional Learning, 2013 CASEL Guide: Effective Social and Emotional Learning Programs, 2012. http://www.casel.org/social-and-emotional-learning/ 1

IMPAQ International

Pillars of Peace SEL Evaluation Report July 2020

Exhibit 2 shows the types of schools funded, which include four Exhibit 2. Types of Grantee Schools regular public schools, four public charter schools, one Hawaiian 1 immersion school, and two private schools. One of the charter schools offers both an English language program and a Hawaiian language 2 4 immersion program. Just as the grantees vary in terms of type of school and location, they also vary greatly in terms of grade levels. As shown in Exhibit 3, some range from K-12 or pre-K–12, four are elementary schools and one is a middle school.

Exhibit 3. Grade Levels of Participating Schools School

Pre-K

Ānuenue ‘Ele‘ele



Ka‘elepulu Kualapu‘u Laupāhoehoe St. Andrew’s

  

University Lab Volcano Waikīkī TOTALS

Public Charter

Private

Hawaiian Immersion

1st

2nd

3rd

4th

5th

6th

7th

8th

9th

10th

11th

12th

 



 



   

  

‘Ewa Makai HNL Waldorf

Regular Public

       

      

How was the Initiative evaluated? As shown in the conceptual overview below (Exhibit 4), the evaluation began with an understanding of schools/grantees and the students, families, teachers, and communities they serve. It is critical to understand the context within which SEL programs are implemented in order to accurately capture implementation lessons and examine outcomes. The middle of the framework describes the inputs or the SEL program elements. This might be thought of as the “intervention” we are examining with this evaluation. SEL programs vary across schools, but most include some common elements such as teacher/administrator trainings, in-class lessons, and integration with academic instruction. Some grantees include teachers’ own SEL skills as a core part of their SEL intervention. The implementation study examined these elements and how they were carried out in schools. The boxes in blue represent the intended outcomes of SEL programming. While some models might consider one or another of these as an intermediate outcome contributing to another outcome (such as considering changes in behavior to be a contributing factor to school climate, or SEL skills as a contributing factor to behavior) this model sees all three of these as interrelated – each contributing to the others. Working with schools/grantees, we collected outcomes data that showed changes in school IMPAQ International

Pillars of Peace SEL Evaluation Report July 2020

climate, as well as student behaviors and, in some schools, SEL skills. We also analyzed academic achievement data from the Hawaiâ&#x20AC;&#x2DC;i Department of Education (HIDOE) for public schools and public charter schools. Since schools had different specific goals for their programs, only a few grantees may have collected data for each measure, so our analysis focused on: 1) within-school changes over time; and 2) outcomes across groups of schools that collected a common outcome measure.

Exhibit 4. Conceptual Overview of the Evaluation

For this evaluation, we collected and analyzed primarily aggregate data rather than individual studentlevel data, wherever possible. That is, the outcomes analysis focused on school-level data, with the school as the unit of analysis, with some sub-group disaggregation, where possible, such as comparing elementary, middle and high school students, or comparing survey responses from students, teachers and parents.

Evaluation Questions In this section we present the revised evaluation questions that guided this evaluation. Based on discussions with grantees and HCF, several additional evaluation questions were identified, including questions about how good of a fit schools found the SEL programs to be, and their plans for sustaining SEL programming beyond the grant. It is important to note that the key evaluation question for this study was not whether or not the SEL interventions were effective. Since grantees were required to implement evidence-based SEL approaches, research had already shown these interventions to be effective, at least IMPAQ International

Pillars of Peace SEL Evaluation Report July 2020

in the context in which they were tested. Rather, the evaluation questions addressed in this report are included in Exhibit 5 below.

Exhibit 5. Matrix of Study Questions by Data Sources Data Sources Doc Review

Evaluation Questions Implementation 1. What SEL competencies, student behaviors and school climate concerns were being targeted by grantees? 2. What SEL programs were being implemented by grantees? 3.

What were the grantees’ key activities?

How was successful implementation of SEL programming affected by school conditions? What conditions helped and what conditions impeded successful implementation?

What were the project designs and curricula that grantees planned to use? How did actual program implementation match the intended design?

What types of resources were needed to implement high quality SEL programs? What were teachers’ initial perceptions of the SEL programs? How did these change over time?

7. 8.

What challenges were encountered by schools/grantees in implementing the SEL programs, and how were they addressed?

What promising practices emerged in design or delivery of SEL programming, including teacher and staff training? 10. How good of a fit were the programs selected by the grantees for meeting their school's needs? What adaptations did they make to improve the fit? 11. How appropriate were the SEL programs that grantees implemented for the local Hawaiian culture? What efforts did grantees make to adapt these programs for greater cultural relevance? 12. How can grantees sustain their SEL programs? What plans did they have in place for addressing sustainability?

Admin Data

Survey Data

Interviews



Observations



 



Outcomes 1. Behaviors: How did the numbers of behavioral incidents change after SEL programs were implemented? 2. School Climate: How did school climate change after implementing SEL programs? 3. Academic Achievement: How did student academic achievement change after implementing SEL programs? 4. SEL Skills: How did individual student SEL skills change after implementing SEL programs?



5. What other kinds of outcomes did grantees report?

IMPAQ International



 



Pillars of Peace SEL Evaluation Report July 2020

Data Sources Exhibit 5 above also indicates the data sources that were used to address each of the evaluation questions. Briefly, these data sources included the following: •

Document review — The IMPAQ team gathered and reviewed a variety of documents from grantees that provided data about the design, implementation, and documentation of each grantee’s SEL program. Examples of relevant documents include grant applications, program descriptions, progress reports, evaluation reports, and other program documentation.

•

Administrative data — IMPAQ used administrative data from the HIDOE for regular public schools and some charter schools including: ―

School Quality Survey (SQS) and Tripod Survey (Tripod) (school climate data)

―

Strive HI data (student academic achievement)

―

Student Status and Improvement Report (SSIR) data (absence and attendance data)

We recognize that attendance, absence and incident data are limited proxies for student behavior. However, since they were available for almost all of the schools, we did track them over time. •

School Climate Surveys — Regular public schools and most charter schools administered a School Quality Survey (SQS) distributed by HIDOE. In Year 2 the SQS was replaced by the Tripod, along with a few items retained from the SQS. Grantees all agreed to provide school climate data, either using the HIDOE survey or some other instrument. We worked with individual schools to assist them in deciding what types of data to collect and with a group of four schools administering the Education for the Future (EFF) school climate survey. IMPAQ provided guidance to the EFF schools in logistics of survey administration, and modifications and additions of selected survey questions. For example, the group selected and agreed on the wording of a question asking about student’s race and identified some additional survey items to supplement the EFF core items to include respect for diversity and cultural identity.

•

Phone interviews — IMPAQ conducted a round of telephone interviews in the fall of each year to learn about grantees’ SEL implementation plans for the coming school year.

•

Site visit interviews — The IMPAQ team conducted a round of site visits to the schools in the spring each year to learn about how the program was implemented during the year and to learn about challenges and how they were addressed, factors affecting effective implementation, and lessons learned. We also used these interviews to learn about topics such as cultural considerations, alignment with the Nā Hopena A‘o (HĀ) 2 framework, and plans for sustainability.

•

Other Surveys — Several schools developed their own survey instruments or used surveys that accompany their SEL programs to capture the outcomes that were important to them such as: ―

Ke Kula Kaiapuni ‘o Ānuenue (Ānuenue) developed their own Hawaiian language student pre/post survey and their own English language teacher survey. Both had quantitative and qualitative elements.

IMPAQ International

Pillars of Peace SEL Evaluation Report July 2020

•

―

‘Ele‘ele developed a short qualitative teacher survey about p4c that asked about whether the program is beneficial to students, beneficial to the teachers, and what they learned or gained.

―

Kualapu‘u developed a student survey that focuses on where in the school students feel safe, which cool-down strategies they use, whom they go to for help, how they resolve problems, whether they have friends, and their perception of rules, discipline, respect at the school.

―

Laupāhoehoe administered the Second Step Teacher Survey, which asks about various aspects of curriculum implementation.

―

St. Andrew’s administered the Mindsets, Essential Skills, and Habits (MESH) middle and high school student surveys (used by California’s CORE districts) along with the RULER Teacher survey.

―

Waikīkī Elementary developed Pre-Post SEL Student Growth surveys and a year-end Resiliency Theory (RT) Attribution Survey and distributed them to students, teachers and parents.

Observations — During site visits, to the extent possible, we conducted observations of SEL program activities and strategies being carried out at school sites. Some examples of the activities we observed include: ―

Staff meeting about incorporating Tribes strategies at Ānuenue School

―

Sixth grade graduation program at Ānuenue, displaying the culmination of students’ SEL learnings by presenting gifts they made for their parents, performing hula and mele for their families, reciting their genealogy to their family, and eating a meal prepared by the students

―

P4c (Philosophy for Children) Class Council in a 4th grade class at ‘Ele‘ele

―

An 8th grade social studies class at ‘Ewa Makai Middle School discussing assignment to write about an issue emotionally important to them (such as domestic violence/sexual assault) and connecting that to the call to action projects they were working on in English

―

7th and 8th grade classes at ‘Ewa Makai making ti leaf lei and posters for “peace gathering” honoring school shooting victims

―

Teacher/staff training in the No Bully program at Honolulu Waldorf School

―

School-wide faculty and staff professional development and reflections at Ka‘elepulu Elementary School

―

Forgiveness and Courage lessons of the Choose Love Enrichment Program in a 4th grade class at Ka‘elepulu

―

Getting Along together (GAT) Sharing Circle (I feel happy because…; I feel sad because…) and a listening game and Mirror in a first-grade class at Kualapu‘u Charter School

―

Counselor working new “chart” system of intensive support, and meeting of Hawaiian Immersion teachers discussing SEL at Kualapu‘u

―

A staff meeting at Laupāhoehoe Elementary school where staff discussed the results of their schoolwide school climate survey

―

A teacher reflection meeting at University Laboratory School

―

A teacher SEL curriculum integration planning meeting at University Laboratory School

―

A 2nd grade class developing a charter (a component of RULER) around a class project chickens at St. Andrew’s

―

Second Step lessons at in middle school classrooms at Volcano School of Arts & Sciences

IMPAQ International

Pillars of Peace SEL Evaluation Report July 2020

―

A student leadership class at Volcano where the class discussed peer mediation and how to market peer mediation to their fellow students

―

P4c circle in 4th grade class at Waikīkī Elementary School

•

Implementation data from grantees — Grantees documented program implementation using a variety of activity logs and formats for capturing teacher and staff reflections. The IMPAQ team worked with grantees on ways to summarize and report implementation data. We received many teacher and staff reflections which were very useful in the qualitative analysis for providing examples of factors affecting implementation and anecdotal evidence of program effects.

•

Outcome data from grantees — All grantees collected and reported at least two types of outcomes data: 1) a school climate survey, and 2) behavioral outcomes (i.e. attendance, absences, behavioral incidents). Here we present aggregate data by school or across all schools. While it was not the intent of this evaluation to collect and analyze student-level data, during Years 1 and 2 the IMPAQ team received individual student, teacher and parent level school climate data from the schools that administered the EFF. We analyzed the data for each school and provided each with summary exhibits by domain and grade level. We also provided support to some schools in interpreting the results and presenting the results to school staff. We also conducted a limited amount of analysis of MESH data from St. Andrew’s and pre-post survey data from Waikīkī Elementary. We have also tried to support schools in undertaking their own analysis of the data, which a few have really embraced.

For various reasons including data availability, accessibility, and reporting inconsistencies, not all 11 schools submitted each type of data. Exhibit 6 on the following page depicts the data sources available for each school. In some cases, schools did not collect or report each source for all three years.

What were the major limitations in the study? •

Limited evaluation resources — While there are always funding limitations when evaluating social and educational initiatives, it can be challenging to ensure the evaluation design is appropriate to the size of the initiative being evaluated. In this case, even though some grantees were quite ambitious in their desire to transform school learning environments, they received relatively small grants, ranging from about $50,000-$150,000 over the three-year period. This meant grantees could not be expected to launch large-scale data collection efforts. The evaluation design, therefore, needed to build to the maximum extent possible on data grantees were already collecting or that were available through HIDOE.

•

Limitations in measurement — Perhaps the greatest challenge with this evaluation was selecting appropriate measures both flexible enough to be used in different school contexts and for different SEL programs, while consistent enough for summarizing across the grantees. Initially, some schools had a fairly traditional pedagogical view of SEL as another area of classroom content and that they would simply test student knowledge and skills to measure success. Through exploring different SEL skills assessment tools and developing a deeper understanding of systemic changes in teaching and learning associated with SEL, most grantees felt school climate and student behavior were better indicators, although they did not all use the same instruments to measure these outcomes.

•

Limitations in the depth of the study — As often is the case when attempting to tell the story of a program being implemented in a real world setting, there are many layers of detail and nuance that cannot easily be captured and documented within traditional evaluation resources. For this reason, we conducted four case studies, which allowed us to tell at least some of the grantees’ stories in more detail.

IMPAQ International

Pillars of Peace SEL Evaluation Report July 2020

Exhibit 6. Data Collected by Each School School

Doc Review

Phone Interviews

On-site Interviews

Observation

Survey Data TRIPOD/ SQS

EFF

Behavioral Incidents

Attendance

Academics (STRIVE HI)

Other

Ānuenue



‘Ele‘ele



‘Ewa Makai



HNL Waldorf



Ka‘elepulu



Kualapu‘u



Laupāhoehoe



St. Andrew’s



University Lab



Volcano



Waikīkī



IMPAQ International



 

Pillars of Peace SEL Final Evaluation Report July 2020

2. Program Implementation In this chapter, we address six key evaluation questions about program implementation: 1) What SEL competencies, student behaviors and school climate concerns were being targeted by grantees? 2) What SEL programs were being implemented by grantees? 3) What were the grantees’ key activities? 4) What changes or adaptations did grantees make from their original intended design? 5) What factors or conditions were needed to implement high quality SEL programs? 6) How appropriate were the SEL programs that grantees implemented for the local Hawaiian culture and how did grantees adapt these programs for greater cultural relevance?

2.1 Description of Grantees’ SEL Programs What SEL competencies, student behaviors and school climate concerns were being targeted by grantees? Each grantee targeted different SEL competencies, student behaviors and/or school climate concerns. Each had specific reasons for seeking grant funds to develop or expand SEL programming at their school. Grantees described a variety of goals for their SEL programs including: •

Building empathy in students

•

Reducing aggressive behaviors (including bullying)

•

Teaching conflict resolution and anger control

WHY SEL?

•

Building students’ responsible and ethical behavior skills

•

Developing compassion

•

Developing mindfulness

•

Building respect for each other and a sense of community

Schools generally look to SEL to reduce aggressive behavior and bullying and increase positive, empathetic behaviors to create a save and supportive environment for children and adults alike.

•

Reducing cyber bullying

•

Developing positive peer relationships

•

Teaching self-advocacy skills

•

Preparing students to be contributing members of their communities and the world

Given the different issues grantees were addressing or goals they targeted, they also articulated different program objectives. Some examples of grantees' objectives include: •

Continuing to implement SEL program in a consistent manner, making adjustments as needed based on findings from Years 1 and 2

•

Continuing to build faculty capacity in the concept of SEL and in teaching SEL programs

•

Expanding SEL programs from selected grades or classes to the whole school

•

Rebuilding SEL teams who lost members to attrition

•

Collecting more meaningful data, including pre/post surveys

•

Carving out time for SEL teams to meet

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

•

Compensating teachers for time spent preparing SEL lessons or planning for SEL implementation

•

Reducing behavioral incidents and improving academic growth

•

Focusing on prevention by creating a school culture where children feel safe and supported and learn what it means to be a real friend

•

Translating SEL concepts into the Native Hawaiian language to teach them in alignment with Native Hawaiian ways of knowing and building relationships

•

Sharing the benefits of SEL and specific programs with the wider educational community in the state

What SEL programs were being implemented by grantees? As noted in the Grantee Profiles in Volume II: Appendix A, grantees implemented a variety of SEL programs. In many cases, the grant-funded programs built on SEL initiatives already in place, including some that began through grant funding provided through a 2015 pilot program conducted by HCF that explored the viability of SEL programs for Hawai‘i schools. The overall goal for grantees was to create a safer and more positive learning environment through the SEL programs. Most grantees implemented school-wide programs. Some started with selected grade levels and/or teachers, with plans to expand programs to the entire school over time. Using HCF grant funds, schools adopted a wide variety of evidence-based SEL programs, including: •

Free-standing SEL lessons to provide explicit, step-by step instructions to teach students social and emotional competencies across the five core competencies

•

General teaching practices to create classroom and school-wide conditions that facilitate and support social and emotional development in students

•

Integration of skill instruction and practices to support SEL within the context of an academic curriculum

Exhibit 7. SEL Programs Implemented by Grantees SEL Program

Description

CHARACTER COUNTS!

A student development framework based on behavior change theory to instill academic, social, emotional values, mindsets and character traits to help students succeed. Emphasis is placed on establishing a positive school climate.

CHOOSE LOVE ENRICHMENT PROGRAM

Using positive psychology, mindfulness, neuroscience, and character values, the program focuses on four important character values – Courage, Gratitude, Forgiveness, and Compassion in Action – which cultivate optimism, resilience and personal responsibility.

FACING HISTORY AND OURSELVES

Through historical analysis and the study of human behavior, the program heightens students’ understanding of racism, religious intolerance, and prejudice. It helps students relate history to their own lives and promotes greater understanding of their roles and responsibilities in a democracy.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

SEL Program

Description

GETTING ALONG TOGETHER (GAT)

Students learn thinking and cognitive skills, emotional management, and interpersonal/social skills. These skills and strategies create a peaceful school environment where students are empowered to manage their own behavior, decrease conflict, and increase receptivity to learning.

GROWTH MINDSET

A Growth Mindset is the belief that intelligence can be developed. Students understand they can get smarter through hard work, the use of effective strategies, and help from others when needed. Students with a growth mindset are more motived to take on challenging work, persist in the face of setbacks, and achieve at higher levels.

HABITS OF MIND

Set of 16 problem-solving, life related skills necessary to effectively operate in society. The skills promote strategic reasoning, insightfulness, perseverance, creativity and craftsmanship.

INDIVIDUAL AND GROUP MENTORING

Mentoring initiatives based on Resiliency Theory add powerful protective factors to the lives of students who need an extra dose of SEL instruction. A main principal of Resilience Theory is that children want to belong to a group where they are cared for and appreciated. Targeting small groups of identified students, mixed-grade mentoring groups are built around extra-curricular activities. Adult leaders try to create a caring adult connection for mentored students.

MINDFULNESS

This philosophy helps educators and students maintain a moment-by-moment awareness of thoughts, feelings, bodily sensations, and surrounding environment to reduce stress and improve social connections and learning.

NO BULLY

A step-by-step process and set of kindness and empathy interventions to prevent and stop bullying and cyberbullying in school and after-school programs. The program helps to build an inclusive school culture where every student is accepted for who they are.

PHILOSOPHY FOR CHILDREN (p4c)

The underlying principle of p4c is for young people to experience rational and reasonable dialogue about things that matter to them and their teachers, working together in a â&#x20AC;&#x2DC;community of inquiry.â&#x20AC;&#x2122; The aim is for each child to become clearer, more accurate, less self-contradictory and more aware of other arguments and values.

PEER MEDIATION

Trained student mediators working in pairs rotate shifts on the playground to scout out and address potential student conflicts in need of remediation.

POSITIVE DISCIPLINE

The program teaches important social and life skills to help young people become responsible, respectful, and resourceful members of their communities. Positive Discipline is based on the understanding that discipline must be taught, and that discipline also teaches.

RESPONSIVE CLASSROOM

An approach to teaching focused on engaging academics, positive community, effective management, and developmental awareness to help educators create safe and joyful learning communities where students develop strong social and academic skills, and every student can thrive.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

SEL Program

Description

RESPONSE TO INTERVENTION

Response to Intervention is an academic intervention that provides early, systematic and intensive assistance to students who are at risk for or already underperforming against age-level standards. The approach uses universal screening, early intervention, frequent progress monitoring and intensive instruction for struggling students.

RESTORATIVE JUSTICE PRACTICES

Restorative Justice is a powerful approach to discipline that focuses on repairing harm through inclusive processes that engage all stakeholders. The process shifts the focus of discipline from punishment to learning, and from the individual to the community.

ROOTS OF EMPATHY

An experiential learning program featuring a neighborhood baby and parent/caregiver who visit the participating classroom about once every three weeks over the school year. Through guided observations and interactions with the baby, and pre- and postfamily visit sessions, the children gain essential empathy skills that have been shown to reduce aggressive behaviors.

RULER

Recognizing, Understanding, Labeling, Expressing, and Regulating Emotions (RULER) provides tools to enhance individuals’ ability to understand and regulate their own emotions and to consider and empathize with the feelings of others. The program requires that all adults in the school be trained on their emotional intelligence before delivering the program to the students at the school.

SCHOOL CONNECT

Sixty-lesson multimedia curriculum designed to improve high school students’ SEL and academic skills and strengthen relationships among students and between students and teachers. Lessons focus on social awareness, self-awareness, self-management, relationship skills, and responsible decision making.

SECOND STEP

The program includes interactive strategies such as direct instruction, video modeling, partner and group discussion, behavioral skill practice, and interactive homework assignments. Students learn to work together in groups to practice empathy, communication, and problem-solving.

SEVEN HABITS FOR HIGHLY EFFECTIVE TEENS

Utilizing interactive exercises, humor and videos, students learn how to gain greater control of their lives and build relationships high in trust. The program shows educators how to build students’ self-confidence and interpersonal skills, elevate student achievement, and reduce discipline problems.

SHARED INQUIRY FOR READING

A method of teaching and learning that enables students to explore the ideas, meaning, and information found in text. Shared Inquiry promotes an intellectually stimulating interpretative discussion of a work —a group exploration of meaning that leads to engaging and insightful conversation. It helps participants read actively, articulate probing questions about the ideas in a work, and listen and respond effectively to each other.

SIX MOST IMPORTANT DECISIONS YOU’LL EVER MAKE

A program that gives teens strong advice to make informed and wise decisions about six major challenges they face: gaining self-esteem, dealing with parents, making friends, being wise about sex, coping with substances, and succeeding at school and planning a career.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

SEL Program

Description

S.P.R.I.N.K.L.E.R.S.

In the Student Providers of Responsiveness Involving Kindness, Love, Empathy, and Respect through Stewardship program (S.P.R.I.N.K.L.E.R.S.), students greet incoming, students, staff, and visitors at the front entrance of the school. The children develop positive social and emotional skills and contribute to creating a safe environment.

SUCCESS FOR ALL

Classroom programs based on cooperative learning frameworks that guide instruction in critical academic and social skills. The programs are designed to instill a love of learning in children by giving struggling students the means to achieve literacy.

TOOLBOX

TOOLBOX is a metaphor directing children to the inner “tools” that already exist inside them to help them strengthen resilience, self-mastery, and empathy. The program gives children, teachers, parents, and communities a common language and tools to form a cohesive, collaborative, and caring community.

TRAUMA-INFORMED PRACTICE

In a trauma-informed school, the adults in the school community are prepared to recognize and respond to those who have been impacted by traumatic stress, which can arise from bullying, dramatic weather events, school shootings, and the daily exposure to family trauma such as divorce, homelessness, incarceration, and suicide. Students are provided with clear expectations and communication strategies to guide them through stressful situations, with the goal to not only provide coping tools but to create an underlying culture of respect and support.

TRIBES LEARNING COMMUNITIES

A process for creating a positive school or classroom environment to help students achieve, because they feel included and appreciated, are respected, and are actively involved in their own learning. The program has been adapted to be culturally and linguistically aligned with Hawaiian immersion education.

VOICES LITERATURE AND WRITING CURRICULUM

A literacy and social development program that helps teachers lay the groundwork for student achievement and lifelong learning through high-quality, carefully selected trade books. The program develops character and equips students with core social skills while learning deep comprehension of the texts.

What were the grantees’ key activities? In addition to student-centered activities, grantees invested in training and professional development opportunities for faculty and staff, and schools conducted outreach activities to educate families and their communities on their SEL programs. Below are some examples of each of these kinds of activities: Classroom Based or Student-Focused Activities

•

Several schools chose SEL programs that included lessons delivered through regular classroom instruction such as Second Step, Tribes, Voices, Getting Along Together, and Choose Love.

•

One school created a hybrid SEL/Native Hawaiian program to reflect the school's commitment to Native Hawaiian culture and practices, creating an overall Native Hawaiian Values System as the hybrid program's foundation.

•

Another school expanded the implementation of the SEL program to the Hawaiian Immersion portion of the school.

•

School administrators took four students to the InspirEd Youth Engagement and Leadership Lab, where they learned how to plan and execute a "compliment wall", a project where each K-12

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

student received kind and encouraging words from each other. The four students created and developed their compliment wall for the school, and it was inspirational for everyone to see and read. •

One school increased individual and group counseling services for students.

•

Another school implemented a trauma-informed practice of conducting regular check-ins with students to improve relationships and offer additional SEL support.

•

Nine new teachers at one school started their own p4c classrooms.

•

One school has started to help students apply their SEL skills outside the classroom.

Teacher/Staff Trainings and Activities

•

Five teachers traveled to another school to observe p4c in action. In addition, three p4c trainers visited to observe the program in action and provide targeted teacher training.

•

Faculty/staff attended the Restorative Practices Annual Conference to learn how to improve their handling of student discipline and conflict resolution.

•

One school convened a Choose Love Conference and introduced the program to 175 educators around the state.

•

A school provided SEL professional development to remaining playground, custodial, and cafeteria staff who had not yet received the training.

•

Professional development opportunities for SEL were provided to a Response to Intervention Coordinator, a school counselor, the school wide SEL Coordinator and three teachers.

•

Monthly, paid afterschool meetings of SEL teachers were held to support each other and share best practices.

•

SEL Coordinator hosted a Chat and Chew session for faculty, providing a way for teachers to meet informally during lunch to discuss various SEL-related topics.

•

A middle school counselor coached teachers and faculty in school SEL programs, including Positive Discipline, Responsive Classroom, Second Step, and Mindfulness.

•

A school shared SEL practices at several conferences and to approximately 200 education visitors to the school.

•

Several schools worked to develop methods of gathering more meaningful information on the ways teachers are delivering SEL lessons in their classrooms.

Parent and Community Activities

•

Key faculty and school administration hosted and presented several RULER-related outreach events for the professional development of educators, parents, and the community.

•

Schools held several grade-level-focused parent education events on the SEL program.

•

A school conducted outreach to families via short videos on their SEL program posted to the school's website.

•

School held two parent workshops on the No Bully program, helping families distinguish between bullying and conflict, and stressing the importance of practicing SEL skills at home. Parents

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

requested more educational opportunities, and the school brought in a speaker from the Roots of Empathy program to introduce this new program to families. •

School held a parent session to discuss bullying and safety. 80 participants joined in the robust and inclusive discussion. Representatives from local health facilities and the police department also attended.

•

School hosted second annual TOOLBOX Café event for parents, students, teachers, and the greater school community, drawing about 20 participants. The school also planned to hold a Parenting with TOOLBOX event, but it was cancelled due to lack of response.

2.2 Changes or Adaptations from Grantees’ Original Design Most grantees modified their SEL programs at least to some extent, based on their implementation experience. As SEL started to take root at the schools, faculty and staff continued to tailor programs to meet their own unique needs and culture. The following are examples of some of the changes:

Changes to School Culture •

Ānuenue: The school's original plan was to implement Tribes throughout their school, as it seemed to parallel the Native Hawaiian values of character building, self-management, emotional regulation, etc. But as a Hawaiian immersion school, teachers felt the program favored Western perspectives and did not fit well with a Hawaiian context. In review of the SEL and HĀ frames, Ānuenue picked the term Kūkulu Pilina or “relationship building”, to describe the school wide focus of their SEL initiative. This approach, together with new insights from learning about trauma-informed practices, served to create a different kind of school culture. While still embracing Hawaiian cultural values and practices, staff began to shift away from a traditional adult/child relationship to a greater understanding of students experiencing life crises who looked to the school as their safe place and ʻohana. They began to emphasize a Kuaʻana/Kaikaina (older sibling/younger sibling) framework, building in opportunities for older students to teach younger students, which also contributed to significant changes in school culture.

•

‘Ele‘ele: The SEL Coordinator felt p4c was a good fit with the school. She could not imagine it being a bad fit anywhere. Over the past several years, the school's Principal brought in a number of inquiry-based programs. The inquiry-based nature of p4c dovetailed nicely with their Shared Inquiry reading program, which uses the same discussion protocols as p4c. Additionally, the Next Generation Science Standards are inquiry-based and the lessons lent themselves well to p4c discussions. Social studies was another subject that was becoming inquiry-based.

•

‘Ewa Makai: While the school was satisfied with 7 Habits and 6 Decisions, 'Ewa Makai discovered that finding SEL programs that best fit the needs of teachers and students was challenging. While the teachers liked the 7 Habits of Highly Effective Teens, some felt that it was a bit over the heads of their students. There did not seem to be many SEL programs specifically designed for the middle school grades that addressed the issues that middle schoolers are facing. Many of the SEL programs were designed for the elementary grades and could theoretically reach early teens, but these students rejected those programs as too babyish. While the school was comfortable and confident with their current SEL programs, they were open to other appropriate programs. One group of teachers decided to try out Second Step in SY(school year)2019-20 (the year after the 3year grant).

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

•

Honolulu Waldorf: While the school felt that their initial experiences with No Bully were positive, they found the program more of a reactive intervention and decided to include Roots of Empathy as a pre-emptive tool. They found both programs to be complementary to each other.

•

Ka‘elepulu: The school felt that Choose Love has been a great fit for their school, as evidenced by the total buy-in from staff and faculty. All adults in the school are on board with the reasons they are offering SEL to their students, and why they have adopted Choose Love.

•

Kualapu‘u: The school felt that GAT (Getting Along Together) has been a good fit as a foundational piece to their SEL program. It has given the entire school community a common language and tools for simple problem-solving. The program has given adults more empathy for the challenges faced by their students, which has uncovered the deep trauma that many of their students are experiencing. This led to the school's adding trauma informed practices to their SEL program in Year 2.

•

Laupāhoehoe: Laupāhoehoe had several curricula in their SEL program and felt that they were a good fit with their school because SEL and the HĀ framework are foundational pieces to their school culture. They expressed the perspective that “SEL is the plate, not an extra thing on the plate.”

•

St. Andrew's: Having implemented the RULER program throughout their school and having been exposed to additional SEL programs at various Schools of the Future conferences, St. Andrew's determined that RULER has been a good fit for the school, and that it was the base upon which to add other programs to suit the school and student needs (e.g., Mindfulness, Girls' Circle, Choose Love, YogaEd).

•

University Lab: The school felt TOOLBOX was generally a good fit with the school — especially in the elementary grades. The SEL leadership team investigated how best to address the implementation of SEL in the middle school, and to what extent TOOLBOX would be part of that solution. Middle school students resisted the program, deeming it babyish.

•

Volcano: While Second Step was a positive first step in implementing their SEL program, ultimately they were looking for a school-wide, scaffolded curriculum that would be culturally relevant to their students. They felt that while the content was relevant, much of Second Step delivery, including videos and worksheets, was not relevant to their students. Middle school teachers found that modifying examples in Second Step lessons when needed to make the content more culturally relevant was valuable for increasing student engagement in the lessons.

•

Waikīkī: Having implemented Habits of Mind and p4c at the school for more than 20 years, the school definitely felt these programs were a good fit for the culture of the school — in fact, they had become the culture of the school. The reason why teachers embraced Habits of Mind and p4c is in large part due to the fact that these programs incorporate both the cognitive and academic dimensions of the self. At Waikīkī School SEL is not an add-on — it is part of the fabric of the culture of self. The mentoring programs developed under the grant to target students needing more support proved to be successful and a good fit for the school.

Middle school faculty and staff raised concerns about what they perceived as a dearth of SEL programs specifically designed for middle school students, who reject many elementary-based programs as childish. Several schools found themselves adapting lessons to make them more age appropriate for middle school.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Addition of More Intensive SEL Interventions Several schools realized that some of their students faced one or more significant life crises (homelessness, abuse, family death/incarceration) which limited their SEL growth. These schools sought out and embraced more intensive SEL interventions, such as trauma-informed practices and resiliency theory-based mentoring, to better reach these troubled students and provide additional emotional support. •

After learning about trauma-informed practice one school implemented regular, informal checkins with students to build stronger relationships students by increasing positive adult interactions.

•

Another school found they had a large number of students with serious mental and emotional issues who needed support beyond what the school could provide. They contracted with a licensed counselor to run a Boys Group program for their Tier 2 boys.

•

One school expanded their SEL program to include resiliency theory-based mentoring to add protective factors to their students' lives and to provide additional, intense doses of SEL instruction through individual and group mentoring.

•

Several schools invested in professional development for counselors and/or teachers in traumainformed practices.

•

Several schools increased their counseling staff and reassigned various responsibilities among staff to ensure adequate individual counseling time for students who needed it.

Changes to School Policies and Structures •

One school introduced a new School Wide Plan featuring SEL as a key element. Incorporating SEL into the School Wide plan ensured SEL would continue at the school beyond the life of the grant.

•

Another school updated its behavioral incidents policy from a system of strikes to a demerit system and implemented less punitive discipline options, such as time out in a meditation or sensory room.

•

One K-8 school moved the Vice Principal’s office from the elementary campus to the middle school campus to ensure that at least one school administrator was always present at both campuses.

•

Several schools changed their class schedule to ensure dedicated time for SEL instruction was set aside at a regularly scheduled time each day or each week.

•

Two schools added a data manager to improve data collection and analysis efforts.

•

Another school included their SEL program into their School Vision.

Staffing Changes Some grantees experienced staffing changes with new administrators or key SEL players, which affected the implementation of their SEL programs. •

Ānuenue — The school transitioned to a new Principal midyear, and their Vice Principal, part of their core SEL team, was on medical leave for most of spring semester in Year 2.

•

‘Ewa Makai — The Principal who was there when the school first started planning the SEL program retired, and the Vice Principal who championed the school’s SEL efforts also left the school. They

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

appointed a new SEL Coordinator in Year 2, and there was also significant turnover among teachers. •

Honolulu Waldorf — The school faced changes in school leadership as well as faculty turnover. Some of the key members of the SEL implementation team left for a variety of reasons including new job opportunities, moving away and having babies, leaving only one of five trained solution team coaches left. When the administrative director left, they had two temporary part-time directors and then finally a new director at the end of Year 3.

•

Kualapu‘u — Their loss of the school counselor at the end of Year 2 was a major challenge, as they were unable to find a new counselor in Year 3. The substitute counselor has a long history with the school and was helpful in working with the children most in need but would not be returning the following year. The loss of their SEL champion also meant there was no one at the school to provide SEL training to new teachers.

•

Laupāhoehoe — The school had a change of Director (Principal) who was replaced by an interim director. Their Vice Principal also changed, as did their PTA president, and they lost a grant administrator/writer and the high school grant coordinator. The loss of their data coordinator created challenges for collecting and reporting data. The loss of their financial manager also had an impact, as it created a delay in knowing their financial picture that caused professional development to be put on hold during Year 3.

•

St. Andrew's — They saw the loss of their school counselor in the middle of Year 2, and the leader of the RULER implementation team left the school at the end of Year 2. The new counselor was quickly trained in RULER. They also worked with incoming new teachers to do online training the summer prior to teaching in the fall.

•

University Lab School — Teacher turnover significantly affected their SEL implementation, especially in Year 3.

•

Waikīkī — In Year 2, the school unexpectedly lost two-thirds of their SEL team. The DOE hiring process took a lot longer than expected, so in Year 2 they were short-staffed for much of the year. In Year 3 they enjoyed the difference that continuity of staff made.

Teacher Training and Professional Development •

One school provided two days of professional development for their entire faculty. The school changed a previous requirement that non-educational staff had to attend all of the SEL training workshops, making such training optional if the workshops did not directly pertain to a person’s role.

•

Another school wanted to provide additional teacher professional development in their SEL program, but neither of the two institutions that provided the training offered it in SY2017-18.

•

One school decided to focus on expanding the role of its counseling staff to provide teacher training instead of continuing to invest in outside professional development.

2.3 Factors or Conditions That Supported Effective Implementation During site visits and phone interviews, SEL coordinators, school principals, teachers and other staff identified a number of factors they felt had supported effective SEL program implementation. Conversely, a lack or limitation of these supports presented challenges to effective implementation. Some schools IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

were able to adapt their program implementation to support these conditions. Others experienced major challenges as they struggled to ensure these conditions were in place on an ongoing basis. Many of the schools had to address significant turnover in leadership and identified the need for a strong champion and SEL coordinator as critical to success. In collaboration with the grantees, we have identified fifteen of these conditions: 1. Time

9. Careful Pacing

2. An SEL champion

10. SEL for Adults

3. Teamwork among teachers and staff

11. Reflections and Flexibility

4. An effective SEL coordinator

12. Continuity of staffing

5. Commitment or “buy-in”

13. Continuity of SEL practice

6. Professional development

14. Sufficient “dosage”

7. Administrative support

15. Sufficient counseling staff

8. SEL program resources

1. Time When asked “What does it take to implement SEL programs?” the most common answer was “time”. Time also came up frequently as a challenge or limitation to effective implementation. In not every case was the concern about implementation taking a large amount of time. For example, some interventions take very little investment of time. But even in cases where only 15 minutes was needed for an intervention, there were still logistics to consider, such as making sure teachers were comfortable releasing students for 15 minutes to participate in a “No Bully” solution team, or building in a 10-minute mindfulness activity into regular class time. Here are some ways that time was needed for effective implementation: •

Class preparation time — some of the SEL programs that involve classroom instruction required teachers to prepare the lesson and make copies of handouts

•

Time to plan the implementation

•

Time to build a team

•

Staff training and PD time

•

Coordination time

•

Dedicated time at staff meetings

•

Releasing students from class for solution teams or mentoring

•

Staff time outside of the classroom — such as staff time to run solution teams

•

Sufficient staff support to free up time for interacting with students

2. Champion Another key factor in effective implementation was the presence of a champion – that is, someone who was passionate about SEL implementation and motivated to help everyone stay focused. The key characteristics of a champion seemed to be: IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

•

Passion

•

Determination

•

Belief

•

Energy

•

Enthusiasm

•

Focus/attentiveness

•

Initiative

•

Reliability/consistency

The champion could be the Principal or other administrator, or it could be a teacher or counselor. The role of champion could also be two or three teachers or other staff who championed together.

3. Teamwork Teamwork seemed to be essential to effective implementation. In many schools, staff reported that teamwork increased or improved over time, as administrators, teachers and other school staff became more familiar with the SEL programs being implemented and SEL principles overall. Schools identified several aspects of teamwork or collaboration that were key: •

Coordination — Whether implementing a school-wide strategy or delivering SEL curriculum in the classroom, coordination was key to ensuring consistency. Often, teachers coordinated with other teachers in their grade level or subject area.

•

Communication — Examples of essential communication included:

•

―

Administrators communicating their commitment to SEL to students, teachers and parents

―

Teachers communicating SEL concepts clearly with students and parents, as well as communicating their commitment to other teachers and staff

―

Students communicating more effectively with each other and with teachers, using their SEL skills and tools

Camaraderie — While not a goal of this effort, many schools reported that effective teamwork and modeling SEL skills both contributed to increased camaraderie among teachers, among students, and across all players in the school.

4. SEL Coordinator Some schools built in an SEL Coordinator role from the beginning of their grant. Others recognized the need as implementation unfolded. In most schools with a designated formal SEL Coordinator, the role was not full time. In some schools, the Coordinator was a teacher, in others a counselor. In one school, the SEL Coordinator was an administrator. Here are the key roles for SEL Coordinators identified by school staff: •

Become the expert

•

Provide professional development

•

Support the teachers

•

Generate and sustain focus

•

Keep communication going

•

Generate and sustain enthusiasm

•

Prepare materials

•

Review data for program improvements

•

Teach curriculum

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

5. Commitment No matter which intervention grantees were implementing, no matter how simple or challenging, time consuming or “light touch”, grantees agreed that effective implementation required a strong commitment. Without that, an intervention might seem to be going smoothly, but gradually fade over time. Most grantees call this concept “buyin”, but some mentioned they do not like that term, because “buy-in” language implies “you sell, I buy.” But commitment is a free choice -- individuals commit because they choose to. Stakeholders who are committed will do whatever it takes to make it happen. Commitment was seen as important across a wide range of key players: •

Teachers

•

Other school staff

•

Counselors

•

Students

•

Administrators

•

Parents

Initially teachers’ buy-in seemed to be the most key, but over time, grantees found they were seeking to increase the commitment of all school adults, including administrators and staff. Schools used many different strategies to foster commitment including taking full advantage of PD available through the SEL program’s developer, ongoing reinforcement of commitment and principles during follow-up PD and regular staff meetings, making videos of students and teachers involved in SEL implementation, making time to celebrate successes, reviewing reflections and evaluation data for ideas about program improvement, and sharing lessons learned with other schools.

6. Professional Development (PD) Most grantees mentioned teacher training as being essential and often problematic, given that it involves teachers taking time away from the classroom. Even if there were resources for substitutes, some teachers were reluctant to interrupt their momentum in their classroom or curriculum, trainers might have had to come from the mainland, or they may already have other PD scheduled for the school’s scheduled PD days. In some cases, it was not feasible to get all staff together on the same day. Here are some of thoughts from grantees about PD as a key component of program implementation: •

Training was a key component for most programs – While some interventions, such as safe spaces and peace paths were fairly simple to implement, those that involved implementing specific curriculum or teaching strategies required an investment in training teachers and/or counselors.

•

It was valuable to include special-subject teachers and other school staff – Even if only regular classroom teachers were directly involved in program implementation, involving staff school-wide was important for reinforcing SEL strategies and improving overall school climate.

•

It was important to model SEL lessons and skills – Adult SEL skills and use of SEL strategies was just as or almost as important to implementation as those of the students. Modeling was an important teaching strategy; in several schools we saw Principals modeling SEL strategies for teachers at staff meetings.

•

For some SEL programs, full-day PD was extremely valuable – For schools targeting school-wide change, where it was possible to schedule school-wide PD, staff could develop a common language and understanding as well as peer support and camaraderie that helped support implementation.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

•

For some SEL programs, short PD sessions were very effective — PD could be built into regular staff meetings, where even a 5-minute exercise, a few minutes to touch base and check in, or structured reflections was very valuable for sharing SEL knowledge and maintaining ongoing SEL focus.

7. Support Another key factor affecting effective SEL implementation was the degree of support staff felt they received. Support came in many forms, but here we refer to concepts like encouragement, positive feedback, release time for preparation, visibility of the SEL programming within the school, etc. Teachers and other staff mentioned support from a number of different directions. Support from “the top” was the most frequently mentioned, but a strong sense of support was also reported as coming from other stakeholders: •

Principal and other administrators

•

Teachers

•

SEL Coordinator

•

Parents

•

SEL program staff/trainers

•

Community members

It was especially helpful that administrative leaders continuously articulated WHY the school was doing this work and identify how it was beneficial to the students, families and community. Supportive leadership was essential, as they were critical in finding and setting aside the time for the professional growth and development of faculty and staff.

8. Resources In addition to making sure adequate time was made available for effective program implementation, there were, of course, a variety of other resources needed for effective implementation. Most programs emphasized the important role of the grant funding in making SEL program implementation a reality. •

Grant funding — A few schools said it would be possible, but difficult, to implement SEL programming without grant resources. Some said implementation would have been unlikely or substantially delayed without grant funding.

•

Program or curriculum — Some schools had been implementing SEL-related activities without adopting a specific formal program or curriculum. However, the adoption of evidence-based programming provided an important resource for increasing the school’s focus on both implementation and on focusing on specific intended outcomes.

•

Lesson plans and teacher manuals — Classroom-based SEL programs provided valuable lesson plans for teachers to use. While some schools adapted these to specific grade levels, generally these were ready to use and considered an extremely valuable resource.

•

Book lists — In several schools, teachers were eager to reinforce SEL concepts through additional readings. Some programs provided lists of additional reading materials, and in some schools the teachers built reading lists collaboratively.

•

Kits — Some programs provided quite comprehensive materials and resources. For example, the GAT kit included trade books that emphasized the skills, as well as videos and supportive materials, such as the Chilly Penguin mascot, Cool Kid certificates, Peace Path posters, cool down posters, stop and think posters, obstacle posters, workbooks, strategy cards, and stickers.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

9. Careful Pacing Some grantees found they needed to slow down the pace of implementation. In some cases, significant turnover among school leadership and staff may have contributed to the need to re-assess the most appropriate pace for implementation. In other cases, implementation challenges such as use of space and scheduling conflicts meant adjusting expectations to make sure SEL activities were being implemented carefully and thoroughly. For example, in one “…the positive impact of slowing down the pace of school, teachers decided to slow down the implementation was palpable. We believe that slowing introduction of SEL tools to ensure both students down to implement the rollout of the tools … was a prudent move, as it allowed for faculty members to get on and teachers were fully familiar with and board and comfortable with the program.” comfortable using one tool, before introducing the next. One school provided teachers with a Grantee annual report detailed weekly pacing guide to ensure SEL activities did not get pushed aside by other activities. Staff pointed out that setting the right pace affected attaining and maintaining optimal momentum. 10. SEL for Adults While most SEL programming and much of the SEL literature focus on social and emotional learning skills for students, grantees increasingly understood the importance of addressing the SEL skills of faculty and staff. Although it may have been tempting to begin SEL curriculum with the “Adults create the culture of the students, the full effects of SEL were not evident school. Their attitudes and priorities unless it was implemented as part of a school-wide culture. Most are what the children breathe.” grantees reported that without adults creating a culture facilitating SEL success, little to nothing would change for the students. They SEL Coordinator stressed it was important to begin by providing adequate training to teachers, fellow administrators, and staff on SEL to avoid teachers implementing an SEL curriculum without fully understanding it. Grantees recognized both the importance of the adults modeling SEL practices and the benefits to the adults themselves in increasing their own SEL skills. 11. Reflection and Flexibility Some of the schools incorporated individual or group reflections as part of their implementation process. Where reflections were not already part of program implementation, schools agreed to collect some kind of reflection data, usually from teachers, as part of documenting their program implementation. Not only did these reflections provide valuable insights into the benefits and challenges of SEL programming, but teachers and SEL Coordinators reported that the process of reflecting on program implementation was very valuable to the schools themselves. It encouraged participants to take the time to reflect on and assess how implementation was going, and it encouraged schools to value those reflections and recognize the importance of everyone being engaged in thinking about program improvement. Reflection alone was not sufficient to drive improvement, however. It needed to be accompanied by both the flexibility to adapt to and address the learning that comes from taking that time to reflect, and valuing the input of the key stakeholders enough to give attention to the lessons that emerge and respond to their valuable perspectives. IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

12. Continuity of Staffing One of the biggest challenges most of the grantees have grappled with over the threeyear grant period was staff turnover. In some cases, this was the loss of the SEL Coordinator or key teachers who were SEL champions for the school. In other cases, this “This year we were able to hit the involved multiple teachers, resulting in an ground running and truly hit our stride. influx of new teachers unfamiliar with the school’s SEL Continuity of staffing matters a lot.” programming. In some cases, this involved school leadership SEL Coordinator including Principals and Vice Principals, which resulted in shifts in school priorities. Schools that had little or no turnover credited continuity of staffing as one factor that contributed to their success. 13. Continuity of SEL Practice Some schools found it difficult to sustain continuity in SEL practice over the three-year grant period. Several schools saw a dip in intensity of SEL programming in the second year, although they were able to recover, at least to some extent in the third year. “The average life of any program within Teachers reported that when other school-wide initiatives or the educational system, one of my mentors priorities were introduced, if they resulted in interruptions or once told me, is two years no matter how a reduction in the frequency of SEL activities, they felt they good the program is. … Deep-rooted change lost momentum. They felt that regular, continuous SEL requires patience, commitment, and a practice was an important dimension of program long-term perspective.” implementation. Numerous grantees made the case that SEL School Principal is not a program, it is part of the culture or identity of the school, such that SEL is not a lesson to fit into other programs, but a way of teaching. 14. Sufficient “Dosage” Several schools raised concerns about whether teachers were implementing SEL lessons and practices enough to have an impact on the students and the school climate, especially during Year 2. They struggled with whether it was sufficient to build explicit SEL instruction into the schedule once a week, when daily seemed like it would be better. Others tried to address this challenge by building SEL into regular classes outside of the core academic subjects, such as Kualapu‘u mandating that PE teachers start teaching GAT, since all students took PE. This resulted in school-wide implementation as well as ensuring regular exposure. ‘Ewa Makai built SEL lessons into their students’ advisory period. This definitely seemed to have been beneficial, because even if not as frequent as might be ideal, this helped to make exposure more ongoing by avoiding long time gaps between SEL lessons. 15. Sufficient Counseling Resources Several schools identified that a key role of the HCF grant was to expand their counseling resources. It became apparent over the three years that the counselor can play just as important a role in SEL as the teachers. In some schools, the counselor’s role expanded in

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

identities and backgrounds. In a culturally proficient SEL lesson, self-awareness and social awareness are often tightly linked. Although the focus is on understanding the impact of the SEL programs for the school, there is also the need to understand and acknowledge the school communities, in order to evaluate the schools in a more culturally responsive way. An example would be Waikīkī Elementary having a large (30+%) proportion of Japanese speaking, English language learners. Grantees reflected on how well the SEL programs they selected for their schools fit the cultural context of their students and community. For many schools, the key consideration was appropriateness of the program in the Hawaiian context. For some, the key consideration was how well the approaches used by the schools and the SEL domains they represent align with the HĀ framework. The HIDOE’s HĀ framework seeks “to develop the skills, behaviors and dispositions that are reminiscent of Hawaii’s unique context, and to honor the qualities and values of the indigenous language and culture of Hawaiʻi.” Some schools reflected on how well their SEL programs addressed the needs of other cultural groups as well.

How appropriate were the SEL programs that grantees have been implementing for the local Hawaiian culture? All of the grantee schools built Hawaiian cultural values and practices into their educational environment to some extent, whether or not they had a significant number of Native Hawaiian students. This is common throughout the state of Hawai‘i, as our Hawaiian roots embrace a culture of place. Some schools focused on place-based education more than others, and certainly the extent to which the local Hawaiian culture affected their pedagogy and school climate varied significantly across schools. However, all agreed cultural relevance was an important consideration. Here are some of grantees’ perspectives on the appropriateness of the SEL programs they were implementing for the local Hawaiian culture. •

SEL assessments are in English, so may not be accurate for students in Hawaiian language immersion schools or programs.

•

When choosing an SEL program, ‘Ele‘ele took into account the large cultural and language diversity at their school and found p4c a good fit for encouraging students to choose to express their ideas in the most natural way, rather than being limited to academic English.

•

The Waldorf “way” is very Midwestern and predominantly involves white students on the mainland. To one of the teachers who moved to Hawai‘i from Minnesota, in Hawai‘i, given the cultural diversity, students seem less “grounded”; they tended to get more emotionally “worked up”. She described the students here as sort of like palm trees, swaying in the wind.

•

Laupāhoehoe reported that cultural fit was one of the challenges with the Second Step program. In particular, some students did not like the videos, since they didn’t feel that the kids in the instructional videos looked like them, so some teachers improvised and used their own videos to better reflect their students.

•

Kualapu‘u staff reflected that there may be a lot less dialogue about feelings in the Hawaiian culture, where people are expected to process their feelings more internally. GAT gave the students an opportunity to express their feelings in a safe, structured way.

•

University Lab definitely took their cultural diversity into account when choosing SEL programs. Generally, they felt the programs they chose were a good fit. The point of Facing History is teaching about diversity in culture. TOOLBOX is very neutral and works well with a diverse population.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

•

P4c Hawai‘i seems to be a good fit for Hawaiian schools. It was developed to take Hawaiian culture into account. P4c Hawai‘i has as its foundation a focus on inquiry rather than argument (which is more traditional in Western philosophical discussion), in a way that shifts the pedagogical power from the teacher to the students.

What efforts have grantees made to adapt these programs for greater cultural relevance? •

Volcano middle school implemented Hawaiian practices of Nā Kilo ʻĀina 6 and Piko Wehena 7

•

After trying out Tribes and Second Step, the Hawaiian immersion school Ānuenue decided to shift to having their own teachers develop SEL lessons in the Hawaiian language. They focused not merely on translating SEL into a Hawaiian immersion setting, but also on the ways that Hawaiian spaces can cultivate social and emotional growth within the Native context. They explored the ways in which traditional knowledge might enable the learning and growth within the same SEL skill sets identified by CASEL.

•

One Ānuenue teacher developed lessons that looked at the characteristics of leaders in the Hawaiian community, including both the ali‘i 8 and the children’s own ancestors, to recognize what qualities they had, or wanted to build in themselves.

•

With the Principal as the lead, the Ānuenue faculty also used a Hawaiian cultural story (the moʻolelo or story of Kalapana, the famous keiki hoʻopāpā of Puna) as the basis for continuous exercises and reflection comparing modern views in SEL educational pedagogy with character building and traditional Hawaiian perspectives on goals for the whole child.

•

At Kualapu‘u the teachers on the immersion side of the school translated the GAT program into Hawaiian on the fly, and delivered the lessons in Hawaiian. They also translated some of the tools into Hawaiian, such as the conflict stopper and cooperative learning goals, and created a Hawaiian Peace Path.

•

Although TOOLBOX is generally culturally neutral and was a good fit for University Lab, one small exception was that the TOOLBOX Listening Tool includes the instruction to “meet eyes”. Some students were particularly uncomfortable looking adults in the eyes, especially if they thought they were in trouble. One teacher came up with an exercise for the Listening Tool that gave students the opportunity to practice poor listening skills and experience receiving poor communication to understand what good listening would look like.

•

At Volcano, teachers found that as they became more familiar with the curriculum, they were able to apply the lessons to their own lives and their students’ lives through story-telling and discussion. Initially, students and teachers reported that the videos provided with the program were difficult for students to relate to, as the students in the videos did not look like them, and the schools and culture they portrayed were foreign to them. As the teachers became more effective in encouraging students to share their own experiences and telling their own stories, they no longer heard complaints from students that the videos were irrelevant to them.

6 Nā Kilo ʻĀina is a practice of close observation of the environment as “the places and things that feed us physically, spiritually, and emotionally”. Nā Kilo ʻĀina has been adopted at VSAS to incorporate acute awareness of outer (environmental) with inner (social-emotional) qualities to cultivate knowledge and understanding as well as personal connection and relationship to the land and environment. 7 Piko Wehena (“source opening”) is a school-wide daily morning protocol to start the day. 8 Polynesian chief, noble or king

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

•

Waikīkī Elementary developed a bilingual Japanese-English p4c mentoring group, out of an awareness of the academic and social-emotional challenges faced by a large population of Japanese speaking students. Teachers reported a dramatic growth in the confidence of these students during their regular classroom p4c discussions.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

3. Program Outcomes In this chapter, we address four key evaluation questions concerning program outcomes: 1) How did school climate change after implementing SEL programs? 2) How did student behaviors change after SEL programs were implemented? 3) How did student academic achievement change after implementing SEL programs? 4) What other kinds of outcomes did grantees report?

3.1 School Climate School climate is becoming increasingly common as a measure of the outcomes of SEL programming as schools create more positive, supportive learning environments. Acknowledging that assessments of students’ SEL skills can be valuable for targeting SEL programming to meet students’ needs, most grantees felt the outcomes of their efforts were better measured through schoolwide climate change and student behavior than through measures of individual students’ SEL skills. As with other measures, selection of school climate measures for this evaluation began with data the schools were already collecting. Initially, most schools were collecting data on the School Quality Survey (SQS) distributed by HIDOE. Unfortunately, HIDOE changed the survey used to measure school climate statewide from the School Quality Survey to the Tripod Survey beginning in the second year of the grants. The differences in questions between the student version of these two surveys were so substantial as to make comparing SQS and Tripod essentially meaningless (See Volume II: Appendix B.) Therefore, student assessments of school climate are presented here for public schools using the Tripod survey in SY 2017-18 and SY 2018-19, and for the four schools that used the Education for the Future (EFF) survey for all three years. Following the student ratings, we also present results from teachers’ and parents’ assessments of school climate. Finally, we compare student, parent and teacher ratings at the end of this section.

Tripod School Climate Survey Here we present two years of student school climate data collected on the Tripod survey at eight public and charter schools in SY 2017-18 and SY 2018-19. While this leaves out the SQS data collected in SY 201617, the advantage of this approach is that the survey was consistent across the two years shown. The limitation of this approach is that the time frame is limited to looking at change over only a two-year period. As Exhibit 8 shows, student ratings increased overall, reflecting significant increases in three out of the six school climate domains measured.

Exhibit 8. Change in Student Tripod Ratings by Domain Across Eight Schools

Student climate ratings increased overall and for three climate domains – Rules & Norms, Social Support-Adults, and Support for Learning. (Green indicates increase is statistically significant at the 95% confidence level.)

Overall - % Positive Student Tripod Ratings Across all Schools 62.8%

69.5%

SY2017-18 (n=2,429)

SY2018-19 (n=2,262)

IMPAQ International

Physical Security - % Positive Student Tripod Ratings Across all Schools

61.4%

62.7%

SY2017-18 (n=2,429)

SY2018-19 (n=2,262)

Pillars of Peace SEL Final Evaluation Report July 2020

Rules & Norms - % Positive Student Tripod Ratings Across all Schools

44.5%

SY2017-18 (n=2,429)

Social Support - Adults - % Positive Student Tripod Ratings Across all Schools 67.7%

54.1%

SY2018-19 (n=2,262)

SY2017-18 (n=2,429)

Social Support - Students - % Positive Student Tripod Ratings Across all Schools

38.3%

38.4%

SY2017-18 (n=1,734)

SY2018-19 (n=1,514)

73.1%

SY2017-18 (n=2,429)

SY2018-19 (n=2,262)

Social-Emotional Security - % Positive Student TRIPOD Ratings Across all Schools* 85.0%

85.4%

SY2017-18 (n=2,429)

SY2018-19 (n=2,262)

*Elementary students only – not asked of middle and high school students in SY2018-19

Support for Learning- % Positive Student Tripod Ratings Across all Schools 67.3%

74.9%

The domain that showed the largest increase in student ratings was Rules and Norms, which included questions about whether rules were clear and whether student behavior was consistent with school rules. As shown in Exhibit 9, when testing these changes for statistical significance, we found that increases were significant in three of the six domains.

Exhibit 9. Summary of Significant Changes in Student Tripod Ratings Across Eight Schools Student Tripod ratings increased across domains overall and a significant increase in three of the six domains. (Green indicates increase is statistically significant at the 95% confidence level.) Domain

Change from SY 2017-18 to SY2018-19

Overall Across All Domains Physical Security Rules & Norms Social Support – Adults Social Support – Students Social-Emotional Security Support for Learning

In Exhibit 10 we present student Tripod ratings across Years 2 and 3 by grade level. As the exhibit shows, ratings increased in almost every domain. Only Social-Emotional Security showed no increase in ratings. Unfortunately, the SY2018-19 Tripod survey did not include Social-Emotional Security questions for middle IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

and high school. This was disappointing as this was a domain (which asked about feeling safe from bullying behavior and if the teacher was nice when students asked questions) was of great interest to the schools, especially given such low ratings in SY2017-18.

Exhibit 10. Change in Student Tripod School Climate Ratings by Grade Level Across Eight Schools Student ratings increased across almost all domains for all three grade categories. (Green and red indicate changes are statistically significant at the 95% confidence level.)

Across All Domains - % Positive Student Tripod Ratings by School Level

67.0%

74.8%

61.3%

Elementary

Physical Security- % Positive Student Tripod Ratings by School Level

65.6%

Elementary

70.1%

58.7%

Middle

SY2017-18 (n=2,429)

60.2%

63.5%

41.6%

Elementary

Middle

SY2017-18 (n=2,429)

IMPAQ International

50.4%

Elementary

SY2018-19 (n=2,262)

63.9%

SY2018-19 (n=2,262)

68.6%

High

72.0%

High

Rules & Norms - % Positive Student Tripod Ratings by School Level

45.6%

53.6%

61.4% 44.3%

Middle

SY2017-18 (n=2,429)

High SY2018-19 (n=2,262)

Social Support - Students - % Positive Student Tripod Ratings by School Level

Social Support - Adults - % Positive Student Tripod Ratings by School Level 84.8% 87.3%

68.0%

59.2%

Middle SY2017-18 (n=2,429)

56.9%

66.8%

72.4% 39.7% 36.9%

High

Middle

SY2018-19 (n=2,262)

SY2017-18 (n=1,419)

44.3% 33.4%

High SY2018-19 (n=1,349)

Pillars of Peace SEL Final Evaluation Report July 2020

Social-Emotional Security* - % Positive Student Tripod Ratings by School Level

Support for Learning - % Positive Student Tripod Ratings by School Level

85.0% 85.4%

69.1% 29.5%

Elementary

Middle

SY2017-18 (n=2,429)

77.6%

66.9% 71.0%

64.7% 69.6%

Middle

High

21.8% Elementary

High SY2018-19 (n=913)

SY2017-18 (n=2,429)

SY2018-19 (n=2,262)

*The TRIPOD survey did not include Social-Emotional Security questions for middle and high school in SY2018-19.

In Exhibit 11 we summarize in a single table the results of testing these changes to determine which ones are statistically significant. The exhibit shows that most of the differences shown above are statistically significant. The only significant decrease was in ratings for Physical Security by middle school students. Two multi-level schools with middle school students told us of events in their communities that may have made students feel less safe, including a shooting and a fire. Overall, there were significant increases in school climate ratings by all three grade levels of students.

Exhibit 11. Significant Changes in Student Tripod Ratings by Grade Level* Overall, student TRIPOD scores increased for all grade levels. Middle school ratings decreased in the Physical Security domain. (Green and red indicate change is significant at the 95% confidence level.) Elementary

Middle

High

N/A

Overall Physical Security Rules and Norms Social Support-Adults Social Support-Students

N/A

Social-Emotional Security Support for Learning *Changes are statistically significant at the 95% confidence level. Significant decrease Significant increase No significant difference

In Exhibit 12 we present student Tripod ratings by school. The exhibit shows that the ratings increased to some extent for all the schools that administered this survey for the two years. ‘Ewa Makai, Ka‘elepulu and Waikīkī had the greatest increases.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Exhibit 12. Change in Student Tripod Ratings by School Student climate ratings increased somewhat for all eight schools administering the Tripod survey. (Green and red indicate change is significant at the 95% confidence level.)

Across All Domains - % Positive Student Tripod Ratings by School Across All Grade

63.2% 66.1%

71.1% 75.6%

Ānuenue

‘Ele‘ele

61.5%

68.1%

‘Ewa Makai

70.2%

82.4%

Ka‘elepulu

SY2017-18 (n=2,429)

66.1% 68.7%

Kualapu‘u

59.5%

65.8%

60.4%

Univ Lab

68.7%

Volcano

66.0%

78.1%

Waikīkī

SY2018-19 (n=2,262)

Physical Security - % Positive Student TRIPOD Ratings by School Across All Grades 84.8% 55.8% 51.6%

64.3% 61.4%

Ānuenue

‘Ele‘ele

67.9%

58.5%

‘Ewa Makai

70.4% 69.4%

63.0% 49.0%

57.8%

51.0%

43.7%

Ka‘elepulu

SY2017-18 (n=2,429)

Kualapu‘u

Univ Lab

78.1%

66.9%

Volcano

Waikīkī

SY2018-19 (n=2,262)

Rules & Norms - % Positive Student TRIPOD Ratings by School Across All Grades

42.5% 41.5%

46.7% 49.9%

Ānuenue

‘Ele‘ele

45.7%

43.5% 40.0%

40.3%

‘Ewa Makai

60.3%

57.9%

56.9%

Ka‘elepulu

SY2017-18 (n=2,429)

Kualapu‘u

47.4%

57.9% 38.9% 40.0%

Univ Lab

Volcano

37.5%

Waikīkī

SY2018-19 (n=2,262)

Social Support - Adults - % Positive Student TRIPOD Ratings by School Across All Grades

70.7% 73.8%

82.3% 87.7%

Ānuenue

63.4%

‘Ele‘ele

73.5%

‘Ewa Makai

86.6% 90.8%

63.2%

Ka‘elepulu

SY2017-18 (n=2,429)

IMPAQ International

83.2% 82.8%

Kualapu‘u

69.5%

Univ Lab

70.4%

76.5%

Volcano

87.7% 89.4%

Waikīkī

SY2018-19 (n=2,262)

Pillars of Peace SEL Final Evaluation Report July 2020

Social Support - Students - % Positive Student TRIPOD Ratings by School Across All Grades

34.6%

39.8%

34.9%

Ānuenue

45.0%

47.7%

40.4%

33.2%

‘Ewa Makai

Univ Lab

SY2017-18 (n=1,734)

46.0%

Volcano

SY2018-19 (n=1,514)

Social-Emotional Security - % Positive Student TRIPOD Ratings by School Across All Grades* 85.4% 89.3%

81.7%

88.7% 88.2%

96.7%

89.5%

80.4% 80.3%

85.7% 84.4%

63.2%

58.4% 21.4%

Ānuenue

‘Ele‘ele

Ka‘elepulu

Kualapu‘u

SY2017-18 (n=2,038)

Univ Lab

Volcano

Waikīkī

SY2018-19 (n=1,509)

*At University Lab in SY2017-18 a very low percentage of 346 middle and high school students gave positive ratings. In SY2018-19, only elementary students were asked this domain.

Support for Learning - % Positive Student TRIPOD Ratings by School Across All Grades

67.6% 72.0%

73.6% 78.9%

Ānuenue

‘Ele‘ele

67.4% 72.2%

‘Ewa Makai

74.2%

84.7% 67.4%

Ka‘elepulu

SY2017-18 (n=2,429)

73.5%

Kualapu‘u

63.5% 66.8%

Univ Lab

63.8%

74.8%

Volcano

68.6%

79.7%

Waikīkī

SY2018-19 (n=2,262)

In Exhibit 13 we present the results of testing these changes to determine which ones are statistically significant. The exhibit shows that many of the differences shown above are statistically significant. Four of the elementary schools showed an increase in the Social-Emotional Security domain. This is significant because this domain was identified as one that was below average for most of the grantees in Years 1 and 2, and so was a significant focus for many of the schools. As mentioned earlier, it is unfortunate that this domain was not included in the SY2018-19 middle and high school Tripod surveys.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Exhibit 13. Significant Changes in Student TRIPOD Ratings by School* Student ratings increased in at least one domain at seven of the eight schools with two years of TRIPOD data. Overall student TRIPOD scores increased at three schools. Ānuenue

‘Ewa Makai

‘Ele‘ele

Ka‘elepulu

Kualapu‘u

N/A

Univ Lab

Volcano

Waikīkī

Overall Physical Security Rules & Norms Social Support – Adults N/A

Social Support – Students

N/A

Social-Emotional Security Support for Learning *Changes are statistically significant at the 95% confidence level. Significant decrease Significant increase No significant difference

Education for the Future (EFF) School Climate Survey Of the four schools that used the EFF school climate survey, two showed increases in student school climate ratings, as illustrated in Exhibit 14. Honolulu Waldorf saw a decrease in student ratings in Year 2, but the ratings increased again in Year 3. In the analysis below, we continue to share results in terms of the percentage of positive responses in order to be comparable to the data across surveys presented earlier. In Volume II: Appendix C, we also summarize the results using average ratings.

Exhibit 14. Change in Student EFF Ratings by School Two schools had an overall increase in student climate ratings on the EFF. (Green indicates increase is statistically significant at the 95% confidence level.) 67%

67% 52% n=115

n=82 HNL Waldorf

n=68

67%

60%

n=343

n=255

63%

n=171

Laupāhoehoe SY2016-17 (n=648)

64%

61%

n=248

n=93

49% n=243 St. Andrew's

SY2017-18 (n=588)

72%

71%

n=107

n=129

Volcano

SY2018-19 (n=616)

In Exhibit 15 we present changes in scores across individual measurement domains, and the results of testing these changes to determine which ones are statistically significant. The exhibit shows that Volcano had significant increases in six domains and St. Andrew’s in four and both had a significant increase overall across all domains. Laupāhoehoe had significant increases in three domains, even though their increase across all domains was not significant. Honolulu Waldorf had no significant changes.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Exhibit 15. Summary of Significant Changes in Student EFF School Climate Ratings* Student ratings at one school showed an increase in six domains and overall. Three schools showed significant increases in at least three school climate domains in SY2018-19, while one school had no significant changes. School Climate Domain

HNL Waldorf

Laupāhoehoe

St. Andrew’s

Volcano

Connectedness/Engagement

N/A

Physical Security Respect for Diversity

N/A

Self Confidence/Self Awareness

N/A

Self Management

N/A

Social & Civic Learning

N/A

Social Support - Adults Social Support - Students Social-Emotional Security Support for Learning Across all domains *N/A indicates domains not included in one or both years.

In addition to the Tripod survey data, HIDOE also provides a single climate measure as part of the STRIVE HI data system. Exhibit 16 displays the percentage of students who felt positively about their school for eight public and charter schools. As the chart shows, six of the eight schools showed an increase over the two years for which the data is available. We do not have access to the database itself, so we were unable to calculate for which schools the difference might be statistically significant.

Exhibit 16. Hawai‘i STRIVE HI Data – Percentage of Students Who Feel Positively About Their School* Students in 6 out of 8 schools felt more positively about their schools in SY2018-19 than SY2017-18. 87% 90%

57%

66%

Ānuenue

72% 69%

77% 74% 49%

‘Ele‘ele

‘Ewa Makai

Ka‘elepulu SY2017-18

83% 85%

73% 76%

72% 73%

Univ Lab

Volcano

58%

Kualapu‘u

Waikīkī

SY2018-19

* The Hawai‘i STRIVE HI reports first began including this measure in the SY2017-18 school year.

Teacher Ratings of School Climate Exhibit 17 presents teacher school climate ratings over the three-year grant period. As the exhibit shows, teacher ratings increased significantly in the Physical Security domain, but decreased in the Rules and Norms and Support for Learning domains. Exhibit 17. Change in Teacher School Climate Ratings Across All Schools by Domain IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Teachers ratings were generally a little higher than student ratings. Ratings increased significantly from SY2016-17 to SY2018-19 in the Physical Surroundings domain, but decreased in the Rules & Norms and Support for Learning domains. (Green and red indicate change is statistically significant at the 95% confidence level.)

Across all Domains

Connectedness/Engagement

% Positive Teacher Ratings Across All Schools

79%

74%

75%

SY2016-17 (n=208)

SY2017-18 (n=238)

SY2018-19 (n=210)

% Positive Teacher Ratings Across All Schools

Leadership

SY2016-17 (n=208)

69%

73%

SY2017-18 (n=238)

SY2018-19 (n=210)

SY2016-17 (n=208)

SY2017-18 (n=238)

SY2018-19 (n=210)

86%

93%

91%

SY2016-17 (n=208)

SY2017-18 (n=238)

SY2018-19 (n=210)

Professional Relationships

% Positive Teacher Ratings Across All Schools

SY2016-17 (n=208)

78%

% Positive Teacher Ratings Across All Schools

Physical Surroundings

79%

78%

Physical Security

% Positive Teacher Ratings Across All Schools 77%

85%

% Positive Teacher Ratings Across All Schools

93%

87%

77%

SY2017-18 (n=238)

SY2018-19 (n=210)

SY2016-17 (n=208)

Respect for Diversity

68%

74%

SY2017-18 (n=238)

SY2018-19 (n=210)

Rules & Norms

% Positive Teacher Ratings Across All Schools

82%

85%

86%

85%

81%

75%

SY2016-17 (n=208)

SY2017-18 (n=238)

SY2018-19 (n=210)

SY2016-17 (n=208)

SY2017-18 (n=238)

SY2018-19 (n=210)

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Support for Learning

% Positive Teacher Ratings Across All Schools 73%

SY2016-17 (n=208)

53%

59%

SY2017-18 (n=238)

SY2018-19 (n=210)

Exhibit 18 presents teacher school climate ratings by school. These show variation over time, some of which can be accounted for by the fact that the schools had varying response rates from the teachers from one year to another. Very few differences here are statistically significant, primarily because the number of teachers surveyed at each school was fairly small.

Exhibit 18: Change in Teacher School Climate Ratings by School There were very few changes in teacher ratings over time from the first year of the grant to the end of the grant period. (Green and red indicate change is significant at the 95% confidence level.)

Across All Domains % Positive Teacher School climate Ratings Across All Domains by School 83%

84%

83% 61% 44%

Ānuenue

53%

78% 77% 61% 56% 56%

60%

‘Ele‘ele

75%

82%

‘Ewa Makai

71%

67%

79%

66%

87%

100% 97% 99% 79%

52%

HNL Waldorf

SY2016-17 (n=164)

85%

Ka‘elepulu

Kualapu‘u

SY2017-18 (n=238)

Laupāhoehoe

Volcano

Waikīkī

SY2018-19 (n=210)

Connectedness/Engagement % Positive Teacher School Climate Ratings by School 86% 90% 85% 63%

Ānuenue

56%

66%

‘Ele‘ele

83% 80% 74%

‘Ewa Makai

77%

64%

HNL Waldorf

SY2016-17 (n=164)

IMPAQ International

76%

95% 90% 86%

65%

Ka‘elepulu

Kualapu‘u

SY2017-18 (n=238)

83% 87% 77%

83% 66%

100% 97% 99%

58%

Laupāhoehoe

Volcano

Waikīkī

SY2018-19 (n=210)

Pillars of Peace SEL Final Evaluation Report July 2020

Leadership % Positive Teacher School Climate Ratings by School 82% 83% 79%

80% 55%

Ānuenue

45%

72% 76%

55%

‘Ele‘ele

‘Ewa Makai

61%

67%

75%

80%

66%

62%

46% 46%

HNL Waldorf

SY2016-17 (n=164)

Ka‘elepulu

Kualapu‘u

SY2017-18 (n=238)

55%

89%

99% 97% 99%

82%

45%

Laupāhoehoe

Volcano

Waikīkī

SY2018-19 (n=210)

Physical Security % Positive Teacher School Climate Ratings by School 91% 95% 93%

90% 92% 91%

71% 77% 78%

Ānuenue

‘Ele‘ele

79%

‘Ewa Makai

SY2016-17 (n=127)

100%

90% 92%

Ka‘elepulu

SY2017-18 (n=125)

100%

88%

Kualapu‘u

97%

87%

79%

Ānuenue

93% 92% 91%

93% 90%

69% 74%

‘Ele‘ele

‘Ewa Makai

SY2016-17 (n=127)

SY2018-19 (n=152)

100% 77%

100%

Kualapu‘u

Ānuenue

88% 85% 67% 62% 65%

‘Ele‘ele

77%

71% 54% 49%

‘Ewa Makai

SY2018-19 (n=152)

IMPAQ International

91% 69%

76%

43%

HNL Waldorf

SY2016-17 (n=164)

83%

99%

Waikīkī

Professional Relationships % Positive Teacher School Climate Ratings by School 81% 86% 82%

100%

76%

Ka‘elepulu

SY2017-18 (n=125)

100%

Waikīkī

Physical Surroundings % Positive Teacher School Climate Ratings by School 87%

100%

72%

83% 79%

100% 97% 99%

42% 38%

Ka‘elepulu

Kualapu‘u

SY2017-18 (n=238)

Laupāhoehoe

Volcano

Waikīkī

SY2018-19 (n=210)

Pillars of Peace SEL Final Evaluation Report July 2020

Respect for Diversity % Positive Teacher School Climate Ratings by School 90%

88% 62%

96%

94%

93%

70%

HNL Waldorf

Laupāhoehoe SY2016-17 (n=37)

83%

Volcano

SY2017-18 (n=110)

SY2018-19 (n=58)

Rules and Norms % Positive Teacher School Climate Ratings by School 80%

88%

86% 86% 60%

Ānuenue

73% 72%

68%

46% 50%

‘Ele‘ele

‘Ewa Makai

SY2016-17 (n=127)

62%

54%

Ka‘elepulu

SY2017-18 (n=125)

100% 97% 97%

87%

85%

Kualapu‘u

Waikīkī

SY2018-19 (n=152)

Support for Learning % Positive Teacher School Climate Ratings by School

89% 69%

50%

41%

50%

46%

HNL Waldorf

38%

Laupāhoehoe SY2016-17 (n=37)

66%

SY2017-18 (n=110)

Volcano SY2018-19 (n=58)

Parent Ratings of School Climate Exhibit 19 presents parent school climate ratings over the three years of the grants. Overall, and within each domain, parent ratings were fairly high compared to those of students and teachers. Perhaps because the ratings were already high in SY2016-17, there were very few changes in parent ratings over the grant period. The only changes were significant increases in the Social-Support-Students and SocialEmotional Security domains, but a decrease in the Connectedness/Engagement domain.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Exhibit 19. Change in Parent Ratings Across All Schools by Domain Parent ratings increased in the Social-Support-Students and Social-Emotional Security domains, but decreased in the Connectedness/Engagement domain. (Green and red indicate changes from SY 2016-17 to SY 2018-19 are statistically significant at the 95% confidence level.)

Across All Domains Parent % Positive Scores Across All Schools

Connectedness/Engagement Parent % Positive Scores Across All Schools

83%

82%

80%

85%

SY2016-17 (n=730)

SY2017-18 (n=1,242)

SY2018-19 (n=1,379)

SY2016-17 (n=730)

Leadership Parent % Positive Scores Across All Schools 83%

83%

80%

SY2016-17 (n=730)

SY2017-18 (n=1,242)

SY2018-19 (n=1,379)

SY2016-17 (n=730)

70%

SY2017-18 (n=1,242)

73%

SY2017-18 (n=1,242)

SY2018-19 (n=1,379)

Physical Security Parent % Positive Scores Across All Schools

Respect for Diversity Parent % Positive Scores Across All Schools 80%

75%

86%

88%

86%

SY2016-17 (n=730)

SY2017-18 (n=1,242)

SY2018-19 (n=1,379)

Rules & Norms Parent % Positive Scores Across All Schools

81%

89%

90%

91%

SY2018-19 (n=1,379)

SY2016-17 (n=730)

SY2017-18 (n=1,242)

SY2018-19 (n=1,379)

Social Support - Adults Parent % Positive Scores Across All Schools

Social Support - Students Parent % Positive Scores Across All Schools

87%

89%

88%

92%

93%

SY2016-17 (n=730)

SY2017-18 (n=1,242)

SY2018-19 (n=1,379)

SY2016-17 (n=730)

SY2017-18 (n=1,242)

SY2018-19 (n=1,379)

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Social-Emotional Security Parent % Positive Scores Across All Schools 82%

80%

83%

86%

85%

SY2017-18 (n=1,242)

SY2018-19 (n=1,379)

SY2016-17 (n=730)

SY2017-18 (n=1,242)

SY2018-19 (n=1,379)

75%

SY2016-17 (n=730)

Support for Learning Parent % Positive Scores Across All Schools

In Exhibit 20 we present the results of testing these changes to determine which ones are statistically significant. As shown here and above, parent ratings changed very little across most of the domains. However, there was a significant increase in the percentage of parents giving positive responses for both the Social Support from Students and Social-Emotional Security, and a decrease in the Connectedness/Engagement domain.

Exhibit 20. Summary of Significant Changes in Parent EFF Ratings Across Four Schools* Student ratings increased in the Social Support-Students and Social-Emotional Security domains, but decreased in Connectedness/Engagement. (Green and red indicate change is statistically significant at the 95% confidence level.) Domain

Change from SY2015-16 to SY2018-19

Connectedness/Engagement Leadership Physical Security Respect for Diversity Rules and Norms Social Support-Adults Social Support-Students Social-Emotional Security Support for Learning Across all Domains *Changes are statistically significant at the 95% confidence level. Significant decrease Significant increase No significant difference

In Exhibit 21 we present changes in parent school climate ratings by school. Consistent with the findings above, there were no significant differences in parentsâ&#x20AC;&#x2122; overall ratings overall except for a decrease in parent ratings at â&#x20AC;&#x2DC;Ewa Makai, as shown in the top chart below. There were two domains where a few schools showed significant increases: Social Support-Students and Social-Emotional Security. There were three domains where at least one school had a decrease in positive responses: Leadership, Rules & Norms and Connectedness/Engagement.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Exhibit 21. Change in Parent School Climate Ratings by School For most schools there were no significant changes in overall parent ratings. (Green and red indicate change is statistically significant at the 95% confidence level.)

Overall - % Positive Parent Ratings by School 80% 76% 75%

Ānuenue

85%

78% 78%

‘Ele‘ele

84% 71% 70%

81%

70%

82%

92%

85% 87%

‘Ewa Makai HNL Waldorf Ka‘elepulu SY2016-17 (n=730)

78%

76%

81%

70%

Kualapu‘u Laupāhoehoe

SY2017-18 (n=1,242)

76% 74% 75%

Univ Lab

82% 87% 87%

Volcano

94% 93% 91%

Waikīkī

SY2018-19 (n=1,379)

Connectedness/Engagement - % Positive Parent Ratings by School 87%

84% 68% 68%

Ānuenue

76% 71%

81%

80% 55% 56%

‘Ele‘ele

70%

81%

89%

80% 81%

‘Ewa Makai HNL Waldorf Ka‘elepulu SY2016-17 (n=730)

86%

73%

95%

84% 85%

66% 65%

59%

Kualapu‘u Laupāhoehoe

SY2017-18 (n=1,242)

84% 89% 85%

88%

75%

Univ Lab

Volcano

Waikīkī

SY2018-19 (n=1,379)

Leadership - % Positive Parent Ratings by School 75% 73% 70%

80% 78% 74%

Ānuenue

‘Ele‘ele

84%

77% 75%

‘Ewa Makai

75%

84% 83%

Ka‘elepulu

SY2016-17 (n=665)

91% 93% 92% 72%

75%

Kualapu‘u

SY2017-18 (n=1,114)

79%

75% 77%

64%

Laupāhoehoe

Univ Lab

Waikīkī

SY2018-19 (n=1,238)

Physical Security - % Positive Parent Ratings by School 82% 82%

77%

88%

Ānuenue

81% 85%

‘Ele‘ele

91% 92%

100% 84%

‘Ewa Makai SY2016-17 (n=665)

IMPAQ International

92%

88% 91% 73%

Ka‘elepulu

73%

Kualapu‘u

SY2017-18 (n=1,114)

85%

94% 96% 96% 76% 79% 78%

Laupāhoehoe

Univ Lab

Waikīkī

SY2018-19 (n=1,238)

Pillars of Peace SEL Final Evaluation Report July 2020

Respect for Diversity - % Positive Parent Ratings by School 84%

83%

HNL Waldorf

79%

76%

65%

Volcano

SY2016-17 (n=65)

SY2017-18 (n=128)

SY2018-19 (n=141)

Rules & Norms - % Positive Parent Ratings by School 97%

86% 85% 87%

91%

87% 89%

94%

83% 87%

93%

94%

90%

81%

84% 86% 90%

84%

96% 97% 97%

69%

Ānuenue

‘Ele‘ele

‘Ewa Makai

Ka‘elepulu

SY2016-17 (n=665)

Kualapu‘u

SY2017-18 (n=1,114)

Laupāhoehoe

Univ Lab

Waikīkī

SY2018-19 (n=1,238)

Social Support - Adults - % Positive Parent Ratings by School 83%

89% 88%

Ānuenue

88%

83% 86%

‘Ele‘ele

88% 87%

83%

77%

84%

100% 95% 94%

85%

88%

93%

87%

72%

‘Ewa Makai HNL Waldorf Ka‘elepulu SY2016-17 (n=730)

Kualapu‘u Laupāhoehoe

SY2017-18 (n=1,242)

83% 79% 80%

Univ Lab

81%

88% 87%

Volcano

99% 99% 98%

Waikīkī

SY2018-19 (n=1,379)

Social Support - Students - % Positive Parent Ratings by School 91%

97%

92% 88% 94%

76%

92% 92% 87%

100% 99% 86%

96% 85%

79%

93%

81% 85%

86%

92% 93%

‘Ele‘ele

‘Ewa Makai HNL Waldorf Ka‘elepulu SY2016-17 (n=730)

IMPAQ International

Kualapu‘u Laupāhoehoe

SY2017-18 (n=1,242)

95% 98% 98%

72%

63%

Ānuenue

90% 91%

Univ Lab

Volcano

Waikīkī

SY2018-19 (n=1,379)

Pillars of Peace SEL Final Evaluation Report July 2020

Social-Emotional Security - % Positive Parent Ratings by School 92% 64%

71% 68%

76%

66%

74%

74% 71%

84% 89%

92% 89%

86%

81%

65% 53%

Ānuenue

‘Ele‘ele

‘Ewa Makai HNL Waldorf Ka‘elepulu SY2016-17 (n=730)

68%

59%

91% 91% 87%

89%

95% 94%

53%

Kualapu‘u Laupāhoehoe

SY2017-18 (n=1,242)

74% 78%

Univ Lab

Volcano

Waikīkī

SY2018-19 (n=1,379)

Support for Learning - % Positive Parent Ratings by School 77%

82% 84%

Ānuenue

86% 85% 86%

‘Ele‘ele

83%

91% 78% 78%

75%

78%

86%

92%

84%

87%

65%

‘Ewa Makai HNL Waldorf Ka‘elepulu SY2016-17 (n=730)

82%

72%

Kualapu‘u Laupāhoehoe

SY2017-18 (n=1,242)

80% 77% 79%

Univ Lab

78%

85% 87%

Volcano

95% 95% 94%

Waikīkī

SY2018-19 (n=1,379)

Discussion of School Climate Findings In this chapter we have presented a great deal of school climate data. We have included both the percent positive ratings for each year as well as significance testing to identify which of those changes are statistically significant. We have presented results for different instruments, by grade level and by school. Here we briefly summarize the key school climate findings: •

Overall, students’ school climate ratings increased over the course of the grant.

•

Examining Tripod survey results by grade level revealed significant increases in students’ climate ratings for all three grade categories — elementary, middle and high — with significant increases in almost every climate domain. The only decrease was in middle school ratings of Physical Security. Several middle schools reported outside events that affected students’ overall sense of physical security including a shooting, a fire and the mistaken incoming ballistic missile alert as possible contributing factors.

•

Among the four schools using the EFF school climate survey, Volcano had significant increases in six domains and St. Andrew’s in four and both had a significant increase overall across all

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

domains. Laupāhoehoe had significant increases in three domains, even though their increase across all domains was not significant. Honolulu Waldorf had no significant changes. Honolulu Waldorf saw a decrease in student ratings in Year 2, but the ratings increased again in Year 3. The Social-Emotional Security and Social Support – Adults domains increased significantly for three of the four schools. •

According to HIDOE, the percentage of students reporting they “Feel positively about their school” increased in six out of eight public and charter schools.

•

Teachers tended to give somewhat higher ratings that students.

•

Teacher ratings increased significantly in the Physical Security domain but decreased in the Rules and Norms and Support for Learning domains.

•

When examining teacher ratings by climate domain, very few of the differences were statistically significant, perhaps because at the individual school level, the sample size of teachers was fairly small.

•

Parents tended to respond more positively than either students or teachers.

•

Perhaps because the ratings were already high in SY2016-17, there were very few changes in parent ratings over the grant period. The only changes were significant increases in the SocialSupport-Students and Social-Emotional Security domains, but a decrease in the Connectedness/Engagement domain.

In order to better understand what these results might mean, we asked grantees for their thoughts on why school climate ratings did or did not improve as much as they may have hoped. Here are some possible explanations: •

Ānuenue — A few small decreases in school climate ratings are not surprising, given the school’s population has continued to increase to the extent that space is very crowded. The school’s population has also been changing such that a larger percentage of students are high risk. The curriculum used during the first two years was not a good cultural fit for the school, and the curriculum developed for Year 3 was still being piloted and improved. In addition, administering school climate surveys in English was problematic for many students.

•

‘Ele‘ele — One reason climate results were not higher was because they expanded the school population being surveyed. The 5th grade had a reputation for poor behavior throughout the school. It was challenging to sustain high energy levels and focus on SEL over time.

•

‘Ewa Makai — The school experienced significant turnover among school leadership and administration, and their school climate results worsened, as was reflected in the parent ratings.

•

Honolulu Waldorf — The school attributed decreased school climate ratings in Year 2 to significant turnover both among school leadership and administration, and among teachers. Also, flooding from Hurricane Lane resulted in their having to move the school campus, and all the upheaval and turmoil that entails. Moving to a less desirable neighborhood affected students’ feelings of safety and physical security. They were pleased that student ratings increased again in Year 3.

•

Ka‘elepulu — School climate results improved, even though increased knowledge and familiarity with SEL concepts can often seem to make respondents more critical. Each year they found new ways to incorporate Choose Love on campus, such as adding morning mana‘o and Brave Breath before school.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

•

Kualapu‘u — School climate ratings decreased, perhaps because increased knowledge and familiarity with SEL concepts made respondents more critical. They also had turnover among school leadership and administration. The school also had a large number of very high needs students without adequate support.

•

Laupāhoehoe — School climate results improved, even though they had turnover among school leadership and administration. In addition to implementing SEL curriculum, they have also been tracking positive behaviors via PBIS, and making sure these get increased attention from the teacher, and they have added more intense interventions for their students who need them.

•

St. Andrew’s — Even though it was challenging to sustain high energy levels, focus, or priority, their school climate results improved. They attributed the increases to careful, systematic implementation, teacher commitment and strong administrative leadership.

•

University Lab — Student school climate results increased in two domains, but this was less improvement than they were hoping for, due at least in part to significant turnover among school leadership and administration as well as among teachers. It was challenging to sustain high energy levels, focus, or priority, and their school climate results worsened.

•

Volcano — School climate results improved. As the only school with both Tripod and EFF survey data for comparison over time, it was interesting to note increases across domains on the EFF, and only in one domain on the Tripod. The school attributes improvements to some extent to the loss of some SEL-resistant faculty, teachers’ increasing familiarity with the curriculum and the support of the counselor. They also credit off-campus place-based learning, increased Hawaiian cultural education, and teacher training and professional development as contributing to a more positive school climate.

•

Waikīkī — Even though school climate was already rated higher than most of the other grantees due to the school’s long history of SEL, student ratings increased over the course of the grant period. They attributed this to their mentoring efforts with high needs students. They also acknowledge the need to continue to address the Social Support – Students domain.

3.2 Student Behavior The behavioral measures available for this evaluation were limited to those that schools were either already collecting or would be easy to report. Student behavior was documented through attendance, absences and behavioral incidents.

School Attendance One of the most common measures of student behavior associated with a positive learning environment is school attendance. Not only does school attendance relate to students’ engagement in learning, but it also reflects a family’s commitment to ensuring their students do not miss school. It is also common for students who have experienced bullying to miss more school. Exhibit 22 shows the average daily attendance at the 10 schools for which the data were available for at least two of the three grant years. Although average daily attendance was above 91% for all schools, there was variation across the schools ranging from 91% to 98% average daily attendance. Average daily attendance increased at Volcano, Laupāhoehoe and St. Andrew’s by approximately one percentage point. While there were no large across the board changes in attendance and absences, those schools that specifically targeted attendance as a concern were pleased with the results. For other schools, attendance was not identified as a major concern for the school or focus for their SEL program. IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Exhibit 22. Average Daily Attendance by School Three schools showed a small increase in attendance from Year 1 to Year 3.

94%

95% 95% 95%

95% 96% 95%

97% 97%

97% 97% 97% 95% 95%

95% 95%

93% 93%

Ānuenue

94%

98% 96%

95%

93% 91% 92%

‘Ele‘ele

‘Ewa Makai

HNL Waldorf

Ka‘elepulu

SY2016-17

Kualapu‘u

SY2017-18

Laupāhoehoe

St. Andrew's

95% 96%

92%

Volcano

Waikīkī

SY2018-19

Another measure of student attendance is the average number of daily absences at the school. Exhibit 23 shows variation in the average number of days students were absent. Average daily absences across the nine schools for which this information is available ranged from just under six days per year to 13 days per year. One school, Volcano, had a large decrease in averaged daily absences from 11.6 absences in SY201617 to an average of 7 absences in SY2018-19. None of the grantees cited absenteeism as a big problem, but some hope attendance will improve over time in response to students’ increased motivation and engagement.

Exhibit 23. Average Daily Absences by School Three schools showed a decrease in daily absences from Year 1 to Year 3. 11.2

13.0

12.3 12.6

8.3

9.0 9.0

8.5

7.8

8.4

9.8 9.5

11.6

10.8 9.1

8.4

10.1 8.3

8.2

5.7 5.7 5.8

Ānuenue

‘Ele‘ele

‘Ewa Makai

HNL Waldorf

Ka‘elepulu

SY2016-17

SY2017-18

Kualapu‘u

Laupāhoehoe

8.6 7.0

Volcano

7.2

7.5

Waikīkī

SY2018-19

The HIDOE Strive HI data includes a measure of chronic absence, defined as absent 15 or more days per year. Exhibit 24 summarizes the percentage of students missing 15 or more days per year over a threeyear period. (Note each of these exhibits is on a different scale to reflect the difference in values across schools, so although the slope of the lines may be steep, the differences are actually quite small.) The exhibit shows that most schools had a decrease in chronic absenteeism in 2018 (Year 3 of the grant), compared to 2015 (pre-grant).

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Exhibit 24. Percentage of Students Who Missed 15 or More Days of School* Five of the 9 public and charter schools have reduced chronic absenteeism since the 2015-16 (pre-grant) school year.

Ānuenue - % of Students Who Missed 15+ Days of School 23

23 20

2016

2017

2018

2016

2017

2018

2019

2016

3 2019

6 2017

2016

2018

2017

2018

2019

2016

2019

23 2016

7 2017

2018

2017

2018

2017

2018

2019

Laupāhoehoe - % of Students Who Missed 15+ Days of School

2019

Volcano - % of Students Who Missed 15+ Days of School

Kualapu‘u - % of Students Who Missed 15+ Days of School

University Lab - % of Students Who Missed 15+ Days of School 9

‘Ewa Makai - % of Students Who Missed 15+ Days of School

Ka‘elepulu - % of Students Who Missed 15+ Days of School 6

‘Ele‘ele - % of Students Who Missed 15+ Days of School

2016

14 2017

11 2018

14 2019

Waikīkī - % of Students Who Missed 15+ Days of School

2019

2016

2017

2018

2019

* Data on the percentage of students who missed 15 or more days of school were available for 9 public and public/charter schools through the HIDOE Strive HI reports.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Behavioral Incidents A more direct measure of student behavior is the extent to which students engage in problem behaviors that might result in referral, disciplinary action or even suspension. After discussion with the small evaluation working group and learning about the great variation in how schools document problem behaviors and determine the consequences, we concluded that suspensions would not be a good measure of student behavior. While inconsistencies in school policies related to reporting behavioral infractions and making suspension decisions make this a problematic measure in general, it is even more problematic in the context of implementing new SEL programming, where schools were changing their approaches and policies about how to handle problem behavior. Some implemented positive discipline, peer mediation, restorative justice and other strategies that all but eliminate out-of-school suspensions altogether. Others implemented enhanced reporting systems and new procedures that made referrals even more likely than prior to SEL implementation. After much discussion and consideration, we chose the concept of “behavioral incidents” as potentially more meaningful than suspensions or referrals, even though we recognized that these were not necessarily defined the same way across schools. Rather than simply report the number of incidents, as shown in Exhibit 25, we have computed the number of incidents per 100 students. In some cases, students had multiple incidents over the course of the year, which is why in one school the number of incidents per 100 students was over 100 early in the first two years of the grant. (Ten of the eleven grantees reported this information.) Exhibit 25 shows change in behavioral incidents over the three-year grant period. Rather than report absolute numbers, we report this measure in terms of the number of incidents per 100 students at the school. In most schools the number of reported incidents decreased or stayed the same. In some schools, the number of reported incidents actually increased in Year 2 and then decreased again. These increases were expected and were attributed to changes in the ways that teachers and administrators reported and tracked incidents. Honolulu Waldorf, for example, had been tracking behavioral incidents through a strike system. Their strike rate actually went up in the context of their effort to use the strike system more proactively to more comprehensively identify incidents and problems as they arose, to make sure the school addressed them rather than allowing them to fester or worsen. Other grantees worked towards decreases in behavioral incidents that they attributed to SEL practices. For example, Volcano implemented strategies to change the language used to describe incidents from labels such as “bullying” to descriptions of the actual behaviors that caused distress. They credit this for contributing to their reduction in behavioral referrals, along with expanding ‘Ike Hawai‘i from an elective to a required course for all students, beginning each day with morning Piko Wehena, and using long-term meaningful, place-based projects to develop a sense of responsibility for place through close observation, knowledge and stewardship.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Exhibit 25. Changes in Number of Reported Behavioral Incidents per 100 Students Six schools experienced a decrease in behavioral incidents. 124 102 77 57

48 30

23 2 Ānuenue

16 14

12 0

‘Ele‘ele

‘Ewa Makai HNL Waldorf Ka‘elepulu SY2016-17

26 27 1

Kualapu‘u Laupāhoehoe St. Andrew's

SY2017-18

8 Volcano

16 5

Waikīkī

SY2018-19

Discussion of Behavioral Findings Although the behavioral measures available for this evaluation were limited to those that schools were either already collecting or would be easy to report, the results are encouraging. While there was no large across the board changes in attendance and absences, those schools that specifically targeted attendance as a concern, including Ānuenue and Volcano, were pleased with the results. For the other schools, attendance was not identified as a major concern for the school or focus for their SEL program. As mentioned earlier, evaluating program effectiveness through documenting problem behaviors was somewhat challenging. In order to better understand what these results might mean, we asked grantees for their thoughts on why their behavioral outcomes did or did not improve as much as they may have hoped. Here are some possible explanations: •

Ānuenue — School attendance holding fairly steady was a big win for the school, given that an increasing number of students were bussed in from across the island. Many students had to board the Waiʻanae and Nānākuli buses by 5:30 in the morning, which can be extremely taxing on families. HCF grant monies allowed Ānuenue to experiment with class-wide incentives of snacks for best weekly attendance. They felt the data supported this approach to building a positive school climate around attendance, and the school decided to take over funding of snack purchases to sustain this initiative. They attributed their decrease in behavioral incidents to a fundamental shift in how the school managed student discipline. Their increasing focus on trauma-informed practices included professional development and school-wide embracing of a strong supportive approach to problem behavior, which helped to bring along some of the most challenged students as they learned to understand themselves better and forged more successful relationships with both adults and fellow students.

•

‘Ele‘ele — Attendance/absence numbers changed very little, but attendance has not been problem at the school. They had the behavioral incidents results they expected. In year 2 they improved on reporting these incidents, which resulted in the number of reported incidents increasing, followed by a reduction in Year 3 as SEL programming became more institutionalized and school climate continued to improve.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

•

‘Ewa Makai — The school experienced a decrease in behavioral incidents, as their structure for handling behavioral incidents changed. They were pleased with the result.

•

Honolulu Waldorf — As expected, their attendance remained high. Their behavioral incidents increased substantially in Year 2, which they considered a positive result, as they made schoolwide changes in reporting behavioral incidents. As anticipated, their behavioral incidents decreased in Year 3, which they attributed to a combination of improved procedures for addressing students’ behavioral needs and to their schoolwide SEL efforts.

•

Ka‘elepulu — Their attendance was already high, and they had very few behavioral incidents, so it was not surprising that these measures did not change very much over the course of the grant. In general, they felt they had been very successful in addressing the needs of individual students who struggle with behavior and attribute much of their success to their Choose Love program.

•

Kualapu‘u — Their attendance remained fairly high, which they attributed partially to their history of good attendance and partly to a generally positive school climate. Their behavioral incidents also remained high, which they attributed at least to some extent, to staff turnover.

•

Laupāhoehoe — They were pleased with the improvement in attendance but planned to continue to address chronic absences. They reported that their slight increase in behavioral incidents was what they expected and reflected a combination of becoming better at reporting incidents more accurately, and a more effective process for addressing students’ needs, which kept the increase quite small.

•

St. Andrew’s — Their attendance was already high, and they had very few behavioral incidents, so it is not surprising that these measures did not change very much over the course of the grant. In general, they feel their SEL efforts have helped them to become successful in addressing the needs of the very few individual students who struggle with behavior.

•

University Lab — They had very high attendance (99% in Year 3) and very few behavioral incidents (3 per 100 students in Year 3) and were pleased to have maintained these levels. (They were not included in the charts above because they did not provide the data in Years 1 and 2.) They believed their steadily improving school climate was a contributing factor for both of these measures.

•

Volcano — They decreased their average daily absences despite the effect of Kīlauea eruption at the beginning of Year 3. They also had some difficulty with making sure their buses were in good repair for a short while during the year. (Many of their students take the bus from far distances.) The number of behavioral incidents was very high in Years 1 and 2, when the school changed their referral procedures and were explicit about teachers recognizing and reporting incidents more thoroughly. They were pleased with the reduction in behavioral incidents since then, which they attribute to the teachers and the counselor being better equipped to handle behavior problems themselves, and the focus on SEL.

•

Waikīkī — They did not get the attendance/absence results they hoped for but noted that changes in Year 2 were minimal and not related to the SEL initiative. They got the behavioral incidents results they expected, and felt this was because more teachers, students, and parents were aware of how to get help/support.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

3.3 Academic Achievement The common understanding among grantees was that one key outcome of effective SEL programming is improved academic performance, but most initially saw it as a long-term outcome and did not expect to see a significant change within the grant period. Despite this, we decided to report STRIVE HI data on academic progress, anticipating that some grantees might be pleasantly surprised to see that the changes they are seeing anecdotally had become apparent in academic achievement data. Some schools were already noticing academic changes. For example, the Vice Principal at Waikīkī Elementary was facing a little pushback from a few teachers who did not see the value of students missing class for one of the mentoring events, so the Vice Principal took the year-end test scores and disaggregated the event students. He found that the students in the mentoring group had the biggest academic growth rate of any students in the school — definitely higher than the school average. The Vice Principal has often cited that statistic since, and the teachers became less resistant. Exhibit 26 summarizes academic performance as measured by the percentage of students meeting academic standards. The exhibit clearly illustrates that schools began at different starting places. (Note each of these exhibits is on a different scale to reflect the difference in values across schools.) Some schools, such as Waikīkī Elementary and Ka‘elepulu began SEL programming prior to the Pillars of Peace initiative and started at a higher level of academic achievement than the other schools. As Exhibit 26 shows, only Waikīkī Elementary and Ka‘elepulu did not show any improvements. However, their starting place was so high that even though they had a small decline, they were still the schools with the highest academic achievement across all eleven grantees. All seven of the other public and charter schools showed gains in academic achievement in at least one academic subject. Three of the schools, Ānuenue, ‘Ewa Makai and Volcano, showed gains in all three subject areas.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Exhibit 26. Changes in Percentage of Public and Charter School Students Meeting Math, Language Arts and Science Standards Seven of the nine public and charter schools showed an increase in percentage of students meeting academic standards in at least one subject.

Percentage of Students Meeting Standards at ‘Ele‘ele

Percentage of Students Meeting Standards at Ānuenue

28 21

30 24 19

Science

Language Arts

Math

56 52

Science

53 50

45 44

2016

2017

50 Math 48 Language Arts

2016

2017

2018

2019

Percentage of Students Meeting Standards at ‘Ewa Makai

2018

2019

Percentage of Students Meeting Standards at Ka‘elepulu 100

62 56

50 41

39 36

2016

2017

2018

57 Language Arts

48 Science

40 Math

2016

2019

Percentage of Students Meeting Standards at Kualapu‘u

2017

87 78

2018

85 Science 80 Math 79 Language Arts

2019

Percentage of Students Meeting Standards at Laupāhoehoe 43

52 42

33 34 27

23 2016

32 30

19 2017

IMPAQ International

2018

32 28 28

39 31

Math

26 23

Science Language Arts

Language Arts

26 23

Science

2018

2019

Math

17 2016

2019

2017

Pillars of Peace SEL Final Evaluation Report July 2020

Percentage of Students Meeting Standards at University Lab 70

71 62

48 46

2016

2017

Percentage of Students Meeting Standards at Volcano

Language Arts 48

Science Math

37 2018

2016

2019

27 2017

53 Language Arts 47 Science 40

Math

34 29 2018

2019

Percentage of Students Meeting Standards at WaikÄŤkÄŤ 97

93 Science

90 84 84

79 76

81 74

81 Language Arts 79 Math

72 2016

IMPAQ International

2017

2018

2019

Pillars of Peace SEL Final Evaluation Report July 2020

Achievement Gap Another measure of academic progress is narrowing the achievement gap between high needs students (those who are economically disadvantaged, have limited English and/or disabilities) and non-high needs students. Again, this was not an outcome measure that grantees chose, but one that we decided to look at, given the availability of data. As shown in Exhibit 27, two of the nine public and charter schools (Ānuenue and Laupāhoehoe) reduced the achievement gap in English Language Arts from SY2016-17 to SY2018-19. Four schools (Ka‘elepulu, Kualapu‘u, Laupāhoehoe and Volcano) reduced their achievement gap in math. Exhibit 27. Reducing the Achievement Gap Two schools reduced their English Language Arts (ELA) achievement gap and four reduced their Math achievement gap from SY2016-17 to SY2018-19.

Achievement Gap in ELA by School 43%

45%

43% 33% 26% 20%

28% 26%

35%

33%

31%

29%

25% 18%

15%

36%

34% 27%

26% 19%

14%

33%

38% 29%

26%

18%

Ānuenue

‘Ele‘ele

‘Ewa Makai

Ka‘elepulu

SY2016-17 ELA Achievement Gap

Kualapu‘u Laupāhoehoe University Lab

SY2017-18 ELA Achievement Gap

Volcano

Waikīkī

SY2018-19 ELA Achievement Gap

Achievement Gap in Math by School 45%

43%

19% 13%

18%

23%

19%

24% 26%

28%

24%

18%

29%

28% 13%

Ānuenue

‘Ele‘ele

‘Ewa Makai

SY2016-17 Math Achievement Gap

Ka‘elepulu

16%

18%

17%

22%

27%

24%

24% 18%

20%

11% 10%

Kualapu‘u Laupāhoehoe University Lab

SY2017-18 Math Achievement Gap

Volcano

Waikīkī

SY2018-19 Math Achievement Gap

Discussion of Academic Outcomes The academic achievement outcomes reported here are consistent with the stories of the different schools about the ways they felt their schools had changed since receiving the Pillars of Peace grants. While the measures of changes in school climate and student behaviors were somewhat blunt instruments for measuring change, the schools themselves all seemed to feel that their SEL efforts made a noticeable difference at their schools. Even those schools at risk of not being able to sustain their formal SEL IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

programs beyond the grant were able to point to at least some changes that they believed would persist, at least in how the teachers teach their regular lessons, and in the way students and teachers relate to each other at school. It is not surprising, then, that the schools might see these changes reflected in improved academic outcomes, at least among the schools that did not start out with high academic achievement at the beginning of the grants. Key findings include: •

All but the two highest performing schools saw an increase in percentage of students meeting academic standards in at least one subject area.

•

Three schools saw an increase of students meeting academic standards in all three subject areas.

•

An important observation when assessing academic outcomes is that schools clearly began at very different starting places, with a low of 4-21% of students meeting standards to a high of 83-100% of students meeting standards.

•

Two of nine public and charter schools reduced the achievement gap between high needs and nonhigh needs students in English Language Arts and four reduced the achievement gap in math over the grant period.

3.5 Other Outcome Measures Student SEL Skills and Competencies Measuring students’ SEL skills and competencies is a complex endeavor because schools and SEL programs may choose to focus on many different skills and competencies, and many different instruments have been developed to measure different aspects of SEL. Evaluators disagree about the exent to which it is appropriate to measure the effectiveness of SEL programs through assessing students’ SEL skills. Many say that since SEL is a combination of students’ skills, adults’ SEL skills, teaching and discipline practices and school climate, it is more appropriate to assess changes in student behaviors and the school learning environment than to focus on SEL skills per se. Others feel that if a school is trying to teach the students specific competencies, then assessing the extent to which students have gained those competencies is an appropriate evaluative measure. There seems to be general agreement that student SEL assessments can be useful for identifying the kinds of SEL programming that might be the best fit for the school, as well as identifying students who may need intensive support, but do not lend themselves to being used as high stakes performance measures. The IMPAQ team worked with a small group of grantees to research a wide range of SEL skills surveys and develop a brief student SEL skills survey that includes a small number of items considered to be highest priority among the group. (See Volume II: Appendix D.) Unfortunately, this work wasn’t completed until fairly late in the spring semester, and in the end, none of the schools administered the survey in Year 2 or Year 3. However, working together on the survey did provide the participating grantees with additional insights about possible ways of selecting a small set of proxies for the large and complex set of SEL skills that are currently being measured by schools around the country. Also, several grantees have indicated they now wish they had tried to measures students’ SEL skills, so they may yet find the instrument helpful. During this time, several schools developed or used various instruments to assess their students’ skills and learning. We worked with several of them to help them analyze the data and determine both how they might use the results, and whether it was worth the investment to continue to collect the data over the longer term. Some of these efforts were described in earlier reports. Resources don’t allow for us to analyze and present data from individual grantees in this report, but grantees have provided the Foundation with analysis in their reports. For example: IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

•

Ānuenue developed their own Hawaiian language student pre/post survey and their own English language teacher survey. Both had quantitative and qualitative elements.

•

‘Ele‘ele developed a short qualitative teacher survey about p4c that asked about whether the program is beneficial to students, beneficial to the teachers, and what they learned or gained.

•

At the end of each school year, Ka‘elepulu gave students a test that asked them what improvement, if any, they saw in these areas within themselves (not necessarily directly, but through questions that identified these skills and behaviors).

•

Kualapu‘u Elementary School decided to look closely at school climate to pinpoint more exactly where improvements could be made. They developed a student survey that focuses on where at the school students feel most safe, which cool-down strategies they use, whom they feel they can go to for help, how they resolve problems, whether they have friends, and their perception of rules, discipline, respect at the school.

•

Laupāhoehoe administered the Second Step Teacher Survey, which asks about various aspects of curriculum implementation.

•

St. Andrew’s collected data on SEL competencies through their administration of the Mindsets, Essential Skills, and Habits (MESH) middle and high school student surveys (used by California’s CORE districts) along with the RULER Teacher survey.

•

Waikīkī Elementary developed their own Pre-Post SEL Student Growth Surveys that were distributed to students, teachers and parents. They developed these as an evaluative measure to assess the school’s progress in addressing the needs of students who were targeted for referral to Individual Mentoring, Group Mentoring or both. The surveys include one question addressing each of the five major SEL domains as defined by CASEL. They also developed and administered a yearend Resiliency Theory (RT) Attribution Survey for students, teachers and parents.

Teacher Perceptions The IMPAQ team worked with a small group of four schools to design an online teacher perception survey. (See Volume II: Appendix E.) We set up the survey in Google forms, which creates charts and graphs from the data automatically. University Lab used this survey in Year 2. We helped them to analyze the data and it was sufficiently useful to them that they administered it again in Year 3 to see how teachers’ understanding of and commitment to SEL may have changed over time.

3.6 Outcomes Summary Because of the large number of exhibits and the amount of detail included in this discussion of grantee outcomes, in Exhibit 28 we further summarize the data into just a few small tables. As the first table in the exhibit shows, all eleven grantees showed improvement in at least one of the three major areas of measurement. Seven of the grantees showed improvement in all three areas. The tables below that provide additional detail on each of the three outcome areas.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Exhibit 28. Summary of Grantee Outcomes OUTCOMES SUMMARY

Improvement in at least one School Climate Measure

Improvement in at least one Behavorial Measure

Improvement in at least one Academic measure



          

  

Ānuenue ‘Ele‘ele ‘Ewa Makai Honolulu Waldorf

      

Ka‘elepulu Kualapu‘u Laupāhoehoe St. Andrewʻs University Lab Volcano Waikīkī

SCHOOL CLIMATE & BEHAVIOR

Student Tripod Overall (Ex.12)

SCHOOL CLIMATE Student STRIVE HI Teacher EFF Feel Climate Overall Positive Overall (Ex. 14) (Ex.16) (Ex. 18)

Ānuenue

N/A

‘Ele‘ele

N/A

‘Ewa Makai Honolulu Waldorf Ka‘elepulu

N/A

   N/A

   BEHAVIOR

Parent Climate Overall (Ex. 21)

Avg. Daily Attend. Absences** (Ex. 22) (Ex. 23, 24)



N/A

Kualapu‘u

N/A

Laupāhoehoe

N/A

St. Andrewʻs

N/A

 



 *

University Lab

N/A

Volcano



Waikīkī

N/A

   

  

N/A N/A



Behavioral Incidents (Ex. 25)

N/A

  

N/A

 

   

N/A

 

* St. Andrew’s had a statistically significant increase in ratings among Elementary Students. ** Reduction in either Average Daily Absences or Percentage of Students Who Missed 15+ Days of School.

ACADEMICS

Ānuenue ‘Ele‘ele ‘Ewa Makai

Meeting ELA Standards (Exhibit 26)

Meeting Math Standards (Exhibit 26)

Meeting Science Standards (Exhibit 26)

  



  



ELA Achievement Gap (Exhibit 27)

Math Achievement Gap (Exhibit 27)



  

 

Ka‘elepulu Kualapu‘u Laupāhoehoe University Lab Volcano

   

 



Waikīkī IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

4. Lessons Learned Grantees learned a great deal about implementing SEL programs. While many programs were implemented as planned, for some schools, mid-year course corrections were needed. For others, lessons were learned that may inform program improvements in the years to come. These lessons may also be useful to inform other schools and grantees of issues to be aware of when planning for, or implementing, SEL programs. Here we summarize lessons learned by the grantees in each of the three years of the grants. Lessons learned the previous year still remained relevant, but each year, grantees added new insights.

Year 1 Lessons Learned •

There is no single best starting place — Some grantees took on a large school-wide effort, implementing multiple SEL programs, and working towards school-wide changes. Other grantees started small in their first year, then identified unmet needs and expanded to be more comprehensive or wide reaching.

•

SEL is a school-wide issue — Some programs such as RULER were designed to be school-wide and community based. Implementation has to be “all-hands-on-deck” to be successful. Other programs can be rolled out a little at a time, but ultimately, SEL is a school-wide effort. Grantees told us that students did not feel safe at certain times or in certain parts of the school (hallways, playground at recess, etc.) Therefore, school-wide efforts to improve the school’s learning environment are critical.

•

School climate is a primary focus — While initially some schools focused mostly on teaching students specific SEL skills and competencies, the primary focus for most schools in the first year was improving the overall learning environment. This seemed to be the case regardless of whether a grantee initially embarked on school-wide change or decided to start small with a single intervention or a small selection of teachers or grade levels.

•

SEL is about students AND adults — During Year 1, grantees learned the importance of having adults (teachers, administrators, staff) at the schools examine, reflect upon, and build their own SEL skills. Schools with healthy climates, and effective SEL programs, included adults who modeled SEL competencies such as: positive and collaborative adult-adult relationships and communication; empathy; and respect for adults and children throughout the campus.

•

Teacher and staff commitment are key — Grantees unanimously agreed that SEL programs can only be successful in schools where SEL programs enjoy the full support and commitment of the Principal. However, teacher, staff and community commitment are also important. Teachers needed to be invested in the SEL initiatives; and it was even better to implement programs that were teacher-driven, not mandated from above. As one respondent shared, “You have to have a team. One champion is not enough.”

•

Schools can use data to get “buy-in” for SEL programs — Data that revealed areas needing improvement and showed that the program was having positive results were found to be compelling. This evidence greatly improved the buy-in of the more reluctant teachers.

•

Student commitment is also key — Some schools used students to help communicate the SEL philosophy and its value. Students made SEL seem attractive to other students. For example, one

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

school used a short video explaining and promoting the GAT principles to the students. The video featured students at the school who were respected academically and athletically by the other students, and demonstrating SEL activities to give the program the “cool” stamp. School leaders felt this was instrumental in getting students to buy in. •

Invest in implementation planning — Schools found it valuable to build in time early in the implementation phases for planning, making sure everybody understood the roll-out, and had access to materials and resources. This meant recognizing that time spent upfront was a valuable investment by ensuring time was well used during the school-year implementation. This might mean investing some “pre-season” time in accessing materials, making copies, and preparing supplies and curricula. In schools with limited space, good implementation planning also included making sure there was adequate space available for storing materials, group meetings, and safe spaces for students.

•

Keep it fresh for staff and students — Several schools suggested using at least one faculty meeting per semester to revisit what staff learned in their SEL training, acquire new skills, share experiences, and check in on how SEL program implementation was going. It is important to keep interest and focus high. It is also important to be creative in retaining student interest. This meant not relying solely on scripted SEL program lessons, but supplementing them with humor, reinforcing concepts throughout the day, and adding readings and activities.

Year 2 Lessons Learned •

Planning needs to happen in the spring — It was important to do implementation planning in the spring semester for the coming year. Otherwise, there was the risk of delays in curriculum planning if there was turnover in leadership, faculty and staff during late spring and summer.

•

Leadership and teacher buy-in need to go hand to hand — Several schools experienced significant turnover. Some had changes in leadership; for others, the turnover was mostly among the teachers. Strong leadership commitment was not sufficient without a clear strategy for onboarding new teachers. Experienced teachers easily lost their commitment to SEL programming if new leadership was not on board.

•

Sustaining focus and maintaining the energy can be difficult — Even in schools where implementation went well, it was difficult to keep the program on course. To keep staff invested in SEL, strong champions were needed to lead the effort and sustain momentum. Teachers could easily lose their motivation, decide to switch to some other program or stop implementing SEL altogether, without a strong consistent focus.

•

SEL time needs to be structured into the school day — Helping teachers learn to build SEL concepts and strategies into their everyday instruction and regular lessons was valuable but not sufficient to ensure that SEL got sufficient attention. Even schools using SEL programs that did not include explicit classroom instruction found it valuable to carve out regularly scheduled time dedicated to SEL practice. Without this structure, SEL programming could easily fade over time.

•

Reflection is an important part of the process and worth the time and effort — Taking the time to stop and reflect on what was working well and what could work better was extremely valuable. Not only did reflection for both students and staff provide valuable, actionable feedback, but a regular practice of reflection also provided an opportunity to celebrate and reinforce successes. It also created an opportunity to acknowledge and address unexpected realities, ensured that everyone’s perspective was valued and heard, and developed the skills needed to assess and reflect as a valuable part of the SEL skillset that can be applied throughout daily life.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

•

Students can be empowered to play a crucial role — Some SEL programs such as p4c required that teachers be willing to give up some control and trust the students to lead. Critical thinking, intellectual curiosity and taking greater responsibility were all areas of growth for the students. Peer mediation, peer mentoring, and student leadership opportunities helped foster student empowerment and greater buy-in that contributed to student engagement and motivation to learn.

Year 3 Lessons Learned •

SEL programming can be a good fit with Hawaiian cultural values — Two Hawaiian immersion schools were initially resistant to SEL programs operating from a Western context, but they recognized that many SEL skills aligned well with Native Hawaiian culture. These schools, to varying extents, took the time to translate SEL lessons into the Hawaiian language, and align them with Hawaiian values. Several other schools invested time in identifying ways that SEL programming complements or supports the HIDOE HĀ framework.

•

Build capacity to address the needs of students in crisis — After reviewing their data and realizing that many of their students were dealing with significant trauma, several schools shifted their focus from resolving student conflict to minimizing the impact of student crisis on campus, and several invested in learning more about shifting their approach to positive discipline and traumainformed practices.

•

Strong leadership from the top is essential — Starting SEL programs from the ground up, with teacher-generated ideas and programs, is a great way to introduce a school to SEL, but at some point the school needed to mandate the teaching of SEL to all teachers to create sufficient momentum and ensure the program was implemented with fidelity throughout the school. More directly involving leadership with SEL meetings led to more administrative support and more timely decisions.

•

Start with the infrastructure — A schoolwide SEL program required long-term support structures, and implementation went more smoothly if those were in place before the program was rolled out. Schools needed plans in place to allow for staff meeting time, professional development goals, and realistic data collection plans with meaningful measures. Learning to scale department or small-group initiatives into schoolwide initiatives took planning and the involvement of the whole faculty. Additionally, in dealing with staff turnover, it was important to develop a platform for onboarding new staff and students with the SEL curriculum and SEL’s place in the culture of the school.

•

Build capacity to analyze data — Several schools experienced frustration that although they put in place data collection strategies and collected good data, they lacked the knowledge or skills needed to analyze the data. There needed to be support in place to analyze the data collected and use it to develop actionable steps to proceed further or implement course corrections. Several schools used grant funds to hire outside data analysts to help them understand their data and use it to inform policies moving forward.

•

Bring it back to the Why — To help with teacher buy-in and to maintain momentum, it was imperative to keep bringing the focus back to why the school decided to build in SEL skills in the first place. Positioning the SEL program as helping the students helped the teachers embrace the program.

•

SEL instruction is both explicit and implicit — Traditionally, SEL is delivered to students through explicit means. SEL is what the teachers talk about and something that they teach to students. But SEL can also be delivered through an implicit approach where SEL is caught through the “hidden curriculum” of the how. Implicit teaching creates an environment where SEL is practiced,

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

lived, and, with sufficient exposure, internalized and cultivates SEL competencies, not by talking about them but by immersing the students within an environment that is imbued through and through with these competencies. •

SEL can be transformative, but it doesn’t happen overnight – Most grantees found that the more they invested in implementing SEL programming, the more they recognized the need to invest in a wider range of strategies to improve their learning environment and address the needs of struggling students. Many found their focus shifting from simply implementing an evidence-based program to much broader, systemic change. Most found that the three-year grant period was not long enough to accomplish the kinds of changes emerging as priorities for their school.

Getting the Right Fit Grantees offered many ideas of how to ensure that the SEL programming a school chooses is a good fit for the school. Examples of key considerations include:

“A deep-rooted culture of SEL is a slow play that emerges over the course of several years…” — SEL Coordinator

•

Does this go hand in hand with other school initiatives? Does it fit into the school vision?

•

Is the SEL program easily adaptable to the (specific) values of the school?

•

Start with a target group of your highest needs or most at-risk group of students to help choose a program that has the best chances of reaching/relating to them.

•

What other schools are already using that program, and what have their successes/challenges been?

•

How much does it cost? And what are the professional development costs associated with implementing it?

•

Think outside the box — it does not have to be something "recommended" by the state or district. It could be something that has worked in other settings (corporations, even prisons) that can be adapted to your school community.

•

Consideration of new personnel applies to fit. Fit includes: "Will this teacher/staff member 'fit' into our community's SEL culture?"

•

Is this a curriculum that needs class time? Who will deliver? Which classes? Is the program a preventative model or is it an intervention? How does the training work? A few select advocates or whole faculty/staff? Is there an adult SEL piece? How much time for organization and prep needed? Will you have an SEL coordinator?

•

When choosing an SEL program(s), include staff, especially ALL teachers’ ideas, outcome expectations, concerns, and provide continued support.

•

Consider how it works with all grade levels (K-12 setting). Does program fit within the structures already in place such as the school’s bell schedule, physical space, teacher/staff roles, administrative structure, PD policies, etc.

•

Look at a variety of programs and analyze against CASEL framework.

•

Consider integrated support for adults within the program.

•

Does it help create a positive culture across campus?

•

Involve all stakeholders from the beginning in the selection of curricula, programs, etc.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

•

SEL programs that are rooted in human values are more easily integrated into culturally diverse schools. Consider what values are explicit and implicit in the programs.

Specifically and explicitly plan for SEL and Hawaiian Values integration, using both PBIS and the HĀ framework.

•

Choose Love fits well with the HĀ framework, but it is important to make the connections explicit.

•

Schools found it helpful and even necessary to add "other" practices and language that support cultural appropriateness to the Second Step SEL program.

•

Several schools felt that the mindset of the adults in the school is even more important than the SEL program itself.

•

SEL teams looked for parallels between the local culture/community and the program's values/teachings. For example, restorative practices go hand in hand with ho‘oponopono9.

•

It is very important to inform parents about what is going on in the school. This can be very important not only to elicit their help in reinforcing the SEL lessons, but also a good way to be sure the SEL program reflects the culture of the school community.

•

P4c aligns well with the HĀ framework (aloha, total well-being, belonging etc.) and also with ho‘oponopono and forgiveness through empathy.

•

Roots of Empathy specifically encourages the baby's family to bring in their language, songs, and customs. In this way, it not only acknowledges each student’s temperament, but also serves as another way to bring the culture of the school community into the classroom.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

5. Promising Practices Through reviewing grantees’ reports and our interviews with faculty and staff, the IMPAQ team identified “promising practices” that offer insights into strategies for effective SEL program implementation. While not “best practices”, in the sense that there is limited evidence that these practices necessarily lead to greater program success, these practices are ones that are considered by the grantees to be making a difference and that seem to have promise as being transferable and useful to other schools implementing SEL programs. •

Ānuenue — School leadership found that teachers sometimes seemed to lack empathy and understanding for students with significant behavioral issues. They found that increasing the amount of information shared with teachers about trauma-informed practice in their faculty meetings helped ensure that everyone on campus felt a sense of urgency and responsibility for creating an inclusive environment on campus.

•

Ānuenue — The school never felt that programs created from a Western context were truly representative of the vision and mission of the school. They identified all of the ways that traditional Hawaiian knowledge led to the outcomes of: self-awareness, self-management, social awareness, relationship skills, and responsible decision-making. This led to greater buy-in from the teachers as they became comfortable identifying concepts in the SEL initiaitve which supported traditional knowledge.

•

‘Ele‘ele — A group of 10 p4c teachers met as a group during afterschool hours to talk about what was working, challenges teachers faced, and possible strategies to try. They received professional development during this time, such as learning the Good Thinker’s Toolkit. The SEL Coordinator also shared some best practices observed in different classrooms. Grant funds were used to support teachers for these after-hours meetings.

•

‘Ewa Makai — Each year a group of teachers and counselors met to develop a Pacing Guide. This helped keep the teams on the same page, by laying out exactly which lessons were to be taught each week. With representatives from each team, this was a teacher-driven effort that was shared with the whole staff. The teachers on the Character Development Pacing Committee refined the school’s SEL pacing guide to meld the Covey curriculum with the DOE General Learner Outcomes, TRIBES, AVID strategies, and team bonding activities. This also helped ensure that SEL lessons were not pushed into the background or given lower priority.

•

‘Ewa Makai — Middle school Student for a Day was a teacher student shadowing activity. The teachers wore the uniform, carried a backpack, went from class to class, and students treated them like students. The teachers found it to be a very eye-opening exercise in empathy. What a great way of getting a new perspective. The students wanted to know why the teachers were doing this, and the teachers said they want to understand more about the student experience. Knowing that the adults cared enough to put themselves through the exercise made the students feel very heard and appreciated.

•

Honolulu Waldorf — The faculty used weekly SEL meetings to further integrate and implement SEL throughout the school. Their student support committee adopted a three-legged-stool model, supporting academics, behavior, and SEL in their students. While acknowledging they were interrelated, they put in place a point person for each leg at the lower school as well as the high school. One result of this was a new look at behavioral supports as well as revisiting their punitive sounding “strike” system. They piloted a new approach called TAB, or “take a breath”. Students used a reﬂection page to identify their behavior, how it affected others, identify their emotions and make a plan for the next time.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

•

Ka‘elepulu — The school counselor came into the classrooms to teach Choose Love lessons, and teachers reinforced them by working the concepts into their own lessons. Lessons started with a Tibetan singing bowl inside a p4c circle, and the conversation started off on whichever Choose Love lesson they were learning. The students discussed the concepts in the circle in the very proscribed, respectful p4c manner. The original Choose Love lessons do not take very long, but by incorporating p4c, they lasted a full 30 minutes, giving the students ample time to explore the topics.

•

Kualapu‘u — A 6th grade teacher implemented a system of Team Points, where GAT teams in reading, writing, and social studies earned points for doing their work. They could redeem their points for snacks. The teacher tracked individual and group points via a free computer program called Class Dojo (not affiliated with GAT), which let her enter the reasons why points were awarded or taken away. The computer program was a great tool to solve arguments around points, as everything was documented.

•

Laupāhoehoe — The school changed their bell schedule to accommodate two periods of dedicated SEL instruction per week. All students received 50 minutes of SEL curriculum instruction on Mondays and 30 minutes of SEL reinforcement activities on Wednesdays. This dedicated and consistent SEL time was beneficial to both teachers and students as they practiced their SEL skills.

•

St. Andrew’s — The school created a bimonthly, 2-page newsletter called The RULER Review, to keep faculty and staff up to date on which RULER tools to focus on for the month, and which to start getting ready to teach in the upcoming month. The newsletters included links to pedagogical videos, reminders of upcoming RULER events/PD, and photos of RULER in action in classrooms. The 3-year olds’ pre-school teacher created Mood Meter puppets for the teachers and children to use for addressing emotions. These helped to make the quadrants of the mood meter more concrete and understandable to these young students.

•

St. Andrewʻs — Understanding the importance of building relationships, St. Andrew’s focused the first couple days of school solely on building relationships and community. All grade levels were given space to create their RULER Charters and focus on setting class norms.

•

University Lab — The school established an SEL Grant Committee comprised of two faculty SEL grant coordinators and one administrator who met regularly two to three times a month. They took on planning, organizing and facilitating school community-outreach programs, as well as continuing to guide the SEL program implementation planning and professional development.

•

Volcano — The school implemented strategies to change the language used to describe incidents from labels such as “bullying” to descriptions of the actual behaviors that caused distress. This improved communication and understanding and helped with resolving conflicts. School administrators credited this with contributing to a reduction in behavioral referrals.

•

Volcano — Volcano has embraced Hawaiian cultural education to help students develop a relationship with their environment and community. The school implemented weekly place-based, project-based learning, with middle school students leaving the classroom for a whole day each week. Each English, math, and science teacher comitted to a 3-year environmental/community service project for their students. Initially teachers struggled with the loss of classroom time, but after adapting their curricula, found they were able to cover the material more easily because students became much more engaged, motivated learners. Staff reported that these out-of-school activities focused on nature and community drove the most SEL progress with their middle school students.

•

Waikīkī — The SEL program changed the format of their Art Group to provide higher intensity mentoring for students. In addition to weekly classes, the students attended public exhibitions at the Honolulu Museum of Art, two parent-child weekend classes, and two-night p4c and art classes for 4th and 5th graders, and middle and high school alumni of the program.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

•

Waikīkī — At Waikīkī Elementary, SEL is caught through the “hidden curriculum” of the “how”. Faculty and staff have created an environment where SEL has been practiced, lived, and, with sufficient exposure, internalized. SEL competencies have been cultivated not by talking about them but by immersing the students within an environment that is imbued through and through with these competencies. Waikīkī School certainly talks about SEL competencies and teaches them to children. But they have also helped students to “catch” these competencies by purposefully and repeatedly exposing them to the living practice of these competencies.

Use of evaluation results for program improvement All grantees collected multiple types of evaluation data, using various quantitative and qualitative measures. Common data collection activities included: •

Student surveys assessing student perceptions of school climate and safety

•

Teacher and parent climate surveys

•

Teacher interviews and questionnaires assessing program implementation, challenges, and suggestions for improvements

•

Meeting and activity logs

•

Teacher and staff reflections on the program

Some grantees used evaluation tools and protocols embedded in the SEL program they were implementing. For example, Choose Love and RULER include their own survey instruments. Regular public schools and most charter schools used the Tripod survey (along with a few residual SQS items, as required by HIDOE) to assess yearly progress. IMPAQ worked with a group of grantees during Year 1 to discuss evaluation goals and select a common instrument that private schools and others interested could use to measure yearly progress. The group decided to use EFF surveys to measure school climate. While a few schools have yet to begin analyzing their data in earnest, most schools have used the data collected to measure the effectiveness of their SEL programs and other related systems (such as disciplinary approaches) in order to drive informed decisions for course correction or forward progress. Several schools were frustrated that they lacked the capacity to fully analyze their data to extract meaningful results in evaluating the success of their programs. Two schools hired outside consultants to help them make sense of their data. Schools described using data collection and analysis for program improvement: •

Schools used data to ensure SEL programs were being implemented consistently.

•

Behavior data was used to diagnose issues and evaluate the effectiveness of SEL strategies.

•

Schools realized the value of capturing baseline data for comparison with post-intervention data.

•

Several schools used school climate data to focus their efforts on specific school climate domains.

•

While schools were hungry for feedback, they realized they needed to balance the number of surveys implemented with limited teacher time and to consider student survey fatigue.

•

The data collected by some schools indicated a need for student trauma support, and they introduced ways to help faculty learn about and implement trauma-informed strategies.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

â&#x20AC;˘

One school was looking at their data to see if there is a correlation between higher student selfperceptions of SEL outcomes and the amount of time teachers spend in class on the SEL lessons.

â&#x20AC;˘

Some schools collected large amounts of data but lacked the staff time and/or expertise to analyze the data for program improvement.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

6. Sustainability and Looking to the Future In this chapter, we address key evaluation questions about the sustainability of the SEL efforts grantees put into place with the Pillars of Peace grants: How can grantees sustain their SEL programs? What plans do they have in place for addressing sustainability? What aspects of their SEL efforts are likely to be sustained? We first began discussing issues around sustainability with grantees in Year 2 of the grants. Some had already been working on sustainability plans, others seemed to assume that the best parts of what they were doing would be sustained naturally. By Year 3, all grantees were looking at ways to sustain or even expand their SEL efforts. The following ideas come from the grantees themselves.

Institutionalize SEL SEL strategies and activities as well as associated changes in school culture and climate can best be sustained if these are truly integrated throughout the school. SEL can be institutionalized by making explicit decisions to ensure these efforts become part of the fabric of the school itself. Here are some of the strategies the grantees are using to institutionalize SEL into their schools: •

Integrate SEL into school-wide plan/charter contract/school guiding documents.

•

Build SEL into existing structures in the school day to support explicit SEL instruction (advisory, guidance lessons, activity period, specialties).

•

With strong administrator support SEL becomes a mindset, a culture. SEL needs to be seen as a measure of the school’s overall mission and values.

•

Build a strong SEL team with representatives from lots of different areas within the school (administration, teachers, specialties). This way SEL is not dependent on just one or two champions.

•

Prioritize school funds to continue essential functions such as counseling and SEL coordination.

•

Build what was learned over the past 3-4 years into a new, revised, more data-driven 3-year plan.

•

When you build in sustainability, a “program” is no longer a program, but rather a part of the culture, part of what the school has become.

•

Have teachers deliver their own SEL lessons rather than have a counselor or someone else deliver them. That way the teachers take ownership of SEL and reinforce it throughout the school day.

•

Principal and school staff should be on board — not only with the plan, but also with a structure to monitor the plan, revise over time, keep it current and relevant.

•

Continue to Invest in Professional Development One of the benefits of the Pillars of Peace grants was providing resources for professional development (PD). Because PD time is limited and every year there are conflicting priorities, it is easy to let slide PD that is associated with work done under grants that have ended. Most grantees recognized, especially IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

given issues of staff turnover, that it is essential to find ways to continue PD beyond the grant, if SEL programming is to be sustained. Grantee suggestions include: •

Train teachers to be in-house "trainers" so there is no need to pay for outside PD every year.

•

Make sure to provide PD beyond a small group of teachers. Get the other adults on campus trained as well. Then the adults can all support each other and continue to reinforce the learning.

•

Build SEL into the schools’ regular PD and keep the conversation going with staff (staff meetings, PD days, etc.).

•

Build a process to educate new faculty to the SEL program and continually offer teachers interesting ways to implement the learning so it stays engaging and interesting for both the students and the teachers.

•

Use Data Effectively •

Examine the data more closely and use it to inform a set of measures and milestones (and timeline) that best fits the school’s needs and actual situation.

•

Use the data from this grant to demonstrate how SEL programming supports the students and contributes to their success when seeking future grant funding.

•

One school decided to implement their school climate surveys during the month of November. Receiving feedback in the middle of the year will allow them to address issues sooner. They also hope that a newly formed discipline record and tracking system will help them understand their students better as they work to support them. This information will also help them to track the trends of their school climate as they work towards the next stages of health for their school.

•

Several grantees valued the data collected for these grants and appreciated the grant paying for someone to analyze it. They found it valuable to planning and found it really shifted thinking about the importance of collecting and using data. The challenge will be to continue to analyze the data and use it, but they are committed to finding a way to cover those costs after the grant.

Seek Additional Funding •

Most grantees are seeking funding to continue to invest in resources including programs, training, and SEL advocates. Some are specifically looking to bring in funding so they can designate specific and explicit resources to students with greater needs.

•

One school is seeking funding to continue to implement student-initiated projects.

•

Several schools identified the need seek funding to maintain counselor staffing.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

Continue to Grow •

SEL champions are great, but there needs to be growth or spreading to more champions in order to be flexible or resilient to change. The team approach is much more sustainable than a single champion.

•

The three years helped to get SEL established, and now most schools are in a good place to expand out. Several schools said they could see expansion among their students just through activities like murals and student clubs, and they can see the positive influence these activities have had on them.

•

Several SEL Leadership teams are discussing how to address the school-wide implementation of SEL and to what extent their existing tools will be a part of that solution. In other words, they are taking a step backwards, in order to move ahead more intelligently.

Continue Participating in the Larger SEL Community Most SEL implementation leaders, faculty and counselors we talked to found they appreciated the opportunity to discuss the SEL work they were doing and learned from the experience of others. Many felt somewhat isolated when daily demands made it difficult to stop and reflect or reach out to educators to exchange ideas. Several grantees reported wishing the current HCF SEL cohort could continue. This was seen to have value not only for schools to continue applying the insights they learned from each other to maintain “…there might be considerable value in continuing the and build on the gains from these grants, but also cohort’s fruitful dialogue, building upon established as an opportunity to address the broader question relationships, and the ongoing cultivation of exemplary of how to best nurture the development of SEL SEL model schools.” across the state. It was suggested this could SEL Coordinator perhaps be accomplished through an occasional meeting, or perhaps through continued support of their SEL efforts. Several grantees felt the exchange of experiences and ideas among schools in the cohort was one of the most valuable components of this grant program.

Wishes for the Future We asked grantees to identify what things they wished could have been done differently or could be done differently in the future. Some of these wishes might provide ideas for future directions HCF might take, for the grantees themselves going forward, for other schools or initiatives seeking to incorporate or expand SEL efforts, or for future evaluations of SEL efforts. Grantees’ wishes included: •

I wish the grant period were longer so after we figure out what to do, we have more time to keep track of how we do and see the results over a little longer period.

•

I wish we could include a control group non-grant schools or HIDOE overall.

•

I wish we could see the "average" of all schools in Hawai‘i in the things we're tracking.

•

I wish we could analyze teacher perception over time.

•

I wish we could compare student outcomes by gender and ethnicity.

•

I wish (or hope) that we all continue to collect data beyond Year 3.

•

I wish I could focus more of my time to survey students and staff about SEL at our school.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

•

I wish we could do this type of cohort discussion and information sharing with a group of parents.

•

I wish we could conduct parent focus groups to learn more about their perspective and what they need to help reinforce SEL.

•

I wish we had created an at-risk cohort to track and have another level of comparison within each school.

•

I wish we could track data for alumni/graduates.

•

I wish we could continue having cohort meetings. Learned so much. Cannot wait to share with my team.

•

I wish we could get help analyzing our data so we can make informed plans.

•

I wish we could look at how demographic/high needs schools vs high needs students differ in needs/resources.

•

I wish we could see how our SEL work will look in our students lives 5 years from now? 10 years from now? 15 years from now? 20?

•

I wish we had a good measure of how SEL has impacted individual students.

•

I wish we could look at outcomes for students in "tiered" groups using the same scale, to see if the data show more or less progress for some students rather than just looking at "big picture" snapshots of the whole school.

Sharing Stories One of the aspects of being part of an SEL learning community that grantees greatly appreciated was sharing stories with other grantees. Not only did the stories help illustrate what different schools were doing, but the act of identifying and telling their own stories helped schools focus in on what changes they were seeing in their schools. While this report shares a great deal of data and very insightful reflections and lessons learned by the grantees, sometimes the best way to describe what happened is through story telling. The following are some illustrative stories the grantees shared with us: •

Ānuenue — From a traditional lens, teaching students to hoʻoponopono (“make it right”) when they have made a mistake is an important cultural skill. However, when asking students to apologize to one another, it was evident that some students were not capable of regulating their emotions to reflect and reset. When we embarked on the journey to implement SEL at Ānuenue, we did not know as much about how trauma-informed approaches with students could benefit our community. As we began to broaden our awareness of the needs of our school community through our focus on strengthened relationships, we learned there were students who were not capable of problem-solving to the degree we expected them to, because they were experiencing a life crisis (e.g., homelessness, neglect, abuse, incarceration of a parent, etc.). We began to understand, that for many of our students, school was their safe zone, and the faculty were looked at by our students as ʻohana. School-wide, we are definitely moving to the softer side. Much less “What did you do wrong?” and more “What’s going on today?” and “How are you feeling?” and “What can I do to help?”

•

‘Ele‘ele — The Principal experimented with having p4c-type discussions during faculty meetings. He wanted to wait until p4c was firmly embedded and the teachers had the opportunity to see it in action for a while. He felt the students were modeling the program for the teachers and proving its efficacy, which was ultimately more efficient than trying to navigate all the myriad and random objections that teachers might have. Some teachers let the students generate the topics and questions for the class council, and some teachers generated the topics themselves. One of

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

the interesting topics the students came up with themselves was about the mistaken incoming ballistic missile alert that everybody in Hawai‘i got one Saturday morning in 2018 — what it meant, and how their families reacted to the alert. •

‘Ewa Makai — Some girls came to me (the Principal) and said the bathroom was their safe place, so when they felt sad they went to the bathroom. But what they wanted to do was have something positive in the bathroom, so that if they went in there, and they were upset or crying or whatever, that there was something there that would cheer them up. So I told them they could paint the wall. So they painted up there a picture — and this is all their idea — a picture of a phone, and from the phone is lots of text message that said things like "Create positive vibes." Every text message was something positive, like "Be yourself, you're the best," …all these positive messages around this phone. And they said just, "This will make us feel really good when we come in."

•

‘Ewa Makai — The Empowered Girls group painted a mural called "Keys to Kindness”. They produced little colored paper keys, and then they had people write how to be kind, and then put them all around the Keys to Kindness. They recently decided to change it up and took all the keys down and now put up hearts. They had everybody fill out a heart recently, and they are putting all the hearts up of how to be kind to each other.

•

HNL Waldorf — Roots of Empathy taught the school emotional literacy, temperament, and brought new parts of the community together. One such moment was during a presentation with Mary Gordon from Roots of Empathy that was attended by both high school students and lower school parents. One mother shared her concern about her adopted daughter, asking Mary how to make up for her first year in which her daughter had not formed attachment. A sophomore stood up and told her that he was adopted as well. He advised her to just keep loving her and that even though it is harder, she will eventually understand that she is loved. This moment brought tears to the mother’s eyes.

•

Ka‘elepulu — The health room attendance data was actually a pretty good indicator of the success of Choose Love. Students used to visit the health room all the time throughout the day to deal with their issues, and it was a constant disruption. One first grade teacher went from having 10 students a day visit the health room to maybe one student a week. They only visit the health room if they’re bleeding or have ‘ukus (head lice) now.

•

Ka‘elepulu — One teacher shared “When the children are having a problem or conflict on the playground all I need to say is ‘choose love’ and they know what to do.”

•

Kualapu‘u — The school decided to collect data on which features of their SEL programming students were using. Almost 100% students reported using at least one “cool down strategy” and one “problem-solving strategy”. Almost all children could identify someone from whom they could receive help, most of whom were at school with the child: teachers, friends, the school counselor, the Principal. (One child even mentioned the custodians, and it was evident from the child’s other answers that this was a serious comment.)

•

Kualapu‘u — GAT changed the way Kualapu‘u looked at data. Because of SEL, they developed a survey system that they can use regularly, even after the SEL funds go away, as a way of monitoring what is going on with GAT. They can track which strategies the students know, and which they do not know, which helps the teachers target their instruction. They also learned about how safe students feel on different parts of the campus, which helped them to target monitoring different parts of the school as well as target PD to the staff who need it the most.

•

St. Andrew’s Lower School — An unexpected tragedy in late March 2019 struck the school community when one of the Kindergarten boys died in a kayaking accident. This was a tremendous

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020

shock to the school community as they all struggled to deal with this tragic loss. Thankfully, the groundwork and SEL foundation that they established over the years helped faculty and staff, students, and families process their grief together. The experience made visible the progress they had made in developing collective emotional intelligence, coming together in support of each other, and fostering a nurturing school community. •

St. Andrew’s Upper School — The fall theater production, Zombie Thoughts, featured a character’s journey in dealing with anxiety; the theater director figured out a way to incorporate the Mood Meter and MetaMoment (tools of the RULER program) in the play’s narrative. Following the show, the actors participated in a discussion featuring the school counselor and a community psychologist, addressing issues and complexities with anxiety and SEL. As a result, the school’s production garnered attention from the playwright.

•

University Lab School — Three boys who were best friends got into a scuffle and asked for a quiet space so they could work it out. They worked it out, came up with a resolution amongst them and walked back into the activity as buddies again. They were very proud of themselves. The teacher never even learned what the conflict was about.

•

Volcano — A student remarked that a family member was dying, and another student responded by saying, “Oh well, that’s life.” During the student court session, the defendant testified that he did not mean it in a bad way, that he had gone through really hard times, and he was able to deal with his life challenges by maintaining a philosophy that the hard times were a part of his life and he just had to accept things as they are. During the defendant’s testimony, he shared a lifechanging incident that involved his father ending up in prison. The tone of the room was silent but comfortable. Later student court members reported how touched they were by the defendant’s testimony and that they almost cried. The team reflected on the nature and impact of miscommunication and versus open, sincere, and respectful dialog. In addition to the relationship building this court session had for all the students involved, the defendant reported that he felt good after talking about his experiences out loud to his peers.

•

Waikīkī Elementary — One student mentoring group student said “I think that Live Aloha was good for me because I feel like I was listened [to] and appreciated. For example, when we share about our weekend and answer a question, I feel like everyone pays attention and listens to me. I also feel like I was respected, and I opened up to a lot of people that I probably wouldn’t have otherwise. Another reason is that it made me more relaxed knowing that no matter what happened someone would be there and listen to me and my feelings.”

•

Waikīkī Elementary — In September 2016, a 4th grader sat within his classroom at Waikīkī School and repeatedly screamed “Bitch!” as his shocked classmates looked on. Three years later this same young man sat on a student panel in front of an audience of 100 educators at the Schools of The Future Conference. He spoke with uncommon eloquence and authentic emotion about how his life had changed. He talked about how he used to have outbursts and how embarrassed he was by them. He talked about how, thanks to the mentoring initiatives at Waikīkī School, he had learned to control himself and come to believe in himself. His words validated three years of Waikīkī School SEL team’s work and confirmed that, indeed, they had accomplished what they had aimed to achieve. They had designed and implemented a powerful set of SEL initiatives that had positively altered the trajectory of this young man’s life.

IMPAQ International

Pillars of Peace SEL Final Evaluation Report July 2020