An evidence-based, independent third party research study conducted by PREVNet and funded by the Ont

Ontario Education Pilot 2014 -2015

An evidence-based, independent third party research study conducted by PREVNet and funded by the Ontario Ministry of Education.

PROMOTING RELATIONSHIPS AND ELIMINATING VIOLENCE NETWORK

Table of Contents List of Figures ................................................................................................................................................ 3 Results Summary........................................................................................................................................... 4 Questions Related to Bullying Characteristics .......................................................................................... 5 Questions Related to Legal Issues of Bullying ........................................................................................... 6 Questions on Responding to Bullying ....................................................................................................... 8 Questions Related to Sexual Abuse .......................................................................................................... 9 Questions Related to Confidence ........................................................................................................... 10 Post Survey Follow-up Questions ........................................................................................................... 11 Limitations .................................................................................................................................................. 12 6 Month Follow Up ..................................................................................................................................... 13 Questions Related to Bullying Characteristics ........................................................................................ 13 Questions Related to Legal Issues of Bullying ......................................................................................... 14 Questions on Responding to Bullying ..................................................................................................... 15 Questions Related to Sexual Abuse ........................................................................................................ 17 Real-world application ................................................................................................................................ 19 Conclusions………………………………………………………………………………………………………………………………………….. 20 About PREVNet ………………………………………………………………………………………..…………………………………………. 21 Appendix: The questionnaire…………………………………………………………..………………………………………………….. 22

Ontario Education Pilot 2014-2015

2

List of Figures Figure 1: Questions Related to Bullying Characteristics ............................................................................... 5 Figure 2: Questions Related to Legal Issues of Bullying ................................................................................ 6 Figure 3: Questions on Responding to Bullying ............................................................................................ 7 Figure 4: Questions Related to Sexual Abuse ............................................................................................... 9 Figure 5: Percentage of people who reported that they were 80% or more confident in their ability to ... 9 Figure 6: Post Survey Follow-up Questions ................................................................................................ 11 Figure 7: Participant Satisfaction with Program ......................................................................................... 12 Figure 8: Questions Related to Bullying Characteristics: Follow up .......................................................... 13 Figure 9: Questions Related to Legal Issues of Bullying: Follow up ............................................................ 14 Figure 10: Questions on Responding to Bullying: Follow up ...................................................................... 15 Figure 11: Questions related to Sexual Abuse: Follow up .......................................................................... 17 Figure 12: Percentage of people who reported that they were 80% or more confident in a certain ability: Follow up..................................................................................................................................................... 18 Figure 13: Percentage of people who applied their knowledge of certain topics post-Respect in Schools program ...................................................................................................................................................... 19

Ontario Education Pilot 2014-2015

3

Background Rationale: There are programs available addressing bullying, harassment, and discrimination. Unfortunately a limited number have strong scientific evaluations to ensure that they are having positive effects. The research on Respect in Schools will provide the scientific evidence to evaluate the effects of the program. PREVNet was requested to conduct an independent evaluation of the Respect in School Program.

Objectives of the Evaluation 1. To evaluate the effect of the Respect in School Program on students, educators, and all educational staff over time.

2. To evaluate the knowledge, skills and confidence gains through the Respect in School Program of the participants and how they change over time and influence the prevalence of bullying and victimization.

Design Participants completed the survey prior to the training and post training. A smaller sample completed the follow up survey.

Participants 5,556 participants completed the pre and post module questionnaire. Of those only 668 completed the follow up questionnaire.

Questions Participants completed an online questionnaire comprising items assessing content of the module, their perceived confidence in addressing problems related to the module content, and their satisfaction with the module. The follow up questions had fewer content questions, but also assessed the extent that the participants had implemented or used the knowledge gained from the module (see appendix for the questionnaire).

Results The results are presented in two sections, the pre-post comparisons and the pre-post and follow-up comparisons. The analyses compared the percentages of correct responses to each of the questions using z-tests for proportions and controlling for conducted for multiple tests. Graphs also depict the results of these analyses. Questions are grouped thematically on the graphs.

Ontario Education Pilot 2014-2015

4

Figure 1. Questions Related to Bullying Characteristics 100%

*

* 80%

60%

*

* *

*

40%

Pre-Test

Post-Test 20%

0% Main Leadership Characteristic style for of Bullying confidence

Types of Bullying

Major types Most Development of harrasment important of factor in independence harrasment age group

Questions Related to Bullying Characteristics This figure represents the percentage of correct responses from participants for several questions about bullying. These questions asked participants to identify the main characteristic of bullying and harassment, the type of leadership style most likely to promote positive self-esteem, types of bullying, forms of harassment, the most important factor in harassment, and the age group at which youth start to develop a personal identity. The significant differences from pre to post-test are indicated with an asterisk (*).

Key Findings



Participants significantly improved their knowledge from pre to post program on identifying: o The main characteristic of bullying and harassment (31% to 54%) o Types of bullying (57% to 82%) o The age at which youth start to develop personal identity. o The four major types of harassment (68% to 75%)

Ontario Education Pilot 2014-2015

5

Concerns 



Even after significant increases from the pre to post test, the majority of participants could not identify the main characteristic of bullying (less than 60% were able to correctly answer).

Similarly, even less than 40% of participants could correctly identify the age at which youth develop a sense of identity

Recommendations This section of program information could be strengthened by increasing the focus on Children’s development and increasing the amount of knowledge on Leadership styles for promoting confidence.

Figure 2. Questions Related to Legal Issues of Bullying 100%

*

*

*

*

*

80% 60%

Pre-Test

40%

Post-Test 20% 0% Legal duty to Bullying can be Legal duty to Punishment report abuse considered report other can be abuse harassment school leaders

Probing questions

Questions Related to Legal Issues of Bullying The questions in this section asked participants whether or not school leaders always had a duty to report abuse and neglect; if bullying may be considered harassment if the perpetrator is over the age of 12; if school leaders have a legal duty to report other school leaders suspected of abuse; if punishment could be considered a form of abuse; and if probing questions should be asked when leaders suspect neglect or abuse of a child.

Ontario Education Pilot 2014-2015

6

Concerns 

The number of correct responses in the pre-test for some of the questions in this section was quite high, which may have prevented the study from capturing any increase in knowledge for the participants. There were only two questions were the baseline knowledge was low. In other words, they had high baseline knowledge. A suggestion is to review the materials in this section and provide more in depth information about these issues.

Key Findings   

The majority of post-test responses for this section were correct (over 90%).

Participants were significantly more knowledgeable at post-test in identifying that bullying can be considered harassment if the victim is over 12 years old.

There was a very large increase in the number of participants understanding that probing questions need to be used when abuse or neglect is suspected.

Ontario Education Pilot 2014-2015

7

Questions on Responding to Bullying These questions related to how often bullying stopped when children intervene, what percentage of bystanders typically support a victim, the best way to respond to bullying, and appropriate discipline for bullying/harassment/abuse.

Key Findings 

Participant knowledge significantly increased for all questions in this section.  This increase was particularly large for participant knowledge on: o how often bullying stops when children intervene on the playground (correct responses more than doubled) o the percentage of bystanders that support a victim

Concerns 

Participant knowledge was still low at post test for two of the questions: o Percentage of bystanders that support a victim (< 50%) o Appropriate discipline for bullying/harassment/abuse (< 55%)

Recommendations The program’s focus could be increased in appropriate ways to respond to bullying, including a more indepth promotion of data on how often bystanders intervene to support victims, and more information on when it is appropriate to provide discipline related to bullying/harassment/abuse.

While the percentage of participants that could correctly answer questions related to these topics significantly increased from pre to post test, the percentage of respondents answering these questions correctly was still somewhat low (both below 55%). They may require more knowledge and time spent on this topic in the modules.

Ontario Education Pilot 2014-2015

8

Figure 4. Questions Related to Sexual Abuse *

100%

*

* 80%

60%

Pre-Test Post-Test

40%

20%

0% Age of Consent

Sexual abuse has to include Can students consent with a touching school leader

Questions Related to Sexual Abuse This section included questions on the age of consent for sexual activity, whether or not sexual abuse has to involve touching, and whether or not students can consent for sexual activity with a school leader.

Key Findings   

Participant knowledge of what denotes sexual abuse and whether students can consent to sexual activity with a school leader was above 90% before training.

Participant knowledge significantly increased for all questions in this section.

The largest increase of knowledge in this section was regarding age of consent. o The percentage of correct responses for this question jumped from 50% to 80%.

Recommendations The age of consent remains an area that requires increased focus in this section.

Ontario Education Pilot 2014-2015

9

Figure 5. Percentage of people who reported that they were 80% confident or more in their ability to... *

100%

*

*

*

*

*

80%

60%

Pre-Test

40%

Post-Test 20%

0% Build healthy Identify Identify the Know relationships bullying abuse attributes of a strategies to with students or harassment positive support leader victims

Know when Use power to report positively and bullying responsibly

Questions Related to Confidence Client confidence was measured at two points in time: Pre-test and Post-test. For each question given during the survey, participantsâ&#x20AC;&#x2122; responses ranged from 0-100% at 20% intervals. Here, we include the percentage of respondents that reported they would be 80% confident or more in each scenario for each time point.

The items in this section asked participants if they were confident in a variety of skills including building healthy relationships, identifying bullying and abuse, knowing the attributes of a positive leader; feeling confident knowing how to use strategies to support victims, feeling confident knowing when to report abuse, and confidence in using their power positively.

Key Findings ď&#x201A;ˇ

The majority of participants were 80-100% confident in their abilities rather than 050%. o This trend was even more significant during the post test o Over 90% of participants consistently reported 80-100% confidence in the post-test.

Ontario Education Pilot 2014-2015

10

Figure 6. Post survey followup questions 100%

80%

60%

No Somewhat

40%

Yes 20%

0% Convienient to use

Easy to use

Beneficial Information

Helped me Would become more recommend to child-centered others

Post Survey Follow-up Questions These questions, which were delivered after the post-program survey, asked whether or not the participant found the program was convenient to access, easy to use, beneficial, if the program would help them become a more child-centered school leader, and if the participant would recommend this program to other school leaders.

Participants were able to respond to these questions indicating that they were either ‘Somewhat Satisfied’, ‘Satisfied’, or ‘Not Satisfied’ with the program in each of these areas.

Key Findings  

The majority of participants were satisfied with the program in each of these areas Less than 10% of participants said that they would not recommend this program.

Ontario Education Pilot 2014-2015

11

Figure 7. Total Participant Satisfaction with Program Were you satisfied with the program?

4% 20%

Yes Somewhat No 76%

Key Findings This graph shows the combined responses to the satisfaction survey after completing the program.

The majority of participants (96%) reported either “Satisfied” or “Somewhat Satisfied”. 

This demonstrates that the majority of participants were happy with the program overall.

Limitations Some questions in the survey had a high percentage of correct responses (above 90% correct) before the completion of the program. Due to this, participant knowledge is difficult to assess knowledge gained in some areas (e.g., information on sexual abuse) Further, some questions in the survey did not show an increase of participant knowledge after the program, or even showed a decline in the percentage of correct responses. 

These areas in the program may need a greater focus, perhaps by explaining these concepts with more clarity, or by increasing the amount of content on these particular topics.

Ontario Education Pilot 2014-2015

12

6 Month Follow-up A total of 668 of the original 5556 participants completed the original survey questions at a 6 month follow-up to see how well knowledge from the Respect in Schools program was maintained.

Figure 8. Questions Related to Bullying Characteristics: Follow-up 100%

* *

* *

* 80%

*

*

*

* *

60%

* *

40%

Pre-Test Post-Test Follow up

20%

0% Main Leadership Characteristic style for of Bullying confidence

Types of Bullying

Major types Most Development of harrasment important of factor in independence harrasment age group

Questions Related to Bullying Characteristics Key Findings



Participants significantly improved in their knowledge from pre-program to 6 month follow up for each question related to bullying characteristics.  There was no significant change in participant knowledge on the major types of Concerns harassment and the most important factor in harassment from post-program to 6 month follow up, meaning that participants maintained their knowledge in these areas.  For 3 of the 6 questions related to bullying, correct answers significantly decreased from  At 6 month follow up, at least 60% of participants were getting the majority of questions post-program to characteristics 6 month follow-up, indicating some participants were not retaining the related to bullying correct. information. .  Participants significantly improved in their ability to identify the main characteristic of  bullying Still, even less than 40% of question participants couldparticipants’ correctly identify the age atofwhich 6 months later.

This reflects understanding someyouth of the constructs adolescent development. develop areviewed sense ofon identity.  Ontario Education Pilot 2014-2015

13

Figure 9. Questions Related to Legal Issues of Bullying: Follow-up * *

100%

*

*

80%

60%

Pre-Test Post-Test 40%

Follow up

20%

0% Bullying harassment if over 12

Can use probing questions when suspected abuse

Questions on Legal Issues Related to Bullying Key Findings ď&#x201A;ˇ ď&#x201A;ˇ

Participants significantly improved in their knowledge from pre-program to 6 month follow up for each question related to bullying characteristics For each question, over 80% of participants were able to provide the correct answer 6 months after the program.

Ontario Education Pilot 2014-2015

14

Concerns ď&#x201A;ˇ

The number of correct responses at the 6 month follow-up was significantly lower than the number at post-test, suggesting some participants were not retaining the information

Figure 10. Questions on Responding to Bullying: Follow-up 100%

*

* * *

80%

* * *

60%

Pre-Test Post-Test 40%

Follow up

20%

0% When children intervene

% of bystanders that Best way to respond support victim to bullying

Appropriate discipline

Questions on Responding to Bullying

Ontario Education Pilot 2014-2015

15

Key Findings   

Participants significantly improved in their knowledge from pre-program to 6 month follow up for questions related to bullying characteristics Participants significantly increased in their ability to correctly identify the most appropriate discipline for bullying at 6 month follow-up Over 80% of participants were able to correctly identify the best way of responding to discipline

Concerns   

Participants scored significantly lower at 6 months follow up than at post-test on the majority of questions Participants did not differ in correct responses from pre-test to follow-up on the best way to respond to bullying Participant knowledge was still low at the 6 month follow-up for two questions o Percentage of bystanders that support a victim (< 50%) o How long bullying persists after children intervene (< 55%)

Recommendations Building on this program, there should be a particular focus on communicating how to appropriately respond to bullying, especially considering the importance of this topic. Specifically, since both questions on bystander intervention were both below 55%, the program might spend more time outlining the importance and effectiveness of peer intervention.

Ontario Education Pilot 2014-2015

16

Figure 11. Questions Related to Sexual Abuse: Follow-up *

100%

* * 80%

60%

Pre-Test Post-Test 40%

Follow up

20%

0% Age of Consent

Sexual abuse has to include touching

Questions Related to Sexual Abuse Key Findings 



Participants were significantly higher in their knowledge on the age of consent and what sexual abuse involves at 6 month follow up in comparison to their initial pre-test scores.

At 6 month follow-up, participants’ scores of 70% correct for both questions related to sexual abuse.

Concerns  

Participants scored significantly lower at 6 months follow up than at post-test on both questions.

Participants’ knowledge of what sexual abuse has to include did not differ from pre-test to 6 month follow-up.

Ontario Education Pilot 2014-2015

17

Figure 12. Percentage of people who reported that they were 80% confident or more in certain abilities: Follow-up

*

100%

*

*

*

*

*

* *

*

80% 60%

Pre-Test 40%

Post-Test Follow up

20% 0% Build healthy relationships with students

Identify bullying Identify the Know strategies Know when to abuse or attributes of a to support report bullying harassment positive leader victims

Key Findings ď&#x201A;ˇ

ď&#x201A;ˇ

The majority of participants were 80-100% confident in their abilities rather than 050%. o This trend was even more significant during the post test. o Over 90% of participants consistently reported 80-100% confidence in the post-test.

Participants remained significantly more confident in their abilities at 6 month follow up than at pre-test.

Concerns

ď&#x201A;ˇ

Participants were significantly less confident in their abilities at 6 months follow up than at post-test on the majority of questions. o However, these differences were not large

Ontario Education Pilot 2014-2015

18

Real-world application Figure 13. Percentage of people who applied their knowledge of certain topics post-Respect in Schools program 100% 80% 60% 40%

Not at All

20%

Somewhat

0%

Often

Key Findings ď&#x201A;ˇ ď&#x201A;ˇ

For the majority of questions, at least 60% of participants had applied their knowledge from the program somewhat or often.

For the majority of questions, 20% of participants applied these topics often, and at least 40% applied their knowledge of leadership styles, child emotional development, and selfawareness often.

Ontario Education Pilot 2014-2015

19

Conclusions      



The Respect in School program is providing staff and educators important information that they are implementing in their work.

There is an increase in knowledge in all areas of focus of the program from pre-test to post test.

There is an increase in confidence in applying the knowledge learned from pretest to post test.

The program content and delivery style is valued by the users.

For the sample that completed the follow up questions, they did not retain the information at the same level as they had at post testing, but did demonstrate a higher level of knowledge then they had before the training.

The evaluation provides some direction to modifying the content of the training: o Based on the responses to the characteristics of bullying, it is clear that this knowledge was new to over half of the participants and it is a critical component in the module. There is a need to continue to focus on issues such as effective leadership styles and some general issues concerning development. o Similarly, the training addressing responding to bullying highlighted the importance of the training on the role of peers in bullying and providing strategies to address bullying. Peers play important positive and negative roles in bullying and through understanding the peer dynamics of bullying, interventions can be more effective. o The legal issues regarding bullying and abuse could be more in depth as many of the participants had the knowledge prior to completing the training. However, the age of consent in sexual intercourse is an issue that needs to be highlighted. This information is critical and ideally, all staff should be aware of these legal issues. o There could be more of a focus on strategies There are many strengths to this program o The knowledge is relevant and needed by staff and educators o The delivery method is highly valued o The content of the training is highly valued.

This program provides important training for educators and knowledge increases and is retained post training and at follow up. Participants’ confidence increased as a function of the training and about 60% of the participants are implementing the knowledge gained in their practices.

Ontario Education Pilot 2014-2015

20

About PREVNet PREVNet is an umbrella network of 122 leading Canadian research scientists, over 150 graduate students and 62 national youth-serving organizations. Launched in 2006 with the Networks of Centres of Excellence, PREVNet’s mission is to stop bullying in Canada and to promote safe and healthy relationships for all Canadian children and youth. Created and led by Scientific Co-Directors, Dr. Debra Pepler of York University and Dr. Wendy Craig of Queen’s University, this national network is the first of its kind in Canada, providing an unprecedented opportunity to change the way we understand and deal with bullying problems in this country.

Everyone who is involved in a child’s life, and every place where Canadian children and youth live, work and play, needs information about bullying problems and strategies to promote healthy relationships.

Before PREVNet, there were a number of different bullying prevention activities in use at local, provincial and national levels, all of which operated in isolation without an evidence-based national platform for coordination and implementation. As a national network, PREVNet is now bringing together researchers and national organizations to enhance awareness, build research capacity, assess bullying problems and promote evidence-based programs and effective policies across Canada.

PREVNet has three key messages:  Bullying is wrong and hurtful  Bullying is a relationships problem  Promoting relationships and eliminating violence are everybody’s responsibility

Ontario Education Pilot 2014-2015

21

Appendix: The Questionnaires at pre-post and follow-up

Ontario Education Pilot 2014-2015

22