Selecting and Training Evaluators

District Leader’s Guide

Selecting and Training Evaluators and Conducting Observations

Evaluating professional practice requires substantial investments in training. Š Copyright 2012 Colorado Legacy Foundation. All rights reserved.

District Leader’s Guide

Selecting and Training Evaluators and Conducting Observations Implementing Evaluation Systems: Learning from Pioneering Districts In an effort to help districts implement local educator evaluation systems in line with the requirements of SB191, the Colorado Legacy Foundation is providing a suite of resources to district and school leaders. CLF worked with three Colorado districts who recently revised their evaluation process. Although these districts implemented their new systems prior to passage of SB191, the process that each district went through is informative. We hope that highlighting these districts and the lessons they learned along the way will help other districts leverage their success and avoid re-living their most difficult challenges.

Three case studies provide the foundation for this work. District and school leaders can read the case studies for examples of how different districts have approached similar goals. A series of District Leader’s Guides builds on the case studies and provides more direct and specific guidance to district leaders as they move forward with implementation. The case studies and guides can be found on the Colorado Legacy Foundation website.

1

Brighton Case Study - Brighton educators and administration agreed that their evaluation system was a “dog and pony show.” In 2009 they revised their system, with terrific buy-in from the union, to more meaningfully support teachers.

2

Eagle Case Study - Eagle has spent nearly a decade developing their evaluation system and aligning it to instruction, assessment, and professional development.

3

Harrison Case Study - In 2007, Harrison hired a new superintendent who instituted a new evaluation system along with rigorous instructional supports, interim assessments, and a pay-for-performance system.

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

Who should use this Guide?

This guide has been developed from the lessons learned from Brighton, Eagle and Harrison. District leaders should use it in conjunction with the case studies to prepare the district for a new evaluation system. This guide is developed for district leaders who:

• Have read Preparing Your District to Implement a New Educator Evaluation System, Communicating Effectively with Stakeholders, Selecting and Using Multiple Measures to Evaluate Educators, and Determining Levels of Performance.

• Have established a district evaluation team which is looking for examples, lessons learned and implementation tips from Colorado school districts.

• Are preparing to select and train evaluators.

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

Introduction Evaluators should be trained before evaluating teachers and principals. When measuring quality standards on professional practice, it is critical that both the evaluators and the educators know exactly what different proficiency levels for each standard look like in practice. Observing and rating professional practice standards requires a substantial investment in training evaluators to ensure inter-rater reliability and for administrators to use the results to inform practice and professional learning needs.

Effective evaluator selection and training is essential for the integrity of the system to be maintained, ensuring that the resulting scores are fair and defensible. Evaluator training should include explicit decision rules and examples of evidence that would justify one performance rating over another. Ideally evaluators are trained with video performances of teachers where multiple evaluator trainees score the video and compare and calibrate ratings.

Additional Resources and Tools on Selecting and Training Evaluators Brief: Teacher Evaluators Training and Certification The goal of Teachscape’s practitioner series is to share the lessons from the Measures of Effective Teaching (MET) project, in order to support district leaders and state-level policymakers in the implementation of new, impactful teacher evaluation systems. Teacher Evaluator Training and Certification addresses the often overlooked and complex set of challenges involved in training and certifying the evaluators of classroom teaching.

Guide: Teacher Evaluation Models in Practice This guide, developed by the National Comprehensive Center for Teacher Quality, gives guidance to states and districts on implementing new evaluation systems. Read how districts in other states are training evaluators.

Š Copyright 2012 Colorado Legacy Foundation. All rights reserved.

State Board of Education Rules The State Board of Education rules require districts and BOCES to follow existing law regarding the selection and training of evaluators. The relevant sections of the rules from SB191 follow.

5.03 Training for Evaluators and Educators

and have completed a training on evaluation skills that has been approved by the Department. The Department shall develop a process for approving education and training programs for evaluators that is consistent with the approval process previously developed pursuant to section 22-9-108, C.R.S.

5.03 (A) School Districts and BOCES shall provide training to all evaluators and Educators to provide an understanding of their local evaluation system and to provide the skills and knowledge needed for implementation of the system.

5.03 (C) School Districts and BOCES are encouraged to provide training to Teachers, so that Teachers may conduct peer coaching observations in order to support other Teachers by providing actionable feedback on Professional Practice.

5.03 (B) As required by section 22-9-106 (4) (a), C.R.S., all performance evaluations must be conducted by an individual who has completed a training in evaluation skills that has been approved by the Department. Teachers may fill the role of an evaluator if they are a designee of an individual with a Principal or Administrator license

The rules go on to require that school districts and BOCES clearly communicate to teachers and principals the tools that will be used to measure their performance, how they will be weighted and aggregated to determine a final performance rating, and how student academic growth will be measured.

Guiding Questions for District Leaders Questions for the district evaluation team are: • Who will serve as evaluators? What criteria will be used to select them?

• Will training for evaluators provide opportunities to observe and rate professional practice and provide feedback to educators?

• Will evaluators be certified? If so, how often will they be retrained and recertified?

• How will inter-rater reliability (IRR) between evaluators be assured? If there is evidence that IRR has not been achieved, what will the district do?

• How will evaluators be freed up to conduct annual evaluations?

• Will an evaluator monitoring system be put into place? (This often requires a database.)

• Will evaluators be expected to do post-conferences to help teachers improve their practice?

• What will we do if observations on professional practice do not correlate with student academic achievement? © Copyright 2012 Colorado Legacy Foundation. All rights reserved.

Learning from Pioneering Districts

Although Brighton, Eagle and Harrison undertook reform of their evaluation systems prior to the passage of SB191, the process by which they developed the system and rolled it out district-wide yielded many lessons learned and implementation tips. District evaluation teams are encouraged to read the case studies and use the case study highlights to further inform their discussions.

The following sections include examples and lessons learned from the district case studies.

Evaluator Traning in Eagle County Schools Eagle County Schools uses an adaptation of Teacher Advancement Program’s evaluator training.

Teachers’ classroom instruction is evaluated several times each year by multiple trained evaluators, including administrators and master teachers, using rubrics based on research and effective instructional practices. Training for evaluators occurs annually in August during a four-day workshop where trainees view videos of teaching practice and rate these performances on the district’s quality standards using a rubric.

Evaluators are certified based on performance measures, and leadership teams regularly monitor the reliability and consistency of evaluations in their schools. Data and technology tools support this analysis. Evaluators are recertified every year to ensure inter-rater consistency.

Eagles County Schools use differentiated roles to conduct evaluations: • Mentor teachers spend 70 percent of their time in their classroom and 30 percent of their time coaching teachers as well as planning and leading cluster groups. They are contracted for 10 additional days and receive a $5,000 stipend for these responsibilities. There is one mentor teacher for every six to eight teachers in a school.

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

• Master teachers spend 30 percent of their time in their classroom and the remaining time coaching and evaluating teachers as well as overseeing cluster groups. They are contracted for 20 additional days and receive an $11,500 stipend for these responsibilities. There is a master teacher for about every 14 to 16 teachers in a school.

Teacher Evaluations: Every teacher is evaluated three times during the school year based on the rubric adopted by Eagle County School District’s Board of Education. A mentor teacher conducts a first observation in September or October. This is a low-stakes evaluation that focuses on coaching, feedback and continuous improvement.

For the next two evaluations, a master teacher conducts one announced observation, and the principal conducts one unannounced observation.

Scoring on the rubric identifies specific areas of improvement with detailed evidence from a teacher’s instruction and concrete examples to address these areas. This feedback helps teachers improve their instructional practices and informs the work of cluster groups.

Evaluator Traning in Eagle County Schools (Cont.)

Principal ratings are weighted at 60 percent, master teacher evaluations at 35 percent and teacher selfreflection at 5 percent.

The announced evaluation includes a pre-conference between the evaluator and the teacher; both evaluations include a post-conference lasting up to one hour and provide teachers with two kinds of explicit feedback in addition to formal scores – one “area for reinforcement” and one “area for refinement,” each tied to a specific indicator on the rubric. Evaluators and teachers analyze how a particular strength of the lesson contributed to student learning and discuss how the teacher can continue to build on that area of strength (reinforcement). They also analyze an element of the lesson that could have been improved, and discuss how the teacher can work to better contribute to student learning in that area (refinement).

There are two categories of teacher observations – “snapshot” indicators as seen in a single observation and “evidence over time” where ongoing evidence is collected. When evaluators share results with teachers, they adjust goals and professional learning opportunities to help students meet standards.

Evaluators use an extensive evaluation manual developed by the district’s professional development department. The manual is also a handbook for teachers describing exemplary instructional practices.

Evaluators are certified based on performance measures, and leadership teams regularly monitor the reliability and consistency of evaluations in their schools.

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

Implementation Tips and Lessons Learned

Pioneering districts suggested the following implementation tips and lessons learned: • Train evaluators to use a rubric to reliably rate teachers according to the district’s quality standards so they know what proficient and exemplary practice looks like in action. Videos of teachers scored by all evaluators help calibrate ratings and help them understand what to look for. The Teacher Advance Program produced videos and training materials for schools and districts adopting its program.

• Ensure inter-rater reliability. Ideally, all raters would give the same teacher observation the same score. This is important so teachers feel their rating was fair.

• Train evaluators to conduct in-depth post-conferences that can help teachers improve their effectiveness.

Feedback to teachers on their performance strengthens their instructional skills.

• Provide teachers with targeted follow up support based on their evaluations. Evaluation results can be used to guide school and district professional learning opportunities. Principals and district leaders analyzed evaluation results to identify professional learning needs – and then provided these opportunities either individually or through PLCs. • Provide opportunities for a second evaluation by a different evaluator if there is a disagreement about the rating.

This is relatively easy to do and settles disputes.

• Monitor evaluations for grade inflation. There should be a correlation between an educator’s rating on professional practice and student achievement results. If there isn’t a correlation, more emphasis on training evaluators, and a focus on the rigor of the quality standards, may be required.

• Conduct multiple evaluations. Several 10-15 minute “spot” evaluations conducted over a semester give teachers valuable feedback. They also signal that good instruction, every day, is important.

• The real value of the evaluation system is guiding professional learning and improving teachers’ instructional skills based on results. The primary focus is on improving instruction. However, the evaluation system in some pioneering districts could also be used to dismiss teachers that after significant support could not achieve proficiency. In pioneering districts it was not unusual for teachers to decide to change careers when they understood the definition of effectiveness and the quality standards they would need to meet.

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

Conclusion In summary, training evaluators to make accurate and consistent ratings is another critical part of an evaluation system. Scores must be fair and defensible. Training should include decision rules and examples of practice that meet each of the four performance levels (ineffective, partially effective, effective, and highly effective). Measuring professional practice requires substantial investments in training. One possibility is to credential evaluators based on the accuracy of their ratings and renew these credentials every year. Well-trained external evaluators that are certified is another possibility. The number and type of educator evaluations also impact how well the evaluation system serves its purposes. Multiple evaluations, with thoughtful feedback, will help improve educators’ skills and thus student achievement.

Acknowledgments Author: Jane Armstrong, JM Armstrong and Associates Research Support and Editing: Heather Chikoore, Colorado Legacy Foundation Research Support for the Case Studies: Ulcca Hansen, Colorado Legacy Foundation and Kristen Davidson, University of Colorado at Boulder Central office staff, school board members, principals, teachers, parents and community members participated in interviews that informed the development of this guide. Brighton, Eagle and Harrison School Districts opened their doors to researchers to describe what they were doing, how they were doing it, the challenges they faced and lessons they learned. Interviewees were both gracious and candid in their interviews. Without them, this knowledge could not be captured and shared. i Laura Goe, Lynn Holdhelde and Tricia Miller. A Practical Guide to Designing Comprehensive Teacher Evaluation Systems. (Washington, DC: National Comprehensive Center for Teacher Quality). May 2011. ii Johnathan Eckert. More Than Widgets. TAP: A Systemic Approach to Increased Teaching Effectiveness.

(Washington, DC: National Institute for Excellence in Teaching). December 2009.

Š Copyright 2012 Colorado Legacy Foundation. All rights reserved.