Ensuring Data Integrity and Transparency

District Leader’s Guide

Ensuring Data Integrity and Transparency

Data infrastructure is an essential component of any educator evaluation system. Š Copyright 2012 Colorado Legacy Foundation. All rights reserved.

District Leader’s Guide

Ensuring Data Integrity and Transparency Implementing Evaluation Systems: Learning from Pioneering Districts In an effort to help districts implement local educator evaluation systems in line with the requirements of SB10-191, the Colorado Legacy Foundation is providing a suite of resources to district and school leaders. CLF worked with three Colorado districts who recently revised their evaluation process. Although these districts implemented their new systems prior to passage of SB191, the process that each district went through is informative; Highlighting these districts and the lessons they learned along the way will help other districts leverage their success and avoid re-living their most difficult challenges.

Three case studies provide the foundation for this work. District and school leaders can read the case studies to learn how different districts have approached similar goals. A series of District Leader’s Guides builds on the case studies and provides more direct and specific guidance to district leaders as they move forward with implementation.

1

Brighton Case Study - Brighton educators and administration agreed that their evaluation system was a “dog and pony show.” In 2009 they revised their system, with terrific buy-in from the union, to more meaningfully support teachers.

2

Eagle Case Study - Eagle has spent nearly a decade developing their evaluation system and aligning it to instruction, assessment, and professional development.

3

Harrison Case Study - In 2007, Harrison hired a new superintendent who instituted a new evaluation system along with rigorous instructional supports, interim assessments, and a pay-for-performance system.

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

Who should use this Guide?

This guide is developed for district leaders who: • Have read Preparing Your District to Implement a New Educator Evaluation System, Communicating Effectively with Stakeholders, Selecting and Using Multiple Measures to Evaluate Educators, Determining Levels of Performance, and Training and Selecting Evaluators; • Have established a district evaluation implementation team which is looking for examples, lessons learned and implementation tips from Colorado school districts; • Are prepared to explore, with the district implementation team and technology professionals in the district, the data infrastructure needed to effectively evaluate educators and provide them with targeted support.

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

Ensuring Data Integrity and Transparency

Introduction

Data infrastructure is an essential component of any educator evaluation system. Districts will need to have a system in place that allows the components of an effectiveness rating - student growth measures and educators’ professional practice scores - to be stored, analyzed, and combined. Before teacher performance data can be analyzed and the results shared with stakeholders, districts should thoughtfully decide upon procedures for data collection, validation, and warehousing; these decisions are therefore necessary to inform stakeholders, guide professional learning, and assess the measures used and the evaluation system as a whole. Furthermore, the data elements selected and used, and the decisions about data infrastructure, should be guided by the evaluation goals.i

In thinking about data management, there are several important considerations: 1. First, measures that assess student performance must be reliable and valid. Reliability refers to the consistency of the measure/instrument; in other words, it provides an estimate of how much or how little results would vary across multiple administrations of an instrument. Validity refers to the extent to which an instrument is measuring what it is supposed to measure.

2. Verifying and “scrubbing” existing data is important to ensure that data is as accurate as possible. Teachers need to have confidence that student data being attributed to them is an accurate representation of who they taught. This will facilitate understanding in teachers and principals on what data is being used and how.

3. Measures that provide real time feedback, are accessible and easily understood, and inform teacher practice are more likely to have an immediate impact on teaching and learning.ii 4. Data should also be timely and easy to use. For example, districts using benchmark or common assessments to guide student interventions need to share results with teachers in a timely manner so teachers can differentiate instruction, make decisions about student grouping, or re-teach key concepts.

The easiest and most effective way to ensure that teachers meaningfully engage with achievement data is to provide individual teachers with access to their students’ data using a password protected system.

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

State Board of Education Rules and CDE Guidance

Under SB10-191, CDE is responsible for monitoring the implementation of local evaluation systems to ensure districts are using multiple, fair, transparent, timely, rigorous and valid methods. It is equally important that educators receive adequate feedback and professional development support to provide them a meaningful opportunity to improve their effectiveness. While the rules that govern SB10-191 are not explicit about district requirements for data infrastructures, the reporting requirements for districts shed light on what may be needed (see Section 6.04).

The rules propose how each of the following objectives may be evaluated and the data that may be used. The rules for SB10-191 state that, at a minimum, monitoring efforts shall focus on: 1. Increasing the effectiveness of all educators.

2. Analyzing the correlation between student performance outcomes and the assignment of educators to performance evaluation ratings.

3. Analyzing the equitable distribution of effective and highly effective educators.

4. Analyzing the extent to which principals and teachers understand how they are being evaluated, what they need to do to improve, and how to access resources they need to support their professional development.

The rules also propose how each of the above objectives may be evaluated and the data that may be used. More information from CDE on how districts will demonstrate compliance with the law will be available prior to July 2013, the date by which districts will need to decide if they are using their own system or adopting the state model system.

It is equally important that educators receive adequate feedback and professional development support to provide them a meaningful opportunity to improve their effectiveness.

Š Copyright 2012 Colorado Legacy Foundation. All rights reserved.

Guiding Questions

Questions for the district implementation team are: • What information does CDE provide through SchoolView and other databases that can be used in informing the design of the evaluation system and reporting results?

a. How does this data answer the critical questions and serve the purposes of the evaluation system?

b. What additional data might be needed to meet the needs of the district?

• In what ways might the district’s information technology personnel be brought into the discussion?

• What are the district’s strengths and weaknesses with regard to technology and human capacity needed to collect and report data accurately?

a. What are some potential ways the district might build capacity regarding data collection,

analysis, and reporting?

b. What training will district personnel need to ensure accurate data collection and reporting?

c. What are barriers to building capacity within the district?

• Does the district have a data infrastructure that is easily accessible and used?

• Can the data infrastructure combine different databases such as school-, teacher-, and student-level demographic information; student achievement measures; human resources/personnel records; and financial expenditures?

• How will the district link educator evaluation data and professional learning needs?

• How will educator effectiveness results be shared with key stakeholders?

• How will results be made available for individual learning plans as well as school improvement plans?

Learning from Pioneering Districts Although Brighton, Eagle and Harrison undertook reform of their evaluation systems prior to the passage of SB191, the process by which they developed and implemented their systems provides many lessons and suggests several implementation tips.

The following sections provide examples and lessons learned from the district case studies.

For more detailed information about these lessons, district evaluation teams are encouraged to read the case studies and use the case study highlights to further inform their discussions.

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

Creating and Using a Data Infrastructure for Improvement— Eagle County Schools (ECS) District leaders, principals, and teachers have easy access to student achievement data reported by any number of subgroups and grade levels. For example, student-, classroom-, and school-level reports are available, and can be disaggregated by student demographics (like ELL status, gender, and IEP status), year-to-year cohorts, or specific academic standards.

The commercial data warehouse used by ECS is designed to report Colorado CSAP data and other state reporting requirements. Summative CSAP reports provide growth information grouped by a teacher-student link. The district’s NWEA data is also available. Educators have access to student reports on specific standards, test administrations, longitudinal analyses, and other queries.

Student reports can be generated that provide data to develop response to intervention (RTI) plans, individualized learning plans, and literacy plans. This process is streamlined and very accessible to teachers. As many as ten templates are available to report data aligned to specific learning plans and teachers only need to enter a student ID number and data are automatically pulled into the student report for each teacher’s use.

In the 2011-12 school year, the district has begun using a new data infrastructure that includes a data dashboard function. Instead of logging into multiple databases to access information, the new system will integrate data from multiple sources on a single data dashboard.

Also during school year 2011-12, rigorous curriculum design (curriculum units that align to the new state standards, instruction, and assessment) is being implemented. Common formative assessments are available for each unit of study which allows educators to assess student progress on priority standards during the school year. Formative assessment results are quickly available, encouraging teachers to modify or supplement instruction every four to eight weeks based on student assessment results.

To the left is an example of the data dashboard that teachers can access at any time throughout the school year.

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

Implementation Tips

Pioneering districts suggested the following implementation tips: • Educators must be trained in the use of the data system. Provide one district training tailored to people’s roles and the data they would need to use, rather than training everyone on the whole data system. Training is needed in order for educators to understand how to access data, understand what it means, and how to use data to improve instruction.

• Districts can consider developing standard data displays, also known as “dashboards.” Decisions about what to include in these data displays should be informed by the data most frequently requested by principals, teachers, and district leaders.

• Data systems are most useful when designed to be queried. Flexible data systems allow users to run their own analyses, and personalize their inquiry in ways that are most useful to the decisions they must make in their practice and professional development.

• Teachers should have 24/7 access to their students’ data and their own evaluation data. Timely access solves two issues: (a) teachers’ ability to make fast and informed decisions about their students’ instruction and (b) teachers’ adjustments to their practice and instructional strategies based on feedback.

• Specific software programs may need to be developed to weight data and calculate scores. Programs can also be developed to monitor inter-rater reliability of evaluators.

• Data must be easily retrievable, especially by teachers and principals.

• Educators who know how to use data for instructional improvement will make better use of the data system and have better results.

• District leaders should analyze school data and make decisions on how to better support schools, their leaders, and their teachers. Data that are analyzed quickly allow for targeted learning for teachers with specific needs.

Conclusion Data infrastructures are critical to storing data and providing the flexibility to generate reports that answer the large number of questions facing districts regarding accountability, policy, and practice. Data must be accurate, transparent, and easily understood. Training educators to use the data system is essential and should lead to data-informed decision-making, including targeted learning for individual students and professional learning for school staff.

© Copyright 2012 Colorado Legacy Foundation. All rights reserved.

Acknowledgements

Author: Jane Armstrong, JM Armstrong and Associates Research Support and Editing: Heather Chikoore, Colorado Legacy Foundation Research Support for the Case Studies: Ulcca Hansen, Colorado Legacy Foundation and Kristen Davidson, University of Colorado at Boulder Central office staff, school board members, principals, teachers, parents and community members participated in interviews that informed the development of this guide. Brighton, Eagle and Harrison School Districts opened their doors to researchers to describe what they were doing, how they were doing it, the challenges they faced and lessons they learned. Interviewees were both gracious and candid in their interviews. Without them, this knowledge could not be captured and shared. i Laura Goe, Lynn Holdhelde and Tricia Miller. A Practical Guide to Designing Comprehensive Teacher Evaluation Systems. (Washington, DC: National Comprehensive Center for Teacher Quality). May 2011. ii Laura Goe, Lynn Holdhelde and Tricia Miller. A Practical Guide to Designing Comprehensive Teacher Evaluation Systems. (Washington, DC: National Comprehensive Center for Teacher Quality). May 2011.

Š Copyright 2012 Colorado Legacy Foundation. All rights reserved.