Managing Multi-State Assessment Consortia

Managing Multi-State Assessment Consortia Lessons From American Diploma Project Network Assessment Consortium

■ ■ ■ ■

Background: ADP Assessment Consortium Consortium Organization and Governance Potential Application Requirements: Demonstrating Capacity for Success Increasing Odds of Success

AMERICAN DIPLOMA PROJECT NETWORK

2

American Diploma Project Assessment Consortium ■

Initiated in 2005 by 9 states in Achieve-led ADP Network ●

consortium has grown to 15 states

■

State share broader college- and career-ready policy agenda

■

Started with Algebra II EOC exam, expanded to include Algebra I

■

Agreed-upon policy purposes ●

● ●

■

Improve curriculum and instruction and ensure consistency of content and rigor Enable states to compare results across and within states Indicate readiness for college credit bearing mathematics

Agreement to college-ready cut score set by Achieve

AMERICAN DIPLOMA PROJECT NETWORK

3

ADP Assessment Consortium In summer 2006, nine states issued RFP for the development of an Algebra II EOC Exam: •

•

Arkansas, Indiana, Kentucky, Maryland, Massachusetts, New Jersey, Ohio, Pennsylvania, Rhode Island Ohio acted as “lead state” in unprecedented multi-state procurement arrangement

WA MT

ND

VT

OR ID

SD

WI

WY

CA

NV

•

Pearson awarded contract in 2007 with addition of Algebra I in 2008 Since the consortium began, six additional states have joined: Arizona, Florida, Hawaii, Minnesota, North Carolina, and Washington.

AMERICAN DIPLOMA PROJECT NETWORK

PA IL

CO

AZ

OK

NM

MO

OH VA

KY

NJ DE MD DC

RI CT

NC

TN

AR

SC MS

AK

IN

WV

KS

MA

NY MI

IA

NE UT

TX

•

NH ME

MN

AL

GA

LA FL

HI

More states are expressing interest in the exam and joining the consortium…

4

Consortium Organization and Governance ■ ■ ■ ■ ■ ■

ADP Consortium Member States Ohio – lead state Coordination and Direction Team (CDT) Achieve Center for Assessment – technical advisor Pearson – test developer selected competitively

AMERICAN DIPLOMA PROJECT NETWORK

5

New State Members ■

Membership is extended to any ADP Network state wanting to join

■

New member states must: ● ● ● ●

Agree to terms of Partnership Agreement and MOU Agree to terms stated in the contract (but may add language) Participate in all CDT meetings, item development and review meetings Make firm, long range commitments ◆ Contractual ◆ Personnel ◆ Financial

6 AMERICAN DIPLOMA PROJECT NETWORK

6

Role of the Lead State: Ohio Appoint a procurement officer ■ Conduct procurement ■ Provide guidance to the CDT ■ Administer the contract ■

Manage contract amendments ● Incorporate new states into the contract ●

7 AMERICAN DIPLOMA PROJECT NETWORK

7

The Coordination and Direction Team

Includes assessment directors or other high-ranking policy-making officials from each member state in the consortium ■ Oversees production and implementation of the Algebra I & II program ■ Ensures that legal and policy needs of each state are addressed during team deliberations and decision making ■

8 AMERICAN DIPLOMA PROJECT NETWORK

8

The CDT and Decision Making ■

The CDT states have: Multiple representatives ● One vote ●

■

Process of consensus, collaboration, and compromise: Relating to the policies of each state ● Repeated as necessary ● Less discussion time needed as process matures ●

■

For contractual matters, decisions must be unanimous

9 AMERICAN DIPLOMA PROJECT NETWORK

9

Achieve’s Role ■

Convener and consensus builder ●

■

Brought chiefs, state content and assessment directors, governors & postsecondary leaders to the table

Consortium Management Communications with and among states ● Consensus building ● Project manager/oversee development contract ● Assembled additional expertise (technical, legal) ●

■

Content expertise ADP Benchmark standards ● Model course descriptions ●

AMERICAN DIPLOMA PROJECT NETWORK

10

Achieve’s Role ■

Promote Postsecondary participation Math faculty at 2- and 4-year institutions ● System leadership ● With key national higher education organizations ●

■

Define college readiness Conceptually ● Set achievement levels and cut scores ●

AMERICAN DIPLOMA PROJECT NETWORK

11

Key Lessons for States: Participating in a Consortium Takes Time and Communication ■

Active involvement of chief state school officer in start up phase is essential, so that state policy objectives and improvement strategy drive assessment design

■

Essential but difficult to sustain their engagement over time, as project moves to development and implementation phases

■

Governors and legislatures play critical roles in state testing policy; they must be engaged and informed along the way

■

The transaction costs for SEA staff are high compared with single-state assessment development

AMERICAN DIPLOMA PROJECT NETWORK

12

Key Lessons for States: It’s Not Just About the Test ■ ■

■ ■

Developing common summative assessments is difficult Summative assessments not enough – coherent approach that includes formative and interim assessments, curriculum and instructional materials, professional development is necessary to support instructional improvement That requires more ambitious effort, and broader and deeper consensus among participating states Common standards provides critical foundation for such an approach

AMERICAN DIPLOMA PROJECT NETWORK

13

Key Lessons for States: Shared Vision is Essential ■

States must have common standards and must hold firm to a shared vision: ●

of policy objectives of uses and purposes of assessments

●

of assessment system design

●

■

Scope of the vision will determine other practices must also be held in common. The more comprehensive and coherent the vision, the more issues states must address in common

■

Within a consortium, willingness to compromise on behalf of larger vision and shared goals is essential ●

●

To develop summative tests alone, states must still be willing to compromise on scores of details (e.g. testing windows, testing modes, test security, test guidelines, calculator policy for mathematics, accommodations, etc.)

Additional compromise will be needed in more ambitious approaches

AMERICAN DIPLOMA PROJECT NETWORK

14

Potential Application Requirements: Consortia Should be Asked to Demonstrate: High degree of consensus around: ■

Policy Objectives Instructional Improvement ● Vision of assessment system ●

■

Leadership Capacity in each state Support of Governor ● Active Leadership by and commitment of Chief State School Officer (not just state testing directors) and Postsecondary leaders (for college-readiness) ● Engagement of State Board of Education and Legislature as appropriate in each state ●

AMERICAN DIPLOMA PROJECT NETWORK

15

Potential Application Requirements: Consortia Should be Asked to Demonstrate: Technical capacity across the consortium, matched to assessment system design ■ Consortium Governance and Decision-Making Structure ■ Consortium Procurement Strategy ■ Roles and Responsibilities of Consortium members and partners, including supporting organizations ■

AMERICAN DIPLOMA PROJECT NETWORK

16

Increasing the Odds of Success Over Time Applications should include a sustainability plan that addresses: ■

Anticipated (and unanticipated) leadership turnover, especially for governors and chiefs

■

Strategies for building and sustaining support among key stakeholders ●

■

Annual assessment costs, especially if: ● ●

■

Ongoing communications with teachers, students and parents about purposes and uses of assessments Projected annual costs expected to exceed current costs Continuous development (e.g., new items) is part of design, beyond life of federal grant

Ongoing commitment of partner organizations through grant period

AMERICAN DIPLOMA PROJECT NETWORK

17

Managing Multi-State Assessment Consortia Lessons From American Diploma Project Network Assessment Consortium